On Message Based Architecture

When do you choose to go with a message orchestrator like BizTalk? How do I know when and how best to make use of BizTalk?

There seems to a trend of jumping on the BizTalk bandwagon lately. There also seems to be a lot of misunderstanding about how best to leverage BizTalk and/or even when to use it. I would probably categorize myself as one who has some basic understanding of it's purpose, but would not be in a position to speak with authority on the subject.

That being said, I would like to take a step back and examine some common patterns that get people thinking that they need BizTalk to solve their business problems.

First, and the most obvious, is the desire to abstract out of existing business systems the hard coded workflow that burdens the business with only being able to change as fast as changes can be made and tested in these systems. If the workflow is abstracted into some meta data format and then the business systems can process it's messaging through BizTalk then the idea is the meta data about the workflow can quickly and readily be changed to change how the business systems work to better and more efficiently align them with the business.

I can see the definitely long term value with this scenario. It would likely mean significant rework to the existing business systems, possibly pulling it so far apart that ground up rewrites would yield a better product. The biggest change likely being flipping all that internal message communication that previously happened in-process inside out so that the different areas or modules of the system became their own autonomous services and the orchestrator was responsible for routing the messages to the proper service for processing.

A secondary advantage to such a system is that there may be ways to parallelize a lot of the communication if architected correctly, yielding many more calls that were asynchronous allowing some really cool things like throttling and work distribution over time, taking advantage of idle times of the day for servers and reducing the peak times that might stress the servers, balancing processing out allowing for smoother operation and increased workloads.

There is another pattern that I have witnessed in passing that I think comes from more of a lack of understanding the value of orchestration, that is abstract orchestration. Again, it could just be me that is not understanding, but I want to think out loud for a minute.

There is a certain abstract pattern found in many enterprise applications where there are many processes that are autonomous from each other and can be automated. They don't have to run in serial and multiple requests of the same type can even be threaded out to run in parallel. Think data collection processes that are either querying different databases or other other types of data sources (e.g. web services, etc.). The end goal will be to take all these pieces of data and homogenize them into a single data store for normalization and either user selection of "best fit" data or some algorithmic business rule implementation to automate the process based on different abstract criteria.

In an organization that is already beginning to sell and pursue the typical pattern outlined previously in this post, it becomes a natural inclination to through every solution into the workflow/orchestration cloud to be the magic bullet. Just because an application contains elements of orchestration -- and what application doesn't -- doesn't mean that the architecture of that solution be such that it requires abstracting that orchestration out of code (think if/else blocks or Strategy Pattern implementations that send different messages or invoke different modules based on some condition) and into an enterprise orchestration tool like BizTalk.

Will the operations you are implementing on that sub-application level really change that often to warrant the up front design time to abstract away the orchestration? Making use of Command Pattern Based Messages and MSMQ can offer a pretty powerful message based solution alleviating one from having to get more complicated that necessary. Sure this article was published over two years ago, but sometimes the good classic solutions die hard.

Anyone have differing thoughts or opinions on this topic? I'd love to hear and learn.

Tags: command pattern, messaging, msmq, biztalk, orchestration, message based architecture