Is BizTalk Server an ESB?

Is BizTalk Server Microsoft’s version of the Enterprise Service Bus? Matt Nicholson discusses this together with human workflow and long-running transactions with Scott Woodgate from Microsoft’s Business Process and Integration team.

Originally published on DNJ Online, Dec 2005

Matt: Some people have said that BizTalk Server is Microsoft’s Enterprise Service Bus (ESB). Would you agree with that?

Scott: We can definitely address that issue! But first, let’s rewind. With a Service Oriented Architecture (SOA), creating the services is important, and on our platform, using ASP.NET and Windows Communication Foundation (formerly ‘Indigo’) to create those services is a fine thing. The capabilities of such services, and how you talk to them in a point-to-point manner, will continue to improve through additional layers supporting security, transactions, reliability and so on. WCF is all about creating services and then accessing them point-to-point in a secure, reliable and transactive manner.

However in any SOA the notion of composing a multiple of these services into a broader application, rather than just talking point-to-point, is important. This is where the business process support of BizTalk Server comes in.

The third aspect of SOA is the notion that one service shouldn’t know about another service. All a service needs to know is that it’s sending a message such as, say, a purchase order. Something else ‘in the middle’ can figure out where to deliver that message. It may be that nobody is interested in the purchase order, but more than likely there’s another service, or multiple
services, that will receive it. However as far as the source application or service is concerned, it doesn’t care. That’s important because if I need to go from having one target to two, or replace one with another, then I want to be able to do so without affecting the source. In BizTalk parlance that ‘thing in the middle’ is the publish and subscribe messaging infrastructure.

So putting that together, business process management and message-based solutions become more relevant in an SOA – more significant than in other architectures. The interesting discussion then is around the notion of an ESB, and to have that discussion I need to set some context for my view of ESB, based on the reading I’ve done around the subject.

It soon becomes apparent that the definition of ESB is a little unclear. If you listen to Sonic Software, who initially invented the term in conjunction with Gartner in 2002, they have one definition. Burton Group also has a definition, and a number of other people have further definitions.

If I compress these definitions and look for similarities, then one feature of an ESB is routing support. As I described earlier, a service does not need to know its target service so, in the terminology of ESB, it is the bus that handles the routing. So routing is one of the key things. However we’ve been doing routing since BizTalk Server 2000, and through Web services since BizTalk Server 2002.

On top of routing is Web service support, and the industry generally agrees that supporting Web
services is important. Most of these ESB definitions also include some level of management. Some of them bring in the notion of a ‘bus architecture’, and what that means is that you install a piece of software on all the machines that are taking part in the infrastructure, and this software allows them to talk to a centralised network. Indeed one of the challenges for ESB is that enterprises find the need to install software on every single node of the infrastructure very invasive.

So those are the three primary characteristics of ESB. What’s interesting about ESB is that it is different to most software evolution. Most software evolution is about increased functionality at lower prices. When BizTalk came into the marketplace in 2000 it provided similar features to
products that were millions of dollars, but at a dramatically lower price. As a result we’ve driven down the price that the customer is prepared to pay for those features.

However ESB is not about providing the same features for less, or providing more features for the same price. ESB is about providing less features for a lower price than that of a full-blown Java-based server. Some of the features that ESBs don’t provide include business process orchestration, business rules, business activity monitoring and so on – features that BizTalk Server and some of our more expensive competitors do have.

This is a key point. IBM and BEA and a variety of folks over in Java-land have very expensive but fully featured integration servers. Sonic Software saw an opportunity to take a subset of that functionality and offer it at a lower price point. This was a very sensible play in the Java space, because the price that customers were willing to pay for simple messaging scenarios was
lower than what the vendors were offering. However in the Microsoft space it doesn’t make any sense, as BizTalk Server provides a superset of the features of an ESB at the same price or less.

So when a customer comes to Microsoft and says, “Do you have an ESB?” I say, “No, because I don’t understand why you would just want the routing, Web service support and management.” The customer may respond, “Well, in my first project, all I want to do is route messages backwards and forwards.” And I say, “Well, you can do that just fine with BizTalk Server, and in the sense that you pay per CPU and that CPU is only doing messaging, then that’s all you’r paying for.”

The difference is that if they want to add business process orchestration in their next project, and
most of our customers are adding business process and rules and business activity monitoring by their second project, if not their first, then if they’ve bought an ESB they have to go and find a new product with a new architecture, and re-learn that set of skills. With BizTalk Server you take the same product and you start using those features and it’s all tightly integrated.

So from an ESB perspective, BizTalk Server is more fully featured than an ESB, which means that it’s not an ESB. On the other hand, customers in the marketplace are being educated that they need an ESB, so if a customer comes to me and says, “I need an ESB, can Microsoft help?” I will say, “Absolutely! We do not have an ESB, but we have a superset of that functionality in the combination of BizTalk Server and our Web services stack.” And of course combining BizTalk
Server with WCF, when that becomes the full Web services stack, will be very compelling.

So ESB has come about through a price point issue. I think that as the Java server integration market continues to commoditise, the ESB space will evaporate.

Mapping one message schema onto another in BizTalk Server 2004.

Human workflow

Matt: I understand that BizTalk Server 2006 does not support human workflow. Could you define what you mean by ‘human workflow’ in this context?

Scott: When a customer comes to me and says, “Do you support human workflow in BizTalk Server?” The first thing I ask is, “Can you explain what you mean?” Some customers then say, “I have this ordering process, and as part of this ordering process I need to get this person, called Scott, to approve or deny a purchase order. It’s a single step, and it involves Scott specifically, and then subsequently the process continues, sending messages backwards and forwards.” My response is, “Great, with BizTalk Server, something like Infopath and Sharepoint, you can build a perfect solution for that particular problem.” When Office ‘12’ comes along, together with Windows Workflow Foundation running on Sharepoint, it gets even more powerful.

On the other hand, a customer might come to me and say, “What I really need is a designer that enables my power users to manipulate people in my organisation, pulling information from an Active Directory source to design a flow between humans that may involve hierarchical escalations, such that if a purchase order needs approving and I’m not available then send it to my manager, and so on.” BizTalk as it stands does not have such a designer, doesn’t suppor hierarchical escalation, and can’t pull role information from Active Directory. However all of those features are available from our partners.

The business process in BizTalk Server is targeted at the developer, and actually runs inside Visual Studio .NET. We do have a Visio-based designer that’s targeted at business users, but it’s designed to give the developer a starting point, rather than as a human workflow designer.

The second important feature here is the support for roles so you can support scenarios such as, “I’m Scott and I’m in the role of the developer. When I submit an approval form I want to submit it to the developer manager and this week Matt’s in that role, but when Matt gets promoted to lead product unit manager then somebody else will take that role, or there might be two people in that role and either of them can do it.” BizTalk Server does not natively support the notion of roles, so if roles are important you need to write code.

Matt: That would suggest what is needed is integration with Active Directory, where you’ve got the roles defined.

Scott: Yes, and there are ways to support that level of functionality by building on top of BizTalk Server. However it is probably far more cost effective to purchase it from one of our three key partners, namely SourceCode, Ultimus and Teamplate, who all specialise in that scenario, providing a high-level designer to drag-n-drop people, do hierarchical escalation, and pull roles from Active Directory. In each case those designers plug in to BizTalk Server to handle the system-to-system workflow and to do traditional EAI and B2B. Those tools are appropriate if you’re looking to combine human workflow with system workflow.

With regards to BizTalk Server I would say that we continue to take feedback from customers on human workflow, and we think it’s an important problem strategically for us to investigate. We also want to ensure that, if you’re investing in human workflow today through one of our partners, you have a path forward with that partner.

Handling transactions

Matt: Could you take us through the way in which BizTalk Server handles transactions? Because of course there is a distinction between ATOMIC transactions, where you’re trying to commit the whole thing in as short a time as possible, and more long-term transactions, such as those involved in buying a house.

Scott: Business process is effectively a state engine in the sense that it manages state from the very beginning of the process, for example when I make an offer on a house, to when I finally move in. Some of that state is very short-running, for example where I need a debit/credit scenario with my bank account so the money to buy the house can be transferred. We support ATOMIC transactions in BizTalk Server using the DTC [Distributed Transaction Coordinator]. What BizTalk does is give you the ability to graphically draw that transaction around a set of shapes as part of a business process, as opposed to code it.

On the other hand there’s long-running transactions. A long-running transaction can be dragged across any logic, whether it is transactional (in the ATOMIC sense) or non-transactional. Long-running transactions can’t satisfy the Isolation requirement [the ‘I’ in ‘ATOMIC’], which means they don’t do record locking. However they do give you the ability to manage a process that starts on Day 1, but an error occurs 15 days later because you’re waiting for a response from your mortgage broker, and you better get that response or you aren’t going to be able to close the deal.

In these scenarios a business exception occurs (as opposed to a system.transaction type of exception) and that can be bubbled up through the business process inside BizTalk Server, where a compensation handler may pick it up. However it’s different to traditional short-running transactions where you get automatic rollback. If you’ve already sent an email to your mortgage broker you can’t ‘roll it back’ because email doesn’t work that way.

The key difference is that there is no isolation, because clearly if you had 10,000 business processes all running for a month then there’d be record locking in the database left, right and centre, and it wouldn’t be practical to tie up all those resources like that. However you can have all the other characteristics of the transaction, in terms of Acidity and so on, and then use compensation handlers to either manually undo the logic by looking at the state that’s stored in the process and see if you can undo that update; or what’s more common in these scenarios, re-send the email and then notify somebody to call that person on the phone so that you can handle the business exception gracefully.