Baby Steps to SOA – Step Ten: Riding the ESB
October 14, 2013 12 Comments
In the continuing Baby Steps to SOA series, we follow Doug and the IT team behind BuyMyWidget.com as they take steps to renovate their digital asset architecture. In this final stage, the team moves to using an Enterprise Service Bus (ESB) to handle the inter-application communication. This step allows for an increased ability to manage the disparate systems and scale to an enterprise level while also removing tight coupling between the applications.
The Evolutionary Roadmap
- Step One: Analyze and Plan
- Step Two: Measure It
- Step Three: Three Tiers for the Website
- Step Four: Single Sign-On
- Step Five: The Move to a CMS
- Step Six: Data Services
- Step Seven: Centralizing eCommerce
- Step Eight: Sharing the Business Tier
- Step Nine: Moving beyond the website
- Step Ten: Riding the ESB
Step Ten: Riding the Bus.
For a lot of organizations, there may not be a need to scale up to an Enterprise Service Bus (ESB). You may only have a few services being managed, or the cost of the solution simply doesn’t factor well into the budget. Moving to an ESB to handle the routing of requests for services on the network is not for everyone, but offers a solution to some common problems.
For example: The team has rolled out a dozen or so data and business services, all being consumed by several applications, and then suddenly there is a need to move one of the services to a new domain. The path to this service was probably in a config setting, or database setting, or hard-coded into the applications (please don’t). Now the team needs to update all of the applications that are consuming this single service just because of a simple domain change.
The goal is to remove the knowledge of the where and how from the applications themselves, and allow the ESB to handle these instead. This allows for a centralized management of the routing, among other things.
While direct connections to the services will offer an increased performance by cutting out the overhead of a middleware layer, an ESB can help the team achieve several key goals:
- Abstracted Routing: ESB handling of communication and message routing allows for individual applications to be able to be separated from the knowledge of the how and where of a service. This increases deployment flexibility.
- Message Queueing: While updating a service, or performing maintenance on it, messages to this service can be queued up and delivered when the service becomes available again.
- Prioritization: Not all services are created equally. In order to make efficient use of network bandwidth, the ESB can prioritize which messages should be handled in which order to ensure low-priority calls that are high in volume do not interfere with high priority network requests.
- Monitoring: Health of the components in the architecture can be monitored by the ESB software and logged to allow for teams to have a central view into the network health.
- Agnostic: By performing all of the messaging between applications, the JAVA application doesn’t need to know about the CORBA implementation on the other end, or the .NET web service living on a Windows server. Each application only needs to know about how to talk to the ESB, and the ESB needs to know how to transmit those messages.
In Our Scenario
Lynn has been dying to get a chance to get an ESB implemented into the architecture at BuyMyWidget. Previously, when there were only a few applications and little direct interoperation, the business case for such a move was not one Doug could get behind. Now, with dozens of applications and data services interoperating in multiple environments, the business case is clear.
The first step is for Doug and Lynn to choose their ESB solution. The current architecture is all running on Windows Server 2008 R2 64-bit operating systems, and the development team has built all applications and service communication points using Windows Communication Foundation (WCF) on .NET 4.0. If they stick with their current OS requirements, that will limit their vendors to Microsoft, Talend, and TIBCO.
Choosing to go the Microsoft BizTalk route would allow Doug’s team to continue their Microsoft trend for their solutions. This will probably cost them about $2500 per core on the license, based on their current needs. TIBCO is a well-proven company and ActiveMatrix is a solid solution. The price can be high, and Doug’s team will likely not use most of the features. Talend is an up-and-comer that is beginning to challenge some of the more established players (see Gartner magic quadrants for 2012 and 2013). It also is open source and provides a subscription option.
Lynn would like to go the open source route, while Doug feels more comfortable with the Microsoft product given past investment in Microsoft solutions. Given the cost is within his budget for new software, Doug goes with the Microsoft product over the open source Talend solution and the team gets to work on BizTalk.
With the purchasing decision out of the way, implementation kicks off. Given the standardized WCF messaging already in play, this step is rather straight-forward. Rather than reproducing it here, I’ll point to a great series currently running about how to consume WCF with BizTalk: http://jamescorbould.wordpress.com/2013/09/23/biztalk-and-wcf-consuming-a-wcf-service-part-1-a-look-at-the-service-to-be-consumed/
Once the development work has been done, deployment begins. The ESB application needs to be put into place and configured so that all endpoints are reachable. Lynn and the deployment manager then need to begin rolling out the updates to each application to begin consuming the services via the ESB. To avoid risk, Lynn decides to start with a single application, the events website, and roll out from there. After updating the single application and verifying that all services are being consumed correctly via the ESB, the team can continue working through each application until they are all connected to each other via the ESB.
With everything in place, Doug just needs the dashboards up to monitor the system health. Doing some research on TechNet, Doug finds that BizTalk360 has just come out with a new version and has some great monitoring and support capabilities. The team goes about downloading the trial to figure out how to get it working with the ESB. It seems great, but the cost seems to be quite high for their intended usage. Doug’s team decides to continue to review some of the other options and roll out a basic custom dashboard in the meantime to fill the gap until they find a more robust tool that meets their price range.
How much will this step cost?
Buying off-the-shelf software from Oracle, Microsoft, or IBM may cost you anywhere from $10K to $100K USD, with support in addition to the license cost. Free open-source solutions are also available if you are looking to avoid the license cost. I would recommend reading the Wikipedia comparison of business integration software article which provides a great summary for quick comparison of a variety of integration software.
If the team additionally decides to purchase monitoring tools, this can run for several thousand dollars of licensing costs as well. Each vendor will have its own licensing structure, so make sure to spend the time to evaluate all of them, as most of the “Enterprise” licensing packages are probably overkill for small-to-medium sized businesses.
In addition to the licensing and support costs, you will also need to plan for a senior developer, a senior IT resource, and a project management resource to organize the implementation and refactoring of existing applications and services to take advantage of the new ESB.
I would additionally recommend having a business analyst put together the business case for the ESB implementation and then define some new requirements to take advantage of the ESB capabilities. In particular, the surfacing of dashboards from the monitoring and logging capabilities of the ESB is usually a good start for showing business stakeholders the benefits of the project. Message queues can also be argued to show that valuable eCommerce transactions will not be lost during maintenance or service application outages.
After several months, this series will come to a close with a final retrospective.