In the most pure and simple service oriented scenario, a single consumer interacts directly with a single service provider.

For more complex situations, intermediaries are used to provide additional capabilities:

• Security
• Transactions
• Routing
• Data Mapping
• Interface Mapping
• Reliability
• Protocol Translation
• And etc…

What exactly is an intermediary?

An intermediary is a piece of hardware or software that bridges the gap between a service provider and service consumer to provide one or more value-added capabilities (see bulleted list above). There are three primary options for intermediaries (and a host of variations on these three themes):

• Service Intermediary - Another service acts as the intermediary, providing required enterprise capabilities
• Centralized Bus - Service calls are brokered through a common, centralized focal point
• De-centralized Framework - Enterprise capabilities (security, mapping, etc.) are pushed out to the endpoints

Comparing the three options

As with so many things, there is no clear-cut winner across these three strategies, it all depends upon what you are trying to accomplish.

Option 1: Service Intermediary 

Advantages: Extremely lightweight solution.

Disadvantages: Not as robust or flexible as other approaches and requires more testing to verify proper configuration.

Option 2: Centralized Bus

Advantages: Offers lots of “out-of-the-box” capabilities that are pre-configured and available for use.  Provides a single point of control and service access management.

Disadvantages: May be over-kill in many situations, offering far more functionality and options that are needed.

Option 3: De-centralized Framework

Advantages: Provides the greatest flexibility and configurability for managing endpoints and enforcing policies.

Disadvantages: Requires a more disciplined governance process to ensure that each service is managed properly.

Working with intermediaries

Services do not generally need intermediary support except to provide value-added capabilities such as security, transactions, routing, data/interface mapping, guaranteed delivery, protocol translation, etc.  It is important that organizations carefully evaluate their requirements and select architectural strategies that meet their needs.  There are a variety of valid topologies that could be employed, depending upon your requirements, including services as intermediaries, deployment of a centralized bus (either physical or logical), and the utilization of a decentralized, policy-driven framework (“liberate the endpoints”).

Primitive man and woman were forced to adapt to their environment to survive. Animal hides for clothes, crude weapons and tools made out of stone or bone, and roots or berries for food (perhaps sabertooth steak or terradactyl ribs if company is coming over). Eventually, some nearby tribe discovers fire, but reports of fire-related injuries and property destruction lead our cave dwellers to avoid this new magic. These primitive humans are doing fine with their current capabilities and they are able to meet their basic needs without dabbling with such things. Over time the nearby tribe’s use of fire expands and one very enterprising woman invents pit fired pottery. The more primitive people have heard and seen the potential of this new breakthrough and are interested in using it. Alas, the obstacles to adopting this innovation are not trivial. They must learn how to create fire, handle fire cautiously, locate and stockpile flint and tender, develop fireproof tools, create fire pits, learn to craft with clay and finally develop a process to fire the clay pots successfully. Lack of education, skills, infrastructure, resources, and processes hinder their ability to progress. Perhaps the greatest barrier for these primitive people is fear of the unknown and risk of failure. Thousands of years later, we are faced with the same technology adoption challenges — education, skills, infrastructure, resources, processes, and fear.

Modernizing legacy information systems is not unlike the modernization efforts that have occurred for thousands of years. In fact, legacy or heritage systems are often referred to as ‘dinosaurs’. Correspondingly, the same types of obstacles must be overcome:

• Education — There is a mindset shift that must take place to understand the service oriented way of thinking. At first glance, services are just fancy objects. The reality is that service orientation requires a much broader, end-to-end view of the enterprise, complete with process-centric alignment, layered architecture, contracted interfaces, standards-based connectivity, and full life cycle governance. I have blogged about the alignment of service orientation and the unique qualities of SOA before. Also, David Ing has some interesting thoughts on service orientation.
• Skills — A well documented skills gap exists around service orientation. Joe McKendrick sites it as one of the two things killing SOA in 2008 and the guys over at ZapThink have been alerting the industry to the dangerous SOA skills gap for quite some time. Why is this? Service orientation skills cannot be acquired by attending a conference or reading a book (although I do have a book that I HIGHLY recommend). Academic learning as well as hands-on mentoring is required. Moreover, there are nuances to effective service orientation, service design, process alignment, and enterprise governance that require time and experience to develop. There are new technologies, design patterns, techniques, and methodologies that must be introduced and ultimately absorbed. This will require instructor-led training, research, hands-on mentoring, and practical experience to develop proficiency in these areas.
• Infrastructure — Contrary to popular belief, a service oriented infrastructure cannot be achieved by purchasing an Enterprise Service Bus (ESB) from a vendor and adding water. There are, in fact, a variety of infrastructure elements (service registry/repository, governance suite, business process engine, policy manager, policy enforcer, and etc.). There is a great little SOA infrastructure white paper that a consortium of vendors put together a while back. Additionally, be sure to avoid the trap of assuming that you need to have your SOA infrastructure fully mature on day one. Eric Newcomer has a great post from a couple years back regarding an incremental adoption of SOA infrastructure.
• Resources – There are several ways to slice the resource issue. For effective adoption of SOA, you will need a pool of human resources (with appropriate education and skills), technical resources (infrastructure, knowledge management and collaboration tools, as well as design and development tools), and you will need expert resources (developed in-house, or brought in from an outside source initially).
• Processes — All the education, skills, infrastructure, and resources in the world won’t amount to a hill of Java beans without effective processes for governing the adoption of service orientation. You will need processes for service selection, service design, quality assurance and testing, policy enforcement, and runtime service management. Effective governance can make or break your adoption of SOA. I have blogged on the importance of service oriented governance numerous times.
• Fear – People fear change. Service orientation appears threatening to many people at first due to the changes required in adopting it. New patterns of thinking are required around how to solve customer problems. Skills must be adapted, other skills developed from scratch. New infrastructure and resource pools must be created and processes must be adapted and new processes created. These changes can be intimidating. A considerable degree of people change management must be accepted as a part of a successful transition toward service orientation.

Service orientation adoption is challenging in its own right. It becomes increasingly challenging when you factor in the dinosaurs that must be modernized as a part of the adoption process. Immediately, questions begin to arise:

• What systems and/or business processes should be modernized?
• Should we migrate, expose, or leverage our existing legacy assets?
• Should adoption occur within a particular line of business or across the whole of the enterprise?
• How will modernization impact existing teams and project roles?

These questions and many more must be explored as part of the adoption of service orientation. I explored several of these topics in a recent public webinar - Enterprise Modernization and SOA Concepts (pdf).

Another resource you might consider is a new course that Web Age has been offering since April of this year. It is a two-day workshop that explores the modernization of legacy applications, SOA concepts, modernization challenges and risks, and various strategies for modernizing legacy systems. It is aimed at team leads, architects, managers, and legacy application support personnel. For more details, check out the course details - WA1657 Application Modernization and SOA Concepts.

Adopting service orientation and “moving out of the Stone Age” is not a simple task. It requires intentional allocation of time, energy, and resources. It cannot and should not be approached in an ad-hoc fashion. Technology adoption has always been challenging, just like when ancient man and woman adopted fire. When it comes to service orientation, be sure that you work through the necessary steps of adopting education, skills, infrastructure, resources, processes, and addressing the fear factor. Otherwise, you just might get burned.

I’ve been on the road a lot lately, working with various clients, writing courseware, writing articles, and basically doing everything EXCEPT for blogging.

Here’s some of what I’ve been up to:

• Article: SOA Governance: Start Small and Build Incrementally (published March 6th by SYS-CON)
  
• Presentation: SOA Adoption Planning (1-hour seminar I delivered in February)
• Presentation: Delivered a virtual presentation for the Dallas SOA Users Group. I had a client engagement in Calgary, so we tried out a virtual delivery and it was quite successful.
  
• Book: Service Oriented Architecture: A Field Guide for Executives (August 2008, Wiley Press)
• Courses: Updates to various courses plus new courses coming soon for Project Managers, Mainframe and Legacy Personnel, and some Best Practices content. For more information, check out Web Age’s SOA curriculum.
• Web-based SOA Training: I’ve been working on a new offering at Web Age in which some of our content is made available as recorded, flash-based audio/visual presentations that can be hosted internally within an organization’s intranet. More details to follow.

I’m going to try and be more diligent about blogging in April. I especially hope to share insights and perspectives that I glean from the IBM SOA IMPACT conference in Vegas. I’ll be attending that conference in early April and conducting a Birds of a Feather session on governance.

A rose by any other name would still smell sweet, just as a strategic corporate initiative is no more or less successful due to its use of certain en-vogue terms such as Enterprise Architecture (EA), Business Process Management (BPM), and Service Oriented Architecture (SOA). I was struck by the realities of the semantic terminology game that we play while attending the Open Group’s 17th Annual Enterprise Architecture Practitioners Conference in San Francisco last week.

There was a general consensus at the conference that there are common themes running throughout enterprises that have various labels attached. Those themes include the following:

• Business and IT need tighter alignment
• Visibility between business and IT should be increased
• Governance is needed at various levels of the organization to reduce risk and protect ROI
• Business processes and technology solutions should be more flexible and better able to adapt to changing demands
• Business processes and technology solutions should be designed intentionally and with sufficient rigor

The labels attached to these trends vary from enterprise to enterprise. Some drive toward one or more of these goals under the EA umbrella, others under the BPM label, and still others are talking about SOA. Some organizations use two or even all three labels. Regardless of the attached label, it is a strategic initiative at the business unit or enterprise level that aims to improve business predictability and squeeze more value out of technology investments. This become readily apparent as the SOA Governance panel (other panelists included Michael Nassar of IBM, Stephen Bennett of BEA, and Mats Gejnevall of Capgemini) that I participated in last Tuesday was quickly challenged by the audience to delineate between governing SOA and governing EA. The line between these disciplines is quickly blurring within many enterprises.

So what’s the take-away for us? We need to look past the labels that are applied and take a more business-focused and goal-oriented approach to education, mentoring, and ultimately solution development. We need to probe more intently with our clients to discover their strategic direction, business drivers, and objectives for one-year, two-year, and five-year timeframes. We may have the perfect service offering for them, but because it is labeled as SOA, BPM, or EA, it may not jive. I have several examples of this:

• Back in November, I wrote about Synovus, a mid-sized bank in the southeastern US, that spoke at SOAWorld in San Francisco. They were a non-traditional case study in that they did not set out to adopt SOA. They just knew that they had various business problems that needed to be solved. They set out to solve those problems and in the process created a SOA. If someone had approached them a year or two ago with an offer to enable their enterprise through SOA, they would have dismissed it because they were not doing ‘SOA’, at least not in name.
• Last year, a US Department of Defense (DOD) agency participated in one of my SOA governance workshops. Several of the students made the observation that although we had originally set out to talk about SOA governance, the vast majority of the concepts, principals and best practices that we discussed were applicable to EA. The mentoring engagement quickly broadened in scope and we began speaking in terms of governing the enterprise and aligning with business strategy. The principals, methodologies, and best practices all flowed very nicely into this broader scope. The ‘SOA’ label that we had begun with was immaterial to the objective — better governance of enterprise assets and business processes.
• Throughout last year Web Age Solutions worked with various clients that wanted to service orient their enterprise assets and orchestrate those assets as a part of executable business processes. Some of them called that SOA, others called that BPM. Some wanted us to describe SOA in the context of BPM and others wanted us to introduce BPM in the context of SOA. At the end of the day, the solution was the same. It was the same standards, same tools, and the training was about 90% the same. The difference was found in the labels attached and the emphasis upon certain key concepts.

It is imperative that we understand our client’s needs, perspectives, and bias with respect to trends and labels. At the end of the day, we are enabling them to solve business problems, not in order to check a box next to an acronym.

I spent much of the second day talking with attendees, speakers, and SYS-CON staff about SOA, virtualization, and the conference in general. Overall, there was a general sentiment that the conference was a success and had some valuable content. One presentation from the day that really stood out was a case study from the banking industry — Delivering Big Bank Solutions with Community Banking Intimacy. The bank is Synovus, a regional bank in the southeastern part of the US with about 33 billion in assets spread across multiple lines of business (banking, financial management, brokerage, insurance, and more).

In a thick southern, ‘good ole boy’ style delivery, the speakers from Synovus described their approach as ‘Blue Collar SOA’, explaining: “…it may not be perty, but it works and we get a lot of value out of it.” The Synovus representatives went on to explain that while they did not set out to deploy a SOA solution, their business requirements and technology objectives led them in that direction. They went on to describe several keys to successful SOA adoption, including the following:

• Define and document a set of guiding principles
• Define and document a reference architecture that is consistent with those principles
• Identify standards, technologies, and products that can enable that architecture and communicate and manage this through governance
• Identify a business visible pilot project to convert the strategy and reference architecture into a real implementation
• Define and enforce the implementation strategy through governance processes
• Employ governance toolsets to validate services against guiding principles as well as policies and SLAs

The results of their SOA adoption as well as the importance of service orchestration is outlined in a white paper from Active Endpoints (their service orchestration partner).

The presenters concluded by identifying a huge list of lessons learned. Three things really stood out to me on that list:

1. Identify and document standards and best practices and enforce them through your governance processes
2. SOA changes your team dynamics — you will need more documenters than developers
3. Education is key because SOA is hard

That third one was especially of interest to me given Web Age’s focus upon SOA education. In the Q&A portion I asked for the folks from Synovus to elaborate on the education topic. They were eager to explain that they had discovered a huge gap between legacy skill sets and what it takes to be successful with SOA. They had to figure out how to translate those skills, bring people up to speed on new methodologies and terminologies and get people to think differently about how to solve problems. Synovus also discovered that considerable education and mentoring was necessary to help the leadership that direct their lines of business to “think outside their silo”. Amen!

At Web Age we have seen the same sorts of challenges and often discover that teaching the technologies is fairly simple. Teaching the methodologies is more difficult. Teaching the new mindset (e.g. ‘thinking outside the silo’) and providing business users with new tools and techniques for solving problems is even more difficult. So what is the most challenging? Getting organizations to actually change once they are educated. The technology side of SOA is relatively easy. The human side is where things get tricky.

Overall, the conference was a great experience. If you get a chance to catch the East coast or West coast shows next year, definitely go. I hope to see you there!

The first day of SOA World 2007 - West went very well. Miko Matsumura with Software AG / webMethods kicked things off with the keynote - Time Oriented Architecture: Evolution by Design? And he had some really entertaining 3-D animation and virtual simulations in his presentation. It was pretty cool. The next presentation in the main room was by Theo Beack from BEA - Virtualized SOA: Adaptive Infrastructure for Demanding Applications.

I delivered the third presentation of the day in the main room - A Little SOA Goes A Long Way, discussing the importance of identifying which aspects of your enterprise are ripe for service orientation and which areas are better left alone. The outline for that presentation is as follows:

• Introduction
• Why SOA initiatives fail
• What my kids taught me about SOA
• Adopting SOA selectively
• Bowling for governance
• Review

This presentation seemed to be well-received and I was particularly pleased with the broad acceptance of the Selective SOA Methodology that I presented. This methodology serves as the cornerstone for much of the SOA Adoption and SOA Governance mentoring that Web Age Solutions provides to clients. If you’d like a copy of the presentation, you can download it here — A Little SOA Goes A Long Way (PDF).

After a trip to the Expo Floor and some snacks, we were back in the main room to hear from Mike Pellegrini from Active Endpoints presenting - Your SOA Needs BPEL For Orchestration. He had some great content, but one nugget really stood out to me early in the presentation. Mike was describing services and processes as two kinds of abstractions that are important in SOA. After discussing them separately, he offered the following synthesis: “Services don’t change often, but they are orchestrated and re-orchestrated fairly often to build/modify business processes.” I like that. I would qualify it to say that services SHOULDN’T change often. In other words, I believe that this is indicative of an enterprise that has reached a considerable degree of maturity in their service orientation. Nonetheless, I thought it was a really solid characterization of services and processes.

Following Mike’s presentation, the speaker that was slated to speak in the main room could not be located. After hunting for about ten minutes, SYS-CON got desperate and accepted my offer to deliver another presentation that I had on my laptop. So I hopped up on stage, grabbed a mic, and fired up my laptop with the same presentation that I had delivered at SOAWorld in June early this year in New York - Service Oriented Patterns and Anti-Patterns. Fortunately, I had delivered that presentation about a month ago for a users group in Dallas, so I wasn’t completely unfamiliar with the material. I was a little rushed for time due to the late start, but overall that presentation went rather well.

After a lunch break, Ian Thain of Sybase presented - Model-driven SOA. His presentation included some interesting demonstrations of model-driven SOA using Sybase tools. Next was another Expo Floor break and snacks, followed by the SOA Power Panel.

I had the pleasure of participating on the SOA Power Panel along with Miko Matsumura (Software AG / webMethods), Kevin Hakman (TIBCO), and Sandy Zylka (NextAxiom). Jeremy Geelan moderated and we had some great discussion around SOA, convergence with other trends, economic impacts, and more. The panel session was recorded and I will post a link to it once it is made available.

I don’t know what happened for the rest of the day as I was occupied by various discussions in the late afternoon and then in a bit of sight-seeing in the evening (Coit Tower, Treasure Island, Lombard Street, and more).

One of the challenges in working with companies in their early explorations into new technologies and methodologies is the inevitable backlash that occurs regarding change. Many have argued that at best, SOA brings nothing new to the table, and at worst it will fail to achieve the desired results for the enterprise (i.e. agility, reuse, interoperability, etc.) just as previous attempts such as OO, CORBA, and Web services have failed.

While I certainly believe that SOA is more of an evolution than a revolution, there are several key aspects that give me hope with respect to success for the service orientation of the enterprise.

• Business alignment — While previous attempts around system integration and componentization have had some success at achieving improved capabilities at a very low-level, many failed to achieve any sort of sustainable ROI. A principal reason for this was the lack of business alignment. Developing more flexible, interoperable, and reusable capabilities is great, but if they are not aligned with the business and where it is headed strategically, then failure is inevitable.
• Runtime governance — We’ve made a lot of strides in the areas of technology project management, management of design and user requirements, as well as configuration management of software. Unfortuantely, once we push our systems out into production, all of our plans and controls go out the window. The enablement of runtime governance of services that are managed by automatically enforceable policies is tremendous. This gives us a new dimension to controlling, influencing, and protecting the way in which enterprise components are used and represents a significant different over previous initiatives.
• Gravity — There is sometime to be said for the fact that people have really gotten behind SOA. Vendors, think tanks, standards groups, public and private sector organizations have really taken up the SOA banner and run with it. There have been plenty of good ideas in the past, but without a critical mass, they are doomed for the history books (err… blogs).
• Internet – The Internet has changed the face of computing, business, and social interaction. SOA is inherently network-aware and able to take advantage of standards-based distributed computing frameworks.

Finally, a colleague passed along a great post from IBM Fellow Kerrie Holley that describes several aspects regarding how SOA is different from previous approaches: What is Different about SOA than previous approaches?

Jason Bloomberg and I delivered a webinar yesterday — SOA Governance: The Good, The Bad, and The Ugly

My portion of the webinar identified 10 Steps to Successful SOA Governance:

1. Avoid extremes
2. Involve business stakeholders
3. Develop SOA champions
4. Promote service ownership
5. Govern by policy
6. Shepherd the service portfolio
7. Promote a common vocabulary
8. Invest in proper governance tooling
9. Encourage collaborative governance planning
10. Start small and grow incrementally

Recently, I’ve noticed a theme among a couple of new clients as well as some people that I met at a conference. Governance? Oh sure, we’ll fold that in at some point. I’ve seen this expressed in two ways:

1. Governance is fine, but should only be implemented once some services, a registry, and a few other pieces are in place
2. Governance is a part of the project management life cycle but only once the services have been deployed

I believe that both of these perspectives on governance are flawed. I’ll address each one in turn.

Flawed perspective #1: Governance isn’t needed until after our SOA is up and running

At first glance, this thought process is rather logical. You really need some services (15+) and infrastructure before governance becomes an issue, right? The trouble is that ungoverned services require refactoring once governance is put in place. Additionally, personnel tend to be frustrated when structure is introduced later. They resist the idea that governance is needed today when it wasn’t needed yesterday. The counter to this line of thinking is that instituting governance too early will crush a fledgling service orientation initiative before it gains any traction. The flaw in this line of reasoning is to assume that governance has to be fully-mature at its inception. Governance does not, and in fact should not, be implemented all at once any more than your SOA infrastructure should be deployed in one fell swoop. Experience shows that change should be rolled out incrementally, delivering more mature capabilities gradually within the enterprise.

Flawed perspective #2: Governance isn’t needed until services are deployed
Although I’ve seen this expressed in a variety of ways, no one presented it more clearly than one client I worked with in August when they showed me their SDLC: Design > Build > Deploy > Govern They had successfully avoided the first flawed perspective by recognizing the importance of including governance from the very start. At the same time, however, they fell into another trap in relegating governance to more of an administrative function at the end of the project life cycle. Governance should be a consistent thread throughout the entire project life cycle. Enforcing policies, applying best practices, and ensuring that services and their related artifacts are properly documented begins with the service design phase and continues until service retirement.

Alternative perspective: Lean Governance
Lean governance represents a mindset in which governance is applied as needed. Implement only as much governance as is needed and constantly monitor the environment in order to tweak the degree of guidance and oversight. Governance shouldn’t be implemented as one or more powerless committees, nor as heavyweight bureaucracy. Governance should be focused, lean, and ever-present. Early in the adoption of SOA, governance should be minimal. It could be as simple as a requirement to be WS-I compliant and maintain service contracts for each service. Over time, the governance policies, processes, and procedures, as well as the corresponding infrastructure can be grown incrementally along-side the maturing of the service oriented enterprise. Finally, this governance should be applied throughout the project life cycle. Best practices identify three governance gates - design-time, change-time, and run-time. These three gates serve as check points to ensure that service design, development, and run-time behavior are consistent with enterprise goals and stated best practices.

In my experience, governance is crucial to any significant organizational change and service orientation is no different. I don’t believe that governance has to be at the extremes (powerless committees vs heavyweight bureaucracy), but instead can and should be focused, lean, and ever-present. Initially you don’t need very much governance, but it is much easier to start with an expectation that governance will be present and grow it incrementally than it is to add later down the line. Governance should be planned from the start and it should provide context for the entire project and program life cycle.

Last month, Aberdeen Group released a report detailing the findings from analyzing the service orientation results over a period of eighteen months for approximately 400 companies. The analysis revealed a ‘deep division’ between enterprises that simply deploy ‘a bunch of Web services’ and those that take a more strategic and comprehensive approach to service orientation. A lot of great analysis of these findings has emerged within the blogosphere:

• Survey: Done right, SOA pays by Joe McKendrick
• SOA Beginning to Pay Off by David Linthicum
• Survey points to SOA pay off by Tom Jowitt

I’ll let you dig through the report as well as the above posts for more details, but I would like to highlight one interesting aspect of the report. Aberdeen identifies three classifications of SOA maturity that categorize the survey respondents:

1. Best-in-Class (top 20% of aggregate performance scorers) - Characterized by organizations that prioritize investments in education/training, architecture, SOA middleware and infrastructure, and processes aimed at measuring and tweaking performance
2. Average (middle 50% of aggregate performance scorers) - Characterized by organizations that have made minor investments in SOA middleware and infrastructure, very little in education/training, and have virtually ignored organizational performance measurement metrics for refining the enterprise
3. Laggards (bottom 30% of aggregate performance scorers) - Characterized by organizations that have made little to no investment in SOA tools or training/education, tend to deploy Just a Bunch of Web Services (JBOWS), and virtually ignore the importance of governance

Beyond just identifying and labeling these groups, the report provides some compelling metrics around how these three categories drastically impact the ROI realized from service orientation. For example, every one of the Best-in-class organizations saw a reduction in application development costs, compared with only 59% of Average organizations. Those companies identified as Laggards, however, actually saw an increase in application development costs. Similar trends were identified regarding maintenance costs and impacts to user satisfaction.

So the question remains – Are you a SOA Laggard?