Primitive man and woman were forced to adapt to their environment to survive. Animal hides for clothes, crude weapons and tools made out of stone or bone, and roots or berries for food (perhaps sabertooth steak or terradactyl ribs if company is coming over). Eventually, some nearby tribe discovers fire, but reports of fire-related injuries and property destruction lead our cave dwellers to avoid this new magic. These primitive humans are doing fine with their current capabilities and they are able to meet their basic needs without dabbling with such things. Over time the nearby tribe’s use of fire expands and one very enterprising woman invents pit fired pottery. The more primitive people have heard and seen the potential of this new breakthrough and are interested in using it. Alas, the obstacles to adopting this innovation are not trivial. They must learn how to create fire, handle fire cautiously, locate and stockpile flint and tender, develop fireproof tools, create fire pits, learn to craft with clay and finally develop a process to fire the clay pots successfully. Lack of education, skills, infrastructure, resources, and processes hinder their ability to progress. Perhaps the greatest barrier for these primitive people is fear of the unknown and risk of failure. Thousands of years later, we are faced with the same technology adoption challenges — education, skills, infrastructure, resources, processes, and fear.

Modernizing legacy information systems is not unlike the modernization efforts that have occurred for thousands of years. In fact, legacy or heritage systems are often referred to as ‘dinosaurs’. Correspondingly, the same types of obstacles must be overcome:

• Education — There is a mindset shift that must take place to understand the service oriented way of thinking. At first glance, services are just fancy objects. The reality is that service orientation requires a much broader, end-to-end view of the enterprise, complete with process-centric alignment, layered architecture, contracted interfaces, standards-based connectivity, and full life cycle governance. I have blogged about the alignment of service orientation and the unique qualities of SOA before. Also, David Ing has some interesting thoughts on service orientation.
• Skills — A well documented skills gap exists around service orientation. Joe McKendrick sites it as one of the two things killing SOA in 2008 and the guys over at ZapThink have been alerting the industry to the dangerous SOA skills gap for quite some time. Why is this? Service orientation skills cannot be acquired by attending a conference or reading a book (although I do have a book that I HIGHLY recommend). Academic learning as well as hands-on mentoring is required. Moreover, there are nuances to effective service orientation, service design, process alignment, and enterprise governance that require time and experience to develop. There are new technologies, design patterns, techniques, and methodologies that must be introduced and ultimately absorbed. This will require instructor-led training, research, hands-on mentoring, and practical experience to develop proficiency in these areas.
• Infrastructure — Contrary to popular belief, a service oriented infrastructure cannot be achieved by purchasing an Enterprise Service Bus (ESB) from a vendor and adding water. There are, in fact, a variety of infrastructure elements (service registry/repository, governance suite, business process engine, policy manager, policy enforcer, and etc.). There is a great little SOA infrastructure white paper that a consortium of vendors put together a while back. Additionally, be sure to avoid the trap of assuming that you need to have your SOA infrastructure fully mature on day one. Eric Newcomer has a great post from a couple years back regarding an incremental adoption of SOA infrastructure.
• Resources – There are several ways to slice the resource issue. For effective adoption of SOA, you will need a pool of human resources (with appropriate education and skills), technical resources (infrastructure, knowledge management and collaboration tools, as well as design and development tools), and you will need expert resources (developed in-house, or brought in from an outside source initially).
• Processes — All the education, skills, infrastructure, and resources in the world won’t amount to a hill of Java beans without effective processes for governing the adoption of service orientation. You will need processes for service selection, service design, quality assurance and testing, policy enforcement, and runtime service management. Effective governance can make or break your adoption of SOA. I have blogged on the importance of service oriented governance numerous times.
• Fear – People fear change. Service orientation appears threatening to many people at first due to the changes required in adopting it. New patterns of thinking are required around how to solve customer problems. Skills must be adapted, other skills developed from scratch. New infrastructure and resource pools must be created and processes must be adapted and new processes created. These changes can be intimidating. A considerable degree of people change management must be accepted as a part of a successful transition toward service orientation.

Service orientation adoption is challenging in its own right. It becomes increasingly challenging when you factor in the dinosaurs that must be modernized as a part of the adoption process. Immediately, questions begin to arise:

• What systems and/or business processes should be modernized?
• Should we migrate, expose, or leverage our existing legacy assets?
• Should adoption occur within a particular line of business or across the whole of the enterprise?
• How will modernization impact existing teams and project roles?

These questions and many more must be explored as part of the adoption of service orientation. I explored several of these topics in a recent public webinar - Enterprise Modernization and SOA Concepts (pdf).

Another resource you might consider is a new course that Web Age has been offering since April of this year. It is a two-day workshop that explores the modernization of legacy applications, SOA concepts, modernization challenges and risks, and various strategies for modernizing legacy systems. It is aimed at team leads, architects, managers, and legacy application support personnel. For more details, check out the course details - WA1657 Application Modernization and SOA Concepts.

Adopting service orientation and “moving out of the Stone Age” is not a simple task. It requires intentional allocation of time, energy, and resources. It cannot and should not be approached in an ad-hoc fashion. Technology adoption has always been challenging, just like when ancient man and woman adopted fire. When it comes to service orientation, be sure that you work through the necessary steps of adopting education, skills, infrastructure, resources, processes, and addressing the fear factor. Otherwise, you just might get burned.

A rose by any other name would still smell sweet, just as a strategic corporate initiative is no more or less successful due to its use of certain en-vogue terms such as Enterprise Architecture (EA), Business Process Management (BPM), and Service Oriented Architecture (SOA). I was struck by the realities of the semantic terminology game that we play while attending the Open Group’s 17th Annual Enterprise Architecture Practitioners Conference in San Francisco last week.

There was a general consensus at the conference that there are common themes running throughout enterprises that have various labels attached. Those themes include the following:

• Business and IT need tighter alignment
• Visibility between business and IT should be increased
• Governance is needed at various levels of the organization to reduce risk and protect ROI
• Business processes and technology solutions should be more flexible and better able to adapt to changing demands
• Business processes and technology solutions should be designed intentionally and with sufficient rigor

The labels attached to these trends vary from enterprise to enterprise. Some drive toward one or more of these goals under the EA umbrella, others under the BPM label, and still others are talking about SOA. Some organizations use two or even all three labels. Regardless of the attached label, it is a strategic initiative at the business unit or enterprise level that aims to improve business predictability and squeeze more value out of technology investments. This become readily apparent as the SOA Governance panel (other panelists included Michael Nassar of IBM, Stephen Bennett of BEA, and Mats Gejnevall of Capgemini) that I participated in last Tuesday was quickly challenged by the audience to delineate between governing SOA and governing EA. The line between these disciplines is quickly blurring within many enterprises.

So what’s the take-away for us? We need to look past the labels that are applied and take a more business-focused and goal-oriented approach to education, mentoring, and ultimately solution development. We need to probe more intently with our clients to discover their strategic direction, business drivers, and objectives for one-year, two-year, and five-year timeframes. We may have the perfect service offering for them, but because it is labeled as SOA, BPM, or EA, it may not jive. I have several examples of this:

• Back in November, I wrote about Synovus, a mid-sized bank in the southeastern US, that spoke at SOAWorld in San Francisco. They were a non-traditional case study in that they did not set out to adopt SOA. They just knew that they had various business problems that needed to be solved. They set out to solve those problems and in the process created a SOA. If someone had approached them a year or two ago with an offer to enable their enterprise through SOA, they would have dismissed it because they were not doing ‘SOA’, at least not in name.
• Last year, a US Department of Defense (DOD) agency participated in one of my SOA governance workshops. Several of the students made the observation that although we had originally set out to talk about SOA governance, the vast majority of the concepts, principals and best practices that we discussed were applicable to EA. The mentoring engagement quickly broadened in scope and we began speaking in terms of governing the enterprise and aligning with business strategy. The principals, methodologies, and best practices all flowed very nicely into this broader scope. The ‘SOA’ label that we had begun with was immaterial to the objective — better governance of enterprise assets and business processes.
• Throughout last year Web Age Solutions worked with various clients that wanted to service orient their enterprise assets and orchestrate those assets as a part of executable business processes. Some of them called that SOA, others called that BPM. Some wanted us to describe SOA in the context of BPM and others wanted us to introduce BPM in the context of SOA. At the end of the day, the solution was the same. It was the same standards, same tools, and the training was about 90% the same. The difference was found in the labels attached and the emphasis upon certain key concepts.

It is imperative that we understand our client’s needs, perspectives, and bias with respect to trends and labels. At the end of the day, we are enabling them to solve business problems, not in order to check a box next to an acronym.

One semester in high school, I joined a show choir. After two months, I decided that I had endured all of the step-chasse, step-ball-change, and “jazz hands” that I could stand! I don’t remember any of it now, but I do remember the importance of choreography and coordination among the various participants. What, pray tell, does this have to do with SOA? I’m glad you asked.

Like a well choreographed dance, successful service oriented solutions clearly define roles, responsibilities, and coordinated interaction. Some call it workflow, others choreography, others orchestration, still others talk in more general terms of coordination. There was a time when I thought all of this was a just a semantic game and there was no real difference. While a part of me still questions whether or not we REALLY need all of these different terms, concepts, and standards for the basic idea of ‘making sure everyone and everything does what she/he/it is supposed to’, I am beginning to appreciate the differences among these various concepts.

Orchestration (Executable Business Process)

As organizations mature in their adoption of SOA, they tend to migrate away from a focus upon individual services and toward a focus upon business processes. Orchestration enables an enterprise to fulfill a business process by pulling together a set of service operation calls along with workflow-style business logic and some process-specific variables to maintain persistent state throughout the process. Principally, this is achieved through Business Process Execution Language (BPEL) and a corresponding process engine. BPEL is championed by the likes of Microsoft, IBM, and Oracle (to name a few) and is maintained and actively developed by OASIS.

Choreography (Multi-party Collaboration)

Moving beyond the internal workings of a given organization’s services and orchestrated business processes, choreography is concerned with facilitating the interactions that occur between enterprises. Choreography relates to describing collaboration interfaces between parties. The dominant standard in this space is the W3C’s Web Service Choreography (WS-CDL), a language designed to describe multi-party contracts.

Coordination (Multi-party Context Sharing)

The term ‘coordination’ within the SOA / WS space is far less well understood than either orchestration or choreography. Much of the literature and vendor documentation out there will describe choreography and coordination as being synonymous. This is not quite accurate. To make matters more confusing, the W3C has a Web Services Coordination group. Their job is to coordinate all of the others groups in the WS domain. So what does ‘coordination’ actually mean in the SOA / WS context? It’s really all about sharing context between WS entities, typically to support the prorogation of transaction semantics. OASIS has done a tremendous amount of work in this area under the Web Services Composite Applications Framework (WS-CAF) umbrella. That framework consists of the Web Service Context (WS-CTX), Web Service Coordination Framework (WS-CF), and Web Services Transaction Management (WS-TXM) standards.

Orchestration vs Choreography vs Coordination

So what’s the story here? Are these various workflow-type concepts and corresponding technology standards truly distinct elements that merit unique treatment, or just an example of over-engineering? Although my heart longs for simplicity and clarity in this arena, I do think that there is some value in breaking these notions apart. This is especially true for enterprises that engage in complex, service oriented B2B interactions (typically financial and insurance companies).

Orchestration expresses organization-specific business logic. The entire flow is owned and governed by the organization, even if the flow requires interaction with third parties. Choreography is more collaborative and peer-to-peer in nature, with no central organizing entity. Coordination is a supporting player that can be used in concert with an orchestration or choreography.

So when do you choose one over the other?

• Use orchestration to automate and service orient individual, executable business processes
• Use choreography to describe contractual service interactions between various parties
• Use coordination to define the hand-off of application and business context between various parties

If you’d like to read more on the subject, I recommend the following:

• “Service Orchestration vs Service Choreography”, by John Reynolds
• “Orchestration vs Choreography”, from The ServerSide.com
• “Choreography vs Orchestration” (Orchestra / Dancer Analogy), by Stefan Tilkov
• “Choreography vs Orchestration” (Traffic Light Analogy), by Paul Downey
• “Orchestration, Choreography, Choreostration”, by Jim Alateras

Any thoughts, questions, or war stories to share regarding orchestration, choreography, or similar concepts in SOA?

I spent much of the second day talking with attendees, speakers, and SYS-CON staff about SOA, virtualization, and the conference in general. Overall, there was a general sentiment that the conference was a success and had some valuable content. One presentation from the day that really stood out was a case study from the banking industry — Delivering Big Bank Solutions with Community Banking Intimacy. The bank is Synovus, a regional bank in the southeastern part of the US with about 33 billion in assets spread across multiple lines of business (banking, financial management, brokerage, insurance, and more).

In a thick southern, ‘good ole boy’ style delivery, the speakers from Synovus described their approach as ‘Blue Collar SOA’, explaining: “…it may not be perty, but it works and we get a lot of value out of it.” The Synovus representatives went on to explain that while they did not set out to deploy a SOA solution, their business requirements and technology objectives led them in that direction. They went on to describe several keys to successful SOA adoption, including the following:

• Define and document a set of guiding principles
• Define and document a reference architecture that is consistent with those principles
• Identify standards, technologies, and products that can enable that architecture and communicate and manage this through governance
• Identify a business visible pilot project to convert the strategy and reference architecture into a real implementation
• Define and enforce the implementation strategy through governance processes
• Employ governance toolsets to validate services against guiding principles as well as policies and SLAs

The results of their SOA adoption as well as the importance of service orchestration is outlined in a white paper from Active Endpoints (their service orchestration partner).

The presenters concluded by identifying a huge list of lessons learned. Three things really stood out to me on that list:

1. Identify and document standards and best practices and enforce them through your governance processes
2. SOA changes your team dynamics — you will need more documenters than developers
3. Education is key because SOA is hard

That third one was especially of interest to me given Web Age’s focus upon SOA education. In the Q&A portion I asked for the folks from Synovus to elaborate on the education topic. They were eager to explain that they had discovered a huge gap between legacy skill sets and what it takes to be successful with SOA. They had to figure out how to translate those skills, bring people up to speed on new methodologies and terminologies and get people to think differently about how to solve problems. Synovus also discovered that considerable education and mentoring was necessary to help the leadership that direct their lines of business to “think outside their silo”. Amen!

At Web Age we have seen the same sorts of challenges and often discover that teaching the technologies is fairly simple. Teaching the methodologies is more difficult. Teaching the new mindset (e.g. ‘thinking outside the silo’) and providing business users with new tools and techniques for solving problems is even more difficult. So what is the most challenging? Getting organizations to actually change once they are educated. The technology side of SOA is relatively easy. The human side is where things get tricky.

Overall, the conference was a great experience. If you get a chance to catch the East coast or West coast shows next year, definitely go. I hope to see you there!

One of the challenges in working with companies in their early explorations into new technologies and methodologies is the inevitable backlash that occurs regarding change. Many have argued that at best, SOA brings nothing new to the table, and at worst it will fail to achieve the desired results for the enterprise (i.e. agility, reuse, interoperability, etc.) just as previous attempts such as OO, CORBA, and Web services have failed.

While I certainly believe that SOA is more of an evolution than a revolution, there are several key aspects that give me hope with respect to success for the service orientation of the enterprise.

• Business alignment — While previous attempts around system integration and componentization have had some success at achieving improved capabilities at a very low-level, many failed to achieve any sort of sustainable ROI. A principal reason for this was the lack of business alignment. Developing more flexible, interoperable, and reusable capabilities is great, but if they are not aligned with the business and where it is headed strategically, then failure is inevitable.
• Runtime governance — We’ve made a lot of strides in the areas of technology project management, management of design and user requirements, as well as configuration management of software. Unfortuantely, once we push our systems out into production, all of our plans and controls go out the window. The enablement of runtime governance of services that are managed by automatically enforceable policies is tremendous. This gives us a new dimension to controlling, influencing, and protecting the way in which enterprise components are used and represents a significant different over previous initiatives.
• Gravity — There is sometime to be said for the fact that people have really gotten behind SOA. Vendors, think tanks, standards groups, public and private sector organizations have really taken up the SOA banner and run with it. There have been plenty of good ideas in the past, but without a critical mass, they are doomed for the history books (err… blogs).
• Internet – The Internet has changed the face of computing, business, and social interaction. SOA is inherently network-aware and able to take advantage of standards-based distributed computing frameworks.

Finally, a colleague passed along a great post from IBM Fellow Kerrie Holley that describes several aspects regarding how SOA is different from previous approaches: What is Different about SOA than previous approaches?

I’ve been traveling and teaching a lot since the SOAWorld conference. This past week, a couple of my students challenged me on some statements that I made during the course regarding the notion that Microsoft has arrived a bit late to the SOA table. I find this particularly interesting considering the fact they Microsoft was so instrumental in the early Web services innovations around SOAP, WSDL, etc., but it seems that they have gotten distracted with lawsuits, Office, and Vista, and taken a while to come along with a comprehensive SOA story and supporting infrastructure.

After some interesting and sometimes heated discussions, these students points me toward some resources that shed some interesting light on what Microsoft has been working on. I must admit that I had lost track of .NET 3.0 and the evolution of Indigo into the Windows Communication Foundation (WCF). The work that Microsoft has done in this arena is quite impressive and, in true Redmond fashion, elegantly simple. Additionally, one of the students turned me on to Juval Lowy, who has been recognized by Microsoft as a Software Legend and a Regional Director for Silicon Valley. Juval is the brains behind IDesign, a great resource for .NET 3.0, and provides master-level workshops on .NET 3.0 and especially the WCF. The guy is brilliant, has a fascinating perspective on SOA, and a rather compelling and authoritative delivery style. To get a taste you can check out one of his MSDN Webcasts: Applying Service-Orientation to Your Development Process.

On a parallel thread, I ran across an interesting article from Redmond Magazine: “Microsoft Does Have a SOA Strategy“. Dana Gardner, who is quoted in the article, blogs on this subject and describes the strategy as ’service-enabled’ and ’sorta-SOA’. Another analyst quoted in the Redmond article describes Microsoft’s strategy around SOA as ’standards-based at the edge’. The article also identifies that Microsoft is focusing more upon empowering individuals and small teams rather than ’selling a big, fat SOA stack’. This is certainly consistent with their strategy in other areas of the enterprise. The reality of, course, is that Redmond will not sit idle while the IBMs, Oracle’s, and SAPs of the world carve up the enterprise marketplace. More infrastructure components are coming from Microsoft in the next iteration of Biztalk and the full realization of .NET 3.0, especially the Workflow Foundation (WF) piece.

One final thread regarding Microsoft’s SOA strategy is an apparent attempt to take the ‘ESB leadership’ that the big Java vendors have managed to secure in the industry and turn it around as a liability. Rather than trying to compete against these established ESB vendors directly, Microsoft is attempting to paint the notion of an ‘enterprise’ bus as short-sighted and internally-focused strategy. The next generation, according to Redmond, is the Internet Service Bus (ISB) which turns the attention outward to the extended-enterprise, consisting of partners, clients, and the information superhighway at large. Whereas traditional ESB solutions focus within the enterprise. The ISB turns the attention to interaction between enterprises. I’m not sure how much of that is real and how much is fluffy rhetoric, but it is an intriguing perspective.

It’s been over a month since I’ve posted anything. For the past several weeks I’ve been heavily engaged with clients and several public speaking engagements related to Service Oriented Architecture (SOA), Business Process Management (BPM) and Enterprise Architecture (EA). A resounding theme as of late has been the importance of aligning the business and technology teams along shared enterprise goals. Although architecture and service design patterns are important, more and more enterprises are interested in the cross-functional synergy that SOA enables. In order to truly deliver upon the promise of agility that service orientation offers, it is critical that business and technology personnel work together.

So what exactly does this enterprise alignment look like?

• Cross-functional, enterprise teams must champion the adoption of SOA. Change agents and visionary champions need to understand and clearly communicate the SOA strategy to all levels of the organization. Technology-minded individuals and business-minded individuals need to be involved. This should also be a diverse, enterprise-wide initiative that has input from all lines of business. Some groups choose to establish or leverage an existing Center of Excellence, Competency Center, or similar objective group. Regardless of how formal or informal the team may be, the keys to success are passion, communication, and diversity.
• The Information Technology group must be treated as a strategic partner, not a cost of doing business. IT provides far more value than simply keeping an organization’s servers up and running. They are a key enabler of information and automated systems that allow humans to focus more on analytics and less on low-level details.
• Organizational transparency must be prioritized. The visibility of IT’s business value and internal capabilities must be raised to the business community (minus the technical jargon and useless acronyms). The business capabilities that IT has today and can reasonably make available in the future need to be clearly communicated to the business teams. Visibility is a two-way street though. The business teams need to clearly convey business drivers and relevant metrics so that the technology teams understand where the organization is headed.
• A process-centric approach to business. Business Analysts and Business Engineering teams are gaining increasing credibility within the modern enterprise. In order to truly bridge the gab between business and technology, it is essential for a common frame of reference to exist. Business process models offer a visual mechanism for both business and technology personnel to discuss, explore, and solve business problems without getting caught up in the details. Afterward, each team is then able to translate this into their world. The technology teams, in particular, have a lot of options available to them thanks to standards like Business Process Execution Language (BPEL) and Business Process Modeling Notation (BPMN). On the business-side of things, it is important that business processes become a pervasive element within the enterprise. Business processes should drive budget allocation, project management frameworks, and even human resource strategies.

Service Oriented Architecture has really grown to become an umbrella that encompasses an entire range of enterprise initiatives (service architecture, BPM, EA, governance, etc.). Rather than getting caught up in the technical ‘architecture’ aspect of SOA, I prefer to talk with clients in terms of business and technology alignment along service-oriented business processes. Service orientation is really a people problem more so than it is a technology problem.

As I explore this alignment concept further with clients, I will post some more on this subject. I might even distill my experiences into a white paper later this year. I’ve got a pretty full schedule this summer. If nothing else, I’ll be back in late June / early July to comment on SOAWorld 2007 in NYC.

We’ve been hacking away at SOA in one form or another for seven or eight years. I often wonder where we are in terms of SOA maturity across the industry as well as at individual corporations and government agencies. In some respects, there are many strong signs of maturity. There are copious standards and methodologies, extensive vendor tool support, widespread adoption in the private sector, and even evidence of increasing momentum from the public sector. We also are developing a fairly extensive list of SOA success stories and failures from which to learn. On the other hand, there is still a lot of work to do in the areas of governance, business process management and modeling, service management, service transactions, and service security (to name a few). I often wish that there were some sort of objective measure for the maturity of the industry and a comprehensive roadmap for where the various standards organizations, vendors, and industry think tanks are heading.

On an organization-specific basis there is certainly a strong desire to define SOA adoption roadmaps and develop models for gauging business and technology maturity with respect to SOA. One popular approach is ZapThink’s Service-Oriented Architecture Roadmap. Another popular approach is to construct maturity models based off of CMM/CMMI. Arguably, the most widely accepted of the models that follow the CMMI framework would be the SOA Maturity Model (SOA MM) developed by Progress-Sonic Software and a handful of partners. In evaluating the various maturity models, ZapThink offers their analysis in: “What to look for in a SOA maturity model“.

I have pulled together a brief whitepaper (SOA Maturity Models) that introduces the subject of SOA maturity models, briefly explores three popular models, and then provides some analysis regarding selecting and customizing a model.

As SOA increases in maturity and we understand it better, inevitably we develop standards and common design strategies. One such design strategy that I am closing monitoring is the evolution of the Enterprise Service Bus (ESB). In particular, I am interested in the potential development of a standardized SOA middleware framework framework that can be implemented by multiple vendors to supporting portable process definition, component/service assembly, mediation logic, messaging, and a data service layer.

OSOA seems to be making some considerable headway on Service Component Architecture (SCA) and Service Data Objects (SDO). OASIS has certainly cornered the business process / service orchestration side of things with the Business Process Execution Language [which evidentally now is shunning the all upper-case (BPEL) acronym in favor of the much stranger looking (BPel) case]. This still leaves me wanting to see some standardization in mediation logic and messaging. I hope that Chappell and friends will pick up the ball with Synapse and run with it, but it doesn’t appear to be moving especially fast right now. Until then, or until another initiative comes along, I guess I’ll just have to settle for vendor lock-in on the messaging / mediation middle-tier.