Reusability

• A primary driver of MDM is to allow applications to share data safely and avoid duplication.
• SOA heavily promotes the re-use of services throughout the enterprise.

Discoverability

• The Master Data Repository exposes data to applications in the enterprise.
• The Service Registry and Repository exposes services to a SOA.

Abstraction

• In MDM the complexity of the underlying data model is hidden from consumers.
• In SOA the complexity of the underlying service implementation is hidden from consumers.

Business-IT alignment

• Master data is business-focused by design.
• Creating a master data model provides a common data abstraction across the enterprise.
• This model is exposed through services within a SOA.

SOA enables integration and data exchange through services.  But this integration is useless without a common vocabulary of data content and structure.  This is why so many service oriented methodologies emphasize the importance of a logical data model that is aligned with business needs.  This data model represents the master data in SOA.

MDM drives how the enterprise manages data.

SOA drives how that data is accessed and utilized in a business context.

SOA and MDM were made for each other.

For more complex situations, intermediaries are used to provide additional capabilities:

• Security
• Transactions
• Routing
• Data Mapping
• Interface Mapping
• Reliability
• Protocol Translation
• And etc…

What exactly is an intermediary?

An intermediary is a piece of hardware or software that bridges the gap between a service provider and service consumer to provide one or more value-added capabilities (see bulleted list above). There are three primary options for intermediaries (and a host of variations on these three themes):

• Service Intermediary - Another service acts as the intermediary, providing required enterprise capabilities
• Centralized Bus - Service calls are brokered through a common, centralized focal point
• De-centralized Framework - Enterprise capabilities (security, mapping, etc.) are pushed out to the endpoints

Comparing the three options

As with so many things, there is no clear-cut winner across these three strategies, it all depends upon what you are trying to accomplish.

Option 1: Service Intermediary 

Advantages: Extremely lightweight solution.

Disadvantages: Not as robust or flexible as other approaches and requires more testing to verify proper configuration.

Option 2: Centralized Bus

Advantages: Offers lots of “out-of-the-box” capabilities that are pre-configured and available for use.  Provides a single point of control and service access management.

Disadvantages: May be over-kill in many situations, offering far more functionality and options that are needed.

Option 3: De-centralized Framework

Advantages: Provides the greatest flexibility and configurability for managing endpoints and enforcing policies.

Disadvantages: Requires a more disciplined governance process to ensure that each service is managed properly.

Working with intermediaries

Services do not generally need intermediary support except to provide value-added capabilities such as security, transactions, routing, data/interface mapping, guaranteed delivery, protocol translation, etc.  It is important that organizations carefully evaluate their requirements and select architectural strategies that meet their needs.  There are a variety of valid topologies that could be employed, depending upon your requirements, including services as intermediaries, deployment of a centralized bus (either physical or logical), and the utilization of a decentralized, policy-driven framework (“liberate the endpoints”).

You can find the videocast here: SOA Power Panel - Live from Times Square. There is a brief commercial and then it takes you right into the panel discussion.

I had a ton of fun participating in it and hopefully you’ll find value in viewing it.

The book was a ton of hardwork, but I am quite pleased with the result.  The aim was to address what I considered to be a significant gap in the marketplace — a pragamatic book on SOA that is more business oriented and less technology focused.  So I signed up Bibhas Bhattacharya as my co-author and set out to write such a book. There are vast amounts of resources available for techies that are interested in SOA.  There are even a fair number of resources available for managers.  If, however, you are a business leader, executive, or are actively trying to get the buy-in of one, where do you turn?  What resource is there to help you grapple with important questions like:

• What is SOA really about from a business standpoint?
• When is SOA applicable and when is it not?
• How should an organization go about adopting service orientation?

There are a couple of excerpts available for you to preview the content:

• Excerpt of Chapter 1 “SOA Primer”
• Excerpt of Chapter 3 “SOA Value Proposition”

When you get the opportunity to read through some or all of the book, I would really appreciate your feedback on the content.  I hope that many will find it to be a useful resource to facilitate pragmatic SOA adoption.

One effective technique for defining the schema for services that I have seen is something I describe as a composite schema. There is a rather large assumption that many service designers make, which is that the schema definition for a service must be atomic.  That is to say that you either define the entire schema inside the service interface file, or define the entire schema in a separate file.  This is fine for the first few services, but what happens if you are six or nine months into service orienting your enterprise and a key aspect of your data model changes (perhaps the ‘customer’ element or ‘product’ element)? You’ll have to go and update the schema for each and every service that uses these elements.  The alternative, is to wrestle with each service design team and get them to agree to a single schema that everyone uses (which becomes bulky and unwieldy).  The answer to all of this, is a composite schema.

A composite schema is one that draws its definition from more than one schema file.  Thus a service interface has some of its data elements defined within one schema and some defined within another (some choose to have a single schema per service interface and simply have that single schema file reference several other schema definition files).  This allows service definitions to be reused and common elements to be defined only once.  When changes inevitably need to be made, this keeps the maintenance overhead to a minimum.

So what should the design of this composite schema look like?  There are several possibilities, but a common approach is as follows:

• Core Data — This schema defines common elements that are consistent across the board within your enterprise.
• Division / Line of Business Data — The next layer tends to focus upon some major segmentation such as industry or business division, etc.
• Product or Service Line – Data structures that are common across a product line or service category are defined in this schema.
• Service Data — Unique elements that are specific to the service are defined here.

Obviously, you could define as many or as few categories of schema as you like.  The goal here would be to minimize how much of a service’s data elements are defined as service-specific data types.  The ideal, would be for 80% or more of the data used within your service interfaces to draw from common, reusable schemas.

Composite schema are by no means revolutionary.  This is a capability that has existed for years.  The problem, of course, is that there is a tendency to craft solutions in a traditional, silo-oriented fashion.  Service orientation drives us to think more about exposing and consuming services across business units, in the broadest possibly context.  It also drives us to compose solutions from existing assets before creating new assets.  All of this leads to a need to work smarter, not harder, and look for opportunities to reduce maintenance costs and more effectively utilize resources.  Composite schemas represent one such technique.

The SOAWorld conference was very well attended, including the co-located events: Virtualization World and Data Services World.  I was a bit suprised to see that so many of the presenters were vendors rather than customers or practioners.  If this had been my first SOA conference, this would be a red flag for me that SOA is still just a lot of hype.  As it was, I just found it rather odd and a little more of an empty experience than I have come to expect.  Otherwise, the conference was great and it was quite encouraging to see so much energy around things like SOA and Virtualization in spite of the state of the economy.  This gives me great confidence and reinforces a belief that I have held for some time now, which is that SOA and Virtualization make good economic sense.

My portions of the conference went fairly well.  On Monday, June 23rd, I delivered: “A Composable Service is a Good Service”.  The room was packed and that presentation was extremely well received.  The audience was very engaging, shared some of their own war stories and asked for insight on particular service design dilemas that they have been dealing with.  On Tuesday, SYS-CON staff contacted me and asked if I would be willing to step in for a speaker that would not be able to attend.  I accepted, and delivered: “A Little SOA Goes a Long Way” (my SOAWorld presentation from last November).  I also had the pleasure to participate in two expert panels, one of which was held in the Reuters studio in Time Square.  When one or more of those videos become available, I’ll be sure to provide a link.

Presentation deck downloads: “A Composable Service is a Good Service” (PDF)  and “A Little SOA Goes a Long Way” (PDF)

One final comment about the conference is that there was quite a bit of buzz about virtualization as well as “cloud computing”.  I’ve paid a little bit of attention to the virtualization movement, but thus far I have completly ignored cloud computing.  None of my clients are talking about it, and it may all be vaporware (let the cloud-related puns ensue), but it has sparked my curiosity all the same.

Initially, I would like to focus on the subject of ‘composability’.  Composable services have several important qualities:

• composable services stictly adhere to their service interface, Service Level Agreements(SLAs), and supporting metadata
• service operations always leave enterprise data in a valid business state
• service operations can be called indepently, in parallel, or as a part of a larger service orchestration
• service operations can be called within or without a transaction context

When you boil it down to the essentials, composable services clearly define what they do and they do exactly what they have defined.  Consumers of composable services are then at liberty to determine how they want to use that service’s operations, rather than the service having certain built-in assumptions regarding how the service will be used or what the larger business context ought to be.

To learn more about the merits and details of composable services, I recommend the following resources:

• “Composable Services” (MSDN Library)
• “Service Composability” (SOAprinciples.com)
• “A Composable Service is a Good Service” (Presentation at SOAWorld 2008 next week)

There’s another, less tangible benefit of composable services — they are lower stress.  The frustration and stress associated with consuming a service that later fails due to some unknown buisiness logic violation is supremely frustrating.  Consuming services that are not composable is stressful, because they cannot be depended upon to work in whatever context you choose.  Likewise, publishing non-composable services for consumption is stressful because the service provider is constantly in fear that someone will ‘misuse’ a service.  This is no way to run an enterprise or encourage the adoption of service orientation. 

I’ll explore all of these topics next week in my presentation at SOAWorld on composable services.  I’ll post the completed presentation as well as perspectives from the audience next week.  Have a great weekend and I’ll see you in New York!

Modernizing legacy information systems is not unlike the modernization efforts that have occurred for thousands of years. In fact, legacy or heritage systems are often referred to as ‘dinosaurs’. Correspondingly, the same types of obstacles must be overcome:

• Education — There is a mindset shift that must take place to understand the service oriented way of thinking. At first glance, services are just fancy objects. The reality is that service orientation requires a much broader, end-to-end view of the enterprise, complete with process-centric alignment, layered architecture, contracted interfaces, standards-based connectivity, and full life cycle governance. I have blogged about the alignment of service orientation and the unique qualities of SOA before. Also, David Ing has some interesting thoughts on service orientation.
• Skills — A well documented skills gap exists around service orientation. Joe McKendrick sites it as one of the two things killing SOA in 2008 and the guys over at ZapThink have been alerting the industry to the dangerous SOA skills gap for quite some time. Why is this? Service orientation skills cannot be acquired by attending a conference or reading a book (although I do have a book that I HIGHLY recommend). Academic learning as well as hands-on mentoring is required. Moreover, there are nuances to effective service orientation, service design, process alignment, and enterprise governance that require time and experience to develop. There are new technologies, design patterns, techniques, and methodologies that must be introduced and ultimately absorbed. This will require instructor-led training, research, hands-on mentoring, and practical experience to develop proficiency in these areas.
• Infrastructure — Contrary to popular belief, a service oriented infrastructure cannot be achieved by purchasing an Enterprise Service Bus (ESB) from a vendor and adding water. There are, in fact, a variety of infrastructure elements (service registry/repository, governance suite, business process engine, policy manager, policy enforcer, and etc.). There is a great little SOA infrastructure white paper that a consortium of vendors put together a while back. Additionally, be sure to avoid the trap of assuming that you need to have your SOA infrastructure fully mature on day one. Eric Newcomer has a great post from a couple years back regarding an incremental adoption of SOA infrastructure.
• Resources – There are several ways to slice the resource issue. For effective adoption of SOA, you will need a pool of human resources (with appropriate education and skills), technical resources (infrastructure, knowledge management and collaboration tools, as well as design and development tools), and you will need expert resources (developed in-house, or brought in from an outside source initially).
• Processes — All the education, skills, infrastructure, and resources in the world won’t amount to a hill of Java beans without effective processes for governing the adoption of service orientation. You will need processes for service selection, service design, quality assurance and testing, policy enforcement, and runtime service management. Effective governance can make or break your adoption of SOA. I have blogged on the importance of service oriented governance numerous times.
• Fear – People fear change. Service orientation appears threatening to many people at first due to the changes required in adopting it. New patterns of thinking are required around how to solve customer problems. Skills must be adapted, other skills developed from scratch. New infrastructure and resource pools must be created and processes must be adapted and new processes created. These changes can be intimidating. A considerable degree of people change management must be accepted as a part of a successful transition toward service orientation.

Service orientation adoption is challenging in its own right. It becomes increasingly challenging when you factor in the dinosaurs that must be modernized as a part of the adoption process. Immediately, questions begin to arise:

• What systems and/or business processes should be modernized?
• Should we migrate, expose, or leverage our existing legacy assets?
• Should adoption occur within a particular line of business or across the whole of the enterprise?
• How will modernization impact existing teams and project roles?

These questions and many more must be explored as part of the adoption of service orientation. I explored several of these topics in a recent public webinar - Enterprise Modernization and SOA Concepts (pdf).

Another resource you might consider is a new course that Web Age has been offering since April of this year. It is a two-day workshop that explores the modernization of legacy applications, SOA concepts, modernization challenges and risks, and various strategies for modernizing legacy systems. It is aimed at team leads, architects, managers, and legacy application support personnel. For more details, check out the course details - WA1657 Application Modernization and SOA Concepts.

Adopting service orientation and “moving out of the Stone Age” is not a simple task. It requires intentional allocation of time, energy, and resources. It cannot and should not be approached in an ad-hoc fashion. Technology adoption has always been challenging, just like when ancient man and woman adopted fire. When it comes to service orientation, be sure that you work through the necessary steps of adopting education, skills, infrastructure, resources, processes, and addressing the fear factor. Otherwise, you just might get burned.

What does rational decision-making have to do with adopting service orientation and developing services for reuse by others? I’m glad you asked. The simple fact is that the creation of services as reusable rather than one-off solutions is not free. Experts peg the expense of creating reusable solutions anywhere from base cost +30% to base cost +200%. It is significantly more expensive to build for reuse. On the flip side, it is considerable less expensive to reuse an existing service. Experts peg the expense of reusing a solution rather than creating one from scratch to be anywhere from base cost - 80% to base cost - 20%. In short, developing reusable services is very expensive, but once you have reused a service multiple times, a return on your initial investment will emerge. I blogged about this several months ago (SOA ROI, deconstructed) and provided a link to a whitepaper that explores this subject in more detail.

From an enterprise perspective this all seems very logical and the additional investment is easily offset by the value of the reuse over the long term. The problem is that in the short term, rational self-interest will drive individuals to never develop reusable services.

Consider the following scenario:

Bill is a project leader and is creating a service oriented product portal. He realizes that several of the services he needs could be designed in a fashion to foster reuse. As a good citizen of the enterprise, he directs the team to design these services in a more flexible and enterprise-centric fashion. Additional effort is spent during design and potential enterprise customers are interviewed regarding their potential needs. The added cost puts a strain on his budget but is certainly the best answer for the enterprise overall.

Sharon is another project leader in the same division as Bill. She is creating a product reporting application. She discovers that several of her requirements can be met by simply reusing the services created by Bill’s team. Other services that she needs are then developed as unique, one-off services to contain costs. Sharon uses a fraction of her budget to produce this solution, freeing up resources to meet other goals for the year.

At the end of the year, both Bill and Sharon will be judged based upon what they were able to accomplish with their respective budgets. In the absence of some acknowledgment of Bill’s contribution to the pool of available services or similar affirmation of Bill’s decision, then he is in hot water. Sharon looks like a hero and Bill comes off as a fool. Knowing this, Bill will be less likely to develop reusable services in the future (unless he can directly benefit in a very short timeframe).

Adopting service orientation and encouraging reuse is great, but what will compel divisions and individual project teams to actually play ball? In the absence of some sort of incentive structure, charge-back mechanism for using services, or similar recognition for contributions to the greater good, no one will develop reusable services. Rational self-interest will drive each group and each team to develop services that only help themselves and which will, in effect, re-create the same silos that we are trying to supplant.

All of this points to a very real gap that exists within many enterprises that are adopting or at least exploring SOA – lack of business buy-in. Someone within the business needs to recognize the true cost of SOA and put in place some sort of incentive structure to encourage teams to contribute to the greater good of the enterprise. SOA is not a short-term, quick-win approach. It requires strategic planning and long-term investment windows.

So what if you have no business buy-in? Are you sunk? No. You may not be able to get traction around reuse, but you can go for ease of integration, the benefits of standardization vs proprietary solutions, lower maintenance costs due to the loose-coupling that SOA offers, the configurable nature of services and process, and other benefits that do not depend upon long-term reuse time windows.

Adopting service orientation requires a considerable amount of strategic planning and up-front design work. There are a variety of business drivers around SOA and reuse is certainly a popular one. Is it foolish to develop reusable services? It is if you attempt to do so within an organization that is not viewing things strategically or is unaware of the long-term impacts of service orientation. In the absence of business buy-in and strategic incentives, only a fool would build reusable services.

Here’s some of what I’ve been up to:

• Article: SOA Governance: Start Small and Build Incrementally (published March 6th by SYS-CON)
  
• Presentation: SOA Adoption Planning (1-hour seminar I delivered in February)
• Presentation: Delivered a virtual presentation for the Dallas SOA Users Group. I had a client engagement in Calgary, so we tried out a virtual delivery and it was quite successful.
  
• Book: Service Oriented Architecture: A Field Guide for Executives (August 2008, Wiley Press)
• Courses: Updates to various courses plus new courses coming soon for Project Managers, Mainframe and Legacy Personnel, and some Best Practices content. For more information, check out Web Age’s SOA curriculum.
• Web-based SOA Training: I’ve been working on a new offering at Web Age in which some of our content is made available as recorded, flash-based audio/visual presentations that can be hosted internally within an organization’s intranet. More details to follow.

I’m going to try and be more diligent about blogging in April. I especially hope to share insights and perspectives that I glean from the IBM SOA IMPACT conference in Vegas. I’ll be attending that conference in early April and conducting a Birds of a Feather session on governance.