SOA in Action Blog

Joe McKendrick: Hello, and thanks for joining us today. My name is Joe McKendrick, contributing analyst to ebizQ, and I'll be your moderator for today's panel discussion, part of today's SOA Governance virtual conference.

We just heard a very informative and enlightening keynote presentation by Gartner's Frank Kenney and Software AG's Miko Matsumura on why SOA Governance is so critical to organizations today.

To further explore these issues, we have an all-star line-up panel for you, with whom I would consider some of the premier thought leaders in the SOA space today.

Let me introduce:

Anne Thomas Manes, vice president and research director with Burton Group, and former Chief Technology Officer at Systinet; Ron Schmelzer, managing partner with ZapThink; Frank Martinez, senior vice president, product strategy for SOA Software; David Bressler, SOA Evangelist for the Actional products at Progress Software; Ed Horst, vice president of marketing and product strategy for AmberPoint; and John Michelson, a founder and chief scientist of iTKO LISA.

We've really got a lot to talk about today. One, service oriented architecture is at a turning point, for a couple of reasons. It's been around, in its current form, for several years now. We have many companies that have taken SOA beyond the piloting stage and are applying it against real business problems. And getting real results. Also, with today's economy so turbulent, there is concern as to how to position SOA going forward - as a cost-cutting strategy, or as a mechanism for business growth. I want to get a read from our panelists on how far along we really are with SOA, and what is needed to take things to the next level.

Second, I'd like to get practical, and explore the tools and techniques that are essential to getting started with SOA governance.

Third, we need to discuss the implications of SOA governance in an environment that seems to getting seemingly more ungovernable all the time - we have all those Web 2.0-ish activities everyone seems to be engaging in, well under the radar of corporate IT management. Then there's virtualization, in which the actual management of various systems gets taken to a whole different level.

And, now, here's the crazy part - we hope to talk about all these things all within one hours' time. We invite our listeners to join in as well. We urge everyone to participate -- just use the "Ask a Question" area there on your gray console at any time during the presentation.

Okay, welcome panelists. Let's get started. First, a question on semantics. I've been hearing another term to describe the concept of SOA governance, and that is "Service Lifecycle Management." Some have even said that the term "governance" may be a little too overbearing for what we need to accomplish with SOA. Should we, perhaps, start referring to the concept as Service Lifecycle Management to better describe how we're managing services?

Ron Schmelzer: In general, people's perspectives on governance are probably colored by the role that governance plays in their organization as a whole...

Part of the reason why you're seeing perhaps some resistance to the concept of SOA governance is in the past organizations in the past have Tried other flavors of IT governance that actually have been heavy-handed. As they say, everybody likes governance, nobody likes to be governed...

Part of the reason is that it's hard to impose certain controls on parts of the organization and achieve flexibility and continuous change, and at the same time have the Visibility...

There are a lot of concrete things.. that are specific to SOA governance, especially around design time, runtime and change time aspects, of dealing with a continuously changing IT environment.

And within that environment you want to give people the freedom to achieve agility they want, without introducing chaos, without losing control. If I had to put the terms governing and Governance and government in that context. I think that's what it is, it's managing the balance between freedom and flexibility and agility and control, to avoid chaos.

And to the extent that companies can find the right balance between the two things... I don't really see governance as being itself as being a term that's particularly heavy handed... any more so than government... certainly we have handy handed government... but we also have other governments that are much more responsive... to their electorate... within that concept, I think that's helpful...

Service lifecycle management is an aspect of SOA governance, especially viewed within the larger context of IT governance and corporate governance.

Joe McKendrick: Okay, let's ask Ed Horst. Do you see organizations struggling to achieve a balance between freedom and control?

Ed Horst: Certainly. It partly depends on the existing organizational structure of the organization, what they're used to. Is it a very decentralized IT organization... is it more centralized... requires different styles of governance, or at least tolerate different styles of governance...

But there is a couple of keys to governance... success... it really boils down to you need an approach that is somewhat automated... it has to be automatic in some ways in picking up the information and staying in synch with what's going on.

It can't be too onerous... which is Ron's earlier comment.. everybody wants government, but they don't want to be governed.. that certainly is true for the people that are impacted by SOA governance in the broad sense...

and it has to be incented in some way. Fundamentally, you have to answer the question what's in it for me. if you expect to people to participate in the governance process. There has to be something in it for them other than something in it for the other guy.

Joe McKendrick: Anne Thomas Manes, I've heard you address these topics, and hear echoes of your statements in Ed Horst's remarks, that governance should be not too onerous.

Anne Thomas manes: I think my point is you want to make governance as transparent as possible. Back to the original question that you asked... There are certain organizations where you cant use the term governance... because the culture and environment just reacts badly to it. In that case if you want to call it service lifecycle management instead of governance, that's what you have to call it.

I think from an industry perceptive, the term we want to use is SOA governance. It's the term we want to use.

But if you want to make the governance work within your organization, you want to make sure that the governance is not too heavy handed, as Ed was saying. You want to make sure its not too onerous. You want to make sure that it doesn't impend progress. Therefore if you can make governance as automatic as possible and as helpful as possible, then a governance program is going to be a lot more successful.

Ed Horst: Anne, I've also heard you talk about incentives:

Anne Thomas Manes: Anytime you want to get somebody to do something for you, you have to answer the what's in it for me? If you're trying to ask the business guys to give up a certain amount of self-determination by accepting the use of another service, or if you're asking them to perhaps give up something special that they wanted, for something more generic, why are you asking them to do this, and why would they even consider doing that if they have to give something up? Then it's really important to explain what's in it for me, and here's what you want to do.

When it comes to a governance program, if you can make the governance program helpful to the people who are required to adhere to is, as opposed to something onerous, then they're going to be a lot more happy about following the rules.

Jon Michelson: We found that very analogous to the governance analogy in the physical world, that federating or layering the governance policy. Is how it happens. If you think about it, my homeowners association wants to know where I'm going to put my flag. But at the federal level that would be an absurd notion for them to decide such a thing. So we are more tolerant of greater control closest to us than we are far away.

so if governance is federated as opposed to this ..single tiered notion... where there's this very distant group of people... who decides on how were supposed to build and consume services... then we will tend to react negatively toward that... but if there is even if it's a ..fairly onerous set of policies.. but it's shared... and its owned by me and my close constituents... I tend to be more willing to do it.

that does get to what ed and Anne both mentioned... they're both getting something for it... we can see if I do participate in my local governance, I'm going to get better reuse... and I'm going to get faster cycles and such... at the federal level... we don't get that kind of promise.. instead we just get the reaction of, that you're controlling my behavior... and I don't see an upside to it....

David Bressler: ...semantics of SOA governance... service lifecycle management... is better...

one of the challenges about layering policies, if you limit it to service lifecycle management... you're actually forgetting about a lot of other aspects... of SOA governance, which include ...policy lifecycle management, or persona management, or process management... all of those tings... are aspects of SOA governance that service lifecycle management ....doesn't apply from a semantics perspective at least...

Joe McKendrick: Let's talk a bit about taking those first steps into SOA governance. First, what happened to the idea of SOA Registry/Repository as the "first step" toward SOA Governance, and are today's enterprises really leveraging these components to their potential? And if not, why not? Do you need to have this foundation in place before proceeding with the other stuff?

Frank Martinez: I think it depends... there's certainly no shortage of customers... in the marketplace today, that went out and started with the goal of rolling out an SOA registry repository...

What were finding that in practice today... is that while that's something that's pretty easy to wrap your arms and your head around... it really doesn't speak to the goal... registries and repositories really are a mechanism... so there's certainly no shortage of ...folks and took...that as a. first step...

when looking at the opportunity around SOA governance, what you're really trying to do is make sure... that you're addressing people, policy, process, metrics, and organizational aspects of your program... that begins at the planning stages.. and it continues on through the development lifecycle, and it has to be supported by your operational governance efforts...

so while SOA registries and repositories certainly are a necessary mechanism... they by no measure... deliver governance on their own... you really have to make investments kin those areas...

going back to the first question.. It's really about encouraging desired behavior... ..if you've got a culture in your organization that is averse to being governed... is much more likely to want to focus around the aspect of service lifecycle management... then what you need to do is make sure that you're encouraging that behavior by whatever means necessary..

...make the governance mechanism as transparent as you can... ...you make the path of least resistance the governance path.. in other words, good governance is really about making sure that ... your governance path is the path of least resistance versus the path of greatest resistance... so the registry repository mechanisms are certainly useful and necessary... by no ,means sufficient without the other areas... being addressed simultaneously...

Ron Schmelzer: I definitely want to back that up.... In general, the recurring theme we see with service oriented architecture... ...you have to start with the problem first, figure out how the architecture and the services are going to solve those problems... ...and then think about how the infrastructure you're going to buy or build is going to support those services... you'll find it doesn't matter if its registries... or any sort of SOA infrastructure product at all... if you Start with the tools... then work our way back... then try to figure out what services you're going to create... then figure out what kind of problems you're going to solve... Its not that you're going to be unsuccessful, but ...the odds of success are going to be pretty low... you'll probably be pretty lucky if you do it that way....

you can always find the greatest chance of success... by finding the problems first... and figuring out how to solve those problems... and using those problems are going to determine your architecture... that demands a registry, if that demands a federated registry.. if that demands an esb of one type of another... then great. At least you'll know the decision you've made will solve the business problem.... is a shortcut to buy the tool first, and hope that... you're going to manage to shoehorn the problem... into the solution you've already found....

Jon Michelson: ...I totally agree with Ron... ....I would add... we know that if we're traveling to very warm climate, that there's a requirement for us to have the appropriate clothes... and if were going to a cold weather climate... then we're going to need different types of clothes..

If we're going down an SOA path... ...there are certain things that are obvious to plan and prepare for... I would suggest that some way of understanding how to provide access to control... ... provide access and control... around service definitions...

I think we need to know before we get there... that were going to need it ... so it should be part of the plan that we pack with the right materials...

Clearly it is, in some fashion... whether its registries from the way we currently think of them or something else... ...this is one of those things I think we know we need in order to prepare for the place we are going...

Anne Thomas Manes: I have a somewhat different angle on this... ....governance is about process. It's about defining policies.... And establishing processes that you use to verify compliance with policies...

It's about measuring what you're doing.. and it's about having an organization... that actually provides a certain level of support for the governance program in the first place...

This is a process, and as I'm developing my code... ...I need to make sure that the code is conforming with my principals and with my best practices... and with any legal regulations and requirements that I have...

And therefore since it is a process, I have the ability to automate parts of that process... and I can use infrastructure to help me with that process... and so therefore the products that you buy from the various SOA governance vendors... ..are providing you with tools and technologies that can help you automate that process...

Just as you automate your purchasing process... or your claims processing systems or whatever.. you can always automate these processes...

There's a huge amount of data... metadata and artifacts that you have to manage and maintain in order to accomplish adoption of service oriented architecture... ....you've got to maintain information about all your services... and about all the artifacts that you're building, and about the consumers and providers and the contracts between them... Policies that apply to those different types of services... both at design time and at run time... and where you're going to keep all this information

So registries and repositories are extremely valuable pieces of infrastructure that help you manage and maintain this information....

Can you do governance without them? Yes, but, I suspect that your not going to get it to scale very well... because you really want to have a place where you can go and say, here are the policies we have, and he's where they apply, and here's the information we have about our services...

You don't necessarily have to have a UDDI registry, and certainly don't have to have the latest whiz-bang SOA repository. But I probably need something to help to me manage these assets.

Dave Bressler: You used the word automatic, you just said it has to capture this information, whether its about the service directly, or the metadata about it. In fact ...that is where reg rep has failed.... .... Because its not capturing anything.. its not automatic at all... and frankly its nothing more than a fancy spreadsheet of information about your services that you can email... around once a week to your development team.

When people start. down the line that may be different.. what happened is People started with reg rep because.. they wanted to get a handle on the Chaos... but because its not automatic, they didn't get a handle on the chaos.. ...automatic is not only one of the most important things, but one of the first things that need to be done...

Anne Thomas Manes: you to make sure your governance processes are as automatic as possible...

David Bressler: I don't think ...registries repositories don't do that automatic component of it. ... that's why they haven't been as successful... they are a foundation of a place to start... may be one of the cornerstones... but they're not the cornerstone to start propping up things with...

Anne Thomas Manes: I'm not sure where is the right place to start... somewhere you have to document what your policies are... and you have to specify where those policies apply.... you still have to maintain and manage all this metadata... therefore registries and repositories are a very valuable place to start...

But just going out and buying a governance product is not going to give you governance. Because if you don't have the processes that enable your developers and your consumers and your users ....to populate that registry that repository, you're not going to capture... that information.

Jon Michelson: You still have to address the requirements ... around your people are going to exhibit the right behavior, and ..environment... processes that are going to automate that execution... of those behaviors.... And how the metrics are going to measure the business outcomes of what you are doing....

And ...that's not going to get solved by ...by simply deploying a piece of infrastructure... You still have to understand how... that infrastructure is going to support that model...

Anne Thomas Manes: in fact what you really need to build the SOA governance processes.. into your standard software development lifecycle processes... so that when you check in code, when you do your builds, when you do your unit tests... its going to... automatically do the SOA governance processes as automatically as possible... preferably and verify that things are WS-O compliant... ...or that you're not attempting to tunnel verbs through URLs if you're trying to do RESTful services.... Or whatever else your policies might be...

Ron Schmelzer: I agree completely with Anne and all the rest of the folks that were talking about governance. It's an absolute requirement to make SOA work, although its much more about the processes and technology.

The interesting thing is we've been around this concept before... in a completely different context from SOA... ...CRM.... I remember this very same conversation.,. just buying a CRM system doesn't really help you manage customer relationships ...you cant have a customer relationship management application.... And all of a sudden your sales and customer relationships are going to get better... at best it's a tool to figure out how to help you improve.. your customer relationships.... ...if you have bad information, and bad salespeople... and bad customer support processes, even the best crm system in the world is not going to help you... improve your actual customer relationships...

That what the general theme is here. you need SOA governance.. to make SOA effective in an environment of continuous change.. even the best SOA governance products... wont help if you have bad code governance processes...

Anne Thomas Manes: it's a lot like collaboration... ...you can go out and buy SharePoint, or groove or whatever your favorite collaboration tool is... if people don't want to collaborate... no matter how much infrastructure you provide, that's not going to facilitate collaboration... ..people still use email and they abuse it... because they wind up getting all this data that's lost in their email environment? As opposed to collecting it in workspaces...

Ed Horst: what about the commercial products around registries and repositories.. and some of the more successful SOA governance efforts that we've seen have not been built around commercial products... ...more around existing in-house processes, home-grown solutions and almost more of a wiki-style.. dialog around their SOA efforts, and less about putting everything in an elaborate database schema...

the collaborative nature also comes home to roost here... in the SOA governance side of things... there has to be rules that have to be followed, certainly... there has to be give and take... ..the what in it for me dimension....

Joe McKendrick: Let's look at the whole Web 2.0 Enterprise 2.0 phenomenon. Many of our listeners may have a lot of people engaged in rapid application development, or service development via mashups and other Web 2.0 methods. There's a lot of potential for chaos. Should the emerging governance structures we see for SOA be extended to Web 2.0? Should we go to the Web 2.0 folks and say, 'Hi, I'm from the governance committee, and I'm here to help'? Or will this kill Web 2.0 initiatives? Or, perhaps, Web 2.0ers will simply find a workaround the rules and policies?

Anne Thomas Manes: it's funny, because I've had this argument with a number of my RESTafarian friends... who try to convince me if you're doing REST, then you actually don't need governance... at which point I respond saying, are you kidding? Think about how many people have created really, really bad POX applications that they claim to be rest and actually have almost no representation... of the rest principles involved... ...they don't follow any of the constraints, and they're basically just tunneling RPCs to URLs...

Its really important if you want to build really good resources that can be consumed by an application.. so you can very easily very easily mash tem up, or make them... accessible to anything in the world... then you have to make sure you're designing those resources effectively. that just doesn't mean you're just tunneling RPCs through an URL.. that means that you're going through an effort... to say, what are the proper resources to expose, and how do I expose them? How do I ensure that ... I don't ..destroy all of my security systems in place... by enabling direct access into all of these resources all over the place. So I think that governance is absolutely required in a WOA space... or resource oriented space...

Ron Schmelzer: ...the whole Web 2.0 movement... is a movement towards the idea of bringing the community as a whole into the whole collaborative contribution... to the value of the net.. Web 2.0 is more than just webs and blogs and wikis... but also the ... also the YouTubes and Twitters... and all these things that are shifting control of where this content is generated, and the value of that content to the audience...

I would says that's one of the most powerful and amazing things... about the movement toward Web 2.0... At the same time as evolution to software as a service... at the same time as evolution to SOA... open source.. mobile computing... and all this great stuff...

Is that we're moving towards an environment... where computing is being highly decentralized... ...where were not relying IT to be a sole providers... of capabilities... ..but rather were relying more on the audience to create the value... of that content...
In that environment, its not even a matter if governance will support Web 2.0... its already happening..... organizations like Google and Amazon and YouTube.. simply cant manage that environment where they have millions of people...making millions of contributions on a daily basis.. ...without having some sort of environment where they can have... control... at the design ant the change time and the runtime...

This is especially being the case with something as basic as Google maps.. there are thousands of applications now that are dependant... on Google maps API... even though... its not a web-services-based SOA..., its an XML interface... Google cant version that interface... they cant just decide to up and change the way that the function works... if they do, they could be breaking... some highly critical application that's dependent on it...

so now they have introduce governance... into the mix.. there's really no choice... it's a matter of how they enable... Web 2.0... to continue to grow and provide value without making it brittle... and that's really the argument for governance in a Web 2.0 context...

Ed Horst: ...the way I normally have this conversation is I play the consumer.. ...and talk about that type of scenario.. now that I start trusting services... upon which I depend but I am not in control of... the risk of unintended consequences to their changes... the risk associated with things like versioning and performance deltas.. and all of those different things... they start to affect my application performance... and how it behaves...

All of a sudden, I want governance on them, but not necessarily governance on me... the problem is that consumption ... cannot be an ungoverned activity in order for it to work right.... So if consumers don't have their part .... then the producers are lacking the availability that they need....

So if consumers don't have their part ... ..collection of services... without being responsible to... their own contribution... this has to go both ways... and we have to right size it... Core critical business services need a significant more governance activity, and need more visibility... more visibility and constituents than the Web 2.0 application itself...

That Web 2.0 application... you have to properly vent its use cases, and declare use cases back to service providers... without both sides of that, we're in trouble...

The phrase we often use is consumption can't be free beer type of reuse. Its got to be something that's governed.

Jon Michelson: Your versioning of Google map services is a natural introduction into the two flavors of governance. Kind of the design time and runtime aspect, because there's a little bit of both involved in solving that problem...

You're talking about keeping consumers... in dialog in coordination with the providers is a critical part of the design process, and moving forward from that... and similarly on the runtime side of things... ...there's going to become a point in time... where they that they need to eventually come out with release 2.0... ...they're going to have decisions to make... do they set up both sides of services in parallel.. do they try to hide this transition...

regardless of how the mechanics they do on the runtime side of things... its going to be virtually impossible to guarantee that Every dependent consumer of the service... cuts over to the new version on Sunday morning at 6am... or whatever time they pick to do the cutover ... it just wont happen...

So there's a runtime component... to the problem... where they're going to have to run both systems in parallel... version migration to the problem, where you have to run both systems have version migration from dependent services.... Over some period of time... Six months to a year... maybe multiple years even... it may take before everybody gets converted to the new version...

David Bressler: Motivation. If I'm a consumer of Google maps, and my application is working just fine... in fact I have no motivation ever to move off the old interface... but As Google, I certainly don't want to run two things in parallel... so I want to move people over as quickly as possible...

The need is to do something transparently... to break that dependency between consumer and provider.... And be able to migrate people over... so they can get that stuff out as quickly as possible without impacting people at all and recognizing... people will have different requirements.. different versions... or may never even want to at all...

Joe McKendrick: With Web oriented architecture, SOAs will extend beyond the firewall.. companies will not only be using services from their own domain, but services from the cloud. That brings two levels of governance -- governance on provider side, governance on consuming organization side.

Dave Bressler: I look at things from three different perspectives. Its always about being a service provider. I like to break it down to Three different views of the world. There's companies that make services, and provide those services as a service internally to their organization.

Then there are those that do it more of a traditional B to B perspective... it's a one-to-one relationship. Each relationship from a process perspective is individualized... Then you have a true SaaS type of solution... whether it's a Google maps or Salesforce.com...

Of course in all three of those cases, you have the perspective. If I'm the consumer in that situation, or if I'm the producer in that situation... in all three of those cases... ....so you have the table with six little boxes... requirements.. motivations.. what some of the drivers and objectives... of doing something are...

when Google changes... if I'm the consumer, I want to make sure I'm not impacted... as an enterprise.. if I have a dozen different applications that are using my salesforce interface linking to salesforce.com....I don't want to have to manage.. a dozen different teams at the same team...

if I could somehow abstract that out... do some sort of virtualization, some sort of governance, whatever you want to all it...

it optimizes the way that I used hat service as a consumer... similarly, salesforce.com wants to have a way to govern... a version or control the way their consumers do things with them..

Frank Martinez: I think that's a really good way to think about it. especially from the operational governance perspective, runtime governance. There's certainly some opportunities around our planning activities. When you look at this, it really just goes beyond just runtime governance, design time governance or change time governance. You also have to include plan time as well.

When you're looking at those external relationships... Part of those governance decisions... is deciding when its more efficient ... to leverage externalized capabilities versus internal capabilities.... what sets of criteria when it makes sense... from a business perspective to do that...

ensuring that we actually take advantage of governance opportunities is equally as important... throughout our planning activities... ...it shapes the decision tree... ..how we achieve the appropriate business outcome...

and then making sure that we have mechanisms in place... that help to insulate us from the prospective changes that other parties might be imposing on us... ...is equally as important... and then leveraging that ... things we produce and we own.. within our own administrative boundaries... social boundaries, political boundaries...

so the same thing has to be done for capabilities... that were taking advantage of outside of our own corporate balance sheet, and tap into ....the capabilities that are provided by others...

Ron Schmelzer: to answer your question about Web oriented architecture... and the use of third-party services, external services in the context of governance... I think we should realize that's going to be less the exception and most likely the rule..

..what does it mean in the current economic climate... to really have internal services... if a year ago if you told me that Lehman had internal services for internal use, I would have believed you... ..I don't think that Lehman's customers are necessarily going to want their capabilities to just disappear...

... the current reality... of the current economic climate.... Is more on increasing distribution and use of resources that are location independent, provider independent... probably infrastructure independent.... We cant really count on the fact that we can make a decision... even about what platform we'll use... and have that be constant over any reasonable period of time...

in the area of continuous change.... that's the value of SOA... In that environment of continuous change.. abstracting these various... enabling the business... without having IT to be a primary roadblock... within this environment of continuous change its very hard to enable... this change without building services, and to enable these services without having governance... That's why governance and enterprise architecture in general go hand in hand...

Ed Horst: ...Ron your case in point about real time combining of companies. I'll just got everything from one vendor approach, and that will make everything smooth and efficient... it really points out the flaw in that... usually from an M&A perspective, that throws that all up in the air... an architecture that's defined as homogeneous... oftentimes has serious problems.

The biggest value that's placed on Lehman was the IT infrastructure... article of how Lehman assets were valued... that's the biggest value... that's described there...

Ron Schmelzer: the management team can go, but the IT team has to stay...

Ed Horst: It all comes own to, Whether it's a Web 2.0 thing, or in-house SOA thing or whatever, it boils down to answering the what's in it for me question. There's the obvious incentive kind of approach, you actually pay money to people for their behavior.

It can be as simple as providing visibility, just being able to report accurately what reuse is actually happening and actually getting deployed and reused. Almost turns it into a bragging rights kind of thing. Its almost just personal bragging rights, for one organization versus another for how often services are reused and not another. So the incentive doesn't always have to be monetary.

(A summary of the panel discussion is also available here. An archived replay is available here.)

_____________________________________________________________________