SOA in Action

What is this thing called "Master Data Management" that everyone is talking about, and what's the connection with SOA?

MomentumSI's John Kalogirou connects the dots for us, noting that MDM -- data shared across disparate IT systems and groups -- encompasses a wide range of data sources and systems.

Services access information from a data services layer. A data services layer provides an abstraction layer between producers and consumers of data. The data services layer presents consumers with a virtual aggregated view of data from multiple data sources in a consistent and centralized fashion. The layer's interface supports all consumers (human, application, external parties, or business services) while providing agility to data source providers.

A data service layer offers many benefits. Consumers are insulated from complexity, location, and changes in source data systems through abstraction. Providers have the flexibility to change underlying data schemas without impacting consumers through abstraction. Companies can centrally manage, monitor, measure, and report on the enterprise view of the data and metadata.

John observes that the technical intersection of MDM and SOA occurs at the data services layer. However, wjile MDM and SOA share common design principles, they inherently don't mesh. "MDM typically doesn't embrace SOA's 'loose coupling' principle." He adds that "MDM practitioners contemplating supporting today's SOA systems need to become familiar with SOA standards and strive for loose coupling with external systems. Eliminating point-to-point interfaces and replacing them with service-enabled integration minimizes the impact of changes from integration partners and consumers."

Questions to be asked include the following:

- Does the MDM support extensible data types such as XML?
- Can internal, partner, or client services search, identify, and consume the CDM [canonical data model]?
- What is the effort to replace an external service provider?
- Are data quality and conformance services exposed for use by applications or external parties?
- What is the turnaround for changing the functionality of the current MDM system?
- Can the MDM handle near real-time requests for conformance from participating systems?
- Can the MDM exchange metadata with other MDM systems?
- Can the MDM infer context and take action based on the semantics of the information being exchanged?

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

One of things I try to accomplish at this blogsite on a regular basis is relay stories of outstanding SOA successes, and the unique challenges organizations had to overcome to get there.

The latest SOA success story of interest to come over the wires comes out of Richmond, Va.-based Southern States, a coop owned by more than 300,000 farmers, selling farm supplies in 1,200 retail stores in 23 states.

The business challenge was price lists at the cooperative's retail stores -- stored in Oracle’s OneWorld ERP application, a homegrown point-of-sale application and an online catalog -- had to be manually updated by store employees, Karen Lankford, vice president of information systems, is quoted as saying. This created accounting headaches in trying to reconcile data and made it very difficult to quickly adjust prices to meet changing market conditions.

The company is rolling out a collaborative SOA-based modeling toolset to its 50 developers, which enables them to build and link Web services. The first service allows product pricing data stored in multiple systems to be changed easily.

Lankford estimates that this first service is helping the organization generate about $1.4 million in additional revenues a year. This is an interesting point, since the value proposition of SOA and Web services to date has been in cost savings and productivity, versus actual revenue generation. However, the article does not spell out how the price listing service translated into the additional revenues. It's likely that that employees had difficulty keeping up with market changes that affected the pricing of goods sold through the retail outlets, and as a result, many items may have been sold at lower than market rates. Having a service that monitors such fluctuations and automatically making changes across the entire network would keep things in line.

One of the biggest challenges to the SOA/Web services implementation at Southern States had been training developers to reuse code from other parts of the organization, Lankford told ComputerWorld. She noted that the company's newfound ability to reuse code will mark a “dramatic change” for developers. The coop must conduct “a campaign to convince people that this is the direction we are going,” she said. The company will need to have a few more visible internal “wins” using the software to help convince the developers that the tool can help ease application development work.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

Does your business keep expecting the same kind of ROI from SOA as they get from point-to-point integrations, even though you try to explain until you're blue in the face that SOA is anything but P2P?

Join the club. Unfortunately, P2P integration is less invasive and doesn't mess with business processes, and as a result, does not bear the up-front costs that SOA may incur. Marc Rix, a senior SOA/BPM architect, has just released this white paper that discusses describes the differences between P2P and SOA-based integration, and explains why SOA ROI is vastly different from P2P ROI.

SOA is uncharted territory, "so in the absence of objective SOA performance data, the industry needs a simple model that compares the distinguishing characteristics of SOA and the P2P approach on a fundamental level so that we can make more objective judgments about the fit of SOA in our enterprises," Marc says.

Marc separates the economics of P2P and SOA this way:

Point-to-point offers short-term savings, but long-term pain. "Point-to-point architecture carries seductively low up-front costs, resulting in little impact at the project level... However, microeconomic stability can quickly fester into macroeconomic chaos. As more and more one-off solutions are piled atop one another, IT maintenance costs swell exponentially and connection sprawl strangles the infrastructure."

The costs of individual P2P connections remain constant no matter how often the process is repeated -- therefore P2P costs grow linearly, Marc points out.

SOA means higher up-front expenditures but long-term savings. "Higher up-front expenditures are required in order to ensure loose coupling between linked components, which inflates project budgets. This translates into initial financial losses at the microeconomic level and often catapults projects beyond conventional thresholds of risk tolerance. However... investments in SOA at the microeconomic level can blossom into colossal benefits at the macroeconomic level. By dramatically reducing the number of physical connections in the infrastructure and leveraging no-cost, virtual connections to achieve everything-to-everything connectivity, SOA’s return on investment increases exponentially as the infrastructure grows."

Unlike P2P, each physical connection in the SOA network links one node to all other nodes via virtual connections through a hub. This means that the value of the SOA network increases exponentially as its size, or number of nodes, increases, Marc relates. Thus, "the value of the SOA network is greater than the sum of its parts."

A link to Marc's white paper can be found at his site here.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

Talk about integration challenges. I recently spoke with a number of insurance companies about their SOA strategies for aging policy administration systems for a new report in Insurance Networking News. In some cases, company policy administration systems were still running on mainframes that were almost four decades old.

"I suspect that, like The Hartford, there are many companies that have aging, legacy platforms," said Rich Maynard, property and casualty chief architect for The Hartford Financial Group. This says a lot for the staying power of mainframes, which continue to this day as the workhorses behind 80 percent of the world's data. And, in the case of The Hartford, its commercial lines system is chocked full of logic and applications that gives the carrier a leg up on its competition to this day.

The company will modernize some mainframe-based policy administration systems that are now decades old, while replacing others. In some cases, the commercial lines systems provide competitive advantage, while the consumer systems are more of a commodity application that will be replaced.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

What's the most typical reaction from business managers upon hearing proposals for SOA-enabling their processes? Eyes glazed over?

Look to the experience of a major aerospace manufacturer, with huge backend integration challenges, which managed to bring 100 interfaces to its SOA layer from all parts of the business.

As recently reported in IT Business, Bombardier had a bevy of heavy-duty systems to bring together into a common service layer. Bombardier uses SAP's R/3 ERP system, NetWeaver, and SAP's business warehouse and business intelligence technologies. The aerospace company also uses Lotus Notes for collaboration, EMC's Documentum for content management and technology from Sun Microsystems, Juniper and CA for security services.

A crucial milestone in the SOA project was the creation of the Bombardier Manufacturing Information System, which entailed the integration of eight manufacturing systems resulting in 64 mission-critical real-time interfaces, according to Sean Terriah, manager of IT applications services at Bombardier. The company accomplished 100 mission-critical interfaces using 14 protocols, 10 messaging formats across internal and external systems and network topology.

An additional 40 real-time interfaces were created as part of an extended enterprise that integrates with Bombardier's logistics partners, including Caterpillar Logistics, UPS and Fedex, as well as with the U.S. Customs and the German Customs.

There is a crucial element to Bombardier's success with SOA, said Terriah: His department didn't call these efforts "SOA." Instead, the discussion always evolved around business requirements. "When we talk to the lines of business, we don't start off with the discussion about SOA, “ he explained. “We see SOA as important for our IT group and it's about our approach to developing new solutions…and the lines of business will just benefit from that approach.”

Bombardier is a good example of a company that made SOA an enterprise-wide approach, rather than a siloed IT project.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

Large fish are eating up all the smaller fish -- and that's our loss.

That's the view of Dave Linthicum, who recently took a look across the SOA vendor landscape, and is a little concerned about what he sees -- or is seeing less of.

Dave, who has been part of plenty of vendor acquisitions himself, wonders if with all the acquisitions that are going on, we are losing an innovative edge. "The number of organizations out there that are actually doing service-oriented architecture are diminishing," he said in a new podcast posted here at ebizQ. "This is not due to lack of interest in the space, or a failure of the technology... the larger players out there, the big stack guys, are actually purchasing SOA companies and taking them out of the market."

"Smaller and more innovative SOA vendors are quickly finding themselves as part of larger players," Dave explains. "There were probably three to four dozen acquired just in the last couple of years."

So what's wrong with larger players paying out megabucks for smaller specialized SOA companies? Doesn't that demonstrate how vibrant things are in this space? The problem is, Dave says, is that "most of the innovation, the interesting stuff, takes place among the startup companies." This is being lost as larger vendors take over, and absorb the smaller vendor's products and people into the bellies of their operations. Large vendors, of course, don't like to take risks, and are slow to adapt. Only small startups are in a position to take risks with new technologies and approaches.

Isn't this a problem that every generation has faced? Sure, but things are different today, Dave says, because it's much more difficult to take a company public, due to regulations. The only viable exit strategy for investors is selling the company to the highest bidder.

Managers and professionals need a lot of guidance in service oriented architecture, and large vendors aren't doing a good job of providing guidance, Dave says. "There are lots of things that need to be addressed currently in the SOA marketplace," he said. "I don't see vendors leading the way. As of now, I see them leading the hype. Lots of people are going out there and buying technology without having any idea how its going to fit within their enterprise."

Posted by joemckendrick in SOA  |  Permalink  | Comments (2)  | TrackBacks (0)

Can we, should we, automate SOA policies that are unenforceable by other means?

In my previous post, I discussed the results of a new Aberdeen survey -- as well as ebizQ's own survey from last fall -- which demonstrated advantages when automated SOA governance is put into place.

In a new post, ebizQ's David Kelly observes that automation is the key to effectively managing complex SOA environments. "Like many other parts of the IT infrastructure and software development lifecycle, the more you can automate your SOA governance, the more effective and efficient it will be. It will be easier to enforce, easier to use and will provide greater payback than a non-automated process."

Some governance areas that could be automated include the configuration of new services for security and management, automatic compliance checking to verify whether a new or modified service meets specific compliance requirements, propagation of policies, and automatic detection of dependencies and the relationships among the different services and systems in an SOA environment.

However, in a response to David's post, Todd Biske argues that automating governance essentially "is putting the cart before the horse." He notes that the purpose of automating SOA governance is policy enforcement, which is too much of a hardline approach to SOA.

"Governance isn’t just about enforcement," Todd says. "Governance is about people, policies, and process. People (the legislators) set policy. Policy is enforced through process. That may be a gross oversimplification, but it’s the way it needs to work."

Todd cautions that enforcement often fails "because the people, the policies, or both are not recognized." Even if you automate enforcement, he adds, "if the policies and the authority of the people setting the policies aren’t recognized, your governance efforts are less likely to be successful."

I've heard it said that an uber-governance approach will scare away the very people you need to sign onto an SOA program. In fact, too much governance results in end-user groups performing end-runs and workarounds that bypass the "official" infrastructure -- and severe underuse of the infrastructure. And that means money down the drain.

Enforcement -- whether its a city government or an enterprise architecture board -- is a cultural issue, Todd explains. "Tools can make non-compliance more painful, but they can’t change the culture. If you’re having problems with your governance, I’d look at your people and policies first. If you’ve got recognition on the authorities and the policies, now you can look into minimizing the cost of your enforcement through tooling. It is certainly likely that automated tools will be less costly in the long run than having to schedule two hour meetings with your key legislators for every service review."

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

How much of a difference does effective governance make in SOA? Apparently a big difference.

Aberdeen Group has just completed and published a survey of 950 companies, and concluded that "between a third and half.. are having serious difficulties getting their SOA-enabled applications into stable deployment."

Not everybody is having such trouble, though.

What the research group calls "Best in Class" companies -- the top-performing 20% of the respondent base -- are far more likely to be reporting positive ROI on their SOA investments, as well as lower application development costs. Sixty-eight percent of these companies are reporting good results from SOA, compared to only 23% of the overall survey group.

Of course, SOA success -- especially positive ROI -- is a difficult thing to measure. The Best in Classers may have already had the measurement mechanisms in place to see the success, while the rest of the bunch may be having some success without realizing it. (Or the opposite.)

But the Aberdeen survey also looked at the best practices of the Best in Classers, and found one common denominator, which is highly significant. Almost all had solid governance structures in place, and many had automated this process.

Aberdeen observes that the BIC bunch have implemented design-time governance and re-use policy to minimize lifecycle service costs compared to 26% overall. In addition, more than 80% of the BIC bunch has implemented an automated solution to SOA operations and governance.

Last fall, ebizQ conducted and published a survey that also demonstrated the advantages of automating as much governance as possible.

The survey of 313 companies found that those sites that have runtime and design-time automation are far more likely to report high levels of comfort with their governance solution than those who rely on manual enforcement.

As ebizQ colleague Beth Gold-Bernstein explains it: "It turns out that the biggest differentiator between those that feel their governance solution is sufficient, and those who believe their governance solution is not sufficient, is the degree of runtime automation. Those that have runtime automation have an automated solution that is managing governance policies whenever these services are being accessed and run have far higher levels of confidence." She added that "while design-time automation is good, runtime automation is going to be absolutely essential as organizations move down the path to SOA."

A Webinar on the complete ebizQ survey is archived here.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

It used to be that when we needed help with a work-related problem, we stuck our head over to Jeff in the next cubicle, or made a phone call to a trusted colleague. Now, when we have an issue, we go to a global community of experts, advisers, and knowledgebases. And, likewise, we assist others across the globe.

Welcome to one of the revolutionary shifts in our work brought on by 'wiki-nomics,' in which the world has become a gigantic virtual workplace.

My ebizQ colleague Gian Trotta just posted a compelling interview with John Schmidt, chairman of the Integration Consortium, and senior vice president for enterprise architecture at Wells Fargo Bank, who talked about the implications of wiki-nomics.

Schmidt compared "wiki-nomics for architecture and integration" to "enabling outsourcing at the individual level."

Personal outsourcing -- what a cool concept.

Of course, the term "outsourcing" causes shudders to run up many IT professionals' spines, but Schmidt said he's not talking about sending tasks to lower-paid workers somewhere else on the globe. Unlike the traditional model for outsourcing -- firms contracting out functions or processes to an outside firm -- "individuals are starting to outsource their problem-solving and their own professional development," he says. "They're leveraging things like wikis, blogs, other collaboration events to collaborate in real-time with other individuals."

In today's world, IT professionals go to Google, Wikipedia, and other online sources of support, Schmidt says. "They write out their question in their blog and look for their community to respond and help them. ...they extended their network of peers to outside the four walls of their company. ...they're taking their problems and their professional challenges to the world."

In the same interview, Schmidt also talked about the growing confluence of "integration" and "architecture," which, he says, "have always been two sides of the same coin.... architecture is in fact a key discipline and aspect of any kind of integration in terms of trying to make organizations work together effectively."

Schmidt pointed out that many organizations attempt to break big integration projects into bite-sized segments, which often results in siloed chaos. Architecture can keep all these project pieces aligned, he believes.

The complete interview can be found here.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

Full-fledged SOA adoption -- and even ESB implementations -- is still only a reality at a minority of companies. However, things may be ready to change, and soon at that.

Adoption of SOA is expected to double over the next two years, according to Evans Data Corporation’s recently released Corporate Development Issues Survey. I authored the final survey report for Evans, in which we concluded that interest in SOA is running at high levels. Close to a quarter of enterprise-level developers indicated that they already have service-oriented architecture in place. Another 28% plan to do so within the next 24 months.

Surprisingly, adoption of enterprise service buses is trailing that of full SOAs -- I had expected ESBs to be in the vanguard of SOA early deployments. Currently, 15% of companies have ESBs in place, a number that will more than double during this same time.

Another finding I've seen in some previous studies is that at least four out of ten corporate developers say Web services approaches (we'll include SOA methodologies in there) will significantly reduce application and systems integration costs as adoption grows. This will provide relief to many organizations, as the survey found that one out of five enterprise run mission-critical applications that are more than a decade old.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

ebizQ colleague Beth Gold-Bernstein provides a great analysis of Software AG's announcement that it was buying integration vendor webMethods.

Beth points out that the combination of the two companies' technologies serve up quite an offering, addressing governance, BPM/BAM,application composition, application integration and legacy modernization.

As Beth puts it, "After many years of observing this industry the sad truth is the best technology does not always win. Success much more depends on market strength. Maybe in this case the acquisition will create a stronger company which will propel some leading technologies to the forefront of the market and challenge the other vendors to step up."

However, Neil Ward-Dutton, also posting here at ebizQ, is a little more cynical about what it all means, opining that Software AG wasn't looking for technology, but rather, a foothold in the North American market which webMethods has in abundance.

My view is that Software AG, which has a lot of experience in the legacy systems market, recognizes the potential SOA has in extending and modernizing these systems. webMethods has been strong in this area. Legacy-to-SOA is a vast, and still largely untapped frontier, and ultimately, what will drive SOA adoption.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (1)

One of my favorite quotes about SOA came from Susan Certoma, CIO of Wachovia Bank, who oversaw the deployment of a huge SOA project across the bank's Corporate and Investment Banking division.

“It’s like a roller coaster. You go up and down, throw up a couple times, and at the end of the ride, everything’s great.”

How many times have you thrown up so far?

A new comprehensive account of Wachovoia's SOA has emerged. Financial Technology reports that Wachovia set out to transform its entire Corporate and Investment Banking division through SOA.

This suggests the effort had a lot of top-down support from the executive suite, versus the most oft-taken route to SOA -- building up gradually and incrementally from smaller quick-ROI projects.

Still, the project was not imposed on the organization as a massive, turn-the-organization-upside-down project. Rather, the effort was delivered in bite-size pieces. "Funding and support was built on trust and incremental deliverables," said Tony Bishop, SVP for the CIB technology group. "It was, 'Here is the plan -- each year we are going to deliver increments. If we do, then you will fund the next piece.'"

First, Wachovia integrated its front-end services to create a common interface for sales, trading and banking, followed by services for trade execution, positions, offerings and connectivity to markets.

Wachovia's CIB unit measures its SOA results with two indicators: time to market and the number of reusable parts. So far, he says, it has experienced about a 40 percent cost avoidance over three years using SOA rather than a traditional technology architecture, said Bishop.

Bishop's advice: "You've got to focus. You've got to be committed. You need to do it in incremental building blocks. SOA needs to be business-aligned and needs to have a top-down map and a bottom-up approach."

I imagine this helps ease the ups and downs of the roller coaster ride as well.

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)

A couple of months back, Greg the Architect was taken on a dizzying ride through vendor FUDland as he attempted to find a suitable vendor to initiate his company's SOA effort. (View the video here.)

Now, the same makers of the original Greg video (underwritten by IONA) have brought us another account of Greg's efforts to sell SOA to his CIO and the suits upstairs. The catch? Despite Greg's herculean efforts to transform the way IT interacts with the business, the suits don't seem to get where the ROI is.

Greg gives it all he's got to try to help them understand, with hilarious (but oh so true) outcomes. (Watch the latest video here.)

Posted by joemckendrick in SOA  |  Permalink  | Comments (0)  | TrackBacks (0)