Extremely Distributed Web Services

By Richard Karpinski

So many early Web services projects have been -- somewhat surprisingly -- behind the firewall efforts that use these standards-based protocols to link enterprise applications.

So when National Student Clearinghouse, which aggregates enrollment information for colleges and universities, decided to build Web services interfaces that could literally touch thousands of end-points, it ran into some challenges few enterprises have faced.

The information-intensive company -- with just 45 employees and only a handful of IT staff -- is nonetheless making a major bet on Web services. In the process it has learned lessons about Web services security, management and provisioning (vs. dynamic discovery) from which other enterprises can clearly benefit.

So Where Did You Go To School?

It would seem like a no-brainer for employers, credit card companies, or insurers to determine whether someone is currently enrolled in college. But it's a bit more complicated that that.

There's also plenty of dollars at stake. For instance, lenders have more than $30 billion in student loans out at any time, and they can't collect until students leave school. Insurers can't move students off their parents' policies until they graduate. And companies spend millions of dollars doing background checks on education claims on resumes.

The National Student Clearinghouse was founded as a nonprofit organization in 1993 to tackle these problems. Initially, schools sent paper versions of their records three times a quarter. Later, some schools moved on to FTP'ing flat files and some bigger universities even moved to EDI transactions.

But all of these methods proved less than efficient as the status of student enrollment sometimes changes day to day. The schools could track those changes in their internal systems, but the data remained locked up there.

"This is truly a business that can take advantage of Web services to make our business grow," said Mark Jones, NSC vice president of marketing and business development.

Jones has a deep background in such distributed data challenges. He once worked with IBM's EDI unit, and later moved on to one of its customer's National Computer Systems, where he oversaw that company's conversion to EDI over the Internet. Later he worked at integration vendor Mercator, where among other projects helped sell and support the B-to-B system that runs Amazon.com's back-office operations.

Real-Time Web Services

Initially, Jones focused on opening up browser interfaces to its database, allowing schools and commercial customers to add data and make queries via a Web browser. That certainly provided a nicer interface, but still it wasn't exactly real-time.

Jones saw a clear opportunity to leverage emerging education and human resources XML standards, SOAP-based integration mechanisms, and the Internet as a transport vehicle. The company's developers began working with Web services tools from the Apache Group to build Web services interfaces to its internal systems, leveraging standards like XML, SOAP, and WSDL. NSC created a little Web services "starter kit" to help customers build their end of the Web services connection. "If a school wants to become a real-time provider [of data], they have to implement a Web service and then we ping and get the data from them," Jones said.

Overall, that part of the equation was a no-brainer, Jones said.

"Putting up the Web service was relatively simple, a couple of calls to the database," he said. "It was a simple programming effort that even our skinny staff could figure out."

The real challenge came, however, when Jones and his team began thinking about rolling out the Web service to thousands or potentially even millions of customers.

Today, more than 2,700 colleges representing 91 percent of the nation's enrollment participate in the Clearinghouse. NSC estimates that more than 6 million different commercial entities might like access to its data as well. Overall, the company stores more than 70 million records in a massive Informix database; third parties access the registry more than 100 million times a year to verify student enrollments.

To handle that application and data management challenge, NSC turned to vendor Flamenco Networks, whose Web services management platform provides the security and network management capabilities the company's Web services deployment required, Jones said.

"Even if I could technically provision the server side of this, I still had to worry about getting all of these connections hooked up, getting monitoring in place so I could now how many connections were live, not to mention security. Just managing the certificates is a major deal. Immediately this became a huge problem," Jones said.

Flamenco's network-based approach to managing and securing Web services was a good fit for NSC's massively distributed Web services deployment. All XML messages flow through Flamenco's system, which handles security and provides a real-time management view into the transaction flow to help NSC keep everything up and running.

Provisioning Vs. Dynamic Discovery

But the biggest hurdle to rolling out the Web service widely was provisioning new connections -- and that's where the conventional wisdom surrounding Web services began to fall apart.

Much of the early thinking around Web services was that companies would simply publish their Web services interfaces, for instance, to a Universal Description, Discovery and Integration (UDDI) server. There, as per the acronym, other companies would "discover" the service and link up to it.

NSC, however, found it needed to manually provision its end-point users rather than follow this publish/subscribe model. "UDDI, I could care less about it. It has no relevance" to his project, Jones said. Instead, NSC relied on provisioning tools in the Flamenco product to set up customer links into NSC's Web service. The provisioning process is anything but straightforward, encompassing multiple steps involving service-level agreements, transport protocols, and an array of security specifications, all automated using the Flamenco network, Jones said.

"Connection provisioning seems so obvious now, but we didn't fully appreciate the importance of it," said John Hanger, Flamenco's senior vice president of sales and marketing. "What you really need to drive the business benefits out of these services is a way to systematically get those connections out to people. You can't just throw it out on a UDDI server and expect to get business from it."

Hanger said the need for provisioning doesn't discount the value of more dynamic discovery approaches to Web services. "It is not a repudiation of this approach," Hanger emphasized. "It's just a very clear indicator that we've seen almost universally that when companies initially focus on Web services, they're not doing dynamic discovery but are simplifying integration with existing, trusted trading partners. And that's a little more static in nature."

NSC has begun using the Flamenco platform to roll out the Web service to its first pilot customers and will expand the implementation as time goes by. Right now, the company is in a hybrid stage with a variety of interfaces into its data, some of which it aggregates in its own database and some of which it has begun accessing on a real-time basis at college sites.

"It's going to be a transition," Jones said. "Five years from today, we'll be much more about Web services than today."