E-Commerce Architecture: Schwab Makes The Trade

By SAROJA GIRISHANKAR

The Internet has been good to Charles Schwab & Co. Inc. The brokerage-known for its no-frills discount services in the '70s-is dancing to the Internet tune in the '90s. It hasn't always been a smooth waltz, but the results are an impressive spectacle of leaps and fancy footwork.

With 41 percent of its trades now online-half of that going over the Web-the company has 1.52 million active online accounts valued at $112 billion.

If Schwab is writing the book on online trading, it's a lesson in swift business decisions fully supported and embraced by top management, followed by technical breakthroughs delivered by a savvy IT team that's in step with management and its vision. Neither budget n or time crunches seem to daunt the team effort.

In fact, Schwab is now putting in place an IT infrastructure-including Java-based applications-to handle what it projects will be phenomenal growth for the next decade as well. By 2000, it expects 80 percent to 90 percent of trades to occur online and over the Web, with a mind-boggling rate of 4 million transactions an hour. Quite an ambition for a company with $407 billion in assets.

Many companies struggling with less lofty electronic-commerce goals may ask how such projects really are achieved. InternetWeek asked, too. On the following pages, we trace Schwab's successes and missteps in the past two and a half years as it built an electronic trading legacy.

Breaking Through The Code

It was the first week in October 1995. All that James Chong remembers is that it was a sunny day in San Francisco. So was the mood inside the understated computer laboratory on the eighth floor of 101 Montgomery St.

For a change, the small group of prog rammers had a bit of fun and were exchanging high-fives over a fairly simple programming chore that was not even on their agenda.

Little did they know that this "simple" programming task was to change the strategic course of their financial services company and the industry.

Thirty-seven-year-old Chong, who was then vice president of transaction processing architecture, and his crew of five programmers had just completed the prototype of a Web front end for Schwab's back-end trading system. Chong now calls it a "quick and dirty job" that was done in two weeks and was, in fact, an exercise of sorts to relieve the rigors of a bigger and more strenuous programming project.

Meanwhile, outside of Schwab's world, the Internet was becoming a reality, and, like other corporations, Schwab established an informational corporate Web site.

Times were competitive, and some of Schwab's rivals-Lombard Securities and E*Trade Group Inc.-were starting to offer trading on the Web, but no one knew whether such services would take off.

Security was another major issue. Although Chong and others knew that company chairman "Chuck" Schwab was tracking competitors' road to the Web, he had issued no mandate for Web development efforts.

For the most part, they were urgently focused on building component-based middleware that would link the brokerage firm's back-end host trading system with specific financial services software, such as equity trading. Indeed, the two-year middleware development, which allowed for a plug-and-play component architecture, would eventually become the key factor in the Web trading application.

In fact, that routine week in October eventually transformed the brokerage firm into a leading Web-trading powerhouse.

Dubbed Sentry, the middleware linked business components such as buying and selling equities to the mainframe-based host trading system.

The goal was to add brokerage and financial services quickly through the reuse of business components while shortening the life c ycle of software development.

"We thought the Web software was cool and decided to show it to our CIO and executive vice president Dawn Lepore. But then she decided to collect a few people on her way to the lab," remembers Chong.

Lepore did not bring just a few other programmers. She brought David Pottruck, Schwab president and co-CEO; Chuck Schwab; Roger Neaves, vice president of infrastructure; and Ken Richmond, the project manager of the Sentry host-off-load product. The rest is history.

"Chuck [Schwab] was a little bit surprised that we had created a browser front end. And when he placed and completed a trade for 100 shares of Schwab from a test account from the browser, he got very animated," says Chong.

The executive team stayed at the lab for an hour while Schwab grilled them about the viability and security of Web-based trading. The discussion continued, with Pottruck saying that the risk was not necessarily from outside hackers but from people slipping through the firewalls and accid entally impacting the integrity of databases.

Within six weeks, Chuck Schwab decided to take an aggressive and ambitious leap to the Web. Chong takes little credit and humbly credits the chairman for having the corporate vision. "Chuck had the seed, and we just poured water on it," says Chong.

Dancing For The Chief

Once Charles Schwab saw that secure Web trading could become a reality, wheels started turning fast. On Oct. 7, 1995, four people met at the first brainstorming session to evaluate the Web development. The party included Vincent Phillips, vice president of Web systems; James Chong, vice president and chief architect of the IT group; Ken Richmond, project director of Sentry middleware; and Cynthia Alley, director of electronic brokerage product development.

Impressive as the gathering was, they realized that they had neither the technical skills nor the time to take up a Web development project. So, the consensus was to outsource the task and immediately issue a request for proposal. "It was a hard decision to make because we were a strong engineering team," says Phillips. A month later, on Nov. 7, the RFP went out.

Simultaneously, in late November, Schwab recruited a top gun who knew the Web business and could get cranking on the mission. Gideon Sasson, vice president of information services at IBM, was responsible for Big Blue's proposed Web business and E-mail services. Sasson was anointed as Schwab's executive vice president of electronic brokerage in the last week of November, before such a business existed.

Sasson's honeymoon barely lasted three days before he and his team found themselves in the hot seat: They had underestimated Chuck Schwab's enthusiasm. While Sasson and others were at an off-site meeting at the nearby Clairmont Hotel, they got a call from the boss. "He said Web trading was an important strategy for Schwab and he wanted to launch it in 60 days; on Valentine's Day," recalls Phillips. "When a chairman calls, you put on your dancing shoes."

B argaining with the chairman is never easy. But the Schwab team managed to get another 30 days grace and agreed to deliver a Web trading system on March 31.

The project was dubbed Cupid for the original Valentine's Day launch. Meantime, the RFPs from three major vendors-IBM, MCI and Sun Microsystems-and three smaller vendors came back. Most required nine to 12 months and cost more than $2 million. Schwab did not have the time. "We had no choice but to do it ourselves," says Phillips.

Battle plans were drawn: Chong's team would deliver the component APIs and the Sentry middleware. This would link actual trades with the CICS back-end trading and transaction monitor system by encapsulating and converting TCP/IP data to IBM SNA formats. The architecture would have to allow for linear addition of Web servers on the front end.

Phillips' crew would create the Web trading application using C-based CGI Script and HTML, with some use of Dynamic HTML to build templates to reduce repetitious chores. Schwab chose Netscape Communications' Enterprise Server suite as its platform. The team worked on both Netscape and Microsoft browser front ends; 16-hour days became routine. "We lived and breathed the Web software."

Let The Trading Begin

And then there were hiccups. The first code drop, scheduled for Feb. 7, had bugs in the equity trade software. "All day long, we had all kinds of errors on the trading screen," Phillips recalls.

So, the code was not ready, and Phillips was called into Sasson's office on a Friday afternoon. The scene was not pretty. Sasson wanted to reschedule the code drop and all the other schedules.

Somehow, Phillips persuaded him to extend the deadline until Monday. All six members of the Web team worked all weekend, getting the barest sleep. But, the work got done.

Feb. 26 found the team doing integration testing between the Web system and the middleware, but a lot of work on the infrastructure was still pending. New routers, firewalls and Domain Name Service server s had to be added. Internet services from MCI and BBN Planet were being added.

On March 7, Schwab started testing the Web trading within the company, increasing the number of people to several scores.

D-Day arrived on March 31, and a nervous group of 20 people gathered at 6 a.m. in the conference room outfitted with the actual Web trading server. A preselected group of 50 Schwab customers were ready to trade over the Schwab server.

Bagels and doughnuts were untouched although Alley and Phillips drowned themselves in pots of coffee. At 6:30, the markets opened on the East Coast, and the crew saw trade after trade go smoothly.

"It took us 30 minutes before we realized all systems were go, and it was party time," says Phillips.

The job was done, complete, finito. And the cost? Less than $1 million and in only three months as opposed to the nine to 12 months and the $2 million that outsourcers had asked.

But then Schwab had yet another surprise: huge demand. The company originally exp ected to ramp up 10,000 customers to use its Web trading service by the end of 1996, but that number was reached by the end of May. Once again, the Schwab IT team was sent into a tailspin.

Too Much Success?

Once the anxiety of the launch subsided by the summer of 1996, Phillips' team found itself paying a systems overhead price for its success, adding Web servers on a daily basis at times, going from three in April to a current 50.

Adding 47 Web servers required expansion to the network infrastructure. "We went from two Cisco 7500 routers to eight 7500 routers and on the LAN side, moved from shared Ethernet to 100 megabits Ethernet switching," says Roger Wong, Schwab's technical director of communications services.

In all, Schwab's massive TCP/IP global network uses 550 Cisco routers and several unspecified Catalyst 5500 switches. Metropolitan ATM services link core urban regional offices. A potpourri of network and systems management wares from IBM's Tivoli Systems Inc. unit, Hewlett-P ackard's OpenView, and other performance and network sniffers keep tabs on network health.

April 30, 1996 is another date that Sasson remembers well. The security measures for the Web servers-Netscape's Merchant Servers-required VeriSign's digital certificate technology to comply with the Secure Sockets Layer standard.

"Because the Web service grew in two weeks to the level that we expected to happen in a year, we had major challenges;

especially, we had to outfit eight Web servers with VeriSign's digital certificates. They said they needed 48 hours, and we didn't have 48 hours," says Sasson.

Sasson used his clout as a huge customer and phoned VeriSign's president and CEO directly. The result: Schwab got the digital certificates in a matter of hours.

In fact, with the stock market on the rise, in the first week of May 1996 volume went through the roof and Schwab had to turn away customers.

The success was too much too soon. "You don't turn away customers because they might go to c ompetitors" and that was unacceptable to a company already playing catch-up with Lombard and E*Trade. Sasson again used his connections and turned to IBM, which was supplying the server hardware. IBM had readied SP/2 AIX servers that let new processors be added. In no time, 16 SP/2 servers were installed at Schwab, easing the bottleneck traffic.

Simultaneously, Schwab developers began updating their legacy systems to Java applets that would add richer and livelier interfaces for customers, compared with the flat and noninteractive HTML interface. The Java applets also would allow for newer services, such as Asset Allocation, that would walk customers through a rich set of investment choices.

Code-named Investor Oriented Software, the project was partly outsourced to Agorics Inc., a San Francisco consultancy, and was completed in four months. Service launched in August 1997 and now Schwab has 200 Java developments in progress.

"The customer demand for Web trading services was simply overwhelming a nd we had to move up our plans for expansion," says Sasson.

In a year, the number of active online accounts doubled from 617,000 in December 1996 to 1.2 million, while associated assets ballooned from $42 billion to $80 billion.

Compounding the challenge, the stock market drop of 500 points and the ensuing fiasco of October 1997 was a wake-up call for most financial service firms. Network traffic on Schwab's Web site doubled that day to 10 million transactions, running at 130 percent of the network capacity.

Of course, this meant severe bottleneck pressure on Schwab's Internet pipe, a single 10-Mbps line from BBN Planet-now GTE Internetworking-and several backup T1 links provided by MCI.

For Schwab, reality was put to the test when its trading system had to handle 130 percent of its usual traffic. It meant that 30 percent of customers were not able to connect when they wanted to.

The brokerage's top management team realized that its trading system would have to handle much more than 2 m illion transactions an hour. And the Internet pipes-at T1 and lower rates-that linked Schwab's 1.52 million accounts to the brokerage would not cut it anymore.

It was time to expand the core capacity of the trading system substantially and beef up the Internet connections. Sasson did not want to be caught short again.

Finally, Schwab decided to go with two new ISPs-the company declined to name them-and bigger Internet pipes. Additionally, two 45-Mbps DS3 links, one from each ISP, and several T1s were chosen as backup. "If one of our ISPs goes down we'll be able to pick up services from the other, and we have made sure that even if an entry point to the Internet experiences a problem, we would instantaneously get backup from another place," says Steve Ariana, vice president of electronic brokerage technology.

Bigger Internet links by themselves will not solve all problems. According to Ariana, Schwab has worked out priority routing, service level agreements and quality-of-service guarantees with i ts ISPs to cover all bases. Now, customer service is the focus.

Phillips says that Schwab is resolving a major flaw associated with Java applets. "The question is how long it will take to download these Java applets and will customers wait that long?" Focus groups showed that customers would tolerate a three-minute wait over a 28.8 modem line. In the end, the team was able to exceed customers' requirements.

To sharpen its Java implementation, Schwab this month will use Marimba's Castanet technology to make the applet persistent on the desktop. Only changes to the Java applet will be downloaded, further reducing the download time.

Continuing its breakneck pace, Schwab is working on eliminating the middleware piece altogether by incorporating it in the core host system. The servers, component APIs and Java applications will reside in the host, and existing Cobol programs would have Java wrappers and look like Java-Beans. The client interface also will be a JavaBean. The switchover is scheduled for the fourth quarter.

The goal: reduce software development time and put out competitive services, even faster-if that's possible.