Schwab Outage: IT Wake-Up Call

By JEFFREY SCHWARTZ

A brownout at Charles Schwab & Co.'s online brokerage this week delivered potent lessons to managers of transaction-intensive Web sites.

A configuration error with a new mainframe that was added to increase trading capacity brought down Schwab's trading system for about one hour at the start of trading on Wednesday.

In so doing, it dramatized the need for e-commerce sites of all sizes to rigorously test new systems and components, institute strong failover and backup procedures, and implement those procedures quickly when things go wrong.

Until recently, Schwab seemed relatively safe because of its reliance on that ultrareliable computing behemoth, the mainframe.

"There's almost no excuse for a mainframe environment to be offline," said Giga Information Group analyst Rob Enderle. "These are extremely robust environments. Failover is anything but new. Performance could have degraded but, if done right, they should have been able to fall back on the previous configuration pretty quickly."

Ironically, this week's outage occurred just moments after Schwab president and co-CEO David Pottruck extolled the virtues of the Schwab Web site in a keynote address at the IT for Wall Street '99 event, a financial trade show, while making clear that Schwab and others do not have the reliability issue mastered.

"We spend a lot of time thinking about reliability," Pottruck said. "We are all in a learning mode. As we learn, we are all going to get better."

The new IBM mainframe--the sixth supporting the online brokerage giant's business--was turned on Tuesday night, and a configuration error related to how the new mainframe connects to other systems and applications made the site inaccessible Wednesday morning, officials said, although they declined to be more specific about the error.

Schwab officials insist they tested the system offline but that real-world conditions are difficult to mimic.

"I think the market opening is the true test of how things are going because you have 1,000 people signing on in the same one-minute interval. You can simulate a lot of stuff, but we obviously didn't simulate it perfectly," said Jan Hier-King, Schwab's senior vice president for e-brokerage technology.

At least one observer agreed that the unpredictability of loads in businesses such as online stock brokerages can make simulations a crapshoot.

"In order to design computer systems that will be reliable, we have to be able to predict what we are asking the computers to do," said Amy Wohl, president of consultancy Wohl & Associates.

The one-hour outage was intermittent, Schwab officials said. However, in retrospect, Schwab's technicians should have switched processing over to a failover system to minimize its impact, Hier-King said. Therein lies a lesson for IT managers.

"We were confident that this was a very short-term problem," Hier-King said. "We were so confident it was so short term, we didn't revert to our backup systems."

While it's difficult to draw direct comparisons between Schwab and other e-commerce sites because of Schwab's enormous transaction volumes, another major e-commerce site recently averted disaster through redundancy measures.

When both its primary and mirrored Unix ordering systems failed last month at its primary location hosted by Exodus Communications, National Semiconductor Corp.'s site blacked out for about two minutes before a second mirror site hosted by Digital Island in Hawaii kicked in, according to Phil Gibson, the company's director of interactive marketing.

Gibson acknowledged that his upgrade strategy can be slower and more methodical than that of Schwab.

"We can track over time to see when our servers begin to get a fever, and then act," Gibson said. A mainframe upgrade requires careful planning, and in Schwab's case, in a more compressed timeframe, he said.

Schwab will try the upgrade again, perhaps as early as next week.

It was the second outage in as many months for Schwab. In January, Schwab experienced a brief outage as batch processes extended into the new trading day.

This week's incident was the latest in a series of outages that have hit Ameritrade, Datek, E*Trade and Waterhouse Securities, among others. But Schwab is the largest online brokerage--with more than 2 million user accounts--and is known to have a robust and highly fault tolerant infrastructure.

While many online brokerages have much work cut out for them to provide adequate capacity, Schwab, E*Trade and others have been diligent about upgrading their systems to support their unprecedented growth.

Schwab's system allows 100,000 simultaneous account logons, up from 10,000 just a year ago. Once completed, the upgrade attempted this week will boost that to 250,000.

In January alone, Schwab received 3.7 million trade orders, up from 2.8 million in December, and it signed on 1 million new customers in the past year.

David Joachim contributed to this story.