eBay Retrenches

Devastating outage exposes lack of redundancy, need for simplicity

A lack of server redundancy and software problems proved a lethal combination this week as online auctioneer eBay went dark for more than a day and struggled throughout the week to keep its core applications running.

It was the worst outage to date for any high-flying e-commerce site. eBay had already weathered site failures twice last month and three times in December. This time, it cost the company more than $5 million and 20 percent of its market value.

In response, eBay pledged a complete realignment of its IT priorities and business model. It was forced to temporarily retract feature enhancements such as personalized "My eBay" pages and an enhanced user interface, and company officials said the auction site will focus more on site stability and less on bells and whistles.

"We are seeing multiday outages due to inappropriate design, or the lack of simple redundancy techniques," said Dave Logan, an analyst at consultancy Acuitive Inc. "A lot of these sites just can't keep up with their growth. They don't have enough skills to keep the site running all the time."

Experts familiar with the IT underpinnings of auction sites said that the interactive nature of the traffic, combined with heavy pressure to ramp up quickly, could cause other e-businesses to suffer a similar fate.

"If you look at a site like CNN, the traffic is very asymmetric. Most of the data is flowing in one direction, out to the user," Logan added. "But at eBay, everything is bidirectional. Sellers are listing their items, talking to bidders. Lots of bidders are searching and doing transactions. It's a completely different model."

The nature of that traffic can put an enormous strain on the database that must support an auction-style application, said Larry Schwartz, president of Auction Universe, an eBay competitor. "You have to support a lot of databases, and the hardest hit you can take is when you lose that database server," he said.

eBay's failures could be a harbinger of problems that, left unchecked, could cause problems at other auction sites or even other e-commerce sites that eventually will handle similar patterns of personalization and transaction traffic, experts warned.

eBay suffered site-killing database and CGI server failures just two weeks before it was scheduled to cut over to a high-availability Sun Solaris-based system that would have provided hot backup for the corrupted systems. Even after the site went back online, it was sporadically unsearchable throughout the week.

The hot backup capability "might not have prevented the outage, but it would have shortened [the 22-hour outage] by 20 or 21 hours," said eBay spokesman Kevin Pursglove.

But a source with knowledge of the outage said the lack of redundancy was not the only problem. The Sun Solaris servers that run eBay's auction database and CGI search engine never received a system software patch that might have prevented the file corruption that took eBay staffers so long to repair, the source contended.

In addition, eBay staffers who were troubleshooting an application performance problem ran complex diagnostics on a production Oracle database server, according to the source.

Any one of those conditions--the lack of redundancy, the absence of the Solaris patch, or the running of the diagnostic program on a live server--might not have caused a problem by itself, the source suggested. But in confluence, they dealt a devastating blow to eBay's servers.

eBay officials confirmed the redundancy problem, but spokespeople said the claims of the missing software patch and the diagnostics problem were "inaccurate" but declined to provide further details. The company is planning to release its own report on the cause of the outage, but that report was not available at press time.

Officials at Sun, which is working with eBay to resolve the server problems, would only say that the company is investigating the situation and has not reached any conclusions on the causes of the failure. The officials confirmed that Solaris software was involved in the outage.

eBay officials said they have learned the hard way how much a catastrophic outage can cost. The company was forced to extend many of its auctions by 48 hours, and it will refund fees for all auctions held between June 9 and June 11. The cost of those refunds could total $5 million.

"We're going to have to change our corporate culture and our priorities, because what we've learned from our customers over [the course of the outage] is that stability is their top priority. Today is the start of a new eBay," Pursglove said.

Less tangible is the loss of confidence of eBay's customers, many of whom defected to other auction sites during the outage.

According to Nielsen/NetRatings, page views on eBay dropped by 51 percent on June 11. Much of the traffic went to Yahoo, the second-largest operator of online auctions, which ran radio ads indicating that its site was open for business while others were not. Auction Universe doubled its weekly new membership over the week before, Schwartz said.

eBay's customer base rebounded and continued to grow after outages in December and May. On June 14, eBay sellers listed about 250,000 items on the site, just 50,000 less than a typical day, Pursglove said.

"Everyone on the Internet is forgiving, but only to a point," Schwartz said. "The question is where that point will be."