E-Biz Sites Push For 100% Uptime

By Saroja Girishankar

The world's most successful e-businesses are readying a new layer of Web infrastructure almost as critical as the technology that gave them an online presence in the first place. Their ambitious goal: 100 percent uptime--and the enhancements to their bottom line and brand reputation that accrue from that kind of reliability.

Cisco, Dell Computer, E*Trade Securities Inc., Chemdex Corp., Charles Schwab & Co. Inc. and other captains of e-industry are overhauling their hardware, network infrastructure and applications to handle escalating Web traffic, eliminate expensive outages and prepare for future demand.

No single approach prevails in this second wave of infrastructure buildup. E*Trade, Cisco and Chemdex are rolling out across-the-board redundancy of hardware, software and data centers. Schwab is upgrading its hardware and tuning its operations to keep its site humming without disruption. And Dell is installing new load-balancing software to meet increasing demand.

"E-commerce is our most mission-critical application," said Mark Tonneson, Cisco's senior director of information technology. Cisco handles more than 70 percent of its orders online, at an annual run rate of nearly $8 billion. Dell Online, which does $5 billion annually online, says it's operating close to the 100 percent mark, although chief technology officer Michael S. Dunn declined to say how close.

With those kinds of financial stakes, keeping an e-business site up and running is a high priority. Downtime of just one minute could cost sites as much as $10,000, according to the Standish Group, a research consultancy. By that count, a two-hour blackout carries a price tag of $1.2 million. Perhaps even more detrimental is the potential for losing future business by alienating customers.

Almost all the big e-business players have several backup Internet service providers, redundant servers and applications--and even duplicate data centers in parts of the country served by different ISPs and power grids.

For instance, Cisco, Dell, E*Trade and Schwab each are linked to eight to 10 ISPs over redundant DS-3 links to their respective sites.

"We saw the Tsunami coming, and so we have spent $100 million in redesign and upgrades to counter it," said Lisa Nash, E*Trade vice president of customer management.

E*Trade, which executes 43,000 securities trades per day, opened a data center last month in Alpharetta, Ga., six months ahead of schedule. The Alpharetta center pretty much replicates the databases, servers and applications in E*Trade's Palo Alto and Sacramento, Calif., data centers. Cisco this month will open a new data center in Raleigh, N.C. That center imitates Cisco's data facilities in San Jose as a backup and disaster-recovery site. Dell has two data centers in Texas, one in Europe and one in Asia.

Failover systems kick in when the primary server goes down, or companies mirror multiple active servers--or implement a combination of both approaches.

Schwab, which executes 200,000 online trades per day, has created a mirrored architecture for its 176 Netscape Web servers and application servers running on IBM SP/2 parallel multiprocessors, said Fred Matteson, Schwab's executive vice president of information technology.

Dell fully relies on mirroring techniques, spreading the load across all systems, Dell's Dunn said. To that end, Dell has created Web and SQL Server applications on a farm of hundreds of Dell servers running Windows NT.

For the new kids on the block such as Chemdex, which is hosting a trading hub for selling research equipment to scientists, the only way to go is to build fully redundant and mirrored systems from the start--even though Chemdex handles only four to five orders a minute.

"Putting together a scalable, business-to-business site is a mammoth undertaking," said David Perry, Chemdex's president and CEO.

In addition to building redundancy in its data center in Sunnyvale, Calif., Chemdex--which has more than 300 suppliers and 300,000 products in its catalogs--is rebuilding its data centers and the entire company in an unspecified New Jersey location for backup and disaster-recovery.

"It's about recovering the whole company if we have an earthquake on the West coast," said Pierre Samec, Chemdex's chief information officer. "We cannot afford to lose our massive data."

MedSite.com, an online retailer of medical books which recently launched a trading hub for selling medical supplies to physicians, is learning from the mistakes and implementations of the larger e-commerce sites, said Sandeep Bhan, president and CEO of MedSite.com.

"We are finding that mirroring and clustering techniques are the way to provide round-the-clock uptime," he said.

At the heart of its site is a farm of four Windows NT servers, a SQL Server database, an in-house search engine and Allaire Inc.'s ColdFusion middleware.

Many sites are using or starting to use load-balancing and rerouting software, such as Cisco's Local and Distributed Director, that redirects traffic to alternate sites to prevent bottlenecks.

Dell takes dynamic load balancing down to a granular level. It uses Microsoft's Load-Balancing Service built into NT to divert traffic among servers within a site, Dunn said.

Most e-businesses also are upgrading hardware to handle millions of simultaneous users and deliver subsecond responses.

Cisco is upgrading all its Sun Microsystems E-6500 servers to E-10000 systems to increase its capacity from 350,000 simultaneous users to 2.5 million on a single server. As for Dell, it's no surprise that the computer vendor uses high-end Dell 6300s for its duplicated database repositories.

Schwab is adding two mainframes to the existing six in each of its two data centers in Phoenix, and it's tying them together with IBM's Parallel Sysplex clustering and dynamic load-balancing technology. Eight backup mainframes replicate the front-end systems.

"The whole idea is to have three times the peak capacity so that we can accommodate any kind of unforeseen traffic, and ultimately we want to route traffic based on priority and trading," Matteson said.

Most e-business sites believe in a strategy of overcompensation and underconsumption. Dell runs its systems at 33 percent of capacity. E*Trade, after being stung badly by several outages last quarter, is playing it safe, keeping its systems running at 25 percent of capacity.

Other approaches include redesigning how applications work and upgrading systems in phases.

Two "brownouts" at Schwab, where heavy traffic brought down its trading applications, caused the company to remap the way mainframes process transactions.

Before the mishap, each mainframe was dedicated to processing particular types of transactions or customer accounts. An increase in one kind of transaction could cause a processing bottleneck on some systems, even though other systems had manageable workloads.

So Schwab is now revamping thousands of trading applications, decoupling them from dedicated mainframes. In the new systems, all transactions will be processed by a cluster of eight mainframes acting as a massive, single logical unit. The work is expected to be completed before year's end.

Schwab also has learned another lesson. It no longer simultaneously upgrades all its applications or hardware to newer versions. By design, half its servers and applications run earlier versions of operating systems and applications.

To prevent upgrade blunders, Cisco is working on a new "staging" process that marries and tests content and applications in a near-production network. Previously, content and applications under development were tested separately.

"We are simulating a production environment with a local directory combined with redundant production servers and large number of real users," said Christopher Sinton, director of Cisco's online site, Cisco Connections.

Most companies also are using Web site monitors, such as KeyNote Perspective from KeyNote Systems Inc., and internal performance monitors to generate alerts before set thresholds are reached.

At the end of the day, 100 percent uptime--and total customer satisfaction--remain the goals.

"We will always have outages," said Schwab's Matteson, "but we can ensure that our customers never know it, and that's what all this is about."