In The Eye Of The Storm

Floyd's lessons help sites brace for traffic spikes

Floyd, the most closely tracked hurricane of the Internet Age, is leaving a trail of lessons about how to prepare for and manage Web-site traffic spikes. Among the secrets to stability: on-the-fly site design modifications and proactive infrastructure upgrades.

Weather.com, for instance, stripped its pages of graphical elements like multicolored maps so they could meet basic performance demands. It and other weather sites had already battened down the hatches with infrastructure upgrades in advance of the hurricane season. Those upgrades range from adding server and bandwidth capacity to activating redundancy measures, such as load balancing and caching at multiple sites.

While none of the major weather sites suffered outages, performance degraded under the massive traffic volumes. The top five weather sites--accuweather.com, weather.yahoo.com, weather.com, intellicast.com and wunderground.com--experienced a threefold increase in traffic during the height of the hurricane, according to Internet research firm PC Data Online. Accuweather.com, the most heavily trafficked, saw the daily page views jump 400 percent in one day, to 100 million. This figure includes its free site, as well as the 1,420 other news/weather sites to which it provides information.

Service Metrics Inc., which evaluates site performance in a number of industries, found response times on the most popular weather sites lagging this week as storms continued forming in the Atlantic.

A variety of factors degraded performance. They included weather-induced failures at network backbone providers; simple design problems, such as dynamically created graphics draining CPU cycles; and inaccurate capacity planning.

IT managers at the weather sites were quick to respond to those factors within their control.

Weather.com, the Weather Channel's online subsidiary, continually adjusted its site design throughout Hurricane Floyd, trimming file sizes by eliminating non-critical maps, changing advertising graphics to text-only to minimize bandwidth consumption, and cutting the size of banner headlines. These measures were taken on the fly, in response to performance degradation during the heaviest traffic periods, said Jody Fennell, vice president and executive producer at Weather.com.

Those measures helped, but some experts cautioned that sacrificing content is not always wise.

"I tried to find non-Floyd-related maps, but they weren't even online," said Internet consultant Erik Brandsberg. "You want to make sure you are still providing enough information."

That may be true in a perfect world, but "not many sites have our unique problem of going from a stable environment to huge peaks in one week or month," Fennell said. She said users such as Weather.com must try other approaches if they do not have the networking gear to handle extreme spikes. "It's expensive to have a bunch of boxes sitting underutilized most of the time," she said.

Weather.com nonetheless doubled its server capacity from six to 14 multiprocessor Unix servers in June in preparation for the storm season, Fennell said. Its traffic projections proved to be on target.

Partnering with geographically dispersed service providers also served Weather.com well. The site uses load-balancing switches on the east and west coasts, which enables it to distribute page requests to the least-taxed servers.

Similarly, CNN Interactive successfully employed multiple caching services to conserve bandwidth, according to a CNN official. Individual caching providers, such as Akamai Technologies Inc., can manage pieces of content, such as graphics and live video streams, and serve up that information. CNN's weather page proved to be one of the best performers during Floyd, said Service Metrics, despite peaking at a five-week high of 27 million page views--all related to Floyd--on Sept. 14.

In March, AccuWeather got hurricane-ready by adding to its dozens of Web servers and by upgrading its Ethernet switches to improve the site's sheer processing power. The company also increased its bandwidth by adding a fractional DS3 line and an additional T1 line, said Joel Myers, president of AccuWeather. A variety of carriers supply the DS3 and T1 lines in the event any one fails.

Not everyone was fully prepared. The National Hurricane Center, which provides information to the National Weather Service, colocated two extra Unix servers, purchased a second T1 line, and added a Cisco Data Director to load-balance the traffic. But it wasn't enough.

"We thought we had planned ahead, but we were still caught by surprise," said Webmaster Todd Spindler. "I call Hurricane Floyd the week from hell."

Traffic at the National Hurricane Center site kept rising from Monday Sept. 13 through Wednesday Sept. 15, when it peaked at 3 million page requests. By then, the site had been redesigned as all text, and two mirroring servers had been added.

Of course, even with these preemptive actions, it's impossible to avoid every pitfall. In the case of Floyd, AccuWeather's traffic exceeded projections, said Michael Steinberg, senior vice president. Though he wouldn't provide specific traffic figures, he did say the organization will revise its projection techniques.

AccuWeather also endured the failure of a backbone provider, which caused timeouts and denials of access over one DNS.

The struggles that have occurred should make IT managers all the more ready for the next natural disaster. Said Myers: "Storms will continue to be one of the biggest news events on the Internet. They affect people's lives more than any other story."