E-Trade CIO Discusses The Outages

It's been a grueling week for E-Trade Securities Inc.'s CIO Debra Chrapaty. For three days in a row, outages at the third-largest online broker made the evening news. It was the latest in a spate of outages and trading delays among different online brokers that has even sparked an inquiry of the overall issue by New York State attorney general Eliot Spitzer.

In an interview with InternetWeek senior editor Jeffrey Schwartz, Chrapaty explained what really went wrong at E-Trade and what the incident means going forward.

InternetWeek: So what happened last week?

Chrapaty: We were hearing from our customers that one thing that would make their experience a lot better is if they could receive confirms more quickly. So we took it under consideration, and we were putting in a change to speed up the confirmations.

The actual change was a memory channel adapter to improve I/O with some focus on our trading gateway and trading applications. So it was a memory channel issue that put more throughput on I/O and more pressure on the CPU. That created performance degradation in the application.

What happened essentially is the CPU got overrun a little bit, and it slowed down the application for a lot of our users. It was basically timing out so they couldn't access the application for the trading page.

InternetWeek: Where does the trading gateway reside?

Chrapaty: Our trading gateway sits between the Web-based trading application and BETA Systems [E-Trade's clearing facility]. It runs on Alphas.

InternetWeek: How come it happened three days in a row?

Chrapaty: Needless to say, problems--while they seem obvious after you find them--are not always that easy to detect. So on the first night after we found the issue, we thought it was one of the application layers. Then we realized we were mistaken, that it was a memory channel issue. So it was the normal process of diagnosis. Our systems are fairly complex. And we tried to scientifically, patiently go through each layer and understand the course of events and make well thought out changes. So we made a change one evening and felt confident in the test environment in that it seemed to show improvement. But during the market hours, it revealed itself again on Thursday, and a much smaller issue occurred on Friday.

InternetWeek: So you initially didn't think it was the memory channel upgrade that had caused the problem?

Chrapaty: I am sure I am preaching to the converted, but these systems are very integrated. So if there is an I/O issue, it may show itself in an application slowdown. Or if there is a CPU issue, it may rear itself as a time-out in an application that appears at first diagnosis as an application issue. So we make a change in the application and then look for its impact, and it goes layer by layer.

It is our approach, when we have issues--and understanding the complexity of our environment--we cautiously approach each layer and make changes. We have a full change control and problem management approach. This change was tested and was in production on some of our nodes, and appeared to be working very well. When we added it to a final node it had performance degradation.

InternetWeek: Does this underscore the challenge of trying to keep pace and add new content and capacity while not having these things happen?

Chrapaty: I probably put in 100 changes a month. All are well thought out and well tested, all go into change control that never impact our customers except in a positive way. This is one change that just did not have the impact that we anticipated out of hundreds of changes that we do or thousands throughout the course of the year. I obviously feel badly that it had a negative impact on my customers because I always want them to have a great experience when they come to E-Trade. The intention was to improve that experience. It was well planned and it was implemented off production hours, but it still had a negative impact.

InternetWeek: Is there a need for even more rigorous testing?

Chrapaty: We tested it rigorously. When I go back on what we would have done differently, it was a well thought out change. It was rolled out over a different period of time on different nodes, so we didn't do the memory channel on every single one. It was fully tested on each node and it was rolled out over a period of time, so I would say no. In an environment like this, I would need an environment as large as our entire operation to test every aspect, but I do believe we have a good test process, which is why over the past year we had one issue.

InternetWeek: Is it safe to presume that given these unpredictable occurrences and frequent updates that such an outage will happen again?

Chrapaty: I would say the technology is fallible. I would say it's my job to build as much fault tolerance in my applications as I can to address the fact that technology itself has flaws. Will it happen again? I will make sure to do everything in my power to make sure it doesn't, but certainly it could happen again. Have we learned from this experience to prevent outages like that one? Sure we have. Could something else rear its head in almost every layer of any technology? Sure it could.

InternetWeek: Did anything that happened last week cause you to rethink your architecture?

Chrapaty: To me, this is a one-off. We are back up and cranking. It's given me an opportunity to step back and look at all of the components of my environment in grueling detail. In that respect, I see it as a positive. It helped me to flush through that component of my environment in detail and make it extra, extra strength, because it was already extra strength.