4 Storage Area Networks

In the world of complete solutions, SAN technology falls behind

September 25, 2000

Storage area networking technology is capable of changing the way enterprises think and how they deploy their servers, but so far the technology is in its infancy as standards are still evolving. Although components and even complete solutions exist, the manageability of SANs and their integration into the rest of a business's technology infrastructure is weak.

A typical SAN, of the type we examined in this Review Bunker lab test, consists of a Fibre Channel network connected to servers via Host Bus Adapters (HBA), as well as to storage devices such as disk arrays or tape libraries. Drivers in the servers communicate via the SCSI protocol over Fibre Channel to the SAN switch or hub, which intelligently routes the SCSI commands to a disk array where the commands are transparently mapped to logical units (LUNs), which correspond to individual disks, disk partitions or groups of disks configured as RAID arrays. (Other emerging SAN designs, which no vendor submitted for our lab test, are using Gigabit Ethernet instead of Fibre Channel.)

The beauty of a SAN is that while storage is physically separated from the servers and managed as a central resource, logically it's still under the control of each server's applications software. Whether the data is stored inside the server or resident on the SAN makes no difference to the operating system, the applications or its users. But it can make a tremendous difference to the efficiency and reliability of the solution, as well as to its expandability.

One benefit is that while the server still has transparent access to the storage devices, all the disks can be centrally managed from one location. The amount of disk space mapped to each server can be increased, moved to a different RAID striping scheme or reassigned to other servers on-the-fly, without bringing down the SAN. Also, tape libraries connected to the SAN can back up and restore the disk arrays using the Fibre Channel network without affecting the servers or their applications.

Another benefit is that when coupled with automatic failover‹ using multiple servers, HBAs, SAN switches and disk arrays-- an enterprise can create a very robust and resilient storage system. Since the connections between the servers and the SAN are through fiber-optic links, the various parts of the system can be located hundreds or thousands of feet apart--useful for storage consolidation.

For this server test, we invited SAN vendors with complete SAN solutions to the InternetWeek/ University of Hawaii Advanced Network Computing Lab. We wanted to see not just the individual components, but an entire SAN working, from server, to storage, to management.

Many vendors competed for our five available test slots. The first takers were QLogic Corp., Interphase Corp., JNI Corp. and Compaq, the latter of which requested two test periods for two completely different SAN solutions. At the eleventh hour JNI pulled out, leaving us with four SANs. All four solutions worked, but all were very different.

We learned that although SANs can be deployed today, no vendor offers a complete solution that encompasses multivendor interoperability, multiplatform support, integrated management and the ease of use that we all expect from Ethernet-based LANs.

Compaq RA4100
The smaller of Compaq's solutions, the StorageWorks RA4100 SAN is a solution geared for a handful of servers. Its strength is simplicity. The trade-off is relative inflexibility, as the RA4100 only supports a limited variety (and quantity) of hardware as well as only 11 devices--all proprietary from Compaq.

We found this ironic, as Compaq's hardware/software solution presentation was titled "The Open SAN." Whenever we asked about adding other hardware on the RA4100 SAN solution, Compaq's technicians told us that "that might work, but because we haven't qualified it, it's not supported."

Connectivity within the RA4100 is provided by its Compaq FC-AL (Fibre Channel Arbitrated Loop) Switch 8, which normally contains eight ports, but which can be upgraded through an expansion module with three additional ports. The switch that Compaq brought to the test had all 11 ports and offered redundant power and cooling. At the present time, the RA4100 SAN is limited as it only supports a single FC-AL switch, although Compaq says that larger Fibre Channel fabrics may be supported later this year.

The RA4100 is designed only for x86-based servers. Compaq brought a ProLiant 8500 equipped as a four-way Pentium III Xeon machine running Windows 2000 Server, and we also used two other Compaq servers that we had available for the test--a ProLiant DL380 running Red Hat Linux 6.2 and a DL580 with Windows 2000 Server. Each of the servers was configured with a Compaq Fibre Channel HBA, which required a 66MHz 64-bit PCI slot. According to the company, its SAN also supports Windows NT 4, NetWare, Banyan and OS/2.

On the storage side, Compaq brought two StorageWorks RAID Array 4100 disk enclosures, each of which offered 12 drive bays. One was equipped with 18-GB Ultra3 (Ultra160) SCSI hard drives; the other used the same size drives, but using Ultra2 SCSI disks. Compaq uses the same drive enclosures in its disk enclosures as it uses for internal storage on the modern ProLiant series. The disks were configured as eight logical units (LUNs), four on each disk enclosure, managed by a Compaq RA4000 SAN controller unit.

For backup, they brought a StorageWorks TL891 minilibrary, which contained one 5-MB/sec (native) DLT drive and 35-GB (native) capacity tapes. The SCSI-based library was connected to the FC-AL switch using Compaq's new Modular Data Router, which is a 1U-high Fibre Channel-to-SCSI with a single FC port and two SCSI connections.

There are different administrative tools for different parts of the SAN, such as Compaq's intuitive StorageWorks Command Console for configuring the FC-AL switch. It's a server-side solution that must be running on a server connected to the switch and that provides both a command-line and Web-based interface to configure the switch and the LUNs. A separate utility provides local and remote management (via Ethernet) of the Fibre Channel Host Bus Adapters.

Both of these utilities may be launched from within Compaq's Integrated Management Extended Edition (CIM-XE) management application, and they can roll alerts up to CIM-XE to be handled by the app. For a test, we configured Command Console to fire a thermal alert at 20 degrees Celsius; the alert immediately fired and percolated up to CIM-XE, which then notified us remotely of the problem.

During our tests, the entire RA4100 SAN worked perfectly, but since Compaq didn't configure it for high availability or redundancy, we were not able to test or verify any of those features, although the company does discuss using multiple FC-AL switches and storage enclosures to provide a redundant network.

Our complaint with the Compaq RA4100 SAN is that it's a small system and that it's closed to third-party devices.

Even as Compaq claims that its goal is to drive SANs toward interoperability and open standards, its insistence that only Compaq storage hardware be connected to Compaq's SANs (with the possible exception of other x86 servers, but even those must use Compaq's HBAs) is a striking contrast to Interphase's aggressive openness.

Compaq MA8000
Compaq's second solution, its midsize StorageWorks MA8000 SAN, is a much more comprehensive solution. Not only more scalable, the MA8000 also has higher-end management tools and the ability to support non-x86-based servers. Beyond that capability, it remains as closed as Compaq's smaller RA4100 SAN solution. (The MA8000 is Compaq's midrange SAN; the company also makes a higher-end solution, the MA12000.)

The MA8000 is built around a Fibre Channel fabric switch. Compaq brought two Switch/8 products, though 16-port switches are also available. The Switch/8 is SNMP-enabled, has redundant power and cooling, and can be managed via an intuitive, Web-based management suite. Compaq connected the two switches in a fully parallel arrangement, with two HBAs in each server, dual connections to each system and with no cross-connects between the parallel Fibre Channel networks.

For servers, the Compaq team used the same systems used for the RA4100 SAN test: the ProLiant 8500 Server (running Windows 2000) as well as two other ProLiant servers running Linux and Windows 2000. All three servers used the same 33MHz, 64-bit Compaq Fibre Channel HBA. To demonstrate that the MA8000 SAN is designed to support non-x86 systems, the team brought and connected a Sun Microsystems Ultra5 workstation running Solaris 7, equipped with dual SBus-based FCE-1063 Fibre Channel HBAs from JNI. According to Compaq, the MA8000 also supports AIX, HP-UX, Irix, NetWare, OpenVMS, Tru64 Unix and Windows NT.

For storage, Compaq's MA8000 RAID controller unit can drive up to six SCSI channels, with a maximum of 12 drives per controller. When new firmware for the MA8000 ships, that will be raised to 14 drives per controller. For this test, the company chose to bring its SW4254 disk arrays, each of which can drive as many as 14 one-inch Ultra2 SCSI devices. That means, with current firmware, the MA8000 can support six not-fully-populated SW4254 arrays, for a total of 72 disks, with the potential to upgrade to 84 disks when the firmware is revised later this year.

However, the Compaq technicians who accompanied the lab test recommended each SW4254 be logically partitioned as two seven-disk RAID arrays, by using three SW4254 boxes. Capacity is only 42 drives, but performance is higher due to the smaller number of drives on each SCSI channel.

For this test, the Compaq team brought only a single SW4254, equipped with 14 18.2-GB drives. The company also moved the same TL891 tape library and Modular Data Route (which it used for the RA4100 SAN) over to the MA8000 SAN.

We were impressed with the redundancy and high-availability features built into the MA8000 system. The main MA8000 controller included multiple hot-pluggable, rechargeable battery units for maintaining the state of the RAID subsystem, as well as an environmental monitor. We literally lost track of all the redundant power supplies and cooling systems found in the MA8000 controller and the SW4254 disk arrays.

The system controller contains dual Fibre Channel connections, configured to run both in parallel, failing over all traffic from one port to another in the event of a fault on the Fibre Channel network. However, in order to provide automatic HBA failover on the servers, a pricey software agent called SANworks Secure Path must be installed on each machine.

Another high-availability feature that impressed us was the MA8000's ability to create a pool of global hot-spare drives, which may be on any SCSI channel on the SAN. If a RAID 5 array or RAID 1 mirror set created to service a single server experiences a drive failure, the SAN automatically allocates a spare drive to that array. In our tests, we created a pool of several hot spares and then rapidly disabled a RAID 1 and RAID 5-based logical units by physically pulling the drives; we could visually see the global hot spares being allocated to replace them. We also caused a number of other failures on the SAN, ranging from unplugging GBICs to turning off one of the parallel SAN switches, and after experiencing only a very short pause of less than five seconds, the MA8000 recovered and kept operating each time.

We were also impressed with the breadth and depth of manageability of the MA8000 solution. Most of the system can be administered through Compaq Insight Manager (CIM) using applications either launched from within CIM or running separately, but whose alerts and reports can be monitored by CIM.

Overall control of the MA8000 controller is handled by a Windows NT/2000 agent called Command Console, which must be running on a server or workstation directly connected to the SAN. Command Console can be administered through either a graphical or command-line interface running directly on the machine hosting the agent or via a Win32 graphical applications running out-of-band on the Ethernet network.

The Command Console application not only reports on the SAN's health, but is also the tool used to configure LUNs, global hot spares and administer the storage system. A separate utility is required to manage the SAN switches. There is no central management of the SecurePath failover utility, so it must be configured separately for each server on the SAN, although it can report via SNMP if a failover takes place or if a spare pathway goes down.

When it came right down to it, the MA8000 SAN did it all, with high availability, reliability, scalability and even interoperability with Sun's Solaris. It's expensive--$4,000 per machine just to provide software for Fibre Channel failover! But there's no doubt that it works, and works well, which is why we gave Compaq's MA8000 our Best of Breed award for this lab test, as well as rating it InternetWeek Approved.

Interphase
Interphase brought many different partners' products to our SAN test. The company, which manufactures HBAs and Fibre Channel chipsets, successfully demonstrated interoperability with nine other manufacturers' storage systems, switches, servers, backup hardware and software, Fibre Channel-to-SCSI bridges, and more.

The heart of the Interphase SAN was the company's impressive 5570 SlotOptimizer HBA, which provides both Gigabit Ethernet and SCSI-over-Fibre Channel ports on a single 66MHz 64-bit PCI card. At about $1,500 each, that's more expensive than going with Fast Ethernet and a standard FC HBA, but for high-density systems, full LAN and SAN connectivity would only take a single PCI slot--and a fully redundant solution would take only two slots. This card is ideal for low-profile rack-mounted servers, such as those reviewed last July (July 17)

For our test, however, Interphase didn't build a redundant Fibre Channel SAN. Instead, the cards were inserted into a four-way Compaq ProLiant DL580 running Windows 2000, a dual-processor ProLiant DL380 running Linux and a Dell PowerEdge 4400 running Windows 2000, all brought by Interphase. As the 5570 card was scheduled to begin shipping a few weeks after the test's completion, Interphase was using beta drivers for this test.

The Gigabit Ethernet parts of the HBAs were connected to an Alteon WebSystems' 180 switch, which provided IP connectivity for the test. The Fibre Channel part was initially connected to an eight-port Vixel 7100 series Fibre Channel fabric switch. After the testing was complete, Interphase swapped in an Ancor SANBox-8 switch, just to demonstrate interoperability. Storage on the SAN was provided by two Storagepath SP-8BFC storage systems, each of which contained eight 18-GB Fibre Channel drives. There was tape backup, too: a 108-GB/sec Exabyte Mammoth 2 SCSI tape drive bridged to the Fibre Channel switch via a Crossroads Systems 4200 SCSI-to-FC router.

Software driving the solution ranged from Interphase's own Java-based FibreView Enterprise software for managing the HBAs, to the Legato's Networker for administering storage, Tivoli's SANergy provided shared access to SAN volumes and application failover. Each of the switch manufacturers also had software for managing their switch: Ancor's SAN Surfer and Vixel's SAN InSite.

The Interphase testing went off without a hitch and easily passed nearly all our test requirements such as showing heterogeneous servers playing multiple video streams while also performing file transfers as the Exabyte tape system restored a backup to another SAN volume. Overall, it was a solid performance. As mentioned earlier, Interphase didn't perform a failover test, and according to the company, it's because its current HBA drivers didn't yet offer that feature.

Auto-failover support, the company says, is expected in the fall.

Of course, a SAN means more than hardware. In a real-world deployment, it must be manageable as well. Each of the products comprising the Interphase solution contains its own management software optimized for a specific task.

Interphase's FibreView Enterprise, which we spent the most time with, is a comprehensive and easy-to-use Win32-based package for centrally managing HBAs and the nodes of the Fibre Channel network. The IP-based software works out-of-band over the Ethernet network, not over Fibre Channel.

Interphase demonstrated reasonably tight coupling between FibreView Enterprise and Vixel's SAN InSite software; SAN InSite could launch FibreView, and some alerts from FibreView could roll up into SAN InSite. Still, it would be a mistake to call the packages integrated; there's just some interprocess communication between them.

There is no management-software connectivity between FibreView Enterprise and Ancor's SAN Surfer application, although such integration is being planned for the future, according to Interphase officials.

There was no integration between FibreView Enterprise and the other software used for the SAN, such as Tivoli's SANergy and Legato's NetWorker. Interphase's FibreView Enterprise also didn't provide any native alerting functions about faults on the SAN; if you're not watching the screen, you won't see the errors. For functions such as e-mailed or paged alerts, you'd need to use a third-party package to poll the SAN via SNMP.

Overall, Interphase was able to demonstrate a complete solution, and was the only vendor to bring a diverse multivendor solution representative of what enterprises are likely to deploy in the real world. We were particularly impressed with the product's interoperability and the dual Fibre Channel/Gigabit Ethernet card, which could lead to a fully redundant SAN/ Ethernet solution using low-profile servers. With better-integrated management and with automatic failover within or between Fibre Channel networks, it would be a tough solution to beat.

QLogic
QLogic entered a minimal configuration into our lab test. Its system consisted of two servers on the storage area network, a single switch and a single storage device. Best known as an HBA and chipset manufacturer for both PCI-to-SCSI and PCI-to-Fibre Channel, QLogic had some difficulty in procuring additional hardware for the tests, and was plagued by inexplicable difficulties with the Fibre Channel switch the vendor brought to the test; the consensus was that it was damaged in shipping. Because of their limited hardware, QLogic wasn't able to perform many of the SAN tests that we had requested in the review plan. Even so, QLogic was able to construct and demonstrate a small working SAN solution with solid failover functionality.

QLogic's SAN solution used two servers containing the company's QLA2200F/66 host bus adapters. The QLA2200F series is an optical card containing a 1 Gbps Fibre Channel connection capable of full-duplex (200 Mbps) data transfers. The "66" designation specified that the card requires a 64-bit 66-MHz PCI slot. The servers were provided by InternetWeek: a four-way Pentium III Xeon-based Compaq ProLiant DL580 running Windows NT Server and a Hewlett-Packard LH-6000 six-way Pentium III Xeon also running Windows NT Server. QLogic used a third server as a management console, a two-way Pentium III-based Compaq ProLiant DL380 running Windows 2000; it was connected to the other servers via Fast Ethernet, and didn't have an HBA.

Ancor provided the Fibre Channel fabric switch for the test. We're not sure whether to call the QLogic entry a team or single-vendor effort: Ancor was in the process of being acquired by QLogic during our testing period, and in fact the transaction was completed only a few weeks later. Ancor's SANBox 16HA (High Availability) switch has 16 full-duplex Fibre Channel ports and can be equipped with dual, nonhot-swap power supplies, although the switch used for this test had only a single power supply.

The sole storage device was one of Sun's new StorEdge T3 Fibre Channel RAID disk arrays, which contained nine 18-GB FC- AL drives; this solution was the only one in our lab test to use Fibre Channel drives, rather than SCSI drives. The StorEdge was partitioned into four LUNs for the test, two for each server. During the test, both servers read streaming video from one LUN, while copying data back and forth between the other two LUNs.

For this test, QLogic chose to demonstrate a SAN with redundant paths, with two HBAs in each server, each connected to separate switch ports. There was only one Fibre Channel connection from the Ancor switch to the StorEdge array.

The biggest difficulty with the QLogic test appeared to be faulty switch hardware; its problems limited the amount of time we had for testing. The test was also limited in that QLogic brought only a single storage device and no backup device, and chose to use only Windows NT servers, so we were unable to view cross-platform compatibility and a live data backup/restore operation.

Despite those limitations, QLogic was able to demonstrate a working Windows NT-based SAN capable of handling many simultaneous data transfers. We were impressed by the automatic failover of the redundant fiber links to redundant paths, as configured in its QLconfig management software. With dual HBAs in each server running to the switch, the system was able to recover from a hardware failure (simulated by unplugging a QBIC from the switch) in less than five seconds. However, in another case, where we used the QLview management tool to disable a server's active HBA, the server hung completely for about three minutes before recovering and switching to the redundant HBA and path.

Our biggest complaint was the lack of real management. The QLogic HBAs come with two tools, which can run on Linux, NetWare and Windows: "QLview for Fibre," which provides administrative capabilities for HBAs and logs events to the NT Event Log, and QLconfig, which administers LUNs and failover paths.

Both tools can work on local HBAs as well as HBAs in network-accessible servers using Ethernet or IP-over-Fibre Channel.

Neither presents a complete solution. For example, when we failed one of the servers' redundant links, the only possible notification was a blinking toolbar icon on the management console, there are no e-mail or SNMP alerts, or other notification features. When we pulled up QLconfig and checked a path view, it showed us that a failover happened but not where. To learn why, we had to fire up QLview, check the appropriate server and drill down to the HBA, and all we saw was that traffic wasn't flowing. That's not a true management solution for a production SAN.

The Ancor SANbox included an embedded Web server that downloads a Java applet called SAN Surfer to a management station. The applet currently runs under Windows (we used it on a notebook) and on Linux, although Solaris is promised for the future. It, too, is really a configuration tool, not a management solution: Although it will show the switch's current configuration, there's no alerting and no active monitoring of the SANBox's SNMP traps. To tell if a port goes down, you'd have to be actively managing that switch and visually detect that a state had changed. The Sun StorEdge array had its own management utility, called StorEdge Component Manager, which was not integrated into the rest of the solution.

Overall, we were impressed with the configurability and reliability capabilities of the QLogic/Ancor solution. We can't attest to its backup, scalability or cross-platform abilities, and the management is weak, so we couldn't give QLogic top marks, but there's no doubt that QLogic has the connectivity covered.

Alan Zeichick is principal analyst with Camden Associates and is a contributing editor to InternetWeek. He can be reached at zeichick@camdenassociates.com