Centralized storage provides a huge benefit in its ability to share media between servers, however that transfer requires some sort of transport. The most common sort of shared storage is block level hard drives, and the most common way to read and write to a LUN on a block level device is fibre channel. This is what most people talk about when they refer to storage networking. Let’s talk a little about what it is and how it works.
Storage Area Network
The term “SAN” has almost become a catch-all term for FC disk storage. The SAN itself, however, is simply the switches you use to allow servers access to the devices. Each disk device will usually have two controllers and a bunch of disks in raid groups. Each of these raid groups is split up into multiple volumes (LUNs) which can usually be accessed through either controller.
A fabric is a FC network, defined by switches that keep track of the attached devices. Usually, a SAN will have at least two fabrics, each of which connects to all the servers and disk controllers. This means that if you do a firmware update (or something) to a switch that causes a failure, you still have another switch that’s capable of connecting each device to every other device.
Zoning
Inside a fabric, switches are built with the ability to restrict connections between devices by separating them into “zones”. There are two ways to define zones: hard zoning and soft zoning. Soft zoning is the only one that is recommended by most techs and manufacturers, and the way it works is that each device, no matter where they’re physically plugged in, will get its access list based on it’s permanent identifier, called a “world wide name”. This is similar to MAC addresses in the world of IP networking. Hard zoning allows access to a list of devices in a zone based on the physical port they’re plugged into. This is almost never done.
Importantly, assuming you have a redundant fabric failover SAN, you will usually have to create the zoning on each switch.
Inter Switch Links
ISLs are the technical name for running a fiber cable between two switches and defining it as a switch link. This will cause the switches to merge their fabrics, and share resources (like name servers and zoning tables). This is generally done if you are adding a switch to a fabric to support more devices. You wouldn’t want to do this between two switches set up for redundancy, usually. This leads quite nicely into the next bit, which has a lot to do with switch links:
Interoperability between vendors
Putting Brocade and Cisco into the same fabric by creating an inter-switch link will make the switches drop into something called “interoperability mode”, which basically removes any features from the switch that are not in the FC standards book. That means, particularly, that you can’t trunk multiple links into one ISL. Both Cisco and Brocade have a few nice features that stop working when you plug them into a foreign network. Last I worked with Cisco, however, they could somewhat emulate Brocade switches, allowing you to run the fabric in full Cisco mode even if you had a few Brocades plugged in.
Multi-path drivers
If you plug a server into a fabric that can detect a FC LUN over two ports, it’ll see it twice. This can lead to data corruption if you try to format both “images”. The correct way to have a server access storage through multiple paths is called a “multi-path driver”. This is software you install on your host servers that lets them handle multiple paths intelligently, if their operating system won’t do it by default. Nowadays, even Windows comes with a decently intelligent multi-path driver available (MPIO), so unless you are being forced to install something different by your storage vendor, you should be fine.
Unix and Linux have another layer they can use called LVM (Logical Volume Manager) which can take multiple raw devices (LUNs) and stripe/mirror across them. I haven’t played with it much, but I hear it’s awesome. Mainframes were built from the ground up to use multiple paths to their storage, so using a secondary driver is completely optional.
Anyone who tells you that load balancing between paths is somehow important for your storage is probably not doing the math to prove it. Standard FC links these days are 8Gb/s full duplex. Assuming standard type of IO (small to medium blocks, very little sequential access, with a 70% read ratio), you won’t be able to drive a single link even above 10% utilization. Disks are always the real performance limiter, unless you’re using SSD, and those don’t do very well compared to arrays of disks in raw sequential throughput environments.
Long Distance Relationships
As we talked about in our DR post, sometimes you will want to mirror two disk arrays. This is often done through FC replication. Basically, assuming you have dark fiber between your sites and some long wave ports in your switch, you can send data at an amazing rate. This is pretty much the only way to do synchronous replication without taking your disk controller’s performance down about 3 gears. You can also try FCIP, which encapsulates fibre channel frames into IP packets, but that’s usually something you can only start doing when you are going asynchronous.
If you’re going the dark fiber route, you’ll want to make sure that your long distance ports on your switches are configured to use credits properly. This is a very low level detail, but the number of bits of data that you can get across the dark fiber is partly based on how smart your switches are about sending data while waiting for acknowledgment. Make sure to ask whoever you buy it from about that.
Whether you use dark fiber or FCIP, you will often have to dedicate a port on your controller to serve only replication traffic, and this means ensuring that it’s not in the same zone as any of your hosts. More and more disk controllers are including ethernet ports for asynchronous mirroring these days, though, so this will eventually stop being said.
No related posts.