SANs and Excessive Disk Utilization?

pnutjam asks: "I work for a small to medium mental health company as the Network Administrator. While I think a SAN is a bit of overkill for our dozen servers, it was here when I got here. We currently boot 7 servers from our SAN, which houses all of their disks. Several of them have started to show excessive disk load, notably our SQL server, and our old domain controller (which is also the file/print server). I am in the process of separating our file/print server from our domain controller, but right now I get excessive disk load during the morning when people log on (we use roaming profiles). I think the disks need to be defragged, but should this be done on the servers, or on the SAN itself? When it comes to improving performance, I get conflicting answers when I inquire whether I would get better throughput from newer fibre-channel cards (ours are PCI-x, PCI-e is significantly faster), or mixing in some local disks, or using multiple fibre channel cards. Has anyone dealt with a similar situation or has some expertise in this area?"

Comments

Who's San Box is it?

[haplo21112]

You might want to consider calling the maker for technical support. Some SAN devices require defrag at the BOX level and doing it from the Server will adversely affect your data. Others its OK either way.

Re:Who's San Box is it?

[Anonymous Coward]

Defrag at the box level? Methinks you don't know what you're talking about*

Disk enclosures, from low end to high end, simply employ differing RAID levels to present logical disks as sequential block extents (ranges). A disk enclosure is not in the business of layout... block 001 is immediately before block 002 (although might be RAID-1 or RAID-5 or RAID-6 on the backend).

---
Also, to the submitters question: Throughput is rarely the issue with SANs, 1Gb or more is more than adequate for most apps. The bottleneck is probably the disks themselves, they can only spin so fast. Look at at the disk throughput metrics in Windows PerfMon.

Sometimes there is a mismatch between IO size and stripe widths (readwrites larger than stripe width, meaning that each server IO results in multiple disk IOs), but don't think about this now.

Either add more disks or add more memory to reduce the amount of disk IO. Alternatively, spread the data out. Also, if you're doing database stuff, make sure logs and data are on different disks. Look at whether you are using mirroring or RAID5, and whether this is done in the enclosure or by Windows.

* The only time this could be true is if you're using so-called "thin provisioning", which is pretty uncommon and unlikely for the customer description. Thin provisioning systems work in a log-structured manner to create LUNs out of non-sequential storage, so you don't use up storage that isn't used.

Re:Who's San Box is it?

[Sobrique]

SCSI is the protocol, FC or ethernet is the transport. Those chunky big cables you may be thinking of, might be correctly referred to as 'Parallel SCSI', which might be the source of the confusion.

Storage Area Network

[haplo21112]

For those not in the know, Typically cabled to the machines via fiber connections.

Re:No Acronyms!

[Anonymous Coward]

If you don't know what a SAN is, and are too lazy to consult Google, then why post? He's asking for someone who might be able to help, not trying to teach a lesson.

----

The whole SAN part is a red herring. He just has a storage area network (presumably Fibre Channel, as opposed to iSCSI), which just is a means of connecting servers to storage enclosures. The storage protocol is still SCSI, it's just over a different transport layer.

In other words, he has multiple servers connected to a single storage enclosure, and he's seeing capacity and performance issues.

The disks should be considered just like internal disks: defrag from the respective servers.

I would bet that his problem is simply having insufficient disks (spindles) to serve the morning peak workload... just like if you had a few internal disks.

In short:
- Defrag from each server, if you have a fragging issue
- Add more disks to spread the workload out
- Consider leaving the boot disks in each server, and just put data on the san. One main reason is that swapping to the SAN can be a problem by consuming storage enclosure cache (presuming there is any)

Re:What kind of SAN?

[pnutjam]

Xiotech 3d 1000, w/ qlogic fibre channel cards.

hmm..

[Anonymous Coward]

I am the Sr. Storage Architect for a Fortune 100 company. If you gave the type of array you have specifically, I'd be able to give more specific advice. That said: 1. You should have at least two fibre cards in each box anyway, and it has nothing to do with throughput. 2. Generally, your bottleneck is the disks themselves. If you want to increase performance, You need to increase the number of spindles that the data is striped across. Depending on the type of array, this may be a non-disruptive operation. The other big thing to look at is the type of RAID being used. You can usually get better performance from something striped with RAID10 vs. Raid 5, especially for write intensive data, because RAID 5 incurs an IO write penalty in calculating parity. 3. If you are going to defrag, do it on the server. It could help. There are some defrag functions available in most mid tier storage arrays, but it isn't what you think. The defrag there typically refers to lining up LUNs in a raid group. So, if you have a raid group with 5 LUNs in it, then delete one, you end up with a big empty space in the middle of the group. Defragging that raid group lines up all the LUNs inside that raid group.

SANs

[Sobrique]

The fact you're using a SAN is likely to be fairly irrelevant here. SANs are a way to move data between server and disks. They're not really much more complicated than that.

First question, is what's the symptoms of the problem - how do you know you're 'pegging your disks'? If you're seeing IO load to your HBAs being really high, then yes, you might find that you need to upgrade these. From experience though, HBAs are rarely your limiting factor.

Much more likely is that you're experiencing local disk fragmentation, as you correctly point out. I can't offer specific advise for your array, but in my experience, SANs are 'blind' to filesystems. They work on disks and LUNs. LUNs are the devices a host sees. This can be safely and easily be defragmented, in all the normal ways that you would do normally.

Are you accessing your SAN over fiber channel or iSCSI? IF it's fiber, then again, you _may_ have network contention, but it's unusual in my experience (especially on a 17 servre SAN). If it's network, then you have contention to worry about. Is it possible that your 'gimme profile' requests across your network are also contending with your iSCSI traffic?

You may find that your SAN has 'performance tools' built in. That's worth a look, to see how busy your spindles are. Because of the nature of a SAN, you may find that the LUNs are being shared on the same physical disks. This can be a real problem if you've done something scary like using windows dynamic disk to grow your filesystem - Imagine having two LUNS striped, when in acutality on the back end, they're on two different 'bits' of a RAID 5 set. This is bad, and is worth having a look at.

One place where SANs do sometimes have issues is in page files. Which is possibly a problem if you're SAN booting. SANs have latency, and windows doesn't like high latency on page files. If you really push it, it'll start bluescreening.

This is fixed by local disks for OS, or just moving swap file to local disk.

HBA expansion _might_ improve performance, assuming this is your bottleneck. However you'll need to ensure you are multipathing your HBAs. (Think of them like network cards, and you won't go far wrong - you need to 'cheat' a bit in order to share network bandwidth on multiple cards). But like I say, you probably want to check this is actually a problem. If they're not very old, then it's unlikely, although it might be worth checking which internal bus the HBAs are on. (Resilience and contention).

It's possible your SAN is fragmented, but it's unlikely this is your problem - SANs don't have the same problem with adding and deleting files (LUNs) so all your backend storage will be in contiguous lumps anyway.

And I apologise if I use terminology that you're not familiar with. Each SAN vendor seems to have their own nomenclature when it comes to the 'bits', but they all work in roughly the same way. You have disks, which are ... well disks. RAID groups, which are disks bundled together, with a RAID 1, RAID 1+0, RAID 5 (with variable numbers of parity ratio) and very occasionally RAID 0. You have LUNs. Logical Units. These are ... well, chunks of your bundles of disks. The first 100Mb of a 5 disk RAID 5 group, might be a LUN. The LUN is what the host 'sees', as a single atomic volume. Most disk groups can have multiple LUNs on them, which is why you do need to watch out for how volume management is operating. I have seen a case where a Windows 2000 server added a second LUN, and used dynamic disk to stripe. Not realising that on the back end, both those LUNs were on the same RAID 5 (4+1). Which cause the disks to seek back and forth continually, and really hurt performance.

Oh, and this is also probably a good excuse to be booking SAN training. IMO SANs are fun and interesting, not to mention in demand and well paid :)

A few answers

[sirwired]

1) No, it isn't your Fibre cards. The PCI-whatever bus (or the line speed of the card, for that matter), usually only affect high-bandwidth operations like tape backup. One thing you must remember is that loads that can beat the crap out of disk (random operations spread all over the platters), do not affect the I/O bus of the Fibre adapters at all, which cares only about total throughput.
2) It is far more likely your OS needs defragging than your disk array. Your disk array CAN become fragmented if you add and delete LUNs often, though.
3) Yes, you need multiple fibre cards, but for redundancy, not for bandwidth.
4) Try and put your major workloads on their own RAID arrays on your disk controller.
5) Check to see if you have enough memory in those boxes. If you have one server that keeps swapping out to disk and you are booting from SAN, you are going to get very hosed, very quickly. If these boxes have any internal disk at all, put the swap there.
6) If it is possible with your arrays, max out the segment size. (Engenio/LSI - based arrays can do this.)

This should be enough to get you started.

SirWired

Re:Wrong side of the problem

[Bastardchyld]

I believe he is actually referring to the disk usage caused by having to copy the contents of My Documents out to the workstation, not the network utilization. He is stating that you should redirect My Documents to a file share. Then they are not included in the Roaming Profiles that are copied, this will reduce your disk access at the peak login/logout times. It also could solve your problem if the roaming profiles are the culprit. Although an assessment of your SAN probably would not hurt anything.

Re:What kind of SAN?

[Animixer]

Here are a few things to consider.

1. How many target ports on the magnitude 3d? For that model, I'm not sure but they are probably 2gbit each. Try to balance load across the ports via multipathing software or manual balancing (server a uses port 1, server b uses port 2, etc).

2. What is your SAN switch topology? If hopping across ISLs make sure that you have an adequate amount of trunked bandwidth between the switches.

3. What speed are your SAN switches? Using 1gbit switches would bottleneck a lot faster than 2 or 4g ones.

4. If possible balance multipathing across independent fabrics. e.g. one port from each server to one fabric to one controller on array. this helps for HA.

5. Make sure the port speeds for all links are operational at the fastest common speed. (check that your 2g links are running at 2g, etc)

6. Check your zoning on the fabrics. I find it best in practice to mask by WWN and make sure that each hba port can only see the storage it should see, and nothing else. YES, even if you have masked your luns on the array. No sense having the OS go poking around trying to make device files and whatnot for devices it can't or shouldn't use.

7. Which qlogic cards? QLA21xx are ancient fc-al only. QLA22xx are 1gbit variants. QLA23xx are 2gb variants, the best being the qla234x series. QLA24xx should all be 4gbit.

8. Even a single 66mhz 64bit pci slot should be more than enough bandwidth for a single port 2gb card.

9. the magnitude 3d is if I remember correctly an entry level array. performance on it may be limited depending on how many physical spindles are installed. Also check the array health. if running degraded you will see a perf hit. higher end arrays should handle the failed disks better and not flinch but I've not tried this on a xiotech in practice. They actually may be quite good, I just don't know.