Slashdot Mirror
Ask Slashdot: Best *nix Distro For a Dynamic File Server?
An anonymous reader (citing "silly workplace security policies") writes "I'm in charge of developing for my workplace a particular sort of 'dynamic' file server for handling scientific data. We have all the hardware in place, but can't figure out what *nix distro would work best. Can the great minds at Slashdot pool their resources and divine an answer? Some background: We have sensor units scattered across a couple square miles of undeveloped land, which each collect ~500 gigs of data per 24h. When these drives come back from the field each day, they'll be plugged into a server featuring a dozen removable drive sleds. We need to present the contents of these drives as one unified tree (shared out via Samba), and the best way to go about that appears to be a unioning file system. There's also requirement that the server has to boot in 30 seconds or less off a mechanical hard drive. We've been looking around, but are having trouble finding info for this seemingly simple situation. Can we get FreeNAS to do this? Do we try Greyhole? Is there a distro that can run unionfs/aufs/mhddfs out-of-the-box without messing with manual recompiling? Why is documentation for *nix always so bad?""
Why do you need a unified filesystem. Can't you just share /myShareOnTheServer and then mount each disk to a subfolder in /myShareOnTheServer (such as /myShareOnTheServer/disk1).
I know I’m not going to be the first person to ask this, but if I understand it the plan here was:
1 - buy lots of hardware and install
2 - think about what kind of software it will run and how it will be used
I think you got your methodology swapped around man!
Why is documentation for *nix always so bad?
You are looking for information that your average user won’t care about. Things like boot time don’t get documented because your average user isn’t going to have some arbitrary requirement to have their _file server_ boot in 30 seconds. That’s a very weird use case. Normally you reboot a file server infrequently (unless you want to be swapping disks out constantly..). I’m assuming this requirement is because you plan on doing a full shutdown to insert your drives... in which case you really should be looking into hotswap
Also mandatory: you sound horribly underqualified for the job you are doing. Fess up before you waste even more (I assume grant) money and bring in someone that knows what the hell they are doing.
Use OpenAFS with Samba's modules. Distribution doesn't matter.
Why does it have to be a mechanical hard drive? Why not use an SSD for the boot drive?
Is this a joke? A troll?
Any Linux distribution will boot in less than 30 seconds if you turn off all the services you don't need, which is probably most of them in your case. Any modern distribution will have packages for aufs ready to install. Any modern distribution will tell you, via D-Bus, when a removable disk is plugged in so you can run whatever program you want to handle it e.g. a script that mounts it at the right place in your tree.
Really, singular hard drives are notoriously bad at keeping data around for long. I would make sure you have a copy of everything. So make a file server with RAIDZ2 or RAID6 and script the copying of these hard drives onto a system that has redundancy and is backed up as well.
How many times I have seen scientist come out with their 500GB portable hard drives and they are unreadable... way too much. If you fill 500GB in 24 hours, there is no way a portable hard drive will survive for longer than about a year. Most of our drives (500GB 2.5" portable drives) last a few months, once they have processed about 6TB of data full-time they are pretty much guaranteed to fail.
Custom electronics and digital signage for your business: www.evcircuits.com
CentOS may be your best bet. Its Red Hat Enterprise Linux rebuilt from the Red Hat source code, minus the Red Hat trademark.
unRaid FTW, I use this to handle TB's of data and it works fine.
Another "I don't know how to do my job, but will slag off OSS knowing someone will tell me what to do. Then I can claim to be l337 at work by pretending to know how to do my job".
It's call reverse physiology, don't fall for it! Maybe shitdot will go back to its roots if no one comments in junk like this and the slashvertisments?
Not sure why the 30s boot up requirement is there, so it depends on what you define as "booted" . Spinning up 12 hard drives and making them available through Samba within 30s guarantees your costs will be 10x more than they need to be.
This isn't another example of my tax dollars at work is it?
Why would you want a file server to boot in 30 secs or less? Ok, lets skip the fs check, the controller checks, the driver checks, hell lets skip everything and boot to a recovery bash shell. Why would you not network these collection devices if they are all within a couple of miles and dump to an always on server?
I really fail to see the advantage of a file server booting in under 30 seconds. Shouldn't you be able to hot swap drives?
This really sounds like a bunch of kids trying to play server admin. My apologies if this is not the case, but given the parameters provided this IS what it sounds like.
There's no reason you need a union filesystem. Just mount the data at an appropriate point in a directory tree. Union file systems are designed to solve a different problem.
What you boot from has nothing to do w/ what you read the data from.
Samba is a really strange choice. Given the data volume I'd expect you to be using a large Linux cluster to process the data for which NFS would be more appropriate. It certainly sounds like microseismic data in which case the processing will benefit from making duplicate copies of the data and mounting read only via NFS so the first available server provides the data. Multiple ethernets are needed to get full benefit from doing that though.
*nix documentation is actually very good. But there is a lot of it, so you tend to have grey hair by the time you've read all of it.
BTW Does the CEO play guitar? I play harmonica.
Check out nas4free. It's basically freenas based on newer FreeBSD 9 which has zfs v28. I have been running it in a heavily used production system for 2 months with zero issues. I have 3 raidz2 setups that are shared out via NFs, cifs and afp. This setup is snapshotted every hour and also replicated via zfs send to offsides dr location.
If you go this route invest in a ssd for Zil and one for l2arc if you decide to dedup.
I run 100TB storage using openfiler. It is particular about hardware but rock solid. Match with many nics it is wicked fast. Also I would spread you IO load across a few machines to increase transfer speeds.
Have you considered using a distributed file system such as ceph?
You will need more drives as the data takes twice the space, but on the other hand you won't need to worry about boot times or scalability anymore.
I think there is no single good answer to this and most people will give their personal preference. I'm recommending FreeBSD because it can easily be tweaked and it also has a very good handbook http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ (this is to answer your question about lack of documentation). FreeNAS actually is built on top of FreeBSD.
Booting in under 30 seconds is going to be a bit of a trick for anything servery. Even just putzing around in the BIOS can eat up most of that time(potentially some minutes if there is a lot of memory being self-tested, or if the system has a bunch of hairy option ROMs, as the SCSI/SAS/RAID-generally disk controllers commonly found in servers generally do...) If you really want fast, you just need to suck it up and get hot-swappable storage: even SATA supports that(well, some chipsets do, your mileage may vary, talk to your vendor and kidnap the vendor's children to ensure you get a straight answer, no warranty express or implied, etc.) and SAS damn well better, and supports SATA drives. That way, it doesn't matter how long the server takes to boot, you can just swap the disks in and either leave it running or set the BIOS wakeup schedule to have it start booting ten minutes before you expect to need it.
Slightly classier would be using /dev/disk/by-label or by-UUID to assign a unique mountpoint for every drive sled that might come in from the field(ie. allowing you to easily tell which field unit the drive came from).
If the files from each site are assured to have unique names, you could present them in a single mount directory with unionFS; but you probably don't want to find out what happens if every site spits out identically named FOO.log files, and(unless there is a painfully crippled tool somewhere else in the chain) having a directory per mountpoint shouldn't be terribly serious business.
You might want to consider [url=http://www.openfiler.com]openfiler[/url] too.
But he either likes you, or is setting you up. build one of these instead.: http://hardware.slashdot.org/story/11/07/21/143236/build-your-own-135tb-raid6-storage-pod-for-7384 It's already been talked about.
I know you already stated the hardware is already in place. This is about exercising your new found authority. Go big or go home.
500G in a 24h period sounds like it will be highly compressible data. I would recommend FreeBSD or Ubuntu with ZFS Native Stable installed. ZFS will allow you to create a very nice tree with each folder set to a custom compression level if necessary. (Don't use dedup) You can put one SSD in as a cache drive to accelerate the shared folders speed. I imagine there would be an issue with restoring the data to magnetic while people are trying to read off the SMB share. An SSD cache or SSD ZIL drive for ZFS can help a lot with that.
Some nagging questions though.
How long are you intending on storing this data? How many sensors are collecting data? Because even with 12 drive bay slots, assuming cheap SATA of 3TB a piece. (36TB total storage with no redundancy), lets say 5 sensors, thats 2.5TB a day data collection, and assuming good compression of 3x, 833GB a day. You will fill up that storage in just 43 days.
I think this project needs to be re-thought. Either you need a much bigger storage array, or data needs to be discarded very quickly. If the data will be discarded quickly, then you really need to think about more disk arrays so you can use ZFS to partition the data in such a way that each SMB share can be on its own set of drives so as to not head thrash and interfere with someone else who is "discarding" or reading data.
You are in over your head. Buy a pogoplug and some usb2 hubs. Connect your drives to the hubs and they appear as unified file system on your clients. Or, if you need better performance accessing the data, call an expert.
Unless you're talking about millions of individual files on each drive it should be relatively quick to mount each hard drive and set up symbolic links in one shared directory to the files on each of the mounted drives. Just make sure Samba has "follow symlinks" set to yes and the Windows clients will see just see normal files in the shared directory.
this whole project is a joke, either:
A) you don't know what your looking for(reasonable but silly)
or B) you don't know how to collect data of what your looking for (pointless).
Process your sensor data at the sensor. There is no reason that anyone needs to take more than 10-50 mb of data per sensor device per session.
If it's a signal detection, use a smart filter to capture the event, or a FFT to capture the frequencies.
If it's a measurement, use a buffered slope detection and only capture the change.
If you do need to move 500gb/per day/per sensor. Just install fiber to the sensors and stream it back to a localized collection server. /.).
This saves countless sneaker net headach's, compression issues and the sort. For the collection server Buy It!, there are great products out there that can
take your 500gb of poorly compressed sensor data and make it 500mb of indexed intelligence (totally avoiding the obvious buzz word use here sorry
Aka: I do not want the insecurity of losing my workplace if my boss happens to learn in slashdot how clueless I am.
Seriously... could you send us the resumé that you sent to get that job?
Why can't
Arch, with conservative repo selection, is the best server OS I've ever used.
Why is documentation for *nix always so bad?""
For starters, I'm really tired of this /. *NIX is-too-hard ranting all the time on 'Ask Slashdot' posts. Don't be a n00b douche; if you don't get it, then spend some time and get it. Don't blame the documentation; dig in and figure out something for yourself for once. Sometimes you Nintendo-and-Mt-Dew generation make me want to throw up.
As for your solution, do-not go with some installable appliance-type distro like FreeNAS; yes it's *BSD under the hood, but you're at the mercy of what that 'focused' distro is goign to provide for you. Case in point: since you're undecided, go with a full-blown distro so you have some flexibility to grow and augment the mission and purpose of this server you're hosting data on.
Since you're clearly a n00b since it's coming to picking out a *NIX solution, go with anything Linux at this point, and set up the NAS services yourself (e.g. Samba/SMB, NFS, etc.) In turn, you'll be able to go to get better community support helping you out, you'll have more flexible OS configuration and growth, and you'll probably learn something to boot.
Also, you don't need to do union filesystem. Simple udev rules and auto mounting them under your top-level structure you're sharing out with your NAS services will do you just fine.
Firstly, the troll bit: "Why is documentation for *nix always so bad?" Scuse me? Uni systems have the BEST documentation of ANY current OS's. Why do you say the documentation is poor?
Secondly: If you have to ask slashdot for this you should not be doing this. Get someone who knows what they are doing to do it for you.
Have you read about Hadoop? I'm not altogether sure it fits what you're doing precisely, but depending on how the data will be used and your fault tolerance characteristics, it might be a good fit.
What is the point of 30 second boot on a file server? If this is on the list of 'requirements', then the 'plan' is 1/4 baked. 1/2 baked for buying hardware without a plan, then 1/2 again for not having a clue.
unioning filesystem? what is the use scenario? how about automounting the drives on hot-plug and sharing the /mnt directory?
Now, 500GB/day in 12 drive sleds....so 6TB a day? do the workers get a fresh drive each day or is the data only available for a few hours before it gets sent back out or are they rotated? I suspect that mounting these drives for sharing really isnt what is necessary, more like pull contents to 'local' storage. Then, why talk about unioning at all, just put the contents of each drive in a separate folder.
Is the data 100% new each day? Are you really storing 6TB a day from a sensor network? 120TB+ a month?
Are you really transporting 500GB of data by hand to local storage and expecting the disks to last? reading or writing 500GB isn't a problem, but constant power cycling and then physically moving/shaking the drives around each day to transport is going to put the MTBF of these drives in months not years.
dumb
Saying "only good mp3 player" makes no sense unless you specify your criteria. Amarok, Banshee, VLC, Rhythmbox, or smplayer are all capable mp3 players by various criteria and easily found by googling for "linux mp3 player". If you use Ubuntu, searching for mp3 player in Software Center finds a plethora of good players. Googling "list of linux audio software" easily finds other things besides just mp3 players: maybe something like Audacity satisfies your requirements better. Search for "mp3" on xmms2.org finds the answer in the first link - your xmms2 install needs have the MAD library, maybe your distro does not install that.
Does not seem like the problem is with bad docs.
"When I first heard Daydream Nation it quite frankly scared the living shit out of me." -- Matthew Stearns
Use systems of symbolic links.
Also, why "30 seconds boot time"? This strikes me as a bizarre and unnecessary requirement. Are you sure you have done careful requirements analysis and engineering?
As to the "bad" documentation: Many things are similar or the same on many *nix systems. This is not Windows where MS feels the need to change everything every few years. On *nix you are reasonably expected to know.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
It seems to me that the easiest way is to mount all of the sensor HDDs to a single directory (using ordinary "mount") on the OS and then share that directory over the network.
Even easier would be to write a shell script to handle the mounting & sharing so that all you have to do is connect the drives and then execute that script.
"When these drives come back from the field each day, they'll be plugged into a server featuring a dozen removable drive sleds."
"There's also requirement that the server has to boot in 30 seconds or less off a mechanical hard drive."
So... it takes minutes to hours to get your drives to the server, but then suddenly it's an emergency to get the server booted. That makes no sense to me. Please explain.
You say "a couple of square miles of undeveloped land". You also seems to say that you have around 12 sensor stations/ portable hard drives that are going to be swapped in and out each day. For reasons stated by others this is asking for loosing data. I would think you can't afford to do that.
Why not set up a wi-fi network using wifi links such as ubiquiti nanostation. You can easily cover a a couple of square miles with that sort of equipment; unless we are up in some mountains or in terrain where you are not even close to obtaining line of sight between stations. You local hardware store might be able to help with aluminum mast tubes. Aim to have the nanostations as far away as possible from any nearby objects including trees.
Then use rsync https://en.wikipedia.org/wiki/Rsync to continuously copy your valuable sensor data to your central location equipped with a safe and daily backed up RAID system. With 6 TB of data per day you are in for some data management issues unless you can somehow weed out irrelevant data on the go. And another plus is that you can work on new sensor data all day.
Because I suck at it and I'm too lazy to learn how to do it myself.
Real servers, even file servers, don't get booted often. And when they do, they should take as much time as they need to check the things they need to check.
Suggest you consider a hot-swap option, such as external drives on USB3.
Mount the external drives into some folder using some kind of naming scheme that makes sense. And/Or copy the data into a nice redundant file system, like RAID6 or RAIDZ.
You will want to speak with a professional storage provider about this, as you are in the special needs category.
But he think he has. Disastrous.
Hey. Could we get a bit of a better screening for what Ask Slashdot stories get to FP??? This sucks big time.
As many have noted, your system is junk if it boots in 30 seconds as server grade hardware typically contain a series of initilization steps and validations that blow away 30 seconds.Initializing 12 disks will take equally as long on a cold boot. Anyhow, stop cold booting and learn to use mount.
The unified file system is rather generically phrased and doesn't describe the task adequately enough. Do the files on the disks contain static names? If so, symlink farm from your mount points.
If the files on the hard disks contain random giberrish for names then simply index the files using a scripting language and create those symlinks. There are a number of ways to do this without actually having to execute the script. The less interesting method is to create a cron job that would clean up stale links and create new ones. Slightly more interesting would be to use the hot plug facility to create the index. Magically, the latter method which depends on hot plug would work well with actually avoiding the cold boot altogether.
I'm afraid this problem wouldn't even take an afternoon to solve.
Typically, the "bad documentation" experience is more accurately defined as "I don't know what I'm looking for." Really, once I have a few key words for a subject it's rather easy to find documentation for a particular area.
Ok, lots of folks asking similar questions. In order to keep the submission word count down I left out a lot of info. I *thought* most of it would be obvious, but I guess not.
Notes, in no particular order:
- The server was sourced from a now-defunct project with similar setup. It's a custom box with non-normal design. We don't have authorization to buy more hardware. That's not a big deal because what we have already *should* be perfectly fine.
- People keep harping on the 30 seconds thing.
The system is already configured to spin up all the drives simultaneously (yes the PSU can handle that) and get through the bios all in a few seconds. I *know* you can configure most any distro to be fast, the question is how much fuss it takes to get it that way. Honestly I threw that in there as an aside, not thinking this would blow up into some huge debate. All I'm looking for are pointers along the lines of "yeah distro FOO is bloated by default, but it's not as bad as it looks because you can just use the BAR utility to turn most of that off". We have a handful of systems running winXP and linux already that boot in under 30, this isn't a big deal.
- The drives in question have a nearly identical directory structure but with globally-unique file names. We want to merge the trees because it's easier for people to deal with than dozens of identical trees. There are plenty of packages that can do this, I'm looking for a distro where I can set it up with minimal fuss (ie: apt-get or equivalent, as opposed to manual code editing and recompiling).
- The share doesn't have to be samba, it just needs to be easily accessible from windows/macs without installing extra software on them.
- No, I'm not an idiot or derpy student. I'm a sysadmin with 20 years experience (I'm aware that doesn't necessarily prove anything). I'm leaving out a lot of detail because most of it is stupid office bureaucracy and politics I can't do anything about. I'm not one of those people who intentionally makes things more complicated than they need to be as some form of job security. I believe in doing things the "right" way so those who come after me have a chance at keeping the system running. I'm trying to stick to standards when possible, as opposed to creating a monster involving homegrown shell scripts.
Says somebody hiding behind his alias "LodCrappo" and trolling the AC's for a flame war.
OP rather clearly stated criteria for "good mp3 player". Here it is, since you missed it the first time: "sorts music like XMMS and since I'm used to XMMS does most everything in a similar fashion as well."
If you actually have to boot fast, stay away from any distro that isn't friendly about bare bones installs. Yes, that means ubuntu and redhat among many others. You seem to have a very specific use model for this machine, so run only what you want and don't let the distro bully you into running more.
That said, as many other people have commented, cold booting seems pretty pointless. Unless you are really stuck with pata drives, you will almost certainly be using a hot-swap friendly bus (sata, sas, usb, firewire). If you must power down the server, you can still suspend or hibernate.
Unionfs should be supported by pretty much everything. FreeNAS and other special purpose distros will probably make it really easy, if this is the sort of use model envisioned by the maintainers, or more difficult if its not. So if the docs don't make it look easy, I'd suggest avoiding those and sticking with an environment where its easy to roll your own features.
Personally, I would probably use debian with systemd (I like it a lot more than upstart), or freebsd for something like this. Slackware and gentoo are also nicely minimalistic. If boot time matters, seriously, use a SSD for the os, and keep the spinny disks just for data. If you keep the services light, you should have no trouble getting any of those to boot in 10 seconds (after bios does its stuff).
2.5" notebook hard drives (also sold as 2.5" externals), will probably hold up better than desktop or even enterprise drives, if you're going to bounce them around every day. As people mentioned, don't expect them to last forever. I would suspect you will mostly be writing sequentially to the drives, that should help you get a bit more life out of them. Run the drives' self tests frequently and toss them as soon as you start seeing problems crop up. 1-1.5TB 2.5" external drives are reasonably cheap at this point.
The first thing I thought of was loss of one of the drives during all this moving around. Seems the protection of the data would be of the utmost priority here. Keeping this in mind, I'd go with a RAID 5 or 10 setup. This will eliminate having the data distributed on different "drives", so to speak, and it would appear to the system one single drive. This would increase the drive count, but loosing a drive, ether physically (oops, dropped that one in the puddle) or electronically (oops, this drive crashed because we keep swapping it every day) would be a non-issue, or at least a not-tragic issue. I'm sure you have a swappable tray system now for the number of drives you need, you may need to add a tray or two for this setup. Just make sure you keep the drives in the correct order, or swap out the whole drive unit.
As for the original question. I don't think there's really a "best" distro for this, they'll all pretty much do the above out of the box and almost automagically. What you need to look for is what is the easiest distro to use in this case. What will the users be able to use with the least support from me? Unless you're the one that will be swapping out the drives on a daily basis, then use what you're most comfortable with.
--- Keep the choice with the user..
The nice thing about CentOS is that if/when you wind up on RHEL (comes with hardware, what you hosting provider is using, etc) the migration will be pretty simple.
First of all, xmms seems to be really outdated, and if his distro does not include it and he cannot compile it himself for whatever reason (despit copious info for how to install dev packages and compile for any distro), I fail to see how this is a failure of the un*x docs specifically. Current documentation for WinPlay3 is also rather scarce.
It's also hard to believe that there is not a single mp3 player out there that sorts music like xmms, whatever this is. He stated he tried Rhythmbox and was not too happy, but the allegedly so deficient un*x docs readily list many more, while the Ubuntu software center lets you try them out with one click.
Anyway, the logical conclusion seems to be to use xmms2, docs for which appear in the third google hit (for me) when searching for xmms. Which in turn, as I wrote, would take him to the solution in the first hit when searching for mp3 at xmms2.org. Again, how is this evidence for generally bad docs?
"When I first heard Daydream Nation it quite frankly scared the living shit out of me." -- Matthew Stearns
You have to be able to identify the disks being mounted. Since these are hot swappable, they will not be automatically identifiable.
Also note, not all disks spin up at the same speed. Disks made for desktops are not reliable either - though they tend to spin up faster. Server disks might take 5 seconds before they are failed. You also seem to have forgotten that even with all disks spun up, each must be read (one at a time) for them to be mounted.
Hot swap disks are not something automatically mounted unless they are known ahead of time - which means they have to have suitable identification.
UnionFS is not what you want. That isn't what it was designed for. Unionfs only has one drive that can be written to - the top one in the list. Operations on the other disks force it to copy it to the top disk for any modifications. Deletes don't happen to any but the top disk.
Some of what you discribe is called an HSM (hierarchical storage management), and requires a multi-level archive where some volumes may be on line, others off line, yet others in between. Boots are NOT fast, mostly due to the need to validate the archive first.
Back to the unreliability of things - if even one disk has a problem, your union filesystem will freeze - and not nicely either. The first access to a file that is inaccessable will cause a lock on the directory. That lock will lock all users out of that directory (they go into an infinite wait). Eventually, the locks accumulate to include the parent directory... which then locks all leaf directories under it. This propagates to the top level when the entire system freezes - along with all the clients. This freezing nature is one of the things that a HSM handles MUCH better. A detected media error causes the access to abort, and that releases the associated locks. If the union filesystem detects the error, then the entire filesystem goes down the tubes, not just one file on one disk.
Another problem is going to be processing the data - I/O rates are not good going through a union filesystem yet. Even though UnionFS is pretty good at it, expect the I/O rate to be 10% to 20% less than maximum. Now client I/O has to go through a network connection, so that may make it bearable. But trying to process multiple 300 GB data sets in one day is not likely to happen.
Another issue you have ignored is the original format of the data. You imply that the filesystem on the server will just "mount the disk" and use the filesystem as created/used by the sensor. This is not likely to happen - trying to do so invites multiple failures; it also means no users of the filesystem while it is getting mounted. You would do better to have a server disk farm that you copy the data to before processing. That way you get to handle the failures without affecting anyone that may be processing data, AND you don't have to stop everyone working just to reboot. You will also find that local copy rates will be more than double what the servers client systems can read anyway.
As others have mentioned, using gluster file system to accumulate the data allows multiple systems to contribute to the global, uniform, filesystem - but it does not allow for plugging in/out disks with predefined formats. It has a very high data throughput though (due to the distributed nature of the filesystem), and would allow many systems to be copying data into the filesystem without interference.
As for experience - I've managed filesystems with up to about 400TB in the past. Errors are NOT fun as they can take several days to recover from.
Also, "my criteria is for x to work like ancient app y" is not so workable. Sounds like Microsoft's convoluted standard's document for Office Open XML regarding backward compatibility. "You have to emulate bug x of Word 2, but we can't tell you exactly how that worked". Someone might have helped him if he had given specific requirements.
"When I first heard Daydream Nation it quite frankly scared the living shit out of me." -- Matthew Stearns
One of the things I have learned in my career is that I do not know everything. I do the things I can do well and understand the business and when I need something more, I will call in those that have the skills to help. If you get the right consultant involved with the project, they will bring the knowledge necessary to do the job right.
One of the biggest mistakes you can make is to think that you have to know everything and its some kind of fault to say, “I do not know right now but let me look into it and come back with an answer”. Do not limit yourself to only the knowledge and skills of the in house staff, tap into other sources to bring in new knowledge and skills that can help you solve problems.
My biggest resource was a list of those I could call in to help me with an issue or in the help in locating those that could come up with an answer. It not a fault to say I need to bring in some help on this project.
---
Excuse the grammar, been awake a few to many hours and not thinking to clear right now.
then get OpenBSD.
Well OP, you opened a big can of worms with this one. You may as well dump everything on the table. I see you started to in another post already. Why not add on answers to the other questions like, what kind of "sensors" are producing 500GB per day? What happens on undeveloped land that could produce so much data?
My first guess (considering undeveloped land) is weather plus solar radiation and soil temperature/moisture and perhaps seismic data. But that couldn't take more that 5MB per day. So what produces 500GB per day?
Come on admit it ... area 51 exists
Documentation for "*nix" is not "always so bad". You seem to be confusing the land of linux with the land of all (free and non-free) unices. In fact, the killer thing about unix always was its frankness about its limitations (qv "bugs" and "limitations" sections in manpages). "linux" or actually the gn00 bunch "standardised" themselves on their own texinfo format, complete with a captive viewer that only an emacs-luser could love. But I digress. There are unix-like operating systems Out There that are rather better documented, but they don't come in "distros". I could mention them but I'll leave looking around a bit more as an exercise to you.
Next up, you're in a supposedly scientific field but you're insisting on samba. What gives? While that software is trying its level best to make bricks without straw, the only reason you'd use it is because the clients use sucky software by an uncooperative vendor. Meaning you're going to run into performance problems no matter what you do. There are better solutions available and widely used in the academic world. Time to look into that.
Then, well, why are you so hung up on overly complex "solutions"? What about simply mounting the disks according to the day the data is from, maybe a bit of shell scripting to glue it all together? Or maybe it's more efficient to pump 500GB once to where it can be shared and archived conveniently then pull the sled and send the drive back into the field. I don't know. Have you even tried?
In short, dear sir, your priorities are jumbled and you haven't done your homework, then you insist on blaming the software and moaning about documentation. The easy solution to all that is to shut up and hire someone competent, even if only to consult on a suitable architecture.
Of course you don't have budget for that, because there's never budget to do it right. But somehow there's always budget to do it all over again, and fuck that up too because, well, no budget to do it right. "Ask slashdot" is in fact a symptom of that. Go sort your priorities.
What you're describing sounds like a fairly typical Sensor Net (or Sensor Web) to me, maybe with a little more data logged than is normal per platform. (I believe they call it a 'mote' in that community).
Some of the newer sensor nets use a forwarding mesh wireless system, so that you relay the data to a highly reduced number of collection points -- which might keep you from having to deal with the collection of the hard drives each night (maybe swap out a multi-TB RAID at each collection point each night instead).
I'm not 100% sure of what the correct forum is for discussion of sensor/platform design. I know they have presentations in the ESSI (Earth and Space Science Informatics) focus group of the AGU (American Geophysical Union). Many of the members of ESIPfed (Federation of Earth Science Information Partners) probably have experience in these issues, but it's more about discussing managing the data after it comes out of the field.
On the off chance that someone's already written software to do 90% of what you're looking for, I'd try contacting the folks from the Software Reuse Working Group of the Earth Science Data System community.
You might also try looking through past projects funded through NASA AISR (Adanced Information Systems Research) ... they funded better sensor design & data distribution systems. (unfortunately, they haven't been funded for a few years ... and I'm having problems accessing their website right now). Or I might be confusing it with the similar AIST (Adanced Information Systems Technology), which tends more towards hardware vs. software. ... so, my point is -- don't roll your own. Talk to other people who have done similar stuff, and build on their work, otherwise you're liable to make all of the same mistakes, and waste a whole lot of time. And in general (at least ESSI / ESIP-wide), we're a pretty sharing community ... we don't want anyone out there wasting their time doing the stupid little piddly stuff when they could actually be collecting data or doing science.
(and if you haven't guessed already ... I'm an AGU/ESSI member, and I think I'm an honorary ESIP member (as I'm in the space sciences, not earth science) ... at least they put up with me on their mailing lists)
Build it, and they will come^Hplain.
Yes FreeNAS all the way! ZFS will do just fine. (FreeBSD rulez!)
Since you already bought the hardware, odds are you're going to run into driver issues. Since you're not already a *nix guy my suggestion is just run windows on your server. Next buy big fat USB enclosures, the kind that can hold DVD drives and put the drive sleds in there. Now you don't have to reboot adding the drives.
Cwm, fjord-bank glyphs vext quiz
Assuming "500 gigs per 24 hours" is "500GiB per 24 hours" then we are talking roughly 6MiB per second. Are we talking video of some kind here? If not, I'll bet the data can compress down really well...
Maybe you should think about streaming data directly to tape, and then swap tapes out. Assuming for a second that your 30 second boot time has something to do with cloning a drive or some such nonsense, using tape will allow you to bring the data to the server instead of the other way around. At this point you will get rid of the silly 30 second boot time requirement while also drastically increasing the life of your fileserver drives.
As for the rest of your requirements, learn Linux or hire someone that knows it. Any distro can be modified to boot faster or slower since it's all pretty much the same anyway. Redhat compiles bash/perl/... from the same upstream sources that Debian does. Buying hardware that will allow you to boot faster is the real trick here. You might try to hire someone that knows what coreboot is and what it does. That would be a good indication that they are aware of factors responsible for getting a system up quickly. I would shamelessly offer my skills for hire but given the description of the problem, I'll stay out of the hiring pool.
Pretty much any unix like OS will do fine. Personally I use NetBSD and if linux is a requirement, Debian.
http://michaelsmith.id.au
simple http://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)
... because symbolic links are supported under Windows.
http://en.wikipedia.org/wiki/Beep_Media_Player
Talk to your local University.
Upper-division- and possibly graduate-level students in the following disciplines may be very interested in how this project turns out:
* Computer science / information systems / related
* The particular science related to the data being collected (e.g. meteorology if this is weather data)
* If this is a government project, discplines related to urban planning, politics, and the like
* If it's a business project, students taking courses related to doing tasks using available resources / tight budgets or in organizations that impose hard, non-technical, not-necessarily-wise requirements.
No matter how you ACTUALLY proceed, letting professors and key students from these disciplines watch you and document what you do then turn their results into "what might you do differently if you were in this same situation, and why?" type coursework will be invaluable to future generations.
And, who knows, you may get some valuable ideas you can actually implement.
--
By the way, I assume you have already considered and for the time being rejected the default option: Not go forward with this project on the grounds that doing nothing is a better option than attempting to do it under the constraints that you are being forced to work under.
Please keep this "default option of canceling the project" in the back of your head, and don't be afraid to recommend it should it really become the best option.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
As a former xmms user (some update finally broke it for good, and all current players midi implementations have stability/memory usage problems.), I can vouch for the crappiness of all the current players in one form or another, as well as the infuriating number of prerequisites and conflicts involved in keeping the versions YOU WANT installed.
VLC is excellent as a videoplayer and in fact has replaced ogle for DVDs for me in the recent past (although I've had different dvds which work better in one or the other, so it can vary dramatically.) However as a music player I've found it.. less than ergonomic. And as stated above it's got memory and stability issues with the midi plug, which involves a lot of my 'older' musical interests (game soundtracks and such).
The best explanation I've heard so far for why technical documentation in general (and in this case *nix documentation in particular) is often so poor is from a sci-fi TV series, called Eureka. In one episode, the characters search in vain for a manual to help shut down an antiquated launch system. When they figure there never was a manual, one character asks another why the builders did not bother to write one. The reply he receives is "Well, what do you want: progress or poetry?"
s/t
fak3r.com
Why is documentation for *nix always so bad?
Try reading Slashdot comment threads regarding the liberal arts, writing/English as a major, or even just people whose primary talent isn't in technology -- would you want to devote a significant chunk of time & energy volunteering in a community that felt that way about your field and everyone within it, even if the community is focused on an interest/hobby you have?
Now mostly at Usenet:comp.misc & SoylentNews.org (it's made of people!)
I might recommend a Software SAN solution such as Nexentastor which I have used ( search on google for "40000_FOOT_OO" )
I would need a lot more information on what you are trying to accomplish to give a yes or no for this particular application. RAID Z is one of the best file systems I have seen to date and the storage unit supports ISCSI NFS CIFS and SAMBA (I believe.) ISCSI is great there is initiators for just about every OS out there and its not a large client either. Windows 2003/2008/2008R2/Core 2008R2 all have clients built in to them so does Linux and BSD and Solaris (I assume OSX too).
How that would work with hot swap file systems on disks might be a bit tricky since it would take (a lot more ) than 30 seconds to boot if a drive is missing...Maybe use hot swap as others have suggested?
(I have no Hot swapable SAS drives on my home testing system which cost me 10,000 for everything over a long period of time in assembly.)
I was thinking perhaps you could attach those drives to a secondary system and copy them over to the primary storage system over ISCSI over 10G connections for quick transfers [perhaps even multiple bonded 10G links (short ones).] That way the main system wouldn't be AT ALL and it just becomes a backup restore or snapshot type of copy over ISCSI (by the way the VMWare ESX ISCSI intiator is the best one I have seen (seems to always work) Linux ISCSI initators coming in second Windows isnt bad in recent releases of server 2008 R2.
Utilizing portable flash based drives would cause more frequent replacements and a higher cost but would definitely allow you to come closer to saturating the bus during the transfers. It might even be possible to use NAS devices as the transport devices with multiple drive capabilities with flash drives to do raid 1's on them to double reads from the drives and would better insure the integrity of your data by replacement of the transport devices drives in a time staggered fashion.
(It all seems a little too sneaker net.)
Perhaps if the sensors are within ethernet or fiber range you could link them to your system that way (rather than using wireless which might be a security risk to you, but that all depends on distances you have between connections and whats in between. This would be fairly low cost for deployments that might be as big as half a dozen football fields (fiber) or 1 or 2 football fields (ethernet) you could route using layer 3 switches to extend.
You could also use optical laser links or Microwave (line of site)(this might be cost prohibitive so forget I mentioned it).
If you look me up I might be able to offer better suggestions.
I might recommend a Software SAN solution such as Nexentastor which I have used but this site wont let me point you to my blog (stupidly).
I would need a lot more information on what you are trying to accomplish to give a yes or no for this particular application. RAID Z is one of the best file systems I have seen to date and the storage unit supports ISCSI NFS CIFS and SAMBA (I believe.) ISCSI is great there is initiators for just about every OS out there and its not a large client either. Windows 2003/2008/2008R2/Core 2008R2 all have clients built in to them so does Linux and BSD and Solaris (I assume OSX too).
How that would work with hot swap file systems on disks might be a bit tricky since it would take (a lot more ) than 30 seconds to boot if a drive is missing...Maybe use hot swap as others have suggested?
(I have no Hot swapable SAS drives on my home testing system which cost me 10,000 for everything over a long period of time in assembly.)
I was thinking perhaps you could attach those drives to a secondary system and copy them over to the primary storage system over ISCSI over 10G connections for quick transfers [perhaps even multiple bonded 10G links (short ones).] That way the main system wouldn't be AT ALL and it just becomes a backup restore or snapshot type of copy over ISCSI (by the way the VMWare ESX ISCSI intiator is the best one I have seen (seems to always work) Linux ISCSI initators coming in second Windows isnt bad in recent releases of server 2008 R2.
Utilizing portable flash based drives would cause more frequent replacements and a higher cost but would definitely allow you to come closer to saturating the bus during the transfers. It might even be possible to use NAS devices as the transport devices with multiple drive capabilities with flash drives to do raid 1's on them to double reads from the drives and would better insure the integrity of your data by replacement of the transport devices drives in a time staggered fashion.
(It all seems a little too sneaker net.)
Perhaps if the sensors are within ethernet or fiber range you could link them to your system that way (rather than using wireless which might be a security risk to you, but that all depends on distances you have between connections and whats in between. This would be fairly low cost for deployments that might be as big as half a dozen football fields (fiber) or 1 or 2 football fields (ethernet) you could route using layer 3 switches to extend.
You could also use optical laser links or Microwave (line of site)(this might be cost prohibitive so forget I mentioned it).
Why boot? Figure out how to hot plug the drives. The sleds may be set up for it. or use ones that are.
Why a unified drive? Create an export directory(s) on the server, mount a drive, create links for each file file on the mounted drive into the eport directory(s), Samba Should be able to make it transparent.
12 sleds x 500GB, 6 TB a day, at 500 MB/sec (optmistic) you have 12,000 seconds (or more) to copy or access it all. Will the CPUs be able to keep up?
Will you need 10Gbs networking to get it to the consumers?
Fun archecture problem, this. And there is a lot you didn't tell us.
Ubuntu server
lububuntu-desktop package (hear me out!)
FreeNX + NXMachine to start a remote desktop session
SFTP or Samba for file sharing
Jailkit to jail the users to a single folder that contains the actual shares.
Eiciel for setting up ACL
Make users and read/write groups with the desktop GUI app (I'm a lazy person)
Use FileZilla or WinSCP to connect to the box for SFTP or use Samba
There's also requirement that the server has to boot in 30 seconds or less...
Uh, disable Pulse Audio??
If Unix is documented so poorly... then please show us the documentation for how you're doing this on Windows.
You don't need Unix or Linux. Get a copy of Window Server. Equip your Windows server box with USB 3 ports. Plug your HDDs into USB 3 caddies then plug them into the server. Share the USB 3 drives and you're done.
Why say *nix? We're gonna use Linux anyway, right?
Which kind of worked fine (though it couldn't sort my mp3's in s sane order, which is why I wanted XMMS in the first place) until I got tired of music and wanted to stop. So, I clicked the X in the upper-right corner... the window dissapeared, but the music continued.
If you close the window, Rhythmbox stays in the tray. You have to explicitly quit it from the menu.
Great. Had to ps and kill, and now my beer supply is out of sync with my enthusiam for music.
Heh. I know the feeling!
The boot process is greatly sped up with SystemD. Even though there is a chain of dependancies in the boot process, SystemD manages to boot largely in parallel, still. It is compatible with SysV bootscripts, so no sweat here ;-)
Fedora is a nice distro, but not stable. However, if you boot directly to a prompt, even Ubuntu Servers boots in a few seconds of a mechanical drive.
Now on to the data sharing; take a look at FUSE (Filesystem In UserSpace). This allows for unifying multiple filesystems into on virtual filesystem.
You can PM me for further questions and for asking my e-mail, if you need more direct contact :-)
Here be signatures
The unit docs used to be excellent then the 90's came and the fired technical writers and told engineers to do it. Engineers as expected have piles of domain knowledge and that reflects in there documentation couple that with a general disdain for the mess that most languages are and you get something with a steep learning curve that has a tenancy to be out of date
No sir I dont like it.
You should go with what you know best: Windows + USB sticks!
nobody pays for it and nobody will take the time to explain the system to people who might write it.
To be blunt:
This should be obvious to anyone who gives it any thought at all. You might try thinking about and solving your own problems instead of just posting them to a web site somewhere
Use genkernel, genkernel --install initramfs, genkernel --menuconfig all should be module support for everything. There that should fix you up.
What format are these drives in? Are they flash drives formatted in FAT32... great plug them all into a powered USB hub and share the files... no, well... bummer.
Are they stand alone ZFS pools? Great, drop them into your ZFS SAN and mount the zpool and share away... no, well... bummer.
What file system are they presented in? Could be anything... if it's Plan9 9P then maybe we can say sure what the heck... anything else and you're going to have to be a bit more specific.
For point of reference, I have two SAN systems at work. One is very fast, 4TB and runs on OpenIndiana and uses ZFS for our database and email servers. It takes several minutes to bring all disks online and be fully functioning. These are flash disks and it has 128GB of ram. It's screaming fast but has lots and lots of small disks. It cost $24K and is made for crazy speeds (saturates two 10GBe links and handles 120k IOs read/write simultaneously no problem).
The big SAN is 40TB and boots in about one minute to a useful state and starts bringing disk online with 10 minutes. It cost $2.5 million and is about the size of a minivan. It's made for gigantic simultaneous IO and 5 nines of availability and has dozens of easily removable drives and is extremely tolerant of hot swapping.
Be more specific OP.
[RIAA] says its concern is artists. That's true, in just the sense that a cattle rancher is concerned about its cattle.
How can everyone ignore FTP?
Did you all forget that all OS's involved support servers and client solutions...
I have found FTP to be faster than SAMBA with a majority of hardware/OS/Software solutions.
Also, a 2 Watt Marine grade USB to 802.11 b,g,n radio costs $40 in quantities of 1.... It has a range of easily over 5 miles.
Drivers exist for any PC collecting data...
Terrain is less of an issue when you have the power to overcome it.
Mesh? Why? We still need to know HOW MANY DATA COLLECTION SITES????
You don't need mesh to buffer 15 minutes of data and then send it in on schedule.
You would gain the option of remote control of each collection site if each was part of a network.
2 watts is a smaller power drain than a local drive swapped out every 24 hours, so if the remote units run off grid (you said the site was "undeveloped land") this would lower power consumption at the remote locations.
I have to agree with just about every Linux/BSD user here, Leave the systems running and hot plug... if you knew *nixes you'd know by now that there's no need to shutdown to add drives. (Even more importantly boot time becomes irrelevant if you don't do it more than a few times a year...)
Ummm, why not use a database to store the data instead of flat files in a folder? Data manipulation and filtering would be much simpler...
I also agree that the system was designed backwards...
Given that, what server hardware is involved so we can tell you what you can expect for boot times???
Is it a standard server we can look up specs for, or is it home built from imaginary needs?
How new is the hardware?
Do you have, or can you get a few SSD's for the boot files?
What is your budget?
You have crippled everyone here by lack of details like these...
I still contend that 2 watt radios with 12dBi antennas would be best to send the data directly to your server site. Either all in realtime, or scheduled with rotating buffers. You'd possibly need a few antennas on a small tower at the server site, each setup with their own subnets and channels, depending on how many remote stations there are.
Mesh is not needed for a 10 mile x 10 mile patch of land given enough cheap high powered radios and smart setup of timed payloads to be FTP'd on schedule to the server site. Your choice of database could easily import the data and drop any redundant data on import, would allow multiple access to the same data, and would have the speed required to store all of the data.
There's nothing wrong with mesh, I just don't think a couple miles of unimproved land requires it unless each site is just a couple MB per day and there is a huge number of sites to total 500GB a day. I also believe that number is arbitrary and not your actual exact need. It would help to know EXACTLY what is being stored.
Is the data wind speed, temperature, etc... or are you counting wildlife with motion triggered cameras and video recorded?
More data is needed about this supposed project.
If wireless data acquisition is out because your budget is already spent, then as some have stated, you will likely fail or have an extremely difficult time not only making it work, but also maintaining the system. The more you walk around with HDD's the more chances one or more will get static zapped or otherwise damaged.. That is not good for research. (Each failure drives up cost of new hardware..)
Good luck.
Yes, we absolutely ask.....
IS this a school project that can't be left running?
The amount of data you are talking about copying from these removable drives will take a long time to copy, and hot swapping drives will help speed up the process if you really can't afford to add $40 to each remote station to send the data in over a wireless connection.
http://www.amazon.com/Alfa-Waterproof-Outdoor-wireless-Integrated/dp/B003ILWRLI/ref=pd_sxp_grid_pt_0_1
That's including shipping, and you could probably get a discount if you ordered one for each site...
You already shot yourself in the foot if you have no budget left over.
What is some hardware fails and needs to be replaced? Do you have spares of anything?
How long will the collection project run?
You can give us the answer of why it needs to be started and halted so often without a story.. unless you are a troll....
It's easy... eg. We don't have 24/7 power at the server site... or, We operate the File Server from a mobile environment (close to first example)... or My BOSS can't afford the power to run 24/7... I still am getting the impression you are a troll looking to incite riot between the various BSD/*nix supporters here.
The question does not require a long answer. certainly it can be summed up in 3 or less sentences.
I, for one, am about to walk away from this thread on /.
As others have posted, BOOTING in under 30 seconds is a fake, but doable task for the core OS.... (and just about any *nix/BSD distro)
There is no valid reason that 45 seconds isn't just as acceptable... go get a cup of coffee.... your time is not that valuable or else you'd have a budget for this job.
an extra 15-30 seconds is not unreasonable to give time for a robust server to boot.
I don't believe the number is real. Nobody is going to die, and waiting for all the drives to show up on site from such a large site gives more time for said system to boot. You could even have it wake on a timer event so it could start just before you arrive on site... (if your requirements are not trolling)
For anyone claiming a legitimate science project, 30 seconds to boot is arbitrary and not a genuine need.
I'd like to read just one or two legitimate reasons why a person can't wait, say 60 or 120 seconds for a system to boot that is supposed to manage such large amounts of external drives with the data stored in such an illogical and unprofessional manner. (Given the requirements presented as part of the answer)
It all adds up to a troll or an employee/student being punished by being assigned an impossible task.
Reusing old servers from another project reeks of a school project... That increases the likelihood of hardware and storage failures. Asking a person who has not enough experience with the *nixes to already know which to use is like asking a banker which model of chainsaw is best for cutting oak... he/she may blunder around, and might find the right answer, but when it comes to implementing the device/s they may not even know how to start or maintain it/them....
The OP is already in trouble because of a lack of knowledge and too little time to become competent.
Or we have a troll...
Still think that's what we have here, given the requirements do not match the industry. No budget, used hardware, not enough specs to make an informed decision, and dodgy answering a few simple questions about the supposed project.
So, to reiterate: Don't Ask???? LOL... Yes, we do ask... answer why you need to shut it off, and what stops you from leaving it powered on. If you can have all of these remote sites gathering data 24/7 what's one more machine left on 24/7???.... that makes no sense other than from a troll....
The reason you won't answer is because if you answer why it must be rebooted so often you invalidate the 30 second requirement and show yourself as a troll....
Prove me wrong and answer why the 2 requirements: Why do you need 30 se
"Why is documentation for *nix always so bad?"
You didn't just say that did you? /facepalm
*nix/bsd/*inux's have the best documentation out of the selection of OS's period. Every command, function, ANYTHING that has anything to do with the os has a man page that goes into pretty great detail on what it does, what options it can use, how to use it...etc. Can't say the same about windows, not even close and as for mac os x.?? I have no idea.
The problem is more of, why can't the user think of the proper question to ask.... as in thinking of what to search for in the Ginormous collection of man pages and other help files. This doesn't even count free online sources or any books..etc that could help as well.
also without knowing how many actual sensors putting out this much data you have to work with it seems rather silly to use this type of system to begin with. Isn't it rather cumbersome to have to swap all of these drives in and out of the system every day? why not use wireless? For specific applications that don't really need general wireless standards (802.11 /a/b/c/g/e/f/g....blah blah) you could implement a high speed point to point type relay and not have to deal with a bunch of failure prone hard drives, especially if the sensor units with the drives are open to weather exposure (even if the drives are enclosed inside) you have greatly increased risk of drives failing.
Try the google, it is fairly adept at queries like these...
Even after fucking around with read/write buffer sizes in the .conf files and mount command lines, Samba is only about 30-50% as fast as NFS and FTP. All *nix machines can handle NFS and even Windows 7 ships with NFS support (you just have to enable it). If you want to read the entirety of 12 x 500GB drives every day then you're going to be wanting 71 megabytes per second - you won't get that if you're using Samba over a single gigabit link.
- loop recursively through all the files in the hard drives
- symlink them to another folder with the same structure
- share that folder
lather, rinse, and repeat every time you add/remove a drive. Not the most efficient or fancy solution in the world, but if you now bash you can write that in 10 lines of code
dCache is probably what you want.
A lot of my biggest concerns have been addressed by others. A few things that I haven't seen covered:
The "30 second boot time" limit makes me assume that there is something time-sensitive about this data collection. (Otherwise, why would you be wasting time on it?) So, you need a fast boot, but then you're mucking around with Samba and union mounts, which are both relatively slow. This doesn't make any sense. This is why people are asking questions or making up odd scenarios in their answers.
The odd scenario that I'm assuming is that you have more drives than sleds, so you need to go through a few load-boot-read-shutdown-unload cycles to get all of your data OR the machine's being "borrowed" to read the data, so you need to bring it up with an alternate OS quickly so that you can work through the night before returning it to normal use in the morning.
If that's the case, it really sounds like (as someone else suggested) that you need to separate the collector from the persistent storage. Set up something that can read the data from all of your "dynamic" drives as fast as possible. Depending on the data, something like rsync or even netcat might be the fastest way to get data off of the machine.
--