Slashdot Mirror

Heres all the links i have bookmarked. Im sure you can make it work. The performance is basically bare metal. you have to pull some fuckery with the conf file for nvidia cards or you get error 53 i believe, because theyre cocksuckers that want to milk everybody. but thats a different issue. i hope these work for you.

https://www.pugetsystems.com/l...
https://davidyat.es/2016/09/08...
https://ubuntuforums.org/showt...
https://lime-technology.com/fo...
https://www.reddit.com/r/VFIO/...
https://bufferoverflow.io/gpu-...
http://vfio.blogspot.com/2015/...
https://www.se7ensins.com/foru...
https://forums.lime-technology...
https://www.reddit.com/r/linux...
http://blog.quindorian.org/201...

Heres all the links i have bookmarked. Im sure you can make it work. The performance is basically bare metal. you have to pull some fuckery with the conf file for nvidia cards or you get error 53 i believe, because theyre cocksuckers that want to milk everybody. but thats a different issue. i hope these work for you.

https://www.pugetsystems.com/l...
https://davidyat.es/2016/09/08...
https://ubuntuforums.org/showt...
https://lime-technology.com/fo...
https://www.reddit.com/r/VFIO/...
https://bufferoverflow.io/gpu-...
http://vfio.blogspot.com/2015/...
https://www.se7ensins.com/foru...
https://forums.lime-technology...
https://www.reddit.com/r/linux...
http://blog.quindorian.org/201...

http://www.flexraid.com/
http://lime-technology.com/ (UnRaid)

Best solution for big media collections.

All data is stored seperatly on each drive, and 1 separate parity drive can protect up to 21 drives (as long as its as big or bigger than any 1 of those 21 drives).

Obviously, everything could be set up on a linux box manually, but unraid is designed for this. You've got to pay a small amount for it if you want to use a ton of hard drives (I think you have to pay if you want more than 3 HDDs or if you want fancy features like Active Directory), but it seems like a pretty slick system.

Everything just works and it is perfectly designed for media consolidation and storage (the writes are slow but the reads are fast and it doesn't require matched drives or anything...). I've been looking in to making one for myself to stream media to xbmc and keep my music and photo collections in one place (and then you only have to back up one device). In the past I would have tried to roll my own, but now I kind of want something that just works, and I have a job so paying for it isn't a problem if I exceed 3 drives.

unRAID does not support unlimited drives in any version. It comes in 3 (free), 6, and 21 drive versions.

I've been using it for a year or two and, while it's got some limitations, it's a good choice for this application. Mostly because the guy's using a random collection of old drives and is likely to have bad sectors across multiple drives at some point. There is no striping with unRAID so the worst thing that can happen is he'll have to mount the drives individually and copy the data to a new array.

http://lime-technology.com/forum/index.php?topic=2817.0 ... the main feature of the script is
1. gets a SMART report
2. pre-reads the entire disk
3. writes zeros to the entire disk
4. sets the special signature recognized by unRAID
5. verifies the signature
6. post-reads the entire disk
7. optionally repeats the process for additional cycles (if you specified the "-c NN" option, where NN = a number from 1 to 20, default is to run 1 cycle)
8. gets a final SMART report
9. compares the SMART reports alerting you of differences.

Check it out. Its "original" purpose was to set the drive to all "0's" for easy insertion into a parity array (read: parity drive does not need to be updated if the new drive is all zeros) but it has also shown great utility as a stress test / burn-in tool to detect infant mortality and "force the issue" as far as satisfying the criteria needed for an RMA (read: sufficient reallocated block count)

If your skill level is enough to adapt the script to your own environ then great, otherwise UnRaid Basic is free and allows 3 drives in the array which should allow you to simultaneously pre-clear three drives. You might even be able to pre-clear more than that (up to available hardware slots) since you aren't technically dealing with the array at that point, but with enumerated hardware that the script has access to which should be eveything on the disc. Hardware requirements are minimal and it runs from flash.

Once your data grows past a certain level redundancy becomes a must. For about 3 years I have been running an unRAID server & I don't know how I survived before it. It allowed me to start with 3 drives (approx 250MB each) to my current configuration of 14 hard drives (mostly 1TB but also a couple smaller drives and three 2TB drives). After you have the system setup you don't need a monitor. I have it in my basement which helps with noise issues that result from having 12 drives spinning 24/7. Because this is a SOFTWARE-based raid-like system I can mix/match different types of drives to get the most storage for the money. I HAVE had a drive fail and the rebuild was easy. Another plus is the data is NOT stripped so if the motherboard ever fails the data can still be accessed (any computer that can access data on a "reiserfs" (I think) filesystem). One recommendation... Wire for Gigabit Ethernet! Once you start transferring HD Video (terabyte-size files) you'll find it helpful. Another recommendation... Find a smaller (100GB-200GB) drive to use as a "cache drive." This will speed up the transfers to your unRaid box :) Good luck! PS1: http://lime-technology.com/ PS2: I have been running version 5.0-beta14 without any issues

Check out UnRAID here. It is open source and lets you add disks more easily while providing redundancy. You should also check out the HTPC forum on AVS Forums.

On this topic, does anyone know how to set up something like an unRAID cache drive using only FOSS tools? bcache is the closest thing I've seen and it's not really the same thing either.

Take a look at unRaid if you haven't already. It is has a lot of the features you are looking for--it provides redundancy, will auto spin-down the drives and supports up to 20 drives. I'm in the proces of putting together a media storage server with it. Check it out at http://www.lime-technology.com/.

Okay one more attempt to reply to this

Head over to http://lime-technology.com/ as they have exactly what you want. Pay for play but not uber expensive and it will solve the issue you're trying to solve. I run two of these boxes and 16 drives isn't an issue. It will do spin-down for you as well as data protection. It stores parity on a single drive and uses ReiserFS on the data drives. If you lose a single drive no biggie, swap it and go. If you lose two at once you lose the data from those drives but not the rest of the data and you can use ReiserFS tools to recover from those drives. Check out the forums for more info and give the free version a spin to see if it's compatible with the hardware you have. Because parity is stored on a single drive it must be as big or bigger than the rest but you can run any cat or dog drive in any other slot no problem. Mixing IDE and SATA is also not an issue, the OS boots from a USB stick so no space is taken up by the OS - low end hardware is fine too but DO use Gig ethernet! One caveat - if you're watching a movie on a drive paired with another sleeping drive and THAT sleeping one is needed for something the video will pause while the other drive spins up - this is a Linux issue and one that Limetech it trying to get around through various means. It doesn't ever effect me but something you ought to be aware of no matter what solution you go with using Linux.

IMO this solution is way better than what you are currently planning and it solves the power issue nicely - tailor made for you methinks :-)

I have your solution - unRAID. unRAID uses a standard ResierFS filesystem for all of it's drives and stores parity on a dedicated drive. When a drive isn't in use the drive spins down. If a drive goes bad you're still protected and it can be swapped out without data loss. If you lose TWO drives at once you do lose data - from those two drives only. Unlike many RAID solutions this doesn't span data across all of the drives along with parity so you don't lose ALL of your data. Since the f/s is standard you can also use data recovery tools on the data to try and recover it offline. the software is pay for play but not uber expensive and it WILL support 16 drives. There's a free version you can use to test hardware compatibility. It boots from a USB stick too so no issues with the OS taking up space you would prefer to use for data storage. It's fast enough to feed media no problem although to max it out you'll want a gig ethernet network. I've been using this for YEARS without issue and have two of these servers running now. Good stuff IMO! http://lime-technology.com/ They have a forum for discussion hardware and software stuff if you've got questions.

IMO this is way better than storing multiple copies of the media as yuo plan...

I've been using unRAID for 3 months now, and it fills the bill wonderfully. You don't need matching drive sizes, don't need a RAID controller, don't need powerful hardware, and the list goes on and on. http://lime-technology.com/

No not so complex as you make it sound -> http://www.lime-technology.com/ and you can build one yourself if you choose. Don't use USB and DO have something to protect it but I don't like normal RAID for this - too power and space hungry.

I'd highly suggest you check out unRaid. It's an inline expandable raid system which allows up to one drive to fail without losing data. I've been using mine for quite a while and I love it!

I have two of these servers now. Each server can hold as many as 16 disks (possibly more actually as the programmer keeps bumping that up) with one disk reserved for parity. Data is NOT striped and parity is ONLY stored on the one drive. If a disk fails I lose no data, if two fail I lose two disks of data but nothing else. No hot spares or any other crap. If a disk isn't being used it goes to sleep and saves me heat and power. Disks can be ANY size but the parity disk must be as big or bigger than any of the data disks. Runs on a pretty decent selection of hardware although keeping the list of what works and what doesn't up to date is apparently tough since hardware changes so fast. It's Linux based but pay for play, yes he's followed the GPL. It's not super expensive and it boots from a USB drive to be web administered. I use full tower cases with SuperMicro 5n1 trays, 2gig of memory, Celeron CPU, power saving PSU, and supported mobo that have onboard video and GigE which you WILL need.

Their forums are a big help and active, users are working to expand the capabilities of these NAS and the programmer is working on making that easier too. Check it out, I've not found anything better yet and with some of the newer versions of SAMBA in the code it's pretty fast too! Perfect for a HTPC but not so great for a big transactional database

http://www.lime-technology.com/

For anyone looking at drobo I highly recommend checking out unRAID: http://www.lime-technology.com/

... and I'll say it again: unRAID

If you're willing to get your hands a little dirty, check out unRAID. You put the OS (linux) and software on a flash drive and boot from that. You have a big parity drive and a bunch of data drives that are just ReiserFS. The user share feature can aggregate all your files into one big virtual filesystem. When you run out of space you just pop another drive in, or pop out a small drive and put a larger on in, then wait for the data to be rebuilt from parity. You don't have to worry about your RAID controller dying, or a two disk failure (you'll still have the other disks of data). I built my machine for maybe a couple hundred bucks, and just added my 7th disk. I have smaller disks in there, but don't bother to remove them.

"And yet MD RAID1 does have better performance on single-threaded reads than single-disk reads. I think there's a bit more here than what you're describing."

That would be interesting to see, do you have any benchmarks? That doesn't jive with my (limited) understanding of how MD RAID1 handles read requests.

"Second, the same principles apply, just scaled down."

Again, a several disk raid5 solution really doesn't apply here.

"It's a cost/benefit tradeoff, and many companies do prefer to put the onus on the employee not to delete stuff that he needs"

Many companies also don't implement firewalls or don't regularly patch systems. That doesn't make it a recommended business practice. _Most_ companies take daily incremental/differential backups along with good, regular (usually weekly) full back ups. As is recommended by any competent IT professional.

"Which I explained in the previous post. You seem to be assuming that the alternative to a good backup system is no backup system? That's a reasonable misunderstanding, but I clarified that by "good" I was referring to backup frequency. Poor wording on my part."

So I should assume your other suggestion was to use RAID10 and a "bad backup solution" ? Of course the alternative to a "good backup solution" would be no backup solution. Anyway, semantics, getting nowhere.

Yes, RAID6 is fantastic, RAID-Z is also pretty amazing and a little company called lime-technology makes a great piece of software called unRAID. It lets you mix and match disks, uses a system similar to RAID-4 but each disk has it's own filesystem. So you could lose every disk but one and the data on that disk is still readable. Great for a small home setup for media and such.

http://lime-technology.com/

Lime-Technology makes a really interesting piece of software. It's a bootable NAS using their raid implementation called unRAID, which is similar to RAID4 except that each disk has an independent filesystem. So, you can pull a single drive out and it's readable. Also, you can mix and match any drive sizes you like, the only condition being the largest drive is used for parity.

Also the basic version is free

http://lime-technology.com/

Okay, unRaid is not particularly fast compared to an optimized system, but it's expandable, had redundancy, is expandable, is web managed, plays nice with windows, sets up in about 20 minutes, costs $0 for a three disc license and $69(?) for a 6 disk license.

My total unoptimized box on an utterly unoptimized Gb network (stock cards, settings, with 100 and 1000 nodes) and unmanaged switches just transferred an 8.3GB file in a hair under three minutes. From a single, cheap SATA drive to a Vista box with an old EIDE drive. Now 380Mb/s is not blazingly fast, but remember that it took almost no effort.

http://lime-technology.com/

No connection except as a happy customer with a 4TB media server that took longer to assemble the case than to get the SW running. If only my Vista Media Center install has been this easy.

Uhhh, I'm guessing that my UnRaid will be needing a new filesystem pretty soon then!

I had some old hardware, disks of various sizes. I liked what I saw with Drobo, (Use drives of different sizes, redundant), but they're pricey for just a chassis with no drives. I found unraid by http://lime-technology.com/ I can use different drives with different speeds, interfaces, etc. Boots from USB, configures Samba, FTP, etc. Not as fast with writes as raid5, but reads are pretty fast over the LAN. With some tweaking, I was able to put it on a full distro and retain the easy configuration, UI, etc. Worth checking out their free edition.

http://lime-technology.com/ offers UnRAID which looks very interesting. There's even a free version to try. To me, it's not just that you need X amount of storage, it's also about growth. What seems like a lot now, won't be a lot in a couple of years.

unRAID from http://www.lime-technology.com/ is what you want... and the 3-drive version is free.

It is not as *fast* as RAID 5 but it is 1) more reliable (since in a multi-drive failure, data from other drives is still readable) and 2) more flexible since you can add more drives when you need, and they do *not* have to all be the same size.

And it spins-down the drives if you want, and they stay spun down until needed... unlike a Windoze box. And it runs well on the $200 Wal*Mart PC, but is a little slow when building the parity disk at the beginning on a slow CPU. But after that, it is fine.

I use whatever disks I had laying arouind to build my NAS but the data is still protected. The software I use is developed by Lime-Technology http://www.lime-technology.com/. It's NOT RAID and instead is a JBOD setup with the first drive being a PARITY drive. This means that if one of my drives fails I still have access to the data. If TWO drives fail I lose TWO drives worth of data - *not* the whole damned thing. The data is not striped and is stored in a ReiserFS F/S so I can pull a drive and mount it elsewhere if I desire. This also means that if a drive isn't being acively used it can be spun down - try that with a striped RAID :-) When you write only the parity disk and the disk being written to need to be spinning, love that. The system can hold more than 12 disks if you use their top of the line software - mine only holds 12 total for a bit over 4.5TB worth of storage. Boots a customized Linux off of a memory stick and yeas source for mods is distributed but not the source for the WEB management stuff - he appears to be GPL compliant.

Some limitations: Parity drive must be as big or bigger than all others. Each drive is a seperate mount point unless you use a funky sort of shared folder feature. The system doesn't have as high a transfer speed as a RAID would, however it streams video for me to an XBMC XBOX1 just fine. It doesn't have a super robust system to notify you of failed drives out of the box although some users have added this functionality. Not a whole lot of security although I've met someone who has added this on and the developer is also working on expanding this in the future. Pretty decent support overall IMO and he's just moved to the 2.6 kernel - I've yet to upgrade though.

All in all this system seems to be perfect for HTPCs and I also use it to store backup images of all my workstations. All of my music and DVDs are stored on it and I'm about to build a second one as I need still more storage and have "spare" drives that I've pulled from the existing one as I've upgraded that I'd like to put to good use :-) Check out the user forums on the site, the developer is pretty responsive...

I can't believe no one has suggested an unRAID server. You get redundancy, storage that can grow by just adding another drive, low power consumption, affordability, and the ability to telnet in. (Plus it runs Linux!) I really like this solution since the data isn't spread out over a bunch of disks in a way that only the RAID controller can understand. Instead it's just a bunch of files on a bunch of disks, with an extra parity drive for reliability.

If a drive goes down, you can just pop a new one in and recover the lost data from the parity drive. If two drives simultaneously fail (unlikely), you lose the data on the drives that failed. Compare that to the nightmare if your RAID controller fails.

Here's my unRAID server, built for $400 plus the drives. I love being able to do backups by just running rsync. Once the author gets sshd built into the system, I can even do automatic incremental snapshot backups using rsync --link-dest.

Lime Technology has a pretty cool product for this. As I understand it, it is a version of RAID 4. You don't get the performance benefit of striping, but performance is still more than adequate for the typical home media server.

A couple of nice points about it are that the array can be as small as two disks (1 parity and 1 data), it's easily expandable and drives can be different sizes (though not larger than the parity drive).

Their prebuilt box is a little pricy, but you can also just buy their software and build your own box.

-pischke
(no affiliation, just a happy customer)

I've never used it though. I wish it was part of md, so I could use it on a standard distro.

http://lime-technology.com/

There's a long thread about this over on AVSForum.com. LB

http://lime-technology.com/unraid.htm

They're just starting out and they're targeting the HTPC crowd. There's a HUGE thread on the AVS HTPC forums about it. So far it seems almost everyone is happy with theirs. The cool part is that you can add hard drives on really easily, so it grows as you need more storage. Pretty affordable considering the other options.

From their website:
Lime Technology believes that there is a market for a new wrinkle on the old RAID technology. Our approach uses a storage organization we call unRAID(TM). unRAID(TM) is similar to RAID-4 in that for every n hard drives, there are n-1 data drives, and a single fixed parity drive. Unlike other RAID organizations, however, files are not striped across the data drives. Instead, each data drive is formatted normally with its own file system.

• Coolermaster Cavalier 2 case
  
• ABIT KV8 ATX AMD Motherboard (any motherboard with SPDIF out works fine)
  
• AMD Sempron 2600+
  
• Corsair Value Select 512mb (I've since added another 512)
  
• Any Radeon 9600 or higher. Must support ATI dongle. I have a Radeon 9600 SE.
  
• ATI Component Video Dongle (there are better options for video now)
  
• 2 Logitech Rumblepad 2 game controllers
  
• Windows XP (or Linux)
  
• Meedio (or MythTV)
  

There's a discussion here. I like the basic concept, RAID-4 without striping, but I'm not rushing to buy their package. I'd like to see an open source driver that does that though.