In addition to what you correctly stated about BackupPC being straightforward for static files, it also works very well for backing up databases. For this purpose, you can have BackupPC execute a dump-command to the database server over SSH, so the dumpfile can then be backup safely.
If a short interruption during the night is acceptable, you can even have BackupPC shut down the database service (and restart it afterwards, of course) to backup the actual database files themselves without need for an additional dump. If you have replicated database servers, there even isn't any noticable interruption for end users with this method.
Apart from the versioning that BackupPC does, it also reduces the amount of data that has to be moved across the network thanks to it's clever transport mechanisms.
Thanks to checksums for every file in BackupPC's storage pool as well as rsync as a recommended transport protocol, only the actual changes to the files are transferred. Thus, BackupPC lends itself well for use over low-bandwidth VPN connections, allowing for off-site backup without the need to take tapes or DVDs somewhere.
Firstly, the harddisks storing the backups aren't the same ones containing the original data being backed up, of course. Their not even supposed to be in the same physical location, let alone the same server! As I said, the data is being backup by means of a network (LAN, VPN et al), which is perfectly feasible thanks to the clever rsync algorithm being used by BackupPC.
Secondly, the.tar.gz export mechanism I mentioned is perfectly capable of creating partial files of 650MB or 4GB size for example. Thanks to the par2 information being generated at the same time, you get lots of checksum data to verify that your exports actually contain something worth taking off-site. If you don't want to actually slice the exports you can always generate them to a locally mounted removable harddisk as one big chunk as well.
Feel free to reply if anything remains unclear in your view.
For automated backups, you really should give BackupPC a try. There are no tapes, CD or DVD media involved, everything is tucked away efficiently on the harddisks. If you want (I never did so far), you can export a.tar.gz from the pool and burn that somewhere for off-site keeping.
As for setting up a fileserver, have a good long look at www.freenas.org or www.openfiler.com! They make it really easy to set things up without any hassle whatsoever.
Indeed it does, but not on a system like that. BackupPC relies heavily on MD5-checksums and does on the fly (de-)compression of archived files, so a little more horsepower is necessary for smooth operation.
But other than that, there's nothing like BackupPC for a pain- and effortless networkbased backup system.
Were you able to determine where the power surge entered your wired network? I use online UPS units to filter my voltage, so I've never had such problems.
Indeed, running proper Cat6 wires is a good investment if you're serious about having multiple networked devices.
Another problem with wireless networks is that all of the clients share the available bandwith most of the time. This means that you'll find yourself running a number of wireless accesspoints, which all have to be connected somehow to a switch, most probably using conventional wires.
The way I see it, there's no way around having a wired "backbone", considering the rising bandwith demands of future wireless clients.
I was assuming that somebody knowledgable enough to know what they could use a router with torrent capabilities for would have things like monitor, keyboard and mouse at home, just in case.
Some of these Mini-ITX boards come with a composite TV-out, so if it needs to be really cheap, hook the server to be set up to the TV and you just need to get an el-cheapo used keyboard to start. As for the CD-ROM drive, one can use an USB-stick instead, which can be used for other things than booting installation media as well.
You are definitely not alone. 2 years ago, I was struggling with the exact same dilemma: How to set up a proper home network that did basically what you listed in your bullets?!
I struggled with all kinds of black boxes and turnkey solutions for years, only to find that they didn't play together too well and that I felt somehow limited in what I could achieve this way.
So I decided to replace all of these routers, NASes, etc. with a set of Linux servers which I was going to install myself. Buying regular PCs for this was (like you pointed out in your post) out of the question. Too much heat and wasted electricity. Too much noise. Too many wasted parts.
In my quest for low-power servers, I became familiar with the Via Epia set of motherboards and the Mini-ITX form factor. What a god send! I ordered some parts from www.mini-itx.com (3 Travla C146 rackmount cases with Via Epia PD6000E, amongst others) and I was set.
Those things consume only a fraction of the power a regular PC would take, plus they're completely headless. No CDROM, only harddisks in a software-RAID1. No screen, keyboard or mouse during normal operation, either. I installed Debian Sarge on them using a bootable USB stick I made, the rest was done over the LAN.
To sum it up, it can be done, but it's a matter of picking the right parts.
Indeed, BackupPC is really a jewel! I have already set it up at a number of sites and it really works like a charm. Best of all, it's much easier to configure and there is no need to install any client software on the computers being backed up.
Efficient on-disk storage is where BackupPC really shines. On average and over time, the backup pool will contain 10 times it's physical storage, thanks to compression and pooling across multiple clients.
As for drawbacks, BackupPC does not play well if your main focus is doing regular tape backups.
Use Linux on the older computers, XP on the newer computers. You're already running three OSs, this way you're moving to two, and giving them an opportunity to see what each one does.
Great idea, all this will achieve is making sure that Linux is perceived by the kids as being sluggish and slow, in contrast to those speedy Windows XP computers that also let them play the games they brought in from home.
Guess what they are going to use in the future?
If Joe Sixpack buys a USB gadget at BestBuy, plugs it into his Linux PC, and it doesn't work, he will ditch Linux.
Strange, do you think he'd also ditch his beloved Windows when he plugs something in for which there are no drivers?
I can only speak for myself, but I usually check the HCL for my operating system and the requirements for the gadget I'm about to buy before actually buying it. Anyone not doing so deserves to be left with an overpriced paperweight.
Amen, brother! Debian's "slow" release cycle is actually what makes it so smooth, because this way there is no need to rush things out the door just because some marketing schedule says so.
I honestly hope that they stick to what they've been doing so far: Release an updated distro when it is ready!
In addition, I'm also very happy about the fact that I don't have to upgrade our servers twice a year. Running apt-get update && apt-get dist-upgrade is all I need to do once a week to stay on the updated side of things.
This is a sales channel where Jane and Joe Sixpack usually watch, just check some of the other products on that page. The channel runs 24h a day and at least every 2 days they're promoting Zeta.
They even sold the pre-release versions there, when Zeta didn't even have a proper repartitioning tool.;) I wonder how many Joes wiped their existing Windows installation in favour of Zeta and found that they lost all their data for something that doesn't even run on their box.
While I'm not entirely sure if they're doing their customers a favour in claiming that Zeta is the best alternative for Windows, they say they'll take it back 30 days with your money back.
Re:how do failures behave?
on
Basics of RAID
·
· Score: 3, Informative
I've been using software-RAID with ATA drives on Linux for quite some time, so I can comment on the behaviour of an array containing a faulty drive.
First off, let me emphasize how important it is to set up proper email notification (or pager etc.) for such cases! If you don't know about the failure, you're certain to get nice phonecalls from affected users.
If you've set up the notification system (smartd and mdadm come to mind), you'll eventually get an email saying something like "Device:/dev/hdc, ATA error count increased from 0 to 1" and that it would be a good idea to check up on the hosts syslog.
Checking up on the system, you'll find that the average system load has increased substantially, which is due to the system trying to persuade the disk to write to a faulty sector and the software RAID having to compensate, queuing the errors.
Depending on how often the defective sector is tried to be written to, the load can increase to values of 10 and above, rendering the system unusable. This is a good time to halt it and replace the defective drive, partition it and "mdadm/dev/md0 -a/dev/hdc1" the new one into the array, starting to resync it right away.
This may sound really horrible, but in practise it's usually less 60 minutes (counting from receipt of the first notification email) until normal operation can resume with such a system. This is assuming you have all spare parts stored somewhere on site.
In genereal, I've found software RAID1 and software RAID5 on Linux to be exceptionally stable. I'm also very happy about the performance, given that all I'm using is a bunch of el-cheapo ATA disks. As for reliability, I'm convinced it can't be beat in the consumers' price range, since I've seen too many consumer grade hardware RAID controllers go down in a swirl when putting more than a light load on them.
In the enterprise, I've seen companies move to software RAID on their Linux systems, because they found out that their only 5 years old enterprise hardware won't be getting any new spare parts anymore, which includes motherboards, CPU and IO controllers. Moving to software RAID on enterprise grade SCSI stuff allows them to move the entire system to another piece of hardware simply by moving the harddisk to it.
Oh dear... I can't imagine that a typical viewer of that station is computer-literate enough to cope with a wiped harddisk after an "install" of this "Zeta OS". Boy, I sure hope none of my friends installs it and asks me to fix things afterwards!
And this channel is brave, too. Imagine, they've been selling Zeta for over a year now, which also means that they've been selling Release Candidate software with a 30-day money-back guarantee... I wonder how many CDs get sent back...
As for the presentation, it's absolutely hilarious! They're showing off tons of small programs which don't need anything more than a P2 with 64 MB RAM anyway. As for hardware compatibility, they claim that driver development is so easy with Zeta OS, that they'll spit one out for any customer's hardware in no time at all... Good luck, you'll need it, guys.
I'm not sure whether that channel's aware of it, but they also have a positioning problem. All of the other stuff they sell is made to work solely on Windows. Now imagine what happens when somebody buys a copy of Zeta OS together with other stuff, like an el-cheapo digicam (it has 12 megapixels (!), but on the close-up you can see it only has a 3.2 megapixel sensor built in;) ) or a GDI-printer. Again, good luck guys, it won't work and your poor callcenter employees will only be asked a ton of stupid questions on how to get the whole junk to work together.
The best part is when the moderator claims that Zeta OS can easily be installed alongside your existing OS, be it Windows or that "Linux".
Strangely enough, there are actually people who appreciate long release cycles! I have servers running woody which absolutely need nothing newer and I'm happy about the fact that I don't have to change everything every 18 months.
If the release cycle were to be shortened to said 18 months, it would be nice if Debian were to maintain older releases and not only the previous release, like it it now.
I recommend Debian to my customers as a server platform, exactly because it has the finest package management and the longest release cycles. When stability is the goal, Debian is the right choice!
What most people fail to realize is that having a RAID-subsytem for storage doesn't relieve you from organizing a proper backup scheme.
Any RAID, be it software or hardware, can only protect an individual failure of harddisks, but not against corrupt filesystems or users deleting files deliberately.
I think you're missing the point here. The term "engineer" as it is used nowadays implies that someone knows the scientific theory and is able to apply it in real world situations.
Most certifications abuse the term "engineer" as far as the scientific theory is concerned. The attendants are fed with pseudo-information and half-truths, leading them to believe they are worth their salt in real-life situations.
Suffice it to say that I have seen enough of those "engineers" fail when confronted even with the simplest of their original test scenarios.
This is indeed very true. The company I work at also has this tendency to look for such pseudo-engineers. Being an engineer myself (MSc. in manufacturing automation) I keep telling them to look for real qualification that lasts, not just short lived and narrow focused stuff that can be acquired in a few weeks' time.
Becoming a real engineer takes time - a lot of time - digging down deep into the core of the matter, not just scratching the surface and pretending to know what's going on.
Unfortunately, the original poster is absolutely right in assuming that the average Joe on the street (and it seems in some personnel departments as well) does not make the right distinction in this regard.
Slashdot Mirror
In addition to what you correctly stated about BackupPC being straightforward for static files, it also works very well for backing up databases. For this purpose, you can have BackupPC execute a dump-command to the database server over SSH, so the dumpfile can then be backup safely.
If a short interruption during the night is acceptable, you can even have BackupPC shut down the database service (and restart it afterwards, of course) to backup the actual database files themselves without need for an additional dump. If you have replicated database servers, there even isn't any noticable interruption for end users with this method.
Apart from the versioning that BackupPC does, it also reduces the amount of data that has to be moved across the network thanks to it's clever transport mechanisms.
Thanks to checksums for every file in BackupPC's storage pool as well as rsync as a recommended transport protocol, only the actual changes to the files are transferred. Thus, BackupPC lends itself well for use over low-bandwidth VPN connections, allowing for off-site backup without the need to take tapes or DVDs somewhere.
OMG, no wonder you got bogged down in the complexity of the server-side setup using that HowTo!
/etc/backuppc/hosts and copy/edit one of the sample client-config.pl for each client to the /etc/backuppc directory.
Looks like setting up BackupPC on Fedora is much more complicated than on Debian Sarge.
On Debian, all I had to do was edit
On a sidenote, I even use rsync to back up my Windows PCs. The nice people at http://www.itefix.no/phpws/index.php?module=pagema ster&PAGE_user_op=view_page&PAGE_id=6&MMN_position =150:150 put together "cwRsyncServer" for Windows for free using rsync and Cygwin, which works like a charm.
It installs itself as a service and allows for really smooth backups using BackupPC.
OK, time to clear a few things up.
.tar.gz export mechanism I mentioned is perfectly capable of creating partial files of 650MB or 4GB size for example. Thanks to the par2 information being generated at the same time, you get lots of checksum data to verify that your exports actually contain something worth taking off-site. If you don't want to actually slice the exports you can always generate them to a locally mounted removable harddisk as one big chunk as well.
Firstly, the harddisks storing the backups aren't the same ones containing the original data being backed up, of course. Their not even supposed to be in the same physical location, let alone the same server! As I said, the data is being backup by means of a network (LAN, VPN et al), which is perfectly feasible thanks to the clever rsync algorithm being used by BackupPC.
Secondly, the
Feel free to reply if anything remains unclear in your view.
For automated backups, you really should give BackupPC a try. There are no tapes, CD or DVD media involved, everything is tucked away efficiently on the harddisks. If you want (I never did so far), you can export a .tar.gz from the pool and burn that somewhere for off-site keeping.
As for setting up a fileserver, have a good long look at www.freenas.org or www.openfiler.com! They make it really easy to set things up without any hassle whatsoever.
Indeed it does, but not on a system like that. BackupPC relies heavily on MD5-checksums and does on the fly (de-)compression of archived files, so a little more horsepower is necessary for smooth operation.
But other than that, there's nothing like BackupPC for a pain- and effortless networkbased backup system.
Were you able to determine where the power surge entered your wired network? I use online UPS units to filter my voltage, so I've never had such problems.
I'd be interested to hear on what statistical data you have based your statements and what you think the current value of "x" is at the moment.
Indeed, running proper Cat6 wires is a good investment if you're serious about having multiple networked devices.
Another problem with wireless networks is that all of the clients share the available bandwith most of the time. This means that you'll find yourself running a number of wireless accesspoints, which all have to be connected somehow to a switch, most probably using conventional wires.
The way I see it, there's no way around having a wired "backbone", considering the rising bandwith demands of future wireless clients.
OK, you've got a point there.
I was assuming that somebody knowledgable enough to know what they could use a router with torrent capabilities for would have things like monitor, keyboard and mouse at home, just in case.
Some of these Mini-ITX boards come with a composite TV-out, so if it needs to be really cheap, hook the server to be set up to the TV and you just need to get an el-cheapo used keyboard to start. As for the CD-ROM drive, one can use an USB-stick instead, which can be used for other things than booting installation media as well.
It doesn't have to be rackmount, if you don't have the space. Instead of the rackmount Travla C146 cases, check out the Travla C134 (http://www.mini-itx.com/store/images/c134-black.j pg) or C137 (http://www.mini-itx.com/store/images/c137-black.j pg) cases, they're really small.
t ml
As for setting up the necessary software, there's a number of projects that scratch that itch:
http://www.collax.com/en/resources/download-cbs.h
http://www.openfiler.com/
http://www.freenas.org/
http://www.skolelinux.org/portal/index_html
http://www.ubuntu.com/server
All of these are really easy to install and work exceptionally well.
You are definitely not alone. 2 years ago, I was struggling with the exact same dilemma: How to set up a proper home network that did basically what you listed in your bullets?!
I struggled with all kinds of black boxes and turnkey solutions for years, only to find that they didn't play together too well and that I felt somehow limited in what I could achieve this way.
So I decided to replace all of these routers, NASes, etc. with a set of Linux servers which I was going to install myself. Buying regular PCs for this was (like you pointed out in your post) out of the question. Too much heat and wasted electricity. Too much noise. Too many wasted parts.
In my quest for low-power servers, I became familiar with the Via Epia set of motherboards and the Mini-ITX form factor. What a god send! I ordered some parts from www.mini-itx.com (3 Travla C146 rackmount cases with Via Epia PD6000E, amongst others) and I was set.
Those things consume only a fraction of the power a regular PC would take, plus they're completely headless. No CDROM, only harddisks in a software-RAID1. No screen, keyboard or mouse during normal operation, either. I installed Debian Sarge on them using a bootable USB stick I made, the rest was done over the LAN.
To sum it up, it can be done, but it's a matter of picking the right parts.
Indeed, BackupPC is really a jewel! I have already set it up at a number of sites and it really works like a charm. Best of all, it's much easier to configure and there is no need to install any client software on the computers being backed up.
Efficient on-disk storage is where BackupPC really shines. On average and over time, the backup pool will contain 10 times it's physical storage, thanks to compression and pooling across multiple clients.
As for drawbacks, BackupPC does not play well if your main focus is doing regular tape backups.
I can only speak for myself, but I usually check the HCL for my operating system and the requirements for the gadget I'm about to buy before actually buying it. Anyone not doing so deserves to be left with an overpriced paperweight.
Amen, brother! Debian's "slow" release cycle is actually what makes it so smooth, because this way there is no need to rush things out the door just because some marketing schedule says so.
I honestly hope that they stick to what they've been doing so far: Release an updated distro when it is ready!
In addition, I'm also very happy about the fact that I don't have to upgrade our servers twice a year. Running apt-get update && apt-get dist-upgrade is all I need to do once a week to stay on the updated side of things.
Well that's exactly what they're doing on German television: http://www.rtlshop.de/rtlshop/servlet/~tvm5/rtlsho p/subsites/articleDetail.html?command=display&btUi d=bt_Article&iDf_id=7f001:-4d972e39:1047e548e3d:78 80&iDf_relayClientId=4&startPage=true
;) I wonder how many Joes wiped their existing Windows installation in favour of Zeta and found that they lost all their data for something that doesn't even run on their box.
This is a sales channel where Jane and Joe Sixpack usually watch, just check some of the other products on that page. The channel runs 24h a day and at least every 2 days they're promoting Zeta.
They even sold the pre-release versions there, when Zeta didn't even have a proper repartitioning tool.
IMHO most existing Windows systems are actually used as single user machines, even though they'd be capable of being used by multiple user.
o p/subsites/articleDetail.html?command=display&btUi d=bt_Article&iDf_id=7f001:-4d972e39:1047e548e3d:78 80&iDf_relayClientId=4&startPage=true, where things are usually quite simple or simplyfied.
With this in mind, one can understand why YellowTab is using sales channels like http://www.rtlshop.de/rtlshop/servlet/~tvm5/rtlsh
While I'm not entirely sure if they're doing their customers a favour in claiming that Zeta is the best alternative for Windows, they say they'll take it back 30 days with your money back.
I've been using software-RAID with ATA drives on Linux for quite some time, so I can comment on the behaviour of an array containing a faulty drive.
/dev/hdc, ATA error count increased from 0 to 1" and that it would be a good idea to check up on the hosts syslog.
/dev/md0 -a /dev/hdc1" the new one into the array, starting to resync it right away.
First off, let me emphasize how important it is to set up proper email notification (or pager etc.) for such cases! If you don't know about the failure, you're certain to get nice phonecalls from affected users.
If you've set up the notification system (smartd and mdadm come to mind), you'll eventually get an email saying something like "Device:
Checking up on the system, you'll find that the average system load has increased substantially, which is due to the system trying to persuade the disk to write to a faulty sector and the software RAID having to compensate, queuing the errors.
Depending on how often the defective sector is tried to be written to, the load can increase to values of 10 and above, rendering the system unusable. This is a good time to halt it and replace the defective drive, partition it and "mdadm
This may sound really horrible, but in practise it's usually less 60 minutes (counting from receipt of the first notification email) until normal operation can resume with such a system. This is assuming you have all spare parts stored somewhere on site.
In genereal, I've found software RAID1 and software RAID5 on Linux to be exceptionally stable. I'm also very happy about the performance, given that all I'm using is a bunch of el-cheapo ATA disks. As for reliability, I'm convinced it can't be beat in the consumers' price range, since I've seen too many consumer grade hardware RAID controllers go down in a swirl when putting more than a light load on them.
In the enterprise, I've seen companies move to software RAID on their Linux systems, because they found out that their only 5 years old enterprise hardware won't be getting any new spare parts anymore, which includes motherboards, CPU and IO controllers. Moving to software RAID on enterprise grade SCSI stuff allows them to move the entire system to another piece of hardware simply by moving the harddisk to it.
Oh dear... I can't imagine that a typical viewer of that station is computer-literate enough to cope with a wiped harddisk after an "install" of this "Zeta OS". Boy, I sure hope none of my friends installs it and asks me to fix things afterwards!
;) ) or a GDI-printer. Again, good luck guys, it won't work and your poor callcenter employees will only be asked a ton of stupid questions on how to get the whole junk to work together.
And this channel is brave, too. Imagine, they've been selling Zeta for over a year now, which also means that they've been selling Release Candidate software with a 30-day money-back guarantee... I wonder how many CDs get sent back...
As for the presentation, it's absolutely hilarious! They're showing off tons of small programs which don't need anything more than a P2 with 64 MB RAM anyway. As for hardware compatibility, they claim that driver development is so easy with Zeta OS, that they'll spit one out for any customer's hardware in no time at all... Good luck, you'll need it, guys.
I'm not sure whether that channel's aware of it, but they also have a positioning problem. All of the other stuff they sell is made to work solely on Windows. Now imagine what happens when somebody buys a copy of Zeta OS together with other stuff, like an el-cheapo digicam (it has 12 megapixels (!), but on the close-up you can see it only has a 3.2 megapixel sensor built in
The best part is when the moderator claims that Zeta OS can easily be installed alongside your existing OS, be it Windows or that "Linux".
Strangely enough, there are actually people who appreciate long release cycles! I have servers running woody which absolutely need nothing newer and I'm happy about the fact that I don't have to change everything every 18 months.
If the release cycle were to be shortened to said 18 months, it would be nice if Debian were to maintain older releases and not only the previous release, like it it now.
I recommend Debian to my customers as a server platform, exactly because it has the finest package management and the longest release cycles. When stability is the goal, Debian is the right choice!
What most people fail to realize is that having a RAID-subsytem for storage doesn't relieve you from organizing a proper backup scheme.
Any RAID, be it software or hardware, can only protect an individual failure of harddisks, but not against corrupt filesystems or users deleting files deliberately.
I think you're missing the point here. The term "engineer" as it is used nowadays implies that someone knows the scientific theory and is able to apply it in real world situations.
Most certifications abuse the term "engineer" as far as the scientific theory is concerned. The attendants are fed with pseudo-information and half-truths, leading them to believe they are worth their salt in real-life situations.
Suffice it to say that I have seen enough of those "engineers" fail when confronted even with the simplest of their original test scenarios.
This is indeed very true. The company I work at also has this tendency to look for such pseudo-engineers. Being an engineer myself (MSc. in manufacturing automation) I keep telling them to look for real qualification that lasts, not just short lived and narrow focused stuff that can be acquired in a few weeks' time.
Becoming a real engineer takes time - a lot of time - digging down deep into the core of the matter, not just scratching the surface and pretending to know what's going on.
Unfortunately, the original poster is absolutely right in assuming that the average Joe on the street (and it seems in some personnel departments as well) does not make the right distinction in this regard.