The rest of the world would just point at a different set of root servers.
Yes, they certainly could. They wouldn't. Most of Europe has been taken in completely by the Hobbesian Philosophy which posits that all things good come from a strong central authority, and splitting authority is inherently bad since it leads to chaos. It is through an implicit "social contract" that individuals form societies and collectively surrender authority over their possessions and themselves to the central authority, so that that authority may dispose of both in the best interests of the society. Breaking this "social contract" is tantamount to attacking society. Thus the Europeans will not build their own root servers unless they could destroy the authority of the existing root servers, because this would be a division of authority.
Obviously, Hobbesianism and Socialism complement each other beautifully. Less obviously (to some, at least), both are horribly broken and wrong.
I believe that concentration of authority invites its abuse, through incompetence or maliciousness or both. I believe that contracts should be signed by all parties involved before they should be considered binding. I believe that people own what they buy, build, discover, or contract, and that people are the authorities over what they own; this is what ownership means. I believe that charity is good, but that government enforcement of mandatory acts of charity are bad.
I also believe that it would be immoral to try to force these beliefs on anyone else. This is IMO the most important distinction between libertarians and fascists, that a true libertarian lets others walk their own paths.
I second this, too. We use Nagios at The Internet Archive, and a trivial Nagios plugin wrapped around/usr/sbin/hddtemp which can be used to monitor the temperature of your servers' hard drives.
Nagios can be made to do any of a variety of things when a plugin returns a "CRITICAL" status, like send emails, call beepers/cellphones, etc.
I totally second this motion. I've only done it once myself, after the CFO of The Sausalito Group had committed some unsavory, malicious, and illegal actions against me and my co-workers (retroactively declared our last few months' worth of paychecks to be loans, especially heinous in light that when TSG declared bankruptcy, its creditors could then seek to recover these sums from the employees). I asked the CFO, John Harrington, if I could record our conversation. He complied, and proceeded to admit to his misactions, and make various confused and contradictory assertions. He's no dummy, but he got really flustered and said all the wrong things anyway.
The moral here is to go ahead and try to get any potentially incriminating conversation on tape. Even though the other party could clam up and only say nonincriminating things, they could just as easily fuck up and spill the beans. You never know until you're there doing it.
The Petabox is shipped to a customer running Debian Linux by default (though of course you can install whatever you want), so there are a number of solutions to choose from. OpenAFS and (as you pointed out) GFS are made specifically for this kind of setup, providing fairly good abstraction of the underlying cluster and easy access to random data. Within The Archive, we have experimented with different approaches, the one currently in production using an API based on a UDP locator service and rsync.
Another approach uses a/net directory under which remote filesystems are NFS-mounted on demand (I'm not sure how it works, our chief sysadmin set it up for testing, but if/net/ia105783/0/foo is not mounted, and then you type 'ls/net/ia105783/0/foo' (or any other command which opens a hypothetical file off/net), the remote filesystem is automagically NFS-mounted so that the command can complete).
I'm not sure that we'll ever use it in production to access our distributed information, though; NFS has a very, very low error rate, but when you have thousands of NFS mounts going on at once (as we do NFS-mount users'/home directories everywhere), "very, very low" translates to "tripping over errors every few days". I've seen some really weird NFS failures and partial failures at The Archive, and I've written some software to be tolerant of them, but most of our software is not, and realistically speaking never will be. It's written to be tolerant of rsync errors instead. *shrug*, six of one, half a dozen of the other. This is one of those things where you need to just pick a solution and use it, whether it's OpenAFS, GFS, NFS, or some homespun thing. All have their pros and cons, and you'll learn to deal with their problems as you use them.
Aluminized plastics laminate the aluminum layer inside the plastic, so that the aluminum is not exposed to the outside. Building a laminate with aluminum on one side and plastic on the other would be significantly weaker than a plastic/aluminum/plastic composite, and would expose the aluminum to chemical change during the part of its lifetime not spent in hard vacuum. Thus it is important, in this application, that the plastic have a high degree of optical clarity.
Yeah.. I was gnashing my teeth a little when I read they were making it from mylar (polyethylene terephthalate). Not only will it degrade quickly, but it is also heavier, weaker, and less resilient than other available materials (16% denser than polycarbonate, 33% denser than polyurethane, either of which would have been stronger + more resilient).
I'm guessing, though, that they went with an off-the-shelf solution for the material to lower costs and expedite production. DuPont already mass manufactures aluminized mylar at this thickness, and I don't know if anyone manufactures similarly thin polycarbonate films, aluminized or not. Optically clear polyurethane is probably too new for anyone to be manufacturing it in film.
Yeah, that was more or less what I was thinking too, except I'd be tempted to use CPU temperature instead of power consumption, just because most hardware platforms already incorporate heat sensors for CPU's, and I don't know of any commodity hardware which will tell you how many watts your CPU is drawing.
It would be imperfect, because other things can influence CPU temperature -- some CPU's have variable-speed fans (especially on laptops), and other factors can cause temperatures to rise inside the case (though in my experience, CPU temperature only changes a little as ambient case temperature increases). So you want to figure on a "jigger factor" to account for this variance, and understand that your measurement isn't going to be totally reliable.
Moreover, surface temperature is not related perfectly linearly with watts dissipated. You should run some microbenchmarks to figure out what the curve looks like at 0%, 25%, 50%, 75%, 100% idle, and build a correction function to map temperature back to idleness fraction.
So on application startup, you would first hold the CPU idle for a couple of seconds, and measure its temperature. Call that 10% idle (or whatever jigger you empirically determine). Then run openssl for a few seconds, and measure its temperature. Call that 90% idle (or whatever jigger you empirically determine). Then start the application, periodically checking the temperature of the CPU. Use your correction function to map the measured temperature to idleness fraction.
I'm not sure how well it would work, but that's where I would start.
Whatever you decide on, please write Slashdot back and let us know what you did and how it turned out!
There is a good, high quality, low cost alternative to buying expensive load-balancing hardware. You can run LVS/IPVS on a linux box and turn it into an intelligent load-balancing router.
At The Archive we have dedicated LVS servers, but if you don't want to spend any extra $$ you can use a machine that is already providing some other service. You can use keepalived to make multiple LVS servers failover for each other.
I wrote a (very brief) HOWTO for setting up LVS/Keepalived. It is Archive-centric, but should be somewhat useful outside The Archive too. Just use rc.local or rc.inet2 or whatever instead of rc.final.
LVS/Keepalived (which is both, free and Free) has worked very well for us thusfar. Our www farm typically handles 30 to 60 http requests per second, with intermittent spiking above 250 http requests per second, and lvs01 sits at 99% idle all day.
You may not realize it, but the time you've spent in highschool has developed habits of self-discipline which are all too easily lost, and hard to regain. I would recommend against taking a year out from school for this reason.
To ease the transition, I would heartily recommend doing what I did, which is to take a year or two of classes at a Junior College (two in the California Bay Area are DeAnza (where I went after highschool) and Homestead). The atmosphere was a pleasant middle-ground between highschool and a "real" college, and they offered many "prep" courses specifically for students making the transition. It is also a good way to get some of your pre-req classes out of the way (the really basic, low-level classes) for a fraction of the cost (tuition at a Jr. College can be a tenth that of a Univerisity). I also lucked out in that some of the courses at DeAnza (higher levels of calculus, et al) were taught as three classes, whereas UCSC crammed them into one or two quarters. I feel that I retained more from these spread-out classes than I did from the crammed courses at UCSC. About 80% of the units I took at DeAnza wound up transferring to UCSC. Also, there were some courses offered at DeAnza (like Asian History) which were not offered by UCSC, so I got exposed to a greater variety of subjects this way.
One of my mistakes was to take too many pre-req's at DeAnza, which left me with nothing but hard courses to take at UCSC. This made my quarters more difficult than they should have been (taking four hard-core CS courses per quarter wore me down).
Some people say that the quality of education at a Jr. College is inferior to that from a University, but I disagree. Both will have some good instructors, and some not-so-good instructors, and the low-level classes are something you can learn pretty well from anywhere -- Calculus, for instance, hasn't really changed much since the 19th century:-) and the books will probably be the same no matter which school you attend. I know that my career has not suffered in any way for having attended two years of Jr. College before going to UCSC.
I have often looked to MySQL's html documentation as a shining example of what documentation should be like. It has a pretty good API, too. I usually haven't the time to do a really knock-up job of my own documentation, but I do try to look at MySQL's for my general approach, including the format (html). Here's an example of some of my documentation. I borrowed some pointers from the standard UNIX man page format, too, because it's been in use for a long time and developed into something reasonably complete and useful.
Another good example (imo) is the RFC which defines the NNTP protocol, rfc-977.
Know your audience -- the HOWTO I wrote was primarily for nonprogrammers with rudimentary knowledge of UNIX command line use (waybackup's primary expected users), but also for programmers who might be trying to debug or extend my code.
The most important thing with a SDK or any other tool, in my opinion, is use it a lot before publishing it, or even considering its development complete. Don't just come up with artificial examples, but actually use it internally to solve real-world problems. Your developers will unavoidingly find really annoying little problems in need of fixing, and come up with time-saving functions (perhaps just wrappers around already-existing API functions) which might need to be added to the SDK. Perhaps there's a function which seemed reasonable at the time, but in actual practice leads to runaway memory consumption. Maybe there are several functions which often get used together, but require the programmer to keep track of parameters which could get hidden internally instead. A nice long beta test, with the expectation of many programmer hours spent in reaction to user-reported errors/suggestions, is also often a good thing.
In fact, as a programmer I usually tailor my development effort towards getting something minimally useful first, and then actually use it, and let my use define further development. Features that sound good "on paper" are often a waste of time to develop because they don't actually get used. Also, thinking real hard at code does not necessarily make it better than code which has been shaped by real-world usage.
Anyway, I'll shut up now. Good luck with your SDK!
Well, the UI (the page linked to by slashdot here) isn't hosted at The Archive, for better or for worse.
The non-waybackmachine web servers (ten, at present) at archive.org proper are load-balanced via keepalived, and should stand up okay before a slashdotting. We learned things about the limitations of our webfarm from hosting the tsunami videos in the wake of that disaster, and beefed them up significantly.
Non-waybackmachine web traffic usually hovers around 40 to 60 hits per second, here, and we should be able to handle four or six times that before we start dropping anyone's connections.
What I posted was only half the story.. in full, the Archive gets its funding the same way that traditional libraries do, through grants from foundations, donations from private entities, and in-kind donations from corporations. The K/A Foundation and crawls-by-contract are just the sources of income with which I am most familiar (I'm just a dumb-ass programmer, so that part of the business isn't very visible to me, thus the oversight).
The Internet Archive has been around since 1996. We're funded by webcrawls-by-contract and by the Brewster Kahle Foundation. The Archive is a non-profit organization. We have no creditors. So relax.
It is in our charter to perpetuate our archives forever, and it's a charge we take seriously. As our hard drives go bad (and oh do they ever!) they are replaced by new ones, and we are protected from data loss by mirroring our archives across machines, and across data centers in different countries.
I think there are three factors coming into play, here:
They can make forking sound bad,
Forking actually can be bad for application developers,
Appeal to Hobbesian bias.
First and foremost, forking is an issue that not many people understand, and is therefore subject to demonization in the press. Since the objective of the authors of these FUD pieces is to make people want to not use Linux, they will pick on whatever aspect they can make sound bad. In this article, they never talk about why forking is bad, but they certainly talk about it as if it were a bad thing. And anyone who is paying attention to Linux has to admit that yes, forking is not only likely, but ongoing. This is seen as confirmation of the supposed problem.
That aside, forking does make life a little more difficult for developers of applications which would like to target multiple Linux distributions. Files are located in different places, init scripts are organized differently, different libraries or modules are installed (or not installed), and when some of the same libraries are installed, they are often different versions. Especially for shrink-wrapped software companies, which are accustomed to the relatively monolithic specifications of Windows (despite its own forking) or MacOS or even Solaris systems, these can make targetting the Linux market more difficult.
Also, I think we can blame Thomas Hobbes for infecting the world with his "centralized good, anarchy bad, mmmkay?" meme. Especially among the more highly educated, Hobbesian philosophy is accepted in America and outright embraced in England and other countries whose educational systems have been heavily influenced by England. Even those who have never had formal schooling in Hobbesian philosophy get exposed to Hobbesian bias, (e.g., when a news reporter talks about the "anarchy" in countries where there is social upheaval and widespread random violence, the listener is left with the impression that that is what anarchy (literally, "without center") means, which is right in line with the Hobbesian notion that anarchy inevitably causes social upheavel and violence). So to thinkers of Hobbesian bent, two Linuxes is necessarily, axiomatically, worse than one Linux, because it creates two "centers" for Linux. All things good come from having one strong center, according to Hobbes.
MIME-encoded text, actually. Which can be a pain-in-the-ass to deal with, especially for attachments.
mbox formatting doesn't care about the format of the text inside a text message. MIME is a standard, but it's just text in the end, so it gets stored into mbox-formatted files just fine.
As for attachments, yes, MIME is the RFC-compliant internet standard for storing attachments. So you should store them that way, and use any of a number of utilities conversant in this standard for extracting them as needed. Two alternatives are to either store them in decoded form (which is fine), or use some proprietary format (which is worse). If you store them in decoded form, then some structure will be necessary for storing them separately so that they can be easily matched with letters. It should be pretty easy to come up with a convenient format. My predisposition would be towards making a directory for each mbox file (perhaps with a.d suffix instead of.txt), with a subdirectory for each letter that contains attachments, and the decoded attachments in the subdirectory. To make them easy to carry around, you could zip or tar them up.
On the other other hand, you could use a format which is trying to become an internet standard, and is making some headway in that direction -- ARC format, which was made for archiving documents of various formats into a standard format. It's a little harder to manipulate than simple text files, though. Its complexity has discouraged casual home use, though many libraries, universities, and companies are adopting it for their archives.
I agree, text is the best way. Besides, the RFC-compliant internet standard email format is text. I store all of my mail in standard mbox format, in files whose names take the format:
[address].[end-date].[n].txt
For instance: billm@ciar.org.2005-04-12.00.txt
That file contains all of my email messages received to that account from some earlier date to 2005-04-12, inclusive. Getting the start date is easy (just head the file and grep out Date) so I don't bother encoding that in the filename. Their sizes range from a few hundred KB to about 100MB, depending on how many days' worth of email is in there. The [n] is there to avoid name collision when saving multiple files from the same account on the same day. When I feel the need to compress them, I use bzip2 on entire files. If you're more concerned about portability, then you can use gzip2 --best, which is only mildly worse and runs on all significant platforms (*nix/BSD/Solaris, MacOS9, MacOSX, Windows95 and later).
Interesting comments regarding RAID. They seem to defy common sense, but common sense is not always correct.
Yeah, though I'm not necessarily correct, either. There are plenty of smart IT professionals who disagree with with The Archive's conclusions regarding RAID. It may just be a contextual thing -- our data storage clusters are friggin' huge, and we only have three sysadmins, two of whom work part-time. A smaller system with more manpower and better discipline about following good procedures may fare better with RAID than we have.
Just out of curiosity, why did you end up going with your third choice for OS (Debian) rather than your first or second choices?
What I listed were my personal choices. At my own company (which only has two employees, one of them me) we use Slackware, and I periodically make sure all our software will "just work" on FreeBSD and Solaris, just in case we need to switch. But the PetaBox program at The Archive is/was very much a group effort. What I said about "go with what you know, go with what your supporting friends know" applies to business use as well. Joerg, our chief sysadmin, knows and likes Debian a lot, and had already put in a good chunk of effort on his own time tweaking a Debian box to play nicely in The Archive's larger framework. Our other sysadmins liked Debian okay, and at least knew a little more about it than Slackware (which is very old-school BSDish in ways). Brewster and Jon, two very important individuals at The Archive, were very much enamored of Debian's apt-get packaging system. FreeBSD wasn't even on the map; The Archive is very much Linux-centric.
Given all that, when I was told to go make the architectural decisions on this new cluster, Debian seemed like a no-brainer. I didn't even waste our time trying to convince anyone that we should go with anything else. I'm fairly nonpolitical about technology, but a lot of people feel very strongly that their choices are the only good choices. Debian was acceptable to everyone who needed to be able to work on the new systems, and choosing it meant that everyone could slide easily into development and administration without kicking up any fuss. I hate it when engineers argue, and am very much willing to suppress my own ideas about what constitutes a "best direction" and throw my support behind someone else's good idea, if it means the team can pull together and do what engineers are supposed to do. (qv this page outlining my philosophies on engineering.)
All in all, Debian was not a bad selection. It is a good, solid distribution, and it has served us very well.
When we developed the PetaBox at The Archive, the idea was to use off-the-shelf PC hardware and maximize GB/buck, while keeping cooling and power costs low. It's worked out pretty well. See also my unofficial PetaBox web page.
It turns out that you really don't need much of a PC to serve files. We underclocked the cheap little Via C3 processors to 800MHz to reduce power and heat, and they still troop along nicely. SATA is not necessary, since you're going to be bottlenecked on the network connection anyway. We used 512MB of RAM per node, but only because our system runs a gaggle of perl scripts to provide a variety of services (file searches, XML-based metadata updates, etc). If you're just going to be running NFS or Samba, 256MB is probably plenty (unless you choose to run Gigabit over a mere 32-bit PCI bus, in which case 512MB or 1GB would be better, so that you're reading more from filesystem cache and pounding the hard drives over your overloaded bus less). Gigabit ethernet is a must (we used 100bT for the PetaBox, which is annoying at times, but the cheaper 100bT 48-port switches were instrumental in keeping the overall price of the system low). We stuck four hard drives in each case, mostly from previous bad experiences trying to work with eight-disk machines. I can't say too much about the disk failure rate statistics which incited us to switch to Hitachi Deskstars, but I will say that I'm glad our PetaBox is using Deskstars and I will only use Deskstars in my workstation at home.
If you really, really want to keep the gigabit pipe full while pounding on your disks, then a newer bus like PCI-Express is necessary. Otherwise, I'd be tempted to go with an older, cheaper (and imo, more reliable) Pentium-II or -III based PC. You can get solid, reliable, well-cooled and well-dustfiltered early model VA Linux servers with 500MHz Pentium-III's for $200 or less. I must stress the importance of buying a really solid, rigid case. Over time, normal computer cases get all bendy-wendy, turning every part into a moving part, including parts you don't want to have moving at all. Fans will start sticking, motherboard traces will start breaking, etc. Most of the rack-mountable cases are made of good thick solid steel panels, which makes them heavy as f**kall, but IMO that's a small price to pay for a system that will run forever.
For operating system, the most important thing is to get something you know how to run and maintain, or can get help running and maintaining. If you have geek friends who are willing to provide technical assistance, find out what they know best and use that. A well-known operating system will probably be of more use to you than a technically better, but less well understood, operating system.
Having said that, my personal preference is Slackware Linux, because I appreciate its philosophy of keeping things simple, and preferences for packages which are the most stable, as opposed to newest versions or lots of features. My second choice would be FreeBSD. Third would be the OS we decided to use at The Archive for the PetaBox nodes, Debian Linux. But if all you know is Windows, then go ahead and use Windows.
Regarding RAID, it's been my experience working at The Archive that RAID is often more trouble than it's worth, especially when it comes to data recovery. In theory, recovery is easy, you just replace a bad disk and it will rebuild the missing data, and you're good to go. In practice, though, you will often not notice that one of your disks are borked until two disks or borked (or however many it takes for your RAID system to stop working), and then you have a major pain in the ass on your hands. At least with one filesystem per disk, you can attempt to save the filesystem by dd'ing the entire raw partition contents onto a different physical drive of same make + model, skipping bad sectors, and then running fsck on the good drive. But if you have
Screw public keys and I don't care how long a password I have to type: What's the best way to encrypt my network drive dedicated to porn?
IMO? You can "stretch" a relatively short OTP into the equivalent of a really long OTP by filtering it through a nonlinear feedback shift register algorithm, effectively turning an N-bit OTP into a slightly less than N**2 OTP. Keep this OTP in a thumb USB drive around your neck or on your keyring. The trivial perl script you use to apply it to data could also be on there, or you can keep it on the net. It doesn't matter too much if the adversary can find it, as long as they don't get their hands on your OTP. You can then safely encrypt TB's of pr0n or nookleear weppens schematics or whatever, using just a few MB's of OTP (which will stretch to TB's long via your NLFSR). You then need a process by which you can decrypt select data, make use of it, and then wipe the plaintext without leaving any of it behind to be recovered later when they kick down your door and drag all your equipment away.
How long before some big bad TLA gubmint agency will be able to break that encryption?
Until they either pull the USB drive from the chain on your neck, or flood the market with common hardware which spies on you (eg, USB controllers with radio transmitters or very large buffers for later retrieval) if they haven't already, or until they point the appropriate TEMPEST gear at your computer while you're accessing your sooper sekrit information.
I cannot recommend Bruce Schneier's Applied Cryptography enough. Buy it and read it and understand it, and you should have a pretty good idea of how to make your pr0n collection pretty safe.
An alternative to using RSA, DSA, or other encryption methods which rely on the difficulty of factoring is Elliptic Curve Encryption. As far as I know, nobody has come up with a theory for how quantum computing might be used to crack ECE. What I do not know is how hard it might be to come up with such a thing, or how long it might take to become available once the theory is mapped out.
RSA-oriented solutions are going to get the lion's share of the attention and development for a while, because it is so popular. Unless ECE catches on in a big way, and/or RSA is widely viewed as useless, there won't be much market incentive to develop an ECE-oriented solution. At least, that's my opinion. So in the meantime, ECE-encrypted data might be safer than RSA/DSA-encrypted data.
Slashdot Mirror
The rest of the world would just point at a different set of root servers.
Yes, they certainly could. They wouldn't. Most of Europe has been taken in completely by the Hobbesian Philosophy which posits that all things good come from a strong central authority, and splitting authority is inherently bad since it leads to chaos. It is through an implicit "social contract" that individuals form societies and collectively surrender authority over their possessions and themselves to the central authority, so that that authority may dispose of both in the best interests of the society. Breaking this "social contract" is tantamount to attacking society. Thus the Europeans will not build their own root servers unless they could destroy the authority of the existing root servers, because this would be a division of authority.
Obviously, Hobbesianism and Socialism complement each other beautifully. Less obviously (to some, at least), both are horribly broken and wrong.
I believe that concentration of authority invites its abuse, through incompetence or maliciousness or both. I believe that contracts should be signed by all parties involved before they should be considered binding. I believe that people own what they buy, build, discover, or contract, and that people are the authorities over what they own; this is what ownership means. I believe that charity is good, but that government enforcement of mandatory acts of charity are bad.
I also believe that it would be immoral to try to force these beliefs on anyone else. This is IMO the most important distinction between libertarians and fascists, that a true libertarian lets others walk their own paths.
-- TTK
I second this, too. We use Nagios at The Internet Archive, and a trivial Nagios plugin wrapped around /usr/sbin/hddtemp which can be used to monitor the temperature of your servers' hard drives.
Nagios can be made to do any of a variety of things when a plugin returns a "CRITICAL" status, like send emails, call beepers/cellphones, etc.
-- TTK
I totally second this motion. I've only done it once myself, after the CFO of The Sausalito Group had committed some unsavory, malicious, and illegal actions against me and my co-workers (retroactively declared our last few months' worth of paychecks to be loans, especially heinous in light that when TSG declared bankruptcy, its creditors could then seek to recover these sums from the employees). I asked the CFO, John Harrington, if I could record our conversation. He complied, and proceeded to admit to his misactions, and make various confused and contradictory assertions. He's no dummy, but he got really flustered and said all the wrong things anyway.
The moral here is to go ahead and try to get any potentially incriminating conversation on tape. Even though the other party could clam up and only say nonincriminating things, they could just as easily fuck up and spill the beans. You never know until you're there doing it.
-- TTK
Hi!
I'd be interested in corresponding about this, if you're willing. Could you please email me at "t t k at archive dot org"?
Thanks,
-- TTK
The Petabox is shipped to a customer running Debian Linux by default (though of course you can install whatever you want), so there are a number of solutions to choose from. OpenAFS and (as you pointed out) GFS are made specifically for this kind of setup, providing fairly good abstraction of the underlying cluster and easy access to random data. Within The Archive, we have experimented with different approaches, the one currently in production using an API based on a UDP locator service and rsync.
Another approach uses a /net directory under which remote filesystems are NFS-mounted on demand (I'm not sure how it works, our chief sysadmin set it up for testing, but if /net/ia105783/0/foo is not mounted, and then you type 'ls /net/ia105783/0/foo' (or any other command which opens a hypothetical file off /net), the remote filesystem is automagically NFS-mounted so that the command can complete).
I'm not sure that we'll ever use it in production to access our distributed information, though; NFS has a very, very low error rate, but when you have thousands of NFS mounts going on at once (as we do NFS-mount users' /home directories everywhere), "very, very low" translates to "tripping over errors every few days". I've seen some really weird NFS failures and partial failures at The Archive, and I've written some software to be tolerant of them, but most of our software is not, and realistically speaking never will be. It's written to be tolerant of rsync errors instead. *shrug*, six of one, half a dozen of the other. This is one of those things where you need to just pick a solution and use it, whether it's OpenAFS, GFS, NFS, or some homespun thing. All have their pros and cons, and you'll learn to deal with their problems as you use them.
-- TTK
Aluminized plastics laminate the aluminum layer inside the plastic, so that the aluminum is not exposed to the outside. Building a laminate with aluminum on one side and plastic on the other would be significantly weaker than a plastic/aluminum/plastic composite, and would expose the aluminum to chemical change during the part of its lifetime not spent in hard vacuum. Thus it is important, in this application, that the plastic have a high degree of optical clarity.
-- TTK
Yeah .. I was gnashing my teeth a little when I read they were making it from mylar (polyethylene terephthalate). Not only will it degrade quickly, but it is also heavier, weaker, and less resilient than other available materials (16% denser than polycarbonate, 33% denser than polyurethane, either of which would have been stronger + more resilient).
I'm guessing, though, that they went with an off-the-shelf solution for the material to lower costs and expedite production. DuPont already mass manufactures aluminized mylar at this thickness, and I don't know if anyone manufactures similarly thin polycarbonate films, aluminized or not. Optically clear polyurethane is probably too new for anyone to be manufacturing it in film.
-- TTK
Yeah, that was more or less what I was thinking too, except I'd be tempted to use CPU temperature instead of power consumption, just because most hardware platforms already incorporate heat sensors for CPU's, and I don't know of any commodity hardware which will tell you how many watts your CPU is drawing.
It would be imperfect, because other things can influence CPU temperature -- some CPU's have variable-speed fans (especially on laptops), and other factors can cause temperatures to rise inside the case (though in my experience, CPU temperature only changes a little as ambient case temperature increases). So you want to figure on a "jigger factor" to account for this variance, and understand that your measurement isn't going to be totally reliable.
Moreover, surface temperature is not related perfectly linearly with watts dissipated. You should run some microbenchmarks to figure out what the curve looks like at 0%, 25%, 50%, 75%, 100% idle, and build a correction function to map temperature back to idleness fraction.
So on application startup, you would first hold the CPU idle for a couple of seconds, and measure its temperature. Call that 10% idle (or whatever jigger you empirically determine). Then run openssl for a few seconds, and measure its temperature. Call that 90% idle (or whatever jigger you empirically determine). Then start the application, periodically checking the temperature of the CPU. Use your correction function to map the measured temperature to idleness fraction.
I'm not sure how well it would work, but that's where I would start.
Whatever you decide on, please write Slashdot back and let us know what you did and how it turned out!
-- TTK
There is a good, high quality, low cost alternative to buying expensive load-balancing hardware. You can run LVS/IPVS on a linux box and turn it into an intelligent load-balancing router.
At The Archive we have dedicated LVS servers, but if you don't want to spend any extra $$ you can use a machine that is already providing some other service. You can use keepalived to make multiple LVS servers failover for each other.
I wrote a (very brief) HOWTO for setting up LVS/Keepalived. It is Archive-centric, but should be somewhat useful outside The Archive too. Just use rc.local or rc.inet2 or whatever instead of rc.final.
LVS/Keepalived (which is both, free and Free) has worked very well for us thusfar. Our www farm typically handles 30 to 60 http requests per second, with intermittent spiking above 250 http requests per second, and lvs01 sits at 99% idle all day.
-- TTK
Don't worry about it until it actually happens -- which just might be never.
FCS's failure seemed likely to me years ago, and that impression has only been reinforced with time.
Let's just focus on doing constructive things, ourselves.
-- TTK
The Hun is a free online portal to sources of free porn:
The Hun's Thumbs-Up
The Hun's Yellow Pages
Happy Hunning!
-- TTK
The Internet Archive's Texts collection has about 25,000 books online, and is working with many partners to get more.
I have a list of about 1300 interesting texts hosted by The Archive, here, which has links to The Archive's details pages for those books.
Enjoy!
-- TTK
You may not realize it, but the time you've spent in highschool has developed habits of self-discipline which are all too easily lost, and hard to regain. I would recommend against taking a year out from school for this reason.
To ease the transition, I would heartily recommend doing what I did, which is to take a year or two of classes at a Junior College (two in the California Bay Area are DeAnza (where I went after highschool) and Homestead). The atmosphere was a pleasant middle-ground between highschool and a "real" college, and they offered many "prep" courses specifically for students making the transition. It is also a good way to get some of your pre-req classes out of the way (the really basic, low-level classes) for a fraction of the cost (tuition at a Jr. College can be a tenth that of a Univerisity). I also lucked out in that some of the courses at DeAnza (higher levels of calculus, et al) were taught as three classes, whereas UCSC crammed them into one or two quarters. I feel that I retained more from these spread-out classes than I did from the crammed courses at UCSC. About 80% of the units I took at DeAnza wound up transferring to UCSC. Also, there were some courses offered at DeAnza (like Asian History) which were not offered by UCSC, so I got exposed to a greater variety of subjects this way.
One of my mistakes was to take too many pre-req's at DeAnza, which left me with nothing but hard courses to take at UCSC. This made my quarters more difficult than they should have been (taking four hard-core CS courses per quarter wore me down).
Some people say that the quality of education at a Jr. College is inferior to that from a University, but I disagree. Both will have some good instructors, and some not-so-good instructors, and the low-level classes are something you can learn pretty well from anywhere -- Calculus, for instance, hasn't really changed much since the 19th century :-) and the books will probably be the same no matter which school you attend. I know that my career has not suffered in any way for having attended two years of Jr. College before going to UCSC.
Good luck figuring out your major!
-- TTK
I have often looked to MySQL's html documentation as a shining example of what documentation should be like. It has a pretty good API, too. I usually haven't the time to do a really knock-up job of my own documentation, but I do try to look at MySQL's for my general approach, including the format (html). Here's an example of some of my documentation. I borrowed some pointers from the standard UNIX man page format, too, because it's been in use for a long time and developed into something reasonably complete and useful.
Another good example (imo) is the RFC which defines the NNTP protocol, rfc-977.
Know your audience -- the HOWTO I wrote was primarily for nonprogrammers with rudimentary knowledge of UNIX command line use (waybackup's primary expected users), but also for programmers who might be trying to debug or extend my code.
The most important thing with a SDK or any other tool, in my opinion, is use it a lot before publishing it, or even considering its development complete. Don't just come up with artificial examples, but actually use it internally to solve real-world problems. Your developers will unavoidingly find really annoying little problems in need of fixing, and come up with time-saving functions (perhaps just wrappers around already-existing API functions) which might need to be added to the SDK. Perhaps there's a function which seemed reasonable at the time, but in actual practice leads to runaway memory consumption. Maybe there are several functions which often get used together, but require the programmer to keep track of parameters which could get hidden internally instead. A nice long beta test, with the expectation of many programmer hours spent in reaction to user-reported errors/suggestions, is also often a good thing.
In fact, as a programmer I usually tailor my development effort towards getting something minimally useful first, and then actually use it, and let my use define further development. Features that sound good "on paper" are often a waste of time to develop because they don't actually get used. Also, thinking real hard at code does not necessarily make it better than code which has been shaped by real-world usage.
Anyway, I'll shut up now. Good luck with your SDK!
-- TTK
Well, the UI (the page linked to by slashdot here) isn't hosted at The Archive, for better or for worse.
The non-waybackmachine web servers (ten, at present) at archive.org proper are load-balanced via keepalived, and should stand up okay before a slashdotting. We learned things about the limitations of our webfarm from hosting the tsunami videos in the wake of that disaster, and beefed them up significantly.
Non-waybackmachine web traffic usually hovers around 40 to 60 hits per second, here, and we should be able to handle four or six times that before we start dropping anyone's connections.
-- TTK
What I posted was only half the story .. in full, the Archive gets its funding the same way that traditional libraries do, through grants from foundations, donations from private entities, and in-kind donations from corporations. The K/A Foundation and crawls-by-contract are just the sources of income with which I am most familiar (I'm just a dumb-ass programmer, so that part of the business isn't very visible to me, thus the oversight).
-- TTK
Correction: Kahle-Austin Foundation, not "Brewster Kahle Foundation" :-P my mistrake
-- TTK
The Internet Archive has been around since 1996. We're funded by webcrawls-by-contract and by the Brewster Kahle Foundation. The Archive is a non-profit organization. We have no creditors. So relax.
It is in our charter to perpetuate our archives forever, and it's a charge we take seriously. As our hard drives go bad (and oh do they ever!) they are replaced by new ones, and we are protected from data loss by mirroring our archives across machines, and across data centers in different countries.
-- TTK
I think there are three factors coming into play, here:
They can make forking sound bad,
Forking actually can be bad for application developers,
Appeal to Hobbesian bias.
First and foremost, forking is an issue that not many people understand, and is therefore subject to demonization in the press. Since the objective of the authors of these FUD pieces is to make people want to not use Linux, they will pick on whatever aspect they can make sound bad. In this article, they never talk about why forking is bad, but they certainly talk about it as if it were a bad thing. And anyone who is paying attention to Linux has to admit that yes, forking is not only likely, but ongoing. This is seen as confirmation of the supposed problem.
That aside, forking does make life a little more difficult for developers of applications which would like to target multiple Linux distributions. Files are located in different places, init scripts are organized differently, different libraries or modules are installed (or not installed), and when some of the same libraries are installed, they are often different versions. Especially for shrink-wrapped software companies, which are accustomed to the relatively monolithic specifications of Windows (despite its own forking) or MacOS or even Solaris systems, these can make targetting the Linux market more difficult.
Also, I think we can blame Thomas Hobbes for infecting the world with his "centralized good, anarchy bad, mmmkay?" meme. Especially among the more highly educated, Hobbesian philosophy is accepted in America and outright embraced in England and other countries whose educational systems have been heavily influenced by England. Even those who have never had formal schooling in Hobbesian philosophy get exposed to Hobbesian bias, (e.g., when a news reporter talks about the "anarchy" in countries where there is social upheaval and widespread random violence, the listener is left with the impression that that is what anarchy (literally, "without center") means, which is right in line with the Hobbesian notion that anarchy inevitably causes social upheavel and violence). So to thinkers of Hobbesian bent, two Linuxes is necessarily, axiomatically, worse than one Linux, because it creates two "centers" for Linux. All things good come from having one strong center, according to Hobbes.
Anyway, just my two cents.
-- TTK, Anarcho-capitalist with biases of his own
MIME-encoded text, actually. Which can be a pain-in-the-ass to deal with, especially for attachments.
mbox formatting doesn't care about the format of the text inside a text message. MIME is a standard, but it's just text in the end, so it gets stored into mbox-formatted files just fine.
As for attachments, yes, MIME is the RFC-compliant internet standard for storing attachments. So you should store them that way, and use any of a number of utilities conversant in this standard for extracting them as needed. Two alternatives are to either store them in decoded form (which is fine), or use some proprietary format (which is worse). If you store them in decoded form, then some structure will be necessary for storing them separately so that they can be easily matched with letters. It should be pretty easy to come up with a convenient format. My predisposition would be towards making a directory for each mbox file (perhaps with a .d suffix instead of .txt), with a subdirectory for each letter that contains attachments, and the decoded attachments in the subdirectory. To make them easy to carry around, you could zip or tar them up.
On the other other hand, you could use a format which is trying to become an internet standard, and is making some headway in that direction -- ARC format, which was made for archiving documents of various formats into a standard format. It's a little harder to manipulate than simple text files, though. Its complexity has discouraged casual home use, though many libraries, universities, and companies are adopting it for their archives.
-- TTK
I agree, text is the best way. Besides, the RFC-compliant internet standard email format is text. I store all of my mail in standard mbox format, in files whose names take the format:
[address].[end-date].[n].txt
For instance: billm@ciar.org.2005-04-12.00.txt
That file contains all of my email messages received to that account from some earlier date to 2005-04-12, inclusive. Getting the start date is easy (just head the file and grep out Date) so I don't bother encoding that in the filename. Their sizes range from a few hundred KB to about 100MB, depending on how many days' worth of email is in there. The [n] is there to avoid name collision when saving multiple files from the same account on the same day. When I feel the need to compress them, I use bzip2 on entire files. If you're more concerned about portability, then you can use gzip2 --best, which is only mildly worse and runs on all significant platforms (*nix/BSD/Solaris, MacOS9, MacOSX, Windows95 and later).
-- TTK
Interesting comments regarding RAID. They seem to defy common sense, but common sense is not always correct.
Yeah, though I'm not necessarily correct, either. There are plenty of smart IT professionals who disagree with with The Archive's conclusions regarding RAID. It may just be a contextual thing -- our data storage clusters are friggin' huge, and we only have three sysadmins, two of whom work part-time. A smaller system with more manpower and better discipline about following good procedures may fare better with RAID than we have.
Just out of curiosity, why did you end up going with your third choice for OS (Debian) rather than your first or second choices?
What I listed were my personal choices. At my own company (which only has two employees, one of them me) we use Slackware, and I periodically make sure all our software will "just work" on FreeBSD and Solaris, just in case we need to switch. But the PetaBox program at The Archive is/was very much a group effort. What I said about "go with what you know, go with what your supporting friends know" applies to business use as well. Joerg, our chief sysadmin, knows and likes Debian a lot, and had already put in a good chunk of effort on his own time tweaking a Debian box to play nicely in The Archive's larger framework. Our other sysadmins liked Debian okay, and at least knew a little more about it than Slackware (which is very old-school BSDish in ways). Brewster and Jon, two very important individuals at The Archive, were very much enamored of Debian's apt-get packaging system. FreeBSD wasn't even on the map; The Archive is very much Linux-centric.
Given all that, when I was told to go make the architectural decisions on this new cluster, Debian seemed like a no-brainer. I didn't even waste our time trying to convince anyone that we should go with anything else. I'm fairly nonpolitical about technology, but a lot of people feel very strongly that their choices are the only good choices. Debian was acceptable to everyone who needed to be able to work on the new systems, and choosing it meant that everyone could slide easily into development and administration without kicking up any fuss. I hate it when engineers argue, and am very much willing to suppress my own ideas about what constitutes a "best direction" and throw my support behind someone else's good idea, if it means the team can pull together and do what engineers are supposed to do. (qv this page outlining my philosophies on engineering.)
All in all, Debian was not a bad selection. It is a good, solid distribution, and it has served us very well.
-- TTK
When we developed the PetaBox at The Archive, the idea was to use off-the-shelf PC hardware and maximize GB/buck, while keeping cooling and power costs low. It's worked out pretty well. See also my unofficial PetaBox web page.
It turns out that you really don't need much of a PC to serve files. We underclocked the cheap little Via C3 processors to 800MHz to reduce power and heat, and they still troop along nicely. SATA is not necessary, since you're going to be bottlenecked on the network connection anyway. We used 512MB of RAM per node, but only because our system runs a gaggle of perl scripts to provide a variety of services (file searches, XML-based metadata updates, etc). If you're just going to be running NFS or Samba, 256MB is probably plenty (unless you choose to run Gigabit over a mere 32-bit PCI bus, in which case 512MB or 1GB would be better, so that you're reading more from filesystem cache and pounding the hard drives over your overloaded bus less). Gigabit ethernet is a must (we used 100bT for the PetaBox, which is annoying at times, but the cheaper 100bT 48-port switches were instrumental in keeping the overall price of the system low). We stuck four hard drives in each case, mostly from previous bad experiences trying to work with eight-disk machines. I can't say too much about the disk failure rate statistics which incited us to switch to Hitachi Deskstars, but I will say that I'm glad our PetaBox is using Deskstars and I will only use Deskstars in my workstation at home.
If you really, really want to keep the gigabit pipe full while pounding on your disks, then a newer bus like PCI-Express is necessary. Otherwise, I'd be tempted to go with an older, cheaper (and imo, more reliable) Pentium-II or -III based PC. You can get solid, reliable, well-cooled and well-dustfiltered early model VA Linux servers with 500MHz Pentium-III's for $200 or less. I must stress the importance of buying a really solid, rigid case. Over time, normal computer cases get all bendy-wendy, turning every part into a moving part, including parts you don't want to have moving at all. Fans will start sticking, motherboard traces will start breaking, etc. Most of the rack-mountable cases are made of good thick solid steel panels, which makes them heavy as f**kall, but IMO that's a small price to pay for a system that will run forever.
For operating system, the most important thing is to get something you know how to run and maintain, or can get help running and maintaining. If you have geek friends who are willing to provide technical assistance, find out what they know best and use that. A well-known operating system will probably be of more use to you than a technically better, but less well understood, operating system.
Having said that, my personal preference is Slackware Linux, because I appreciate its philosophy of keeping things simple, and preferences for packages which are the most stable, as opposed to newest versions or lots of features. My second choice would be FreeBSD. Third would be the OS we decided to use at The Archive for the PetaBox nodes, Debian Linux. But if all you know is Windows, then go ahead and use Windows.
Regarding RAID, it's been my experience working at The Archive that RAID is often more trouble than it's worth, especially when it comes to data recovery. In theory, recovery is easy, you just replace a bad disk and it will rebuild the missing data, and you're good to go. In practice, though, you will often not notice that one of your disks are borked until two disks or borked (or however many it takes for your RAID system to stop working), and then you have a major pain in the ass on your hands. At least with one filesystem per disk, you can attempt to save the filesystem by dd'ing the entire raw partition contents onto a different physical drive of same make + model, skipping bad sectors, and then running fsck on the good drive. But if you have
Screw public keys and I don't care how long a password I have to type: What's the best way to encrypt my network drive dedicated to porn?
IMO? You can "stretch" a relatively short OTP into the equivalent of a really long OTP by filtering it through a nonlinear feedback shift register algorithm, effectively turning an N-bit OTP into a slightly less than N**2 OTP. Keep this OTP in a thumb USB drive around your neck or on your keyring. The trivial perl script you use to apply it to data could also be on there, or you can keep it on the net. It doesn't matter too much if the adversary can find it, as long as they don't get their hands on your OTP. You can then safely encrypt TB's of pr0n or nookleear weppens schematics or whatever, using just a few MB's of OTP (which will stretch to TB's long via your NLFSR). You then need a process by which you can decrypt select data, make use of it, and then wipe the plaintext without leaving any of it behind to be recovered later when they kick down your door and drag all your equipment away.
How long before some big bad TLA gubmint agency will be able to break that encryption?
Until they either pull the USB drive from the chain on your neck, or flood the market with common hardware which spies on you (eg, USB controllers with radio transmitters or very large buffers for later retrieval) if they haven't already, or until they point the appropriate TEMPEST gear at your computer while you're accessing your sooper sekrit information.
I cannot recommend Bruce Schneier's Applied Cryptography enough. Buy it and read it and understand it, and you should have a pretty good idea of how to make your pr0n collection pretty safe.
-- TTK
An alternative to using RSA, DSA, or other encryption methods which rely on the difficulty of factoring is Elliptic Curve Encryption. As far as I know, nobody has come up with a theory for how quantum computing might be used to crack ECE. What I do not know is how hard it might be to come up with such a thing, or how long it might take to become available once the theory is mapped out.
RSA-oriented solutions are going to get the lion's share of the attention and development for a while, because it is so popular. Unless ECE catches on in a big way, and/or RSA is widely viewed as useless, there won't be much market incentive to develop an ECE-oriented solution. At least, that's my opinion. So in the meantime, ECE-encrypted data might be safer than RSA/DSA-encrypted data.
-- TTK