I am a software engineer working at The Internet Archive, and I write parallel software every day (sometimes with PVM for "real" applications, but more often as throwaway perl slammed out on the command line, using open3() to open several simultaneous subprocesses, sometimes fed data by the parent but more often each reading from a different data file). Much of what I do is "trivially parallelizable", meaning it's pretty easy to make scale across multiple processors or machines. It is my impression that most real-life problems seen by most businesses are trivially parallelizable, with the rare exceptions hogging all the attention by dint of being more interesting.
My workstation is a single-processor machine, but I have at my exclusive disposal a dual-xeon machine and two AMD dual-core machines. I'm always scp'ing my work up to them from my workstation so I can take advantage of their multi-process goodness. (Developing while ssh'd into those machines is usually not a good idea, since the network likes to go down or slow down a lot between Archive HQ and our datacenters, and our HQ firewall blocks PVM so I can't just make my workstation the PVM master node with the other three machines slaves.)
When I read this article, my initial reaction was "Enthusiasts, hell! I want as many of these as I can get for servers!" (assuming this 4x4 product is significantly cheaper than current dual-opteron products -- we're a non-profit, without a lot to spend on hardware, and we're always running on the edge of starvation. But maybe that's a bad assumption and these will be prohibitively pricey).
If someone offered me a 4x4 or 8x8 for my desktop, though, I'd accept it gladly, and make good use of it, parsing/analyzing Archive metadata, processing multiple simultaneous http streams (we use a lot of http-rpc here, and xml data representation which means each http-rpc stream can suck down a lot of processing power), md5'ing multiple files in parallel, and the like. I'd probably also make more extensive use of bzip2 than I do currently:-)
My datasets commonly consist of hundreds or thousands of files, each of which can be processed in parallel, so I can keep throwing cores at the problem with near-linear scalability until I grind against disk or bus bandwidth limits (at which point the data needs to start out distributed in order to keep scaling).
The Geode under consideration is the Geode NX 1500, which draws 7 watts at load. AFAIK we are only benchmarking it on how well it processes book scans (which is the majority of our processing load for now and the foreseeable future), and of course how easily it can fill gig-e.
Thanks for forwarding the comparison article:-) it made for an interesting read (the references to Cyrix gave me quite the nostalgic kick!).
The gig-e NIC in question was an Intel EtherPro1000, which is our standard card at The Archive (it's reliable, easy to fill to near-theoretical capacity without resorting to jumbo frames, and well-supported under linux). We measured its performance over "native rsync" (which is to say, without ssh encryption), without compression, which is how data is most commonly transferred between hosts at The Archive. (qv: "rsync ia300130.us.archive.org::items_0" to see what a data node looks like through rsync.. the directories with "d---------" permissions are "darkened" items, taken offline but not deleted.)
I don't have the exact numbers in front of me, but IIRC the C3 was able to fill it to about 35 megabytes/second.
Our current solution is based on the VIA EPIA-M micro-itx motherboard, which uses the VIA Apollo CLE266 chipset. It and four hard drives fit nicely into a moderate-depth 1U case, and it does PXE boot (which we (and Capricorn) use to mass-image new racks with their operating systems and filesystem layouts). Its topology is quite simple, but frankly we didn't need anything more at the time (plain-jane PCI, with embedded 100bT and ATA/133), and its power and heat characteristics are very good.
If you really, really don't need anything more than a simple dumb storage brick, and don't need full gig-E bandwidth, then this hardware does just fine. What we found is that, after deciding on this hardware, subsequent decisions about the handling of metadata on the storage nodes and some other things increased our need for storage nodes' processing capability. Also, increases in storage density (1TB per node in early 2004, vs 3TB per node today) renders 100bT insufficient for some of the operations we need to occasionally perform.
As for what might replace the VIA EPIA-M at The Archive, I cannot yet say. There's a lot of testing of prospective hardware needing to get done. I wouldn't want to suggest a chipset which later turned out to have some horrible flaw that only showed up during stress-testing.
It's been almost exactly two years since we put together the first petabox rack, and both the technology and our understanding of the problem have progressed since then. We've been working towards agreement on what the next generation of petabox hardware should look like, but in the meantime there are a few differing opinions making the rounds. All of them come from very competent professionals who have been immersed in the current system, so IMO all of them are worthy of serious consideration. Even though we'll only go with one of the options, a different option might be better suited to your specific needs.
One that is a no-brainer is: SATA. The smaller cables used by SATA are a big win. Their smaller size means (1) better air flow for cooling the disks, and (2) fewer cable-related hardware failures (fewer wires to break, and more flexibility). Very often when Nagios flags a disk as having gone bad, it's not the disk, but rather the cable that needs reseating or replacing.
Choosing the right CPU is important. The VIA C3's we started with are great for low power consumption and heat dissipation, but we underestimated the amount of processing power we needed in our "dumb storage bricks". The two most likely successors are the Geode and the (as-yet unreleased, but RSN) new low-power version of AMD's dual-core Athlon 3800+. But depending on your specific needs you might want to just stick with the C3's (which, incidentally, cannot keep gig-e filled, so if you wanted full gig-e bandwidth on each host, you'll want something beefier than the C3).
It has been pointed out that we could get better CPU/U, disks/U, $$/U, and power/U by going with either a 16-disks-in-3U or 24-disks-in-4U solution (both of which are available off-the-shelf), compared to 4-disks-in-1U (our current hardware). This would also make for fewer hosts to maintain, and to monitor with the ever-crabby Nagios, and require us to run fewer interconnects. Right now it looks like we'll probably stick with 4-in-1U, though, for various reasons which are pretty much peculiar to our situation.
I heartily recommend Capricorn's petabox hardware for anyone's low-cost storage cluster project, if for no other reason than because a lot of time, effort, brain-cycles, and experimentation was devoted to designing the case for optimizing air flow over the disks (and figuring out which parts of the disks are most important to keep cool). Keeping the disks happy will save you a lot of grief and money. When you're keeping enough disks spinning, having a certain number of them blow up is a statistical certainty. Cooler disks blow up less frequently. Most cases do a really bad job of assuring good air flow across the disks -- the emphasis is commonly on keeping the CPU and memory cool. But in a storage cluster it's almost never the CPU or memory that fails, it's the drives.
Even though the 750GB Seagates appear to provide less bang-for-buck than smaller solutions (400GB, 300GB), the higher data storage density pays off in a big way. Cramming more data into a single box means amortizing the power/heat cost of the non-disk components better, and also allows you better utilization of your floorspace (which is going to become very important, if you really are looking to scale this into the multi-petabyte range).
When dealing with sufficiently large datasets, silent corruption of data during a transfer becomes inevitable, even using transport protocols which attempt to detect and correct such corruptions (since the corruption could have occurred "further upstream" in the hardware than the protocol is capable of seeing). We have found it necessary to keep a record of the MD5 checksums of the contents of all our data, and add a "doublecheck" step to transfers: perform the transfer (usually via rsync), make sure the data is no longer being cached in memory (usually by performing additional transfers), and then recompute the MD5 checksums on the destination host and c
Perhaps you're right -- I have to admit that the only experiences I have had with XML have been with C and perl DOM parsers, and PHP5 (which handles XML as first-class data entities, and for all I know may be using the C DOM XML library underneath). I have not worked with a streaming XML parser -- though, are you sure these parsers are as flexible as you've made them out? The XML specification, section 5.1 seems to imply that a parser that parses a later section of an XML document without parsing some earlier section is out-of-spec (and for good reason -- earlier data can change the meaning of later data, and there's no table of contents in an XML document which gives the lengths of independent sections (which would mostly solve the issue)).
Sure there are other ways to represent data -- but not everyone will understand them, and there won't always be easy to understand rules to expand them [..]
This is perhaps the strongest argument in favor of using XML. Everyone's already using it, and expanding a dataset is straightforward. All I can do is wish that there was a sane, well-crafted, easily-parsed, fault-tolerant binary specification which enjoyed the same ubiquity. One of the points I was trying to make in my original post, though, is that if your situation was already well-suited to using XML, then you were in a position to create and use a nonstandard specification which lacks XML's intrinsic faults.
As Jesapoo points out, it's not about size, but as important as orbit eccentricity is material composition. Planets are historically categorized into two buckets based on their composition -- "terrestrial", which are mostly rock (mercury, venus, earth, mars), and "jovian", which are mostly gas (jupiter, saturn, uranus, and neptune). And then there are comets, which are mostly dirty ice and frozen gas with some rocks.
Pluto is cometary in composition, which has led some to classify it as a comet rather than as a planet. Frankly I can see the argument. Perhaps the best way out is to define "planet" such that some comets can be planets?
XML isn't a bad idea, for instance-- it gives a standard method of defining data transport, for instance.
I've been wondering whether XML is really all that great of an idea. It makes sense to use it when, as you say, you need a standard way of representing data across multiple dissimilar systems. But a key notion behind XML is that unless an XML dataset is well-formed, attempts to parse that XML should fail.
This means XML makes sense to use when you need to represent data across multiple dissimilar systems, and you have control over the formation of datasets. Otherwise, if one system generates imperfectly formed XML, the whole system of systems grinds to a halt. Therefore, you either need sufficient control over all sources of data to be able to fix the way the data representation is generated, or you need to only use bug-free software.
If you have sufficient control over the way data is represented to be able to fix it when it misgenerates the XML, then you don't really need XML, and can instead choose a data representation more appropriate for your needs -- something that doesn't bloat your data out to 5x its original size, and doesn't require you to parse N records before you can parse out the N+1'th record, and doesn't require you to throw out an entire dataset if there is a problem with some part of it (which is like refusing to extract any files from a tarball if the last record in the tarball is truncated (if this hasn't happened to you yet, just wait, it will! and then you'll be glad that tar will extract all the files it can)).
If your system only uses bug-free software, and is sufficiently complex to do something useful,.. I don't know, I'll buy you a drink or something. Congradulations, you're ahead of the rest of us.
That having been said, there certainly seem to be a lot of people out there who are perfectly happy using XML. Maybe my experiences with it have just been unusually bad, or maybe those people don't mind XML's drawbacks. It's been my experience that representation errors are common (and sometimes what an XML parser considers a representation error is actually a desirable feature), and that software is more useful when it proceeds despite adverse conditions, when it can. But my mind is not closed on the subject. There may be something I'm missing, and I don't want to miss it if life throws it at me.
As for web 2.0 as a whole, I see a more complex picture. Yes, it's been unduly hyped, but it's also putting a label on a body of concepts with which the industry is trying to come to terms. There's a vague notion that dynamic web services which share information across contexts can be good, but the why and when of it is still unclear. I do not fault those who try to make more sense of it. Fault lies with those who focus unduly on the tools people have used thusfar to create useful services (Javascript, XML, PHP, Python, etc), to the neglect of the reasons those services have been useful (which are partly technological, but mostly social). I suspect the missing piece is something very simple, like "develop services which satisfy an existing need", but time will tell.
At The Internet Archive we have about 120,000 audio and live music shows, occupying about 53TB of disk space. We're always trying to think of new and better ways to present it to our users.
I'm going to look at all the solutions people have suggested here and try to glean some usability tips which might be implementable on top of our existing interface. Please keep up the good suggestions!
My usual criterion for the quality of a video card is: "how well does XFree86 support it?" (or I guess XOrg, now). A $50 or $30 card which works well for making xterms and Netscape appear on the screen is exactly what I want (and need).
An advantage to being happy with inexpensive cards is that it becomes feasible to purchase a few of them, so that you can standardize sets of machines on them. That goes double for network cards. It's handy to be able to swap harddrives between machines with impunity, and have video + network "just work" without needing to fiddle with modprobe (or with XF86Config/xorg.conf).
My old crop of machines was standardized on ne2k-pci compatible cards, but I'm transitioning to eepro1000:-) It's a wonderful world, where gig-e cards can be had for only $50 (or $25, if I wasn't stuck on the high quality etherpros), and 8-port gig-e switches for only $100. My standard video card is still the ATI Xpert98, though. Maybe it's time to restandardize there too.
I read the article you referenced, and I read the proposed amendment itself, and I do not see where it requires, or even mentions, logging the connections ISP customers make. It appears to only require recording who the ISP's customers are, and retaining information on a customer for up to a year after that customer stops subscribing to the service.
Not that I like the amendment, nor do I trust our government to refrain from abusing their access to this information, but it seems to be a much less big of a deal than everyone is assuming.
The text of the proposed amendment (it's short!):
edited because Slashdot originally refused to accept, saying: "Lameness filter encountered. Post aborted! Reason: Please use fewer 'junk' characters.. now what does that say about the state of legislation?:-)
TITLE VI RECORDS RETENTION
SEC. 601. RECORD RETENTION REGULATIONS REQUIRED.
Title VII of the Communications Act of 1934 (47
USC 601 et seq.) is further amended by adding after
section 718 (as added by section 501 of this Act) the following new section:
SEC. 719. RECORD RETENTION BY PROVIDERS OF INTERNET ACCESS SERVICE.
(a) REGULATIONS REQUIRED Within 90 days
after the date of enactment of this section, the Commission shall prescribe regulations requiring each provider of
Internet access services to retain records to permit the
identification of subscribers to such services for appropriate law enforcement purposes. Such records shall, in
accordance with such regulations, be retained for not less
than one year after a subscriber ceases to subscribe to
such services.
(b) DEFINITION For purposes of this section:
(1) INTERNET The term `Internet' means the combination of computer facilities and electromagnetic transmission media, and related equipment and software, comprising the interconnected worldwide network of computer networks that employ the Transmission Control Protocol/Internet Protocol or any successor protocol to transmit information.
(2) INTERNET ACCESS SERVICE The term `Internet access service' means a service that enables users to access content, information, electronic mail, or other services offered over the Internet, and may also include access to proprietary content, information, and other services as part of a package of services offered to consumers. Such term does not in-
clude telecommunications services.''.
not to nitpick either, but those Russkie defence systems are useless against most US weapon systems that rely on the kinetic penetrator concept (though they MIGHT snap the rod of a TOW missile, the latter being the cheapest weapon of choice here).
If you're going to nitpick, you might want to know what you're talking about, first:-)
The earliest generations of Eexplosive Reactive Armor (ERA) were not effective against Armor Piercing, Fin Stabilized, Discarding Sabot weapons (APFSDS, KE weapons, aka long-rod penetrators), but later generations of Soviet ERA were (and are) quite effective against KE weapons. Kontakt-5 "Heavy ERA", for instance, utilizes a hinged flyplate which snaps off the front the penetrator and induces significant yaw (tumble) in the remainder. This loss of length reduces the KE penetrator's ability to defeat the passive armor underneath the ERA (hypervelocity long-rod penetration is proportional to rod length, see Anderson's and Odermatt's publications. It is common practice in penetration studies to normalize penetration figures to P/L (the ratio of penetration depth to penetrator rod length)). Kontakt-5 was first fielded in 1985, so ERA that protects against KE weapons has been with us for a while.
The most modern APFSDS use a segmented rod which minimizes the impact of this kind of ERA, but ERA has advanced in the meantime too. The very latest design, the ukranian Nozh ERA uses linear shaped charges to cut the penetrator, potentially in multiple locations down its length, and also induces yaw.
Also, the TOW missile uses a shaped charge to achieve penetration, and is not a KE penetrator. It has no "rod" (unless you're talking about the spur for the detonation probe, which sticks way out in front so that the shaped charge can detonate an optimal distance from the target). TOW missiles are vulnerable to Drozd, ARENA, and all of the various flavors of ERA used today, qv:
Just a little nit to pick: Drozd has been deployed on T-55 and T-80 family tanks, but T-90 uses the newer ARENA system. Also, using ARENA precludes mounting Explosive Reactive Armor modules, the latest versions of which are useful against APFSDS threats (which Drozd and ARENA are not), so it's not exactly a silver bullet.
ObPlug: more on various kinds of active defense systems can be found on this page.
It's more like 0.15%, if they use the same compression and content selection criteria as the Internet Archive. If they eschewed with all non-html content (graphics files, pdf's, etc) that would go up quite a bit. If they used better compression (the Archive uses gzip) it would go up some more.
An average crawl of the public web, minus files which are "too large" (not sure what the threshold is for that), makes about 55TB of gzipped archive. 80GB / 55000GB = 0.0014545, or about 0.15%.
Google on "tank armor" and my site pops right up. I spend a lot of time researching tank technology, especially composite armors, both online and in print. Much of my effort is spent on Russian, Ukranian, and German websites, patent filings, and company brochures. The International Journal of Impact Engineering is great when I can get it, but there's a wealth of information to be mined from analyzing the latest German innovations, or the flood of ex-Soviet designs previously hidden behind the iron curtain, now revealed to the west for the very first time.
German defense engineers are masters of the composite armor arrangement. Being a western country, they have access to highly precise fabrication facilities and the most advanced technical materials money can buy, and they have the money to buy it. Seeing what they buy, how they shape it, and how much they put where is invaluable to my interests.
Russia and Ukraine, on the other hand, faced the prospect of fighting NATO on a shoestring budget. Their materials were modest, and they couldn't rely on their manufacturing facilities to consistently produce high-precision machining of components, but their engineers were (and continue to be) top-notch. They accepted the limitations of the materials and tools they had to work with, and found ways to arrange relatively humble metals, ceramics, and plastics such that they synergistically produced very advanced composite armor effects. They designed their armor systems such that they would continue to provide high levels of protection even if the factory mass-producing them couldn't weld a straight line to save their lives. While engineers in the west pushed the limits of what factories were capable of producing, often with disasterous results (qv the failed joint German/American MBT-70 project) their eastern counterparts made damn sure their own designs could be churned out quickly, cheaply, and in vast numbers (exceptions like the T-64/T-80 family notwithstanding). This holds special appeal to me because these are armor effects that can be replicated in a humble home workshop, using inexpensive commodity plastics and ceramics. If I don't measure it perfectly well, or the edges don't line up quite right, hey! It still works! That is more valuable to me than some super-advanced design that would take $1000/lb and a bunch of unobtainable federal licenses before I could even consider putting tab A into slot B.
Besides lots of oil, orphans, and bodybags, one of the things Americans found in our latest tragic military misadventure was a bunch of tanks which had been upgraded with armor modules purchased from Russia and welded into place in Iraq. These modules had been encountered in the 1991 war, but for some reason detailed analysis of these modules' composition didn't see the light of public attention until now. These "Enigma" armor modules repeatedly resisted the shaped charge warheads of Dragon and Milan anti-tank missiles and the depleted uranium projectiles issuing from 30mm Bushmaster autocannons, though they did not fare so well against Abrams' 120mm main guns. Still, that's not bad at all for a relatively lightweight steel box welded onto the turret of a forty year old tank (the T-55). Enigma modules were pried loose and cut open, and do you know what was found inside? Simple layers of steel, natural rubber, and construction-grade aluminum, with airgaps between each three-layer array. Some boxes were missing a layer or two here and there, and the adhesion between layers was sloppy at best. This was an example of the soviet bulging armor, doing what it does best, on the cheap. The effect this combination produces belongs in the realm of rocket science, but putting the boxes themselves together hardly takes a highschool diploma.
I sit corrected.. not flash animations. That's what I get for posting before actually downloading. I'll throw these samples at the deriver and see if it breaks any teeth.
I intend to at least take a look at it. They'd make a nice addition to our movies collection (currently hosting about 30,000 moving image items). Our deriver software (which generates those Mpeg4's et al) doesn't currently grok flash, but that might change.
I would use this to connect 10Ge switches together, not to connect directly to individual servers in the network. Three 32-port 100Ge switches, linked to 32 32-port 10Ge switches by three links (ie, so each 10Ge switch had three links to different 100Ge switches and one link to each of 29 hosts) would allow any two hosts in a 928-node cluster to communicate at full 10Ge capacity (or, more likely, communicate with any number of hosts in that cluster at 10Ge aggregate capacity).
I have traditionally been willing to trade off between interesting work, more responsibility, informal atmosphere, better commute, and compensation. When I was at TSG, the commute was pretty good, and I had a lot of responsibility (I was the entire IT staff -- sysadmin, programmer, and webmonkey), but the salary was lower than at FCI and the work was less challenging. Right now at The Archive, the work is really, really interesting, with a fair measure of responsibility, and a great amount of informality, but the commute is the worst I've ever had (75 to 90 minutes each way) and the compensation is barely better than TSG's. I've thought about leaving and making half again as much at a for-profit corporation, but it would mean giving up on too many interesting prospective projects which The Archive would pay me to pursue.
Sometimes, though, more interesting work can mean better pay further down the road. If you work in a specialty field which interests you more, you will learn more about it and accomplish more, which looks very good on a resume or performance evaluation. If this specialty field is more lucrative than what you had previously been doing, then you can net a higher salary in the long term.
For instance, I already knew how to make money developing compiler technology, but decided distributed processing would be more interesting, so I switched gears and started over as a newbie. It was a good decision, and has brought me more satisfaction, and potentially more salary (if I ever leave The Archive), than continuing my career in compiler technology could have delivered.
If you need a search tool, look around for a solution that someone else has already wasted years of their life on rather than have yourself do the same. Why recode, when you can download?
I have never in my professional life run into a nontrivial production business application which was perfect. They all have bugs. They all need more work. So it doesn't matter whether he downloads an open source system, or inherits his employer's legacy system -- he will need to learn the principles behind the technology in order to move that system forward.
I think that he is confusing the private keys used to prove authenticity of code (usually binary) or media for restrictive purposes (DRM) and keys that prove authenticity of code (source) to protect against modifications.
Hopefully Torvalds will clarify this soon enough, but I got the impression that he was referring to companies which use private keys to implement DRM as "developers" (which makes sense -- engineers at these companies develop Linux code, and if the company's suits couldn't deploy DRM with this code then the engineers might be told to stop developing Linux code and develop for some other platform (VxWorks, *BSD, OpenServer, WinCE, whatever)). I could be totally misreading him, but perhaps he thinks this source of development is good for Linux.
Or perhaps he's objecting on principle.. hopefully he will clarify soon.
I know exactly what you mean. I've worked in the directed marketing industry, using only a pittance of data compared to what Google will soon have at its disposal.
As an informed, educated computer professional, Google is getting to scare the bejezus out of me. If Google Analytics becomes even somewhat widespread, Google will be able to track the web usage of millions of people in considerable detail. The opportunity to correlate this data with other databases, and with gmail message contents, is tremendous and disturbing.
I don't seriously think that Google is a front for a government law enforcement agency, but I do seriously believe that Google is becoming the only webservice the FBI needs to subpoena in order to learn anything about anyone with an online presence.
There is still a lot we do not know about the human brain. It is not clear if synaptic firing is a binary event (either it's firing, or it isn't), or if it contributes a waveform from which the neuron determines further firings (and if the latter, with how much resolution). It is not clear whether the neuron is a simple threshold device, or a stateless map, or a stateful machine. It is not clear that we have identified all of the neurotransmitters. It is not clear how the brain determines that neurotransmitter levels need to change. It is not clear why relatively minor abnormalities in neurotransmitter levels correspond to severe forms of insanity in some people, but not in others. It is not clear if "long" axions are created at random, and then used opportunistically, or if they're created with predetermined roles.
The best neurologists in the world just plain don't know.
Furthermore, the architecture of the human brain is not a good fit with our best information technologies. It is an incredibly dense, yet high-latency, parallel computing device. Microprocessors and optical networks provide less-dense parallelism, but perform serialized tasks with orders of magnitude less latency.
It seems to me that we need a working (practical, useful) theory of intelligence, and then we need to use the strengths of our technology to bring theory to practice as best we can.
I host a mix of special interest sites, personal webpages, and a commercial site on my server. Mostly non-geek stuff. According to our webcounter, these are our users' resolutions and color depths for the last three months:
Slashdot Mirror
I am a software engineer working at The Internet Archive, and I write parallel software every day (sometimes with PVM for "real" applications, but more often as throwaway perl slammed out on the command line, using open3() to open several simultaneous subprocesses, sometimes fed data by the parent but more often each reading from a different data file). Much of what I do is "trivially parallelizable", meaning it's pretty easy to make scale across multiple processors or machines. It is my impression that most real-life problems seen by most businesses are trivially parallelizable, with the rare exceptions hogging all the attention by dint of being more interesting.
My workstation is a single-processor machine, but I have at my exclusive disposal a dual-xeon machine and two AMD dual-core machines. I'm always scp'ing my work up to them from my workstation so I can take advantage of their multi-process goodness. (Developing while ssh'd into those machines is usually not a good idea, since the network likes to go down or slow down a lot between Archive HQ and our datacenters, and our HQ firewall blocks PVM so I can't just make my workstation the PVM master node with the other three machines slaves.)
When I read this article, my initial reaction was "Enthusiasts, hell! I want as many of these as I can get for servers!" (assuming this 4x4 product is significantly cheaper than current dual-opteron products -- we're a non-profit, without a lot to spend on hardware, and we're always running on the edge of starvation. But maybe that's a bad assumption and these will be prohibitively pricey).
If someone offered me a 4x4 or 8x8 for my desktop, though, I'd accept it gladly, and make good use of it, parsing/analyzing Archive metadata, processing multiple simultaneous http streams (we use a lot of http-rpc here, and xml data representation which means each http-rpc stream can suck down a lot of processing power), md5'ing multiple files in parallel, and the like. I'd probably also make more extensive use of bzip2 than I do currently :-)
My datasets commonly consist of hundreds or thousands of files, each of which can be processed in parallel, so I can keep throwing cores at the problem with near-linear scalability until I grind against disk or bus bandwidth limits (at which point the data needs to start out distributed in order to keep scaling).
Just my $0.02
-- TTK
The Geode under consideration is the Geode NX 1500, which draws 7 watts at load. AFAIK we are only benchmarking it on how well it processes book scans (which is the majority of our processing load for now and the foreseeable future), and of course how easily it can fill gig-e.
Thanks for forwarding the comparison article :-) it made for an interesting read (the references to Cyrix gave me quite the nostalgic kick!).
-- TTK
The gig-e NIC in question was an Intel EtherPro1000, which is our standard card at The Archive (it's reliable, easy to fill to near-theoretical capacity without resorting to jumbo frames, and well-supported under linux). We measured its performance over "native rsync" (which is to say, without ssh encryption), without compression, which is how data is most commonly transferred between hosts at The Archive. (qv: "rsync ia300130.us.archive.org::items_0" to see what a data node looks like through rsync .. the directories with "d---------" permissions are "darkened" items, taken offline but not deleted.)
I don't have the exact numbers in front of me, but IIRC the C3 was able to fill it to about 35 megabytes/second.
-- TTK
Our current solution is based on the VIA EPIA-M micro-itx motherboard, which uses the VIA Apollo CLE266 chipset. It and four hard drives fit nicely into a moderate-depth 1U case, and it does PXE boot (which we (and Capricorn) use to mass-image new racks with their operating systems and filesystem layouts). Its topology is quite simple, but frankly we didn't need anything more at the time (plain-jane PCI, with embedded 100bT and ATA/133), and its power and heat characteristics are very good.
If you really, really don't need anything more than a simple dumb storage brick, and don't need full gig-E bandwidth, then this hardware does just fine. What we found is that, after deciding on this hardware, subsequent decisions about the handling of metadata on the storage nodes and some other things increased our need for storage nodes' processing capability. Also, increases in storage density (1TB per node in early 2004, vs 3TB per node today) renders 100bT insufficient for some of the operations we need to occasionally perform.
As for what might replace the VIA EPIA-M at The Archive, I cannot yet say. There's a lot of testing of prospective hardware needing to get done. I wouldn't want to suggest a chipset which later turned out to have some horrible flaw that only showed up during stress-testing.
-- TTK
It's been almost exactly two years since we put together the first petabox rack, and both the technology and our understanding of the problem have progressed since then. We've been working towards agreement on what the next generation of petabox hardware should look like, but in the meantime there are a few differing opinions making the rounds. All of them come from very competent professionals who have been immersed in the current system, so IMO all of them are worthy of serious consideration. Even though we'll only go with one of the options, a different option might be better suited to your specific needs.
One that is a no-brainer is: SATA. The smaller cables used by SATA are a big win. Their smaller size means (1) better air flow for cooling the disks, and (2) fewer cable-related hardware failures (fewer wires to break, and more flexibility). Very often when Nagios flags a disk as having gone bad, it's not the disk, but rather the cable that needs reseating or replacing.
Choosing the right CPU is important. The VIA C3's we started with are great for low power consumption and heat dissipation, but we underestimated the amount of processing power we needed in our "dumb storage bricks". The two most likely successors are the Geode and the (as-yet unreleased, but RSN) new low-power version of AMD's dual-core Athlon 3800+. But depending on your specific needs you might want to just stick with the C3's (which, incidentally, cannot keep gig-e filled, so if you wanted full gig-e bandwidth on each host, you'll want something beefier than the C3).
It has been pointed out that we could get better CPU/U, disks/U, $$/U, and power/U by going with either a 16-disks-in-3U or 24-disks-in-4U solution (both of which are available off-the-shelf), compared to 4-disks-in-1U (our current hardware). This would also make for fewer hosts to maintain, and to monitor with the ever-crabby Nagios, and require us to run fewer interconnects. Right now it looks like we'll probably stick with 4-in-1U, though, for various reasons which are pretty much peculiar to our situation.
I heartily recommend Capricorn's petabox hardware for anyone's low-cost storage cluster project, if for no other reason than because a lot of time, effort, brain-cycles, and experimentation was devoted to designing the case for optimizing air flow over the disks (and figuring out which parts of the disks are most important to keep cool). Keeping the disks happy will save you a lot of grief and money. When you're keeping enough disks spinning, having a certain number of them blow up is a statistical certainty. Cooler disks blow up less frequently. Most cases do a really bad job of assuring good air flow across the disks -- the emphasis is commonly on keeping the CPU and memory cool. But in a storage cluster it's almost never the CPU or memory that fails, it's the drives.
Even though the 750GB Seagates appear to provide less bang-for-buck than smaller solutions (400GB, 300GB), the higher data storage density pays off in a big way. Cramming more data into a single box means amortizing the power/heat cost of the non-disk components better, and also allows you better utilization of your floorspace (which is going to become very important, if you really are looking to scale this into the multi-petabyte range).
When dealing with sufficiently large datasets, silent corruption of data during a transfer becomes inevitable, even using transport protocols which attempt to detect and correct such corruptions (since the corruption could have occurred "further upstream" in the hardware than the protocol is capable of seeing). We have found it necessary to keep a record of the MD5 checksums of the contents of all our data, and add a "doublecheck" step to transfers: perform the transfer (usually via rsync), make sure the data is no longer being cached in memory (usually by performing additional transfers), and then recompute the MD5 checksums on the destination host and c
Perhaps you're right -- I have to admit that the only experiences I have had with XML have been with C and perl DOM parsers, and PHP5 (which handles XML as first-class data entities, and for all I know may be using the C DOM XML library underneath). I have not worked with a streaming XML parser -- though, are you sure these parsers are as flexible as you've made them out? The XML specification, section 5.1 seems to imply that a parser that parses a later section of an XML document without parsing some earlier section is out-of-spec (and for good reason -- earlier data can change the meaning of later data, and there's no table of contents in an XML document which gives the lengths of independent sections (which would mostly solve the issue)).
Sure there are other ways to represent data -- but not everyone will understand them, and there won't always be easy to understand rules to expand them [..]
This is perhaps the strongest argument in favor of using XML. Everyone's already using it, and expanding a dataset is straightforward. All I can do is wish that there was a sane, well-crafted, easily-parsed, fault-tolerant binary specification which enjoyed the same ubiquity. One of the points I was trying to make in my original post, though, is that if your situation was already well-suited to using XML, then you were in a position to create and use a nonstandard specification which lacks XML's intrinsic faults.
-- TTK
As Jesapoo points out, it's not about size, but as important as orbit eccentricity is material composition. Planets are historically categorized into two buckets based on their composition -- "terrestrial", which are mostly rock (mercury, venus, earth, mars), and "jovian", which are mostly gas (jupiter, saturn, uranus, and neptune). And then there are comets, which are mostly dirty ice and frozen gas with some rocks.
Pluto is cometary in composition, which has led some to classify it as a comet rather than as a planet. Frankly I can see the argument. Perhaps the best way out is to define "planet" such that some comets can be planets?
-- TTK
XML isn't a bad idea, for instance-- it gives a standard method of defining data transport, for instance.
I've been wondering whether XML is really all that great of an idea. It makes sense to use it when, as you say, you need a standard way of representing data across multiple dissimilar systems. But a key notion behind XML is that unless an XML dataset is well-formed, attempts to parse that XML should fail.
This means XML makes sense to use when you need to represent data across multiple dissimilar systems, and you have control over the formation of datasets. Otherwise, if one system generates imperfectly formed XML, the whole system of systems grinds to a halt. Therefore, you either need sufficient control over all sources of data to be able to fix the way the data representation is generated, or you need to only use bug-free software.
If you have sufficient control over the way data is represented to be able to fix it when it misgenerates the XML, then you don't really need XML, and can instead choose a data representation more appropriate for your needs -- something that doesn't bloat your data out to 5x its original size, and doesn't require you to parse N records before you can parse out the N+1'th record, and doesn't require you to throw out an entire dataset if there is a problem with some part of it (which is like refusing to extract any files from a tarball if the last record in the tarball is truncated (if this hasn't happened to you yet, just wait, it will! and then you'll be glad that tar will extract all the files it can)).
If your system only uses bug-free software, and is sufficiently complex to do something useful, .. I don't know, I'll buy you a drink or something. Congradulations, you're ahead of the rest of us.
That having been said, there certainly seem to be a lot of people out there who are perfectly happy using XML. Maybe my experiences with it have just been unusually bad, or maybe those people don't mind XML's drawbacks. It's been my experience that representation errors are common (and sometimes what an XML parser considers a representation error is actually a desirable feature), and that software is more useful when it proceeds despite adverse conditions, when it can. But my mind is not closed on the subject. There may be something I'm missing, and I don't want to miss it if life throws it at me.
As for web 2.0 as a whole, I see a more complex picture. Yes, it's been unduly hyped, but it's also putting a label on a body of concepts with which the industry is trying to come to terms. There's a vague notion that dynamic web services which share information across contexts can be good, but the why and when of it is still unclear. I do not fault those who try to make more sense of it. Fault lies with those who focus unduly on the tools people have used thusfar to create useful services (Javascript, XML, PHP, Python, etc), to the neglect of the reasons those services have been useful (which are partly technological, but mostly social). I suspect the missing piece is something very simple, like "develop services which satisfy an existing need", but time will tell.
-- TTK
At The Internet Archive we have about 120,000 audio and live music shows, occupying about 53TB of disk space. We're always trying to think of new and better ways to present it to our users.
I'm going to look at all the solutions people have suggested here and try to glean some usability tips which might be implementable on top of our existing interface. Please keep up the good suggestions!
-- TTK
My usual criterion for the quality of a video card is: "how well does XFree86 support it?" (or I guess XOrg, now). A $50 or $30 card which works well for making xterms and Netscape appear on the screen is exactly what I want (and need).
An advantage to being happy with inexpensive cards is that it becomes feasible to purchase a few of them, so that you can standardize sets of machines on them. That goes double for network cards. It's handy to be able to swap harddrives between machines with impunity, and have video + network "just work" without needing to fiddle with modprobe (or with XF86Config/xorg.conf).
My old crop of machines was standardized on ne2k-pci compatible cards, but I'm transitioning to eepro1000 :-) It's a wonderful world, where gig-e cards can be had for only $50 (or $25, if I wasn't stuck on the high quality etherpros), and 8-port gig-e switches for only $100. My standard video card is still the ATI Xpert98, though. Maybe it's time to restandardize there too.
-- TTK
I read the article you referenced, and I read the proposed amendment itself, and I do not see where it requires, or even mentions, logging the connections ISP customers make. It appears to only require recording who the ISP's customers are, and retaining information on a customer for up to a year after that customer stops subscribing to the service.
Not that I like the amendment, nor do I trust our government to refrain from abusing their access to this information, but it seems to be a much less big of a deal than everyone is assuming.
The text of the proposed amendment (it's short!):
edited because Slashdot originally refused to accept, saying: "Lameness filter encountered. Post aborted! Reason: Please use fewer 'junk' characters .. now what does that say about the state of legislation? :-)
That is all!
-- TTK
not to nitpick either, but those Russkie defence systems are useless against most US weapon systems that rely on the kinetic penetrator concept (though they MIGHT snap the rod of a TOW missile, the latter being the cheapest weapon of choice here).
If you're going to nitpick, you might want to know what you're talking about, first :-)
The earliest generations of Eexplosive Reactive Armor (ERA) were not effective against Armor Piercing, Fin Stabilized, Discarding Sabot weapons (APFSDS, KE weapons, aka long-rod penetrators), but later generations of Soviet ERA were (and are) quite effective against KE weapons. Kontakt-5 "Heavy ERA", for instance, utilizes a hinged flyplate which snaps off the front the penetrator and induces significant yaw (tumble) in the remainder. This loss of length reduces the KE penetrator's ability to defeat the passive armor underneath the ERA (hypervelocity long-rod penetration is proportional to rod length, see Anderson's and Odermatt's publications. It is common practice in penetration studies to normalize penetration figures to P/L (the ratio of penetration depth to penetrator rod length)). Kontakt-5 was first fielded in 1985, so ERA that protects against KE weapons has been with us for a while.
The most modern APFSDS use a segmented rod which minimizes the impact of this kind of ERA, but ERA has advanced in the meantime too. The very latest design, the ukranian Nozh ERA uses linear shaped charges to cut the penetrator, potentially in multiple locations down its length, and also induces yaw.
Also, the TOW missile uses a shaped charge to achieve penetration, and is not a KE penetrator. It has no "rod" (unless you're talking about the spur for the detonation probe, which sticks way out in front so that the shaped charge can detonate an optimal distance from the target). TOW missiles are vulnerable to Drozd, ARENA, and all of the various flavors of ERA used today, qv:
http://www.globalsecurity.org/military/systems/mun itions/tow.htm
-- TTK
Just a little nit to pick: Drozd has been deployed on T-55 and T-80 family tanks, but T-90 uses the newer ARENA system. Also, using ARENA precludes mounting Explosive Reactive Armor modules, the latest versions of which are useful against APFSDS threats (which Drozd and ARENA are not), so it's not exactly a silver bullet.
ObPlug: more on various kinds of active defense systems can be found on this page.
-- TTK
It's more like 0.15%, if they use the same compression and content selection criteria as the Internet Archive. If they eschewed with all non-html content (graphics files, pdf's, etc) that would go up quite a bit. If they used better compression (the Archive uses gzip) it would go up some more.
An average crawl of the public web, minus files which are "too large" (not sure what the threshold is for that), makes about 55TB of gzipped archive. 80GB / 55000GB = 0.0014545, or about 0.15%.
-- TTK
Google on "tank armor" and my site pops right up. I spend a lot of time researching tank technology, especially composite armors, both online and in print. Much of my effort is spent on Russian, Ukranian, and German websites, patent filings, and company brochures. The International Journal of Impact Engineering is great when I can get it, but there's a wealth of information to be mined from analyzing the latest German innovations, or the flood of ex-Soviet designs previously hidden behind the iron curtain, now revealed to the west for the very first time.
German defense engineers are masters of the composite armor arrangement. Being a western country, they have access to highly precise fabrication facilities and the most advanced technical materials money can buy, and they have the money to buy it. Seeing what they buy, how they shape it, and how much they put where is invaluable to my interests.
Russia and Ukraine, on the other hand, faced the prospect of fighting NATO on a shoestring budget. Their materials were modest, and they couldn't rely on their manufacturing facilities to consistently produce high-precision machining of components, but their engineers were (and continue to be) top-notch. They accepted the limitations of the materials and tools they had to work with, and found ways to arrange relatively humble metals, ceramics, and plastics such that they synergistically produced very advanced composite armor effects. They designed their armor systems such that they would continue to provide high levels of protection even if the factory mass-producing them couldn't weld a straight line to save their lives. While engineers in the west pushed the limits of what factories were capable of producing, often with disasterous results (qv the failed joint German/American MBT-70 project) their eastern counterparts made damn sure their own designs could be churned out quickly, cheaply, and in vast numbers (exceptions like the T-64/T-80 family notwithstanding). This holds special appeal to me because these are armor effects that can be replicated in a humble home workshop, using inexpensive commodity plastics and ceramics. If I don't measure it perfectly well, or the edges don't line up quite right, hey! It still works! That is more valuable to me than some super-advanced design that would take $1000/lb and a bunch of unobtainable federal licenses before I could even consider putting tab A into slot B.
Besides lots of oil, orphans, and bodybags, one of the things Americans found in our latest tragic military misadventure was a bunch of tanks which had been upgraded with armor modules purchased from Russia and welded into place in Iraq. These modules had been encountered in the 1991 war, but for some reason detailed analysis of these modules' composition didn't see the light of public attention until now. These "Enigma" armor modules repeatedly resisted the shaped charge warheads of Dragon and Milan anti-tank missiles and the depleted uranium projectiles issuing from 30mm Bushmaster autocannons, though they did not fare so well against Abrams' 120mm main guns. Still, that's not bad at all for a relatively lightweight steel box welded onto the turret of a forty year old tank (the T-55). Enigma modules were pried loose and cut open, and do you know what was found inside? Simple layers of steel, natural rubber, and construction-grade aluminum, with airgaps between each three-layer array. Some boxes were missing a layer or two here and there, and the adhesion between layers was sloppy at best. This was an example of the soviet bulging armor, doing what it does best, on the cheap. The effect this combination produces belongs in the realm of rocket science, but putting the boxes themselves together hardly takes a highschool diploma.
American armor engineering s
I sit corrected .. not flash animations. That's what I get for posting before actually downloading. I'll throw these samples at the deriver and see if it breaks any teeth.
-- TTK
Will/can archive.org mirror the videos?
I intend to at least take a look at it. They'd make a nice addition to our movies collection (currently hosting about 30,000 moving image items). Our deriver software (which generates those Mpeg4's et al) doesn't currently grok flash, but that might change.
-- TTK, Archive Engineer
I used xfig.
-- TTK
I would use this to connect 10Ge switches together, not to connect directly to individual servers in the network. Three 32-port 100Ge switches, linked to 32 32-port 10Ge switches by three links (ie, so each 10Ge switch had three links to different 100Ge switches and one link to each of 29 hosts) would allow any two hosts in a 928-node cluster to communicate at full 10Ge capacity (or, more likely, communicate with any number of hosts in that cluster at 10Ge aggregate capacity).
Something like this: 928 node network
-- TTK
Me too .. and right now, I am.
I have traditionally been willing to trade off between interesting work, more responsibility, informal atmosphere, better commute, and compensation. When I was at TSG, the commute was pretty good, and I had a lot of responsibility (I was the entire IT staff -- sysadmin, programmer, and webmonkey), but the salary was lower than at FCI and the work was less challenging. Right now at The Archive, the work is really, really interesting, with a fair measure of responsibility, and a great amount of informality, but the commute is the worst I've ever had (75 to 90 minutes each way) and the compensation is barely better than TSG's. I've thought about leaving and making half again as much at a for-profit corporation, but it would mean giving up on too many interesting prospective projects which The Archive would pay me to pursue.
Sometimes, though, more interesting work can mean better pay further down the road. If you work in a specialty field which interests you more, you will learn more about it and accomplish more, which looks very good on a resume or performance evaluation. If this specialty field is more lucrative than what you had previously been doing, then you can net a higher salary in the long term.
For instance, I already knew how to make money developing compiler technology, but decided distributed processing would be more interesting, so I switched gears and started over as a newbie. It was a good decision, and has brought me more satisfaction, and potentially more salary (if I ever leave The Archive), than continuing my career in compiler technology could have delivered.
-- TTK
If you need a search tool, look around for a solution that someone else has already wasted years of their life on rather than have yourself do the same. Why recode, when you can download?
I have never in my professional life run into a nontrivial production business application which was perfect. They all have bugs. They all need more work. So it doesn't matter whether he downloads an open source system, or inherits his employer's legacy system -- he will need to learn the principles behind the technology in order to move that system forward.
-- TTK
I think that he is confusing the private keys used to prove authenticity of code (usually binary) or media for restrictive purposes (DRM) and keys that prove authenticity of code (source) to protect against modifications.
Hopefully Torvalds will clarify this soon enough, but I got the impression that he was referring to companies which use private keys to implement DRM as "developers" (which makes sense -- engineers at these companies develop Linux code, and if the company's suits couldn't deploy DRM with this code then the engineers might be told to stop developing Linux code and develop for some other platform (VxWorks, *BSD, OpenServer, WinCE, whatever)). I could be totally misreading him, but perhaps he thinks this source of development is good for Linux.
Or perhaps he's objecting on principle .. hopefully he will clarify soon.
-- TTK
I know exactly what you mean. I've worked in the directed marketing industry, using only a pittance of data compared to what Google will soon have at its disposal.
As an informed, educated computer professional, Google is getting to scare the bejezus out of me. If Google Analytics becomes even somewhat widespread, Google will be able to track the web usage of millions of people in considerable detail. The opportunity to correlate this data with other databases, and with gmail message contents, is tremendous and disturbing.
I don't seriously think that Google is a front for a government law enforcement agency, but I do seriously believe that Google is becoming the only webservice the FBI needs to subpoena in order to learn anything about anyone with an online presence.
-- TTK
There is still a lot we do not know about the human brain. It is not clear if synaptic firing is a binary event (either it's firing, or it isn't), or if it contributes a waveform from which the neuron determines further firings (and if the latter, with how much resolution). It is not clear whether the neuron is a simple threshold device, or a stateless map, or a stateful machine. It is not clear that we have identified all of the neurotransmitters. It is not clear how the brain determines that neurotransmitter levels need to change. It is not clear why relatively minor abnormalities in neurotransmitter levels correspond to severe forms of insanity in some people, but not in others. It is not clear if "long" axions are created at random, and then used opportunistically, or if they're created with predetermined roles.
The best neurologists in the world just plain don't know.
Furthermore, the architecture of the human brain is not a good fit with our best information technologies. It is an incredibly dense, yet high-latency, parallel computing device. Microprocessors and optical networks provide less-dense parallelism, but perform serialized tasks with orders of magnitude less latency.
It seems to me that we need a working (practical, useful) theory of intelligence, and then we need to use the strengths of our technology to bring theory to practice as best we can.
-- TTK
I host a mix of special interest sites, personal webpages, and a commercial site on my server. Mostly non-geek stuff. According to our webcounter, these are our users' resolutions and color depths for the last three months:
39.3% 1024x768 @ 32bit
11.9% 1280x1024 @ 32bit
10.6% 800x600 @ 32bit
9.7% 1024x768 @ 16bit
6.3% unknown (javascript disabled)
3.6% 800x600 @ 16bit
3.5% 1152x864 @ 32bit
3.4% 1280x800 @ 32bit
1.6% 800x600 @ 24bit
1.5% 1600x1200 @ 32bit
1.5% 1024x768 @ 24bit
1.3% 1280x1024 @ 16bit
0.9% 1440x900 @ 32bit
4.9% misc other (not going to bother transcribing)
-- TTK