I use plenty of stuff for which I have the source code. Going back to the 4.2mumble BSDs, through SunOS, Linux, Solaris, the various x86 BSDs, and plenty of applications (this is Mozilla I'm/.ing with, and before that a long line of other open source browsers). I have no problem with installing large Linux farms, using Apache for an enterprise web deployment, using MySQL for moderate sized databases (or PostgreSQL, though I haven't deployed it personally).
Tape backup... NBU wins. Legato's a close second. Sorry, charlie. Open source as a category does not suck. The open source backup stuff doesn't suck, for small to medium sized sites. It's not enterprise class, though, and most of the trick to succeeding in IT is knowing when the tools you use aren't applicable anymore and how to figure out what are.
NBU can't RAIT, but it can stream across multiple tapes, and can write duplicate tapes if you want redundancy. And you can extract the files off tape with tar if you have to.
I've seen attempts to build large enterprise backup environments with "simple open" software. They melt down somewhat short of the size that the original questioner is asking about, typically.
I've built environments with NBU and used Legato, at large sites (much larger than the original questioner). They just work. Configuring them initially can be non-trivial if you have no prior experience with them, but once set up right they just work.
Throwing a bunch of open source tech at the wall and seeing if it sticks will kill you here. I've been at places which were big enough to use 40, 60, 180, multi-hundred, 5000 tape changers. They use professional grade stuff. It works. If you can't make it work, don't go to work for large sites. Don't wire your whole five hundred cube building up with daisy-chained 8 port 10/100 switches, and don't use toy backup equipment if you're an enterprise class environment. Data backups aren't a tech game: they're how you survive the statistically likely disk outages and statistically unlikely building fires/floods/earthquakes/tornados/etc. This is important, and half-assed solutions shouldn't apply.
For that many systems, use a professional, enterprise grade, commercial solution. The open source stuff doesn't supply the same manageability.
AND FOR GOD'S SAKE, REGULARLY VERIFY THAT YOU CAN READ THE TAPES BACK... More sites have been screwed by backup tapes that weren't readable than any other failure mode. Verifying every tape is best. Second best is every weekly. Random samples, but covering every single drive's tape output at least once a month, are poor third place.
The two obvious software suggestions are Veritas/Symantec NetBackup and Legato Networker.
Weekly fulls and daily incrementals are good. Your offsite schedule should be checked to ensure that you have a relatively recent restore point both onsite (in case of data loss) and offsite (in case of building loss).
In terms of offsites, having a prepared plan for where and how to restore (Disaster Recovery and Business Continuity) is also important. But those all start with "Go get the tapes...".
Is this internet access for desktop users? People from outside coming in to your corporate website? VPN connections to other offices? How many users? Are you attempting to syncronize any data across the link? In real time, or overnight?
The possible set of right answers depends a lot on what you're doing with it.
Policy based routing plus any number of DSL lines will work for splitting up desktop web access.
Inbound traffic for the corporate website is pretty much the antithesis of that... outbound traffic is the target, and that ends up being T-1 optimized for small sites and bonded T-1s or faster links for bigger ones.
VPNs can be symmetrical or asymmetrical. Your mileage may vary.
What started Silicon Valley was that it had critical mass, of everything that modern tech companies needed to grow out of.
The article lists a lot of that, but misses some other things. Pre-existing tech and engineering companies... before it was Silicon Valley, HP and Varian already started here, IBM was a major major force in the area (one of IBM's bigger research centers), GE was here in force. Lockheed is here, doing unmentionable space stuff, and Space Systems Loral's predecessors.
These were all more traditional tech companies, but the untraditional tech companies were in a sense a spinoff from the density of skills and suppliers and environment that the larger tech companies had been growing in for decades previously.
Lame. Try looking out of areas which were orchards 30 years ago. Half the south bay was agricultural back then (237 used to be a surface street, through fields mostly... and was the least preferred route for decades, until it finally got the freeway treatment). The route that 85 takes that now connects to 101 on the south side of San Jose... used to be through mostly open fields, not that long ago.
All (nearly) those former agricultural regions are now high density housing or business parks. But the rest of the valley, where houses haven't been torn down and rebuilt with property-line hugging multistory monsters, has older homes with character.
Los Altos, where I grew up, is succumbing to the property line multistory monsters slowly. But Mountain View and Sunnyvale and Cupertino and Santa Clara mostly aren't.
See, that's the problem, and it's the programmers that caused it. Stop making subroutines. Monolithic code be da shizzle on da bizzle.
You no doubt think that's funny, but the flight control systems on the JAS-39 Grippen jet fighter from Sweden and John Carmack's flight control computers switched over to that type of programming logic. One large loop, no functions, you only go forwards towards the end and then start again at the beginning.
UC Santa Cruz has had a tech writing for computer scientists program, with both an undergrad and graduate level class, for years now. See for example http://www.soe.ucsc.edu/classes/cmpe185/.
There's a pretty senior professor behind it, though they use lecturers for most of the sections. They probably would love to help out other schools...
There were enough of WAFL's methods and algorithms published in 1996 to reimpliment to those specifications in a cleanroom. WAFL really isn't all that complicated.
That's no more of a copyright violation than Linus coding Linux to the SysV Interface Definition.
Not really, you still need big honking multi-processor machines to run big honking databases. A quad-proc dual-core opteron still isn't there yet in being able to match a fully loaded E25K for chewing on a big database.
More importantly, a rack full of quad-proc dual-core opterons still isn't there yet in being able to match a fully loaded E25K for chewing on a big database. Ten racks full of them, and a room full of them, either, unless you can partition the database efficiently.
It is still far easier to do Oracle RAC wrong, and end up with a flat performance curve as you add nodes past 8 or so, than to do it right. It's possible to do RAC for some databases right and get reasonably, monotonically increasing performance out to many many nodes, but it's not common yet, or practical if you look at it statistically in terms of how many projects end up having to back it out and go back to large monolithic SMP servers.
Some databases are partitionable and easily splittable among systems without clustering them. Those, it's already cost effecitve to move to large stacks of small servers. But those aren't the typical data models for large commercial databases.
From commercial experience: Multiple MTA machines fine. Multiple MUA (IMAP, POP, Webmail) machines fine. Don't use a clustered filesystem; use NFS. All the software of note use locking which plays well with NFS.
Scaling can be done easily by adding more NFS boxes and managing the directory structure with links or whatnot.
...he rolls out to volunteer fire department calls in it all the time...
Unless you're hauling a bunch of stuff,...
I would guess that you aren't familiar with that volounteer fire department people have to carry sometimes, and the sort of truck it can sometimes take to get in and out of places with several thousand feet of vertical gradient between towns.
According to statistics I've seen, that's about 2% of SUV owners.
20 years ago, you had things like the Suburban, the various Jeeps (Cherokee, Wagoneer, Grand Wagoneer etc), the Blazer, etc. You saw a few normal people driving them, but by and large they were a slightly nicer more enclosed pickup truck for people who had legit offroad or hauling or 4x4 needs. Most of those people drove pickup trucks, though. Some with camper shells, but pickups not protoSUVs.
SUVs now have taken over the role of the family van or station wagon. And that, that's a shame, because vans have more space and wagons are a lot safer to drive and get a lot better mileage.
I didn't mind it when friends had a Jeep Grand Wagoneer that was used to haul a horse trailer every weekend, or the other GW we were using to support amateur filmmaking up in the hills (and once, slid backwards down a 30 degree slope in the mud and rain, bumping the uphill cliff the whole way down...). I didn't mind it when mom got an Explorer for Search and Rescue work. I don't mind it that my best friend has a big honking pickup truck; he rolls out to volunteer fire department calls in it all the time, and sometimes that's in the rain or snow or off road.
I mind the thousand other SUVs I drive past on the way to work, which do roughly none of those things to justify being build that way.
If you know what the remote IP addresses are going to be (consumer grade but fixed IP addresses at remote ends) then ssh would be an adequate solution by itself, and a lot simpler than most of the alternatives. With its ability to forward ports and X windows displays, it can handle pretty much anything.
If you need constant monitoring and interaction a real VPN may make more sense, but... think carefully about how much complexity you add in the management layer here. Does that overall improve or degrade the total environment's reliability and managability?
I think it's clear that they're not addressing the same market space. But my point was that this release by Sun is qualatatively different than prior embedded CPU releases under open source rules. Fewer people may end up synthesizing and using T1's than will use either FPGA or synthesized embedded CPU designs, but the availability of T1's design does open up interesting new possibilities.
For which, I agree, the "market demand" is unclear. At the very least I know that a lot of researchers are looking at it closely for ideas and comparisons. And given the licensing, someone who had a high-thread-count embedded application might find a use for. It's not entirely clear to me that the IP is in the noise... it's been a decade, but I was working around chip IP issues in the mid-90s, and a large chunk of the budget was the IP, and another was verifying the level of stuff which is already verified with the T1 design. You still need to synthesize and tape it out, but that's a large chunk of effort taken out.
I agree that the total number of shipped products from the T1 design release could end up being zero. But I wouldn't bet any serious money on that.
I'm a bit out of it on the latest design requirements for CPUs - is the technology of these folks actually good enough to make a reasonably modern CPU?
Yes. I believe that the best MOSIS process is the
IBM 90 nm process, which is 7 metal layer, pretty flexible. The T-1 SPARC we're talking about (Niagra) is a 90 nm, 9 metal layer Copper wire fab design (see Sun's Specs). You can't quite fab a T-1 as Sun laid it out with IBM's process, but it's pretty close. You could produce a roughly the same size, slightly larger and/or slower version of the same chip with a new detail layout, using the same chip level "circuit diagram" but a different physical design with fewer layers of metal used etc.
AMD uses 0.13 and 0.09 u (90 nm) processes for their
current Opteron line,
though theirs are Silicon on Insulator fab processes. Again, different design details, but the same general scale and capabilities.
The newest Intel Zeon MP processors are at 65 nm processes, one step past the IBM 90 nm process (components on the average taking roughly half the surface area per step). But Intel still produces a lot of slightly older 90 nm and larger CPUs, and industry consensus is that the 90 nm AMD and 65 nm Intel chips are still roughly at equal performance.
A few thousand bucks and a working chip design will get you parts these days, in suprisingly modern fab processes (a few tens of thousand for 0.13u and 90nm).
Sun does not produce chips. Their business is to define a standard and have others implement the standard, which Sun then uses in their systems.
No, Sun has always designed the majority of the CPUs that they use in their systems. They are fabless; they don't own the chip fabrication factories... but they have buildings full of people working on chip design.
Others have designed SPARC CPU chips (Fujitsu / HAL; Ross) which Sun used, as well. But Sun engineers designed UltraSPARC-I, II, IIi, IIe, III, IIIi, IV, T-1, and going backwards the SuperSPARC, MicroSPARC-I and II, most of the Sun-4 and Sun-4c generation stuff.
The difference here is that we're seeing current generation high end processors start to appear here... there's a huge difference between low end, embedded CPU class cores being available and a Niagra T-1 SPARC. T-1 isn't the fastest single thread CPU out there, but it may well be the fastest total multithreaded throughput CPU out there on the market today. Whether that's appropriate for most users / workloads or not, it is clearly a huge difference compared to embedded CPUs.
MMORPGs weren't a significant market player until the late 1990s, when the Internet became ubiquitous and fast enough for most people to actually play high content games.
MTG's effects were clear and openly known and discussed in the game industry. Game store owners loved it: Sales went up 50% or more. Except that what really happened was that game store sales of Magic and competitors went up to about 75-100% of prior sales levels, and the sales of miniatures and paper/dice roleplaying games dropped 25% to 50%.
SJ Games was still recovering from the Secret Service raid when MTG came out. If they hadn't reacted by announcing and then selling INWO they'd have gone under in 1995 or 96 along with GDW. Steve Jackson has admitted that INWO carried the rest of their line for years; they'd have been much more profitable if they'd abandoned most of the rest, but they didn't want to do that.
I grossly disagree with the comment that it was collapsing in on itself; there were a number of firms and a small, and not particularly high growth market. Nobody was truly thriving, but SJG, GDW, TSR, White Wolf were all hanging out and producing stuff and paying their people's salaries. Not all these companies were profitable, but most were, and there was no serious threat of an industry implosion.
MTG arrives and bang, there goes the neighborhood. Wave of companies folding.
I wasn't a major player by any means, but I was there and involved. I'd been selling stuff to GDW every few months for Challenge magazine and looking at getting more involved with projects for them and SJG; a friend had just done a book for SJG. When GDW went down, they owed me some (not serious) money. I'd been talking to them, and got the authors-level financial explanation associated with the corporate shutdown. I also saw the equivalents out of SJG explaining how the Illuminati CCG saved them when the rest of the market tanked.
It's true that the paper/dice RPG market wasn't thriving and growing seriously; it's also true that it wasn't imploding prior to MTG, and was afterwards.
Slashdot Mirror
I use plenty of stuff for which I have the source code. Going back to the 4.2mumble BSDs, through SunOS, Linux, Solaris, the various x86 BSDs, and plenty of applications (this is Mozilla I'm /.ing with, and before that a long line of other open source browsers). I have no problem with installing large Linux farms, using Apache for an enterprise web deployment, using MySQL for moderate sized databases (or PostgreSQL, though I haven't deployed it personally).
Tape backup... NBU wins. Legato's a close second. Sorry, charlie. Open source as a category does not suck. The open source backup stuff doesn't suck, for small to medium sized sites. It's not enterprise class, though, and most of the trick to succeeding in IT is knowing when the tools you use aren't applicable anymore and how to figure out what are.
NBU can't RAIT, but it can stream across multiple tapes, and can write duplicate tapes if you want redundancy. And you can extract the files off tape with tar if you have to.
Amanda certainly doesn't suck, but it's not NBU.
BMR has been standard for years.
I've seen attempts to build large enterprise backup environments with "simple open" software. They melt down somewhat short of the size that the original questioner is asking about, typically.
I've built environments with NBU and used Legato, at large sites (much larger than the original questioner). They just work. Configuring them initially can be non-trivial if you have no prior experience with them, but once set up right they just work.
Throwing a bunch of open source tech at the wall and seeing if it sticks will kill you here. I've been at places which were big enough to use 40, 60, 180, multi-hundred, 5000 tape changers. They use professional grade stuff. It works. If you can't make it work, don't go to work for large sites. Don't wire your whole five hundred cube building up with daisy-chained 8 port 10/100 switches, and don't use toy backup equipment if you're an enterprise class environment. Data backups aren't a tech game: they're how you survive the statistically likely disk outages and statistically unlikely building fires/floods/earthquakes/tornados/etc. This is important, and half-assed solutions shouldn't apply.
What are your tape drives doing from 9am until 6pm?
That's not enough time for a full check, but that's enough time for checking half the tapes...
For that many systems, use a professional, enterprise grade, commercial solution. The open source stuff doesn't supply the same manageability.
AND FOR GOD'S SAKE, REGULARLY VERIFY THAT YOU CAN READ THE TAPES BACK... More sites have been screwed by backup tapes that weren't readable than any other failure mode. Verifying every tape is best. Second best is every weekly. Random samples, but covering every single drive's tape output at least once a month, are poor third place.
The two obvious software suggestions are Veritas/Symantec NetBackup and Legato Networker.
Weekly fulls and daily incrementals are good. Your offsite schedule should be checked to ensure that you have a relatively recent restore point both onsite (in case of data loss) and offsite (in case of building loss).
In terms of offsites, having a prepared plan for where and how to restore (Disaster Recovery and Business Continuity) is also important. But those all start with "Go get the tapes...".
Is this internet access for desktop users? People from outside coming in to your corporate website? VPN connections to other offices? How many users? Are you attempting to syncronize any data across the link? In real time, or overnight?
The possible set of right answers depends a lot on what you're doing with it.
Policy based routing plus any number of DSL lines will work for splitting up desktop web access.
Inbound traffic for the corporate website is pretty much the antithesis of that... outbound traffic is the target, and that ends up being T-1 optimized for small sites and bonded T-1s or faster links for bigger ones.
VPNs can be symmetrical or asymmetrical. Your mileage may vary.
What started Silicon Valley was that it had critical mass, of everything that modern tech companies needed to grow out of.
The article lists a lot of that, but misses some other things. Pre-existing tech and engineering companies... before it was Silicon Valley, HP and Varian already started here, IBM was a major major force in the area (one of IBM's bigger research centers), GE was here in force. Lockheed is here, doing unmentionable space stuff, and Space Systems Loral's predecessors.
These were all more traditional tech companies, but the untraditional tech companies were in a sense a spinoff from the density of skills and suppliers and environment that the larger tech companies had been growing in for decades previously.
All (nearly) those former agricultural regions are now high density housing or business parks. But the rest of the valley, where houses haven't been torn down and rebuilt with property-line hugging multistory monsters, has older homes with character.
Los Altos, where I grew up, is succumbing to the property line multistory monsters slowly. But Mountain View and Sunnyvale and Cupertino and Santa Clara mostly aren't.
UC Santa Cruz has had a tech writing for computer scientists program, with both an undergrad and graduate level class, for years now. See for example http://www.soe.ucsc.edu/classes/cmpe185/. There's a pretty senior professor behind it, though they use lecturers for most of the sections. They probably would love to help out other schools...
There were enough of WAFL's methods and algorithms published in 1996 to reimpliment to those specifications in a cleanroom. WAFL really isn't all that complicated.
That's no more of a copyright violation than Linus coding Linux to the SysV Interface Definition.
Solaris, or in extremis NetApp or the other dedicated NAS boxes, do just fine, up to pretty darn big sized installations.
It is still far easier to do Oracle RAC wrong, and end up with a flat performance curve as you add nodes past 8 or so, than to do it right. It's possible to do RAC for some databases right and get reasonably, monotonically increasing performance out to many many nodes, but it's not common yet, or practical if you look at it statistically in terms of how many projects end up having to back it out and go back to large monolithic SMP servers.
Some databases are partitionable and easily splittable among systems without clustering them. Those, it's already cost effecitve to move to large stacks of small servers. But those aren't the typical data models for large commercial databases.
From commercial experience: Multiple MTA machines fine. Multiple MUA (IMAP, POP, Webmail) machines fine. Don't use a clustered filesystem; use NFS. All the software of note use locking which plays well with NFS.
Scaling can be done easily by adding more NFS boxes and managing the directory structure with links or whatnot.
I've run oracle on 32 processor Sun E10Ks with reasonably linear speedup from few-processor performance, back in the Solaris 7 days.
I've run (now obsolete) ATG Dynamo on the same, with similar results.
I've run Apache (1.3.x) on the same, with similar results.
I've seen applications which stopped scaling well at much less than that.
"Large business applications" isn't specific enough.
SUVs now have taken over the role of the family van or station wagon. And that, that's a shame, because vans have more space and wagons are a lot safer to drive and get a lot better mileage.
I didn't mind it when friends had a Jeep Grand Wagoneer that was used to haul a horse trailer every weekend, or the other GW we were using to support amateur filmmaking up in the hills (and once, slid backwards down a 30 degree slope in the mud and rain, bumping the uphill cliff the whole way down...). I didn't mind it when mom got an Explorer for Search and Rescue work. I don't mind it that my best friend has a big honking pickup truck; he rolls out to volunteer fire department calls in it all the time, and sometimes that's in the rain or snow or off road.
I mind the thousand other SUVs I drive past on the way to work, which do roughly none of those things to justify being build that way.
If you know what the remote IP addresses are going to be (consumer grade but fixed IP addresses at remote ends) then ssh would be an adequate solution by itself, and a lot simpler than most of the alternatives. With its ability to forward ports and X windows displays, it can handle pretty much anything.
... think carefully about how much complexity you add in the management layer here. Does that overall improve or degrade the total environment's reliability and managability?
If you need constant monitoring and interaction a real VPN may make more sense, but
I think it's clear that they're not addressing the same market space. But my point was that this release by Sun is qualatatively different than prior embedded CPU releases under open source rules. Fewer people may end up synthesizing and using T1's than will use either FPGA or synthesized embedded CPU designs, but the availability of T1's design does open up interesting new possibilities.
For which, I agree, the "market demand" is unclear. At the very least I know that a lot of researchers are looking at it closely for ideas and comparisons. And given the licensing, someone who had a high-thread-count embedded application might find a use for. It's not entirely clear to me that the IP is in the noise... it's been a decade, but I was working around chip IP issues in the mid-90s, and a large chunk of the budget was the IP, and another was verifying the level of stuff which is already verified with the T1 design. You still need to synthesize and tape it out, but that's a large chunk of effort taken out.
I agree that the total number of shipped products from the T1 design release could end up being zero. But I wouldn't bet any serious money on that.
Or MOSIS....
A few thousand bucks and a working chip design will get you parts these days, in suprisingly modern fab processes (a few tens of thousand for 0.13u and 90nm).
No, Sun has always designed the majority of the CPUs that they use in their systems. They are fabless; they don't own the chip fabrication factories... but they have buildings full of people working on chip design.
Others have designed SPARC CPU chips (Fujitsu / HAL; Ross) which Sun used, as well. But Sun engineers designed UltraSPARC-I, II, IIi, IIe, III, IIIi, IV, T-1, and going backwards the SuperSPARC, MicroSPARC-I and II, most of the Sun-4 and Sun-4c generation stuff.
Hey Nick, it's been a while.
The difference here is that we're seeing current generation high end processors start to appear here... there's a huge difference between low end, embedded CPU class cores being available and a Niagra T-1 SPARC. T-1 isn't the fastest single thread CPU out there, but it may well be the fastest total multithreaded throughput CPU out there on the market today. Whether that's appropriate for most users / workloads or not, it is clearly a huge difference compared to embedded CPUs.
MMORPGs weren't a significant market player until the late 1990s, when the Internet became ubiquitous and fast enough for most people to actually play high content games.
MTG's effects were clear and openly known and discussed in the game industry. Game store owners loved it: Sales went up 50% or more. Except that what really happened was that game store sales of Magic and competitors went up to about 75-100% of prior sales levels, and the sales of miniatures and paper/dice roleplaying games dropped 25% to 50%.
SJ Games was still recovering from the Secret Service raid when MTG came out. If they hadn't reacted by announcing and then selling INWO they'd have gone under in 1995 or 96 along with GDW. Steve Jackson has admitted that INWO carried the rest of their line for years; they'd have been much more profitable if they'd abandoned most of the rest, but they didn't want to do that.
I grossly disagree with the comment that it was collapsing in on itself; there were a number of firms and a small, and not particularly high growth market. Nobody was truly thriving, but SJG, GDW, TSR, White Wolf were all hanging out and producing stuff and paying their people's salaries. Not all these companies were profitable, but most were, and there was no serious threat of an industry implosion.
MTG arrives and bang, there goes the neighborhood. Wave of companies folding.
I wasn't a major player by any means, but I was there and involved. I'd been selling stuff to GDW every few months for Challenge magazine and looking at getting more involved with projects for them and SJG; a friend had just done a book for SJG. When GDW went down, they owed me some (not serious) money. I'd been talking to them, and got the authors-level financial explanation associated with the corporate shutdown. I also saw the equivalents out of SJG explaining how the Illuminati CCG saved them when the rest of the market tanked.
It's true that the paper/dice RPG market wasn't thriving and growing seriously; it's also true that it wasn't imploding prior to MTG, and was afterwards.
Wow. I guess the age before collecting card games is prehistory, now.