Sometimes I reply politely and let them know of the error. Sometimes I play games to see whats the most crazy thing I can rely and get people to believe (only to update later and let them know).
The tube would be a vacuum. In order to leave the pod you would have to open a hatch which would suck all of the air out of the pod then walk through the dark, mostly airless pipe looking for another hatch to the outside which when opened would blow air rapidly into the pipe until the air equalizes.
Granted "temporary air" canisters can be provided, and an "emergency stop" system can be implemented that automatically breaks the vacuum in the pipe. This is not an impossible problem to solve but a "hatch ever xyz meters" is an over simplified answer to the proposed issue.
wait wait wait.. Give google the credit it deserves.. It actually (accidentally) launched a pgp feature in gmail in 2009 See this blog posting.. Problem was that it was only validation of signatures. There was a bunch of talk on making this a full GPG install but the authors were such crypto nerds that they couldn't get past a "pure" implementation.. which basically breaks if you are using web mail the way it was intended. They wanted a Gears app to read your keys, perform the crypto, then upload messages to Gmail.. etc. I wanted them to implement a "first step", meaning that google would keep the keys for people that wanted PGP in a web mail like experience.. Google could sign the keys at level 1 (minimal trust) and never expose them to end user so the private keys remain private to Google. My argument was that this would get hundreds of millions of users to start having at least a chance to understand encryption, and will provide a way to encrypt for @gmail.com addresses in a way that at least protects up until Google. Now, I know this is nowhere near perfect, but for "perfect" why not just use an IMAP client that actually supports full PGP. I wanted this option pushed because it would have introduced users to the concepts without requiring them to eat the sun completely so to speak.
This is basically the approach that most container systems use. A scratch space is mounted on top of the various container objects that is a partition on LVM. Interacting with the file system will only impact your locally allocated space.
Docker may be like jail() in a way, but true linux cgroups/namespaces are far more powerful. For one, they can be set on individual processes (including threads). So you can create a thread which has a different view of the filesystem than say the main thread. Sure, the attack vector exists to share information between them but now you can basically make one more hop for an attacker. You can make threads which have no network access, or make a thread which has no access to the process list on a system.
So picture using this with a web browser. You can make that crappy module run in a process which has no network access, a root file system that is empty (/var/empty or some such) and can not see any of the other processes on the system. Its only access to the outside world is through a SOCKS proxy passed in as a file descriptor. Even better this can be done with minimal system calls and no setup from the end user so you don't need any of the real infrastructure that jails require. Just recently they added user namespaces as well so uid "0" in a namespace isn't uid 0 on the host OS.
I love that you can harden a web server by having all the threads accept a "resolver" thread have no network access, and have all the threads except a logging thread have no file system access (or limited file system access), while also limiting the resolver thread to say 50M of memory, the main processing thread to 80% CPU and 12G of memory, and the logging thread to 10% CPU and 10k file system operations per second.. etc.
The per thread aspect of the whole setup is way cool, but the zero administrative overhead for a large chunk of it is even cooler. =)
You can call the creditors and ask for proof of debt. If they can nto provide you documentation that you own the money then you can force it to be removed from your credit report.
You would think this but there is a component of the US government doing significant nuclear research in Idaho (http://en.wikipedia.org/wiki/Idaho_National_Laboratory). They had installed something like 50 nuclear reactors in the last 50 years. On top of that they are currently working on building the "next generation" reactor design.. etc. Just because commercial plants were not being built didn't mean that research stopped or that we stopped building reactors all together. Hell, we are currently building something like 5 just for submarines alone: http://www.navy.mil/navydata/f...
Let me explain how a DDoS mitigation strategy works for many of the companies listed in the summary. They setup datacenters in 10, 15, or more places all hosting a proxy. Some of these solutions use DNS to route traffic around problems (GSLB) while others like CloudFlare use Anycast which is awesome and super hard to get right. Each of these services are typically setup with tons of bandwidth capacity, well over 10Gb, but often times into the 100Gb range. They also often have deals with upstream providers that can filter traffic at the edges meaning it never makes it onto the internet in the first place.
Since you servers are not exposed to the internet, and the ones are are have far, far more horsepower to deal with this than a DDoS will even manage from the client side they can easily just churn through the attack, discarding connections and never letting them hit your limited servers. This is how they can easily survive Anonymous style DDoS attacks.
The other thing is to ensure you have turned of every "feature" your load balancer is giving you. SSL termination at the LB, full session management, etc. All of these cost load balancer CPU which is easy to take advantage of, even if there is a DDoS mitigation system in front of your site. You can't just add a few more servers either. Adding capacity to a load balancer is nearly impossible to do mid-attack.
Even more interesting is that you can often times trick the crappy ddos software by doing things like excessively slow responses (tarpitting) making its loop take ages to try again. This is pretty much using the tactics of a DDoS directly against the attackers.
Another common tactic is to add attackers to a view in your bind config that resolves your hostname to 127.0.0.1 just for them. This works if you do not have long TTLs and they are using hostnames. If they are using direct IPs then you simply move your traffic to a second IP and drop the one they are attacking. Best case is if you can do this via BGP announcements so the traffic simply will fail to route and everybody wins.
And yes, I do this professionally but not for any commercial product.
Speaking as somebody that has done hardware qualifications and burn-in development at very large scale for companies you ahve heard of let me tell you the tools I use:
fio: The _BEST_ tool for raw drive performance and burnin testing. A couple of hours of random access will ensure the drive head can dance, then a full block by block walk through with checksum verification will ensure that all blocks are readable and writable.. I usually do 2 or 3 passes here. You can tell fio to reject drives that do not perform to a minimum standard. Very useful for finding functional yet not quite up to speed drives. The statistics produced here are awesome as well.. Something like 70 stats per device per test.
stressapptest: This is google's burn in tool and virtually the only one I have ever found that supports NUMA on modern dual socket machines. This is IMPORTANT as its easy to ignore issues that come up with the link between the CPUs. The various testing modes give you the ability to tear the machine to pieces which is awesome. Stressapptest also is the most power hungry test I have ever seen, including the intel Power testing suite that you have to jump through hoops to get.
Pair this with a pass of memtest and you get a really, really nice burn in system that can burtalize the hardware and give you scriptable systems for detecting failure.
The map does not appear to actually mark the areas of the country where it is completely impossible to setup service. In Idaho, where I grew up, there are huge tracts of government property with restrictions and limitations that make it impossible to have cell service, let alone 3G.
Craters of the Moon is one of the largest exposed lava rock flats in the world. If you go to Google maps and search for "idaho", you will see a huge black spot in the bottom right. The flow is actually much larger than that and its all one big preserve. Its impossible to run underground cables since its all basically solid rock, and running overhead wires is pretty damn challenging as well given the lack of roads.
The Frank Church wilderness area which makes up a large chunk of the middle of the state specifically bans wires and electricity, cell towers, wheels, and pretty much any other modern technology. There is no way it will have 3G coverage any time soon.
Montana has the Bob Marshal wilderness area, Wyoming has Yellowstone, California has Yosemite, etc.
Hell, even the south western part of Idaho is just a big flat desert with virtually no farms, roads, or people. Why should we worry about its 3g coverage?
Why is it that all the people working at scale seems to be going with NoSQL solutions? Are all the devs at Google, Facebook, Twitter, Digg, Redit, etc total idiots or in fact is there a problem that they face that is actually real?
Anybody that sites Amazon, Walmart or any large retailer as an example of why SQL scales is missing the point. Retails have very few write operations compared to the read load. The vast majority of the load hits databases that serve reads and have a high tolerance for write latency. This is a field SQL is good at solving.
On the other hand, social sites that have massive cross user data ties and constant write updates where latency is very important don't fit this model that well. Sure, you can remove SQL replication from the mix, use independent instances of MySQL serving fractions of the overall site, with redundancy between them but if you do that you have functionally built a NoSQL data store. The concept isn't to get right of SQL, its to get rid of the relational aspect of data storage. You can no longer rely on all your data being available to a single SQL statement.
Being an operations guy though I should point out the number one failing of SQL in my world. If you assume that, on average, a machine will either crash or have some sort of hardware failure once a year and you consider a site with 1,000 machines then you see that nearly 3 machines will die every day. Even if you count on 2 years of continuous uptime that is over 1 a day. with 10,000 machines your failure rate is 27 per day, 100,000 machines is 273. This means that any database layer that requires a large number of machines has to build in a recovery layer. Clients need to know that a node is down, when it comes back it needs to have data uploaded to it.. etc. The NoSQL solutions like Cassandra manage this automatically. Trying to do this with MySQL becomes really complicated and you end up implementing all the same logic and constraints in NoSQL solutions anyways. I have seen this happen twice now.
So then tell your friend you won't email him at GMail. I am sorry, email is NOT something you can easily protect based on the very nature of how it is delivered and how much control there is at every point along its delivery route. Concerned about that? Encrypt your emails. Expecting email to be "private" is a joke. Its just like saying that your posts on a blog are private because you turn on some control lists.
Also, have you ever read Googles privacy policies? Its the only company that doesn't blanket state that they will change the privacy policy at any time without notice. They actually say that they will not reduce your rights if they change the policy. They also state that they will only use personal information for the intended purpose and if they decide to use it some other way they will get your consent.
Yea same here. I have had several purchases go completely south with Pay Pal. Money got locked up and I couldn't get refunds until the problem was completely resolved. I have never had any problems with Google Checkout merchants so I have not had a chance to see Google react to a bad purchase, but the TOS for Google Checkout seems much friendlier for both Buyers and Sellers (See the disputes section here. The ability to do dispute resolution actually seems nice compared to PayPal's hit or miss resolution and that makes me trust Google a bit more. Then again, it helps that most Google merchants actually have some screening done on them, where paypal allows anybody to anybody money transfers.
This is what I would expect. When I bugged apple about their broken NFS support on servers they told us that engineers would get back to us. They never did. So I started asking on forums and mailing lists to see if I could get an answer and as soon as I brought it up the thread would get killed or the post would be deleted. Then when we had issues with MPICH it as the same dang thing. Eventually they admitted that MPICH2 works much nicer on Mac OS than MPICH 1 due to some network implementations stuff. Every time I brought it up on the forums though the thread would get killed.
(For the curious, the problem that we where having was that an Apple server running NFS would always seem to forget about the last file in a directory when it cached the directory contents. so running "mkdir a; cd a; touch 1 2 3 4 5 6 ; cd.. ; rm -rf a" would fail one out of four times when being done over NFS. If you waited a half an hour then ran rm -rf a it would work great. This issue didn't happen when Mac OS systems mounted Linux NFS shares, but happened every time a Linux or Mac OS system mounted a NFS share off of a Mac OS based system. This was still happening to all of our PPC based systems as of last summer when we finally switched them over to PPC Linux, which made the problem go away)
I guess what I am saying is that it is not surprising. Apple has always nuked threads that made them look bad so why not this one?
I can not agree more! After a recent interview with a 'darling' tech company I decided that I didn't want to work there. Vague questions and poorly worded rebuttals to answers scared me away. Plus they fell into the cardinal sin of computer science. Never.. Ever.. ask somebody to write answers on a white board. It is not natural. If I prefer to place open/close brackets right when start a function then white boards do not fit that pattern. Little things like this make it more 'English' and less 'code'. As such it is a very poor way to identify good developers. During my entire interview I never once was told what the job would actually be. Generic terms and very vague guidelines where thrown around but nobody could even describe basic work conditions. I have a job. I like my job. Why would I risk taking a job I know nothing about, for very little added pay?
Because of this I decided to not take the job. I contacted them after the interview and let them know. They seemed confused that somebody would interview and then not want the job. They have contacted me and are trying to work with me to resolve my problems. Who knows..
Back on point though. If the company has a terrible hiring process then it is likely a disorganized internal structure that is feeding it. The first tech company I worked for LIED to me in my interview. I was told that I would work days, I ended up working nights, I was told that they would work with me for continued education, they wouldn't do any shift adjusting, nor would they help pay for classes. If I would have paid attention I could have clearly seen the problems. Granted at the time I couldn't have done anything about it and it turned out that the job got me experience that I couldn't have gotten anywhere else, and got me in the Trade Act when I got laid off. So in the end they paid for me to go to school as well as the cost of the education..:)
I run Beowulf Clusters for a living.. Three to be exact. Two run Gentoo and one runs Mac OS. I see Mac OS as a far more likely product in clusters than Microsoft. And even then Mac OS is missing huge chunks of functionality in the cluster world. Checkpointing is broken using Condor and there is no third party apps for Grid Engine. Most programs fail to compile without some massaging. Often programs attempt to compile against native libraries rather than X11. This prevents remote users from using the apps.
Even with all of this though programs can be made to work. I have something like 100 custom programs that needed installed on my clusters. NCBI tools, Bio apps, stuff like that. All of them are coded to Unix environments. Compiling them on windows would be a total pain in the butt! I keep hearing that new programs will be made to work but I don't see that happening all that much. Most new programs are forks of old programs. (At least in the Bio/Geo worlds.) I still have TONS of fortran stuff out there. Lots and lots of stuff that only compiles against GCC 2.95. These things need modified in order to work with a newer version of the SAME OS.. you think a total change is going to happen?
Plus.. The cost of the OS can be killer. When you are talking $1200-$3400 a node an added $500 is huge! Our Mac OS cluster cost us $50k in software licenses. And its 50 nodes. Even if Microsoft drops the price to $100 a pop that is still REALLY expensive. $100 a pop across 50 nodes pays for a bunch more nodes!
So I guess what I am saying is that unless Microsoft starts writing tons of its own apps it won't break into the cluster world very fast. They will be luck to grow as fast as Apple has (%1 of the top 500 list in 4 years).
"Gentoo makes me so much more productive."
Gentoo makes my clusters easy to manage. less time spent compiling custom bio apps means more time I get to spend doing real work.
"Gentoo is more in the spirit of open source!"
Seeing as how I have contributed many bug fixxes and new package builds to the gentoo portage tree without actually coding a single line I would say that I _AM_ involved. There are many others like me.
"Heh, my system is soooo much faster after installing Gentoo."
Benchmarks don't lie. On our clusters we where able to see a ~10% performance increase going from RedHat to Gentoo. Much of this was simply because of the tightly bound environment and low memory footprint between the two distro's.
"You Red Hat guys must get sick of dependency hell..."
I have not had to specify two rpms on the command line to get past a dependancy problem in ages. yum manages that all very nicely. However, I have had to dance and jump through hoops to get odd things like R modules, perl modules, and flakey programs installed. Before you scream cpan and cran bear in mind that I run clusters.. It must be done in mass. Odd module installation in Gentoo usually involves:
emerge -b 'package name' && pdsh -a emerge -K 'package name'
"All the other distros are soooo out of date."
All of the other distro's do not have the volumes of science and biology programs that gentoo has. Screw the up to date software argument I am more interested in the ability to install odd programs that are virtually unheard of with a single command even ROCKS doesn't come close to matching the program listing available for Gentoo.
To completly compile the image that we use here takes ~ 8 hours. That includes all the odd programs and strange things that you would have to find if you where using any other linux. The total install time is about the same as other distro's and it takes far far less of my time to manage the installation. Granted I am an oddball because I do cluster work and not common desktop stuff. You say that Gentoo will never be used in buisness but it actually is.. the last place I worked ran it.. Universities around the country run it.. Just because you think its hard to use doesn't mean that it is.. Gentoo isn't the perfect distro by any means but there is a reason why the people that use it love it. Its nice and clean, easy to manage, and offers lots and lots of configurability for both the admin and the users.
I would so love to prove you wrong right now. Turns out you are completly correct. both of our XServe RAID devices use 7200RPM Ultra IDE Hitachi drives. This invalidates at least some of our benchmarking as it was done single drive on our XServe systems (what where supposed to be like drives but are serial ATA drives.) all of our single drive benchmarks are invalid then (or rather, are meaningless to this discussion =) However, the XServe RAID still performed very well when doing a 5 disk vs 5 disk RAID setup.
What really sucks is that we just found out that our nifty Cisco switch that was purchased before I got here has a severe bandwidth restriction making it nearly useless for MPI communication on our clusters.. 6GB/s between 48 port gig switches. who thought that was a good idea?
Yes, but what I was saying is that blanket excluding drives because they use a SATA->FC converter is limiting the project and not gaining anything. We benchmarked our drives against SCSI drives and other fibre channel solutions. The SATA drives in the XServe RAID kicked the 10K RPM SCSI drives around the block. The 15K SCSI drives did better.. but not "drasticly better" when compared 5 disk array vs 5 disk array.
So my question is this: why limit yourself? Did you look at the "SATA" specifications on the XServe RAID and think "slow" or did you actually throughly look through the design documents on apple.com? I work with a guy that swears up and down that only SCSI/Fibre Channel is the only fast interconnect solution but our benchmarking clearly showed that SATA is often times faster and worst case is only a big slower.
There is no real speed advantage to using FC drives vs SATA drives. The only advantage is when you start chaining a bunch of them together. But the XServe RAID uses a Fibre Channel bus to connect to everything outside of the actual RAID device.
Don't confuse me with somebody that is a die hard apple fan boy. Right now our Mac's are actually running Linux and only account for less than 15% of our computing hardware. We just looked at the XServe RAID and realized that it was by far the best solutin for us being that we wanted a cheap and fast drive array.. exactly what the poster seems to be looking for.
Speaking as the owner of two XServe RAID devices (5TB and 7TB models) as well as several other Fibre Channel devices I can say that the Apple Fibre Channel is by no means slow. Each SATA drive has pretty much equal performance to the SCSI drives we use in our Dell head node. Combined together there are times where we can pull several hundred megs a second off the XRAID's. Plus our XRAID has been fairly immune to failures thusfar. I have yanked drives out of it and it just keeps right on going.
Another little hint, if you are really worried about speed you can just install large high RPM sata drives yourself. Its not that hard to do at all.
Now, when ever I go out and buy a drive, I'm leaning towards Maxtor simply because I have a lot of them and one hasn't failed me with crucial data on it. I'm a lot better prepared to deal with that now as I'm older and wiser so maybe I won't ever feel that level of pain again.
I am sitting in my office lookign at a pile of 20 Maxtor drives.. They all died between 1-2 years old. Maxtor will not return or replace these drives. So while you may have never had problems with your Maxtor drives we have. Take this as you will.
Umm.. Minor point I know. The issue with Micron wasn't with Flash memory. It was with SDRAM. I know this because when Micron layed off 10% of its work force in 2003 the entire flash group got axed. Flas wasn't making money at all.. Of course, the company that bought all Micron's flash assets made tons of cash and now flash is a total cash cow..
Good old Micron.. The thing they are best at is making bad decisions =) (Lehi.. Flash.. SRAM.. QDR.. etc) I watched Micron just sit on some of the best products it had.
Why does a city need to provide WiFi access? Why the push? In Salt Lake City a company called Xmission (I do not work for themnor use there service, though I would if I didn't alread use the University of Utah for both) has provided free WiFi access across the entire downtown area. They contact buisnesses and set it all up free of charge. There motive? Get the Xmission name all over the place by having buisnesses put up "Free WiFi provided by Xmission" signs.
Slashdot Mirror
I have a five character gmail address that is both a common first and last name. I get misdirected emails every day. So far I have gotten emails for:
* Companies (same name)
* Doctors.
* Sons, daughters, dads, etc.
* Contract details.
* Job offers.
* Politicians.
Sometimes I reply politely and let them know of the error. Sometimes I play games to see whats the most crazy thing I can rely and get people to believe (only to update later and let them know).
The tube would be a vacuum. In order to leave the pod you would have to open a hatch which would suck all of the air out of the pod then walk through the dark, mostly airless pipe looking for another hatch to the outside which when opened would blow air rapidly into the pipe until the air equalizes. Granted "temporary air" canisters can be provided, and an "emergency stop" system can be implemented that automatically breaks the vacuum in the pipe. This is not an impossible problem to solve but a "hatch ever xyz meters" is an over simplified answer to the proposed issue.
In the early days an employee lived in an RV in the parking lot. His RV was nicknamed the "Weaverplex". It wasn't really secret either.
wait wait wait.. Give google the credit it deserves.. It actually (accidentally) launched a pgp feature in gmail in 2009 See this blog posting.. Problem was that it was only validation of signatures. There was a bunch of talk on making this a full GPG install but the authors were such crypto nerds that they couldn't get past a "pure" implementation.. which basically breaks if you are using web mail the way it was intended. They wanted a Gears app to read your keys, perform the crypto, then upload messages to Gmail.. etc. I wanted them to implement a "first step", meaning that google would keep the keys for people that wanted PGP in a web mail like experience.. Google could sign the keys at level 1 (minimal trust) and never expose them to end user so the private keys remain private to Google. My argument was that this would get hundreds of millions of users to start having at least a chance to understand encryption, and will provide a way to encrypt for @gmail.com addresses in a way that at least protects up until Google. Now, I know this is nowhere near perfect, but for "perfect" why not just use an IMAP client that actually supports full PGP. I wanted this option pushed because it would have introduced users to the concepts without requiring them to eat the sun completely so to speak.
This is basically the approach that most container systems use. A scratch space is mounted on top of the various container objects that is a partition on LVM. Interacting with the file system will only impact your locally allocated space.
Docker may be like jail() in a way, but true linux cgroups/namespaces are far more powerful. For one, they can be set on individual processes (including threads). So you can create a thread which has a different view of the filesystem than say the main thread. Sure, the attack vector exists to share information between them but now you can basically make one more hop for an attacker. You can make threads which have no network access, or make a thread which has no access to the process list on a system.
So picture using this with a web browser. You can make that crappy module run in a process which has no network access, a root file system that is empty (/var/empty or some such) and can not see any of the other processes on the system. Its only access to the outside world is through a SOCKS proxy passed in as a file descriptor. Even better this can be done with minimal system calls and no setup from the end user so you don't need any of the real infrastructure that jails require. Just recently they added user namespaces as well so uid "0" in a namespace isn't uid 0 on the host OS.
I love that you can harden a web server by having all the threads accept a "resolver" thread have no network access, and have all the threads except a logging thread have no file system access (or limited file system access), while also limiting the resolver thread to say 50M of memory, the main processing thread to 80% CPU and 12G of memory, and the logging thread to 10% CPU and 10k file system operations per second.. etc.
The per thread aspect of the whole setup is way cool, but the zero administrative overhead for a large chunk of it is even cooler. =)
You can call the creditors and ask for proof of debt. If they can nto provide you documentation that you own the money then you can force it to be removed from your credit report.
You would think this but there is a component of the US government doing significant nuclear research in Idaho (http://en.wikipedia.org/wiki/Idaho_National_Laboratory). They had installed something like 50 nuclear reactors in the last 50 years. On top of that they are currently working on building the "next generation" reactor design.. etc. Just because commercial plants were not being built didn't mean that research stopped or that we stopped building reactors all together. Hell, we are currently building something like 5 just for submarines alone: http://www.navy.mil/navydata/f...
This is a bit of a naive explanation.
Let me explain how a DDoS mitigation strategy works for many of the companies listed in the summary. They setup datacenters in 10, 15, or more places all hosting a proxy. Some of these solutions use DNS to route traffic around problems (GSLB) while others like CloudFlare use Anycast which is awesome and super hard to get right. Each of these services are typically setup with tons of bandwidth capacity, well over 10Gb, but often times into the 100Gb range. They also often have deals with upstream providers that can filter traffic at the edges meaning it never makes it onto the internet in the first place.
Since you servers are not exposed to the internet, and the ones are are have far, far more horsepower to deal with this than a DDoS will even manage from the client side they can easily just churn through the attack, discarding connections and never letting them hit your limited servers. This is how they can easily survive Anonymous style DDoS attacks.
The other thing is to ensure you have turned of every "feature" your load balancer is giving you. SSL termination at the LB, full session management, etc. All of these cost load balancer CPU which is easy to take advantage of, even if there is a DDoS mitigation system in front of your site. You can't just add a few more servers either. Adding capacity to a load balancer is nearly impossible to do mid-attack.
Even more interesting is that you can often times trick the crappy ddos software by doing things like excessively slow responses (tarpitting) making its loop take ages to try again. This is pretty much using the tactics of a DDoS directly against the attackers.
Another common tactic is to add attackers to a view in your bind config that resolves your hostname to 127.0.0.1 just for them. This works if you do not have long TTLs and they are using hostnames. If they are using direct IPs then you simply move your traffic to a second IP and drop the one they are attacking. Best case is if you can do this via BGP announcements so the traffic simply will fail to route and everybody wins.
And yes, I do this professionally but not for any commercial product.
Speaking as somebody that has done hardware qualifications and burn-in development at very large scale for companies you ahve heard of let me tell you the tools I use:
fio: The _BEST_ tool for raw drive performance and burnin testing. A couple of hours of random access will ensure the drive head can dance, then a full block by block walk through with checksum verification will ensure that all blocks are readable and writable.. I usually do 2 or 3 passes here. You can tell fio to reject drives that do not perform to a minimum standard. Very useful for finding functional yet not quite up to speed drives. The statistics produced here are awesome as well.. Something like 70 stats per device per test.
stressapptest: This is google's burn in tool and virtually the only one I have ever found that supports NUMA on modern dual socket machines. This is IMPORTANT as its easy to ignore issues that come up with the link between the CPUs. The various testing modes give you the ability to tear the machine to pieces which is awesome. Stressapptest also is the most power hungry test I have ever seen, including the intel Power testing suite that you have to jump through hoops to get.
Pair this with a pass of memtest and you get a really, really nice burn in system that can burtalize the hardware and give you scriptable systems for detecting failure.
The map does not appear to actually mark the areas of the country where it is completely impossible to setup service. In Idaho, where I grew up, there are huge tracts of government property with restrictions and limitations that make it impossible to have cell service, let alone 3G.
Craters of the Moon is one of the largest exposed lava rock flats in the world. If you go to Google maps and search for "idaho", you will see a huge black spot in the bottom right. The flow is actually much larger than that and its all one big preserve. Its impossible to run underground cables since its all basically solid rock, and running overhead wires is pretty damn challenging as well given the lack of roads.
The Frank Church wilderness area which makes up a large chunk of the middle of the state specifically bans wires and electricity, cell towers, wheels, and pretty much any other modern technology. There is no way it will have 3G coverage any time soon.
Montana has the Bob Marshal wilderness area, Wyoming has Yellowstone, California has Yosemite, etc.
Hell, even the south western part of Idaho is just a big flat desert with virtually no farms, roads, or people. Why should we worry about its 3g coverage?
Joe Stump wrote a post that is a perfect response to this insanity.
http://stu.mp/category/nosql
Why is it that all the people working at scale seems to be going with NoSQL solutions? Are all the devs at Google, Facebook, Twitter, Digg, Redit, etc total idiots or in fact is there a problem that they face that is actually real?
Anybody that sites Amazon, Walmart or any large retailer as an example of why SQL scales is missing the point. Retails have very few write operations compared to the read load. The vast majority of the load hits databases that serve reads and have a high tolerance for write latency. This is a field SQL is good at solving.
On the other hand, social sites that have massive cross user data ties and constant write updates where latency is very important don't fit this model that well. Sure, you can remove SQL replication from the mix, use independent instances of MySQL serving fractions of the overall site, with redundancy between them but if you do that you have functionally built a NoSQL data store. The concept isn't to get right of SQL, its to get rid of the relational aspect of data storage. You can no longer rely on all your data being available to a single SQL statement.
Being an operations guy though I should point out the number one failing of SQL in my world. If you assume that, on average, a machine will either crash or have some sort of hardware failure once a year and you consider a site with 1,000 machines then you see that nearly 3 machines will die every day. Even if you count on 2 years of continuous uptime that is over 1 a day. with 10,000 machines your failure rate is 27 per day, 100,000 machines is 273. This means that any database layer that requires a large number of machines has to build in a recovery layer. Clients need to know that a node is down, when it comes back it needs to have data uploaded to it.. etc. The NoSQL solutions like Cassandra manage this automatically. Trying to do this with MySQL becomes really complicated and you end up implementing all the same logic and constraints in NoSQL solutions anyways. I have seen this happen twice now.
So then tell your friend you won't email him at GMail. I am sorry, email is NOT something you can easily protect based on the very nature of how it is delivered and how much control there is at every point along its delivery route. Concerned about that? Encrypt your emails. Expecting email to be "private" is a joke. Its just like saying that your posts on a blog are private because you turn on some control lists.
Also, have you ever read Googles privacy policies? Its the only company that doesn't blanket state that they will change the privacy policy at any time without notice. They actually say that they will not reduce your rights if they change the policy. They also state that they will only use personal information for the intended purpose and if they decide to use it some other way they will get your consent.
Googles private PolicyYea same here. I have had several purchases go completely south with Pay Pal. Money got locked up and I couldn't get refunds until the problem was completely resolved. I have never had any problems with Google Checkout merchants so I have not had a chance to see Google react to a bad purchase, but the TOS for Google Checkout seems much friendlier for both Buyers and Sellers (See the disputes section here. The ability to do dispute resolution actually seems nice compared to PayPal's hit or miss resolution and that makes me trust Google a bit more. Then again, it helps that most Google merchants actually have some screening done on them, where paypal allows anybody to anybody money transfers.
This is what I would expect. When I bugged apple about their broken NFS support on servers they told us that engineers would get back to us. They never did. So I started asking on forums and mailing lists to see if I could get an answer and as soon as I brought it up the thread would get killed or the post would be deleted. Then when we had issues with MPICH it as the same dang thing. Eventually they admitted that MPICH2 works much nicer on Mac OS than MPICH 1 due to some network implementations stuff. Every time I brought it up on the forums though the thread would get killed. (For the curious, the problem that we where having was that an Apple server running NFS would always seem to forget about the last file in a directory when it cached the directory contents. so running "mkdir a; cd a; touch 1 2 3 4 5 6 ; cd .. ; rm -rf a" would fail one out of four times when being done over NFS. If you waited a half an hour then ran rm -rf a it would work great. This issue didn't happen when Mac OS systems mounted Linux NFS shares, but happened every time a Linux or Mac OS system mounted a NFS share off of a Mac OS based system. This was still happening to all of our PPC based systems as of last summer when we finally switched them over to PPC Linux, which made the problem go away)
I guess what I am saying is that it is not surprising. Apple has always nuked threads that made them look bad so why not this one?
Remember though, Lots of domains are hosted by GMail, like *@google.com and such. The new hosted system is what this is targeting.
I can not agree more! After a recent interview with a 'darling' tech company I decided that I didn't want to work there. Vague questions and poorly worded rebuttals to answers scared me away. Plus they fell into the cardinal sin of computer science. Never.. Ever.. ask somebody to write answers on a white board. It is not natural. If I prefer to place open/close brackets right when start a function then white boards do not fit that pattern. Little things like this make it more 'English' and less 'code'. As such it is a very poor way to identify good developers. During my entire interview I never once was told what the job would actually be. Generic terms and very vague guidelines where thrown around but nobody could even describe basic work conditions. I have a job. I like my job. Why would I risk taking a job I know nothing about, for very little added pay?
Because of this I decided to not take the job. I contacted them after the interview and let them know. They seemed confused that somebody would interview and then not want the job. They have contacted me and are trying to work with me to resolve my problems. Who knows..
Back on point though. If the company has a terrible hiring process then it is likely a disorganized internal structure that is feeding it. The first tech company I worked for LIED to me in my interview. I was told that I would work days, I ended up working nights, I was told that they would work with me for continued education, they wouldn't do any shift adjusting, nor would they help pay for classes. If I would have paid attention I could have clearly seen the problems. Granted at the time I couldn't have done anything about it and it turned out that the job got me experience that I couldn't have gotten anywhere else, and got me in the Trade Act when I got laid off. So in the end they paid for me to go to school as well as the cost of the education.. :)
As a cluster admin that runs a little under 250 Gentoo systems I guess that makes me a Chinese farmer or something =)
I run Beowulf Clusters for a living.. Three to be exact. Two run Gentoo and one runs Mac OS. I see Mac OS as a far more likely product in clusters than Microsoft. And even then Mac OS is missing huge chunks of functionality in the cluster world. Checkpointing is broken using Condor and there is no third party apps for Grid Engine. Most programs fail to compile without some massaging. Often programs attempt to compile against native libraries rather than X11. This prevents remote users from using the apps.
Even with all of this though programs can be made to work. I have something like 100 custom programs that needed installed on my clusters. NCBI tools, Bio apps, stuff like that. All of them are coded to Unix environments. Compiling them on windows would be a total pain in the butt! I keep hearing that new programs will be made to work but I don't see that happening all that much. Most new programs are forks of old programs. (At least in the Bio/Geo worlds.) I still have TONS of fortran stuff out there. Lots and lots of stuff that only compiles against GCC 2.95. These things need modified in order to work with a newer version of the SAME OS.. you think a total change is going to happen?
Plus.. The cost of the OS can be killer. When you are talking $1200-$3400 a node an added $500 is huge! Our Mac OS cluster cost us $50k in software licenses. And its 50 nodes. Even if Microsoft drops the price to $100 a pop that is still REALLY expensive. $100 a pop across 50 nodes pays for a bunch more nodes!
So I guess what I am saying is that unless Microsoft starts writing tons of its own apps it won't break into the cluster world very fast. They will be luck to grow as fast as Apple has (%1 of the top 500 list in 4 years).
"Gentoo makes me so much more productive."
Gentoo makes my clusters easy to manage. less time spent compiling custom bio apps means more time I get to spend doing real work.
"Gentoo is more in the spirit of open source!"
Seeing as how I have contributed many bug fixxes and new package builds to the gentoo portage tree without actually coding a single line I would say that I _AM_ involved. There are many others like me.
"Heh, my system is soooo much faster after installing Gentoo."
Benchmarks don't lie. On our clusters we where able to see a ~10% performance increase going from RedHat to Gentoo. Much of this was simply because of the tightly bound environment and low memory footprint between the two distro's.
"You Red Hat guys must get sick of dependency hell..."
I have not had to specify two rpms on the command line to get past a dependancy problem in ages. yum manages that all very nicely. However, I have had to dance and jump through hoops to get odd things like R modules, perl modules, and flakey programs installed. Before you scream cpan and cran bear in mind that I run clusters.. It must be done in mass. Odd module installation in Gentoo usually involves:
emerge -b 'package name' && pdsh -a emerge -K 'package name'
"All the other distros are soooo out of date."
All of the other distro's do not have the volumes of science and biology programs that gentoo has. Screw the up to date software argument I am more interested in the ability to install odd programs that are virtually unheard of with a single command even ROCKS doesn't come close to matching the program listing available for Gentoo.
To completly compile the image that we use here takes ~ 8 hours. That includes all the odd programs and strange things that you would have to find if you where using any other linux. The total install time is about the same as other distro's and it takes far far less of my time to manage the installation. Granted I am an oddball because I do cluster work and not common desktop stuff. You say that Gentoo will never be used in buisness but it actually is.. the last place I worked ran it.. Universities around the country run it.. Just because you think its hard to use doesn't mean that it is.. Gentoo isn't the perfect distro by any means but there is a reason why the people that use it love it. Its nice and clean, easy to manage, and offers lots and lots of configurability for both the admin and the users.
I would so love to prove you wrong right now. Turns out you are completly correct. both of our XServe RAID devices use 7200RPM Ultra IDE Hitachi drives. This invalidates at least some of our benchmarking as it was done single drive on our XServe systems (what where supposed to be like drives but are serial ATA drives.) all of our single drive benchmarks are invalid then (or rather, are meaningless to this discussion =) However, the XServe RAID still performed very well when doing a 5 disk vs 5 disk RAID setup.
What really sucks is that we just found out that our nifty Cisco switch that was purchased before I got here has a severe bandwidth restriction making it nearly useless for MPI communication on our clusters.. 6GB/s between 48 port gig switches. who thought that was a good idea?
Yes, but what I was saying is that blanket excluding drives because they use a SATA->FC converter is limiting the project and not gaining anything. We benchmarked our drives against SCSI drives and other fibre channel solutions. The SATA drives in the XServe RAID kicked the 10K RPM SCSI drives around the block. The 15K SCSI drives did better.. but not "drasticly better" when compared 5 disk array vs 5 disk array.
So my question is this: why limit yourself? Did you look at the "SATA" specifications on the XServe RAID and think "slow" or did you actually throughly look through the design documents on apple.com? I work with a guy that swears up and down that only SCSI/Fibre Channel is the only fast interconnect solution but our benchmarking clearly showed that SATA is often times faster and worst case is only a big slower.
There is no real speed advantage to using FC drives vs SATA drives. The only advantage is when you start chaining a bunch of them together. But the XServe RAID uses a Fibre Channel bus to connect to everything outside of the actual RAID device.
Don't confuse me with somebody that is a die hard apple fan boy. Right now our Mac's are actually running Linux and only account for less than 15% of our computing hardware. We just looked at the XServe RAID and realized that it was by far the best solutin for us being that we wanted a cheap and fast drive array.. exactly what the poster seems to be looking for.
Speaking as the owner of two XServe RAID devices (5TB and 7TB models) as well as several other Fibre Channel devices I can say that the Apple Fibre Channel is by no means slow. Each SATA drive has pretty much equal performance to the SCSI drives we use in our Dell head node. Combined together there are times where we can pull several hundred megs a second off the XRAID's. Plus our XRAID has been fairly immune to failures thusfar. I have yanked drives out of it and it just keeps right on going.
Another little hint, if you are really worried about speed you can just install large high RPM sata drives yourself. Its not that hard to do at all.
Check out Alien Raid for more information.
Now, when ever I go out and buy a drive, I'm leaning towards Maxtor simply because I have a lot of them and one hasn't failed me with crucial data on it. I'm a lot better prepared to deal with that now as I'm older and wiser so maybe I won't ever feel that level of pain again.
I am sitting in my office lookign at a pile of 20 Maxtor drives.. They all died between 1-2 years old. Maxtor will not return or replace these drives. So while you may have never had problems with your Maxtor drives we have. Take this as you will.
Umm.. Minor point I know. The issue with Micron wasn't with Flash memory. It was with SDRAM. I know this because when Micron layed off 10% of its work force in 2003 the entire flash group got axed. Flas wasn't making money at all.. Of course, the company that bought all Micron's flash assets made tons of cash and now flash is a total cash cow..
Good old Micron.. The thing they are best at is making bad decisions =) (Lehi.. Flash.. SRAM.. QDR.. etc) I watched Micron just sit on some of the best products it had.
Why does a city need to provide WiFi access? Why the push? In Salt Lake City a company called Xmission (I do not work for themnor use there service, though I would if I didn't alread use the University of Utah for both) has provided free WiFi access across the entire downtown area. They contact buisnesses and set it all up free of charge. There motive? Get the Xmission name all over the place by having buisnesses put up "Free WiFi provided by Xmission" signs.