All software is consistent. Whether you can predict it is more a function of your own intelligence and understanding.
Not that I'm disparaging you, though. I agree that software should be predictable by the user, and that's why I only support this feature if it will include lots of configuration, so the software can also understand the user's desires appropriately. After spending an hour or two configuring Compiz to know that (for example) my mouse follows my attention, I increased my productivity. I expect this to be no different.
After all, how long does it take for a human to be able to predict another human's actions, with any reasonable accuracy?
My understanding is that it's the re-connecting difficulty. Mangled pipes, broken in several places and mixed in with the remains of a sunken drilling rig, are not very nice to work with. It's easier just to drill a new well a few miles away.
I'm no DBA, but my limited understanding is that sharding requires advance knowledge of the data being stored and the application thereof. My understanding also is that there are some designs that can't be effectively sharded without a huge cost penalty.
That's great if your programmers are sitting right next to the DBAs, but it's pretty bad for systems where other have to access the database as well. A project I'm working on now involves a painfully-slow SQL query, because we need to query by date, and the table is partitioned by an id number. Politics and other issues mean we won't be getting an index just to help our very-limited use case.
Hadoop (and more directly equal to SQL databases, HBase) distributes data with no prior knowledge. The distribution is done automatically, with no concern for what the data actually is or how it might be used. It keeps generality up and costs down, going back to my original comment about practical solutions.
Maybe it's the $200,000 investment from the community that "forced them to release" whatever they could as soon as possible. Having something to show for that initial investment is likely to bring in more support. There are two kinds of projects: Those that need to show progress to get support, and Duke Nukem Forever.
Except for the name, the partly-finished projects that aren't yet distributed, the reputation, the control of the project, etc... If Novell were to keep a group doing "the same thing", they wouldn't be much better off than someone else just forking the project.
VMWare would be "buying a Linux company", not "buying Linux". Like any corporate acquisition, it makes sense if the assets are worth more to VMWare than the amount they're paying for it.
Consider what VMWare might be able to do with SuSE. SuSE has a good reputation and a relatively devoted user base. My speculation is that VMWare will add a new awesome feature to their virtualization products, but say that it's only fully supported under SuSE. Their corporate customers, being familiar with SuSE's current reputation, will likely be willing to convert from Red Hat or other distros over to SuSE, just because of the added feature. VMWare has then turned that user base into a new set of paying customers.
Without the SuSE name, VMWare would have just another magic feature. Without the magic feature, SuSE's just a name. Together, they can make a profitable product.
Not if it's anything like every big project I've worked on.
First, projects go through a phase of "how can we do this" where various components are mashed together with the expectation that things will work later. That's a good thing to do while gathering initial funding.
Then they go through the phase of "we can do this" where some parts of the project work, but most is broken.
That's followed by the "demonstration" phase, where things work under perfect circumstances. That seems to be where Diaspora is at now.
Next is the "we can do this well" phase, where the once-connected components are split up and divided into their appropriate layers and security is locked down, now that there's a clear idea of what the security model must support.
Finally is the "continued development" phase, where the project is stable enough that new components don't need major changes to security, and extra features can be added.
I've had a few projects that started with the frameworks and various layers of abstraction, and they've invariably failed after many refactorings and revisions. Heck, one project I worked on was a web-based game engine, which turned into a giant security model, and finally died without a single line of actual game code written. It took eight months to fail miserably. Projects change, and requirements change. Going into a security model too early can be worse than not having one.
Street View: really useful for avoiding comments like "I was running right on time for the job interview, but I couldn't find your office and spent a half an hour wandering around the block"
I'm waiting for the law requiring mining companies to pay for the damages to the families of people killed with knives and bullets. They're obviously benefiting from such crimes and doing nothing about it.
Assuming the maximum configuration is thousands of cores, how does it compare in other aspects to Facebook's 23,000 cores and 36 petabytes of data, with unlimited scalability to come?
For all intents and purposes, mainframes are still mainframes. They're parallel, and they grow, but they still have those limits that clusters just don't have.
Sure, insurance companies have kept claims data around for many years. They make some pretty good observations about obvious correlations. People who speed too much tend to hit more things. People with chronic diseases tend to die.
What about the data they couldn't handle, though? What about the effects of someone's purchases? Did they buy quality brake pads? What about the circuit breakers installed in their house the last time it was remodeled? What contractor did they hire? How many of that contractor's other remodels had later electrical problems?
Such questions used to be far outside the scope of what was feasible to handle. That's Big Data, and it comes from all sorts of sources, like the mechanic who installed the brake pads. An insurance company can purchase data from the mechanic, and get a better idea of their customers' overall risk. That (ideally) leads to more accurate pricing. A little data here, a little there, and a lot of connections.
That's a lot of data, though. Just think how many car repairs are being done right now. Hadoop helps make the data manageable by keeping all the maintenance behind a nice layer of abstraction. Virtualization helps make Hadoop manageable, since small businesses can purchase server time based on use, rather than invest in a huge cluster right off the bat. Having a huge dataset is now easy, and that qualifies as "stuff that matters".
Think about the term "mirror" for a while. In most mirrored setups, all data goes everywhere. Yes, we've had mirrors for quite a while, and they've provided linear speedup, and increase the cost of upgrades. A cluster doesn't just copy the data everywhere. Hadoop, for example, replicates each block only three times by default. On a large cluster, it's very likely that two randomly-selected nodes won't share any common data. The idea of having parallel data access isn't new, but the technology to do it efficiently while maintaining a low cost is. That's why I included the word "practically" in my first sentence.
It's a new and more efficient use of an old concept. Bicycles have been around for about two hundred years, and yet the Tour de France still makes the news.
Mainframes don't offer the data capacity that a large cluster does. The topic here is handling big datasets (measured in the billions of records), and single machines just can't do that as efficiently as a cluster, no matter how beefy they are. The future is parallel.
It isn't about Facebook so much as it's a shift in what problems are practically solvable.
First, realize that traditional approaches like SQL are limited mostly by the single box (or the few mirrors) the platform runs on. Querying a large (a billion rows) table can take minutes on a very fast machine, hours if there's significant disk access needed, and months if the query's complex enough. Clusters can process those same billion records far faster, bringing that time down from months to hours, or even seconds for a simple scan. Advances in cluster computing over the last few years have made this parallel processing much easier.
The promise is that problems that were previously too big to even think about are now easy. If your solved problem is something people want, like showing what their friends are up to, your product will do well.
First, I'm not really sure what you mean by "copy it into HDFS", since that's usually where data is stored in the first place. Copying in lots of data without giving the cluster time to stabilize can cause it to go into safe mode, where it won't make changes until everything's properly distributed. It's version 0.20. Don't expect perfection just yet.
There are also experts out there who will be quite happy to help get things running better. My company has been using Hadoop quite successfully with their help.
Dells are cheaper than IBM, for the amount of performance they provide. Big Data is best handled in a large cluster, which provides reliability and parallelism to process a highly-scalable amount of data in a very short time. Beyond a certain point, high-end machines just raise the cost of failure, where low-end machines provide only marginally lower performance.
Be a nice professor. Make microwave popcorn during the exam for the students. Use an old low-power microwave, because I find it makes the popcorn taste just a bit better. If your class is large enough and hungry enough, you'll have no cheating (via wi-fi, at least) for the duration of the exam.
So, in order to protect American soldiers, we shouldn't overreact to this and jump on the "OMG CENSORSHIP!" bandwagon. Instead, we ought to just say "it's not a big deal" and move on.
I guess you did miss it. As a quick review, acts of Congress become laws. The first amendment to the Constitution specifically limits the powers of Congress. The judicial branch of government is charged with determining if a given law is in line with the restrictions in the Constitution.
The relevant details are in the case Brandenburg v. Ohio, which held that the Espionage Act can indeed limit speech if that speech is likely to result in "imminent lawless action", the precise definition of which has not been declared. In fact, the Brandenburg case modifies the ruling of Schenck v. United States, which is where the common example of 'yelling "fire" in a crowded theater' comes from.
In short, it's not the act of placing others at risk that's the problem, but creating a situation that's likely to lead to illegal action. Releasing the names and details of intelligence officers puts them at risk of murder, which is illegal everywhere I know of. Sorry for the lack of clarity.
National security is why we have a government in the first place. It falls under the mandate to "insure domestic Tranquility, [and] provide for the common defence" that the Constitution was written for in the first place. I'm not saying it's a magic phrase that excuses heinous breaches of civil rights, but now and then there's a point to it.
If one copy will be seen by 100,000 people, then two copies leaking is 200,000, and that's twice the risk. The government screwed up (big surprise) by approving the book completely in the first place. Now they're trying to clean up their mistake. A comparison to 1984 is almost (but not quite) appropriate here. The government isn't trying to change recorded history, but merely to delay facts from coming out while they're still risky.
The publisher could do something underhanded, like deciding to only sell the books overseas, and would probably get a lot of love from the anti-government folks here. It'd also guarantee their next military-related book gets a much more thorough review. Is it really worth the trouble and risk, just so a few civilians can get a real name that they'll forget an hour later? I don't think so.
For that, you'll have to look at the Espionage Act, and its amendments in the Sedition act. In summary, it's illegal to aid groups the American government has declared to be enemies.
I'm afraid I don't remember the details involved (coincidence, I swear), but I seem to remember a biographical book a few years ago causing quite a ruckus, as it indicated that an elected official (in California, I think) had a relationship with a prostitute. It turned out to be false, but the guy's career was ruined. Similar stories abound for abortion clinic doctors. It seems having a real identity in the public spotlight is indeed a risk.
Relatedly, this is why so many fictional works have the disclaimer that "any relation to actual persons, living or dead, is entirely coincidental." Being responsible about others' lives does not have to get in the way of telling a story.
Slashdot Mirror
All software is consistent. Whether you can predict it is more a function of your own intelligence and understanding.
Not that I'm disparaging you, though. I agree that software should be predictable by the user, and that's why I only support this feature if it will include lots of configuration, so the software can also understand the user's desires appropriately. After spending an hour or two configuring Compiz to know that (for example) my mouse follows my attention, I increased my productivity. I expect this to be no different.
After all, how long does it take for a human to be able to predict another human's actions, with any reasonable accuracy?
My understanding is that it's the re-connecting difficulty. Mangled pipes, broken in several places and mixed in with the remains of a sunken drilling rig, are not very nice to work with. It's easier just to drill a new well a few miles away.
I'm no DBA, but my limited understanding is that sharding requires advance knowledge of the data being stored and the application thereof. My understanding also is that there are some designs that can't be effectively sharded without a huge cost penalty.
That's great if your programmers are sitting right next to the DBAs, but it's pretty bad for systems where other have to access the database as well. A project I'm working on now involves a painfully-slow SQL query, because we need to query by date, and the table is partitioned by an id number. Politics and other issues mean we won't be getting an index just to help our very-limited use case.
Hadoop (and more directly equal to SQL databases, HBase) distributes data with no prior knowledge. The distribution is done automatically, with no concern for what the data actually is or how it might be used. It keeps generality up and costs down, going back to my original comment about practical solutions.
Maybe it's the $200,000 investment from the community that "forced them to release" whatever they could as soon as possible. Having something to show for that initial investment is likely to bring in more support. There are two kinds of projects: Those that need to show progress to get support, and Duke Nukem Forever.
Except for the name, the partly-finished projects that aren't yet distributed, the reputation, the control of the project, etc... If Novell were to keep a group doing "the same thing", they wouldn't be much better off than someone else just forking the project.
VMWare would be "buying a Linux company", not "buying Linux". Like any corporate acquisition, it makes sense if the assets are worth more to VMWare than the amount they're paying for it.
Consider what VMWare might be able to do with SuSE. SuSE has a good reputation and a relatively devoted user base. My speculation is that VMWare will add a new awesome feature to their virtualization products, but say that it's only fully supported under SuSE. Their corporate customers, being familiar with SuSE's current reputation, will likely be willing to convert from Red Hat or other distros over to SuSE, just because of the added feature. VMWare has then turned that user base into a new set of paying customers.
Without the SuSE name, VMWare would have just another magic feature. Without the magic feature, SuSE's just a name. Together, they can make a profitable product.
Not if it's anything like every big project I've worked on.
First, projects go through a phase of "how can we do this" where various components are mashed together with the expectation that things will work later. That's a good thing to do while gathering initial funding.
Then they go through the phase of "we can do this" where some parts of the project work, but most is broken.
That's followed by the "demonstration" phase, where things work under perfect circumstances. That seems to be where Diaspora is at now.
Next is the "we can do this well" phase, where the once-connected components are split up and divided into their appropriate layers and security is locked down, now that there's a clear idea of what the security model must support.
Finally is the "continued development" phase, where the project is stable enough that new components don't need major changes to security, and extra features can be added.
I've had a few projects that started with the frameworks and various layers of abstraction, and they've invariably failed after many refactorings and revisions. Heck, one project I worked on was a web-based game engine, which turned into a giant security model, and finally died without a single line of actual game code written. It took eight months to fail miserably. Projects change, and requirements change. Going into a security model too early can be worse than not having one.
After a few months, a big project has bugs? Really? That's amazing! After all, Windows has been around for only 20 years and it's perfect, right?
I think I'll reserve judgment for sometime in 2012...
Street View: really useful for avoiding comments like "I was running right on time for the job interview, but I couldn't find your office and spent a half an hour wandering around the block"
And Linux users now. Nothing new here.
I'm waiting for the law requiring mining companies to pay for the damages to the families of people killed with knives and bullets. They're obviously benefiting from such crimes and doing nothing about it.
Assuming the maximum configuration is thousands of cores, how does it compare in other aspects to Facebook's 23,000 cores and 36 petabytes of data, with unlimited scalability to come?
For all intents and purposes, mainframes are still mainframes. They're parallel, and they grow, but they still have those limits that clusters just don't have.
(I consider price to be a limit as well)
Money and hardware are interchangeable. Price per cycle is still a measure of efficiency.
You're missing the point.
Sure, insurance companies have kept claims data around for many years. They make some pretty good observations about obvious correlations. People who speed too much tend to hit more things. People with chronic diseases tend to die.
What about the data they couldn't handle, though? What about the effects of someone's purchases? Did they buy quality brake pads? What about the circuit breakers installed in their house the last time it was remodeled? What contractor did they hire? How many of that contractor's other remodels had later electrical problems?
Such questions used to be far outside the scope of what was feasible to handle. That's Big Data, and it comes from all sorts of sources, like the mechanic who installed the brake pads. An insurance company can purchase data from the mechanic, and get a better idea of their customers' overall risk. That (ideally) leads to more accurate pricing. A little data here, a little there, and a lot of connections.
That's a lot of data, though. Just think how many car repairs are being done right now. Hadoop helps make the data manageable by keeping all the maintenance behind a nice layer of abstraction. Virtualization helps make Hadoop manageable, since small businesses can purchase server time based on use, rather than invest in a huge cluster right off the bat. Having a huge dataset is now easy, and that qualifies as "stuff that matters".
Think about the term "mirror" for a while. In most mirrored setups, all data goes everywhere. Yes, we've had mirrors for quite a while, and they've provided linear speedup, and increase the cost of upgrades. A cluster doesn't just copy the data everywhere. Hadoop, for example, replicates each block only three times by default. On a large cluster, it's very likely that two randomly-selected nodes won't share any common data. The idea of having parallel data access isn't new, but the technology to do it efficiently while maintaining a low cost is. That's why I included the word "practically" in my first sentence.
It's a new and more efficient use of an old concept. Bicycles have been around for about two hundred years, and yet the Tour de France still makes the news.
Mainframes don't offer the data capacity that a large cluster does. The topic here is handling big datasets (measured in the billions of records), and single machines just can't do that as efficiently as a cluster, no matter how beefy they are. The future is parallel.
It isn't about Facebook so much as it's a shift in what problems are practically solvable.
First, realize that traditional approaches like SQL are limited mostly by the single box (or the few mirrors) the platform runs on. Querying a large (a billion rows) table can take minutes on a very fast machine, hours if there's significant disk access needed, and months if the query's complex enough. Clusters can process those same billion records far faster, bringing that time down from months to hours, or even seconds for a simple scan. Advances in cluster computing over the last few years have made this parallel processing much easier.
The promise is that problems that were previously too big to even think about are now easy. If your solved problem is something people want, like showing what their friends are up to, your product will do well.
First, I'm not really sure what you mean by "copy it into HDFS", since that's usually where data is stored in the first place. Copying in lots of data without giving the cluster time to stabilize can cause it to go into safe mode, where it won't make changes until everything's properly distributed. It's version 0.20. Don't expect perfection just yet.
There are also experts out there who will be quite happy to help get things running better. My company has been using Hadoop quite successfully with their help.
Dells are cheaper than IBM, for the amount of performance they provide. Big Data is best handled in a large cluster, which provides reliability and parallelism to process a highly-scalable amount of data in a very short time. Beyond a certain point, high-end machines just raise the cost of failure, where low-end machines provide only marginally lower performance.
Be a nice professor. Make microwave popcorn during the exam for the students. Use an old low-power microwave, because I find it makes the popcorn taste just a bit better. If your class is large enough and hungry enough, you'll have no cheating (via wi-fi, at least) for the duration of the exam.
Meh. Recorded health information is already of such awful quality that it's practically useless without a long interview verifying major points.
So, in order to protect American soldiers, we shouldn't overreact to this and jump on the "OMG CENSORSHIP!" bandwagon. Instead, we ought to just say "it's not a big deal" and move on.
s/grant/guarantee/g
I guess you did miss it. As a quick review, acts of Congress become laws. The first amendment to the Constitution specifically limits the powers of Congress. The judicial branch of government is charged with determining if a given law is in line with the restrictions in the Constitution.
The relevant details are in the case Brandenburg v. Ohio, which held that the Espionage Act can indeed limit speech if that speech is likely to result in "imminent lawless action", the precise definition of which has not been declared. In fact, the Brandenburg case modifies the ruling of Schenck v. United States, which is where the common example of 'yelling "fire" in a crowded theater' comes from.
In short, it's not the act of placing others at risk that's the problem, but creating a situation that's likely to lead to illegal action. Releasing the names and details of intelligence officers puts them at risk of murder, which is illegal everywhere I know of. Sorry for the lack of clarity.
National security is why we have a government in the first place. It falls under the mandate to "insure domestic Tranquility, [and] provide for the common defence" that the Constitution was written for in the first place. I'm not saying it's a magic phrase that excuses heinous breaches of civil rights, but now and then there's a point to it.
If one copy will be seen by 100,000 people, then two copies leaking is 200,000, and that's twice the risk. The government screwed up (big surprise) by approving the book completely in the first place. Now they're trying to clean up their mistake. A comparison to 1984 is almost (but not quite) appropriate here. The government isn't trying to change recorded history, but merely to delay facts from coming out while they're still risky.
The publisher could do something underhanded, like deciding to only sell the books overseas, and would probably get a lot of love from the anti-government folks here. It'd also guarantee their next military-related book gets a much more thorough review. Is it really worth the trouble and risk, just so a few civilians can get a real name that they'll forget an hour later? I don't think so.
For that, you'll have to look at the Espionage Act, and its amendments in the Sedition act. In summary, it's illegal to aid groups the American government has declared to be enemies.
I'm afraid I don't remember the details involved (coincidence, I swear), but I seem to remember a biographical book a few years ago causing quite a ruckus, as it indicated that an elected official (in California, I think) had a relationship with a prostitute. It turned out to be false, but the guy's career was ruined. Similar stories abound for abortion clinic doctors. It seems having a real identity in the public spotlight is indeed a risk.
Relatedly, this is why so many fictional works have the disclaimer that "any relation to actual persons, living or dead, is entirely coincidental." Being responsible about others' lives does not have to get in the way of telling a story.