Okay, Macroeconomics 101. We'll start with the way things work now, then explain the difference, then discuss why fixing it won't work.
Currently, with fiat money, billions of dollars every day are "created" and "destroyed" through the fractional reserve banking system. Like virtual particles of matter and antimatter, there is an equal amount of debt created and destroyed as well. The net value of what the banks hold does not change, because their created debt and created assets cancel each other out. If a bank's brought in only $1 million in deposits, its net assets are only $1 billion, regardless of how much the bank has borrowed from the Federal Reserve Bank.
When someone (Acme, Inc.) wants to do something that they don't have money for (develop a new model of rocket-powered roller skate, for example), they can get a loan for it from a bank. Thanks to fractional reserve banking, that loan is practically unlimited in size, as long as two conditions are met:
The lending bank is willing to loan out that much
The lending bank has enough other loans out that the risk of this one loan defaulting is not a threat to the bank's stability
By creating a sufficient supply of pure cash, the desired transactions can take place - contracts are signed, parts are ordered, workers are paid, and paychecks are deposited in the bank, just in time to pay back that bank's loan from the Fed. In the grand scheme of things, Acme gave money directly into the workers' accounts, and now has a huge debt to fill. Fortunately, those workers will soon be working, parts will be delivered, the contracts fulfilled, and sales will bring in enough revenue for Acme to negate that "anti-money" debt.
Of course, with debt effectively being "anti-money", it can function just like money can... Contracts can be arranged for deferred payment, and Parts can be bought on credit, and workers' checks can be delayed until after the work is done. Debt works just like money, with the only difference being who gets to hold it while obligations are fulfilled.
Note that through this whole process, the only money involved that doesn't have a loan to counter it is what the eventual customers pay Acme, Inc.for the shiny new roller skates. That's perfectly fine, since money is just a tool for facilitating trade. Why mandate that money be attached to a shiny metal when it's not going to exist as anything more than a placeholder for a few days before being annihilated?
Whew.
Now let's contrast that with a fixed currency like a gold standard. Since all money must necessarily be tied to something real, there is no possibility for a large loan. Banks simply cannot loan out more than they've been given by depositors, and since they'll need to keep cash on hand to cover bank runs, their capacity for loans is effectively crippled. When Acme asks for a loan to cover the investment to make new roller skates, too bad. The money for a loan simply isn't available, and never will be. After all, if a depositor comes asking for their money and the bank can't provide it, the bank goes bankrupt, so their fraction of loan-able money is measured against the risk of a run.
Without the loan to Acme, the contracts aren't signed, parts aren't made, and workers aren't paid. All trade is effectively limited by how much money one company can gather at one time, and then how much it's willing to spend to make some product. That's a liquidity crisis.
Liquidity crises are bad. The majority of damage in 2008 and 2009 (which we're still feeling today) was caused by a liquidity crisis. With no trade, jobs are lost, people get worried, and they turn to the savings in the banks... raising the risk of a bank run, and therefore lowering the amount the banks will loan out, which damages trade even more.
So why not simply declare an ounce of gold to be worth $1,000,000, so the government could still make huge loans?
No matter how much gold is valued at, the economy's growth will always eventually outpace it. As
Still lazy, and this isn't really important anyway, but as I recall it was a few loans to banks that had gone bankrupt after getting the loan, or had turned their overnight loans into something longer that was still reported in the same figure... Special circumstances with so little money as not to be significant.
It's right there in the title of the graph: Log scale. GDP growth is exponential, while gold production is generally linear over the long term. No matter what value you peg gold at, the economy's need for liquid money will eventually surpass it.
Setting the value at ridiculously high values also impacts the value of existing stockpiles, until someone's old earring is enough to retire on. That's an enormous amount of short-term inflation, which means that everyone whose fortunes were not in gold would be relatively broke. The sociopolitical effects of such changes have been seen before, and are seldom peaceful.
It's worth noting that almost 100% of those overnight loans were repaid, as well. I'm too lazy to look up the details, but out of the trillions of total dollars loaned (a few billion at a time, as you said), a few hundred thousand dollars were outstanding last time I checked.
Having lived in Michigan in 2008, I can offer some insight into those million disappearing jobs. They're real. They're almost all in the northern Midwest, in the supply chains heading into Detroit. Machine shops in the area are commonly contracted out for a year at a time making a single set of parts for a car, with full expectation of getting a new contract when that one's filled. Around the massive part plants, other dependent industries grow to serve the hundreds of workers at the big plant - restaurants, grocery stores, strip malls... anything a worker would be likely to stop at on his way home from work.
While the idea that a single car company's reorganization would directly affect that many jobs is indeed absurd, the indirect jobs are just as dependent on Detroit's stability. One lost contract at a small plant can put a few thousand people out of work, and most of them aren't comfortably over the union's safety net. Those contracts aren't exactly easy to come by, either, so if a major manufacturer like GM or Chrysler has to back out of their contracts, it could easily be a few years before the plant could start up again.
Gold isn't feasible. There simply isn't enough of it available to back all the money necessary for the modern economy.
With a gold-backed currency, a billion-dollar deal needs a billion dollars' worth of gold. That's about 0.5% of the world's total supply of gold, currently valued at about $180 billion. For reference, the United States alone currently has a money supply of about $2 trillion, more than ten times the world's supply of gold. Since the amount of available gold is relatively fixed (growing slowly, but nowhere near as fast as the economy grows), the increasing demand for money far outpaces the supply of gold. That means the gold standard deflates, where by simply holding on to money its value increases. Since trade is unlikely to be as profitable as doing nothing, the whole economy grinds to a halt. That means no commerce, no jobs, and a massive recession. But at least you have a shiny chunk of metal!
Diamonds are just as bad, but differently. Diamonds aren't rare. They aren't even particularly uncommon, except for enormous ones. Rather, their scarcity is artificially maintained by DeBeers, whose existence depends on maintaining that scarcity. Heck, synthetic diamonds are cheaply available for industrial use, so any use of diamonds as money is effectively turning tools into cash overnight.
Specifically, I'm wondering which government program (or anybody else, for that matter) provided the funds for loan #2? I doubt it'd be the Federal Reserve Bank, which wouldn't lend out that much money in the first place (hence necessitating the bailouts) without more cash being in the JPM's reserves.
Apart from the normal everyday fractional-reserve loans from the Federal Reserve Bank, JPMorgan probably owes close to $0. I'm assuming you were referring to the $25 billion in bailout money, which was apparently paid in full.
No, I'm an experienced developer who's actually worked with NoSQL databases, while that's a retarded straw-man argument.
It's almost as retarded as whoever came up with this "web scale" buzzword in the first place. Unless you're as big as Facebook or Google, your
website probably doesn't need to use a NoSQL database. You're probably better off with a nice and easy RDBMS, where the tools are already built for
you and everything interfaces nicely. The whole Big Data approach likely isn't even really appropriate in the first place, because even collecting
massive amounts of data you probably don't have enough to make your answers statistically significant.
The primary feature of NoSQL databases for Big Data applications is that their write performance usually scales up linearly with the number of nodes in the cluster. In contrast, most RDBMSs either cannot scale write performance (being limited by the master server's throughput), scale more slowly (as bandwidth and processing demands increase exponentially), or scale unpredictably (as data distribution changes). While write performance is therefore almost unlimited for NoSQL, it comes at the expense of consistency because the bandwidth savings come from removing synchronization. With enough data, missing a few data points is an error below statistical significance, so the loss of full ACID doesn't matter. Again, if you're not working with enough data to actually max out your server, you probably don't need a NoSQL database in the first place.
Databases are tools, and as always you should choose the right tool for the job, through a thorough consideration of the strengths and weaknesses of each option. No, it's not as much fun as parroting buzzwords or spamming links to funny videos, but it's what actually gets the job done right.
And how are we measuring the size? What sizes are measured for typical 'big data'?
The last Big Data system I worked on was a new system. Our initial load pulled in a billion rows of data over two days. It used a few dozen terabytes, but again, that's only for a small new database.
Are we talking about detailed information, or inefficient data formats?
As much detail as possible. In the case of a web crawler, every header, parameter, and circumstance of a page visit. For a medical system, every nurse visit and every note recorded. For an insurance agency, that could include every mechanic visited, every recall, every ticket, and every oil change.
Are we talking about high-resolution long-term time series, or are we talking about data that is big because it has a complex structure?
Both, depending on the application. Generally, translating a complex structure into the key-value form preferred by the NoSQL data stores (which scale better for fast data gathering than most RDBMSs) is difficult, so the former is more common.
Is the data big because it has been engineered so, or is it begging for a more refined system to simplify?
The data is big because the system is explicitly not simplified. All of the source data is preserved and kept for later analysis. Where a traditional application might discard a user's mouse movements as trivial, a Big Data system could collect all the mouse movements (since cursors have been found to follow where users' eyes are looking), and analyze them to determine which features users spend the most time looking for. Those features could then be moved to a more obvious place in a future version.
"Big Data" is not so much a term for some particular threshold in a database's size, but rather an approach to problem-solving based on gathering as much information as possible, rather than just what the architect considers relevant at the time. It's caught the attention of business managers, because it allows developers to start gathering data before the managers have to nail down exactly what questions they want to ask. The data is all stored, and new questions just mean new algorithms for analyzing it.
Re:Big Data is the new place where magic happens
on
How Big Data Became So Big
·
· Score: 5, Interesting
No mod points, so I'll just post instead: You seem to be blissfully ignorant of what you're talking about.
Big Data isn't just gathering tons of data, then running it through the same old techniques on a big beefy cluster hoping that answers will magically fall out. Rather, it's a philosophy that's used throughout the architecture to process a more complete view of the relevant metrics that can lead to a more complete answer to the problem. If I'd only mentioned "empowering" and "synergy", that would be a sales pitch, so I'm just going to give an example from an old boss of mine.
A typical approach to a problem, such as determining the most popular cable TV show, might be to have each cable provider record every time they send a show to a subscriber. This is pretty simple to do, and generates only a few million total events each hour. That can easily be processed by a beefy server, and within a day or two the latest viewer counts for each show can be released. Now, it doesn't measure how many viewers turned off the show halfway through, or switched to another show on the commercials, or who watched the same channel for twelve hours because they left the cable box turned on. Those are just assumed to be innate errors that cannot be avoided.
Now, though, with the cheap availability of NoSQL data stores, widespread access to high-speed Internet access, and new "privacy-invading" TV sets, much more data can be gathered and processed, at a larger scale than ever before. Now, a suitably-equipped TV can send an event upstream for not just every show, but every minute watched, every commercial seen, every volume adjustment, and possibly even a guess of how many people are in the room. The sheer volume of data to be processed is about a thousand times greater, and coming in about a thousand times as fast, to boot.
The Big Data approach to the problem is to first absorb as much data as possible, then process it into a clear "big picture" view. That means dumping it all into a write-centric database like HBase or Cassandra, then running MapReduce jobs on the data to process it in chunks down to an intermediate level - such as groupings of statistics for each show. Those intermediate results can answer some direct questions about viewer counts or specific demographics, but not anything too much more complicated. Those results, though, are probably only a few hundred details for each show, which can easily be loaded into a traditional RDBMS and queried as needed.
In effect, the massively-parallel processing in the cluster can take the majority of work off of the RDBMS, so the RDBMS has just the answers, rather than the raw data. Those answers can then be retrieved faster than if the RDBMS has to process all of the raw data for every query.
Rather than dismissing errors of reality as unavoidable, a Big Data design relies on gathering more granular data, then distilling accurate answers out of that. For situations where there is enormous amounts of raw data available, this is often beneficial, because the improved accuracy means that some old impossible questions can now be answered. If enough data can't be easily collected (as in the case of so many small websites (almost anybody short of Facebook and Google)), Big Data is probably not the right approach.
I always seem to forget ZFS, which is a shame. It certainly seems magical, and far more reliable than plain RAID. I'm afraid I don't know enough about it, though, to determine if the same resilvering risks apply, or if there's enough additional layers of error correction to mitigate the risk of read errors.
As I write this, a question occurs to me: Does ZFS even do disk-at-once resilvering to recover from a failed disk? If not, what's it do?
Quite the contrary, and that's my point. The errors here aren't just "let's try again" failures. They're unrecoverable, final, data-is-gone-forever errors, and the chances of encountering one are very high with so much data. Resilvering such a large array is practically impossible (as described in the article I linked to). Without resilvering and having blocks spread among disks, losing one disk means you've lost a little bit of everything, so all your data is corrupt, rather than just the fraction that was stored on the failing drive. Add to that hassle the extra expense of having more disks, controllers, and setup time, and the submitter would be better off writing a few thousand DVDs.
What if you're in the middle of nowhere and it is simply some debris that rolls over in your path is detected as obstacle. Now you're on the highway doing 65 and your car suddenly slows down/stops for no reason and you get rear-ended.
That's the fault of the idiot tailgating you. If you're driving close enough to the car in front of you that you can't come to a full stop if he does, you're too damned close!
Also, where can I find this awesome magical computer vision & depth perception equipment that can constantly scan 300 or so feet (average stopping distance at highway speeds) and accurately (well lets say >90% confidence) identify any random object in the real world.
The hardware is available from most robot suppliers (being just a few LIDAR units and high-res cameras) and the software can be found within Google's well-secured vaults, being most likely an edge-detection algorithm applied to the pictures, then those shapes projected onto the 3D map from the LIDAR to identify objects, then a pattern-recognition engine to identify the object and its risk. If the object can't be identified with high confidence, assume it's dangerous and slow down until a more accurate assessment can be made.
As mentioned already, RAID is not a backup solution. While it will likely work fine for a while, the risk of a catastrophic failure rises as drive capacity increases. From the linked article:
With a twelve -terabyte array the chances of complete data loss during a resilver operation begin to approach one hundred percent - meaning that RAID 5 has no functionality whatsoever in that case. There is always a chance of survival, but it is very low.
Granted, this is talking about RAID 5, so let's naively assume that doubling the parity disks for RAID 6 will halve the risk... but then since we're trying to duplicate 24 terabytes instead of twelve, we can also assume the risk doubles again, and we're back to being practically guaranteed a failure.
Bottom line is that 24 terabytes is still a huge amount of data. There is no reliable solution I can think of for backing it all up that will be cheap. At that point, you're looking at file-level redundancy managed by a backup manager like Backup Exec (or whatever you prefer) with the data split across a dozen drives. As also mentioned already, the problem becomes much easier if you're able to reduce that volume of data somewhat.
Considering that the words "forty" and "fifty" are very similar, yes, I expect it was a slip of the tongue. Every president (and indeed, every other politician, executive, and other celebrity) has made just as many mistakes. They're human. It happens.
Do you expect the act of inauguration to somehow free a person from all distractions? Or does it magically provide an extra few hours a day to set aside for practicing upcoming speeches? Or is it just this particular person that's under examination, when the last president was just as bad?
It's a simple fact of life that people often mix up thoughts while speaking. It's so common that nobody notices everyday occurrences unless they're particularly egregious. When the speaker is a celebrity, though, suddenly it's a big deal that supposedly reflects negatively on the intelligence of the person. We all know perfectly well what the speaker intended, so why is a misspoken phrase so important?
Once everyone has their every waking moment recorded, let's see how many offhand comments, politically-incorrect statements, and slip-of-the-tongue mistakes you make. I bet it'll be more than 28 in four years.
Write a book. But please, I beseech you, stop sitting watching pointless crap on TV.
How about I write a screenplay? Maybe I'll even take that screenplay and some actors, and make a TV show. Maybe someone will watch it, and understand the artistic message that inspired my show. Maybe it will give someone hope that this difficult time in their life isn't that bad. Maybe it will convince someone to finally get outside and exercise. My art could change the world, if not for that jackass who thinks it's just pointless crap.
So I haven't yet tested it, but it sounds like a fancy interface to netstat, diff, and a wee bit of HijackThis thrown in for good measure. From the download site:
Some of the checks performed by the tool include analysis of changed or newly added files, registry keys, services, Microsoft ActiveX controls, listening ports and other parameters that affect a computer's attack surface.
The actual assessment of an attack surface is far more complex than any single system, and there's a heavy user-education component that no automated tool can test. While I'm sure this will have some use for admins who don't run firewalls or are under typically-asinine requirements to describe in detail the impact of a package, it looks more useful for ensuring programs actually uninstall completely.
You don't get it. The next presidential term will start in 2013, which is 21 years after 1992. This is obviously meant to commemorate the fact that the National Minimum Drinking Age act was passed two years after Obama turned 21, after it safely wouldn't affect him.
Don't you see? This is the Illuminati once again ensuring that its elite members can go out and party while us common folk get screwed, and then they have the balls to remind us all of how we're just complacent sheep!
Don't you see? Conspiracy theories make perfect sense if sanity isn't an issue for you!
Re:Windows 8 seems like a solid product
on
Windows 8 Is Ready
·
· Score: 1
...just like moving from... old Office to new Ribbon style Office, you will get used to it and then you see the better sides of things.
How long is that process supposed to take? Five years later, I still detest the ribbon interface, and still can't just explain to users where to find the function they're asking about. There's always the chance that something's moved, and knowing the folks I work with, that always seems to happen.
"It's on the view tab of the ribbon. No, up top. The toolbar thing. Click 'View'. Good. Now... Oh, it's blank? There's one button on the side? I'll just come down and take a look..."
I have no idea how can anyone trust them with their data.
The vast majority of the population either doesn't care about their data security, or doesn't know enough about Dropbox's shortcomings to be concerned. As for myself, my most recent use of dropbox was to synchronize work on a group project. We had one team at remote locations uploading data, and another two teams retrieving the data and processing it. All our data was practically worthless to anyone else, and not too private to us.
Dropbox operates very much like a real physical dropbox: You can stick stuff in it, and retrieve it later. Somebody else might break into it, so don't use it for vitally confidential stuff.
I recently was in Ghana, and saw the same thing... almost every bulb was a CFL. There were a few incandescent bulbs, but they were mostly in lamp fixtures on abandoned buildings.
Slashdot Mirror
Okay, Macroeconomics 101. We'll start with the way things work now, then explain the difference, then discuss why fixing it won't work.
Currently, with fiat money, billions of dollars every day are "created" and "destroyed" through the fractional reserve banking system. Like virtual particles of matter and antimatter, there is an equal amount of debt created and destroyed as well. The net value of what the banks hold does not change, because their created debt and created assets cancel each other out. If a bank's brought in only $1 million in deposits, its net assets are only $1 billion, regardless of how much the bank has borrowed from the Federal Reserve Bank.
When someone (Acme, Inc.) wants to do something that they don't have money for (develop a new model of rocket-powered roller skate, for example), they can get a loan for it from a bank. Thanks to fractional reserve banking, that loan is practically unlimited in size, as long as two conditions are met:
By creating a sufficient supply of pure cash, the desired transactions can take place - contracts are signed, parts are ordered, workers are paid, and paychecks are deposited in the bank, just in time to pay back that bank's loan from the Fed. In the grand scheme of things, Acme gave money directly into the workers' accounts, and now has a huge debt to fill. Fortunately, those workers will soon be working, parts will be delivered, the contracts fulfilled, and sales will bring in enough revenue for Acme to negate that "anti-money" debt.
Of course, with debt effectively being "anti-money", it can function just like money can... Contracts can be arranged for deferred payment, and Parts can be bought on credit, and workers' checks can be delayed until after the work is done. Debt works just like money, with the only difference being who gets to hold it while obligations are fulfilled.
Note that through this whole process, the only money involved that doesn't have a loan to counter it is what the eventual customers pay Acme, Inc.for the shiny new roller skates. That's perfectly fine, since money is just a tool for facilitating trade. Why mandate that money be attached to a shiny metal when it's not going to exist as anything more than a placeholder for a few days before being annihilated?
Whew.
Now let's contrast that with a fixed currency like a gold standard. Since all money must necessarily be tied to something real, there is no possibility for a large loan. Banks simply cannot loan out more than they've been given by depositors, and since they'll need to keep cash on hand to cover bank runs, their capacity for loans is effectively crippled. When Acme asks for a loan to cover the investment to make new roller skates, too bad. The money for a loan simply isn't available, and never will be. After all, if a depositor comes asking for their money and the bank can't provide it, the bank goes bankrupt, so their fraction of loan-able money is measured against the risk of a run.
Without the loan to Acme, the contracts aren't signed, parts aren't made, and workers aren't paid. All trade is effectively limited by how much money one company can gather at one time, and then how much it's willing to spend to make some product. That's a liquidity crisis.
Liquidity crises are bad. The majority of damage in 2008 and 2009 (which we're still feeling today) was caused by a liquidity crisis. With no trade, jobs are lost, people get worried, and they turn to the savings in the banks... raising the risk of a bank run, and therefore lowering the amount the banks will loan out, which damages trade even more.
So why not simply declare an ounce of gold to be worth $1,000,000, so the government could still make huge loans?
No matter how much gold is valued at, the economy's growth will always eventually outpace it. As
Still lazy, and this isn't really important anyway, but as I recall it was a few loans to banks that had gone bankrupt after getting the loan, or had turned their overnight loans into something longer that was still reported in the same figure... Special circumstances with so little money as not to be significant.
It's right there in the title of the graph: Log scale. GDP growth is exponential, while gold production is generally linear over the long term. No matter what value you peg gold at, the economy's need for liquid money will eventually surpass it.
Setting the value at ridiculously high values also impacts the value of existing stockpiles, until someone's old earring is enough to retire on. That's an enormous amount of short-term inflation, which means that everyone whose fortunes were not in gold would be relatively broke. The sociopolitical effects of such changes have been seen before, and are seldom peaceful.
It's worth noting that almost 100% of those overnight loans were repaid, as well. I'm too lazy to look up the details, but out of the trillions of total dollars loaned (a few billion at a time, as you said), a few hundred thousand dollars were outstanding last time I checked.
Having lived in Michigan in 2008, I can offer some insight into those million disappearing jobs. They're real. They're almost all in the northern Midwest, in the supply chains heading into Detroit. Machine shops in the area are commonly contracted out for a year at a time making a single set of parts for a car, with full expectation of getting a new contract when that one's filled. Around the massive part plants, other dependent industries grow to serve the hundreds of workers at the big plant - restaurants, grocery stores, strip malls... anything a worker would be likely to stop at on his way home from work.
While the idea that a single car company's reorganization would directly affect that many jobs is indeed absurd, the indirect jobs are just as dependent on Detroit's stability. One lost contract at a small plant can put a few thousand people out of work, and most of them aren't comfortably over the union's safety net. Those contracts aren't exactly easy to come by, either, so if a major manufacturer like GM or Chrysler has to back out of their contracts, it could easily be a few years before the plant could start up again.
Gold isn't feasible. There simply isn't enough of it available to back all the money necessary for the modern economy.
With a gold-backed currency, a billion-dollar deal needs a billion dollars' worth of gold. That's about 0.5% of the world's total supply of gold, currently valued at about $180 billion. For reference, the United States alone currently has a money supply of about $2 trillion, more than ten times the world's supply of gold. Since the amount of available gold is relatively fixed (growing slowly, but nowhere near as fast as the economy grows), the increasing demand for money far outpaces the supply of gold. That means the gold standard deflates, where by simply holding on to money its value increases. Since trade is unlikely to be as profitable as doing nothing, the whole economy grinds to a halt. That means no commerce, no jobs, and a massive recession. But at least you have a shiny chunk of metal!
Diamonds are just as bad, but differently. Diamonds aren't rare. They aren't even particularly uncommon, except for enormous ones. Rather, their scarcity is artificially maintained by DeBeers, whose existence depends on maintaining that scarcity. Heck, synthetic diamonds are cheaply available for industrial use, so any use of diamonds as money is effectively turning tools into cash overnight.
[citation needed]
Specifically, I'm wondering which government program (or anybody else, for that matter) provided the funds for loan #2? I doubt it'd be the Federal Reserve Bank, which wouldn't lend out that much money in the first place (hence necessitating the bailouts) without more cash being in the JPM's reserves.
In fact, JPM's corporate debt is at its lowest since the bailout, regardless of the source.
That's okay... you can feel free to keep ranting about politics if it makes you feel better.
Apart from the normal everyday fractional-reserve loans from the Federal Reserve Bank, JPMorgan probably owes close to $0. I'm assuming you were referring to the $25 billion in bailout money, which was apparently paid in full.
No, I'm an experienced developer who's actually worked with NoSQL databases, while that's a retarded straw-man argument.
It's almost as retarded as whoever came up with this "web scale" buzzword in the first place. Unless you're as big as Facebook or Google, your website probably doesn't need to use a NoSQL database. You're probably better off with a nice and easy RDBMS, where the tools are already built for you and everything interfaces nicely. The whole Big Data approach likely isn't even really appropriate in the first place, because even collecting massive amounts of data you probably don't have enough to make your answers statistically significant.
The primary feature of NoSQL databases for Big Data applications is that their write performance usually scales up linearly with the number of nodes in the cluster. In contrast, most RDBMSs either cannot scale write performance (being limited by the master server's throughput), scale more slowly (as bandwidth and processing demands increase exponentially), or scale unpredictably (as data distribution changes). While write performance is therefore almost unlimited for NoSQL, it comes at the expense of consistency because the bandwidth savings come from removing synchronization. With enough data, missing a few data points is an error below statistical significance, so the loss of full ACID doesn't matter. Again, if you're not working with enough data to actually max out your server, you probably don't need a NoSQL database in the first place.
Databases are tools, and as always you should choose the right tool for the job, through a thorough consideration of the strengths and weaknesses of each option. No, it's not as much fun as parroting buzzwords or spamming links to funny videos, but it's what actually gets the job done right.
The data is as big as it can be.
And how are we measuring the size? What sizes are measured for typical 'big data'?
The last Big Data system I worked on was a new system. Our initial load pulled in a billion rows of data over two days. It used a few dozen terabytes, but again, that's only for a small new database.
Are we talking about detailed information, or inefficient data formats?
As much detail as possible. In the case of a web crawler, every header, parameter, and circumstance of a page visit. For a medical system, every nurse visit and every note recorded. For an insurance agency, that could include every mechanic visited, every recall, every ticket, and every oil change.
Are we talking about high-resolution long-term time series, or are we talking about data that is big because it has a complex structure?
Both, depending on the application. Generally, translating a complex structure into the key-value form preferred by the NoSQL data stores (which scale better for fast data gathering than most RDBMSs) is difficult, so the former is more common.
Is the data big because it has been engineered so, or is it begging for a more refined system to simplify?
The data is big because the system is explicitly not simplified. All of the source data is preserved and kept for later analysis. Where a traditional application might discard a user's mouse movements as trivial, a Big Data system could collect all the mouse movements (since cursors have been found to follow where users' eyes are looking), and analyze them to determine which features users spend the most time looking for. Those features could then be moved to a more obvious place in a future version.
"Big Data" is not so much a term for some particular threshold in a database's size, but rather an approach to problem-solving based on gathering as much information as possible, rather than just what the architect considers relevant at the time. It's caught the attention of business managers, because it allows developers to start gathering data before the managers have to nail down exactly what questions they want to ask. The data is all stored, and new questions just mean new algorithms for analyzing it.
No mod points, so I'll just post instead: You seem to be blissfully ignorant of what you're talking about.
Big Data isn't just gathering tons of data, then running it through the same old techniques on a big beefy cluster hoping that answers will magically fall out. Rather, it's a philosophy that's used throughout the architecture to process a more complete view of the relevant metrics that can lead to a more complete answer to the problem. If I'd only mentioned "empowering" and "synergy", that would be a sales pitch, so I'm just going to give an example from an old boss of mine.
A typical approach to a problem, such as determining the most popular cable TV show, might be to have each cable provider record every time they send a show to a subscriber. This is pretty simple to do, and generates only a few million total events each hour. That can easily be processed by a beefy server, and within a day or two the latest viewer counts for each show can be released. Now, it doesn't measure how many viewers turned off the show halfway through, or switched to another show on the commercials, or who watched the same channel for twelve hours because they left the cable box turned on. Those are just assumed to be innate errors that cannot be avoided.
Now, though, with the cheap availability of NoSQL data stores, widespread access to high-speed Internet access, and new "privacy-invading" TV sets, much more data can be gathered and processed, at a larger scale than ever before. Now, a suitably-equipped TV can send an event upstream for not just every show, but every minute watched, every commercial seen, every volume adjustment, and possibly even a guess of how many people are in the room. The sheer volume of data to be processed is about a thousand times greater, and coming in about a thousand times as fast, to boot.
The Big Data approach to the problem is to first absorb as much data as possible, then process it into a clear "big picture" view. That means dumping it all into a write-centric database like HBase or Cassandra, then running MapReduce jobs on the data to process it in chunks down to an intermediate level - such as groupings of statistics for each show. Those intermediate results can answer some direct questions about viewer counts or specific demographics, but not anything too much more complicated. Those results, though, are probably only a few hundred details for each show, which can easily be loaded into a traditional RDBMS and queried as needed.
In effect, the massively-parallel processing in the cluster can take the majority of work off of the RDBMS, so the RDBMS has just the answers, rather than the raw data. Those answers can then be retrieved faster than if the RDBMS has to process all of the raw data for every query.
Rather than dismissing errors of reality as unavoidable, a Big Data design relies on gathering more granular data, then distilling accurate answers out of that. For situations where there is enormous amounts of raw data available, this is often beneficial, because the improved accuracy means that some old impossible questions can now be answered. If enough data can't be easily collected (as in the case of so many small websites (almost anybody short of Facebook and Google)), Big Data is probably not the right approach.
I always seem to forget ZFS, which is a shame. It certainly seems magical, and far more reliable than plain RAID. I'm afraid I don't know enough about it, though, to determine if the same resilvering risks apply, or if there's enough additional layers of error correction to mitigate the risk of read errors.
As I write this, a question occurs to me: Does ZFS even do disk-at-once resilvering to recover from a failed disk? If not, what's it do?
Quite the contrary, and that's my point. The errors here aren't just "let's try again" failures. They're unrecoverable, final, data-is-gone-forever errors, and the chances of encountering one are very high with so much data. Resilvering such a large array is practically impossible (as described in the article I linked to). Without resilvering and having blocks spread among disks, losing one disk means you've lost a little bit of everything, so all your data is corrupt, rather than just the fraction that was stored on the failing drive. Add to that hassle the extra expense of having more disks, controllers, and setup time, and the submitter would be better off writing a few thousand DVDs.
What if you're in the middle of nowhere and it is simply some debris that rolls over in your path is detected as obstacle. Now you're on the highway doing 65 and your car suddenly slows down/stops for no reason and you get rear-ended.
That's the fault of the idiot tailgating you. If you're driving close enough to the car in front of you that you can't come to a full stop if he does, you're too damned close!
Also, where can I find this awesome magical computer vision & depth perception equipment that can constantly scan 300 or so feet (average stopping distance at highway speeds) and accurately (well lets say >90% confidence) identify any random object in the real world.
The hardware is available from most robot suppliers (being just a few LIDAR units and high-res cameras) and the software can be found within Google's well-secured vaults, being most likely an edge-detection algorithm applied to the pictures, then those shapes projected onto the 3D map from the LIDAR to identify objects, then a pattern-recognition engine to identify the object and its risk. If the object can't be identified with high confidence, assume it's dangerous and slow down until a more accurate assessment can be made.
As mentioned already, RAID is not a backup solution. While it will likely work fine for a while, the risk of a catastrophic failure rises as drive capacity increases. From the linked article:
With a twelve -terabyte array the chances of complete data loss during a resilver operation begin to approach one hundred percent - meaning that RAID 5 has no functionality whatsoever in that case. There is always a chance of survival, but it is very low.
Granted, this is talking about RAID 5, so let's naively assume that doubling the parity disks for RAID 6 will halve the risk... but then since we're trying to duplicate 24 terabytes instead of twelve, we can also assume the risk doubles again, and we're back to being practically guaranteed a failure.
Bottom line is that 24 terabytes is still a huge amount of data. There is no reliable solution I can think of for backing it all up that will be cheap. At that point, you're looking at file-level redundancy managed by a backup manager like Backup Exec (or whatever you prefer) with the data split across a dozen drives. As also mentioned already, the problem becomes much easier if you're able to reduce that volume of data somewhat.
Considering that the words "forty" and "fifty" are very similar, yes, I expect it was a slip of the tongue. Every president (and indeed, every other politician, executive, and other celebrity) has made just as many mistakes. They're human. It happens.
Do you expect the act of inauguration to somehow free a person from all distractions? Or does it magically provide an extra few hours a day to set aside for practicing upcoming speeches? Or is it just this particular person that's under examination, when the last president was just as bad?
It's a simple fact of life that people often mix up thoughts while speaking. It's so common that nobody notices everyday occurrences unless they're particularly egregious. When the speaker is a celebrity, though, suddenly it's a big deal that supposedly reflects negatively on the intelligence of the person. We all know perfectly well what the speaker intended, so why is a misspoken phrase so important?
Remarkably on-topic.
Once everyone has their every waking moment recorded, let's see how many offhand comments, politically-incorrect statements, and slip-of-the-tongue mistakes you make. I bet it'll be more than 28 in four years.
Write a book. But please, I beseech you, stop sitting watching pointless crap on TV.
How about I write a screenplay? Maybe I'll even take that screenplay and some actors, and make a TV show. Maybe someone will watch it, and understand the artistic message that inspired my show. Maybe it will give someone hope that this difficult time in their life isn't that bad. Maybe it will convince someone to finally get outside and exercise. My art could change the world, if not for that jackass who thinks it's just pointless crap.
Fair point. A few cruise missiles will automatically expose a significant denial-of-service attack surface.
So I haven't yet tested it, but it sounds like a fancy interface to netstat, diff, and a wee bit of HijackThis thrown in for good measure. From the download site:
Some of the checks performed by the tool include analysis of changed or newly added files, registry keys, services, Microsoft ActiveX controls, listening ports and other parameters that affect a computer's attack surface.
The actual assessment of an attack surface is far more complex than any single system, and there's a heavy user-education component that no automated tool can test. While I'm sure this will have some use for admins who don't run firewalls or are under typically-asinine requirements to describe in detail the impact of a package, it looks more useful for ensuring programs actually uninstall completely.
You don't get it. The next presidential term will start in 2013, which is 21 years after 1992. This is obviously meant to commemorate the fact that the National Minimum Drinking Age act was passed two years after Obama turned 21, after it safely wouldn't affect him.
Don't you see? This is the Illuminati once again ensuring that its elite members can go out and party while us common folk get screwed, and then they have the balls to remind us all of how we're just complacent sheep!
Don't you see? Conspiracy theories make perfect sense if sanity isn't an issue for you!
...just like moving from ... old Office to new Ribbon style Office, you will get used to it and then you see the better sides of things.
How long is that process supposed to take? Five years later, I still detest the ribbon interface, and still can't just explain to users where to find the function they're asking about. There's always the chance that something's moved, and knowing the folks I work with, that always seems to happen.
"It's on the view tab of the ribbon. No, up top. The toolbar thing. Click 'View'. Good. Now... Oh, it's blank? There's one button on the side? I'll just come down and take a look..."
I have no idea how can anyone trust them with their data.
The vast majority of the population either doesn't care about their data security, or doesn't know enough about Dropbox's shortcomings to be concerned. As for myself, my most recent use of dropbox was to synchronize work on a group project. We had one team at remote locations uploading data, and another two teams retrieving the data and processing it. All our data was practically worthless to anyone else, and not too private to us.
Dropbox operates very much like a real physical dropbox: You can stick stuff in it, and retrieve it later. Somebody else might break into it, so don't use it for vitally confidential stuff.
I recently was in Ghana, and saw the same thing... almost every bulb was a CFL. There were a few incandescent bulbs, but they were mostly in lamp fixtures on abandoned buildings.
Obvious astroturfer only managed to make me terrified of ever working with the beast...