Than engines designed for massive parallelism in dealing with workloads which can be effectually processed in parallel.
Operating on what hypothetical database schema with how many records spread across how many tables?
Generally NoSQL engines use schema on read techniques not schema on write. The table structure comes during the read. To get some sort of fair comparison something like a typical star schema with a much too large fact table (think billions or trillions of rows) and a half dozen dimension tables.
Or if you really want to make it worse. The same query where the table is getting 1m writes / second and you want an accurate stream.
SQL engines have problems with massive parallelism? Why? Which ones?
Because SQL by its nature operates on the table not the individual rows. Older database technologies that were row oriented like what you see on a mainframe on in SaS work better when the ratio of table size to computation speed is low. Today because disk storage size per dollar has gone up so fast, we disk we face many of the same problems systems in the 1980s faced with tape.
And the next question is pretty much all of them. The big data SQL engines have the least problems though and via. their execution plans turning into map-reduces might present a viable long term solution.
How well do you *really* know SQL in general and the capabilities of different database engines in particular?
Assume I don't know anything. Oracle, which has the best engine and SQL people on the planet has a guide for hybridization to handle things their engine can't handle well. IBM which probably comes in second and invented the relational database produces their own Hadoop / R to handle queries that DB2 (which is BTW far better than Oracle at stream) can't handle. Teradata's engine which was originally written specifically for larger amounts of data for a decade has had specific features of another subsystem to do enhanced big data, they also have guides for hybridization for things even their enhanced engine can't handle And Microsoft which writes the 3rd most popular engine has spent many millions on hybridization strategies. Enterprise DB (postgres) fully supports the IBM strategy.
I don't know anyone in the space who does agree with the/. "SQL can do everything" attitude.
but that portion off the article was ridiculous, and thus far all of the comments in support of it have demonstrated a similar lack of familiarity with actual databases, their operation, or performance tuning.
The article was ridiculous I said as much in another response. However the comment I was responding to went much too far in the other direction. As for performance tuning -- performance tuning is designed to avoid full table scans and expensive joins. To goal of many hybridization strategies is to take a raw data flow and convert it into a relational ETL using a big data engine which can take advantage of indexing and a better execution plan. It doesn't do much good when the initial goal is to do a full table scan.
I know SQL pretty well. I agree with you it handles most stuff. That doesn't mean it handles everything.
SQL engines are often slower. SQL engines have problems with massive parallelism (i.e often at around 12 CPUs you stop gaining much at all by adding addition CPU). SQL engines have problems with complex in document (i.e. in blob) searches etc...
I'm a big data advocate. I like the idea of engines designed for unstructured data. But the two examples in the article barely even register as difficulties of relational databases, "What if two people share the same address but not the same account? What if you want to have three lines to the address instead of two? Who hasn’t tried to fix a relational database by shoehorning too much data into a single column? Or else you end up adding yet another column, and the table grows unbounded.".
As for his comments on denormalizing, I'm wondering if he has ever head of a data warehouse and a star / snowflake schema both of which handle the "I want cheaper joins" problem without having to denormalize the dimension tables.
Polls are having trouble determining which candidate is going to win. They are having no trouble determining the distinction between candidates with no change, irrelevant alternatives, and viable contenders. Polling isn't the reason people aren't voting for 3rd parties.
Turnout matters a lot. Talking about what what they really believe on most issues is likely to assist their opponent's turnout without doing much for their own. Terrible strategy.
The department of defense runs servers out of house. Lockheed Martin runs a cloud provider. Many of the country's banks handle it. There is no question you can buy better security than any company has internally.
As for running an internal cloud that's pretty easy and they could ask a vendor to run the financial it while keeping all the servers physically on their prem.
You don't have a label (except the one you own) and you have never released it. Unless your self-owned label signed a distribution agreement Apple won't have access at all.
One way to handle that is to not own your infrastructure and just rent month to month from the vendor who provides a pool of servers. What you are likely facing is the problem of how to prevent the administrative cost from going above X% by preventing the IT administrative cost from going about Y% by slowing down acquisitions... Better yet is just to guarantee Y and save the labor.
At this point for almost all companies good quality colo space is infinite. Most times a company isn't even using a meaningful fraction of their colo's space and so they could double or triple instantly without hassle much less an extra 33%. And even if their colo doesn't other's direct connected to it do have extra space... So consider space infinite once you are willing to rent.
That being said, I have problems believing the 1/3rd of severs figures from the article. That's not my experience at all.
It can in better type systems and those exist today (example "safe division via. the Maybe monad") You can have a NAN added to the integers ( but then your integral math code can't execute directly on the arithmetic logic unit in the CPU, it becomes an abstract type. For higher level languages that often won't matter anyway.
The issue is your type system sucks. What you want are types that guarantee they are non-zero.
So what you want is a type "divisor double" or "divisor integer" where putting a 0 in such a variable generates a type error.
A more sane approach to ignoring the type avoid the error checking you want would be to have this defined using the maybe Monad. Then you could just define
div' takes 2 numbers of types A and returns a Maybe A where div' x y = if y != 0
then Just (div x y)
else Nothing
Programs then automatically lift if f takes type B to type C then f' from Maybe B to Maybe C becomes f' Just (b) = Just (f b) f' Nothing = Nothing
etc... Then you get rid of the division by 0 when it makes sense. If you want to cast Nothing to 0 (which I think is dangerous) do it explicitly at some point where you don't want a Maybe object but insist on a base object. You can't change math, but at least it makes the call explicit.
I like the fact that we have unified ecosystem. Stuff breaking means forced upgrades and allows developers to know that everyone is on the latest version of everything. The cost of the upgrades are low because it helps everyone to be upgrading and we get features far faster than the Windows eco system where the time between first availability of a feature in the OS and every application depending on it can be 20 years.
Apple expects you to upgrade. Part of what makes a unified ecosystem is that everyone is essentially on the same version of everything. They are pretty open about that and its been policy for a long time. 3 years of studio recordings were not lost btw Pro Tools has a version which supports the current operating system.
Just as you need glue code to tie together libraries to make an application you need glue code to tie together SaaS services to make a unified service. The libraries now are just SaaS APIs.
Well that's sort of the situation that accidentally exist on iOS and even on that platform there has been some demand for different engines. There are tradeoffs between engines. Say for example an engine has a bug. Do you continue to process data in line with that bug for application compatibility or fix it? Should the engine focus on newest features or on standards compliance? Should it focus on security vs. compatibility? Should it focus on minimizing time to render or minimizing power drain?
True for consumer. But I think server is a counter example. In RMS's world end users are consumers of some technology and producers of others. They are technologically interested. My getting a sink diagram does nothing for me since I still would rather just pay a plumber.
So for example RMS AFAIK has never once written about all the exciting stuff going on with open source networking in the last decade+. But it is a wonderful example of where his vision has been achieved. At this point the standard has become that networking technologies are implemented and start maturing in the open source community as open code running on open operating systems supporting mostly open source applications. And only after are those components reimplemented in semi-closed, semi-hardware (RMS has never been as concerned about software implemented in hardware because it isn't IP in the same way) solutions for ultra high performance.
So anyway. I think we agree on the fact. But I think RMS achieved far more success than he originally intended for the community he was interested in. Free Software dominates the technical community and the software they use and manage. Consumers don't care about the 4 freedoms and there...
What's going to be the interesting battle over the next 20 years is the Internet of Things. How open is the software running on all sorts of devices that don't have traditional computer interfaces? So far the tendency is towards more openness than has previously existed as Free Software / Open Source is becoming the standard for devices.
So? What gives the USA govt the right to punish (meaningfully or otherwise) a French company?
You were arguing that this French company could be meaningfully regulated. If you don't want governments punishing them, then they can't regulate them on their shore or off.
As long as those dozens of subsidiaries follow USA law, then there's no connection to the specific subsidiary (Orange-USA) that is breaking the law and being punished.
And if they aren't following the law then what? They just shift assets between them everytime one of them gets caught.
But when have you ever seen it not occur?
Everyday. I work in telco. Regulators cross borders and revolve disputes all the time. Same with airline industry who handle new problems. Shipping and packaging. Oil shipping and refining (though extraction often gets quite violent). It happens most of the time.
. So governments are in a position where they either have to give up their sovereignty (well technically try to convince other nations to give up their sovereignty, but that certainly won't be a one-way street,) or give up their ability to censor random things that they don't like
Or create an effectual system of global regulation like what happens for most industries.
No, they can't. The USA government has no more control over Google France than the French government has over Google USA
They have full authority over the parent company. They could order Google USA to shut it down. Same power they have over any other owner of foreign stuff that commits international crimes. Their are plenty of Americans in American prisons for things they did abroad.
nobody would want to instigate a whole treaty negotiation process to deal with a single company's policies
I'm not saying a single company I'm saying all multi-nationals. And then another specific one having to do with multi-national internet businesses.
I agree the WTO is a good place as well to deal with this. But that is a new power and so likely requires at least an agreement.
And why would the US government need to go through two (like long and legally intensive) battles in order to regulate shit that happens on American soil?
Because they don't have a choice if we are going to have foreign corporations. Orange-USA doesn't really exist. One of the core powers that gives the government the ability to regulate is the ability to punish. The USA government can't meaningfully punish Orange by punishing Orange-USA because Orange can create dozens of subsidiaries or buy shares in other companies and move resources and assets between them. Orange is the "immortal person" the USA subsidiary companies are like their clothes easily changed. So for the USA government to be able to regulate they need the help of France. Which means that this negotiation has to take place
As for long legally intensive battles... Why does that need to occur?
Google already has (or at least has threatened) to do exactly that in several countries already when they couldn't come to terms with local laws
Not really. Because the people in those countries continue to use Google services. Google pulling out is mostly a technical rearrangement. It isn't them meaningfully completely pulling out. And so the countries often can't accomplish their objective.
Now I happen to like how Google is handling corrupt EU regulations but Google is a perfect case in point of the futility of trying to regulate international corporations in each country. The USA government is the government that can effectually regulate Google. If Germany or France or Spain wants a regulate the USA government is who they should be negotiating with, if they want their regulation to actually happen.
Then under that interpretation mobile has proven what is mostly true elsewhere. Guys who run servers like the 4 freedoms and take advantage of them, developers take advantage of them while end user applications mostly consist of a mixture of unfree code with free under layers. Which means that the 4 freedoms are valuable freedoms for computer professionals and mostly worthless for everyone else.
The one disadvantage about quiet leadership is that you will much less talked and written about.
I suspect Osama bin Laden who was well known for his quiet, respectful and thoughtful conversation style (I'm not kidding, his policies and his personality don't match) might beg to differ with that. Much as I like Jobs and Torvalds I suspect 200 years from now they may be mostly forgotten while bin Laden is still remembered for mainstreaming Qutbism. Heck our likely next president, Mrs. Clinton, is rather introverted and tends to quietly guide the people close to her.
For enterprise Apple sells (well below cost) the server components for iOS. Companies can setup their own applications stores and distribute applications through those with no 30% going to Apple. The 100 limit the GP mentioned applies to individual devices. If the devices register as belonging to X company X company can distribute applications through their infrastructure to all of them. Apple governs their app store. They make it easy to setup up alternatives.
FOSS is shinning on mobile. Virtually every cloud that is being used to support those apps is 95% FOSS or more. Many of the frameworks are FOSS. The mobile operating systems themselves are about 80% FOSS.
Non technical end user tweaking is an area where FOSS still gets its butt kicked.
Slashdot Mirror
Teradata, Vertica ... use hybridization and big data strategies inside their engines. They don't refute the point, they prove it.
Than engines designed for massive parallelism in dealing with workloads which can be effectually processed in parallel.
Generally NoSQL engines use schema on read techniques not schema on write. The table structure comes during the read. To get some sort of fair comparison something like a typical star schema with a much too large fact table (think billions or trillions of rows) and a half dozen dimension tables.
Or if you really want to make it worse. The same query where the table is getting 1m writes / second and you want an accurate stream.
Because SQL by its nature operates on the table not the individual rows. Older database technologies that were row oriented like what you see on a mainframe on in SaS work better when the ratio of table size to computation speed is low. Today because disk storage size per dollar has gone up so fast, we disk we face many of the same problems systems in the 1980s faced with tape.
And the next question is pretty much all of them. The big data SQL engines have the least problems though and via. their execution plans turning into map-reduces might present a viable long term solution.
Assume I don't know anything. Oracle, which has the best engine and SQL people on the planet has a guide for hybridization to handle things their engine can't handle well. IBM which probably comes in second and invented the relational database produces their own Hadoop / R to handle queries that DB2 (which is BTW far better than Oracle at stream) can't handle. Teradata's engine which was originally written specifically for larger amounts of data for a decade has had specific features of another subsystem to do enhanced big data, they also have guides for hybridization for things even their enhanced engine can't handle And Microsoft which writes the 3rd most popular engine has spent many millions on hybridization strategies. Enterprise DB (postgres) fully supports the IBM strategy.
I don't know anyone in the space who does agree with the /. "SQL can do everything" attitude.
The article was ridiculous I said as much in another response. However the comment I was responding to went much too far in the other direction. As for performance tuning -- performance tuning is designed to avoid full table scans and expensive joins. To goal of many hybridization strategies is to take a raw data flow and convert it into a relational ETL using a big data engine which can take advantage of indexing and a better execution plan. It doesn't do much good when the initial goal is to do a full table scan.
I know SQL pretty well. I agree with you it handles most stuff. That doesn't mean it handles everything.
SQL engines are often slower.
SQL engines have problems with massive parallelism (i.e often at around 12 CPUs you stop gaining much at all by adding addition CPU).
SQL engines have problems with complex in document (i.e. in blob) searches
etc...
I'm a big data advocate. I like the idea of engines designed for unstructured data. But the two examples in the article barely even register as difficulties of relational databases, "What if two people share the same address but not the same account? What if you want to have three lines to the address instead of two? Who hasn’t tried to fix a relational database by shoehorning too much data into a single column? Or else you end up adding yet another column, and the table grows unbounded.".
As for his comments on denormalizing, I'm wondering if he has ever head of a data warehouse and a star / snowflake schema both of which handle the "I want cheaper joins" problem without having to denormalize the dimension tables.
Polls are having trouble determining which candidate is going to win. They are having no trouble determining the distinction between candidates with no change, irrelevant alternatives, and viable contenders. Polling isn't the reason people aren't voting for 3rd parties.
Turnout matters a lot. Talking about what what they really believe on most issues is likely to assist their opponent's turnout without doing much for their own. Terrible strategy.
The department of defense runs servers out of house. Lockheed Martin runs a cloud provider. Many of the country's banks handle it. There is no question you can buy better security than any company has internally.
As for running an internal cloud that's pretty easy and they could ask a vendor to run the financial it while keeping all the servers physically on their prem.
You don't have a label (except the one you own) and you have never released it. Unless your self-owned label signed a distribution agreement Apple won't have access at all.
One way to handle that is to not own your infrastructure and just rent month to month from the vendor who provides a pool of servers. What you are likely facing is the problem of how to prevent the administrative cost from going above X% by preventing the IT administrative cost from going about Y% by slowing down acquisitions... Better yet is just to guarantee Y and save the labor.
At this point for almost all companies good quality colo space is infinite. Most times a company isn't even using a meaningful fraction of their colo's space and so they could double or triple instantly without hassle much less an extra 33%. And even if their colo doesn't other's direct connected to it do have extra space... So consider space infinite once you are willing to rent.
That being said, I have problems believing the 1/3rd of severs figures from the article. That's not my experience at all.
What's wrong is that 0*infinity is undefined the same as 0/0. It has no single value as your example shows.
It can in better type systems and those exist today (example "safe division via. the Maybe monad") You can have a NAN added to the integers ( but then your integral math code can't execute directly on the arithmetic logic unit in the CPU, it becomes an abstract type. For higher level languages that often won't matter anyway.
The issue is your type system sucks. What you want are types that guarantee they are non-zero.
So what you want is a type "divisor double" or "divisor integer" where putting a 0 in such a variable generates a type error.
A more sane approach to ignoring the type avoid the error checking you want would be to have this defined using the maybe Monad. Then you could just define
div' takes 2 numbers of types A and returns a Maybe A where
div' x y = if y != 0
then Just (div x y)
else Nothing
Programs then automatically lift
if f takes type B to type C then
f' from Maybe B to Maybe C becomes
f' Just (b) = Just (f b)
f' Nothing = Nothing
etc... Then you get rid of the division by 0 when it makes sense. If you want to cast Nothing to 0 (which I think is dangerous) do it explicitly at some point where you don't want a Maybe object but insist on a base object. You can't change math, but at least it makes the call explicit.
I like the fact that we have unified ecosystem. Stuff breaking means forced upgrades and allows developers to know that everyone is on the latest version of everything. The cost of the upgrades are low because it helps everyone to be upgrading and we get features far faster than the Windows eco system where the time between first availability of a feature in the OS and every application depending on it can be 20 years.
Apple expects you to upgrade. Part of what makes a unified ecosystem is that everyone is essentially on the same version of everything. They are pretty open about that and its been policy for a long time. 3 years of studio recordings were not lost btw Pro Tools has a version which supports the current operating system.
Just as you need glue code to tie together libraries to make an application you need glue code to tie together SaaS services to make a unified service. The libraries now are just SaaS APIs.
Well that's sort of the situation that accidentally exist on iOS and even on that platform there has been some demand for different engines. There are tradeoffs between engines. Say for example an engine has a bug. Do you continue to process data in line with that bug for application compatibility or fix it? Should the engine focus on newest features or on standards compliance? Should it focus on security vs. compatibility? Should it focus on minimizing time to render or minimizing power drain?
True for consumer. But I think server is a counter example. In RMS's world end users are consumers of some technology and producers of others. They are technologically interested. My getting a sink diagram does nothing for me since I still would rather just pay a plumber.
So for example RMS AFAIK has never once written about all the exciting stuff going on with open source networking in the last decade+. But it is a wonderful example of where his vision has been achieved. At this point the standard has become that networking technologies are implemented and start maturing in the open source community as open code running on open operating systems supporting mostly open source applications. And only after are those components reimplemented in semi-closed, semi-hardware (RMS has never been as concerned about software implemented in hardware because it isn't IP in the same way) solutions for ultra high performance.
So anyway. I think we agree on the fact. But I think RMS achieved far more success than he originally intended for the community he was interested in. Free Software dominates the technical community and the software they use and manage. Consumers don't care about the 4 freedoms and there ...
What's going to be the interesting battle over the next 20 years is the Internet of Things. How open is the software running on all sorts of devices that don't have traditional computer interfaces? So far the tendency is towards more openness than has previously existed as Free Software / Open Source is becoming the standard for devices.
You were arguing that this French company could be meaningfully regulated. If you don't want governments punishing them, then they can't regulate them on their shore or off.
And if they aren't following the law then what? They just shift assets between them everytime one of them gets caught.
Everyday. I work in telco. Regulators cross borders and revolve disputes all the time. Same with airline industry who handle new problems. Shipping and packaging. Oil shipping and refining (though extraction often gets quite violent). It happens most of the time.
Or create an effectual system of global regulation like what happens for most industries.
They have full authority over the parent company. They could order Google USA to shut it down. Same power they have over any other owner of foreign stuff that commits international crimes. Their are plenty of Americans in American prisons for things they did abroad.
I'm not saying a single company I'm saying all multi-nationals. And then another specific one having to do with multi-national internet businesses.
I agree the WTO is a good place as well to deal with this. But that is a new power and so likely requires at least an agreement.
Because they don't have a choice if we are going to have foreign corporations. Orange-USA doesn't really exist. One of the core powers that gives the government the ability to regulate is the ability to punish. The USA government can't meaningfully punish Orange by punishing Orange-USA because Orange can create dozens of subsidiaries or buy shares in other companies and move resources and assets between them. Orange is the "immortal person" the USA subsidiary companies are like their clothes easily changed. So for the USA government to be able to regulate they need the help of France. Which means that this negotiation has to take place
As for long legally intensive battles... Why does that need to occur?
Not really. Because the people in those countries continue to use Google services. Google pulling out is mostly a technical rearrangement. It isn't them meaningfully completely pulling out. And so the countries often can't accomplish their objective.
Now I happen to like how Google is handling corrupt EU regulations but Google is a perfect case in point of the futility of trying to regulate international corporations in each country. The USA government is the government that can effectually regulate Google. If Germany or France or Spain wants a regulate the USA government is who they should be negotiating with, if they want their regulation to actually happen.
Then under that interpretation mobile has proven what is mostly true elsewhere. Guys who run servers like the 4 freedoms and take advantage of them, developers take advantage of them while end user applications mostly consist of a mixture of unfree code with free under layers. Which means that the 4 freedoms are valuable freedoms for computer professionals and mostly worthless for everyone else.
I suspect Osama bin Laden who was well known for his quiet, respectful and thoughtful conversation style (I'm not kidding, his policies and his personality don't match) might beg to differ with that. Much as I like Jobs and Torvalds I suspect 200 years from now they may be mostly forgotten while bin Laden is still remembered for mainstreaming Qutbism. Heck our likely next president, Mrs. Clinton, is rather introverted and tends to quietly guide the people close to her.
For enterprise Apple sells (well below cost) the server components for iOS. Companies can setup their own applications stores and distribute applications through those with no 30% going to Apple. The 100 limit the GP mentioned applies to individual devices. If the devices register as belonging to X company X company can distribute applications through their infrastructure to all of them. Apple governs their app store. They make it easy to setup up alternatives.
iOS apps don't have unneeded access and need to function well if semi-needed access is denied. It is a vastly more secure ecosystem.
FOSS is shinning on mobile. Virtually every cloud that is being used to support those apps is 95% FOSS or more. Many of the frameworks are FOSS. The mobile operating systems themselves are about 80% FOSS.
Non technical end user tweaking is an area where FOSS still gets its butt kicked.