The A4 chip doesn't really seem to have any really fancy technologies in it. Mostly, it's just repackaging and combination of other components that already exist, but instead of combining them in the generic, general purpose manner they normally are, putting them all together in one chip allows a bunch of superfluous stuff to be eliminated.
This means that algorithms in the real world can scale worse than their theoretical "worst-case", if that theoretical worst-case scaling is based on the assumption of constant memory access cost, since that assumption does not hold in the real world.
I agree with all of your post except this part. Algorithmic complexity theory is about orders of magnitude, not about precise numbers. That's why we have O(n) as a complexity class but not O(2n), O(3n) as separate classes; saying that an algorithm has O(n) worst-case time performance is saying that the time it takes to run it is approximated by some linear function of the problem size. We don't particularly care which linear function it is, because that depends too closely on your assumptions about the computational model and/or hardware. What we really care about when we call it O(n) is that that's better than something like O(n^2) (a.k.a. polynomial time or just "P") or O(log n), no matter what computational model you assume.
As long as the slow memory accesses in the physical hardware still respect some bound, you can treat that upper bound as the constant worst-case memory access cost, and use the algorithmic complexity analysis to calculate an upper bound on the algorithmic complexity. If we turn your argument on its head, we'd say instead that the actual physical cost of running algorithms is often faster than the complexity class indicates because many memory accesses are much faster than the constant upper bound, but that doesn't make the linear upper bound invalid. The best you might be able to do is to assume a finer-grained computational model with variable cost memory access, and prove that you can get a lower upper bound there, but the original higher upper bound is still an upper bound for the computational model chosen, and can still be useful when reasoning about a system.
Treating the database as a black box is the problem, not the solution.
Then the database is broken.
No, the database has never been supposed to be a black box. It is a abstraction that divides the data management problem into two parts: the logical and the physical. The logical side, very roughly, is schema definitions and queries. The physical side is physical data arrangement, indexing, data statistics, cost-based optimization, etc. To get good performance of the database, you must be ready to use the tools provided on the physical side.
Aside from the effect of inflation, real estate has a limited supply, but the population continues to increase (i.e., demand continues going up). Thus the price of housing is going to continue to go up until either we find a way to make more land or the population stops increasing.
There is plenty of land. Better transport makes it possible for people to live further afield from their jobs. It's also always possible to grow vertically by tearing down single-story dwellings and putting up multistory buildings--i.e., make dramatically more efficient use of some of the land we already use for housing.
Also, the US population growth rate is about 0.9%. Even with a constant housing supply, this puts an upper limit on how much you could expect the housing demand to grow per year--and it's less than the historical return of bonds and stocks, and the historical inflation rate.
You don't seem to understand what's being talked about. GP said that sometimes removing columns from SELECT clauses in Oracle causes drastic slowdowns. You respond with a bunch of cut-and-paste about join removal in PostgreSQL 9.0, which apart from just not being relevant, is a feature that potentially causes speedups when you remove columns from SELECTs (the opposite effect of what's being discussed).
I _think_ what they are saying, is if the column you remove from the select clause, is the only column from that table being used/displayed other than the column used in the join, then the optimizer will remove the entire join from the query, thereby speeding up the query. But I could be very wrong because I've been wondering the same thing as you since it was first mentioned.
I think you're mixing up two things:
The explanation of how join removal works in PostgreSQL 9.0, which you are stating the very basics of.
The thread's original poster's claim that, in Oracle, removing a column from a SELECT clause sometimes results in a modified query that performs much worse than the original.
You're basically right in your understanding of (1), though the details are considerably more complex.
I expressed my skepticism about (2) in another comment in this thread. Then the comment you're replying to was my sympathetic yet skeptical reply to another poster who assumed that the column removal from the SELECT clause might cause a cartesian join (a join of two tables without any conditions restricting the joins, which tends to perform very badly and is usually an indicator that your query is written wrong).
Robert Haas comments on this in one of the blog posts linked in the article:
In some cases you can prove that the inner join can't affect the row count - foreign keys are an essential part of deducing this.
...and, as a sidenote, this is a perfect example of why it sucks that SQL allows for duplicate rows in tables and result sets. This optimization would be (a) easier and (b) far more generally applicable if the database only returned unique rows, instead of having to worry about returning the correct number of duplicate rows.
Certain optimizations use page reads to improve performance when returning a result set. It will try to access pinned/blocked columns before accessing trivial columns there by reducing collection times. Although these columns may not be indexed per se, they are ordered so retrieval pretty much sequential. Removing those pinned columns (either from the query or from thjs DB in general) may cause the DB to to start 'thrashing' and reduce the performance. This is another reason why vectored indexes are a good thing.
You know, you've lost me there, and I don't think it's me. You're going to have to explain that better if I'm going to believe it.
The best I can make it, you're assuming that there is a table that's structured/ordered/partitioned/whatever based on the values of the column that gets removed from the query. I assume that's what you mean by "blocked" or "pinned" columns.
However, I still fail to understand how including such a column in the SELECT list of a query provides any optimization opportunities. The only ways I can see it affecting query execution (e.g., table access paths and join methods) still involve use of the column in the join conditions in the FROM, restrictions in the WHERE, or use of the column in GROUP BY.
You could... You know... Take a look at the execution plan before executing your new query, see that big cartesian join and notice there's something wrong.
I'm still waiting for the fantastic explanation of how removing a column from a SELECT clause in a query that executes with no cartesian joins will result in a query that does.
This bites me occasionally in Oracle where you've got a big query that has lots of tables joined together, and then at some point one of the columns is removed from the select part, and the query performance suddenly goes to hell. Then you have to go through and verify that each table is actually being used (even worse if the column that was removed from the select came from deep joins).
Do you have an actual example? I simply don't see how removing a column from a select clause would make a query slower, unless you're talking about uses of aggregate functions, and even there I'm having a hard time seeing how a removal could make things worse.
The thing that determines how much work the database has to do in order to produce the results is the FROM, the WHERE and the GROUP BY, because those are the ones that determine what's going to be accessed, joined, sorted and how. The SELECT (except for the use of aggregate functions) primarily just decides what information to present from the join results and how to present it.
It's all smoke & mirrors (and some processing in the background). The primary limitation right now is the lens size, not the sensors or the quality of the lenses. You won't get a significantly better picture without increasing the size of the lens, which it doesn't look like Apple is doing.
The liveblog I watched vaguely suggested that the sensor is bigger, but I'd still seek confirmation of that by this point.
What was clear, however, is that the iPhone 4 is using a back illuminated sensor, which is supposed to be a significant incremental advance in sensor performance, equivalent to increasing the sensor diagonal by 1.4. Though again, I haven't yet seen independent confirmation of this (haven't been looking, though)--my figures come from press releases.
It's true that with a camera that small, the primary limitation is indeed the lens size as you said. But you can possibly get a significantly better picture by improving the sensor and lens technology at a constant size--it's just that if you actually cared primarily about image quality, you'd always go for a much larger sensor and lens.
It may have gone faster if not for the fact that during the entire negotiation, Steve Jobs kept screaming "THAT'S NOT MY CAR!".
Except that never happened. Read the affidavit for the search warrant. Once Gizmodo asked Apple for a letter saying that the phone was theirs, Apple sent it.
To make it worse, even if Apple had denied ownership of the phone, that's not relevant, because the Good Samaritan knows that the car is not his.
If they can spell their own names, of course they will be a whiz at regular words. "Bill", "Joe", and "Mary" haven't a chance against "Dhananjay" "Jagannatha" or "Chandrakanta".
I know you're joking and all, but Indian names are usually phonetic. If they know to pronounce their own name, they can spell it.
Spelling bees are a very American phenomenon, because very few languages have an orthography that's as messed up as English does. Most other languages have spelling systems that make such a competition pretty boring.
This isn't correct. a UI is a human / machine interface. An API is a machine / machine interface which is implemented by a human. The documentation for the API is more properly compared to UI.
No, the abstractions and conceptual models used by an API to make itself understandable to programmers are perfectly analogous to a UI in the narrow sense you're insisting. The machine doesn't care what abstractions the API uses to make problem solving easier for developers.
I agree that complex tasks require complex APIs. I just don't see why this is such a complex task. We're not using SSL, namespaces or storing a gigantic file here, and I don't see any reason why those features should make the process that much harder.
The bigger question, really, is how come if somebody thought that that list of steps to copy a file was good enough to be an example of how to perform a simple task using the API, then why aren't those steps encapsulated as a reusable function that just gets called with the source, the destination and the objects to copy.
It just strikes me as programming at the wrong level of abstraction--which is, quite frankly, the most common flaw of programmers everywhere.
If your stock holdings goes up in value, you have not yet made a profit. You don't make a profit until you sell, which is money that comes from someone else. Just because they haven't sold at a loss doesn't change the fact that they still have to buy-in for you to sell. It's not a net loss, but it's not a net gain, either. The money you gain comes from others, and the money you lose goes to others. No actual wealth is created or destroyed, just transferred. It is zero-sum.
The problem with this picture is that stocks entitle the holder to a share of the profits of the corporation, which come from creating new and valuable goods and services outside of the market. Part of those profits are fed back into the market, which increases the pool of money in the market and therefore turns it into a non-zero-sum game.
That is, you can only call the stock market a zero-sum game if you assume it's a closed system, but a growing economy will continually pump new money into it (and a shrinking economy will take money away from it).
The present value of the future payments that will be made to the stockholder.
If your answer has anything to do with the future (future revenue, future earnings, etc.), please explain how you're able to know the "true value" of anything which has yet to happen.
We are not able to know the true value today, but that does not entail that the true value of a stock is undefined, but simply that is unknown. And I don't think that infirms GP's argument that excess liquidity introduces extra noise into the market. The stock market price is supposed to be the aggregation of many actors' estimate of the true value and their confidence on their estimates (as measured by their willingness to put their money behind those estimates). Trades that are not motivated by such estimates add noise to the market.
In your example, the implicit assumption is that Joe and Mary have very little actual wealth, which is why a long delay between buying and selling is a meaningful discriminator of speculation.
No, not really. All I really assume is that both are using a long-term buy-and-hold strategy, which is pretty much the prototype of an "investor" as opposed to a "speculator."
However, if Joe and Mary are megacorporations with eg 365,000 shares available, then over a period of 10 years, Mary can buy or sell 100 shares every day, and still hold each share for at least 10 years from the time it is bought, provided no activity was logged in the first ten years of its existence.
But, modulo cost basis issues, different shares of the same stock issue are fungible, so I don't think this really tells us anything. The important thing is what position Joe and Mary hold on the stock; think of it as a graph with time on the x axis, and the value of their holdings on the stock on the y axis (long is positive, short is negative). As long as they remain long on that stock for the whole time period (i.e., the graph doesn't spend any stretches of time close to 0 or below), they're basically acting like long-term buy-and-hold investors. They may very well be trading: perhaps adding to their position periodically (like people who set up automatic investment in mutual funds for each paycheck), rebalancing their exposure to the stock, or changing their portfolio allocation (e.g., selling a significant portion of one stock to add a new long-term holding to the portfolio).
There's a common thread to my two remarks here: it's not about trades, it's about strategies. An investor seeks to profit from the profits of the enterprises they invest in. A speculator seeks profit by other means: exploiting short-term crowd psychology, finding obscure arbitrage opportunities, etc.
Honestly, if you are trading anything less than thousands of shares at a time, this most likely isn't going to effect you to the tune of more than a dollar or two tops per trade unless you are trading in some really thin issues.
But since most people invest through mutual funds, which do often perform large transactions, this is in fact a problem for most people.
But for every investor, there has to also be a speculator, otherwise, the transaction won't ever get made.
No, this is just not true. The market can work without speculators, with investors selling to each other. Why? Because some investors will need to sell their investments in order to consume the return, while others will defer consumption in order to invest. If Joe bought AAPL at $10 on 1990, and then Mary then bought it from him at $45 on 2005 and holds it to date, both Joe and Mary are investors, not speculator.
Trades faster than a day should be simply outlawed
You just bought Ford stock an hour ago, and now you just learned that the company is declaring bankruptcy effective 5 o'clock today. Do you really want to be forced to keep that stock until 23 hours from now (when it will be worthless)? Of course not. You want to dump it as fast as possible while the price is still high and you can recover some of your money.
While I certainly agree that once-a-day is too strict, your example is very flawed. If Ford's 5 o'clock bankruptcy is public knowledge, then its price will be zero-ish now, because nobody's going to want to pay a high price for a company that goes bankrupt in the next few hours. If it's not public knowledge, then you'd be trading on insider information, which is fraud. So being able to trade a second time that day only helps you if crime helps you.
I'm pretty libertarian, but I agree these should be stopped. As the other poster said, it gives real estate closer to the market servers an advantage, I'm not quite clear how it works, but it is evident that it does because people are doing it. I assume they can recognize short term patterns and jump in ahead of anyone else who might try to take advantage of them.
When a buyer or seller places a limit order, their limit price is supposed to be a secret, and the market is supposed to deliver the best possible price for them relative to that limit. Flash trades and "immediate-or-cancel" orders allows high frequency traders to issue a flurry of really quick orders to discover a slower trader's limit price, and then trade at that limit instead of the price that the slower guy would have otherwise gotten.
So if ACME is trading at $26.10, slow buyer A enters a limit buy order for $26.40, and slow seller B enters a market sell order, the high speed trader is able to use really fast trades to discover A's $26.40 limit, buy B's shares at $26.10, and then sell them right away to A for $26.40, all before A can learn about B's more favorable sell offer and accept it.
It's allowed because the current theory is that anybody who wants to do it, can. I think the best argument against that is it takes real estate close to the market computers in order to have a fast enough ping time to trade by the millisecond.
No, the best argument against it is that it allows automated front-running by allowing the high-frequency traders to issue and cancel small orders in quick succession to discover an ordinary buyers or seller's limit price, and then profiting by offering a sale that would have otherwise happened at a price more favorable to the initiator. To quote the link (which is highly recommended):
Let's say that there is a buyer willing to buy 100,000 shares of BRCM with a limit price of $26.40. That is, the buyer will accept any price up to $26.40. But the market at this particular moment in time is at $26.10, or thirty cents lower.
So the computers, having detected via their "flash orders" (which ought to be illegal) that there is a desire for Broadcom shares, start to issue tiny (typically 100 share lots) "immediate or cancel" orders - IOCs - to sell at $26.20. If that order is "eaten" the computer then issues an order at $26.25, then $26.30, then $26.35, then $26.40. When it tries $26.45 it gets no bite and the order is immediately canceled.
Now the flush of supply comes at, big coincidence, $26.39, and the claim is made that the market has become "more efficient." Nonsense; there was no "real seller" at any of these prices! This pattern of offering was intended to do one and only one thing - manipulate the market by discovering what is supposed to be a hidden piece of information - the other side's limit price!
Slashdot Mirror
The A4 chip doesn't really seem to have any really fancy technologies in it. Mostly, it's just repackaging and combination of other components that already exist, but instead of combining them in the generic, general purpose manner they normally are, putting them all together in one chip allows a bunch of superfluous stuff to be eliminated.
I agree with all of your post except this part. Algorithmic complexity theory is about orders of magnitude, not about precise numbers. That's why we have O(n) as a complexity class but not O(2n), O(3n) as separate classes; saying that an algorithm has O(n) worst-case time performance is saying that the time it takes to run it is approximated by some linear function of the problem size. We don't particularly care which linear function it is, because that depends too closely on your assumptions about the computational model and/or hardware. What we really care about when we call it O(n) is that that's better than something like O(n^2) (a.k.a. polynomial time or just "P") or O(log n), no matter what computational model you assume.
As long as the slow memory accesses in the physical hardware still respect some bound, you can treat that upper bound as the constant worst-case memory access cost, and use the algorithmic complexity analysis to calculate an upper bound on the algorithmic complexity. If we turn your argument on its head, we'd say instead that the actual physical cost of running algorithms is often faster than the complexity class indicates because many memory accesses are much faster than the constant upper bound, but that doesn't make the linear upper bound invalid. The best you might be able to do is to assume a finer-grained computational model with variable cost memory access, and prove that you can get a lower upper bound there, but the original higher upper bound is still an upper bound for the computational model chosen, and can still be useful when reasoning about a system.
No, the database has never been supposed to be a black box. It is a abstraction that divides the data management problem into two parts: the logical and the physical. The logical side, very roughly, is schema definitions and queries. The physical side is physical data arrangement, indexing, data statistics, cost-based optimization, etc. To get good performance of the database, you must be ready to use the tools provided on the physical side.
There is plenty of land. Better transport makes it possible for people to live further afield from their jobs. It's also always possible to grow vertically by tearing down single-story dwellings and putting up multistory buildings--i.e., make dramatically more efficient use of some of the land we already use for housing.
Also, the US population growth rate is about 0.9%. Even with a constant housing supply, this puts an upper limit on how much you could expect the housing demand to grow per year--and it's less than the historical return of bonds and stocks, and the historical inflation rate.
Somebody doesn't understand the difference between mean and median.
You don't seem to understand what's being talked about. GP said that sometimes removing columns from SELECT clauses in Oracle causes drastic slowdowns. You respond with a bunch of cut-and-paste about join removal in PostgreSQL 9.0, which apart from just not being relevant, is a feature that potentially causes speedups when you remove columns from SELECTs (the opposite effect of what's being discussed).
I think you're mixing up two things:
You're basically right in your understanding of (1), though the details are considerably more complex.
I expressed my skepticism about (2) in another comment in this thread. Then the comment you're replying to was my sympathetic yet skeptical reply to another poster who assumed that the column removal from the SELECT clause might cause a cartesian join (a join of two tables without any conditions restricting the joins, which tends to perform very badly and is usually an indicator that your query is written wrong).
...and, as a sidenote, this is a perfect example of why it sucks that SQL allows for duplicate rows in tables and result sets. This optimization would be (a) easier and (b) far more generally applicable if the database only returned unique rows, instead of having to worry about returning the correct number of duplicate rows.
You know, you've lost me there, and I don't think it's me. You're going to have to explain that better if I'm going to believe it.
The best I can make it, you're assuming that there is a table that's structured/ordered/partitioned/whatever based on the values of the column that gets removed from the query. I assume that's what you mean by "blocked" or "pinned" columns.
However, I still fail to understand how including such a column in the SELECT list of a query provides any optimization opportunities. The only ways I can see it affecting query execution (e.g., table access paths and join methods) still involve use of the column in the join conditions in the FROM, restrictions in the WHERE, or use of the column in GROUP BY.
I'm still waiting for the fantastic explanation of how removing a column from a SELECT clause in a query that executes with no cartesian joins will result in a query that does.
There is no objective sense in which Homo sapiens can be said to be the "dominant life form." It's a value judgement. Don't try to pass it as a fact.
Do you have an actual example? I simply don't see how removing a column from a select clause would make a query slower, unless you're talking about uses of aggregate functions, and even there I'm having a hard time seeing how a removal could make things worse.
The thing that determines how much work the database has to do in order to produce the results is the FROM, the WHERE and the GROUP BY, because those are the ones that determine what's going to be accessed, joined, sorted and how. The SELECT (except for the use of aggregate functions) primarily just decides what information to present from the join results and how to present it.
The liveblog I watched vaguely suggested that the sensor is bigger, but I'd still seek confirmation of that by this point.
What was clear, however, is that the iPhone 4 is using a back illuminated sensor, which is supposed to be a significant incremental advance in sensor performance, equivalent to increasing the sensor diagonal by 1.4. Though again, I haven't yet seen independent confirmation of this (haven't been looking, though)--my figures come from press releases.
It's true that with a camera that small, the primary limitation is indeed the lens size as you said. But you can possibly get a significantly better picture by improving the sensor and lens technology at a constant size--it's just that if you actually cared primarily about image quality, you'd always go for a much larger sensor and lens.
Except that never happened. Read the affidavit for the search warrant. Once Gizmodo asked Apple for a letter saying that the phone was theirs, Apple sent it.
To make it worse, even if Apple had denied ownership of the phone, that's not relevant, because the Good Samaritan knows that the car is not his.
I know you're joking and all, but Indian names are usually phonetic. If they know to pronounce their own name, they can spell it.
Spelling bees are a very American phenomenon, because very few languages have an orthography that's as messed up as English does. Most other languages have spelling systems that make such a competition pretty boring.
No, the abstractions and conceptual models used by an API to make itself understandable to programmers are perfectly analogous to a UI in the narrow sense you're insisting. The machine doesn't care what abstractions the API uses to make problem solving easier for developers.
The bigger question, really, is how come if somebody thought that that list of steps to copy a file was good enough to be an example of how to perform a simple task using the API, then why aren't those steps encapsulated as a reusable function that just gets called with the source, the destination and the objects to copy.
It just strikes me as programming at the wrong level of abstraction--which is, quite frankly, the most common flaw of programmers everywhere.
The problem with this picture is that stocks entitle the holder to a share of the profits of the corporation, which come from creating new and valuable goods and services outside of the market. Part of those profits are fed back into the market, which increases the pool of money in the market and therefore turns it into a non-zero-sum game.
That is, you can only call the stock market a zero-sum game if you assume it's a closed system, but a growing economy will continually pump new money into it (and a shrinking economy will take money away from it).
The present value of the future payments that will be made to the stockholder.
We are not able to know the true value today, but that does not entail that the true value of a stock is undefined, but simply that is unknown. And I don't think that infirms GP's argument that excess liquidity introduces extra noise into the market. The stock market price is supposed to be the aggregation of many actors' estimate of the true value and their confidence on their estimates (as measured by their willingness to put their money behind those estimates). Trades that are not motivated by such estimates add noise to the market.
No, not really. All I really assume is that both are using a long-term buy-and-hold strategy, which is pretty much the prototype of an "investor" as opposed to a "speculator."
But, modulo cost basis issues, different shares of the same stock issue are fungible, so I don't think this really tells us anything. The important thing is what position Joe and Mary hold on the stock; think of it as a graph with time on the x axis, and the value of their holdings on the stock on the y axis (long is positive, short is negative). As long as they remain long on that stock for the whole time period (i.e., the graph doesn't spend any stretches of time close to 0 or below), they're basically acting like long-term buy-and-hold investors. They may very well be trading: perhaps adding to their position periodically (like people who set up automatic investment in mutual funds for each paycheck), rebalancing their exposure to the stock, or changing their portfolio allocation (e.g., selling a significant portion of one stock to add a new long-term holding to the portfolio).
There's a common thread to my two remarks here: it's not about trades, it's about strategies. An investor seeks to profit from the profits of the enterprises they invest in. A speculator seeks profit by other means: exploiting short-term crowd psychology, finding obscure arbitrage opportunities, etc.
But since most people invest through mutual funds, which do often perform large transactions, this is in fact a problem for most people.
No, this is just not true. The market can work without speculators, with investors selling to each other. Why? Because some investors will need to sell their investments in order to consume the return, while others will defer consumption in order to invest. If Joe bought AAPL at $10 on 1990, and then Mary then bought it from him at $45 on 2005 and holds it to date, both Joe and Mary are investors, not speculator.
While I certainly agree that once-a-day is too strict, your example is very flawed. If Ford's 5 o'clock bankruptcy is public knowledge, then its price will be zero-ish now, because nobody's going to want to pay a high price for a company that goes bankrupt in the next few hours. If it's not public knowledge, then you'd be trading on insider information, which is fraud. So being able to trade a second time that day only helps you if crime helps you.
No, it's much worse than this. High frequency trading allows its practicioners to cheat, quite literally, as I pointed out in another comment that linked to this blog post.
When a buyer or seller places a limit order, their limit price is supposed to be a secret, and the market is supposed to deliver the best possible price for them relative to that limit. Flash trades and "immediate-or-cancel" orders allows high frequency traders to issue a flurry of really quick orders to discover a slower trader's limit price, and then trade at that limit instead of the price that the slower guy would have otherwise gotten.
So if ACME is trading at $26.10, slow buyer A enters a limit buy order for $26.40, and slow seller B enters a market sell order, the high speed trader is able to use really fast trades to discover A's $26.40 limit, buy B's shares at $26.10, and then sell them right away to A for $26.40, all before A can learn about B's more favorable sell offer and accept it.
No, the best argument against it is that it allows automated front-running by allowing the high-frequency traders to issue and cancel small orders in quick succession to discover an ordinary buyers or seller's limit price, and then profiting by offering a sale that would have otherwise happened at a price more favorable to the initiator. To quote the link (which is highly recommended):