All dedup operations have a trade-off between disk I/O and memory use. The less memory you use the more disk I/O you have to do, and vise-versa.
Think of it like this: You have to scan every block on the disk at least once (or at least scan all the meta-data at least once if the CRC/SHA/whatever is already recorded in meta-data). You generate (say) a 32 bit CRC for each block. You then [re]read the blocks whos CRCs match to determine if the CRC found a matching block or simply had a collision.
The memory requirement for an all-in-one pass like this is that you have to record each block's CRC plus other information... essentially unbounded from the point of view of filesystem design and so not desirable.
To reduce memory use you can reduce the scan space... on your first pass of the disk only record CRCs in the 0x0-0x7FFFFFFF range, and ignore 0x80000000-0xFFFFFFFF. In other words, now you are using HALF the memory but you have to do TWO passes on the disk drive to find all possible matches.
The method DragonFly's HAMMER uses is to allocate a fixed-sized memory buffer and start recording all CRCs as it scans the meta-data. When the memory buffer becomes full DragonFly dynamically deletes the highest-recorded CRC (and no longer records CRCs >= to that value) to make room. Once the pass is over another pass is started beginning with the remaining range. As many passes are taken as required to exhaust the CRC space.
Because HAMMER stores a data CRC in meta-data the de-dup passes are mostly limited to just meta-data I/O, plus data reads only for those CRCs which collide, so it is fairly optimal.
This can be done with any sized CRC but what you cannot do is avoid the verification pass.. no matter how big your CRC is or your SHA-256 or whatever, you still have to physically verify that the duplicate blocks are, in fact, exactl duplicates, before you de-dup their block references. A larger CRC is preferable to reduce collisions but diminishing returns build up fairly quickly relative to the actual amount of data that can be de-duplicated. 64 bits is a reasonable trade-off, but even 32 bits works relatively well.
In anycase, most deduplication algorithms are going to do something similar unless they were really stupidly written to require unbounded memory use.
Because, really, virtually nobody here actually understands what is going on. So I will throw some light on the matter.
There are two big problems with solar energy in the U.S. right now.
The #1 problem is that the U.S. is in the middle of a natural gas bonanza that started about a decade ago but really only started ramping up around 2007ish... and kept ramping up right on through the crash and is still ramping up today, with no end in sight. This has caused domestic NG production to go through the roof and domestic NG prices to fall through the floor.
Power companies switching from coal are going right to natural gas, which has 30% (or better) lower emissions and low prices for at least the next decade. It will take that long for our LNG export infrastructure to ramp up enough for world markets to relieve the downward pressure on domestic natural gas prices. In anycase, just switching to NG allows power companies to meet EPA requirements for probably the next decade (or longer), and that's without any magic technology to make it even cleaner.
Natural gas used to only be used for peaking plants, except in California where they started to also be used for base load earlier than other parts of the country. Now natural gas is being used for base load across the board (along with nuclear). Nearly all of those coal plants undergoing decommissioning are being replaced with natural gas plants, not solar or anything else.
The #2 problem with solar energy is China's overproduction of solar panels. China has no problem destroying their environment with the chemical leavenings from the production of solar panels. Chinese companies are facing the same problems that U.S. companies are facing with prices plunging, but they have a much lower cost of production. The result is that non-chinese companies basically can't compete (under ANY circumstances).. because we aren't willing to destroy our environment like the Chinese are.
This has created a massive drop in price for the bulk panels that large businesses (like Google for example) can purchase. Consumers cannot get at these prices, we just don't buy enough panels, but even so prices are going down for us too.
This, in turn, has created a huge problem for solar thermal power companies, because the price of bulk panels has dropped below or near par to the cost of constructing a solar thermal (mirror based) energy plant. This makes the ongoing cost of a solar thermal plant, which requires significant maintainence and has parts with limited life spans due to thermal cycling, higher than the near zero running cost of an installed conventional solar panel closer to the buyer (typically on the roof of the business premises).
Numerous other factors are also creating issues. Germany was the single biggest purchaser of solar panels for the last decade due to massive government subsidies. Those subsidies are now winding down, which only makes the market glut worse.
And that's the problem in a nutshell. Even the most environmentally minded person has to realize by now that it is impossible to go from coal to solar in one step. Natural gas is the only thing we have in-country that is even remotely capable of replacing coal at the generation levels required. I will applaud government investment in solar energy but anyone who thinks that solar can actually replace fossil fuel is fooling themselves.
Well, you definitely CAN tell when one or more buffers along the path begins to fill up, because latency increases. Packet loss is not necessary and, in fact, packet loss just makes the problem worse since many TCP connections implement SACK now and can keep the bandwidth saturated even in the face of packet loss.
The ideal behavior is probably not to start dropping packets immediately... eventually, sure, but definitely not immediately. Ideally what you want to do is to attempt to shift the problem closer to the edges of the network where it is easier to fairly apportion bandwidth between customers.
Send-side bandwidth limiting is very easy to implement since TCP already has a facility to collect latency information in the returned acks. I wrote a little beastie to do that in FreeBSD many years ago, and I turn it on in DragonFly releases by default.
The purpose of the feature is not to completely remove packet buffering from the network, because doing so would put the sending server at a severe disadvantage verses other servers that do not implement similar algorithms (which is most of them).
The purpose is to unload the buffers enough such that the algorithms in the edge routers aren't overloaded by the data and can do a better job apportioning bandwidth between streams.
Our little network runs this coupled with fair queueing in both directions... that is, we not only control the outgoing bandwidth, we also pipe all the incoming bandwidth through a well connected colo and control that too, before it runs over the terminal broadband links. This allows us to run FAIRQ in both direction in addition to reserving bandwidth for TCP acks and breaking down other services. FAIRQ always works much better when links are only modestly overloaded and not completely overloaded. Frankly we don't have much of a choice, we HAVE to do this because our last-leg broadband links are 100% saturated in both directions 24x7. Anything short of that and even a single video stream screws up the latency for other connections beyond hope.
This sort of solution works great near the edges.
For the center of the network, frankly, I think about the best that can be done is modest buffering and RED and then trying to reduce the load on the buffers in the center with algorithms run on the edges (that can sense end-to-end latency). The modest buffering is needed for the edge algorithms to be able to operate without bits of the network having to resort to dropping packets. In otherwords, you want the steady state load for the network to not have to drop packets. Dropping packets should be reserved for the case where the load changes too quickly for the nominal algorithms to react. That's my opinion anyhow.
You are absolutely right. Plus the iTouch is only $200 brand spanking new for the 8GB generation 4 model. Front and rear-facing cameras, microphone, accelerometer and Gyro. The iTouch doesn't have a GPS or compass (awwww) or 3G, so it is feature-comparable. The display has just about as many pixels but of course is considerably smaller. But yah, it does fit in the pocket and for anything larger that doesn't I'd rather have an iPad's bigger display anyhow.
To be fair, trying to read a book with an iPhone or iTouch would not be fun, but neither do I consider a 7" active display fun for reading. The only reason I have a Kindle at all is for the liquid paper display for outside reading. I'm sure many people will love the color Kindle for reading but that's not a reason for buying it vs a liquid paper kindle.
Amazon's fire is already right smack in the middle of a squeeze between Apple's small and Apple's large.
p.s. I'm not against Amazon per-say, but as an investor I just don't find their tablet compelling.
Compare it against an 16G iPad 1 on Amazon then (I don't know of any 8G iPads). Around $350, wifi-only (about the same as on E-Bay). Around $400 for Wifi+3G.
That's $150 more for a MUCH nicer screen, double the storage, and a microphone (and thus Skype too). $200 more for 3G on top of that.
Here's the problem: Consumers who buy these things are already paying between $150-$250/mo for their phone+internet+TV services. So you're talking, literally, a 1-2 months difference in price against a readily available iPad 1 in terms of consumer cash flow.
Worse, we are talking about NO difference in price against a brand new iTouch (iPod touch) with its smaller screen but also with a microphone AND front-facing camera (meaning video calls can be done with it).
The reason Apple is making money hand over fist with their products is that they sell them in the right form factor at the right price point.
Now is Amazon selling their pad at the right price point? I personally don't think they are. Why would I want to get an Amazon pad verses a brand new 4th generation iPod touch? Let alone a used iPad 1. All the Amazon pad has going for it is a screen that's a little bigger but still not big enough. I don't think so.
Apple has something like 85% of the pad market. Most of the android tablet vendors have volumes in the 500,000 - 800,000 range (each) whereas Apple's volumes are in the ten's of millions.
The market is something like 12 million pads this year and expected to be something like 40 million pads next year, if I remember correctly. It's a very big pie.
Both the Nook and Amazon appear to have moved away from liquid paper for their color displays. I don't have a nook but I gotta say that the Amazon B&W Kindle w/liquid paper is really wonderful when reading outside in bright sunlight, and just fine everywhere else. It's the ONLY reason why I have a kindle. I bring it along on vacations and don't even bother to bring a charging cord.
If these color devices don't have wonderful visibility in bright sunlight then I have no interest in buying them relative to buying an iPad. It's the only reason why I read books on the kindle in the first place!
Amazon might want their new gadget to compete in the Nook space, and I'm sure it will do well in that space, but if it is large enough to be considered a pad and yet can't compete in the Pad space, and its display isn't as readable as their B&W liquid paper display, then their volume improvements will only be incremental at best.
Customers often view products very differently than companies would like them to, and I think that is going to be the case for the Amazon pad.
Deadly serious. Consumers these days either have a native HDMI/HDCP-enabled TV or they don't. If you don't, then composite is your only other choice for most device-to-TV connections. Sure, laptops have VGA output, and a lot of TVs have VGA input, but DVI connectors don't typically work with HDCP and modern flat screen TVs don't even work well with the original HDMI standard.
So the only two reliable choices are a VERY new HDMI/HDCP-enabled TV or composite video. And that's it.
Also, most video producers (Starz, HBO, etc) only allow composite output for unencrypted / non-DRM video.
Only a small percentage of the population have the most modern flatscreen TVs with native HDMI and a relatively recent HDCP implementation. So if you are an electronic gadget company those are your only two choices. That's why you still see composite inputs and outputs available on a lot of brand new equipment.
I actually use my iPad in this configuration. Just about all cases support three positions: vertical on edge, horizontal on edge, and flat at a modest angle.
The ssh apps are really only useful when you have a wireless keybaord. Without the keyboard there isn't enough screen real-estate to be able to do real work, and you can't really type all that quickly using the on-screen keyboard. I can type at 80wpm (used to be 100 but I'm getting older now:-)) on the wireless keyboard without making an error. I can type at maybe 10wpm on the on-screen keybaord.
In an emergency I can ssh in with just the ipad and its on-screen keyboard. It is painful as hell but it works. I can do the same thing on my android phone... the pain, strangely enough, is about the same either way, even though the pad has a bigger screen. Go figure.
But with the keyboard the pad is very usable. EMACS people will be thrilled (it's a really good combo). Too bad I'm a VI guy:-).
And Apple has Apple TV... the TV boxes are not competitors to the pads. That is, the existence of a TV box (I have two) does not prevent a consumer from buying a pad, and the TV box's features will not cause a consumer to buy a less capable pad.
Switching video on the fly is something Apple already supports. It's a feature I don't really use all that often and it's irrelevant for most video apps that aren't movies. For movie apps, say when playing via NetFlix, then the service itself remembers where you were... sync is a fancy do-nothing name that's meaningless when video is streamed from a cloud service. Of COURSE the cloud service is going to remember where you left off.
8G of storage... arguable, but also considerably less flexible. The minimum these days is 16G and 32G is really preferable. The problem is that the biggest seller of pads are the games you can run on them. At least that is the case for Apple. It isn't JUST media services. Games are a huge portion of Apple's income stream. Games tend to be considerably larger than non-game Apps. At only 8G Amazon is seriously mis-reading the applications market for pads. Is Amazon not interested in supporting a rich gaming base for their pads?
People buy pads for portability, and part of that portability is being able to use the internet while outside of your personal Wifi range (or starbucks or your folks house or whatever). And even for the workplace... I know a great many people who use 3G at work instead of their company's wifi network due to privacy concerns.
Another huge part of that portability are location services. Maps. Navigation. Google contextual searches. Even games use location services.
Amazon's pad has a serious market placement problem. By removing (big) games, GPS (location services, contextual services, navigation), and 3G (workplace privacy, true portable use), basically the only thing Amazon has left is in-home use and kids playing pre-loaded videos in the car in a world where preloading videos are already mostly obsolete (and with only 8G even preloading videos is severely limited).
These are things that consumers expect these days. Pads without them are just toys with a very, VERY limited shelf life.
I'm sorry, I just don't see any reason to own the Amazon pad. Not at $200. Maybe at $50 I'd get a few. Not at $200.
Resale value of 16G Wifi iPad 2 after 2 years: $350
Resale of Amazon Kindle Fire after 2 years: $0
See the problem? The iPad 1 on E-Bay RIGHT NOW can be had for ~$325. 16G, wifi-only version. 3G version only a bit more. 2 years from now the iPad 2 is going to be in the same situation.
The Kindle Fire as a media-only player is going to have a resale value of basically nothing in 2 years. They'll be on E-Bay for $50 at most.
So, yes, the Kindle Fire is in fact competing against the iPad, RIGHT NOW.
AppleTV is similar, and costs about the same as the Roku (both are about $100). The AppleTV supports Netflix, NBA, MLB.TV, YouTube, Vimeo, Flickr, and a radio service (not sure whos) out of the box. And of course accepts airplay streams from other apple devices. However, the AppleTV 2 basically only works with HDMI-native/HDCP-enabled devices (no more composit out). I'm not sure about the Roku.
My brother loves his Roku. It's the way to go for in-home entertainment if you don't want to buy into Apple's model.
Not just all tablets... all smart portable devices, period. Even my old iTouch 2 (oh, I'm sorry, I mean 'iPod Touch':-)) is a media consumption gadget and plays video on its tiny screen just fine (and even over airplay to a big screen for some apps with the recent iOS update).
And for in-home there's of course AppleTV or a Roku box or any one of several other fully integrated boxes (don't buy actual flatscreens with the stuff built-into the TV, it won't be easily upgradeable).
There's nothing special about being able to play videos that gives Amazon any sort of advantage here. Their only claim to fame is that the screen is bigger than the screen on my iTouch or Android phone.
People might buy it for $200 but its questionable whether it is worth even that much, and its resale value a year or two down the line is even MORE questionable. Even though an iPad is ~$600-$900 the resale value after ~2 years is still going to be north of $350, so even cost conscious consumers with a little intelligence can buy into the series.
Amazon is competing against iPad 1's on E-Bay. Literally. I know people are pounding the dirt saying that Amazon is really competing against the color nook, and they will certainly eat a cuhnk out of the nook's sales, but I think their pad is too close to other pads (and most especially the iPad) for the consumer space to see it that way.
What kind of stupid ass comment is that? Of course I was talking about the ipod touch... which everyone calls the itouch and has for years. And you know what? I will continue to call it the itouch going into the future.
Ok, maybe I was a year off. But, Mr idiot, you can argue all you want over stupid issues like that and it doesn't change the fact that I actually have this baby in my hand and airplay DOES WORK.
Let me repeat that: air play works. It doesn't have iPad 2's mirroring mode so it doesn't work with all apps, but it works with some, such as YouTube (and probably more later) and the audio works for everything (which is a nice option in addition to blue-tooth audio). I know because, gee, I actually have the product and use it.
The point though is that Apple has given their iTouch product good longevity by continuing to support the hardware and most of the software apps with their new OS releases. Nobody else does that. Even Android phones get long in the tooth because vendors don't update the OS images. You would be lucky to see any of these other pad or smart phone offerings stay up to date for more than ~2 years before their vendors give up on them.
This also gives Apple products tremendous resale value. The iPad 1 got airplay with the last OS update and, again, while it doesn't have the mirroring mode that the iPad 2 is able to support a lot of apps can still route video over it and more as the apps get updated. I know because, again, I have an iPad 1 (which by the way still has a $350 resale value) and I use those functions on it.
Pricey cellular data plan? You mean $25/month for 2G? Doesn't sound pricey to me compared to the average cable or smart-phone bill. Though you can't run video over a 2G plan (when on the road, since you have wifi at home most likely) you can pretty much do everything else without any worries.
I use my android phone to hot-spot other devices, including older iPads and other people's devices who don't have 3G when on road trips. It works... but it's unreliable as heck compared to native 3G and it's also a pretty big hassle as it requires the smart phone to be plugged into power (the battery drain is horrendous).
So as a nominal solution hot-spotting from another device works, but for real convenience that $25/month plan works a lot better.
Hey, you could turn it around and try to justify millions of people buying a $50 pad with a paper overlay instead of a screen and complain about that being apples and oranges too.
It doesn't matter that this Amazon fire is only $200. There isn't anything there that makes it actually be useful verses products which already exist. There is no direct relationship between cost and usefulness... you could buy it and still not be happy, even if it were free.
Or to be more blunt. People who can't afford a used iPad 1 should be just as happy using the smart phone they probably already have which is a ton more capable then the amazon pad whos only claim to fame is a bigger screen.
There's another problem too. Amazon has made their pad obsolete even before it goes on sale. It has a shelf life of less than a year. That isn't $200 you are spending for a piece of equipment you are going to keep for a long time, that's $200 you are spending for a piece of equipment that you are going to throw away in a year.
You know something? I still use my 4-year-old 2nd generation iTouch. I used to carry it around with me. These days it sits in a speaker dock and mostly runs pandora. But even this 4-year-old model can run most apps and can even run video out via Airplay (though without iPad 2's mirroring mode only some apps can do that). All the Amazon pad has over this 4-year-old apple product is a bigger screen.
Essentially. There's no reason to buy it. To read books? I have my black and white kindle with it's 4-week+ battery life to read books with, and you don't want to use an active display for serious reading anyway.
Hey, I don't necessarily need a camera but... no microphone? Are they nuts? I use google voice search all the time on all my devices.
And no GPS? So it can't reliably be used for navigation, even hot-spotted, while in the car.
And no way to export video? Gee, nice to be able to watch movies on such a tiny screen while sitting in one's own home! NOT!
So what's the new Amazon pad good for? Not a damned thing.
You aren't required to sign up for a wireless contract with an iPad. I have a couple of the wifi + cellular versions (because the WIFI-only iPads do not support GPS), but I only turned up a data plan on one of them. $25/mo for 2G of 3G data is a nice price point. It's not as though I'm going to be streaming video all the time while outside 3G range after all, but there are a ton of apps that need internet access (like google maps) that are useful when on the road.
In anycase, while it is possible to hot-spot a smart phone to give a wifi-only pad internet access when on the road (I've done this many times), it isn't nearly as convenient and the phone needs to be connected to a power source to avoid running its battery down in 1hr while in hot-spot mode. I wouldn't buy the Amazon pad, not even for $200, without cellular connectivity.
Ok, I understand the price point is very good, but this device doesn't even compete with the iPad 1. It basically just competes against the Kindle and the color Nook.
* No video output that I can see (vs the ipad's composite output) * No TV box streaming solution (vs airplay) * Is there even any blue tooth for wireless audio? * No 3G, it is wifi only * Only 8G of storage (vs max of 64G for iPad 1/2) * GPS? Accelerometers? Compass? Microphone? * Short battery life when wifi is enabled. 8 hours is only with wifi turned off. The kindle is made for off-line reading and is great with wifi turned off, but this device is worthless with wifi turned off.
Let alone compete with the iPad 2 with it's front and rear facing cameras for video calls. And both the iPad 1 and 2 have cheap ($25/mo for 2G) 3G data plans for when you aren't in wifi range. I usually don't use more than ~200MB outside of wifi range but it's damned convenient. Even with just my android phone I run Pandora and Google maps in the car all the time, and on longer trips with passengers having the iPad with google maps and internet surfing is great fun.
I use my iPad every day and I guarantee you that while the video might be nice on these devices, you really really want video output solution like Apple's AirPlay and Apple's composite outputs when you want to sit down and stream a show from your pad to your home TV.
Well, it balances out in many respects because California also has strict efficiency requirements that reduce average household electricity use by a lot. Since manufacturers don't like to build different versions of the same model a large chunk of those efficiency improvements also benefit the rest of the country.
One good example... all those wall warts that used to weigh a ton because they contained power-hungry transformers have gone away, replaced by far more efficient switching power supplies. Most electronic gadgets which stay plugged-in and ready to go 24x7 (such as TVs and DVD players) now also have sleep modes that cut power use by a lot.
Similarly for dish washers, dryers, washing machines, and refrigerators.
And, of course, the mild weather makes a big difference too, though I should point out the statistics below are for electricity use and don't count heating oil or natural gas.
This is why California has the lowest per-capita electricity use in the U.S. (lower even than Hawaii), at 6721 kWh a year. The U.S. average is 12145 kWh a year. Wyoming has the highest use at 27457 kWh a year.
So even though electricity is expensive in California, the proportional cost is still inline with the rest of the nation. Californians are a lot more careful about their electricity use just in general.
Grid-tie solar systems aren't going to be counted in those numbers (PG&E only meters the net, they don't collect statistics on how large a chunk the solar takes out of it). Only solar power plants were counted in those statistics.
Still, grid-tie systems probably do not reduce total system demand by a whole lot. From my read we have ~252MW of residential solar installed and ~356MW non-residential, which comes to around ~600MW. It's unclear whether the non-residential is just counting commercial retain installations or whether it is also counting solar power stations but I think it's just commercial retail (the number would be too low if it counted both). And, of course, that's just peak generation.
Well, more realistically, what is happening is that China is producing solar panels well in excess of demand (and doing a number on their environment in the process, but that's another story), and it's forced prices for panels down so much that all the other panel producers are dropping like flies. Has nothing whatsoever to do with U.S. regulations (unless you want U.S. creeks to run black with industrial chemicals too), or conspiracy theories about siphoning government money or anything like that.
Even worse, panel prices are now low enough to compete against large industrial-scale mirror/tower systems. So THOSE companies are also getting crushed as contracts get canceled and buyers go with panels. Think about that for a moment. It isn't that panels are less expensive than mirrors, it's that panels are now less expensive than mirrors + tower system + workforce required to keep it maintained.
* HFT is basically situating your own servers in the same machine room as the exchange computers. Then they are given ammunition in the form of 'flash orders'... orders which either fill instantly or are aborted. The HFT computers basically post these flash orders as quickly as they can. When the SEC tested raising the transaction limit for HFT traders to 1 milllion transactions a second the HFT computer's order volume instantly filled the gap and ran at 1 million transactions a second.
* Most of these transactions don't actually do anything. They are 'discovery' transactions or they are simply designed to cause other computer trading algorithms to go haywire by delaying the consolidated tape.
* HFT does not add liquidity to the market. Think about that a moment... if cycling a few millions dollars continuously around in a circle added liquidity to the market then anyone could do it and the market would magically be infinity liquid. There is no argument here. Only people who don't understand how the market works thinks HFT might be able to add liquidity to it. It absolutely does not. This was obvious in several flash crashes where algorithmic trading systems tried to trade into what they thought was liquidity and the HFT algorithms withdrew that liquidity in an instant. Boom, gone. Liquidity you can withdraw in 1 microsecond is not liquidity.
* The primary mechanism for HFT to make money is to create delays in the consolidated tape in order to create wider differences between exchanges for the same ticker symbol, then use those differences to arbitrage a small profit using colocated servers at each location. In this manner HFT might actually aid in price-discovery to some degree, if not for the fact that the price 'discovery' you see on your screen is going to be a few seconds old due to the delay in the consolidated tape. So... no.
* HFT can cause normal algorithmic trading to blow up spectacularly. Normal algorithmic trading depends on prices being accurate (think about that... most computer programs go haywire if presented with inaccurate input). HFT causes prices to be inaccurate, sometimes by a lot, allowing HFT algorithms to blow up algorithmic trading algorithms. I would say they do this on purpose... but the more confusion they can cause the more profit they make.
* HFT primarily hurts institutional traders because HFT algorithms can use flash orders to probe the limits of the institutional trader's orders and then front-run them.
* HFT *IS* front-running. That's what a flash order basically allows them to do. They can come in $0.001 in front of normal traders (who can only quote prices in cents.. $0.01) and take the market making away from the legitimate seller. Many exchanges have incentives for adding liquidity. HFT can take that market making away from the legitimate buyer or seller and get the incentive. P.S. this is one reason why institutional investors are screaming angry at the exchanges for allowing HFT. HFT is stealing their exchange incentives.
* HFT does not really hurt individual investors like you and me, unless we're stupid or we try to day-trade against them or we're stupid and use market orders. Our orders are just too small and as human beings we can often see when algorithmic breakdowns create massive buying or selling opportunities and take advantage of that. I've traded these opportunities pretty much since the crash and never lost. They don't happen very often but they're fun to trade when they do. Mostly, though, for those of us who work investments in longer time spans (months or years instead of minutes or hours or days), the algorithmic and HFT trading shenanigans only create more obvious opportunities.
Institutional traders can't really take advantage of these opportunities unless the breakdown is of truly massive proportions. It's happened in a few cases but, oddly enough, the 'flash crash' the media talks about isn't one of them. It simply didn't last long enough. Action on Bo
You're missing the point, though. The i7 is not a $1000 cpu. Intel differentiates their chipsets and locks the multiplier on many of them BECAUSE THEY HAVE THE PRICING POWER TO DO SO. This is why their lowest-end cpus don't have virtualization and why their highest-end consumer cpus don't have ECC, and why they have *three* different lines of SandyBridge consumer cpus (i3, i5, i7) instead of one.
In anycase, the i5 beats the crap out of AMD's high-end offering too. Just because you can overclock an unlocked AMD phenom doesn't mean that it's actually a good idea to do so. The phenoms generate an enormous amount of heat just idling when overclocked to the point where they can compete with even just an i5, while the entire i5 system (cpu, mobo, the works) is idling at 45W.
There's just no comparison. This is why Intel has the pricing power to charge people for relatively minor differences and AMD does not.
I really do love AMD, which is why nearly every machine I've bought prior to this year was AMD-based. It was cheap to upgrade, everything cost less, and you got competitive performance out of it for the life cycle of the cpus (which tended to be less than 2 years).
That changed in 2011. All of those benefits no longer apply. AM3+ is a joke when there's no point upgrading to get the best performance out of AMDs new line of chips (which aren't even out yet for the consumer). At this point in the product cycle Intel's power consumption is so low that it easily warrants the cost of a new mobo.
You can't overclock an 1100t to run ALL 6 cores at high speed at once without water cooling and without it drawing such an extreme amount of power that it doesn't make sense to do it in the first place. I've done this myself.
You can overclock an Intel i7 to almost 5 GHz with water cooling and it will use half the power of an overclocked phenom in a similar configuration.
There's no comparison.
The only place where the phenom x 6 might win running all 6 cores is with floating point, because the 4 core x 2 hyperthreaded intel i7 only has 4 fp cores. But even there AMD tends to lose because a lot of programs written for e.g. windows don't even bother using AMD's advanced floating point features, whereas every single one of them are optimized for Intel's.
Slashdot Mirror
All dedup operations have a trade-off between disk I/O and memory use. The less memory you use the more disk I/O you have to do, and vise-versa.
Think of it like this: You have to scan every block on the disk at least once (or at least scan all the meta-data at least once if the CRC/SHA/whatever is already recorded in meta-data). You generate (say) a 32 bit CRC for each block. You then [re]read the blocks whos CRCs match to determine if the CRC found a matching block or simply had a collision.
The memory requirement for an all-in-one pass like this is that you have to record each block's CRC plus other information... essentially unbounded from the point of view of filesystem design and so not desirable.
To reduce memory use you can reduce the scan space... on your first pass of the disk only record CRCs in the 0x0-0x7FFFFFFF range, and ignore 0x80000000-0xFFFFFFFF. In other words, now you are using HALF the memory but you have to do TWO passes on the disk drive to find all possible matches.
The method DragonFly's HAMMER uses is to allocate a fixed-sized memory buffer and start recording all CRCs as it scans the meta-data. When the memory buffer becomes full DragonFly dynamically deletes the highest-recorded CRC (and no longer records CRCs >= to that value) to make room. Once the pass is over another pass is started beginning with the remaining range. As many passes are taken as required to exhaust the CRC space.
Because HAMMER stores a data CRC in meta-data the de-dup passes are mostly limited to just meta-data I/O, plus data reads only for those CRCs which collide, so it is fairly optimal.
This can be done with any sized CRC but what you cannot do is avoid the verification pass.. no matter how big your CRC is or your SHA-256 or whatever, you still have to physically verify that the duplicate blocks are, in fact, exactl duplicates, before you de-dup their block references. A larger CRC is preferable to reduce collisions but diminishing returns build up fairly quickly relative to the actual amount of data that can be de-duplicated. 64 bits is a reasonable trade-off, but even 32 bits works relatively well.
In anycase, most deduplication algorithms are going to do something similar unless they were really stupidly written to require unbounded memory use.
-Matt
Because, really, virtually nobody here actually understands what is going on. So I will throw some light on the matter.
There are two big problems with solar energy in the U.S. right now.
The #1 problem is that the U.S. is in the middle of a natural gas bonanza that started about a decade ago but really only started ramping up around 2007ish... and kept ramping up right on through the crash and is still ramping up today, with no end in sight. This has caused domestic NG production to go through the roof and domestic NG prices to fall through the floor.
Power companies switching from coal are going right to natural gas, which has 30% (or better) lower emissions and low prices for at least the next decade. It will take that long for our LNG export infrastructure to ramp up enough for world markets to relieve the downward pressure on domestic natural gas prices. In anycase, just switching to NG allows power companies to meet EPA requirements for probably the next decade (or longer), and that's without any magic technology to make it even cleaner.
Natural gas used to only be used for peaking plants, except in California where they started to also be used for base load earlier than other parts of the country. Now natural gas is being used for base load across the board (along with nuclear). Nearly all of those coal plants undergoing decommissioning are being replaced with natural gas plants, not solar or anything else.
The #2 problem with solar energy is China's overproduction of solar panels. China has no problem destroying their environment with the chemical leavenings from the production of solar panels. Chinese companies are facing the same problems that U.S. companies are facing with prices plunging, but they have a much lower cost of production. The result is that non-chinese companies basically can't compete (under ANY circumstances).. because we aren't willing to destroy our environment like the Chinese are.
This has created a massive drop in price for the bulk panels that large businesses (like Google for example) can purchase. Consumers cannot get at these prices, we just don't buy enough panels, but even so prices are going down for us too.
This, in turn, has created a huge problem for solar thermal power companies, because the price of bulk panels has dropped below or near par to the cost of constructing a solar thermal (mirror based) energy plant. This makes the ongoing cost of a solar thermal plant, which requires significant maintainence and has parts with limited life spans due to thermal cycling, higher than the near zero running cost of an installed conventional solar panel closer to the buyer (typically on the roof of the business premises).
Numerous other factors are also creating issues. Germany was the single biggest purchaser of solar panels for the last decade due to massive government subsidies. Those subsidies are now winding down, which only makes the market glut worse.
And that's the problem in a nutshell. Even the most environmentally minded person has to realize by now that it is impossible to go from coal to solar in one step. Natural gas is the only thing we have in-country that is even remotely capable of replacing coal at the generation levels required. I will applaud government investment in solar energy but anyone who thinks that solar can actually replace fossil fuel is fooling themselves.
-Matt
Well, you definitely CAN tell when one or more buffers along the path begins to fill up, because latency increases. Packet loss is not necessary and, in fact, packet loss just makes the problem worse since many TCP connections implement SACK now and can keep the bandwidth saturated even in the face of packet loss.
The ideal behavior is probably not to start dropping packets immediately... eventually, sure, but definitely not immediately. Ideally what you want to do is to attempt to shift the problem closer to the edges of the network where it is easier to fairly apportion bandwidth between customers.
Send-side bandwidth limiting is very easy to implement since TCP already has a facility to collect latency information in the returned acks. I wrote a little beastie to do that in FreeBSD many years ago, and I turn it on in DragonFly releases by default.
The purpose of the feature is not to completely remove packet buffering from the network, because doing so would put the sending server at a severe disadvantage verses other servers that do not implement similar algorithms (which is most of them).
The purpose is to unload the buffers enough such that the algorithms in the edge routers aren't overloaded by the data and can do a better job apportioning bandwidth between streams.
Our little network runs this coupled with fair queueing in both directions... that is, we not only control the outgoing bandwidth, we also pipe all the incoming bandwidth through a well connected colo and control that too, before it runs over the terminal broadband links. This allows us to run FAIRQ in both direction in addition to reserving bandwidth for TCP acks and breaking down other services. FAIRQ always works much better when links are only modestly overloaded and not completely overloaded. Frankly we don't have much of a choice, we HAVE to do this because our last-leg broadband links are 100% saturated in both directions 24x7. Anything short of that and even a single video stream screws up the latency for other connections beyond hope.
This sort of solution works great near the edges.
For the center of the network, frankly, I think about the best that can be done is modest buffering and RED and then trying to reduce the load on the buffers in the center with algorithms run on the edges (that can sense end-to-end latency). The modest buffering is needed for the edge algorithms to be able to operate without bits of the network having to resort to dropping packets. In otherwords, you want the steady state load for the network to not have to drop packets. Dropping packets should be reserved for the case where the load changes too quickly for the nominal algorithms to react. That's my opinion anyhow.
-Matt
You are absolutely right. Plus the iTouch is only $200 brand spanking new for the 8GB generation 4 model. Front and rear-facing cameras, microphone, accelerometer and Gyro. The iTouch doesn't have a GPS or compass (awwww) or 3G, so it is feature-comparable. The display has just about as many pixels but of course is considerably smaller. But yah, it does fit in the pocket and for anything larger that doesn't I'd rather have an iPad's bigger display anyhow.
To be fair, trying to read a book with an iPhone or iTouch would not be fun, but neither do I consider a 7" active display fun for reading. The only reason I have a Kindle at all is for the liquid paper display for outside reading. I'm sure many people will love the color Kindle for reading but that's not a reason for buying it vs a liquid paper kindle.
Amazon's fire is already right smack in the middle of a squeeze between Apple's small and Apple's large.
p.s. I'm not against Amazon per-say, but as an investor I just don't find their tablet compelling.
-Matt
Compare it against an 16G iPad 1 on Amazon then (I don't know of any 8G iPads). Around $350, wifi-only (about the same as on E-Bay). Around $400 for Wifi+3G.
That's $150 more for a MUCH nicer screen, double the storage, and a microphone (and thus Skype too). $200 more for 3G on top of that.
Here's the problem: Consumers who buy these things are already paying between $150-$250/mo for their phone+internet+TV services. So you're talking, literally, a 1-2 months difference in price against a readily available iPad 1 in terms of consumer cash flow.
Worse, we are talking about NO difference in price against a brand new iTouch (iPod touch) with its smaller screen but also with a microphone AND front-facing camera (meaning video calls can be done with it).
The reason Apple is making money hand over fist with their products is that they sell them in the right form factor at the right price point.
Now is Amazon selling their pad at the right price point? I personally don't think they are. Why would I want to get an Amazon pad verses a brand new 4th generation iPod touch? Let alone a used iPad 1. All the Amazon pad has going for it is a screen that's a little bigger but still not big enough. I don't think so.
-Matt
Apple has something like 85% of the pad market. Most of the android tablet vendors have volumes in the 500,000 - 800,000 range (each) whereas Apple's volumes are in the ten's of millions.
The market is something like 12 million pads this year and expected to be something like 40 million pads next year, if I remember correctly. It's a very big pie.
Both the Nook and Amazon appear to have moved away from liquid paper for their color displays. I don't have a nook but I gotta say that the Amazon B&W Kindle w/liquid paper is really wonderful when reading outside in bright sunlight, and just fine everywhere else. It's the ONLY reason why I have a kindle. I bring it along on vacations and don't even bother to bring a charging cord.
If these color devices don't have wonderful visibility in bright sunlight then I have no interest in buying them relative to buying an iPad. It's the only reason why I read books on the kindle in the first place!
Amazon might want their new gadget to compete in the Nook space, and I'm sure it will do well in that space, but if it is large enough to be considered a pad and yet can't compete in the Pad space, and its display isn't as readable as their B&W liquid paper display, then their volume improvements will only be incremental at best.
Customers often view products very differently than companies would like them to, and I think that is going to be the case for the Amazon pad.
-Matt
Deadly serious. Consumers these days either have a native HDMI/HDCP-enabled TV or they don't. If you don't, then composite is your only other choice for most device-to-TV connections. Sure, laptops have VGA output, and a lot of TVs have VGA input, but DVI connectors don't typically work with HDCP and modern flat screen TVs don't even work well with the original HDMI standard.
So the only two reliable choices are a VERY new HDMI/HDCP-enabled TV or composite video. And that's it.
Also, most video producers (Starz, HBO, etc) only allow composite output for unencrypted / non-DRM video.
Only a small percentage of the population have the most modern flatscreen TVs with native HDMI and a relatively recent HDCP implementation. So if you are an electronic gadget company those are your only two choices. That's why you still see composite inputs and outputs available on a lot of brand new equipment.
-Matt
I actually use my iPad in this configuration. Just about all cases support three positions: vertical on edge, horizontal on edge, and flat at a modest angle.
The ssh apps are really only useful when you have a wireless keybaord. Without the keyboard there isn't enough screen real-estate to be able to do real work, and you can't really type all that quickly using the on-screen keyboard. I can type at 80wpm (used to be 100 but I'm getting older now :-)) on the wireless keyboard without making an error. I can type at maybe 10wpm on the on-screen keybaord.
In an emergency I can ssh in with just the ipad and its on-screen keyboard. It is painful as hell but it works. I can do the same thing on my android phone... the pain, strangely enough, is about the same either way, even though the pad has a bigger screen. Go figure.
But with the keyboard the pad is very usable. EMACS people will be thrilled (it's a really good combo). Too bad I'm a VI guy :-).
-Matt
And Apple has Apple TV ... the TV boxes are not competitors to the pads. That is, the existence of a TV box (I have two) does not prevent a consumer from buying a pad, and the TV box's features will not cause a consumer to buy a less capable pad.
Switching video on the fly is something Apple already supports. It's a feature I don't really use all that often and it's irrelevant for most video apps that aren't movies. For movie apps, say when playing via NetFlix, then the service itself remembers where you were... sync is a fancy do-nothing name that's meaningless when video is streamed from a cloud service. Of COURSE the cloud service is going to remember where you left off.
8G of storage... arguable, but also considerably less flexible. The minimum these days is 16G and 32G is really preferable. The problem is that the biggest seller of pads are the games you can run on them. At least that is the case for Apple. It isn't JUST media services. Games are a huge portion of Apple's income stream. Games tend to be considerably larger than non-game Apps. At only 8G Amazon is seriously mis-reading the applications market for pads. Is Amazon not interested in supporting a rich gaming base for their pads?
People buy pads for portability, and part of that portability is being able to use the internet while outside of your personal Wifi range (or starbucks or your folks house or whatever). And even for the workplace... I know a great many people who use 3G at work instead of their company's wifi network due to privacy concerns.
Another huge part of that portability are location services. Maps. Navigation. Google contextual searches. Even games use location services.
Amazon's pad has a serious market placement problem. By removing (big) games, GPS (location services, contextual services, navigation), and 3G (workplace privacy, true portable use), basically the only thing Amazon has left is in-home use and kids playing pre-loaded videos in the car in a world where preloading videos are already mostly obsolete (and with only 8G even preloading videos is severely limited).
These are things that consumers expect these days. Pads without them are just toys with a very, VERY limited shelf life.
I'm sorry, I just don't see any reason to own the Amazon pad. Not at $200. Maybe at $50 I'd get a few. Not at $200.
-Matt
Resale value of 16G Wifi iPad 2 after 2 years: $350
Resale of Amazon Kindle Fire after 2 years: $0
See the problem? The iPad 1 on E-Bay RIGHT NOW can be had for ~$325. 16G, wifi-only version. 3G version only a bit more. 2 years from now the iPad 2 is going to be in the same situation.
The Kindle Fire as a media-only player is going to have a resale value of basically nothing in 2 years. They'll be on E-Bay for $50 at most.
So, yes, the Kindle Fire is in fact competing against the iPad, RIGHT NOW.
-Matt
AppleTV is similar, and costs about the same as the Roku (both are about $100). The AppleTV supports Netflix, NBA, MLB.TV, YouTube, Vimeo, Flickr, and a radio service (not sure whos) out of the box. And of course accepts airplay streams from other apple devices. However, the AppleTV 2 basically only works with HDMI-native/HDCP-enabled devices (no more composit out). I'm not sure about the Roku.
My brother loves his Roku. It's the way to go for in-home entertainment if you don't want to buy into Apple's model.
-Matt
Not just all tablets... all smart portable devices, period. Even my old iTouch 2 (oh, I'm sorry, I mean 'iPod Touch' :-)) is a media consumption gadget and plays video on its tiny screen just fine (and even over airplay to a big screen for some apps with the recent iOS update).
And for in-home there's of course AppleTV or a Roku box or any one of several other fully integrated boxes (don't buy actual flatscreens with the stuff built-into the TV, it won't be easily upgradeable).
There's nothing special about being able to play videos that gives Amazon any sort of advantage here. Their only claim to fame is that the screen is bigger than the screen on my iTouch or Android phone.
People might buy it for $200 but its questionable whether it is worth even that much, and its resale value a year or two down the line is even MORE questionable. Even though an iPad is ~$600-$900 the resale value after ~2 years is still going to be north of $350, so even cost conscious consumers with a little intelligence can buy into the series.
Amazon is competing against iPad 1's on E-Bay. Literally. I know people are pounding the dirt saying that Amazon is really competing against the color nook, and they will certainly eat a cuhnk out of the nook's sales, but I think their pad is too close to other pads (and most especially the iPad) for the consumer space to see it that way.
-Matt
What kind of stupid ass comment is that? Of course I was talking about the ipod touch... which everyone calls the itouch and has for years. And you know what? I will continue to call it the itouch going into the future.
Ok, maybe I was a year off. But, Mr idiot, you can argue all you want over stupid issues like that and it doesn't change the fact that I actually have this baby in my hand and airplay DOES WORK.
Let me repeat that: air play works. It doesn't have iPad 2's mirroring mode so it doesn't work with all apps, but it works with some, such as YouTube (and probably more later) and the audio works for everything (which is a nice option in addition to blue-tooth audio). I know because, gee, I actually have the product and use it.
The point though is that Apple has given their iTouch product good longevity by continuing to support the hardware and most of the software apps with their new OS releases. Nobody else does that. Even Android phones get long in the tooth because vendors don't update the OS images. You would be lucky to see any of these other pad or smart phone offerings stay up to date for more than ~2 years before their vendors give up on them.
This also gives Apple products tremendous resale value. The iPad 1 got airplay with the last OS update and, again, while it doesn't have the mirroring mode that the iPad 2 is able to support a lot of apps can still route video over it and more as the apps get updated. I know because, again, I have an iPad 1 (which by the way still has a $350 resale value) and I use those functions on it.
Man, the morons who post these days.... sigh.
-Matt
Pricey cellular data plan? You mean $25/month for 2G? Doesn't sound pricey to me compared to the average cable or smart-phone bill. Though you can't run video over a 2G plan (when on the road, since you have wifi at home most likely) you can pretty much do everything else without any worries.
I use my android phone to hot-spot other devices, including older iPads and other people's devices who don't have 3G when on road trips. It works... but it's unreliable as heck compared to native 3G and it's also a pretty big hassle as it requires the smart phone to be plugged into power (the battery drain is horrendous).
So as a nominal solution hot-spotting from another device works, but for real convenience that $25/month plan works a lot better.
-Matt
Hey, you could turn it around and try to justify millions of people buying a $50 pad with a paper overlay instead of a screen and complain about that being apples and oranges too.
It doesn't matter that this Amazon fire is only $200. There isn't anything there that makes it actually be useful verses products which already exist. There is no direct relationship between cost and usefulness... you could buy it and still not be happy, even if it were free.
Or to be more blunt. People who can't afford a used iPad 1 should be just as happy using the smart phone they probably already have which is a ton more capable then the amazon pad whos only claim to fame is a bigger screen.
There's another problem too. Amazon has made their pad obsolete even before it goes on sale. It has a shelf life of less than a year. That isn't $200 you are spending for a piece of equipment you are going to keep for a long time, that's $200 you are spending for a piece of equipment that you are going to throw away in a year.
You know something? I still use my 4-year-old 2nd generation iTouch. I used to carry it around with me. These days it sits in a speaker dock and mostly runs pandora. But even this 4-year-old model can run most apps and can even run video out via Airplay (though without iPad 2's mirroring mode only some apps can do that). All the Amazon pad has over this 4-year-old apple product is a bigger screen.
-Matt
Essentially. There's no reason to buy it. To read books? I have my black and white kindle with it's 4-week+ battery life to read books with, and you don't want to use an active display for serious reading anyway.
Hey, I don't necessarily need a camera but... no microphone? Are they nuts? I use google voice search all the time on all my devices.
And no GPS? So it can't reliably be used for navigation, even hot-spotted, while in the car.
And no way to export video? Gee, nice to be able to watch movies on such a tiny screen while sitting in one's own home! NOT!
So what's the new Amazon pad good for? Not a damned thing.
-Matt
You aren't required to sign up for a wireless contract with an iPad. I have a couple of the wifi + cellular versions (because the WIFI-only iPads do not support GPS), but I only turned up a data plan on one of them. $25/mo for 2G of 3G data is a nice price point. It's not as though I'm going to be streaming video all the time while outside 3G range after all, but there are a ton of apps that need internet access (like google maps) that are useful when on the road.
In anycase, while it is possible to hot-spot a smart phone to give a wifi-only pad internet access when on the road (I've done this many times), it isn't nearly as convenient and the phone needs to be connected to a power source to avoid running its battery down in 1hr while in hot-spot mode. I wouldn't buy the Amazon pad, not even for $200, without cellular connectivity.
-Matt
Ok, I understand the price point is very good, but this device doesn't even compete with the iPad 1. It basically just competes against the Kindle and the color Nook.
* No video output that I can see (vs the ipad's composite output)
* No TV box streaming solution (vs airplay)
* Is there even any blue tooth for wireless audio?
* No 3G, it is wifi only
* Only 8G of storage (vs max of 64G for iPad 1/2)
* GPS? Accelerometers? Compass? Microphone?
* Short battery life when wifi is enabled. 8 hours is only with wifi turned off. The kindle is made for off-line reading and is great with wifi turned off, but this device is worthless with wifi turned off.
Let alone compete with the iPad 2 with it's front and rear facing cameras for video calls. And both the iPad 1 and 2 have cheap ($25/mo for 2G) 3G data plans for when you aren't in wifi range. I usually don't use more than ~200MB outside of wifi range but it's damned convenient. Even with just my android phone I run Pandora and Google maps in the car all the time, and on longer trips with passengers having the iPad with google maps and internet surfing is great fun.
I use my iPad every day and I guarantee you that while the video might be nice on these devices, you really really want video output solution like Apple's AirPlay and Apple's composite outputs when you want to sit down and stream a show from your pad to your home TV.
Amazon has a ways to go.
-Matt
Well, it balances out in many respects because California also has strict efficiency requirements that reduce average household electricity use by a lot. Since manufacturers don't like to build different versions of the same model a large chunk of those efficiency improvements also benefit the rest of the country.
One good example... all those wall warts that used to weigh a ton because they contained power-hungry transformers have gone away, replaced by far more efficient switching power supplies. Most electronic gadgets which stay plugged-in and ready to go 24x7 (such as TVs and DVD players) now also have sleep modes that cut power use by a lot.
Similarly for dish washers, dryers, washing machines, and refrigerators.
And, of course, the mild weather makes a big difference too, though I should point out the statistics below are for electricity use and don't count heating oil or natural gas.
This is why California has the lowest per-capita electricity use in the U.S. (lower even than Hawaii), at 6721 kWh a year. The U.S. average is 12145 kWh a year. Wyoming has the highest use at 27457 kWh a year.
So even though electricity is expensive in California, the proportional cost is still inline with the rest of the nation. Californians are a lot more careful about their electricity use just in general.
-Matt
Grid-tie solar systems aren't going to be counted in those numbers (PG&E only meters the net, they don't collect statistics on how large a chunk the solar takes out of it). Only solar power plants were counted in those statistics.
Still, grid-tie systems probably do not reduce total system demand by a whole lot. From my read we have ~252MW of residential solar installed and ~356MW non-residential, which comes to around ~600MW. It's unclear whether the non-residential is just counting commercial retain installations or whether it is also counting solar power stations but I think it's just commercial retail (the number would be too low if it counted both). And, of course, that's just peak generation.
http://californiasolarstatistics.ca.gov/reports/agency_stats/
The wind numbers are quite impressive. There are some excellent wind corridors in CA.
-Matt
Well, more realistically, what is happening is that China is producing solar panels well in excess of demand (and doing a number on their environment in the process, but that's another story), and it's forced prices for panels down so much that all the other panel producers are dropping like flies. Has nothing whatsoever to do with U.S. regulations (unless you want U.S. creeks to run black with industrial chemicals too), or conspiracy theories about siphoning government money or anything like that.
Even worse, panel prices are now low enough to compete against large industrial-scale mirror/tower systems. So THOSE companies are also getting crushed as contracts get canceled and buyers go with panels. Think about that for a moment. It isn't that panels are less expensive than mirrors, it's that panels are now less expensive than mirrors + tower system + workforce required to keep it maintained.
-Matt
Here's a quick primer:
* HFT is basically situating your own servers in the same machine room as the exchange computers. Then they are given ammunition in the form of 'flash orders'... orders which either fill instantly or are aborted. The HFT computers basically post these flash orders as quickly as they can. When the SEC tested raising the transaction limit for HFT traders to 1 milllion transactions a second the HFT computer's order volume instantly filled the gap and ran at 1 million transactions a second.
* Most of these transactions don't actually do anything. They are 'discovery' transactions or they are simply designed to cause other computer trading algorithms to go haywire by delaying the consolidated tape.
* HFT does not add liquidity to the market. Think about that a moment... if cycling a few millions dollars continuously around in a circle added liquidity to the market then anyone could do it and the market would magically be infinity liquid. There is no argument here. Only people who don't understand how the market works thinks HFT might be able to add liquidity to it. It absolutely does not. This was obvious in several flash crashes where algorithmic trading systems tried to trade into what they thought was liquidity and the HFT algorithms withdrew that liquidity in an instant. Boom, gone. Liquidity you can withdraw in 1 microsecond is not liquidity.
* The primary mechanism for HFT to make money is to create delays in the consolidated tape in order to create wider differences between exchanges for the same ticker symbol, then use those differences to arbitrage a small profit using colocated servers at each location. In this manner HFT might actually aid in price-discovery to some degree, if not for the fact that the price 'discovery' you see on your screen is going to be a few seconds old due to the delay in the consolidated tape. So... no.
* HFT can cause normal algorithmic trading to blow up spectacularly. Normal algorithmic trading depends on prices being accurate (think about that... most computer programs go haywire if presented with inaccurate input). HFT causes prices to be inaccurate, sometimes by a lot, allowing HFT algorithms to blow up algorithmic trading algorithms. I would say they do this on purpose... but the more confusion they can cause the more profit they make.
* HFT primarily hurts institutional traders because HFT algorithms can use flash orders to probe the limits of the institutional trader's orders and then front-run them.
* HFT *IS* front-running. That's what a flash order basically allows them to do. They can come in $0.001 in front of normal traders (who can only quote prices in cents.. $0.01) and take the market making away from the legitimate seller. Many exchanges have incentives for adding liquidity. HFT can take that market making away from the legitimate buyer or seller and get the incentive. P.S. this is one reason why institutional investors are screaming angry at the exchanges for allowing HFT. HFT is stealing their exchange incentives.
* HFT does not really hurt individual investors like you and me, unless we're stupid or we try to day-trade against them or we're stupid and use market orders. Our orders are just too small and as human beings we can often see when algorithmic breakdowns create massive buying or selling opportunities and take advantage of that. I've traded these opportunities pretty much since the crash and never lost. They don't happen very often but they're fun to trade when they do. Mostly, though, for those of us who work investments in longer time spans (months or years instead of minutes or hours or days), the algorithmic and HFT trading shenanigans only create more obvious opportunities.
Institutional traders can't really take advantage of these opportunities unless the breakdown is of truly massive proportions. It's happened in a few cases but, oddly enough, the 'flash crash' the media talks about isn't one of them. It simply didn't last long enough. Action on Bo
You're missing the point, though. The i7 is not a $1000 cpu. Intel differentiates their chipsets and locks the multiplier on many of them BECAUSE THEY HAVE THE PRICING POWER TO DO SO. This is why their lowest-end cpus don't have virtualization and why their highest-end consumer cpus don't have ECC, and why they have *three* different lines of SandyBridge consumer cpus (i3, i5, i7) instead of one.
In anycase, the i5 beats the crap out of AMD's high-end offering too. Just because you can overclock an unlocked AMD phenom doesn't mean that it's actually a good idea to do so. The phenoms generate an enormous amount of heat just idling when overclocked to the point where they can compete with even just an i5, while the entire i5 system (cpu, mobo, the works) is idling at 45W.
There's just no comparison. This is why Intel has the pricing power to charge people for relatively minor differences and AMD does not.
I really do love AMD, which is why nearly every machine I've bought prior to this year was AMD-based. It was cheap to upgrade, everything cost less, and you got competitive performance out of it for the life cycle of the cpus (which tended to be less than 2 years).
That changed in 2011. All of those benefits no longer apply. AM3+ is a joke when there's no point upgrading to get the best performance out of AMDs new line of chips (which aren't even out yet for the consumer). At this point in the product cycle Intel's power consumption is so low that it easily warrants the cost of a new mobo.
-Matt
You can't overclock an 1100t to run ALL 6 cores at high speed at once without water cooling and without it drawing such an extreme amount of power that it doesn't make sense to do it in the first place. I've done this myself.
You can overclock an Intel i7 to almost 5 GHz with water cooling and it will use half the power of an overclocked phenom in a similar configuration.
There's no comparison.
The only place where the phenom x 6 might win running all 6 cores is with floating point, because the 4 core x 2 hyperthreaded intel i7 only has 4 fp cores. But even there AMD tends to lose because a lot of programs written for e.g. windows don't even bother using AMD's advanced floating point features, whereas every single one of them are optimized for Intel's.
-Matt
SSD command latency over AHCI, round trip, is around 18uS, or 0.018 ms.
-Matt