I've found SMP-related bugs in our disk drivers simply related to the fact that SSDs are so blasted fast. Other than that (and once fixed), and discounting the crucial POH bug which a firmware update fixed, I have not seen any of our SSDs drop-off the bus.
I had several crucial's with the power-on-hours firmware bug. Basically onc they have accumulated enough power on hours they brick. You can power cycle them and they will work for ~1 hour before bricking again.
Upgrading the firmware fixes the problem. I am amazed that SSD vendors don't just provide USB disk key images for firmware upgrading. It took a while for me to construct a dr-dos based boot stick (since Microsoft is so rabid about letting people format bootable removable devices these days).
But once I had a working image the upgrade process was painless. The machine booted into dr-dos from the usb stick, automatically detected the SSD, and asked if I wanted to upgrade the firmware. 60 seconds later it was done.
Generally speaking I think it's a good idea to keep the firmware up-to-date for Intel and Crucial and possibly other SSDs. Bricking issues aren't the only things they fix. Numerous firmware fixes seem to relate to flash cell failure handling or other bugs related to data integrity. On the otherhand, it's usually NOT a good idea to upgrade the firmware on an OCZ as you are just as likely to introduce new bugs as you are to fix existing ones.
Yes, and that is precisely what happens. But it means that we had to size-down the shared-memory segment in order to take into account that the machine had 7GB less memory available with that many servers running.
There is a secondary problem here... not as bad, but still bad, and that is the fact that each one of those servers has to separately fault-in the entire 6GB. That's a lot of faults. There would be 1/60th as many faults if the servers were threaded. This is a secondary problem because it only really matters in the warmup phase. Once the pages are faulted in you're done. It doesn't seem to effect performance much on DFly (mainly because our VM fault path is fine-grain locked for SMP now), but it does effect cpu use.
There are a couple of possible solutions. We can take the linux route of algorithmically reverse-mapping via higher-level data structures (vm_map_entry in our case), but this leads to inefficiencies in other areas since more page tables have to be scanned than necessary.
FreeBSD uses a much smaller pv_entry structure than we do, but that doesn't really fix the problem, it just allows one to scale a little bit bigger before you hit it again.
There's also using 2MB pages, but that has fragmentation issues if they aren't dedicated at boot time. FreeBSD has been experimenting with those.
Theoretically one could aggregate pv_entry's to cover more than one page, but that is even more complex than the normal PV scheme.
Finally there's the solution that I think I'm going to try to implement for DragonFly, which is to detect shared mappings that are multiples of the segment size which are mapped the same way (R+W), and allow those page table pages to be shared across UNRELATED pmaps. That is, attach the page table pages to the related VM objects. This solves the problem neatly with the only downside being when/if a PTE has to be removed, which since the pmap's are unrelated requires synchronizing the invlpg instruction across all cpus on the system to avoid various cpu bugs and races.
This latter solution would work equally well for SysV SHM and mmap(... MAP_SHARED), and work whether a server forks or threads. It lets us keep our fine-grained pv_entry abstraction. This is the one I'm going for.
I don't think this is true any more. Threads are light weight... that's the whole point. They all share the same pmap (same hardware page table). Switching overhead is very low compared to switching between processes.
The primary benefit of the thread is to allow synchronous operations to be synchronous and not force the programmer to use async operations. Secondarily, people often don't realize that async operations can actually be MORE COSTLY, because it generally means that some other thread, typically a kernel thread, is involved. Async operations do not reduce thread switches, they actually can increase thread switches, particularly when the data in question is already present in system caches and wouldn't block the I/O operation anyway.
There is no real need to match the number of threads to the number of cpus when the threads are used to support a synchronous programming abstraction. There's no benefit from doing so. For scalability purposes you don't want to create millions of threads (of course), but several hundred or even a thousand just isn't that big a deal.
In DragonFly (and in most modern unix's) the overhead of a thread is sizeof(struct lwp) = 576 bytes of kernel space, +16K kernel stack, +16K user stack. Everything else is shared. So a thousand threads has maybe ~40MB or so of overhead on a machine that is likely to have 16GB of ram or more. There is absolutely no reason to try to reduce the thread count to the number of cpu cores.
--
There are two reasons for using lock memory for a database cache. The biggest and most important is that the database will be accessing the memory while holding locks and the last thing you want to have happen is for a thread to stall on a VM fault paging something in from swap. This is also why a database wants to manage its own cache and NOT mmap() files shared... because it is difficult, even with mincore(), to work out whether the memory accesses will stall or not. You just don't want to be holding locks during these sorts of stalls, it messes up performance across the board on a SMP system.
Anonymous memory mmap()'s can be mlock()'d, but as I already said, on BSD systems you have the pv_entry overhead which matters a hell of a lot when 60+ forked database server processes are all trying to map a huge amount of shared memory.
Having a huge cache IS important. It's the primary mechanism by which a database, including postgres, is able to perform well. Not just to fit the hot dataset but also to manage what might stall and what might not stall.
In terms of being I/O bound, which was another comment someone made here... that is only true in some cases. You will not necessarily be I/O bound even if your hot data exceeds available main memory if you happen to have a SSD (or several) between memory and the hard drive array. Command overhead to a SSD clocks in at around 18uS (verses 4-8mS for a random disk access). SSD caching layers change the equation completely. So now instead of being I/O bound at your ram limit, you have to go all the way past your SSD storage limit before you truly become I/O bound. A small server example of this would be a machine w/16G of ram and a 256G SSD. Whereas without the SSD you can become I/O bound once your hot set exceeds 16G, with the SSD you have to exceed 256G before you truly become I/O bound. SSDs can essentially be thought of as another layer of cache.
Here's the problem in a nutshell... any memory mapping that is NOT a sysv shm mapping with the use_phys sysctl set to 1 requires a struct pv_entry for each pte.
Postgres servers FORK. They are NOT threaded. Each fork attaches (or mmap in the case of this patch) the same shared memory region, but because the processes fork instead of thread each one has a separate pmap for the MMU.
If you have 60 postgres forked server processes each mapping, say, a 6GB shared memory segment and each trying to fault in the entire 6G, that is 60 x 6GB worth of page tables, around 87 MILLION page table entries, and thus 87 MILLION struct pv_entry structures.
This is why it blows up. That's just too much. The sysv use_phys hack removes the need for a pv_entry structure. Thus removing several gigabytes of memory that the kernel would otherwise have to allocate to track all those pv_entry's.
There are optimizations that can be done to reduce memory use in the BSDs, but the main problem here is that postgres is using fork() instead of threads. If it used threads the management overhead would be 1/60th what it is now and having 1/60'th the pv_entries would not be a big deal.
* First, on NCQ. *ALL* modern SATA hard drives implement NCQ and have ~31 tags.
* Bridge chips. *NO* modern motherboard uses a bridge chip any more. Bridge chips used on devices is another matter. Some devices still use bridge chips. Many DVDs and CDs used bridge chips (which is why early SATA DVDs and CDs were so broken), though I think that is finally dying out. The most famous was one of the OCZ SSD models which used a bridge chip to tie two controllers together. The controllers could handle ~31 tags, the bridge chip could not so the host probe would indicate no NCQ support. Also, multi-physical-interface devices such as netbooks (hard drives in a 'book' with SATA, USB, and Firewire interfaces)... those generally use bridge chips that often don't support NCQ.
* BIOS 'RAID'. So-called soft-raid. This is fake-raid. It isn't real. Don't expect it to actually work properly in a failure case. It's still talking to the AHCI controller, it's just hiding the fact from the OS. BIOS soft-raid 'controllers' are usually pretty horrible, avoidance is best.
* On data loss from caches. It isn't the caches that you need to battery-back (unless you are REALLY dependent on fsync() times in e.g. a database application). I think the trend is more towards off-host cache redundancy these days because it gets you to approximately the same place without the need for expensive gear. A large percentage of modern filesystems use write barriers and have no problem handling drive cache loss.
That isn't the problem. It's the physical power to the device being dropped while the write IO is in progress that is the problem. Devices, particularly SSDs but also many HDDs, cannot retire meta-data (for a SSD) or even the current sector (for a HDD) if a sudden loss of power occurs. In addition, a sudden power loss on a HDD can cause UNRELATED sectors to fail depending on how the HDD is writing (whether it is doing a full-track write or not). This can lead to serious corruption of the drive, even outright destruction. I've had quantum drives go through sudden power loss during a write with HUNDREDS of sectors lost instead of just one or two. That was a while ago, but it was still in the SATA-era... they were modern drives.
So for UPS/power concerns the only thing that really really matters is that the drive remain powered for at least a second or two. Even that is no guarantee. Barring that you want redundant storage on separate UPS's so someone kicking a plug out or crow-baring the UPS's output doesn't take you out.
* Super-caps... e.g. as Intel advertises on newer SSDs. These are primarily to retire meta-data so the SSD doesn't brick when you power it back up. Intel SSDs have very tiny ram caches so it might be able to retire those too, but most other SSDs have larger ram caches and no super-cap has enough suds to retire the entire cache. The idea is to not end up with a partially corrupt sector here, not necessarily to be able to retire the entire ram cache. Also, SSDs often do background cleanup when idle so not having any pending writes to an SSD doesn't make it safe, necessarily. This is what the super-cap idea primarily addresses.
Battery-backed ram comes under the same category. Well, in this case perhaps super-cap-backed ram (good for maybe ~a week to ~a month with low power static ram). Lots of options here that don't cost an arm and a leg, but again what matters the most is that the drive be able to retire whatever it is currently writing and not necessarily whatever is currently in its caches.
* SAS vs SATA. There is lots of talk about this all the time. I've never noticed any real difference in reliability, probably because the only real difference between the two is firmware. Drive vendors will talk-up using more robust parts but I believe that about as much as I believe that the moon is made out of cheese. There are so few components in HDDs that it is fairly difficult to differentiate consumer from enterprise these days.
That is, NCQ (for SATA) does not have enough command slots available. Only ~32 or so per port. The SAS stuff works a bit better but I think still limits you to ~32 slots per target device (instead of ~32 per port on the driver side).
So what happens if you try to use the so-called media completion feature is that write operations eat up all your tags and crowd out the far more important read operations. This makes the feature almost worthless in my view. Not to mention that there is no way to determine, with the SATA spec, that the drive actually honors the bit. Just like the old ATA stuff, drive vendors play fast and loose with the AHCI/SATA specs in order to try to force people to use the far more expensive SAS drives, even though the actual hardware is the same.
What we do in DragonFly is split available tags into read-dedicated tags and write-dedicated tags, approximately 3:1. Writes only last as long as it takes the device to copy the data to its internal ram caches, so there's no real need to reserve more than a few tags for writing. This leaves the remaining tags available for reading.
If you don't do this what happens is that you can stall your read I/O by saturating all available tags with write IOs... the writes get retired instantly to the drive's ram cache UNTIL that cache is full, then suddenly the newly issued write tags stall and sit there until the drive can flush some data out. If you don't control how many tags you use for pending writes, you can completely lock out read activity. A simple 'dd' to a file... hell, a simple file copy, for a large file, is big enough to exhibit this behavior.
This leaves even fewer tags available for writing. At best, if you want to maintain read performance, you can't really use more than ~8 or so tags for writing.
For a SSD maximum performance can be achieved with even fewer tags since in this case all you are doing is soaking up the command overhead by pipe-lining the IOs. Meaning that 2 write tags is sufficient, 3 to be safe.
All of this pretty much precludes being able to use the media completion bit with AHCI/SATA and still have good performance. To really be able to make use of such a bit one needs to support ~256 tags per target... that's to the actual physical device, NOT ~256 tags to the driver or ~256 tags to the SATA/SAS controller on the host.
In DragonFly, for HAMMER1, there were numerous ordering constraints that requires at least two DISK FLUSH commands per volume header sync. For HAMMER2 there are essentially no ordering constraints except for the volume header write itself so only one DISK FLUSH is required to create a recovery point. In both cases all the writes leading up to the required demarcation point could complete in any order. When combined with read:write tag reservation performance remains good even though the DISK FLUSH doesn't operate NCQ.
For SATA only read and write commands can use NCQ. All other commands require serialization and cannot run concurrently with NCQ commands, including unfortunately the DISK FLUSH command. You can blame Intel for this bit of stupidity... they intentionally broke the AHCI/SATA spec in order to artificially differentiate between SATA and SAS, so drive manufacturers could pump up the prices for SAS drives (even though both the hardware and the physical attachment is exactly the same). Intel broke the AHCI chipset spec in other ways to differentiate it, particularly when it comes to error recovery. They'll tell people that it was to 'maintain compatibility with the ATA command set' but IMHO they are lying. Some of the things Intel did in the AHCI spec were just phenomenally stupid. It is still much, much better than the ATA stuff, but they had a chance to make something really robust and blew it.
For example, with AHCI/SATA error recovery requires serialization, which means that if you use a port multiplier and one drive is having problems you have to stall out I/O to ALL OTHER DRIVES while you deal with the one that is having problems.
That's really all that matters these days. Everything else can be easily corrected in software. Chromatic aberrations are more difficult to deal with nicely.
The reason is that the exchanges get quite a bit of revenue offering these direct-connect services to the HFTs. Basically it's a backdoor way of stealing revenue from the real market makers. The HFTs steal the revenue from the real market markers by interposing themselves in the middle of many of the transactions (thereby getting the market making fee from the exchange), and then turn around and pay the exchanges a fee for the privilages. Thus the exchanges are able to offer the market making fee but get quite a lot of it back in the fees from the HFTs.
It's obvious to anyone with a brain that HFTs do not add any liquidity to the market. I mean, come on, how stupid are we? A small company worth $100M-$1B doing high frequency trading is not going to add any liquidity to the market. All they do is add phantom volume that looks like liquidity right up the moment where a real event happens, then it disappears, PUFF, just like that.
It comes down to the idiot regulators just not caring about making the markets fair to everyone.
In some sense this has been hurting the exchanges too, as the larger players have slowly pulled their trading out of the public exchanges and into dark pool exchanges that don't play the same games. The public exchanges will still be closely priced to the dark pool exchanges due to arbitrage, but retail investors definitely get the short end of the stick in this new reality. That isn't the real problem... the real problem is that the HFTs add so much noise to the system (literally their entire M.O. is to try to force the exchanges to lag other players) that it can result in excessive volatility and screw up price discovery to the HFT's advantage.
Another reality, however, is that market making margins have gone down to $0.01 over the last decade. Before the exchanges went to the decimal system the middle men were pulling down much more revenue... higher margins. This is a huge benefit to the retail investor regardless of the other shenanigans going on. But the psychology is such that it gets lost in the wind when investors and traders see specific participants being given an unfair, leg-up advantage.
There is no 'solar tech'. China mass-produces standard panels for the most part, and those plants produce large quantities of toxic byproducts in doing so which they pretty much just dump into the environment. The solar industry in China also seriously over-produced their panels and wound up with piles of them as other countries (aka Europe, Germany in particular) pulled back subsidies for various reasons.
Solar panel companies in the US simply cannot compete against that, not unless you want to create environmental destruction on a large scale.
The second big nail in the coffin was the crash in natural gas prices. PV panels cannot compete against $3 NG (winds up being ~$4-$5 for an industrial consumer). Hell, not even COAL can compete with NG at current prices.
So this isn't entirey Obama's fault. In fact, not really his fault at all. The subsidies were formulated at a time when other energy inputs were far more costly and before China really started dumping mass quantities of panels onto the market. I'd blame him for trying to continue the subsidies in the face of NG's dominance, but it was a reasonable course of action when it was first initiated.
The raw output from the tar sands is way too heavy to be refineable, it still needs to be diluted with light natural gas liquids first... the same stuff they use to dillute it for pipeline transport, minus a few refinement stages (btw, the US exports some of the lighter NGLs coming out of the shales to Canada for this purpose).
But it makes no sense to refine the output up there far away from consumers. What are they going to do, truck the gasoline down several thousand miles? They'd go out of business trying to do that.
There is much more to the fossil fuel infrastructure than power generation or gasoline refinement. There are a multitude of stages involved and a multitude of outputs (gasoline only being one), and those outputs have to be piped as well. The pipelines for those outputs aren't up north. Trucking is out of the question for these long hauls (not unless you want $15/gal gasoline!).
Trucking is used primarily for short hauls, until local gathering pipeline infrastructure is built, and rails are used for medium-hauls of the liquids, again, only until the pipeline is built to replace it, because of the cost effectiveness of a pipeline. Natural gas can't be trucked or railed in anywhere near the volumes required, it is just too expensive. It has to be piped.
So, basically, it doesn't make any economic sense to try to relocate the supporting infrastructure. Doing so would flush somewhere north of half a trillion dollars down the toilet.
This isn't correct anyway. Peak domestic energy use is in the early evening, when everyone has gotten home and are doing the wash, making dinner, etc. Not during the day.
It's a big problem because power companies that want to charge variable rates would wind up short-changing people with PV installations whos primary output occurs earlier.
I think you are a bit confused over the scale involved, but the main problem is transport cost when it comes to NG. The producers are already underwater with curent well-head NG prices for pure NG plays (called dry-gas wells), any sort of transport other than a pipeline would bankrupt them.
It just seems odd. This is more a business article than anything else, and there is nothing new and cool about buying rail cars.
Our domestic pipeline infrastructure has been on a building spree for a decade. If any of you are investors, that's been the basis for the Oil&Gas MLP buildout that has been maturing at a very fast clip since the mid-2000's, continued right through the crash, and continues to mature at a modestly fast clip today and probably for another 10 years at least before the core-buildout slows down.
Generally speaking transport for OIL and NGLs (Natural Gas Liquids) can start out in tankers and rail cars but ultimately cost efficiency requires a pipeline to be built. And you have no choice for natural gas... its pipeline or nothing pretty much since compression to CNG or LNG levels is way too expensive (and way too dangerous) for domestic transport.
But it takes several years to build a long pipeline, costs billions of dollars, and requires both shippers and receivers to enter into long term 10-year+ contracts with guaranteed volume flow or investors wouldn't finance the pipeline in the first place. Because no actual revenue flows until the pipeline is complete.
There are a dozen major producing areas but in layman's terms the bottleneck is mainly in the North->South direction these days. EastWest has capacity now (though numerous major cities on the east coast still have bottlenecks). Existing pipelines in the north-south direction are essentially maxed out.
The Keystone pipeline saga is your typical talking-head/exaggerated/public-unaware crap. Pipelines criss-cross the U.S. already, there are already numerous (but maxed out) pipelines coming down from Canada all the way to the gulk, and Canada is a major trading partner whos major oil and gas reserves are essentially land-locked. Sure, they have some transport to the coasts for export, but they need to be able to drop down through the border into the U.S. markets and we also have an export market of our own going northward of light NGLs which the Canadians use for a multitude of purposes in their oil-sands operations. It's as much a diplomatic issue with our northern neighbor as it is anything else.
It is true that the direct effect is to reduce your personal wealth.
However, at the same time, the inflation that is creating the personal tax issue also allows your government to pay off debt (and you too if you have debt) at a lower cost or to provide services that you benefit from. Government debt is a proxy to the debt owned by its citizen and services rendered to its citizens, just somewhat indirect.
So while it is annoying from a personal standpoint (because you weren't the one who decided to incur the government debt), a good chunk of the losses incurred by this mechanism are returned indirectly through (e.g.) a better economy.
Also, the tax is not 100%. You don't lose so much wealth that you are back at square one vs having placed the money in your local currency. And compounding still works. Also, again, having cash in any currency is not an 'investment', it's taking work product out of the economy and storing it, which shouldn't entitle the holder to the same value years down the line vs doing something else with the cash (like investing it).
I don't think anyone can make an argument that this mechanism should change because the behavior modification it implies is only beneficial to the country, economy, and population as a whole. Doesn't mean we have to like it, but it isn't evil either.
I understand the concept but it's still complete nonsense. There are a huge number of problems with the concept of bitcoins.
First, scarcity is not a measure of value in of itself unless it is attached to something desirable. A bitcoin is nothing but a bunch of numbers. Arguing that its value is what people are willing to trade for it (with a floor based on the electricity cost) is insufficient. In the stock market there have been multiple instances of this sort of effect when backers of ETFs or ETNs decide to limit the maximum number of issued units, which causes those vehicles to become free-floating.
This is a well known effect that has been observed in the stock market. From what I can tell, bitcoins are no different. Without a regulating entity bitcoins become detached from essentially everything and free-float, and that is basically what we've seen. The problem is that this makes them highly unstable (and we've seen this too).
Even in-game currencies have an economic attachment... the time-value of the gamer (and is far more consistent than the time-value of a cycle of cpu time). Fiat currencies have dozens of major economic attachments. bitcoins have nothing.
Second is the so-called cost of generating a new bitcoin. This is supposed to be the regulating entity but, in fact, it is only HALF of a regulating entity. It puts a floor on the price but no ceiling. A major problem with this is that even the floor is not a fixed value because the work-product required to produce the bitcoin is highly variable. The cost of power is highly variable. The efficiency of cpus and gpus is highly variable.
Finally, the lack of a ceiling on the value attached to a bitcoin and, more importantly, the entire concept of a bitcoin increasing in value over time creates instability. Currencies which go up in value are just as bad as currencies which go down in value. The reason is that holding a currency, just like holding a commodity, removes the related work-product from the economy and sets it aside. When work-product is removed from an economy you effectively reduce your investment in the economy. This might be fine from the point of view of a bitcoin trader but it's a disaster for anyone trying to use bitcoins to drive an economy, which is one reason why nobody takes it seriously. You can't drive an economy with a commodity that increases in value because it reduces investment in the economy you are trying to drive. Reduced investment equals reduced jobs, lost efficiencies, and an inability to compete with other economies.
We have seen this effect in numerous economies, particularly since the crash as currency traders have shifted vast amounts of cash into 'safe havens'. The result? The so-called safe havens destabilized (hard currency or not) and their governments were forced to print money to prevent the rapidly increasing value of those currencies from destroying their economies. Think about that for a moment! The Swiss have been especially impacted, as have other hard currencies (Australian dollar, Canadian dollar, etc). And, of course, so has the US dollar as international investors flee Europe (but the US was easing already so it played into our hand).
This isn't true. Any bitcoin transaction that involves translation to or from another currency is subject to a trading spread and liquidity issues.
Bitcoins have horrible spreads and essentially no real liquidity compared to any other currency. The spreads are conveniently not displayed on most sites.
Thus any transaction held in bitcoin form for only a short duration is going to have huge overheads. Efficiencies can only be gained by holding a fairly large bitcoin balance so most of your activity cycles within the bitcoin universe and doesn't get translated in or out. That, in turn, makes you subject to another huge problem with bitcoin... that being the ridiculously high volatility in value.
This makes bitcoins basically worthless for any real business.
It depends what you mean by Tangible. I assume you are talking about fiat currency here. But everything, even fiat currencies, have some tangibility. Don't be fooled by people who believe that a fiat currency will just collapse out of the blue for no reason, or because inflation might occur (the most typical reason stated).
There mere fact that we are using a fiat currency does not implicate a particular weakness verses other currencies. It does not necessarily make an economy more vulnerable to mismanagement. Strong currencies can create huge problems in mismanaged economies, too. Look at the Euro, for example, which until very recently was managed like a hard currency... a very strong currency for the last few decades. Now observe Ireland, Spain, Italy, Greece, and numerous other countries. Spain at 25% unemployment! Clearly the currency didn't prevent mismanagement within those governments, and the US has (so far) fared much better, so nobody should even remotely operate under the assumption that the fiat nature of the dollar is in any way responsible for our current predicament.
The dollar bill, or any fiat currency, is backed by the entire economy of the issuing nation. Literally. Why? Because nearly all goods and services within the issuing nation are traded using that currency. Because taxes are paid in that currency. Because use of that currency is far more convenient than other things. Because the government might have capital controls in place (Argentina, for example). Because a large chunk of short-term working capital is necessarily kept in that currency. Because it is convenient (think about the extra overhead a vendor would incur trying to use barter... what does he do with the 30 chickens he was just handed? How does he store that gold safely so it doesn't get stolen? How long will those eggs last before they go bad?).
As long as the above is true then the currency has value. Real value. It isn't fake. I can go out and buy a hot-chocolate from Starbucks right now, or a boat, or a house, using dollars. Real value.
People who assume that a fiat currency is worthless miss out on this basic fact, but more importantly people who try to contrast fiat currencies with other currencies within the developed nations also forget that ANYONE these days can have a brokerage account and invest in the domestic and/or world economy (investment in most large-cap domestic names these days is basically an investment in the world economy).
These investments have not been any more stable than fiat currencies. In fact, in the world today, HARD currencies are considerably less stable as well. The dollar is actually one of the more stable currencies (the Yen is probably the most stable of them all, at least up until now). Pick your poison. Anyone with a brokerage account can move their wealth out of the fiat currency system, but shouldn't expect the results to by any more stable.
Currencies remove some risks, but add others. People use currency (fiat or otherwise) as a vehicle for trade precisely BECAUSE currencies are very stable over the short-term. The period of time the transaction operates within the currency is typically short, and thus not subject to inflation.
The real problem with a fiat currency occurs when you try to use it store your hard-earned work-product over a LONG period of time. And here we aren't talking about people living pay-check-to-pay-check (for them the currency is ridiculously stable because they have no savings). Too many people these days believe that, somehow, magically, they should be granted the right to 'save' their money simply by keeping it in cash... keeping it as the fiat currency, effectively removed from the economy, verses investing it in the economy. That's ass-backwards.
Similarly when we look at a hard currency, or any currency which tends to appreciate in value even relative to average base commodities (bitcoin comes to mind)... those currencies encourage people to store their work product in the currenc
I think you have a point here, but at the same time you are talking about an endemic problem with the very small numbers of people at the margins of society who are both poor and also completely or nearly completely undocumented AND have no wherewithal to fix the problem.
Even without documentation you can certainly get on the train, even if it comes to petitioning the courts or finding someone to counter-sign the account, but the people in those situations are there for other reasons. Fixing it won't fix their situations.
You can't base an entire society's working methodology on the needs of the people at the fringe. It's a lost cause no matter how first-rate the society is. Technology is going to move on regardless, and those people are there for more reasons than simply not having any money.
So don't try to change conversation. This isn't about the relatively few who are missing documentation, this is about the relatively MANY who don't have even the minimal sense to manage their finances properly.
Energy fluctuates in value just like everything else, with time, location, and quantity as major components of that value. It might not be subject to inflation per-say, but energy fluctuates in value far, far, FAR more than e.g. dollars. By several orders of magnitude.
So energy would make a rather poor basis for a currency.
Inflation is not really an issue for a currency used as a trading medium, it is only an issue if you try to 'save' your wealth in that currency. There is a trade-off there... people accept inflation in trade for medium-term stability, whereas with a commodity you avoid inflation but must accept short and medium term volatility in value (medium term being ~20 years). Since people have limited life spans, choosing one vs the other is NOT a slam dunk.
Yup, there they are, right next to the lotto machine:-)
Of course, pre-paid debit cards are a pretty big scam unto themselves with the fees involved in buying and reloading the cards. They cater to the stupidity of people.
A standard credit or debit card with a branch-less bank, plus ~2-3 free ATM reimbursements per month, is the best value for anyone capable of maintaining a minimum balance.
I don't use cash at every opportunity but I don't trust every vendor I use with my credit card either, so I always keep some cash in my pocket and tend to make smaller purchases with that cash. I might visit an ATM once a week or so, so the actual physical cash I have on-hand is only 'live' for a week.
In fact, I rarely use checks now either. All my bills use my bank's free bill-pay service. 100% electronic. The vendors often don't even have to handle a physical check, it just goes between banks.
Even so, most of my regular purchases are simply with a credit card. Swipe, squiggle, and its done. Fast and convenient. A lot of vendors don't even bother with a signature for small purchases any more (e.g. Starbucks or Whole Foods), but even when they do I just doodle (and have for 20 years). No reason to use my actual signature. I've even started experimenting with a big 'X' recently and clerks don't care. Smiley-faces are more problematic, though.
Honestly, why should I care whether something is traceable or not? I kinda like being able to sort my credit card statements by purchase to determine how much I'm really spending on things.
In anycase, bank power is overblown... typical talking head media hype. It's a business just like everything else, and as more and more people go branch-less the choice and competition is higher than almost any other industry.
Kind of a ridiculous statement. Bank accounts and debit cards are trivial to get, even if you never had one before. Credit cards are also fairly easy to get when you are starting out, just don't expect the bank to give you a high limit.
Fee avoidance is also trivial, you simply keep a minimum amount of cash in the account. There are plenty of banks (particularly branch-less banks) with low minimums.
The problem that most people have these days is that they are addicted to credit, can't manage money, and aren't willing to learn how to. Either they will maintain a credit card balance and be in-hoc for their entire lives paying interest, or they will constantly border on a $0 account balance, overdraw their accounts every few months trying to juggle it, and then complain that they are too poor (here's a hint: If you can make do with your account always bordering on $0 then you can make do with your account always bordering on $1000... you aren't actually spending any less each month by maintaining a higher balance).
The whole cash/cash-less argument was dead years ago. People who have so little money that they can't make use of existing low-fee/zero-fee mechanisms are hardly going to have enough money to make effective use of a more esoteric currencies like bitcoin, let alone be smart enough to make better use of such a mechanism verses existing and more convenient government sponsored mechanisms.
Has anyone looked at the spreads for converting bitcoin to a real currency? They are insane. The people making the money are the people working the spreads, not the people trying to use bitcoin as an actual currency.
Similarly, there is absolutely no need to keep your money in $USD if you don't want to. It's a fine form of exchange, I use it every day, but these days you can invest your money anywhere... in equities, in foreign currencies, in commodities, and only keep a few months worth of 'cash' around for immediate needs. A few months burn worth of cash is essentially not subject to inflation. The currency is almost irrelevant at that point except for the fact that you have to have it in a currency you buy things with to avoid translation fees.
Of course, anyone who hasn't done this is probably going to be a bit surprised at how volatile the relative value of EVERYTHING moves around these days. Gold, USD, Euro, an equity, a piece of a mine, other commodities, whatever... everything has a relative value based on supply and demand. Nothing is immune. There is no such thing as 'intrinsic' value for anything (not even bits of 'energy' as someone tried to suggest). People who invest in gold, for example, not knowing how volatile its value is, can suddenly find their investment worth less in dollar terms while their burn rate in dollars doesn't change, over a few days or weeks or months, and quickly realize the fallacy of relying on some idiotic notion of absolute or intrinsic value.
p.s. recently drug gangs and cartels have begun using bit-coin to launder money. Lets see how long it lasts as a viable currency when the larger transactions wind up being made by criminals instead of normal users. If you want practical untraceability then cash from an ATM works just fine. Theoretically its traceable but after a couple of transactions it's effectively impossible to track. Just because something is theoretically possible doesn't mean there are government drones hiding behind every flower and bush doing it.
So far IPMI has managed to survive all the crap vendors try to add to it. Chalk one up for standards! It's gone through some growing pains (the original standard was badly broken and precursors tried to share the ethernet port on the mobo), but has settled down into a usable implementation in the last few years, as long as you only use open-source IPMI tools like 'ipmitool'. There are typically still hicups with the video sniffing implementations (particularly when the OS switches video modes), but serial port forwarding seems to work quite well.
In anycase, IPMI has been a godsend for those of us with smaller server installations. It removes the need for an addressable PDU and removes the need for a console server. It removes a lot of cables.
Once the BIOS is setup you don't need to connect a keyboard or monitor to the machine ever again, even when you are physically in the machine room. You just plug you laptop into the switch or Wifi and poof.
There are numerous ways to secure the IPMI net. If configured properly IPMI 2.0 has reasonable security but still can be DOS'd. The best thing about the standard is that it is trivial to insert machines, IP filters, and other things between the IPMI network and the rest of the world but the fact that it can connect directly to the internet also ensures that those extra gadgets are going to be fairly cheap in order to compete. Always only use the IPMI 2.0 UDP protocol... if the thing has a web server or ssh access (easy to test), turn those off straight away.
Mobos with IPMI tend to cost ~$20-$50 more and the IPMI module itself (when not built in) typically costs $30-$60. Considering what you get for it, it's damn cheap. IPMI has already saved me thousands of dollars worth of equipment, gas, and time.
Slashdot Mirror
I've found SMP-related bugs in our disk drivers simply related to the fact that SSDs are so blasted fast. Other than that (and once fixed), and discounting the crucial POH bug which a firmware update fixed, I have not seen any of our SSDs drop-off the bus.
-Matt
I had several crucial's with the power-on-hours firmware bug. Basically onc they have accumulated enough power on hours they brick. You can power cycle them and they will work for ~1 hour before bricking again.
Upgrading the firmware fixes the problem. I am amazed that SSD vendors don't just provide USB disk key images for firmware upgrading. It took a while for me to construct a dr-dos based boot stick (since Microsoft is so rabid about letting people format bootable removable devices these days).
But once I had a working image the upgrade process was painless. The machine booted into dr-dos from the usb stick, automatically detected the SSD, and asked if I wanted to upgrade the firmware. 60 seconds later it was done.
Generally speaking I think it's a good idea to keep the firmware up-to-date for Intel and Crucial and possibly other SSDs. Bricking issues aren't the only things they fix. Numerous firmware fixes seem to relate to flash cell failure handling or other bugs related to data integrity. On the otherhand, it's usually NOT a good idea to upgrade the firmware on an OCZ as you are just as likely to introduce new bugs as you are to fix existing ones.
-Matt
Yes, and that is precisely what happens. But it means that we had to size-down the shared-memory segment in order to take into account that the machine had 7GB less memory available with that many servers running.
There is a secondary problem here... not as bad, but still bad, and that is the fact that each one of those servers has to separately fault-in the entire 6GB. That's a lot of faults. There would be 1/60th as many faults if the servers were threaded. This is a secondary problem because it only really matters in the warmup phase. Once the pages are faulted in you're done. It doesn't seem to effect performance much on DFly (mainly because our VM fault path is fine-grain locked for SMP now), but it does effect cpu use.
There are a couple of possible solutions. We can take the linux route of algorithmically reverse-mapping via higher-level data structures (vm_map_entry in our case), but this leads to inefficiencies in other areas since more page tables have to be scanned than necessary.
FreeBSD uses a much smaller pv_entry structure than we do, but that doesn't really fix the problem, it just allows one to scale a little bit bigger before you hit it again.
There's also using 2MB pages, but that has fragmentation issues if they aren't dedicated at boot time. FreeBSD has been experimenting with those.
Theoretically one could aggregate pv_entry's to cover more than one page, but that is even more complex than the normal PV scheme.
Finally there's the solution that I think I'm going to try to implement for DragonFly, which is to detect shared mappings that are multiples of the segment size which are mapped the same way (R+W), and allow those page table pages to be shared across UNRELATED pmaps. That is, attach the page table pages to the related VM objects. This solves the problem neatly with the only downside being when/if a PTE has to be removed, which since the pmap's are unrelated requires synchronizing the invlpg instruction across all cpus on the system to avoid various cpu bugs and races.
This latter solution would work equally well for SysV SHM and mmap(... MAP_SHARED), and work whether a server forks or threads. It lets us keep our fine-grained pv_entry abstraction. This is the one I'm going for.
-Matt
I don't think this is true any more. Threads are light weight... that's the whole point. They all share the same pmap (same hardware page table). Switching overhead is very low compared to switching between processes.
The primary benefit of the thread is to allow synchronous operations to be synchronous and not force the programmer to use async operations. Secondarily, people often don't realize that async operations can actually be MORE COSTLY, because it generally means that some other thread, typically a kernel thread, is involved. Async operations do not reduce thread switches, they actually can increase thread switches, particularly when the data in question is already present in system caches and wouldn't block the I/O operation anyway.
There is no real need to match the number of threads to the number of cpus when the threads are used to support a synchronous programming abstraction. There's no benefit from doing so. For scalability purposes you don't want to create millions of threads (of course), but several hundred or even a thousand just isn't that big a deal.
In DragonFly (and in most modern unix's) the overhead of a thread is sizeof(struct lwp) = 576 bytes of kernel space, +16K kernel stack, +16K user stack. Everything else is shared. So a thousand threads has maybe ~40MB or so of overhead on a machine that is likely to have 16GB of ram or more. There is absolutely no reason to try to reduce the thread count to the number of cpu cores.
--
There are two reasons for using lock memory for a database cache. The biggest and most important is that the database will be accessing the memory while holding locks and the last thing you want to have happen is for a thread to stall on a VM fault paging something in from swap. This is also why a database wants to manage its own cache and NOT mmap() files shared... because it is difficult, even with mincore(), to work out whether the memory accesses will stall or not. You just don't want to be holding locks during these sorts of stalls, it messes up performance across the board on a SMP system.
Anonymous memory mmap()'s can be mlock()'d, but as I already said, on BSD systems you have the pv_entry overhead which matters a hell of a lot when 60+ forked database server processes are all trying to map a huge amount of shared memory.
Having a huge cache IS important. It's the primary mechanism by which a database, including postgres, is able to perform well. Not just to fit the hot dataset but also to manage what might stall and what might not stall.
In terms of being I/O bound, which was another comment someone made here... that is only true in some cases. You will not necessarily be I/O bound even if your hot data exceeds available main memory if you happen to have a SSD (or several) between memory and the hard drive array. Command overhead to a SSD clocks in at around 18uS (verses 4-8mS for a random disk access). SSD caching layers change the equation completely. So now instead of being I/O bound at your ram limit, you have to go all the way past your SSD storage limit before you truly become I/O bound. A small server example of this would be a machine w/16G of ram and a 256G SSD. Whereas without the SSD you can become I/O bound once your hot set exceeds 16G, with the SSD you have to exceed 256G before you truly become I/O bound. SSDs can essentially be thought of as another layer of cache.
-Matt
Here's the problem in a nutshell... any memory mapping that is NOT a sysv shm mapping with the use_phys sysctl set to 1 requires a struct pv_entry for each pte.
Postgres servers FORK. They are NOT threaded. Each fork attaches (or mmap in the case of this patch) the same shared memory region, but because the processes fork instead of thread each one has a separate pmap for the MMU.
If you have 60 postgres forked server processes each mapping, say, a 6GB shared memory segment and each trying to fault in the entire 6G, that is 60 x 6GB worth of page tables, around 87 MILLION page table entries, and thus 87 MILLION struct pv_entry structures.
This is why it blows up. That's just too much. The sysv use_phys hack removes the need for a pv_entry structure. Thus removing several gigabytes of memory that the kernel would otherwise have to allocate to track all those pv_entry's.
There are optimizations that can be done to reduce memory use in the BSDs, but the main problem here is that postgres is using fork() instead of threads. If it used threads the management overhead would be 1/60th what it is now and having 1/60'th the pv_entries would not be a big deal.
-Matt
Lets clear up some things:
* First, on NCQ. *ALL* modern SATA hard drives implement NCQ and have ~31 tags.
* Bridge chips. *NO* modern motherboard uses a bridge chip any more. Bridge chips used on devices is another matter. Some devices still use bridge chips. Many DVDs and CDs used bridge chips (which is why early SATA DVDs and CDs were so broken), though I think that is finally dying out. The most famous was one of the OCZ SSD models which used a bridge chip to tie two controllers together. The controllers could handle ~31 tags, the bridge chip could not so the host probe would indicate no NCQ support. Also, multi-physical-interface devices such as netbooks (hard drives in a 'book' with SATA, USB, and Firewire interfaces)... those generally use bridge chips that often don't support NCQ.
* BIOS 'RAID'. So-called soft-raid. This is fake-raid. It isn't real. Don't expect it to actually work properly in a failure case. It's still talking to the AHCI controller, it's just hiding the fact from the OS. BIOS soft-raid 'controllers' are usually pretty horrible, avoidance is best.
* On data loss from caches. It isn't the caches that you need to battery-back (unless you are REALLY dependent on fsync() times in e.g. a database application). I think the trend is more towards off-host cache redundancy these days because it gets you to approximately the same place without the need for expensive gear. A large percentage of modern filesystems use write barriers and have no problem handling drive cache loss.
That isn't the problem. It's the physical power to the device being dropped while the write IO is in progress that is the problem. Devices, particularly SSDs but also many HDDs, cannot retire meta-data (for a SSD) or even the current sector (for a HDD) if a sudden loss of power occurs. In addition, a sudden power loss on a HDD can cause UNRELATED sectors to fail depending on how the HDD is writing (whether it is doing a full-track write or not). This can lead to serious corruption of the drive, even outright destruction. I've had quantum drives go through sudden power loss during a write with HUNDREDS of sectors lost instead of just one or two. That was a while ago, but it was still in the SATA-era... they were modern drives.
So for UPS/power concerns the only thing that really really matters is that the drive remain powered for at least a second or two. Even that is no guarantee. Barring that you want redundant storage on separate UPS's so someone kicking a plug out or crow-baring the UPS's output doesn't take you out.
* Super-caps... e.g. as Intel advertises on newer SSDs. These are primarily to retire meta-data so the SSD doesn't brick when you power it back up. Intel SSDs have very tiny ram caches so it might be able to retire those too, but most other SSDs have larger ram caches and no super-cap has enough suds to retire the entire cache. The idea is to not end up with a partially corrupt sector here, not necessarily to be able to retire the entire ram cache. Also, SSDs often do background cleanup when idle so not having any pending writes to an SSD doesn't make it safe, necessarily. This is what the super-cap idea primarily addresses.
Battery-backed ram comes under the same category. Well, in this case perhaps super-cap-backed ram (good for maybe ~a week to ~a month with low power static ram). Lots of options here that don't cost an arm and a leg, but again what matters the most is that the drive be able to retire whatever it is currently writing and not necessarily whatever is currently in its caches.
* SAS vs SATA. There is lots of talk about this all the time. I've never noticed any real difference in reliability, probably because the only real difference between the two is firmware. Drive vendors will talk-up using more robust parts but I believe that about as much as I believe that the moon is made out of cheese. There are so few components in HDDs that it is fairly difficult to differentiate consumer from enterprise these days.
That is, NCQ (for SATA) does not have enough command slots available. Only ~32 or so per port. The SAS stuff works a bit better but I think still limits you to ~32 slots per target device (instead of ~32 per port on the driver side).
So what happens if you try to use the so-called media completion feature is that write operations eat up all your tags and crowd out the far more important read operations. This makes the feature almost worthless in my view. Not to mention that there is no way to determine, with the SATA spec, that the drive actually honors the bit. Just like the old ATA stuff, drive vendors play fast and loose with the AHCI/SATA specs in order to try to force people to use the far more expensive SAS drives, even though the actual hardware is the same.
What we do in DragonFly is split available tags into read-dedicated tags and write-dedicated tags, approximately 3:1. Writes only last as long as it takes the device to copy the data to its internal ram caches, so there's no real need to reserve more than a few tags for writing. This leaves the remaining tags available for reading.
If you don't do this what happens is that you can stall your read I/O by saturating all available tags with write IOs... the writes get retired instantly to the drive's ram cache UNTIL that cache is full, then suddenly the newly issued write tags stall and sit there until the drive can flush some data out. If you don't control how many tags you use for pending writes, you can completely lock out read activity. A simple 'dd' to a file... hell, a simple file copy, for a large file, is big enough to exhibit this behavior.
This leaves even fewer tags available for writing. At best, if you want to maintain read performance, you can't really use more than ~8 or so tags for writing.
For a SSD maximum performance can be achieved with even fewer tags since in this case all you are doing is soaking up the command overhead by pipe-lining the IOs. Meaning that 2 write tags is sufficient, 3 to be safe.
All of this pretty much precludes being able to use the media completion bit with AHCI/SATA and still have good performance. To really be able to make use of such a bit one needs to support ~256 tags per target... that's to the actual physical device, NOT ~256 tags to the driver or ~256 tags to the SATA/SAS controller on the host.
In DragonFly, for HAMMER1, there were numerous ordering constraints that requires at least two DISK FLUSH commands per volume header sync. For HAMMER2 there are essentially no ordering constraints except for the volume header write itself so only one DISK FLUSH is required to create a recovery point. In both cases all the writes leading up to the required demarcation point could complete in any order. When combined with read:write tag reservation performance remains good even though the DISK FLUSH doesn't operate NCQ.
For SATA only read and write commands can use NCQ. All other commands require serialization and cannot run concurrently with NCQ commands, including unfortunately the DISK FLUSH command. You can blame Intel for this bit of stupidity... they intentionally broke the AHCI/SATA spec in order to artificially differentiate between SATA and SAS, so drive manufacturers could pump up the prices for SAS drives (even though both the hardware and the physical attachment is exactly the same). Intel broke the AHCI chipset spec in other ways to differentiate it, particularly when it comes to error recovery. They'll tell people that it was to 'maintain compatibility with the ATA command set' but IMHO they are lying. Some of the things Intel did in the AHCI spec were just phenomenally stupid. It is still much, much better than the ATA stuff, but they had a chance to make something really robust and blew it.
For example, with AHCI/SATA error recovery requires serialization, which means that if you use a port multiplier and one drive is having problems you have to stall out I/O to ALL OTHER DRIVES while you deal with the one that is having problems.
That's really all that matters these days. Everything else can be easily corrected in software. Chromatic aberrations are more difficult to deal with nicely.
-Matt
The reason is that the exchanges get quite a bit of revenue offering these direct-connect services to the HFTs. Basically it's a backdoor way of stealing revenue from the real market makers. The HFTs steal the revenue from the real market markers by interposing themselves in the middle of many of the transactions (thereby getting the market making fee from the exchange), and then turn around and pay the exchanges a fee for the privilages. Thus the exchanges are able to offer the market making fee but get quite a lot of it back in the fees from the HFTs.
It's obvious to anyone with a brain that HFTs do not add any liquidity to the market. I mean, come on, how stupid are we? A small company worth $100M-$1B doing high frequency trading is not going to add any liquidity to the market. All they do is add phantom volume that looks like liquidity right up the moment where a real event happens, then it disappears, PUFF, just like that.
It comes down to the idiot regulators just not caring about making the markets fair to everyone.
In some sense this has been hurting the exchanges too, as the larger players have slowly pulled their trading out of the public exchanges and into dark pool exchanges that don't play the same games. The public exchanges will still be closely priced to the dark pool exchanges due to arbitrage, but retail investors definitely get the short end of the stick in this new reality. That isn't the real problem... the real problem is that the HFTs add so much noise to the system (literally their entire M.O. is to try to force the exchanges to lag other players) that it can result in excessive volatility and screw up price discovery to the HFT's advantage.
Another reality, however, is that market making margins have gone down to $0.01 over the last decade. Before the exchanges went to the decimal system the middle men were pulling down much more revenue... higher margins. This is a huge benefit to the retail investor regardless of the other shenanigans going on. But the psychology is such that it gets lost in the wind when investors and traders see specific participants being given an unfair, leg-up advantage.
-Matt
This is Karma in action. The little guy gets some payback!
-Matt
There is no 'solar tech'. China mass-produces standard panels for the most part, and those plants produce large quantities of toxic byproducts in doing so which they pretty much just dump into the environment. The solar industry in China also seriously over-produced their panels and wound up with piles of them as other countries (aka Europe, Germany in particular) pulled back subsidies for various reasons.
Solar panel companies in the US simply cannot compete against that, not unless you want to create environmental destruction on a large scale.
The second big nail in the coffin was the crash in natural gas prices. PV panels cannot compete against $3 NG (winds up being ~$4-$5 for an industrial consumer). Hell, not even COAL can compete with NG at current prices.
So this isn't entirey Obama's fault. In fact, not really his fault at all. The subsidies were formulated at a time when other energy inputs were far more costly and before China really started dumping mass quantities of panels onto the market. I'd blame him for trying to continue the subsidies in the face of NG's dominance, but it was a reasonable course of action when it was first initiated.
-Matt
The raw output from the tar sands is way too heavy to be refineable, it still needs to be diluted with light natural gas liquids first... the same stuff they use to dillute it for pipeline transport, minus a few refinement stages (btw, the US exports some of the lighter NGLs coming out of the shales to Canada for this purpose).
But it makes no sense to refine the output up there far away from consumers. What are they going to do, truck the gasoline down several thousand miles? They'd go out of business trying to do that.
There is much more to the fossil fuel infrastructure than power generation or gasoline refinement. There are a multitude of stages involved and a multitude of outputs (gasoline only being one), and those outputs have to be piped as well. The pipelines for those outputs aren't up north. Trucking is out of the question for these long hauls (not unless you want $15/gal gasoline!).
Trucking is used primarily for short hauls, until local gathering pipeline infrastructure is built, and rails are used for medium-hauls of the liquids, again, only until the pipeline is built to replace it, because of the cost effectiveness of a pipeline. Natural gas can't be trucked or railed in anywhere near the volumes required, it is just too expensive. It has to be piped.
So, basically, it doesn't make any economic sense to try to relocate the supporting infrastructure. Doing so would flush somewhere north of half a trillion dollars down the toilet.
-Matt
This isn't correct anyway. Peak domestic energy use is in the early evening, when everyone has gotten home and are doing the wash, making dinner, etc. Not during the day.
It's a big problem because power companies that want to charge variable rates would wind up short-changing people with PV installations whos primary output occurs earlier.
-Matt
I think you are a bit confused over the scale involved, but the main problem is transport cost when it comes to NG. The producers are already underwater with curent well-head NG prices for pure NG plays (called dry-gas wells), any sort of transport other than a pipeline would bankrupt them.
-Matt
It just seems odd. This is more a business article than anything else, and there is nothing new and cool about buying rail cars.
Our domestic pipeline infrastructure has been on a building spree for a decade. If any of you are investors, that's been the basis for the Oil&Gas MLP buildout that has been maturing at a very fast clip since the mid-2000's, continued right through the crash, and continues to mature at a modestly fast clip today and probably for another 10 years at least before the core-buildout slows down.
Generally speaking transport for OIL and NGLs (Natural Gas Liquids) can start out in tankers and rail cars but ultimately cost efficiency requires a pipeline to be built. And you have no choice for natural gas... its pipeline or nothing pretty much since compression to CNG or LNG levels is way too expensive (and way too dangerous) for domestic transport.
But it takes several years to build a long pipeline, costs billions of dollars, and requires both shippers and receivers to enter into long term 10-year+ contracts with guaranteed volume flow or investors wouldn't finance the pipeline in the first place. Because no actual revenue flows until the pipeline is complete.
There are a dozen major producing areas but in layman's terms the bottleneck is mainly in the North->South direction these days. EastWest has capacity now (though numerous major cities on the east coast still have bottlenecks). Existing pipelines in the north-south direction are essentially maxed out.
The Keystone pipeline saga is your typical talking-head/exaggerated/public-unaware crap. Pipelines criss-cross the U.S. already, there are already numerous (but maxed out) pipelines coming down from Canada all the way to the gulk, and Canada is a major trading partner whos major oil and gas reserves are essentially land-locked. Sure, they have some transport to the coasts for export, but they need to be able to drop down through the border into the U.S. markets and we also have an export market of our own going northward of light NGLs which the Canadians use for a multitude of purposes in their oil-sands operations. It's as much a diplomatic issue with our northern neighbor as it is anything else.
-Matt
This is both true and false.
It is true that the direct effect is to reduce your personal wealth.
However, at the same time, the inflation that is creating the personal tax issue also allows your government to pay off debt (and you too if you have debt) at a lower cost or to provide services that you benefit from. Government debt is a proxy to the debt owned by its citizen and services rendered to its citizens, just somewhat indirect.
So while it is annoying from a personal standpoint (because you weren't the one who decided to incur the government debt), a good chunk of the losses incurred by this mechanism are returned indirectly through (e.g.) a better economy.
Also, the tax is not 100%. You don't lose so much wealth that you are back at square one vs having placed the money in your local currency. And compounding still works. Also, again, having cash in any currency is not an 'investment', it's taking work product out of the economy and storing it, which shouldn't entitle the holder to the same value years down the line vs doing something else with the cash (like investing it).
I don't think anyone can make an argument that this mechanism should change because the behavior modification it implies is only beneficial to the country, economy, and population as a whole. Doesn't mean we have to like it, but it isn't evil either.
-Matt
I understand the concept but it's still complete nonsense. There are a huge number of problems with the concept of bitcoins.
First, scarcity is not a measure of value in of itself unless it is attached to something desirable. A bitcoin is nothing but a bunch of numbers. Arguing that its value is what people are willing to trade for it (with a floor based on the electricity cost) is insufficient. In the stock market there have been multiple instances of this sort of effect when backers of ETFs or ETNs decide to limit the maximum number of issued units, which causes those vehicles to become free-floating.
This is a well known effect that has been observed in the stock market. From what I can tell, bitcoins are no different. Without a regulating entity bitcoins become detached from essentially everything and free-float, and that is basically what we've seen. The problem is that this makes them highly unstable (and we've seen this too).
Even in-game currencies have an economic attachment... the time-value of the gamer (and is far more consistent than the time-value of a cycle of cpu time). Fiat currencies have dozens of major economic attachments. bitcoins have nothing.
Second is the so-called cost of generating a new bitcoin. This is supposed to be the regulating entity but, in fact, it is only HALF of a regulating entity. It puts a floor on the price but no ceiling. A major problem with this is that even the floor is not a fixed value because the work-product required to produce the bitcoin is highly variable. The cost of power is highly variable. The efficiency of cpus and gpus is highly variable.
Finally, the lack of a ceiling on the value attached to a bitcoin and, more importantly, the entire concept of a bitcoin increasing in value over time creates instability. Currencies which go up in value are just as bad as currencies which go down in value. The reason is that holding a currency, just like holding a commodity, removes the related work-product from the economy and sets it aside. When work-product is removed from an economy you effectively reduce your investment in the economy. This might be fine from the point of view of a bitcoin trader but it's a disaster for anyone trying to use bitcoins to drive an economy, which is one reason why nobody takes it seriously. You can't drive an economy with a commodity that increases in value because it reduces investment in the economy you are trying to drive. Reduced investment equals reduced jobs, lost efficiencies, and an inability to compete with other economies.
We have seen this effect in numerous economies, particularly since the crash as currency traders have shifted vast amounts of cash into 'safe havens'. The result? The so-called safe havens destabilized (hard currency or not) and their governments were forced to print money to prevent the rapidly increasing value of those currencies from destroying their economies. Think about that for a moment! The Swiss have been especially impacted, as have other hard currencies (Australian dollar, Canadian dollar, etc). And, of course, so has the US dollar as international investors flee Europe (but the US was easing already so it played into our hand).
-Matt
This isn't true. Any bitcoin transaction that involves translation to or from another currency is subject to a trading spread and liquidity issues.
Bitcoins have horrible spreads and essentially no real liquidity compared to any other currency. The spreads are conveniently not displayed on most sites.
Thus any transaction held in bitcoin form for only a short duration is going to have huge overheads. Efficiencies can only be gained by holding a fairly large bitcoin balance so most of your activity cycles within the bitcoin universe and doesn't get translated in or out. That, in turn, makes you subject to another huge problem with bitcoin... that being the ridiculously high volatility in value.
This makes bitcoins basically worthless for any real business.
-Matt
It depends what you mean by Tangible. I assume you are talking about fiat currency here. But everything, even fiat currencies, have some tangibility. Don't be fooled by people who believe that a fiat currency will just collapse out of the blue for no reason, or because inflation might occur (the most typical reason stated).
There mere fact that we are using a fiat currency does not implicate a particular weakness verses other currencies. It does not necessarily make an economy more vulnerable to mismanagement. Strong currencies can create huge problems in mismanaged economies, too. Look at the Euro, for example, which until very recently was managed like a hard currency... a very strong currency for the last few decades. Now observe Ireland, Spain, Italy, Greece, and numerous other countries. Spain at 25% unemployment! Clearly the currency didn't prevent mismanagement within those governments, and the US has (so far) fared much better, so nobody should even remotely operate under the assumption that the fiat nature of the dollar is in any way responsible for our current predicament.
The dollar bill, or any fiat currency, is backed by the entire economy of the issuing nation. Literally. Why? Because nearly all goods and services within the issuing nation are traded using that currency. Because taxes are paid in that currency. Because use of that currency is far more convenient than other things. Because the government might have capital controls in place (Argentina, for example). Because a large chunk of short-term working capital is necessarily kept in that currency. Because it is convenient (think about the extra overhead a vendor would incur trying to use barter... what does he do with the 30 chickens he was just handed? How does he store that gold safely so it doesn't get stolen? How long will those eggs last before they go bad?).
As long as the above is true then the currency has value. Real value. It isn't fake. I can go out and buy a hot-chocolate from Starbucks right now, or a boat, or a house, using dollars. Real value.
People who assume that a fiat currency is worthless miss out on this basic fact, but more importantly people who try to contrast fiat currencies with other currencies within the developed nations also forget that ANYONE these days can have a brokerage account and invest in the domestic and/or world economy (investment in most large-cap domestic names these days is basically an investment in the world economy).
These investments have not been any more stable than fiat currencies. In fact, in the world today, HARD currencies are considerably less stable as well. The dollar is actually one of the more stable currencies (the Yen is probably the most stable of them all, at least up until now). Pick your poison. Anyone with a brokerage account can move their wealth out of the fiat currency system, but shouldn't expect the results to by any more stable.
Currencies remove some risks, but add others. People use currency (fiat or otherwise) as a vehicle for trade precisely BECAUSE currencies are very stable over the short-term. The period of time the transaction operates within the currency is typically short, and thus not subject to inflation.
The real problem with a fiat currency occurs when you try to use it store your hard-earned work-product over a LONG period of time. And here we aren't talking about people living pay-check-to-pay-check (for them the currency is ridiculously stable because they have no savings). Too many people these days believe that, somehow, magically, they should be granted the right to 'save' their money simply by keeping it in cash... keeping it as the fiat currency, effectively removed from the economy, verses investing it in the economy. That's ass-backwards.
Similarly when we look at a hard currency, or any currency which tends to appreciate in value even relative to average base commodities (bitcoin comes to mind)... those currencies encourage people to store their work product in the currenc
I think you have a point here, but at the same time you are talking about an endemic problem with the very small numbers of people at the margins of society who are both poor and also completely or nearly completely undocumented AND have no wherewithal to fix the problem.
Even without documentation you can certainly get on the train, even if it comes to petitioning the courts or finding someone to counter-sign the account, but the people in those situations are there for other reasons. Fixing it won't fix their situations.
You can't base an entire society's working methodology on the needs of the people at the fringe. It's a lost cause no matter how first-rate the society is. Technology is going to move on regardless, and those people are there for more reasons than simply not having any money.
So don't try to change conversation. This isn't about the relatively few who are missing documentation, this is about the relatively MANY who don't have even the minimal sense to manage their finances properly.
-Matt
Energy fluctuates in value just like everything else, with time, location, and quantity as major components of that value. It might not be subject to inflation per-say, but energy fluctuates in value far, far, FAR more than e.g. dollars. By several orders of magnitude.
So energy would make a rather poor basis for a currency.
Inflation is not really an issue for a currency used as a trading medium, it is only an issue if you try to 'save' your wealth in that currency. There is a trade-off there... people accept inflation in trade for medium-term stability, whereas with a commodity you avoid inflation but must accept short and medium term volatility in value (medium term being ~20 years). Since people have limited life spans, choosing one vs the other is NOT a slam dunk.
-Matt
Yup, there they are, right next to the lotto machine :-)
Of course, pre-paid debit cards are a pretty big scam unto themselves with the fees involved in buying and reloading the cards. They cater to the stupidity of people.
A standard credit or debit card with a branch-less bank, plus ~2-3 free ATM reimbursements per month, is the best value for anyone capable of maintaining a minimum balance.
-Matt
I don't use cash at every opportunity but I don't trust every vendor I use with my credit card either, so I always keep some cash in my pocket and tend to make smaller purchases with that cash. I might visit an ATM once a week or so, so the actual physical cash I have on-hand is only 'live' for a week.
In fact, I rarely use checks now either. All my bills use my bank's free bill-pay service. 100% electronic. The vendors often don't even have to handle a physical check, it just goes between banks.
Even so, most of my regular purchases are simply with a credit card. Swipe, squiggle, and its done. Fast and convenient. A lot of vendors don't even bother with a signature for small purchases any more (e.g. Starbucks or Whole Foods), but even when they do I just doodle (and have for 20 years). No reason to use my actual signature. I've even started experimenting with a big 'X' recently and clerks don't care. Smiley-faces are more problematic, though.
Honestly, why should I care whether something is traceable or not? I kinda like being able to sort my credit card statements by purchase to determine how much I'm really spending on things.
In anycase, bank power is overblown... typical talking head media hype. It's a business just like everything else, and as more and more people go branch-less the choice and competition is higher than almost any other industry.
-Matt
Kind of a ridiculous statement. Bank accounts and debit cards are trivial to get, even if you never had one before. Credit cards are also fairly easy to get when you are starting out, just don't expect the bank to give you a high limit.
Fee avoidance is also trivial, you simply keep a minimum amount of cash in the account. There are plenty of banks (particularly branch-less banks) with low minimums.
The problem that most people have these days is that they are addicted to credit, can't manage money, and aren't willing to learn how to. Either they will maintain a credit card balance and be in-hoc for their entire lives paying interest, or they will constantly border on a $0 account balance, overdraw their accounts every few months trying to juggle it, and then complain that they are too poor (here's a hint: If you can make do with your account always bordering on $0 then you can make do with your account always bordering on $1000... you aren't actually spending any less each month by maintaining a higher balance).
The whole cash/cash-less argument was dead years ago. People who have so little money that they can't make use of existing low-fee/zero-fee mechanisms are hardly going to have enough money to make effective use of a more esoteric currencies like bitcoin, let alone be smart enough to make better use of such a mechanism verses existing and more convenient government sponsored mechanisms.
Has anyone looked at the spreads for converting bitcoin to a real currency? They are insane. The people making the money are the people working the spreads, not the people trying to use bitcoin as an actual currency.
Similarly, there is absolutely no need to keep your money in $USD if you don't want to. It's a fine form of exchange, I use it every day, but these days you can invest your money anywhere... in equities, in foreign currencies, in commodities, and only keep a few months worth of 'cash' around for immediate needs. A few months burn worth of cash is essentially not subject to inflation. The currency is almost irrelevant at that point except for the fact that you have to have it in a currency you buy things with to avoid translation fees.
Of course, anyone who hasn't done this is probably going to be a bit surprised at how volatile the relative value of EVERYTHING moves around these days. Gold, USD, Euro, an equity, a piece of a mine, other commodities, whatever... everything has a relative value based on supply and demand. Nothing is immune. There is no such thing as 'intrinsic' value for anything (not even bits of 'energy' as someone tried to suggest). People who invest in gold, for example, not knowing how volatile its value is, can suddenly find their investment worth less in dollar terms while their burn rate in dollars doesn't change, over a few days or weeks or months, and quickly realize the fallacy of relying on some idiotic notion of absolute or intrinsic value.
p.s. recently drug gangs and cartels have begun using bit-coin to launder money. Lets see how long it lasts as a viable currency when the larger transactions wind up being made by criminals instead of normal users. If you want practical untraceability then cash from an ATM works just fine. Theoretically its traceable but after a couple of transactions it's effectively impossible to track. Just because something is theoretically possible doesn't mean there are government drones hiding behind every flower and bush doing it.
-Matt
So far IPMI has managed to survive all the crap vendors try to add to it. Chalk one up for standards! It's gone through some growing pains (the original standard was badly broken and precursors tried to share the ethernet port on the mobo), but has settled down into a usable implementation in the last few years, as long as you only use open-source IPMI tools like 'ipmitool'. There are typically still hicups with the video sniffing implementations (particularly when the OS switches video modes), but serial port forwarding seems to work quite well.
In anycase, IPMI has been a godsend for those of us with smaller server installations. It removes the need for an addressable PDU and removes the need for a console server. It removes a lot of cables.
Once the BIOS is setup you don't need to connect a keyboard or monitor to the machine ever again, even when you are physically in the machine room. You just plug you laptop into the switch or Wifi and poof.
There are numerous ways to secure the IPMI net. If configured properly IPMI 2.0 has reasonable security but still can be DOS'd. The best thing about the standard is that it is trivial to insert machines, IP filters, and other things between the IPMI network and the rest of the world but the fact that it can connect directly to the internet also ensures that those extra gadgets are going to be fairly cheap in order to compete. Always only use the IPMI 2.0 UDP protocol... if the thing has a web server or ssh access (easy to test), turn those off straight away.
Mobos with IPMI tend to cost ~$20-$50 more and the IPMI module itself (when not built in) typically costs $30-$60. Considering what you get for it, it's damn cheap. IPMI has already saved me thousands of dollars worth of equipment, gas, and time.
-Matt