2000MiB/s is pretty low end. The newest "cheap" drives are reaching 3500MiB/s, which is maxing out the PCIe 3.0 4x slot. They're already working on PCIe 4.0 and and 8x version, which will quadruple the max IO, which should be trivial with future tech.
I'm personally averaging about 1-2 reported bugs per 1kloc in production. I have no QA team, most of the time I am given no documentation, I'm pretty much a "we don't know how this works and we don't have time to research it, so you need to make magic happen" kind of guy. I primarily specialize in high performance multi-threaded code that needs to work. I have several projects in production that have never had any bugs reported in the 5+ years they've been deployed and in heavy use, but they have had feature creep, of which I typically had 1-2 days turn-arounds for what most people thought was a difficult task.
On the other hand, my code has actually found bugs in other programmer's code. At first they would report a bug about my code, but they must not have taken the time to read the overly verbose exception explaining why it failed. I do a lot of logic checking. If a field is supposed to only be of certain values, I make sure it's only one of those values. I'm not a garbage-in-garbage-out person, I'm a garbage-in THROW EXCEPTION with useful message person.
You are free to create your own proposed language, but so far no one has been successful. C is still used where performance and low level access is a requirement.
Most senior people are equally incompetent. Senior programmers tend to be pattern-anti-pattern cargo-cult programmers. They seem competent to those who are incompetent themselves. They also like to use buzzwords. "I'm going to use a no-sql database to increase the scaling performance" only to reinvent a relational database on top of a no-sql database and get strange data oddities because they didn't realize that "transactions" in their chosen no-sql DB does not mean the same thing as "transactions" in a proper relational database. Then after fixing the transnational issues, the no-sql DB is now slower because synchronization is more expensive in no-sql.
Knowledge in an of itself is useless. The Google datacenter is more knowledgeable than any human, yet it is stupid, about as smart as an insect. Wisdom is what one gains from making mistakes, talent allows you to skip making mistakes in the first place. A wise person can correctly recreate what they have already done in the past, talented person can correctly create something entirely new.
Except you can delay when you cut the car into the future and still get the same result. They can send entangled photons in two different directions, and use different types of measurements to cause statistical influences on the other photon, no matter the distance, and even if it happens in the future. Example is if you measure for vertical polarization, you will get a different distribution than if you measure for horizontal, simplified.
You're confusing budgeted and actually spent. Wartime expenditures are not restricted to budget. For example. On paper, we budgeted only a few billion for wartime expenditures between 2001 and 2006, yet the actual money spent was in the trillions. My poli-sci teacher said in those 5 years, we spent enough money on the war to fund all healthcare and college for all of the USA for 10 years.
DWDM does have great bandwidth, but can only transfer about 400Gb/s per channel right now. A single 1Tb/s channel, even if less than the 32+Tb/s DWMD can handle, still has its usefulness.
Currently commercially available is 32Tb/s with 6km ranges, and has been for 5+ years now. There is little demand because backbone bandwidth supply has completely outpaced demand. No point in increasing the backbone's bandwidth if it's mostly idle already. They are expecting 100Gb/s FTTH over 5Tb/s shared fiber by 2020. I'm sure that will start to drive up demand.
It gets better than that. You ball may be a wave or a particle, and you can record the state of the ball now, but in a scrambled form. Then send the paired ball to some someone else who may be millions of light years away. And depending on if they observe it or not, will decide if your ball is a wave or a particle. From your point of view, someone possibly measuring the twin ball millions of years in the future decides if your current ball is a particle or wave.
The issue is that you can't know until data about what they read is sent back allowing you to decipher what you measured. That has to be sent via classical means.
I visualize all kinds of logical problems, which conflicts with my navigational abilities. I have a hard time thinking and walking. Sometimes my balance is thrown off because what I'm "seeing" and what I'm feeling don't match. I would not describe what I see as normal 3d images. I see "n dimensional" images where "n" is the number of variables.
If I'm thinking really hard, even my hearing and touch gets hijacked and I can experience strange sensations. An example would be when thinking about how network flows interfere with each other. The rate of packets being sent can be thought of as "sound", then all of the "sound frequencies" of the network flows converge on a single point, and then I visualize the resulting sound of these overlapping frequencies and can "see" where peaks get too spiky, resulting in jitter or packetloss. I've done this several times when trying to visualize why I was getting incredibly rare transient packetloss. Effectively microbusts of roughly synchronized senders. Where the "n" dimensional comes in is I can see multiple versions of these at the same time, like many steady flows, many starting flows, etc etc. I can think of the corner cases ahead of time and see all of these cases concurrently without having to rethink of the issue for each case.
I just used networking as an example, but i'm a programmer and do this same thing to pretty much all problems.
Fiber ISP uses IPTV to send me 20Mb/s 1080p. I did a side-by-side of ABC's Harry Potter weekend against my Blueray on the same monitor and I could not tell a difference. The TV was distinctly better than my DVD version, even my wife chimed in on the quality differences. My STB does not support 4k. It will be a bit before they start moving in that direction, but bandwidth is not an issue.
98% of the cost of an ISP has nothing to do with bandwidth or network equipment and speeds are increasing 100% every 9 months. It is so cheap it has been described as "selling air". Are you proposing that people who consume way more air than others should have to pay more? Stop running! You're breathing too much!
They play games with the laws. There is no exclusive franchise, but they do make right of way laws that make the barrier to entry incredibly high. AT&T successfully sued my state government to disallow ISPs access to right of way easements. But you ask, isn't AT&T an ISP? Nope, they're a telcom, completely different. They offer internet services, but they are not an "ISP" by right of way definition. Funny how that works. AT&T wasn't alone in suing the government, so "cable" companies also have access, but not ISPs.
Then you couple that with another common city law that states that unless you have right of way access, there is a limit of how many companies have access. They don't want lots of competition coming through and tearing up everyone's lawns. So we have this law that effectively limits the number of ISPs right of way access to exactly 1, but AT&T and Charter aren't ISPs, but because they offer ISP services, they still count towards that "1". So yeah, good luck.
Incorrect comparison. I have exactly one road connected to my drive way. Whomever owns that has a natural monopoly. A subway is more like a bus than a road.
Cable companies spend 10x more money upgrading their cable networks every 5-8 years than the cost of a fiber network that won't need to be upgraded for several decades. They're throwing money in the wind. Average cost to install 1Gb/1Gb fiber, $1.5k/house, cost to upgrade an existing cable network from DOCSIS 2.0 to DOCSIS 3.0, $10k/house. DOCSIS 3.1 will be more expensive for anywhere they plan to actually provide the "3.1" speeds, otherwise you're stuck with 3.0 speeds until they split nodes, which is hugely expensive.
That's not even including the ~20% industry average reduction in Op-ex costs going fiber, which is a lot of something that consumes 60% of your revenue.
"more upstream equipment"? ISP's router can support up to 64 400Gb ports for a city of 8000 households. How soon are you expecting them to have to upgrade? You seriously underestimate how fast modern ISP equipment is. The linecards are upgradable to 1Tb/s ports. Entry level gear can handle 4Tb/s and high end gear can handle about 900Tb/s. Ever see 300+ 1Tb/s ports in a single router? Yes, they make them.
They're already working on 10Gb/s FTTH and expect it to not only to easily scale to 100Gb by around 2018-2020, but that is 100Gb per customer. The entire fiber will support about 5Tb/s of "shared" bandwidth. Unless they plan to have 50+ customers per fiber, everyone will have dedicated bandwidth.
A "massive" cost that is a small percentage of their over all costs. I think it was Nasdaq that had a report breaking down Netflix' costs that showed bandwidth was pennies on the dollar before then went Open Connect. Most of Netflix' costs have to do with content licensing. Reducing bandwidth is more of a scalability issue and opportunity cost issue, since customers have crap connections and unhappy customers will not be customers for long. You also have the whole "cool factor" from the tech side of things and when you're consuming 1/3rd of the Internet, it's your duty.
If the (then) future hadn't gone the way that it did
If there is anything that I have learned is if something can go wrong it WILL go wrong. Don't let it go wrong in the first place. Spend the extra 1 second and think about your code before you vomit your crap code into production.
It isn't "premature optimization" until it makes it some degree harder to understand than the non-optimized version. Most of what people call "premature optimization" is really just writing correct code. Should I use String.EndsWith or String.Contains to find if a string has a certain suffix? Should I use an explicit cross join and filter in the WHERE clause or use inner joins? These are examples of the level of boneheadedness that most people write code. Don't get me started on multi-threading. If you don't find it intuitive, please don't do it.
Most of the time I can make someone's code several factors faster with virtually zero effort while making it easier to read. And when I say "faster" I don't mean as in micro-benchmarking the code in simplistic situations, I mean production where the entirety of the code is being called many times on many threads. Sure, your "read the entire file into memory and write it back out" works fine in your simple tests, but throw it in production and the 400MiB files put massive pressure on memory when there are hundreds of jobs running at the same time all trying to load the entirety of files into memory before writing them out. STREAM the files.
Then you have the issue of many times good code runs slower in synthetic benchmarks but runs faster in production. Everyone focuses on empirical evidence, but then compare apples and oranges when thinking their dev boxes or servers are the same as prod in that the computers are only running their code in tight loops
I've had queries that took 3 minutes and got them down to under 10 seconds, for a nightly job that no one cared about. Until some large dataset came along and made that 3 minute run go for 24+ hours before a DBA killed it. At which point I tossed my version in and it ran in less than a minute.
In my experience, good code not only runs much faster, but scales linearly, while bad code scales exponentially. All of those scrubs and their "micro benchmarks" showing that their code is fast enough, then you throw it into prod and it blows up a few weeks later and takes days of debugging to nail down the cause.
The fixed function ASIC is magnitudes faster, more efficient, so few transistors, it's practically free to add. It's almost like saying that dedicated floating-point is "a shame" because they could just make the CPU faster and just emulate it. This is just one of those cases were ASICs can beat the crap out of any generalized implementation, even SIMD.
Slashdot Mirror
2000MiB/s is pretty low end. The newest "cheap" drives are reaching 3500MiB/s, which is maxing out the PCIe 3.0 4x slot. They're already working on PCIe 4.0 and and 8x version, which will quadruple the max IO, which should be trivial with future tech.
I'm personally averaging about 1-2 reported bugs per 1kloc in production. I have no QA team, most of the time I am given no documentation, I'm pretty much a "we don't know how this works and we don't have time to research it, so you need to make magic happen" kind of guy. I primarily specialize in high performance multi-threaded code that needs to work. I have several projects in production that have never had any bugs reported in the 5+ years they've been deployed and in heavy use, but they have had feature creep, of which I typically had 1-2 days turn-arounds for what most people thought was a difficult task.
On the other hand, my code has actually found bugs in other programmer's code. At first they would report a bug about my code, but they must not have taken the time to read the overly verbose exception explaining why it failed. I do a lot of logic checking. If a field is supposed to only be of certain values, I make sure it's only one of those values. I'm not a garbage-in-garbage-out person, I'm a garbage-in THROW EXCEPTION with useful message person.
You are free to create your own proposed language, but so far no one has been successful. C is still used where performance and low level access is a requirement.
against what the senior people advice
Most senior people are equally incompetent. Senior programmers tend to be pattern-anti-pattern cargo-cult programmers. They seem competent to those who are incompetent themselves. They also like to use buzzwords. "I'm going to use a no-sql database to increase the scaling performance" only to reinvent a relational database on top of a no-sql database and get strange data oddities because they didn't realize that "transactions" in their chosen no-sql DB does not mean the same thing as "transactions" in a proper relational database. Then after fixing the transnational issues, the no-sql DB is now slower because synchronization is more expensive in no-sql.
Knowledge in an of itself is useless. The Google datacenter is more knowledgeable than any human, yet it is stupid, about as smart as an insect. Wisdom is what one gains from making mistakes, talent allows you to skip making mistakes in the first place. A wise person can correctly recreate what they have already done in the past, talented person can correctly create something entirely new.
Except you can delay when you cut the car into the future and still get the same result. They can send entangled photons in two different directions, and use different types of measurements to cause statistical influences on the other photon, no matter the distance, and even if it happens in the future. Example is if you measure for vertical polarization, you will get a different distribution than if you measure for horizontal, simplified.
I mean 6,000km ranges. I heard "kilo" in my head but forgot a km is the base unit. So 6kkm?
cost $123.9 million...per year
You're confusing budgeted and actually spent. Wartime expenditures are not restricted to budget. For example. On paper, we budgeted only a few billion for wartime expenditures between 2001 and 2006, yet the actual money spent was in the trillions. My poli-sci teacher said in those 5 years, we spent enough money on the war to fund all healthcare and college for all of the USA for 10 years.
DWDM does have great bandwidth, but can only transfer about 400Gb/s per channel right now. A single 1Tb/s channel, even if less than the 32+Tb/s DWMD can handle, still has its usefulness.
Currently commercially available is 32Tb/s with 6km ranges, and has been for 5+ years now. There is little demand because backbone bandwidth supply has completely outpaced demand. No point in increasing the backbone's bandwidth if it's mostly idle already. They are expecting 100Gb/s FTTH over 5Tb/s shared fiber by 2020. I'm sure that will start to drive up demand.
It gets better than that. You ball may be a wave or a particle, and you can record the state of the ball now, but in a scrambled form. Then send the paired ball to some someone else who may be millions of light years away. And depending on if they observe it or not, will decide if your ball is a wave or a particle. From your point of view, someone possibly measuring the twin ball millions of years in the future decides if your current ball is a particle or wave.
The issue is that you can't know until data about what they read is sent back allowing you to decipher what you measured. That has to be sent via classical means.
My 5.5" cellphone has a 2k screen. Give it another year or two.
I visualize all kinds of logical problems, which conflicts with my navigational abilities. I have a hard time thinking and walking. Sometimes my balance is thrown off because what I'm "seeing" and what I'm feeling don't match. I would not describe what I see as normal 3d images. I see "n dimensional" images where "n" is the number of variables.
If I'm thinking really hard, even my hearing and touch gets hijacked and I can experience strange sensations. An example would be when thinking about how network flows interfere with each other. The rate of packets being sent can be thought of as "sound", then all of the "sound frequencies" of the network flows converge on a single point, and then I visualize the resulting sound of these overlapping frequencies and can "see" where peaks get too spiky, resulting in jitter or packetloss. I've done this several times when trying to visualize why I was getting incredibly rare transient packetloss. Effectively microbusts of roughly synchronized senders. Where the "n" dimensional comes in is I can see multiple versions of these at the same time, like many steady flows, many starting flows, etc etc. I can think of the corner cases ahead of time and see all of these cases concurrently without having to rethink of the issue for each case.
I just used networking as an example, but i'm a programmer and do this same thing to pretty much all problems.
Fiber ISP uses IPTV to send me 20Mb/s 1080p. I did a side-by-side of ABC's Harry Potter weekend against my Blueray on the same monitor and I could not tell a difference. The TV was distinctly better than my DVD version, even my wife chimed in on the quality differences. My STB does not support 4k. It will be a bit before they start moving in that direction, but bandwidth is not an issue.
98% of the cost of an ISP has nothing to do with bandwidth or network equipment and speeds are increasing 100% every 9 months. It is so cheap it has been described as "selling air". Are you proposing that people who consume way more air than others should have to pay more? Stop running! You're breathing too much!
They play games with the laws. There is no exclusive franchise, but they do make right of way laws that make the barrier to entry incredibly high. AT&T successfully sued my state government to disallow ISPs access to right of way easements. But you ask, isn't AT&T an ISP? Nope, they're a telcom, completely different. They offer internet services, but they are not an "ISP" by right of way definition. Funny how that works. AT&T wasn't alone in suing the government, so "cable" companies also have access, but not ISPs.
Then you couple that with another common city law that states that unless you have right of way access, there is a limit of how many companies have access. They don't want lots of competition coming through and tearing up everyone's lawns. So we have this law that effectively limits the number of ISPs right of way access to exactly 1, but AT&T and Charter aren't ISPs, but because they offer ISP services, they still count towards that "1". So yeah, good luck.
The 9 month doubling doesn't even include long haul fiber speed increases, which has been more like 10x per year, not 2.
Incorrect comparison. I have exactly one road connected to my drive way. Whomever owns that has a natural monopoly. A subway is more like a bus than a road.
Cable companies spend 10x more money upgrading their cable networks every 5-8 years than the cost of a fiber network that won't need to be upgraded for several decades. They're throwing money in the wind. Average cost to install 1Gb/1Gb fiber, $1.5k/house, cost to upgrade an existing cable network from DOCSIS 2.0 to DOCSIS 3.0, $10k/house. DOCSIS 3.1 will be more expensive for anywhere they plan to actually provide the "3.1" speeds, otherwise you're stuck with 3.0 speeds until they split nodes, which is hugely expensive.
That's not even including the ~20% industry average reduction in Op-ex costs going fiber, which is a lot of something that consumes 60% of your revenue.
"more upstream equipment"? ISP's router can support up to 64 400Gb ports for a city of 8000 households. How soon are you expecting them to have to upgrade? You seriously underestimate how fast modern ISP equipment is. The linecards are upgradable to 1Tb/s ports. Entry level gear can handle 4Tb/s and high end gear can handle about 900Tb/s. Ever see 300+ 1Tb/s ports in a single router? Yes, they make them.
They're already working on 10Gb/s FTTH and expect it to not only to easily scale to 100Gb by around 2018-2020, but that is 100Gb per customer. The entire fiber will support about 5Tb/s of "shared" bandwidth. Unless they plan to have 50+ customers per fiber, everyone will have dedicated bandwidth.
I could get dedicated 20/20 for $40/m 5 years ago, now it's dedicated 150/150 for $50
A "massive" cost that is a small percentage of their over all costs. I think it was Nasdaq that had a report breaking down Netflix' costs that showed bandwidth was pennies on the dollar before then went Open Connect. Most of Netflix' costs have to do with content licensing. Reducing bandwidth is more of a scalability issue and opportunity cost issue, since customers have crap connections and unhappy customers will not be customers for long. You also have the whole "cool factor" from the tech side of things and when you're consuming 1/3rd of the Internet, it's your duty.
If the (then) future hadn't gone the way that it did
If there is anything that I have learned is if something can go wrong it WILL go wrong. Don't let it go wrong in the first place. Spend the extra 1 second and think about your code before you vomit your crap code into production.
It isn't "premature optimization" until it makes it some degree harder to understand than the non-optimized version. Most of what people call "premature optimization" is really just writing correct code. Should I use String.EndsWith or String.Contains to find if a string has a certain suffix? Should I use an explicit cross join and filter in the WHERE clause or use inner joins? These are examples of the level of boneheadedness that most people write code. Don't get me started on multi-threading. If you don't find it intuitive, please don't do it.
Most of the time I can make someone's code several factors faster with virtually zero effort while making it easier to read. And when I say "faster" I don't mean as in micro-benchmarking the code in simplistic situations, I mean production where the entirety of the code is being called many times on many threads. Sure, your "read the entire file into memory and write it back out" works fine in your simple tests, but throw it in production and the 400MiB files put massive pressure on memory when there are hundreds of jobs running at the same time all trying to load the entirety of files into memory before writing them out. STREAM the files.
Then you have the issue of many times good code runs slower in synthetic benchmarks but runs faster in production. Everyone focuses on empirical evidence, but then compare apples and oranges when thinking their dev boxes or servers are the same as prod in that the computers are only running their code in tight loops
Your connection to them is probably choking and down-grading the resolution.
I've had queries that took 3 minutes and got them down to under 10 seconds, for a nightly job that no one cared about. Until some large dataset came along and made that 3 minute run go for 24+ hours before a DBA killed it. At which point I tossed my version in and it ran in less than a minute.
In my experience, good code not only runs much faster, but scales linearly, while bad code scales exponentially. All of those scrubs and their "micro benchmarks" showing that their code is fast enough, then you throw it into prod and it blows up a few weeks later and takes days of debugging to nail down the cause.
The fixed function ASIC is magnitudes faster, more efficient, so few transistors, it's practically free to add. It's almost like saying that dedicated floating-point is "a shame" because they could just make the CPU faster and just emulate it. This is just one of those cases were ASICs can beat the crap out of any generalized implementation, even SIMD.