And everyone else. Germany is now realizing that broke customers don't buy german stuff, so the expectation is that all the eurozone will be in recession by the end of this year. France is hanging by a thread. Spain will be rescued by the IMF, it's a matter of time. The recipes you point already have been applied here, without much success. Small high-debt countries are a great target for investment and speculation, specially if they are inserted in some kind of insulation such as the Euro. Speculators can strangle the country's economy, without compromising their paycheck (because someone else will pay the bill). That said, this crisis is a sort of wakeup call for Portugal - we have great potential for business, but so far without producing results. Smaller companies that can't compete are being forced out of business aggressively. Looking two or three decades ahead, this is not a bad thing - the ones that resist will be major players, and new ones will be aware of mistakes of the past. Most of what's being done in Germany (and a part of it is filling up import forms from China) can be done today here at a fraction of the cost. But the mentality of local business (instead of iberic or european business) still strong among the older generation of managers - that are being weed out by the crisis.
You should. Printing money depreciates currency, and on a country highly dependent on energy imports and raw materials such as Portugal, the result would be a massive increase on the cost of energy, transport and production. Yes, even worse than now (I'm also portuguese).
English isn't my native language, but it seems the problem lies with people thinking that breseham's implementation is "traditional". It's not. To be blunt, the base algorithm isn't even integer based and envolves a div (to calculate the slope), so, by all metrics, the original problem is ignorance. The integer bresenham algorithm is quite popular, but it amazes me that people think is a "traditional" approach, instead of solving a 1st degree equation. (Wtf are people learning in schools nowadays?).
Regarding your div remarks - yes, division is a complex process, but easily solved by throwing more transistors at it. The dependency chain between instructions is only relevant to high-penalty cpus (such as P4), or when you have multiple execution pipes, but no out-of-order execution support. Well, the Pentium mechanism is pretty simple, if instructions can be paired on the U and V pipes, they will, if not, tough luck - they'll be executed sequentially. That is the motive I used it - it is simple to understand. Pentium Pro and later processors do feature multiple "all purpose" pipes, but also feature out-of-order execution. Not that matters much, because the point I made was precisely about the futility of single-processor optimization - what is clever on one cpu release may be utterly stupid on another (and almost everything was stupid on the P4 aberration).
Tiny corrections:D
- Traditional line drawing uses a div per pixel. Check pseudocode at the wikipedia page (http://en.wikipedia.org/wiki/Line_drawing_algorithm). There are some "hybrid" algorithms that implement one division per line, but usually are a variation of bresenham's algorithm with better precision.
- I'm not familiarized with Heaven7's internals, but it is somewhat obvious that uses tricks. As most other rtr implementations out there. And some non-realtime raytracers. If you can simplify global illumination (using "post-production" effects), scene complexity (by camera angles and removal of invisible objects), and recursion (limit interpolation and reflection/refraction), you still can get quite impressive results, even with "old" cpus, by today's standards.
I guess you are confusing the FreeBSD TCP stack optimizations with stateful packet inspection - routing, by default, does not touch firewalling.
But hey, straight from the horse's mouth: http://www.openbsd.org/faq/pf/ru/perf.html . That's why there's a new (NetBSD) project called NPF, that aims the creation of a more sofisticated and scalable packet filter.And, afaik, FreeBSD 9.0 TCP optimizations relate to congestion algorithms, not necessarily firewalling.
I often see these misconception of trying to translate internal/testing/development benchmarks into real-case usage. Just because a piece of software measured 100k connections/s on a controlled environment and selected hardware, it doesn't mean it will perform similarly on your machine. Local network application tests are done with clean, well-formed, unrouted traffic. In internet-facing production servers, you'll get all kinds of garbage. The real amount of hits/s you'll get will depend of a plethora of factors, such as average content size, network latency, I/O latency, design of the application, quality of the hardware, quality of the drivers, etc. The choice of webserver is relevant, but not that much. TCP attacks can drown your machine (by network exaustion) long before you see any real advantage of using Nginx regarding DDOS scenarios. It is also very common to see people switching to nginx and being forced to learn what they are doing, so the idea they have of apache is "all modules loaded" and some random options on a control panel.
I use both webservers, for different applications. Nginx is faster in static content, and seems to handle stress very well. Apache is feature-complete with a plethora of third-party modules (such as mod_security) and behaves very well as a general-purpose server (I actually use a lot mpm_itk, so every vhost has its own uid/gid). Sometimes you can squeeze extra performance by trading Apache for Nginx, sometimes it will break your app (complex mod_rewrite rules, for example). But that's relevant for legitimate users - if your network pipes are clogged by requests, both webservers will appear unavailable.
It seems Slashdot stopped notifying me of AC replies, weird:P
If you don't have the infrastructure and/or the staff, outsourcing DDOS protection seems a good choice. Trying to protect against the 3Gbps mark is a bit silly, since next time you may get hit by 10x that traffic. It is important to learn the nature of the attack (are we talking about hundreds of ip addresses, or a dozen? are they "residential connections" or compromised servers? is it a TCP handshake attack, or requests to specific pages? do those pages exist, or is the fancy 404 rewrite eating up resources by pulling the page from the database?), and probably some optimizations can be made to your ecommerce platform (such as reducing requests, pulling images from CDNs, etc - check tools.pingdom.com for basic hints) so that the traffic spent with legitimate customers can be lowered.
Ask yourself this: how much do you lose for each day your ecommerce platform is down? Since you seem to have a single-server solution, if you consider let's say, 5 days of downtime per year, how much would you lose? Do the losses are greater than 50% of cost of a fancy redundant infrastructure? If so, upgrade immediately.
Since you say you are from the UK (hi neighbour, I'm from Portugal), you may also want to check available other providers such as Rackspace. They probably have some cloud-based solution that may fit your budget.
Modern ARM (v7) cpus already have an integer divide instruction (and many other low power architectures already have them, including modern versions of traditional 8-bit microcontrollers and microprocessors, such as Z80). Most of the optimization tricks you'd use on x86 aren't very good on a RISC platform (I have no experience with ARM; but I've played with Intel RISC processors). Versions with SMID support are widely available, and it is a far more important feature than FP for most intensive algorithms. Also, most devices already have some form of GPU, so I don't expect to find OpenGL compatibility, as an example, implemented in software.
I honestly do not know how deep pipelines relate positively with performance (during the P4 GHz wars, AMD used a much shorter pipeline, so it could get the same performance at a fraction of the frequency - and that's why a single core of a modern [pentium-m descendant such as core 2] 2.0Ghz processor blows out of the water any >3.0GHz Prescott).
And of course, although demos targeted at old CPUs don't use new CPUs in the most ideal way, they still run much faster on new CPUs than they did on the CPUs for which they were intended.
Yes, but all the cleverness of the algorithm optimization is somewhat lost, isn't it? If the result isn't faster than the reference implementation, why waste time optimizing it at that level? And even if you do waste that time and go the extra mile, how do you feel everytime you need to basicly rewrite all your libraries because the rules of the game have changed?
Thus, the old demos should still work perfectly well.
Except many don't. Because some rely on instruction timing for music sync, some rely on "padding" instructions for delay, some rely on vertical retrace to sync the graphics, as an example of many hardware artifacts that were actually used as a feature. Probably since DX/OpenGL demos compatibility has improved, but try to run an old DOS demo on a modern computer. Or one that relies, as an example, on a processor "defect" such as the 386 "unreal" mode.
I guess I was misinterpreted. I regard scene programmers more like "code artists". The software product itself is unique, in the sense that is not usually maintained or ported. Choosing long-term algorithms and performance compromises is usually not a problem, since it is made to be awesome with a specific combination of hardware. The algorithms have to be fast enough to implement the desired funcionality on a specific point in time (the party or the release date) and usually on a specific processor, not on hardware that will exist 6 months from now.
During the nineties, I've done a lot of scene-related assembly code (mostly graphics and infrastructure - extenders, memory managers, etc), so I'm well aware of the limitations imposed and the usual workarounds. Most sourcecode I've seen is an ugly mess, even if it works. There are some true clever and elegant algorithms in the lower categories (128/256b and 4k), but what I've learned since then is that computer processing power is cheap, but software lasts. One of my favourite demos of all time is Heaven7, mostly because it implements a realtime raytracer than runs smoothly on a P200. But probably many of the optimizations that allowed it to run smoothly in such a limited processor aren't valid for a P3. Or a P4 (handcode optimization for any of the P4 lines is a _nightmare_). So, if it was a maintained application, the highly optimized algorithms would probably have to be rewritten, to take advantage of the latest features, and keeping "pushing the boundaries".
As an example (and more in line with the nineties), a lot of effort was put into highly optimized bresenham line algorithms, because traditional implementations implied a div operation per pixel, and integer division was awfuly slow (like 40 cycles on a 486). So, even if bresenham's requires extra instructions, it would still be a lot faster. Well, on a pentium, not only the div instruction took 1 clock cycle (like most of the other instructions), some instructions could be paired for execution (the processor had 2 execution pipes), so the bresenham implementation is usually a lot slower than using the div instruction. If you had to write maintainable software, would you worry about implementation details that could double your development time, but be obsolete on the next processor to be released? I guess not.
Demos are a kind of development on its own. They require no user interaction, no external data pulling (other than already packaged resources), have no error checking whatsoever and usually are buggy as hell - slightly different hardware may give you completely different results, or just don't run at all. So, that's why I don't like to call scene coders "real programmers". They are more of a class of "code artists", and yes, they should deserve more merit than "regular" coders, not only because their algorithmic skills, but also because of their creative way of implementing them.
So, being aware that are issues when using nginx as a reverse proxy instead of varnish, is useless? And those comments were made here, on slashdot, so god forbid you use a search engine to confirm what issues are those...
Demo scene is the perfect example of what real programmers can do.
I'd say demoscene is the perfect example of what can be done, software-wise. I'd guess you've never peeked at a demo sourcecode, or you wouln't call them generically "real programmers" (incidentally, there is usually much talent behind the code, but no structure or discipline whatsoever).
It's not uncommon for multiple colo customers to be on a shared switch with 1gb uplink especially when each individual customer only has a 100mbit port
Yes, you are right, but if you have DDNS worries, you'll keep away from those providers, and/or opt for a beefier connection. Many of them will happily provide you with detailed information regarding network structure, so you can be aware of what you are actually paying. Also, many of those "cheaper" providers have somewhat frequent DoS attacks inside their own network (some machines get compromised, and then try to scan/infect everybody else), so, again, handling/filtering a 3Gbps attack isn't usually a big deal.
Also the isp has to pay for the bandwidth usage that hits them, even if most of it never actually reaches the end customer...
That's the ISP's problem, and I seriously doubt that the pricing doesn't reflect these "hidden costs".
There are several available solutions, depending on your budget. You could opt for a cloud hosting solution (AWS, GoGrid, etc) so your cluster could scale dynamically according to demand. You could roll your own solution (and there are several options with different price ranges, from simple web cluster to redundant replicated clusters in different datacenters), preferably with a provider that won't panic over 3Gbps traffic. You could use a proxy service like those mentioned in the article.
To put it in perspective, many modern datacenters have 10G pipes to the rack. A modern server can handle 10G traffic with iptables (assuming somewhat simple rules) without issues. A midrange NAS probably can pull out more than 3Gbps of data.
The problem is, a DDOS targets resources. If your provider doesn't have the pipes to handle it, it won't even get close to your server, and all their infrastructure will probably drown. If the provider can handle it, then you can actually try to do something about it - front-end caching servers with round-robin DNS, firewalling and/or segment blackholing (some providers can easily configure AS blocking), and a scalable multi-tier architecture, so you don't get offline by TCP resource starvation, database or I/O overload. There are some nifty appliances for web filtering (like Citrix NetScaler or whatever is called), but I have no experience with them. They seem to work for big companies, so it may be an option.
Oh, and 100Mbps won't cut it (specially if it's those 10Mbs burstable to 100Mbps). I probably could DoS your server with my internet connection alone:D
Yeah, but if you have reverse proxy or caching servers, you might as well skip nginx and use varnish, or any other specialized solution. Or even Apache (with the plus of having mod_security). It's not like nginx automagically now enables you to do stuff that wasn't possible before. I actually never used nginx as a reverse proxy, but I've read some comments from people that had some issues with it, and switched to varnish instead.
1) Yeah, in test environments with empty or very very small packets. The truth is, PF isn't threaded, so not only you're bound by ethernet I/O, but also by core processing power. Also, there are open port limits, internal buffer limits, and actual traffic limits (data). If you're pulling your data from a cluster (shared storage, database, whatever), it usually translates to more tcp connections. For real workloads and usual datacenter limits (100Mbps or 1Gbps uplink), you won't get nowhere near that value, unless you have multiple servers and/or your own upstream.
2) TCP connection limits (which FreeBSD handles somewhat gracefully), and if you have a fast link, you'll probably have PF performance issues
3) Requests/s is only a part of the equation, and it is meaningless in a DDOS context. Imagine your server receive a flood of requests from another continent - the RTT alone for the 3-way TCP handshake will probably exaust the available ports. So again, yes, I highly doubt you get those numbers in real-world usage (70k requests of a 1Kb file will be real close to saturating a Gigabit link, and that's assuming it's an instantaneous process).
4) That is a good solution, but it's not really nginx-awesomeness related, is it?
What you describe are the usual steps when you have the money to do it, the infrastructure, and the staff to mantain it. Installing nginx as a way of solving DDOS is plain stupid, specially when most optimizations you can apply to nginx are also valid in apache, and the result will be the same - a properly configured environment will usually have - as you said - bandwidth as its bottleneck, regardless of the web technology used.
I'd suggest stop using a shared host, and looking for a real provider. Choose a dedicated server or colo solution (or AWS if you need the redundancy and HA), hire a good reputable company to set it up and mantain it for you. 3Gbps traffic is child's play even for small companies, and even if your dedicated server/colo is hit by this kind of stuff (3Gbps), I doubt it will be prejudicial to other clients.
So, are you saying nginx will work when you receive more TCP requests than your server can handle? Or the upstream router? Or when evey page render is a database hit? Nginx is a lot faster than apache when serving static content, and at the cost of some flexibility. Guess what? Most of the web isn't static content, even if it appears so. Do you think sessions and agent info are logged into ether? Get real.
And yes, I DO use nginx, and it rocks. It's just not the silver bullet you're talking about.
If you read some kind of print media while you poo (a magazine, a newspaper, the toilet paper itself), you are already being targeted by advertisers. How is this different?
It already is a standard feature in many workstation lines (HP and such). Unfortunely, if you want ECC and an Intel CPU, you need to buy Xeon gear, so it gets expensive fast. There is no actual consensus in how reliable DRAM is, but articles like this http://www.zdnet.com/blog/storage/dram-error-rates-nightmare-on-dimm-street/638 reveal how important is to have at least some simple error-detection mechanism.
The Z80 had no standard chipset:D It usually had some auxiliary IC's (mainly from intel), or some custom ASIC, such as spectrum's ULA. But hey, you could actually get Z80 to work with whatever RAM you wanted, using a paging mechanism (akin to EMS on early PCs).
The extra PCIe lanes probably are the most exciting feature, since they can easily be converted to external PCIe. We're talking high-speed cpu->device interconnection, or even in some cases, cpu->cpu (upto 25m) interconnection, at a fraction of the price of 10G or competing technologies. From displays with embedded graphics card to external storage, the applications are much beyond the gamers realm.
From what I've seen from the specs of the latest socket 2011 boards, they allow upto 64GB on desktop models, but I'm not shure I'd like to have that amount of RAM without ECC.
In the MFM/RLL days, usually all disks came with a list of known bad locations, often handwritten on a table label on the top of the driver. There was no true skipping involved (a low-level BIOS format would wipe all the bad sector markings). Modern drives have "intelligent" skipping - all the relocation data is handled internally by the controller (including spare sectors for relocation), and the CHS notion doesn't exist, because the drives have variable density - the inner zones of each plate map to fewer sectors than the outer zones.
Slashdot Mirror
And everyone else. Germany is now realizing that broke customers don't buy german stuff, so the expectation is that all the eurozone will be in recession by the end of this year. France is hanging by a thread. Spain will be rescued by the IMF, it's a matter of time. The recipes you point already have been applied here, without much success. Small high-debt countries are a great target for investment and speculation, specially if they are inserted in some kind of insulation such as the Euro. Speculators can strangle the country's economy, without compromising their paycheck (because someone else will pay the bill). That said, this crisis is a sort of wakeup call for Portugal - we have great potential for business, but so far without producing results. Smaller companies that can't compete are being forced out of business aggressively. Looking two or three decades ahead, this is not a bad thing - the ones that resist will be major players, and new ones will be aware of mistakes of the past. Most of what's being done in Germany (and a part of it is filling up import forms from China) can be done today here at a fraction of the cost. But the mentality of local business (instead of iberic or european business) still strong among the older generation of managers - that are being weed out by the crisis.
You should. Printing money depreciates currency, and on a country highly dependent on energy imports and raw materials such as Portugal, the result would be a massive increase on the cost of energy, transport and production. Yes, even worse than now (I'm also portuguese).
English isn't my native language, but it seems the problem lies with people thinking that breseham's implementation is "traditional". It's not. To be blunt, the base algorithm isn't even integer based and envolves a div (to calculate the slope), so, by all metrics, the original problem is ignorance. The integer bresenham algorithm is quite popular, but it amazes me that people think is a "traditional" approach, instead of solving a 1st degree equation. (Wtf are people learning in schools nowadays?).
Regarding your div remarks - yes, division is a complex process, but easily solved by throwing more transistors at it. The dependency chain between instructions is only relevant to high-penalty cpus (such as P4), or when you have multiple execution pipes, but no out-of-order execution support. Well, the Pentium mechanism is pretty simple, if instructions can be paired on the U and V pipes, they will, if not, tough luck - they'll be executed sequentially. That is the motive I used it - it is simple to understand. Pentium Pro and later processors do feature multiple "all purpose" pipes, but also feature out-of-order execution. Not that matters much, because the point I made was precisely about the futility of single-processor optimization - what is clever on one cpu release may be utterly stupid on another (and almost everything was stupid on the P4 aberration).
Tiny corrections :D
- Traditional line drawing uses a div per pixel. Check pseudocode at the wikipedia page (http://en.wikipedia.org/wiki/Line_drawing_algorithm). There are some "hybrid" algorithms that implement one division per line, but usually are a variation of bresenham's algorithm with better precision.
- I'm not familiarized with Heaven7's internals, but it is somewhat obvious that uses tricks. As most other rtr implementations out there. And some non-realtime raytracers. If you can simplify global illumination (using "post-production" effects), scene complexity (by camera angles and removal of invisible objects), and recursion (limit interpolation and reflection/refraction), you still can get quite impressive results, even with "old" cpus, by today's standards.
Read what you quoted. Then read again. got it? People used bresenham because traditional algoritms implied a div.
I guess you are confusing the FreeBSD TCP stack optimizations with stateful packet inspection - routing, by default, does not touch firewalling.
But hey, straight from the horse's mouth: http://www.openbsd.org/faq/pf/ru/perf.html . That's why there's a new (NetBSD) project called NPF, that aims the creation of a more sofisticated and scalable packet filter.And, afaik, FreeBSD 9.0 TCP optimizations relate to congestion algorithms, not necessarily firewalling.
I often see these misconception of trying to translate internal/testing/development benchmarks into real-case usage. Just because a piece of software measured 100k connections/s on a controlled environment and selected hardware, it doesn't mean it will perform similarly on your machine. Local network application tests are done with clean, well-formed, unrouted traffic. In internet-facing production servers, you'll get all kinds of garbage. The real amount of hits/s you'll get will depend of a plethora of factors, such as average content size, network latency, I/O latency, design of the application, quality of the hardware, quality of the drivers, etc. The choice of webserver is relevant, but not that much. TCP attacks can drown your machine (by network exaustion) long before you see any real advantage of using Nginx regarding DDOS scenarios. It is also very common to see people switching to nginx and being forced to learn what they are doing, so the idea they have of apache is "all modules loaded" and some random options on a control panel.
I use both webservers, for different applications. Nginx is faster in static content, and seems to handle stress very well. Apache is feature-complete with a plethora of third-party modules (such as mod_security) and behaves very well as a general-purpose server (I actually use a lot mpm_itk, so every vhost has its own uid/gid). Sometimes you can squeeze extra performance by trading Apache for Nginx, sometimes it will break your app (complex mod_rewrite rules, for example). But that's relevant for legitimate users - if your network pipes are clogged by requests, both webservers will appear unavailable.
It seems Slashdot stopped notifying me of AC replies, weird :P
If you don't have the infrastructure and/or the staff, outsourcing DDOS protection seems a good choice. Trying to protect against the 3Gbps mark is a bit silly, since next time you may get hit by 10x that traffic. It is important to learn the nature of the attack (are we talking about hundreds of ip addresses, or a dozen? are they "residential connections" or compromised servers? is it a TCP handshake attack, or requests to specific pages? do those pages exist, or is the fancy 404 rewrite eating up resources by pulling the page from the database?), and probably some optimizations can be made to your ecommerce platform (such as reducing requests, pulling images from CDNs, etc - check tools.pingdom.com for basic hints) so that the traffic spent with legitimate customers can be lowered.
Ask yourself this: how much do you lose for each day your ecommerce platform is down? Since you seem to have a single-server solution, if you consider let's say, 5 days of downtime per year, how much would you lose? Do the losses are greater than 50% of cost of a fancy redundant infrastructure? If so, upgrade immediately.
Since you say you are from the UK (hi neighbour, I'm from Portugal), you may also want to check available other providers such as Rackspace. They probably have some cloud-based solution that may fit your budget.
There is an interesting presentation made by Mayhem a couple of years ago (mostly touting OpenBSD's awesomeness) regarding a DDOS case (also "small" traffic), and how they handled it. You can read the pdf at http://2009.confidence.org.pl/materialy/prezentacje/alessio_pennasilico_bakeca_ddos_confidence_2009.pdf
replace SMID with SIMD :P
I honestly do not know how deep pipelines relate positively with performance (during the P4 GHz wars, AMD used a much shorter pipeline, so it could get the same performance at a fraction of the frequency - and that's why a single core of a modern [pentium-m descendant such as core 2] 2.0Ghz processor blows out of the water any >3.0GHz Prescott).
And of course, although demos targeted at old CPUs don't use new CPUs in the most ideal way, they still run much faster on new CPUs than they did on the CPUs for which they were intended.
Yes, but all the cleverness of the algorithm optimization is somewhat lost, isn't it? If the result isn't faster than the reference implementation, why waste time optimizing it at that level? And even if you do waste that time and go the extra mile, how do you feel everytime you need to basicly rewrite all your libraries because the rules of the game have changed?
Thus, the old demos should still work perfectly well.
Except many don't. Because some rely on instruction timing for music sync, some rely on "padding" instructions for delay, some rely on vertical retrace to sync the graphics, as an example of many hardware artifacts that were actually used as a feature. Probably since DX/OpenGL demos compatibility has improved, but try to run an old DOS demo on a modern computer. Or one that relies, as an example, on a processor "defect" such as the 386 "unreal" mode.
I guess I was misinterpreted. I regard scene programmers more like "code artists". The software product itself is unique, in the sense that is not usually maintained or ported. Choosing long-term algorithms and performance compromises is usually not a problem, since it is made to be awesome with a specific combination of hardware. The algorithms have to be fast enough to implement the desired funcionality on a specific point in time (the party or the release date) and usually on a specific processor, not on hardware that will exist 6 months from now.
During the nineties, I've done a lot of scene-related assembly code (mostly graphics and infrastructure - extenders, memory managers, etc), so I'm well aware of the limitations imposed and the usual workarounds. Most sourcecode I've seen is an ugly mess, even if it works. There are some true clever and elegant algorithms in the lower categories (128/256b and 4k), but what I've learned since then is that computer processing power is cheap, but software lasts. One of my favourite demos of all time is Heaven7, mostly because it implements a realtime raytracer than runs smoothly on a P200. But probably many of the optimizations that allowed it to run smoothly in such a limited processor aren't valid for a P3. Or a P4 (handcode optimization for any of the P4 lines is a _nightmare_). So, if it was a maintained application, the highly optimized algorithms would probably have to be rewritten, to take advantage of the latest features, and keeping "pushing the boundaries".
As an example (and more in line with the nineties), a lot of effort was put into highly optimized bresenham line algorithms, because traditional implementations implied a div operation per pixel, and integer division was awfuly slow (like 40 cycles on a 486). So, even if bresenham's requires extra instructions, it would still be a lot faster. Well, on a pentium, not only the div instruction took 1 clock cycle (like most of the other instructions), some instructions could be paired for execution (the processor had 2 execution pipes), so the bresenham implementation is usually a lot slower than using the div instruction. If you had to write maintainable software, would you worry about implementation details that could double your development time, but be obsolete on the next processor to be released? I guess not.
Demos are a kind of development on its own. They require no user interaction, no external data pulling (other than already packaged resources), have no error checking whatsoever and usually are buggy as hell - slightly different hardware may give you completely different results, or just don't run at all. So, that's why I don't like to call scene coders "real programmers". They are more of a class of "code artists", and yes, they should deserve more merit than "regular" coders, not only because their algorithmic skills, but also because of their creative way of implementing them.
So, being aware that are issues when using nginx as a reverse proxy instead of varnish, is useless? And those comments were made here, on slashdot, so god forbid you use a search engine to confirm what issues are those...
Demo scene is the perfect example of what real programmers can do.
I'd say demoscene is the perfect example of what can be done, software-wise. I'd guess you've never peeked at a demo sourcecode, or you wouln't call them generically "real programmers" (incidentally, there is usually much talent behind the code, but no structure or discipline whatsoever).
It's not uncommon for multiple colo customers to be on a shared switch with 1gb uplink especially when each individual customer only has a 100mbit port
Yes, you are right, but if you have DDNS worries, you'll keep away from those providers, and/or opt for a beefier connection. Many of them will happily provide you with detailed information regarding network structure, so you can be aware of what you are actually paying. Also, many of those "cheaper" providers have somewhat frequent DoS attacks inside their own network (some machines get compromised, and then try to scan/infect everybody else), so, again, handling/filtering a 3Gbps attack isn't usually a big deal.
Also the isp has to pay for the bandwidth usage that hits them, even if most of it never actually reaches the end customer...
That's the ISP's problem, and I seriously doubt that the pricing doesn't reflect these "hidden costs".
There are several available solutions, depending on your budget. You could opt for a cloud hosting solution (AWS, GoGrid, etc) so your cluster could scale dynamically according to demand. You could roll your own solution (and there are several options with different price ranges, from simple web cluster to redundant replicated clusters in different datacenters), preferably with a provider that won't panic over 3Gbps traffic. You could use a proxy service like those mentioned in the article.
:D
To put it in perspective, many modern datacenters have 10G pipes to the rack. A modern server can handle 10G traffic with iptables (assuming somewhat simple rules) without issues. A midrange NAS probably can pull out more than 3Gbps of data.
The problem is, a DDOS targets resources. If your provider doesn't have the pipes to handle it, it won't even get close to your server, and all their infrastructure will probably drown. If the provider can handle it, then you can actually try to do something about it - front-end caching servers with round-robin DNS, firewalling and/or segment blackholing (some providers can easily configure AS blocking), and a scalable multi-tier architecture, so you don't get offline by TCP resource starvation, database or I/O overload. There are some nifty appliances for web filtering (like Citrix NetScaler or whatever is called), but I have no experience with them. They seem to work for big companies, so it may be an option.
Oh, and 100Mbps won't cut it (specially if it's those 10Mbs burstable to 100Mbps). I probably could DoS your server with my internet connection alone
Yeah, but if you have reverse proxy or caching servers, you might as well skip nginx and use varnish, or any other specialized solution. Or even Apache (with the plus of having mod_security). It's not like nginx automagically now enables you to do stuff that wasn't possible before. I actually never used nginx as a reverse proxy, but I've read some comments from people that had some issues with it, and switched to varnish instead.
1) Yeah, in test environments with empty or very very small packets. The truth is, PF isn't threaded, so not only you're bound by ethernet I/O, but also by core processing power. Also, there are open port limits, internal buffer limits, and actual traffic limits (data). If you're pulling your data from a cluster (shared storage, database, whatever), it usually translates to more tcp connections. For real workloads and usual datacenter limits (100Mbps or 1Gbps uplink), you won't get nowhere near that value, unless you have multiple servers and/or your own upstream.
2) TCP connection limits (which FreeBSD handles somewhat gracefully), and if you have a fast link, you'll probably have PF performance issues
3) Requests/s is only a part of the equation, and it is meaningless in a DDOS context. Imagine your server receive a flood of requests from another continent - the RTT alone for the 3-way TCP handshake will probably exaust the available ports. So again, yes, I highly doubt you get those numbers in real-world usage (70k requests of a 1Kb file will be real close to saturating a Gigabit link, and that's assuming it's an instantaneous process).
4) That is a good solution, but it's not really nginx-awesomeness related, is it?
What you describe are the usual steps when you have the money to do it, the infrastructure, and the staff to mantain it. Installing nginx as a way of solving DDOS is plain stupid, specially when most optimizations you can apply to nginx are also valid in apache, and the result will be the same - a properly configured environment will usually have - as you said - bandwidth as its bottleneck, regardless of the web technology used.
I'd suggest stop using a shared host, and looking for a real provider. Choose a dedicated server or colo solution (or AWS if you need the redundancy and HA), hire a good reputable company to set it up and mantain it for you. 3Gbps traffic is child's play even for small companies, and even if your dedicated server/colo is hit by this kind of stuff (3Gbps), I doubt it will be prejudicial to other clients.
So, are you saying nginx will work when you receive more TCP requests than your server can handle? Or the upstream router? Or when evey page render is a database hit? Nginx is a lot faster than apache when serving static content, and at the cost of some flexibility. Guess what? Most of the web isn't static content, even if it appears so. Do you think sessions and agent info are logged into ether? Get real.
And yes, I DO use nginx, and it rocks. It's just not the silver bullet you're talking about.
If you read some kind of print media while you poo (a magazine, a newspaper, the toilet paper itself), you are already being targeted by advertisers. How is this different?
It already is a standard feature in many workstation lines (HP and such). Unfortunely, if you want ECC and an Intel CPU, you need to buy Xeon gear, so it gets expensive fast. There is no actual consensus in how reliable DRAM is, but articles like this http://www.zdnet.com/blog/storage/dram-error-rates-nightmare-on-dimm-street/638 reveal how important is to have at least some simple error-detection mechanism.
The Z80 had no standard chipset :D It usually had some auxiliary IC's (mainly from intel), or some custom ASIC, such as spectrum's ULA. But hey, you could actually get Z80 to work with whatever RAM you wanted, using a paging mechanism (akin to EMS on early PCs).
The extra PCIe lanes probably are the most exciting feature, since they can easily be converted to external PCIe. We're talking high-speed cpu->device interconnection, or even in some cases, cpu->cpu (upto 25m) interconnection, at a fraction of the price of 10G or competing technologies. From displays with embedded graphics card to external storage, the applications are much beyond the gamers realm.
From what I've seen from the specs of the latest socket 2011 boards, they allow upto 64GB on desktop models, but I'm not shure I'd like to have that amount of RAM without ECC.
In the MFM/RLL days, usually all disks came with a list of known bad locations, often handwritten on a table label on the top of the driver. There was no true skipping involved (a low-level BIOS format would wipe all the bad sector markings). Modern drives have "intelligent" skipping - all the relocation data is handled internally by the controller (including spare sectors for relocation), and the CHS notion doesn't exist, because the drives have variable density - the inner zones of each plate map to fewer sectors than the outer zones.