I frequently do R&D work in this kind of area and I am familiar with the in-the-trenches details here. It really isn't as simple as you're making out.
For example, you referred to using regular expressions for the decoders, but there are several details you're glossing over. The first is that you're presumably referring to application layer processing, but before you can do that you have to get hold of that application layer data and get it to something that can process it.
Just identifying which application protocol is in use and therefore which analysis tools are appropriate may not be straightforward. Consider the number of protocols that use some sort of control channel to establish connections but then send their data over some arbitrary UDP port, for example. You need all kinds of stateful analysis to do this reliably, and each case needs to be custom written.
Assuming you can isolate the application layer data, you then need to process it, at line rates, to extract the required metadata, which needs to be stored at line rates as well. The TCAM-based filtering built into a switch ASIC isn't going to implement any sort of regex matching, because that's not how a TCAM works. Compiling usefully detailed regexes to run on a sufficiently powerful FPGA is more plausible, but it's no trivial exercise either. (If it were, a large part of the network tools industry would not be the shape it is and some colleagues of mine would have retired very rich by now.) You can do some quite clever things bringing network traffic through a general purpose CPU, and you can build a device around it that could cope with a surprising amount of throughput using high-end but off-the-shelf components today and of course offers the best flexibility, but only if you can support the dramatically higher power, cooling and rack space requirements in your data centre.
It's true that without needing any application layer processing it would still be practical to record all the TCP connections that took place between two connected devices over your network with current hardware. With the co-operation of whoever operated the server-side of a communications network you could then reconcile the connections to figure out who was communicating when, at least up to IP address and port number. But in that case the valuable information is primarily what you're relying on the communication network to provide; what is recorded by the ISP doesn't really tell you anything very useful, certainly not on the level government spokespeople have been talking about when they say "metadata".
In short, extracting what you call "unique entropy" is straightforward, but it's also almost worthless without the real data set you care about to correlate with. You're still relying on some much more sophisticated deep packet analysis and/or the co-operation of at least one of the participants in the communication, and both of those things will normally need to be set up on a case-by-case basis.
But security updates are not the same as general feature/look-and-feel updates. You can have one without the other.
And default-to-on updates are not the same as mandatory updates. You can protect people who don't know better just as much either way, but the only reason to mandate updates is to force the user to install something they actively don't want to.
It seems highly unlikely that this is the interpretation the government is looking for. They've been quite explicit that they don't just want to know which communications channels you use, but also who you communicate with and the like.
The trouble is that, as many here will understand but I fear many in the government do not, there is no black and white distinction to be made based on some universal technical test to achieve the results the authorities say they want. Leaving aside the usual issues with encryption and reluctant foreign services, you're effectively talking about deep packet inspection in real time of many gigabits/second of network traffic, applying custom processing based on numerous specific protocols and/or service providers to each packet, and then recording the remaining payload after irrelevant parts are stripped.
They've also been talking about broadband being some sort of fundamental right one minute, with ominous-sounding ideas about cutting people off for dubious IP-related reasons the next, and then moving government services that many people are legally required to use into on-line systems the day after that.
I'm pretty sure it's all just an elaborate episode of Yes Minister at this point.
The masses never voted against Blair. They voted against Brown.
In which country? Because in the UK, Blair only received about 40% of the popular vote even on New Labour's second term, and only about 35% going into the third term. In each case the turnout was around the 60% mark. That "historic third term" with the third big majority of MPs in a row was based on the support of just 1 in 5 of the electorate. And even that was with the clear and unambiguous promise that Blair himself would lead the party for a full third term and wouldn't hand over -- as he ultimately did -- to Gordon Brown mid-term.
I don't run Windows 10, either at home or on my work machines, for much the same reasons many here are posting. But even with the various issues over privacy, forced updates and the like, I still don't think it's reasonable or useful to call it a walled garden today.
Of course, with forced updates, you have no guarantee they won't turn it into one over time. Did I mention I don't run Windows 10?:-)
OS X systems also tend to be updatable more times before the newest accompanying hardware undergoes some major change that prevents the upgrade from running on older systems.
It's not having the ability to update that I object to, it's being forced to update the whole OS just to keep up with basic security and stability issues.
For one thing, upgrading your OS is never a trivial exercise and always carries some risk in the real world, even if you're careful. A major update might fix some security issues but introduce new ones as part of new or modified features, for example.
Just as important, you might simply not like the new version as much as the old one. Maybe that's because of the UI look and feel. Maybe they changed the way some feature you rely on worked, or removed it altogether. Maybe they dropped compatibility with some third party software you use. In any case, you shouldn't have to feel like you're playing the lottery about how long a computer is going to continue working in the same way it was supposed to when you bought it.
Devices for professional use and at this kind of price level should be supplied with software relevant to a generous reasonable working lifetime. If that software was defective (security vulnerabilities, instability, whatever) then it should be properly fixed so it works properly and as advertised. Security screw-ups are not an excuse to get the user to change to other software that wasn't what they signed up for.
iOS is a walled garden. OS X is not. It's basically an adapted BSD under the hood with Apple's custom OS X GUI and other services on top, and it has no more trouble installing third party software, accessing the underlying filesystems, or communicating with remote systems than a Windows system.
There's a certain irony that the one thing that really puts me off Apple gear, both iOS devices and mainstream OS X computers, is the lack of commitment to long term support. I don't want to buy a device and find the OS isn't even getting security patches within five minutes unless I update to some new version that I might or might not want. I want to buy a device where the software is supported for the working lifetime of the machine and whether to install updates for anything other than security/stability/compatibility is up to me and an independent decision.
Whatever else you can say about Microsoft, until very recently they always made a serious effort to support Windows systems long-term. But then with Windows 10 they've baked in the forced updates, which removes the one thing that almost guaranteed I'd be buying Windows and not OS X machines for the foreseeable future.
Don't worry. You can rest assured that your pattern of watching shows relating to civil liberties, legalising recreational drug consumption, and footage of police raids does not in any way reflect upon your character as viewed by the authorities. We were just passing through your neighbourhood offering a free swatting to randomly selected individuals.
Fortunately for me, the local cell reception where I live (about a mile from a city centre) is so awful that any such partnership is unlikely to trouble me. Ha! Take that, corporate evil-doers!
Why the hell are we paying for the privilege of being advertised to?
Because, sadly, it appears to be what the market will bear.
Even a few years ago, when I wanted to buy a new TV I had to explicitly say I wanted one that wasn't "smart", just a good screen and sensible inputs. That narrowed the range I could choose from quite significantly. Today, hardly anything I could buy off the shelf from any local store does not have these "smart" features built-in, even though they are almost invariably poorly functioning, quickly obsolete, or outright customer-hostile as with the privacy and security issues.
Well, they do also finally deliver that watched-in-your-own-home experience that the brochure was trying to sell as far back as 1949.
I mean, you have a camera literally watching your living room, an Internet connection, and a load of software written and maintained (or not) by people whose interests are unlikely to include your security or privacy. What could possibly go wrong? It's not like any major brands have had problems with this already or anything.
Don't worry, in a few years the big brands will all be generating their own mesh networks with home "smart" devices for these purposes anyway. Then they don't need anything other than your house not to be inside a large metal frame, and they won't have to trouble you for a network connection (or permission) to track you, as long as someone somewhere on the mesh helpfully provides one.
If the job is impossible then there will soon be obvious, highly visible problems. To maintain law and order, society will then have to come up with democratically acceptable solutions to those problems, which might include legislating to give more or different powers to law enforcement and accepting the consequences.
But hypothetical problems aren't very interesting, and fear of hypothetical problems should not be allowed undue influence in public debate.
I've heard a few disturbing stories over the years that sound a lot like that, but I always imagined it was just hyperbole, at least in most cases. If that is literally true, then it is hard to see how the police organisations with such policies could ever maintain the good community relations necessary for effective police work. Creating a them-and-us culture seems like a sure route to all kinds of problems, not least losing the willingness of witnesses or other relevant members of the public to step up and help.
This sounds like the comments of someone obviously blind to the realities of stepping into a hostile crowd alone.
Why was the hostile crowd there, why was it hostile, why was it necessary for law enforcement to enter it, and why was the officer doing so alone? If your hypothetical problem situation ever actually happens, it sounds like a whole lot of things probably already went wrong long before the officer stepped into the crowd.
It's odd that this sort of situation seems to happen so much more often in the US than most places, though. It's hard to get an accurate picture from outside based on just what the TV reports, because that will naturally highlight the big wins and big failures but probably most police work fits somewhere in between. Still, the picture of US law enforcement that is shown to the outside world is often not a positive one, and makes me wonder how much of any cultural problems with law enforcement in the US were caused by the past behaviour of the law enforcement organisations themselves. Perhaps borrowing more of the community-based neighbourhood policing that is used in a lot of other places and accepting the greater degree of scrutiny they now operate under will help, at least in the long term.
I've also seen reports that at certain times NetFlix represents a high proportion of overall Internet traffic, though I don't recall any with a figure nearly as high as the 80% you mentioned. Maybe that's been true somewhere at some time. But with the rise of things like cloud services, the volume of data flying around during the business day is surely increasing as well. Also, I suspect you underestimate the bandwidth requirements of, say, a RTS game with half a dozen players, each with hundreds or thousands of units, where there are tens or hundreds of thousands of moving things to sync up six ways. But as I said before, I still don't think the point is really which specific services might require more bandwidth and/or lower latency. The point is just that some services are more demanding than others, and traffic shaping in favour of each service being as useful as possible is not unreasonable.
Well, as I mentioned in another post, I've personally seen significant battery degradation in Apple gear much less than that age, so frankly I don't think their technology is as exceptional as you're making out.
But the thing is, even if we ignore that, and we ignore the mountain of other evidence that Apple tries to build in obsolescence and make its equipment hard to maintain in the long run, it still remains true that any failure in an Apple mobile device is a failure of the entire unit. Since swapping it out isn't a viable option for those with security/privacy concerns, using a more serviceable device is still a potential advantage for those customers.
Presumably that is a legitimate support question that the manufacturer will have to answer here. The article implies that the expected lifetime is five years and that during that time both spare parts and software updates will be available even if newer and possibly incompatible models have been released in the interim.
That depends entirely on the network. In some places I've lived/worked, on-line gaming was a heavy contributor to overall data volumes. In some places, video calls have become so (and obviously aren't amenable to the kind of caching you mentioned). I don't think that any particular examples are really the point anyway, though.
Slashdot Mirror
I frequently do R&D work in this kind of area and I am familiar with the in-the-trenches details here. It really isn't as simple as you're making out.
For example, you referred to using regular expressions for the decoders, but there are several details you're glossing over. The first is that you're presumably referring to application layer processing, but before you can do that you have to get hold of that application layer data and get it to something that can process it.
Just identifying which application protocol is in use and therefore which analysis tools are appropriate may not be straightforward. Consider the number of protocols that use some sort of control channel to establish connections but then send their data over some arbitrary UDP port, for example. You need all kinds of stateful analysis to do this reliably, and each case needs to be custom written.
Assuming you can isolate the application layer data, you then need to process it, at line rates, to extract the required metadata, which needs to be stored at line rates as well. The TCAM-based filtering built into a switch ASIC isn't going to implement any sort of regex matching, because that's not how a TCAM works. Compiling usefully detailed regexes to run on a sufficiently powerful FPGA is more plausible, but it's no trivial exercise either. (If it were, a large part of the network tools industry would not be the shape it is and some colleagues of mine would have retired very rich by now.) You can do some quite clever things bringing network traffic through a general purpose CPU, and you can build a device around it that could cope with a surprising amount of throughput using high-end but off-the-shelf components today and of course offers the best flexibility, but only if you can support the dramatically higher power, cooling and rack space requirements in your data centre.
It's true that without needing any application layer processing it would still be practical to record all the TCP connections that took place between two connected devices over your network with current hardware. With the co-operation of whoever operated the server-side of a communications network you could then reconcile the connections to figure out who was communicating when, at least up to IP address and port number. But in that case the valuable information is primarily what you're relying on the communication network to provide; what is recorded by the ISP doesn't really tell you anything very useful, certainly not on the level government spokespeople have been talking about when they say "metadata".
In short, extracting what you call "unique entropy" is straightforward, but it's also almost worthless without the real data set you care about to correlate with. You're still relying on some much more sophisticated deep packet analysis and/or the co-operation of at least one of the participants in the communication, and both of those things will normally need to be set up on a case-by-case basis.
But security updates are not the same as general feature/look-and-feel updates. You can have one without the other.
And default-to-on updates are not the same as mandatory updates. You can protect people who don't know better just as much either way, but the only reason to mandate updates is to force the user to install something they actively don't want to.
Except for everyone who purchased Windows up until Windows 10, you mean?
It seems highly unlikely that this is the interpretation the government is looking for. They've been quite explicit that they don't just want to know which communications channels you use, but also who you communicate with and the like.
The trouble is that, as many here will understand but I fear many in the government do not, there is no black and white distinction to be made based on some universal technical test to achieve the results the authorities say they want. Leaving aside the usual issues with encryption and reluctant foreign services, you're effectively talking about deep packet inspection in real time of many gigabits/second of network traffic, applying custom processing based on numerous specific protocols and/or service providers to each packet, and then recording the remaining payload after irrelevant parts are stripped.
They've also been talking about broadband being some sort of fundamental right one minute, with ominous-sounding ideas about cutting people off for dubious IP-related reasons the next, and then moving government services that many people are legally required to use into on-line systems the day after that.
I'm pretty sure it's all just an elaborate episode of Yes Minister at this point.
The masses never voted against Blair. They voted against Brown.
In which country? Because in the UK, Blair only received about 40% of the popular vote even on New Labour's second term, and only about 35% going into the third term. In each case the turnout was around the 60% mark. That "historic third term" with the third big majority of MPs in a row was based on the support of just 1 in 5 of the electorate. And even that was with the clear and unambiguous promise that Blair himself would lead the party for a full third term and wouldn't hand over -- as he ultimately did -- to Gordon Brown mid-term.
I don't run Windows 10, either at home or on my work machines, for much the same reasons many here are posting. But even with the various issues over privacy, forced updates and the like, I still don't think it's reasonable or useful to call it a walled garden today.
Of course, with forced updates, you have no guarantee they won't turn it into one over time. Did I mention I don't run Windows 10? :-)
OS X systems also tend to be updatable more times before the newest accompanying hardware undergoes some major change that prevents the upgrade from running on older systems.
It's not having the ability to update that I object to, it's being forced to update the whole OS just to keep up with basic security and stability issues.
For one thing, upgrading your OS is never a trivial exercise and always carries some risk in the real world, even if you're careful. A major update might fix some security issues but introduce new ones as part of new or modified features, for example.
Just as important, you might simply not like the new version as much as the old one. Maybe that's because of the UI look and feel. Maybe they changed the way some feature you rely on worked, or removed it altogether. Maybe they dropped compatibility with some third party software you use. In any case, you shouldn't have to feel like you're playing the lottery about how long a computer is going to continue working in the same way it was supposed to when you bought it.
Devices for professional use and at this kind of price level should be supplied with software relevant to a generous reasonable working lifetime. If that software was defective (security vulnerabilities, instability, whatever) then it should be properly fixed so it works properly and as advertised. Security screw-ups are not an excuse to get the user to change to other software that wasn't what they signed up for.
iOS is a walled garden. OS X is not. It's basically an adapted BSD under the hood with Apple's custom OS X GUI and other services on top, and it has no more trouble installing third party software, accessing the underlying filesystems, or communicating with remote systems than a Windows system.
There's a certain irony that the one thing that really puts me off Apple gear, both iOS devices and mainstream OS X computers, is the lack of commitment to long term support. I don't want to buy a device and find the OS isn't even getting security patches within five minutes unless I update to some new version that I might or might not want. I want to buy a device where the software is supported for the working lifetime of the machine and whether to install updates for anything other than security/stability/compatibility is up to me and an independent decision.
Whatever else you can say about Microsoft, until very recently they always made a serious effort to support Windows systems long-term. But then with Windows 10 they've baked in the forced updates, which removes the one thing that almost guaranteed I'd be buying Windows and not OS X machines for the foreseeable future.
I think most of the way tech is shown in CSI counts as criminal, but the two aren't mutually exclusive. :-)
Don't worry. You can rest assured that your pattern of watching shows relating to civil liberties, legalising recreational drug consumption, and footage of police raids does not in any way reflect upon your character as viewed by the authorities. We were just passing through your neighbourhood offering a free swatting to randomly selected individuals.
Fortunately for me, the local cell reception where I live (about a mile from a city centre) is so awful that any such partnership is unlikely to trouble me. Ha! Take that, corporate evil-doers!
Why the hell are we paying for the privilege of being advertised to?
Because, sadly, it appears to be what the market will bear.
Even a few years ago, when I wanted to buy a new TV I had to explicitly say I wanted one that wasn't "smart", just a good screen and sensible inputs. That narrowed the range I could choose from quite significantly. Today, hardly anything I could buy off the shelf from any local store does not have these "smart" features built-in, even though they are almost invariably poorly functioning, quickly obsolete, or outright customer-hostile as with the privacy and security issues.
Well, they do also finally deliver that watched-in-your-own-home experience that the brochure was trying to sell as far back as 1949.
I mean, you have a camera literally watching your living room, an Internet connection, and a load of software written and maintained (or not) by people whose interests are unlikely to include your security or privacy. What could possibly go wrong? It's not like any major brands have had problems with this already or anything.
Don't worry, in a few years the big brands will all be generating their own mesh networks with home "smart" devices for these purposes anyway. Then they don't need anything other than your house not to be inside a large metal frame, and they won't have to trouble you for a network connection (or permission) to track you, as long as someone somewhere on the mesh helpfully provides one.
If the job is impossible then there will soon be obvious, highly visible problems. To maintain law and order, society will then have to come up with democratically acceptable solutions to those problems, which might include legislating to give more or different powers to law enforcement and accepting the consequences.
But hypothetical problems aren't very interesting, and fear of hypothetical problems should not be allowed undue influence in public debate.
I've heard a few disturbing stories over the years that sound a lot like that, but I always imagined it was just hyperbole, at least in most cases. If that is literally true, then it is hard to see how the police organisations with such policies could ever maintain the good community relations necessary for effective police work. Creating a them-and-us culture seems like a sure route to all kinds of problems, not least losing the willingness of witnesses or other relevant members of the public to step up and help.
This sounds like the comments of someone obviously blind to the realities of stepping into a hostile crowd alone.
Why was the hostile crowd there, why was it hostile, why was it necessary for law enforcement to enter it, and why was the officer doing so alone? If your hypothetical problem situation ever actually happens, it sounds like a whole lot of things probably already went wrong long before the officer stepped into the crowd.
It's odd that this sort of situation seems to happen so much more often in the US than most places, though. It's hard to get an accurate picture from outside based on just what the TV reports, because that will naturally highlight the big wins and big failures but probably most police work fits somewhere in between. Still, the picture of US law enforcement that is shown to the outside world is often not a positive one, and makes me wonder how much of any cultural problems with law enforcement in the US were caused by the past behaviour of the law enforcement organisations themselves. Perhaps borrowing more of the community-based neighbourhood policing that is used in a lot of other places and accepting the greater degree of scrutiny they now operate under will help, at least in the long term.
But... But... If they have nothing to hide, they should have nothing to fear!
I've also seen reports that at certain times NetFlix represents a high proportion of overall Internet traffic, though I don't recall any with a figure nearly as high as the 80% you mentioned. Maybe that's been true somewhere at some time. But with the rise of things like cloud services, the volume of data flying around during the business day is surely increasing as well. Also, I suspect you underestimate the bandwidth requirements of, say, a RTS game with half a dozen players, each with hundreds or thousands of units, where there are tens or hundreds of thousands of moving things to sync up six ways. But as I said before, I still don't think the point is really which specific services might require more bandwidth and/or lower latency. The point is just that some services are more demanding than others, and traffic shaping in favour of each service being as useful as possible is not unreasonable.
Well, as I mentioned in another post, I've personally seen significant battery degradation in Apple gear much less than that age, so frankly I don't think their technology is as exceptional as you're making out.
But the thing is, even if we ignore that, and we ignore the mountain of other evidence that Apple tries to build in obsolescence and make its equipment hard to maintain in the long run, it still remains true that any failure in an Apple mobile device is a failure of the entire unit. Since swapping it out isn't a viable option for those with security/privacy concerns, using a more serviceable device is still a potential advantage for those customers.
Presumably that is a legitimate support question that the manufacturer will have to answer here. The article implies that the expected lifetime is five years and that during that time both spare parts and software updates will be available even if newer and possibly incompatible models have been released in the interim.
That depends entirely on the network. In some places I've lived/worked, on-line gaming was a heavy contributor to overall data volumes. In some places, video calls have become so (and obviously aren't amenable to the kind of caching you mentioned). I don't think that any particular examples are really the point anyway, though.