An interesting gun... Here in People-Who-Are-Actually-Professional-Network-EngineersVille, we'd simply accept that our current cheapest-available-transit-provider has shitty connectivity (really? Cogent? really really? Well done, Netflix. Not pinching any pennies, at all) and get a provider that didn't offer bargain bin connectivity and shitty routes to just about everyone. But hey. It's entirely the receiving network's fault.
FTR, NetFlix uses a number of different transit providers, not just Cogent.
Windows RT was simply Microsoft's hedge in the x86/ARM battle. If ARM had utterly dominated Intel in the low-power processor market and the world moved away from Wintel, then RT would've been Microsoft's safety net. If that had happened and Microsoft hadn't made RT, all the armchair quarterbacks currently criticizing Microsoft for making RT would've been criticizing them for not making RT and missing the ARM boat. RT didn't need to succeed. It just needed to be there.
ARM is utterly dominating the low-power market.
Fact is, Windows can't compete in that market as its designed for a high-power environment.
From a security point-of-view, you need to be able to validate that a pointer is valid beyond whether or not it is NULL. You need to know that your application issued the pointer and that the data it points to is valid and within your application space. And this needs to be in real applications, not debug mode.
What do you mean? Under what circumstances is this kind of pointer validation required? It sounds like this is an attempt to detect other bugs, after-the-fact (reading uninitialized or over-written memory, for example).
In implementing a true Secure Programming methodology you don't trust anything. You don't trust that a pointer coming from another part of the program is valid; you need to be able to verify it. Some libc environments give you the ability to validate a heap pointer, but you still cannot validate a stack pointer. So there's a limited ability to really do much with this kind of security testing, which has to be done on-the-fly in the real program (not only debug mode). This is security testing; not program validation, though it can provide that too. It is preventative functionality to keep back actors from being able to break into your program, and if they do extremely limit their ability to go farther by the vary nature of the program itself.
Also, I am still not sure what you mean by "remove unused memory". Remember that the unit of work, in a managed heap, is either the number of live objects or the number of live bytes. "Unused" (or dead or fragmented) memory is not a cost factor.
Unused memory is certainly a cost factor when it comes to systems of programs being able to use the memory on the system. memmapp'd memory is not unlimited. Memory is not unlimited. You only have so much and you have to be a good citizen with its use. Even if you have a high volume of memory allocations, it can be done; and the standard (non-GC) system will certainly keep your memory for only the life-time you need it, no longer; even re-using similarly sized chunks (in some implementations) to minimize calls to the kernel allocator. The kernel can and does notify the libc environment when memory is low, and the libc environment will return free'd memory to the system when that happens. Been there; done that.
more so the "promise of JIT" which gives them a pretty serious win.
So why is Android moving away from using JIT? Why does Android ART binary compile all applications at install?
JIT is a problem. GC is a problem. Combined they make problems worse. Yes, there are situations that they are good for; but not for embedded systems, or phones, or tablets, or most user facing applications. Like many things they are very good for certain kinds of tasks, and that's it.
Not quite. GC are always built on top of malloc+free, not side-stepping them.
This is incorrect. High performance GC implementations are typically built on top of the platform's virtual memory routines, directly (on POSIX, think mmap or shmem). This avoids the waste of "managing the memory manager" and also allows the GC fine-grained control over the heap. On some platforms, this also provides selective disclaim capabilities meaning that the GC actually will give pages back to the OS when contracting the heap, whereas free() wouldn't.
And those versions are few and far between, exception not the rule.
But that doesn't mean that your program should just crash because it failed; the program should degrade gracefully in those situations.
Agreed. To avoid this problem, the better pieces of software I have seen did no dynamic allocation once they reached a steady running state. This meant the failure states were easier to wrangle since you could only fail to allocate during bootstrap or when substantially changing running mode.
Define a steady-state. Not every application has one. This is why real-time stuff doesn't do that - they allocate memory/blocks on the stack at the application (global) level. If you can load the application then everything the application will ever need is allocated. If you cannot load the application, then that's it.
That works for the real-time work; but if you want to do dynamic checking then you need to be able to validate pointers.
Of course, I'd go further and say the standard libs and operating systems should provide a method to validate pointers, but that's more a security concern and the hard part for that is figuring out if the object is valid and on the stack versus valid and on the heap.
The general approach is that a valid pointer is anything non-null. If you need to further introspect the memory mapping or sniff the heap to determine validity, you seriously need to re-think an algorithm. Debugging tools, of course, are exempt from this rule since they are often running on a known-bad address space.
True, that is the general approach. My point is that that is wrong. From a security point-of-view, you need to be able to validate that a pointer is valid beyond whether or not it is NULL. You need to know that your application issued the pointer and that the data it points to is valid and within your application space. And this needs to be in real applications, not debug mode.
Now in doing the reference counter in the GC, then yes - it becomes a lot more expensive to keep it and maintain it because it has no knowledge of the actual use of the pointer.
GCs do not store reference counts since they are completely different from this kind of tracking. They determine validity by reachability at GC time.
Which is a major draw back for using a GC as it now has to crawl everything periodicially.
GC allocation will never be faster than non-GC allocation because it relies on non-GC allocation underneath. Anything the underlying libraries and kernel do, it does as well.
This is incorrect. High performance GCs manage their heap directly and can offer allocation routines based on this reality. The main reason why they can be faster is that they have the ability to move live objects at a later time so fragmentation doesn't need to be proactively avoided in allocation, which is normally what causes pain for malloc+free.
So now you're adding indirect pointers for normal pointer usage...which again now means two calls for every pointer and now you've slowed down the whole application. Smart pointers do the same thing in a sense; as does the PIMPL design pattern - it can still be quite fast, but is still (provably) slower than directly using the poin
By that definition of "wasted", even a basic malloc+free system will waste massive amounts of memory since free rarely actually removes the heap space it is using from the process. Most of the time, someone using a GC configures it to avoid returning this memory, as well, since the environment is typically tuned against the maximum heap size, so giving it back would just mean it has to be requested again, later.
Not quite. GC are always built on top of malloc+free, not side-stepping them.
Further, even if malloc/free didn't return it to the OS, it will still try to alloc from the "free" memory it has before getting more memory from the OS in many implementations. GCs even delay the ability to do that.
Of course, if this is a concern, we are getting more into the debate of whether dynamic allocation should be used, at all (which is something I wish more people thought about - I also like you that you mentioned it can fail since so many people forget that).
Yes. All too often people just code assuming that an allocated memory buffer returns a valid value; usually you'll get something like "it never fails" or "if it fails then there are other issues". But that doesn't mean that your program should just crash because it failed; the program should degrade gracefully in those situations.
Of course, I'd go further and say the standard libs and operating systems should provide a method to validate pointers, but that's more a security concern and the hard part for that is figuring out if the object is valid and on the stack versus valid and on the heap.
I am not sure what you mean by "dynamic allocation time" in a GC and reference counting is NOT "sufficiently quick".
"dynamic allocation" means anything done with malloc/new versus a stack-based allocation, which could be on the local, sem-local (e.g class object), or global stack frames.
Now, regarding reference counting - that all depends on the implementation. Some systems (e.g Ada) do reference counting on the memory allocated - that is, the first couple bytes store a reference count and it's just built-in. Other, like Qt, do it by keeping references (e.g. is X in the child list). And then there is the shared pointer methods (e.g QSharedPointer, std::shared_ptr) which also keep a very quick counter.
Now in doing the reference counter in the GC, then yes - it becomes a lot more expensive to keep it and maintain it because it has no knowledge of the actual use of the pointer.
Actual allocation cost, when using a GC-managed heap is generally incredibly cheap (because the allocator doesn't have to do heap balancing, etc). Reference counting can involve walking massive parts of the heap, and writing to it (sometimes many times). GC is generally very fast but, again, depends entirely on the number of live objects so it becomes more expensive the larger the live set becomes.
GC allocation will never be faster than non-GC allocation because it relies on non-GC allocation underneath. Anything the underlying libraries and kernel do, it does as well.
What do you mean by "as the system needs better and better performance GCs become less and less useful"? A good GC will actually increase the performance of the application as it can improve cache density of the application's live set (not to mention memory-processor affinity).
We do seem to be having 2 conversations here:
1) How does GC compare to other dynamic allocation approaches
2) How does any dynamic allocation approach fare against static or stack-oriented allocation approaches. In this case, I think we both agree that avoiding the issue altogether is preferable, where possible.
GCs will never improve cache performance as it is entirely unrelated and should not be randomly selecting objects. If anything it will decrease cache performance because it will be randomly hitting nodes (during its checks) that the application wants to keep dormant for a time.
Quoting parent; but typically it would be "memory that is not in use by the application but assigned to the application because it has not yet been returned to the OS".
So in a manner, GCs lead to intentional, but temporary, memory leaks. That could be forgiven if there were not other issues regarding using GCs.
If you are generally referring to unpredictable pause times, then that is a real concern of GCs (and some general cases of reference counting and some cases of dynamic allocation). Of course, in the case of the GC, the pause time is a function of the live objects (and, in some cases, their size or topology)
Whether or not you use a GC you still have the dynamic allocation time; to that is a moot point; in some cases you still have the reference counting, and again that is sufficiently quick enough that that too is a moot point.
However, GCs really hurt performance and testability with the unpredictability of the garabage collection time. You can't predict, or control, when a GC decides to check the memory or return it. It could hit at your most critical performance moment, or it may hit at your least critical performance moment, or any time in-between. That is just the nature of the GC.
Yes, certain GC algorithms perform better in some scenarios in others. Some GC implementations provide the ability to control some of it; but they're the exception not the rule.
About the only method of "GC" (and I put that in quotes because it's not really a GC) is how Qt does it with parent objects cleaning up any child object that remains when the parent object is destroyed. It has nearly all the effects of a GC, but all the control and performance of a non-GC system. Yes you can still end up with memory leaks should you forget to parent an object; but even that is generally easy to track down and fix (because something else will usually break as well); and there are very few objects that you have to actually track because they cannot actually have a parent.
Of course, there are GCs which offer predictable pauses but they are typically lower performance. They only matter in real-time environments so they are not often used.
No. They matter in nearly every environment. And I'm pretty certain this is one of main performance issues for the DalvikVM in Android for numerous applications.
If in true Real-Time or even Near-Real-Time application you NEVER allocate memory regardless of whether you use a GC - and especially if you use a GC - because you don't want to have to deal with the possibility the memory may not get allocated. So when you're talking about using a GC, you're not talking about high performance applications. You're talking about everything else.
For everything else your performance comes down to user-perception in most cases (outside of games) and GC collection will hurt that. In games, it can destroy it. User's don't like to have their applications unpredictably stop for a while; they deal with it but they don't like it. And if given a choice between a program that will pause for a while and one that won't, they'll go to the one that won't.
Now GCs can be good for certain environments - where user's don't interact with the system, it's not anywhere near real-time, and the application may have time periods where it simply doesn't matter - e.g a server handling connections where it can clean up after the connection closes and before it handles another; but even then, as the system needs better and better performance GCs become less and less useful.
The problem for Olivier Bloch, Senior Technical Evangelist and Microsoft Open Technologies, Inc. is that they are isolated. The parent company only tells them enough and shows them around enough so that he and others like him can say what they do with a straight face. Meanwhile, the left hand knoweth not what the right hand doth.
I witness this a couple years ago at POSSCON. We had someone that had a similar position at Microsoft give a Keynote. He talked about all the things Microsoft did and everything they showed him and how the teams were open to Open Source. But while he was there saying those things, Microsoft corporate were actively denouncing open source - Balmer and all his lieutenants and even their lieutenants. The speaker truly believed what he was saying - and what he saw. They just sufficiently isolated him such that he was able to believe it and therefore say it. Most of the audience was aware of what Microsoft corporate was saying that directly contracted the speaker they had provided. Sadly, they heckled and booed him instead of showing some professional courtesy, especially since Microsoft was a big sponsor that year.
All that said, it's not that Microsoft can't turn around and become Open Source friendly. It's just that there is such a beaurocratic momentum against it within the company that it will be years before any kind of turn around can effectively happen, unless the have a mass firing of the upper level management, which is not likely.
...and yet, all of Microsoft's flagship products, AFAIK, are the polar opposite of open source. If Microsoft truly thought anything of open source, this should not be the case.
Well, WiX is kind of a flagship product - it's embedded in Visual Studios, utilized by most all of their projects, and Open Source (originally GPLv2, I forget what the current license is).
There is no problem with "wasted" memory backing dead objects between collections since the heap is typically of fixed size between collections (especially in situations like Java where the VM parameters are fixed at start-up time). Reference counting designs not only fail to collect non-trivial object graphs but also do not free memory immediately (although it is typically deterministic with response to call-return behaviour). Now, returning from a function could involve walking every object in the heap, perhaps many times, to update reference counts. GC only incurs cost at GC time, and that is proportional to the number of live objects, not dead objects (which typically dominate).
This is actually a big issue for GCs. It's not that the memory is "wasted" but that the collection of the "wasted" memory leads to unpredictable performance penalties. It's best to avoid GCs entirely for this reason alone.
But it isn't unsupported! The amount of bugs caused directly or indirectly by bad memory management in insecure languages are pretty well documented if you'd be bothered to look.
That one can _badly_ do garbage collection in C/C++ isn't news and not surprising. One can do it in assembly for heavens sake!
Much C++ code uses reference counted garbage collection, some uses this combined with regions - allocating things on the stack and just letting them drop are region based memory management.
But reference counting _does_ have GC overheads, often considerably more than scanning type "traditional" GC algorithm. That includes pauses for GC.
GCs also screw with your timing, which is why any embedded environment does not use GC functionality even in GC languages - you create long lived objects on the stack with tuned array sizes where necessary to use instead of allocating on the heap. (Even in C/C++ that can be common in embedded environments; heap is avoided at all costs.)
And as soon as you work for Facebook or Amazon, you will never have any use for 'bit bashing' again.
False. Having interviewed with Amazon they very well do things that may bit bash and are concerned with that level of performance even from their Java devs. (I typically don't do Java, which was the show stopper from my end.)
Was your encryption device strapped to the side of a transmitter?
Don't know. We just did software that ran on top Linux or Windows; so we referenced FIPS OpenSSL or Windows WinCrypt API and were approved since we didn't do any actual encryption ourselves.
Now if you did the actual encryption yourself such that you essentially re-implemented OpenSSL or WinCrypt then I would certainly expect it to take a long time. But I'm pretty sure there are enough crypto devices within DoD there are numerous certified devices that could be utilized.
Just so that people stop sending me links to conspiracy theories about it. The problem with all of the conspiracy theories about MH370 that I have encountered so far is:
1. There is no motive for a conspiracy.
2. There is no evidence of a conspiracy.
Other than that all of the conspiracy theories are "interesting".
Well, you don't really know any motive until you can identify who did it.
While I'm not one for conspiracies, I think the MH370 incident is a prime example of what could be the next "9/11" - hi-jack an airline in remote region, kill everyone on board, transport it somewhere where you want to make a statement, then load with explosives and make your statement; only after which you announce you were behind it. Such a task could take years to carry out, so it's not going to be evident for potentially a long time who did it or why; and you'll only discover it by looking at the wreckage (if recoverable) after the final terrorist act occurs.
The only thing that gives the above any credence is the fact that they haven't found it yet.
Could there be other explainations? Certainly; but the likelihood is low given the (public) information and critical timing (e.g the plane "disapparing" just after a hand-off between two regions in a remote area that had little if any radar coverage, among government regions that are fighting each other more than helping out so as to point the finger elsewhere, etc).
Do I hope the find it? Certainly; but the current information (or rather lack thereof) only makes my hypothesis (sadly) stronger.
I've had 2 different MS Natural keyboards, both of them had a key that didn't work. Seriously. 2 different ones, purchased way apart from each other, from different vendors.
I searched online and people talked about "fixing" failed keys on these keyboards with a cut piece of a drinking straw. Obviously it was a bad enough flaw to where many other people had issues with them.
Fuck M$ hardware, just like Fuck M$ software.
Funny. I've used them exclusively since 1997. My original I bought in 1997 didn't die until 2004 or so when after the 3rd or 4th spill something shorted the circuitry. I was never a fan of the 1.1 version due to it being a little tighter together on the keys and the odd placement of the arrow keys (I had one at least temporarily...). I've since gotten two more at home - one wireless (which sucks for batteries as its not bluetooth) and another wired. I've been using them at work since 2008 (both present and past employers).
Now, the newer black ones do seem to have the white lettering wear off faster than the old white ones did; but I don't see that being an issue since I'm a touch typist - in part due to using the keyboards themselves.
Also, a quick google search did not show what you describe to be very common, and as with anything there's certain to be an occassional bad one in the mix.
The engineers who designed the Predator were not idiots, adopting Security through Obscurity. The feeds were not encrypted for at least two reasons: The Predator is supposed to be able to go at a moment's notice, and having to wait around to be keyed for the mission at hand as required by NSA, defeats the purpose. I've heard the expression "80% of my intelligence needs that I can have NOW, can share it with coalitions, and don't need a security officer and a safe to transport it around the battlefield is a dream come true..." In any case, the imagery doesn't tell anyone anything they don't already know, and has a quick "half-life" meaning it rapidly becomes irrelevant. It's just not worth encrypting.
Not saying that we don't do stupid things, just that not encrypting the feed was not one of them.
There is one thing it tell them - where the drone is - GPS+Altitude inclusive. Given the speed of the drones, that could be enough time to "get out of the way" (for armed drones) or hide stuff (to limit intelligence gathering). If they're lucky, and the drone is flying low enough, then that could be enough for them to shoot it down too.
"They felt that since the Predator video feeds were being transmitted on frequencies that were not publicly known, no access control, encryption or other security mechanisms would be needed. " -- I am sure it wasn't that simple. As soon as you say 'encryption' in the defense world you open a can of worms that can set your project back as much as 2 years. These aren't technical set backs, but rather paperwork and process set backs. They were probably told by their government program manager to not put 'encryption' in their response because they probably didn't want to deal with the additional process burden. You can't do anything in the defense contracting world such as adding a feature like encryption without the government's program manager signing off on it and often find yourself constrained by law from implementing the best possible solution.
The big issue regarding encryption is showing that the encryption functionality is FIPS-140 compliant; aside from that it's not really that difficult. Been there - we just said "we use product X to do the encryption" and we were done. If they could use a FIPS certified encryption library then they're all done - just enable it.
That said, given how certain other things work by using security through obscurity I wouldn't be surprised if they really did do that intentionally for reasons other than encryption.
They must have forgotten to take the bi-partison approved "rose color glasses" pill sold in "donkey" (blue) or "elephant" (red) variants. A purple pill for independents is also available.
Available from your health care provider today courtesy of ObamaCare.
A good common sense opinion from Mr. Wheeler and the FCC. So where's that common sense when it comes to net neutrality?
Wrong companies. You'll notice his pet companies Verizon, Comcast and Time Werner aren't helped by this. It's more complicated than pushing things in favor of big business, it's pushing things in favor of the biggest businesses. Sprint and T-mobile combined have less market share than either Verizon or AT&T individually, and I'm sure Verizon and AT&T want to keep it that way.
So, what will happen is a smaller company or combined small companies will buy the spectrum, and then get bought out by Verizon or AT&T.
Except that the stance that prohibits Sprint and T-Mobile from merging would also end-up prohibiting AT&T and Verizon from aquiring many other competitors.
The problem is that in the real world, huge companies can afford to acquire resources just to keep them from being available to competitors.
This actually happened in downtown Miami. During the early-2000s building boom, downtown Miami got several dozen new skyscrapers -- nearly ALL of which have huge pedestal garages with more parking spaces per square foot of commercial floorspace than some suburban office parks & shopping malls. Yet, if you go to downtown Miami on Saturday night, you're going to spend $20 to park in a moderately full garage surrounded by 5 or 6 empty garages that are officially closed for the weekend. Basically, a group of investors went in and bought up most of the surplus capacity for a pittance during the real estate crash with the express intent of taking most of the spaces off the market and allowing them to get away with charging a lot more for the few remaining spaces. Making matters worse, they even bulk-leased parking spaces from the goddamn CITY OF MIAMI, so even THOSE garages are "full" despite being mostly unoccupied by actual cars. The problem is the worst in Brickell (the area south of downtown Miami proper), because the City of Miami SOLD OFF the vacant lots it used to own and operate for parking on the assumption that all the new buildings had abundant private parking garage space anyway. In effect, a group of investors figured out how to hack the system, and totally pwn3d Miami residents in the process.
Works only until some unbribed official decides to make a RICO case out of it for racketeering; or even an anti-trust case for abusing the monopoly powers they have.
...typically full of arrogant people that are extremely focused on academics and doesn't have much if anything that applies to the real-world programmer.
Serious, I belong to IEEE which is a lot more practical. I don't belong to IEEE's Computer Society since it is very theoretical and not very useful. However, IEEE has many many benefits which are well worth the money.
I'd love to see a proper Computer Software Engineering organization devoted to the practical every-day programmer, and not academic theories.
The firewall sees another TCP connection. Everything looks great.
Lets say... a malware binary is downloaded with a dynamic load balancing across 2 tcp streams. Everything looks fine to your NG firewall, no malware detected.
This is definitely a game changer. The old security inspections are successfully evaded. Add on the ability to open a stream in the opposite direction.
The countermeasure is to enable deep protocol inspection (and HTTPS inspection!) and your NG firewall and have it try and understand MPTCP and correlate multiple streams. Try doing that with an active/active firewall cluster.
As another poster commented, we are definitely moving towards per-host IPS, hopefully in the hypervisor for your VMs.
So I worked on a file transfer product some years ago that basically did the same thing - it transferred using multiple TCP connections to reliably overcome certain network conditions (e.g satcom, network drops, etc). Our clients built IDS/IPS rules to handle the whole thing without any issue, and we didn't have to provide documentation of our network protocol for them to do it. Of course, it was helped by the fact that we used the same port to open all the connections.
So ultimately I don't see this as an issue from that aspect.
What will be an issue is when people can get two disparate networks to do the transfer - e.g a previous posters WiFi+4G (cell data network) example. Even then, it won't likely be an issue.
Seriously, back in 2004 I was trying to get a phone WITHOUT a camera because I couldn't have a camera at work. Despite being in an area (Washing D.C Metro Area) where this is a requirement for the better part of the majority of the population it was extremely difficult. I never upgraded the phone as a result.
I stopped in several stores and asked about it and was politely informed that the stores do not get to decide what phones they carry. Their corporate parents do - the execs at AT&T, Verizon, Sprint, etc; which means they make it based on what profit margins and policies they want to set forth. So if they want everyone to have a camera, then so be it.
Now adays those same big carriers are trying to push everyone towards the profit centers around the smart phones. AT&T policy is that if you have a keyboard you need an SMS/TXT plan; but if you have a touch screen then you need a (far more costly) data plan.
Slashdot Mirror
An interesting gun... Here in People-Who-Are-Actually-Professional-Network-EngineersVille, we'd simply accept that our current cheapest-available-transit-provider has shitty connectivity (really? Cogent? really really? Well done, Netflix. Not pinching any pennies, at all) and get a provider that didn't offer bargain bin connectivity and shitty routes to just about everyone. But hey. It's entirely the receiving network's fault.
FTR, NetFlix uses a number of different transit providers, not just Cogent.
Errm, it's a dual-core chip, and there's no third core for running the optimizations. They run on the same CPU cores that everything else does.
It's a dual core chip with one core dedicated to doing the optimizations and the other for running the code.
Windows RT was simply Microsoft's hedge in the x86/ARM battle. If ARM had utterly dominated Intel in the low-power processor market and the world moved away from Wintel, then RT would've been Microsoft's safety net. If that had happened and Microsoft hadn't made RT, all the armchair quarterbacks currently criticizing Microsoft for making RT would've been criticizing them for not making RT and missing the ARM boat. RT didn't need to succeed. It just needed to be there.
ARM is utterly dominating the low-power market.
Fact is, Windows can't compete in that market as its designed for a high-power environment.
From a security point-of-view, you need to be able to validate that a pointer is valid beyond whether or not it is NULL. You need to know that your application issued the pointer and that the data it points to is valid and within your application space. And this needs to be in real applications, not debug mode.
What do you mean? Under what circumstances is this kind of pointer validation required? It sounds like this is an attempt to detect other bugs, after-the-fact (reading uninitialized or over-written memory, for example).
In implementing a true Secure Programming methodology you don't trust anything. You don't trust that a pointer coming from another part of the program is valid; you need to be able to verify it. Some libc environments give you the ability to validate a heap pointer, but you still cannot validate a stack pointer. So there's a limited ability to really do much with this kind of security testing, which has to be done on-the-fly in the real program (not only debug mode). This is security testing; not program validation, though it can provide that too. It is preventative functionality to keep back actors from being able to break into your program, and if they do extremely limit their ability to go farther by the vary nature of the program itself.
Also, I am still not sure what you mean by "remove unused memory". Remember that the unit of work, in a managed heap, is either the number of live objects or the number of live bytes. "Unused" (or dead or fragmented) memory is not a cost factor.
Unused memory is certainly a cost factor when it comes to systems of programs being able to use the memory on the system. memmapp'd memory is not unlimited. Memory is not unlimited. You only have so much and you have to be a good citizen with its use. Even if you have a high volume of memory allocations, it can be done; and the standard (non-GC) system will certainly keep your memory for only the life-time you need it, no longer; even re-using similarly sized chunks (in some implementations) to minimize calls to the kernel allocator. The kernel can and does notify the libc environment when memory is low, and the libc environment will return free'd memory to the system when that happens. Been there; done that.
more so the "promise of JIT" which gives them a pretty serious win.
So why is Android moving away from using JIT? Why does Android ART binary compile all applications at install?
JIT is a problem. GC is a problem. Combined they make problems worse. Yes, there are situations that they are good for; but not for embedded systems, or phones, or tablets, or most user facing applications. Like many things they are very good for certain kinds of tasks, and that's it.
Not quite. GC are always built on top of malloc+free, not side-stepping them.
This is incorrect. High performance GC implementations are typically built on top of the platform's virtual memory routines, directly (on POSIX, think mmap or shmem). This avoids the waste of "managing the memory manager" and also allows the GC fine-grained control over the heap. On some platforms, this also provides selective disclaim capabilities meaning that the GC actually will give pages back to the OS when contracting the heap, whereas free() wouldn't.
And those versions are few and far between, exception not the rule.
But that doesn't mean that your program should just crash because it failed; the program should degrade gracefully in those situations.
Agreed. To avoid this problem, the better pieces of software I have seen did no dynamic allocation once they reached a steady running state. This meant the failure states were easier to wrangle since you could only fail to allocate during bootstrap or when substantially changing running mode.
Define a steady-state. Not every application has one. This is why real-time stuff doesn't do that - they allocate memory/blocks on the stack at the application (global) level. If you can load the application then everything the application will ever need is allocated. If you cannot load the application, then that's it.
That works for the real-time work; but if you want to do dynamic checking then you need to be able to validate pointers.
Of course, I'd go further and say the standard libs and operating systems should provide a method to validate pointers, but that's more a security concern and the hard part for that is figuring out if the object is valid and on the stack versus valid and on the heap.
The general approach is that a valid pointer is anything non-null. If you need to further introspect the memory mapping or sniff the heap to determine validity, you seriously need to re-think an algorithm. Debugging tools, of course, are exempt from this rule since they are often running on a known-bad address space.
True, that is the general approach. My point is that that is wrong. From a security point-of-view, you need to be able to validate that a pointer is valid beyond whether or not it is NULL. You need to know that your application issued the pointer and that the data it points to is valid and within your application space. And this needs to be in real applications, not debug mode.
Now in doing the reference counter in the GC, then yes - it becomes a lot more expensive to keep it and maintain it because it has no knowledge of the actual use of the pointer.
GCs do not store reference counts since they are completely different from this kind of tracking. They determine validity by reachability at GC time.
Which is a major draw back for using a GC as it now has to crawl everything periodicially.
GC allocation will never be faster than non-GC allocation because it relies on non-GC allocation underneath. Anything the underlying libraries and kernel do, it does as well.
This is incorrect. High performance GCs manage their heap directly and can offer allocation routines based on this reality. The main reason why they can be faster is that they have the ability to move live objects at a later time so fragmentation doesn't need to be proactively avoided in allocation, which is normally what causes pain for malloc+free.
So now you're adding indirect pointers for normal pointer usage...which again now means two calls for every pointer and now you've slowed down the whole application. Smart pointers do the same thing in a sense; as does the PIMPL design pattern - it can still be quite fast, but is still (provably) slower than directly using the poin
By that definition of "wasted", even a basic malloc+free system will waste massive amounts of memory since free rarely actually removes the heap space it is using from the process. Most of the time, someone using a GC configures it to avoid returning this memory, as well, since the environment is typically tuned against the maximum heap size, so giving it back would just mean it has to be requested again, later.
Not quite. GC are always built on top of malloc+free, not side-stepping them.
Further, even if malloc/free didn't return it to the OS, it will still try to alloc from the "free" memory it has before getting more memory from the OS in many implementations. GCs even delay the ability to do that.
Of course, if this is a concern, we are getting more into the debate of whether dynamic allocation should be used, at all (which is something I wish more people thought about - I also like you that you mentioned it can fail since so many people forget that).
Yes. All too often people just code assuming that an allocated memory buffer returns a valid value; usually you'll get something like "it never fails" or "if it fails then there are other issues". But that doesn't mean that your program should just crash because it failed; the program should degrade gracefully in those situations.
Of course, I'd go further and say the standard libs and operating systems should provide a method to validate pointers, but that's more a security concern and the hard part for that is figuring out if the object is valid and on the stack versus valid and on the heap.
I am not sure what you mean by "dynamic allocation time" in a GC and reference counting is NOT "sufficiently quick".
"dynamic allocation" means anything done with malloc/new versus a stack-based allocation, which could be on the local, sem-local (e.g class object), or global stack frames.
Now, regarding reference counting - that all depends on the implementation. Some systems (e.g Ada) do reference counting on the memory allocated - that is, the first couple bytes store a reference count and it's just built-in. Other, like Qt, do it by keeping references (e.g. is X in the child list). And then there is the shared pointer methods (e.g QSharedPointer, std::shared_ptr) which also keep a very quick counter.
Now in doing the reference counter in the GC, then yes - it becomes a lot more expensive to keep it and maintain it because it has no knowledge of the actual use of the pointer.
Actual allocation cost, when using a GC-managed heap is generally incredibly cheap (because the allocator doesn't have to do heap balancing, etc). Reference counting can involve walking massive parts of the heap, and writing to it (sometimes many times). GC is generally very fast but, again, depends entirely on the number of live objects so it becomes more expensive the larger the live set becomes.
GC allocation will never be faster than non-GC allocation because it relies on non-GC allocation underneath. Anything the underlying libraries and kernel do, it does as well.
What do you mean by "as the system needs better and better performance GCs become less and less useful"? A good GC will actually increase the performance of the application as it can improve cache density of the application's live set (not to mention memory-processor affinity).
We do seem to be having 2 conversations here: 1) How does GC compare to other dynamic allocation approaches 2) How does any dynamic allocation approach fare against static or stack-oriented allocation approaches. In this case, I think we both agree that avoiding the issue altogether is preferable, where possible.
GCs will never improve cache performance as it is entirely unrelated and should not be randomly selecting objects. If anything it will decrease cache performance because it will be randomly hitting nodes (during its checks) that the application wants to keep dormant for a time.
What do you mean by "wasted", in this case?
Quoting parent; but typically it would be "memory that is not in use by the application but assigned to the application because it has not yet been returned to the OS". So in a manner, GCs lead to intentional, but temporary, memory leaks. That could be forgiven if there were not other issues regarding using GCs.
If you are generally referring to unpredictable pause times, then that is a real concern of GCs (and some general cases of reference counting and some cases of dynamic allocation). Of course, in the case of the GC, the pause time is a function of the live objects (and, in some cases, their size or topology)
Whether or not you use a GC you still have the dynamic allocation time; to that is a moot point; in some cases you still have the reference counting, and again that is sufficiently quick enough that that too is a moot point.
However, GCs really hurt performance and testability with the unpredictability of the garabage collection time. You can't predict, or control, when a GC decides to check the memory or return it. It could hit at your most critical performance moment, or it may hit at your least critical performance moment, or any time in-between. That is just the nature of the GC.
Yes, certain GC algorithms perform better in some scenarios in others. Some GC implementations provide the ability to control some of it; but they're the exception not the rule.
About the only method of "GC" (and I put that in quotes because it's not really a GC) is how Qt does it with parent objects cleaning up any child object that remains when the parent object is destroyed. It has nearly all the effects of a GC, but all the control and performance of a non-GC system. Yes you can still end up with memory leaks should you forget to parent an object; but even that is generally easy to track down and fix (because something else will usually break as well); and there are very few objects that you have to actually track because they cannot actually have a parent.
Of course, there are GCs which offer predictable pauses but they are typically lower performance. They only matter in real-time environments so they are not often used.
No. They matter in nearly every environment. And I'm pretty certain this is one of main performance issues for the DalvikVM in Android for numerous applications.
If in true Real-Time or even Near-Real-Time application you NEVER allocate memory regardless of whether you use a GC - and especially if you use a GC - because you don't want to have to deal with the possibility the memory may not get allocated. So when you're talking about using a GC, you're not talking about high performance applications. You're talking about everything else.
For everything else your performance comes down to user-perception in most cases (outside of games) and GC collection will hurt that. In games, it can destroy it. User's don't like to have their applications unpredictably stop for a while; they deal with it but they don't like it. And if given a choice between a program that will pause for a while and one that won't, they'll go to the one that won't.
Now GCs can be good for certain environments - where user's don't interact with the system, it's not anywhere near real-time, and the application may have time periods where it simply doesn't matter - e.g a server handling connections where it can clean up after the connection closes and before it handles another; but even then, as the system needs better and better performance GCs become less and less useful.
They might habe it in the interviews, that does not mean coders actually code that way :) And certainly not every coder, perhaps 1%?
From the methodology of the interview, I can certainly say they are doing those kinds of things in reality.
The problem for Olivier Bloch, Senior Technical Evangelist and Microsoft Open Technologies, Inc. is that they are isolated. The parent company only tells them enough and shows them around enough so that he and others like him can say what they do with a straight face. Meanwhile, the left hand knoweth not what the right hand doth.
I witness this a couple years ago at POSSCON. We had someone that had a similar position at Microsoft give a Keynote. He talked about all the things Microsoft did and everything they showed him and how the teams were open to Open Source. But while he was there saying those things, Microsoft corporate were actively denouncing open source - Balmer and all his lieutenants and even their lieutenants. The speaker truly believed what he was saying - and what he saw. They just sufficiently isolated him such that he was able to believe it and therefore say it. Most of the audience was aware of what Microsoft corporate was saying that directly contracted the speaker they had provided. Sadly, they heckled and booed him instead of showing some professional courtesy, especially since Microsoft was a big sponsor that year.
All that said, it's not that Microsoft can't turn around and become Open Source friendly. It's just that there is such a beaurocratic momentum against it within the company that it will be years before any kind of turn around can effectively happen, unless the have a mass firing of the upper level management, which is not likely.
...and yet, all of Microsoft's flagship products, AFAIK, are the polar opposite of open source. If Microsoft truly thought anything of open source, this should not be the case.
Well, WiX is kind of a flagship product - it's embedded in Visual Studios, utilized by most all of their projects, and Open Source (originally GPLv2, I forget what the current license is).
There is no problem with "wasted" memory backing dead objects between collections since the heap is typically of fixed size between collections (especially in situations like Java where the VM parameters are fixed at start-up time). Reference counting designs not only fail to collect non-trivial object graphs but also do not free memory immediately (although it is typically deterministic with response to call-return behaviour). Now, returning from a function could involve walking every object in the heap, perhaps many times, to update reference counts. GC only incurs cost at GC time, and that is proportional to the number of live objects, not dead objects (which typically dominate).
This is actually a big issue for GCs. It's not that the memory is "wasted" but that the collection of the "wasted" memory leads to unpredictable performance penalties. It's best to avoid GCs entirely for this reason alone.
But it isn't unsupported! The amount of bugs caused directly or indirectly by bad memory management in insecure languages are pretty well documented if you'd be bothered to look. That one can _badly_ do garbage collection in C/C++ isn't news and not surprising. One can do it in assembly for heavens sake!
Much C++ code uses reference counted garbage collection, some uses this combined with regions - allocating things on the stack and just letting them drop are region based memory management.
But reference counting _does_ have GC overheads, often considerably more than scanning type "traditional" GC algorithm. That includes pauses for GC.
GCs also screw with your timing, which is why any embedded environment does not use GC functionality even in GC languages - you create long lived objects on the stack with tuned array sizes where necessary to use instead of allocating on the heap. (Even in C/C++ that can be common in embedded environments; heap is avoided at all costs.)
Personally, I've never had use for GCs.
And as soon as you work for Facebook or Amazon, you will never have any use for 'bit bashing' again.
False. Having interviewed with Amazon they very well do things that may bit bash and are concerned with that level of performance even from their Java devs. (I typically don't do Java, which was the show stopper from my end.)
Was your encryption device strapped to the side of a transmitter?
Don't know. We just did software that ran on top Linux or Windows; so we referenced FIPS OpenSSL or Windows WinCrypt API and were approved since we didn't do any actual encryption ourselves.
Now if you did the actual encryption yourself such that you essentially re-implemented OpenSSL or WinCrypt then I would certainly expect it to take a long time. But I'm pretty sure there are enough crypto devices within DoD there are numerous certified devices that could be utilized.
Just so that people stop sending me links to conspiracy theories about it. The problem with all of the conspiracy theories about MH370 that I have encountered so far is:
1. There is no motive for a conspiracy. 2. There is no evidence of a conspiracy.
Other than that all of the conspiracy theories are "interesting".
Well, you don't really know any motive until you can identify who did it.
While I'm not one for conspiracies, I think the MH370 incident is a prime example of what could be the next "9/11" - hi-jack an airline in remote region, kill everyone on board, transport it somewhere where you want to make a statement, then load with explosives and make your statement; only after which you announce you were behind it. Such a task could take years to carry out, so it's not going to be evident for potentially a long time who did it or why; and you'll only discover it by looking at the wreckage (if recoverable) after the final terrorist act occurs.
The only thing that gives the above any credence is the fact that they haven't found it yet.
Could there be other explainations? Certainly; but the likelihood is low given the (public) information and critical timing (e.g the plane "disapparing" just after a hand-off between two regions in a remote area that had little if any radar coverage, among government regions that are fighting each other more than helping out so as to point the finger elsewhere, etc).
Do I hope the find it? Certainly; but the current information (or rather lack thereof) only makes my hypothesis (sadly) stronger.
I've had 2 different MS Natural keyboards, both of them had a key that didn't work. Seriously. 2 different ones, purchased way apart from each other, from different vendors.
I searched online and people talked about "fixing" failed keys on these keyboards with a cut piece of a drinking straw. Obviously it was a bad enough flaw to where many other people had issues with them.
Fuck M$ hardware, just like Fuck M$ software.
Funny. I've used them exclusively since 1997. My original I bought in 1997 didn't die until 2004 or so when after the 3rd or 4th spill something shorted the circuitry. I was never a fan of the 1.1 version due to it being a little tighter together on the keys and the odd placement of the arrow keys (I had one at least temporarily...). I've since gotten two more at home - one wireless (which sucks for batteries as its not bluetooth) and another wired. I've been using them at work since 2008 (both present and past employers).
Now, the newer black ones do seem to have the white lettering wear off faster than the old white ones did; but I don't see that being an issue since I'm a touch typist - in part due to using the keyboards themselves.
Also, a quick google search did not show what you describe to be very common, and as with anything there's certain to be an occassional bad one in the mix.
The engineers who designed the Predator were not idiots, adopting Security through Obscurity. The feeds were not encrypted for at least two reasons: The Predator is supposed to be able to go at a moment's notice, and having to wait around to be keyed for the mission at hand as required by NSA, defeats the purpose. I've heard the expression "80% of my intelligence needs that I can have NOW, can share it with coalitions, and don't need a security officer and a safe to transport it around the battlefield is a dream come true..." In any case, the imagery doesn't tell anyone anything they don't already know, and has a quick "half-life" meaning it rapidly becomes irrelevant. It's just not worth encrypting. Not saying that we don't do stupid things, just that not encrypting the feed was not one of them.
There is one thing it tell them - where the drone is - GPS+Altitude inclusive. Given the speed of the drones, that could be enough time to "get out of the way" (for armed drones) or hide stuff (to limit intelligence gathering). If they're lucky, and the drone is flying low enough, then that could be enough for them to shoot it down too.
"They felt that since the Predator video feeds were being transmitted on frequencies that were not publicly known, no access control, encryption or other security mechanisms would be needed. " -- I am sure it wasn't that simple. As soon as you say 'encryption' in the defense world you open a can of worms that can set your project back as much as 2 years. These aren't technical set backs, but rather paperwork and process set backs. They were probably told by their government program manager to not put 'encryption' in their response because they probably didn't want to deal with the additional process burden. You can't do anything in the defense contracting world such as adding a feature like encryption without the government's program manager signing off on it and often find yourself constrained by law from implementing the best possible solution.
The big issue regarding encryption is showing that the encryption functionality is FIPS-140 compliant; aside from that it's not really that difficult. Been there - we just said "we use product X to do the encryption" and we were done. If they could use a FIPS certified encryption library then they're all done - just enable it.
That said, given how certain other things work by using security through obscurity I wouldn't be surprised if they really did do that intentionally for reasons other than encryption.
I hope he doesn't do away with their best product, the Microsoft mouse.
It's the one product they have that is Compatible.
Or the Microsoft Natural Keyboards...
Somebody is off their meds again.
They must have forgotten to take the bi-partison approved "rose color glasses" pill sold in "donkey" (blue) or "elephant" (red) variants. A purple pill for independents is also available.
Available from your health care provider today courtesy of ObamaCare.
A good common sense opinion from Mr. Wheeler and the FCC. So where's that common sense when it comes to net neutrality?
Wrong companies. You'll notice his pet companies Verizon, Comcast and Time Werner aren't helped by this. It's more complicated than pushing things in favor of big business, it's pushing things in favor of the biggest businesses. Sprint and T-mobile combined have less market share than either Verizon or AT&T individually, and I'm sure Verizon and AT&T want to keep it that way.
So, what will happen is a smaller company or combined small companies will buy the spectrum, and then get bought out by Verizon or AT&T.
Except that the stance that prohibits Sprint and T-Mobile from merging would also end-up prohibiting AT&T and Verizon from aquiring many other competitors.
The problem is that in the real world, huge companies can afford to acquire resources just to keep them from being available to competitors.
This actually happened in downtown Miami. During the early-2000s building boom, downtown Miami got several dozen new skyscrapers -- nearly ALL of which have huge pedestal garages with more parking spaces per square foot of commercial floorspace than some suburban office parks & shopping malls. Yet, if you go to downtown Miami on Saturday night, you're going to spend $20 to park in a moderately full garage surrounded by 5 or 6 empty garages that are officially closed for the weekend. Basically, a group of investors went in and bought up most of the surplus capacity for a pittance during the real estate crash with the express intent of taking most of the spaces off the market and allowing them to get away with charging a lot more for the few remaining spaces. Making matters worse, they even bulk-leased parking spaces from the goddamn CITY OF MIAMI, so even THOSE garages are "full" despite being mostly unoccupied by actual cars. The problem is the worst in Brickell (the area south of downtown Miami proper), because the City of Miami SOLD OFF the vacant lots it used to own and operate for parking on the assumption that all the new buildings had abundant private parking garage space anyway. In effect, a group of investors figured out how to hack the system, and totally pwn3d Miami residents in the process.
Works only until some unbribed official decides to make a RICO case out of it for racketeering; or even an anti-trust case for abusing the monopoly powers they have.
...typically full of arrogant people that are extremely focused on academics and doesn't have much if anything that applies to the real-world programmer.
Serious, I belong to IEEE which is a lot more practical. I don't belong to IEEE's Computer Society since it is very theoretical and not very useful. However, IEEE has many many benefits which are well worth the money.
I'd love to see a proper Computer Software Engineering organization devoted to the practical every-day programmer, and not academic theories.
The firewall sees another TCP connection. Everything looks great.
Lets say... a malware binary is downloaded with a dynamic load balancing across 2 tcp streams. Everything looks fine to your NG firewall, no malware detected.
This is definitely a game changer. The old security inspections are successfully evaded. Add on the ability to open a stream in the opposite direction.
The countermeasure is to enable deep protocol inspection (and HTTPS inspection!) and your NG firewall and have it try and understand MPTCP and correlate multiple streams. Try doing that with an active/active firewall cluster.
As another poster commented, we are definitely moving towards per-host IPS, hopefully in the hypervisor for your VMs.
So I worked on a file transfer product some years ago that basically did the same thing - it transferred using multiple TCP connections to reliably overcome certain network conditions (e.g satcom, network drops, etc). Our clients built IDS/IPS rules to handle the whole thing without any issue, and we didn't have to provide documentation of our network protocol for them to do it. Of course, it was helped by the fact that we used the same port to open all the connections.
So ultimately I don't see this as an issue from that aspect.
What will be an issue is when people can get two disparate networks to do the transfer - e.g a previous posters WiFi+4G (cell data network) example. Even then, it won't likely be an issue.
Seriously, back in 2004 I was trying to get a phone WITHOUT a camera because I couldn't have a camera at work. Despite being in an area (Washing D.C Metro Area) where this is a requirement for the better part of the majority of the population it was extremely difficult. I never upgraded the phone as a result.
I stopped in several stores and asked about it and was politely informed that the stores do not get to decide what phones they carry. Their corporate parents do - the execs at AT&T, Verizon, Sprint, etc; which means they make it based on what profit margins and policies they want to set forth. So if they want everyone to have a camera, then so be it.
Now adays those same big carriers are trying to push everyone towards the profit centers around the smart phones. AT&T policy is that if you have a keyboard you need an SMS/TXT plan; but if you have a touch screen then you need a (far more costly) data plan.