Dozens of applicants for professorships? I've applied for teaching/generalist English professorships in the last year for which there have been 500-800 applicants. No kidding. Those are extreme cases, but most searches, even in specialist areas, are netting at least 150 applications.
The GP said that the ratio of Ph.D candidates to positions was dozens to one, not that the ratio of applications to positions was dozens to one. The two numbers are not the same, unless each candidate applies to exactly one position on average.
In reality, each candidate applies to dozens of academic positions on average. (Some apply to hundreds, some apply to none; the average is probably on the order of a few dozen.) A few dozen people per position, multiplied by a few dozen applications per person, is entirely consistent with the range of 150-800 applicants per position.
You say that you are (applying to be) an English professor. I am a math professor. I have no sympathy for mathematicians who can't write. Writing is a big part of my research, and every individual on this planet is better off having rudimentary skill in communication. One can even reasonably argue that English in particular is the most important language worldwide. But, by the same token, I also consider foundational math, like English, to be a basic skill that every individual needs. Those who lack mathematics skills are bound to make the same kind of mistake that you displayed.
The solution to this absurdity is to build a time machine, go back to the 80s and define three protocols "http:", "httpe:" (encrypted) and "httpv:" (identity validated) so users don't grow up thinking https: is secure.
Well said. But why do we need a time machine? https is broken and we need to fix it.
Your whole line of posts is based on some sort of premise that we must maintain compatibility with the status quo. My whole point is that the status quo is so irretrievably broken that we must fix it, even if we need drastic steps such as eliminating compatibility with prior notions of "URL" or "https".
Firefox's hysteria against self-signed https goes in the opposite direction. It reinforces the status quo and makes https (or httpe or whatever you would want to call it in an ideal world) even more unusable.
The problem can be fixed. SSH uses no certificates whatsoever, and yet people successfully trust SSH encryption for root-level access. SSH is a far more robust and secure protocol than SSL ever will be.
There's no reason why browsers have to display "https://" or even "http://".
Except, that's a significant part of the address. "http://somesite.org" and "https://somesite.org" could, potentially, point to different content (certainly different vhosts).
Web servers already display different content to users based on their geographical location or their login cookies or any number of state variables, and these content changes are not reflected in the URL. Your point means nothing.
Sites using https generally do so because they want to exchange sensitive data, and the use of a self-signed certificate might indicate that a MiM attack is in progress, or (possibly more likely) that the site is being run by a cowboy outfit who can't be arsed to get proper certificates. So, a self-signed https connection is always slightly fishy (there are plenty of innocent explanations, but identifying those requires human judgement + technical understanding).
This is a tautology. Since today's browsers are so alarmist about self-signed certificates, the use of self-signed certificates is automatically fishy. If you remove the alarmism then the amount of legitimate usage of self-signed certificates would increase dramatically.
self-signed https = someone could be mounting a man-in-the-middle attack or you may have been spoofed/phished to the wrong website.
The same holds for regular http. Someone could be mounting a man-in-the-middle attack with regular http.
Meanwhile, there is one big difference between http and self-signed https that you omitted. With regular http (and only regular http), large-scale attacks like police surveillance and content filtering become possible. https (even self-signed) prevents large-scale passive attacks.
I'd suggest (3) is by far the best place at which to start nagging - most users will rarely encounter this situation (only sites with very small user bases, like home servers or in-development sites have a real excuse for not getting a cert) so you're not going to swamp typical users with bogus warnings. For the typical user, this does mean that something out-of-the-ordinary is happening.
Again, the fact that self-signed certificates are out-of-the-ordinary is a tautology that you helped to set up by insisting that they be treated as out-of-the-ordinary.
And remember at the end of the day, all browsers like firefox actually do is warn you, encourage you to view the certificate and decide whether you want to trust it temporarily or permanently
NO! That's not what firefox does. If firefox did in fact do what you claimed it did, then I would be happy.
In practice, firefox effectively blocks self-signed certificates entirely. It takes five (count them, five) mouse clicks to connect to a self-signed https site in firefox, compared with one mouse click in IE. A regular user is scared away after even one mouse click, much less five. Thus in practice firefox ends up blocking self-signed certificates entirely.
Regular http has no warnings whatsoever, even though every attack against self-signed https is also possible against http, and some attacks against http are not possible against self-signed https. This situation is absurd beyond belief.
Yes, a self-signed https connection can be more dangerous than a plain http one if you see the "https" or the "golden padlock" and assume you have a secure connection.
The obvious solution is: don't display "https" or the "golden padlock."
There's no reason why browsers have to display "https://" or even "http://". The average non-technical user doesn't care about the protocol; they just care about the "golden padlock." On the other hand, the average technical user already knows what's going on.
Nobody here is arguing that self-signed https connections deserve a "golden padlock." That's your own straw man.
The proposal is that we should treat self-signed https connections the same as unencrypted http connections. The same. Not worse. Not better. The same.
I have yet to see anybody articulate an even remotely coherent argument against this proposal.
2 - Defragging: similarly, if you're moving data around in dead space without safely duplicating it or having a filename pointing to the blocks in use at any given time, you're not being careful. Also, which defraggers have random 3-minute gaps in operation that would even allow GC to kick in?
I think it is time to start bringing the discussion to a close, as it appears that we do share at least some common ground.
I will comment only on this one question. Your implication that the 3-minute rule somehow makes the GC "safe" is missing my point entirely. Yes, in practice there are checks and balances such as you describe, that make the GC unlikely to screw up. But, in my view, "unlikely" is not good enough. I want, and need, perfect (logical) block storage and retrieval. This should and must be the design goal. Of course, this goal is impossible to achieve in practice. For example, if the firmware (such as older, pre-SSD firmware) is designed with the goal of providing logical block storage, but fails in this task because of some honest bug, then I can understand that. At least, in this case, the programming code was written unambiguously with the correct goal (and no other directly conflicting goals) in mind, even if this goal was not achieved in practice due to an unintentional bug.
However, when a manufacturer deliberately designs firmware with the goal of deleting logical sectors, no matter how well-intentioned or well-implemented, this design goal (by definition) must come into conflict with the original, core goal of reliable (logical) data retrieval. I do not care what happens in the underlying physical layer, but I do care very greatly about data accuracy at the logical layer. The existence of certain checks and balances to prevent data loss is better than no checks and balances, but it is not better than the REAL alternative, namely, firmware that stores and retrieves logical blocks correctly, and is designed for this and only this purpose, without any other directly contradictory design goals.
No one, not even rocket scientists, has ever succeeded in writing bug-free software. But one should make an effort to minimize the number of opportunities for data loss bugs to arise. Firmware-based logical-sector garbage collection fundamentally and irreconcilably contradicts every reliability design principle known to man. That is why I consider the idea to be so abhorrent.
This GC only works with NTFS filesystems. If you are operating an SSD device using an NTFS file tree to store data, then you (as a programmer) are not using the drive as a block device as you suggest; you're accessing it (or you should be accessing it) via the abstract file tree.
The ATA standard (parallel or serial) does mandate that the drive appear as a logical block device. A drive with an ATA connector must honor that requirement or (IMO) it violates the standard. USB Mass Storage is another example where the standard mandates logical block storage -- what happens if I put one of these defective drives into a USB enclosure?
The only difference is that people seem to take block-level access to the disk for granted;
For many non-edge-case applications, like whole-disk encryption (remember, I'm a cryptographer), block-level access is exactly what you need, and anything less is unacceptable.
I understand that the firmware is supposed to (obviously) ignore non-NTFS volumes and fall back to block storage semantics. But the mere presence of active garbage collection is unwanted to me. It adds another possible failure opportunity.
The problem, from my perspective, is that your arguments hinge on the idea that marking data as 'deleted, and the filesystem can now overwrite it at some random future point, perhaps instantly or never' (the HDD model) is in some way better than 'deleted, purge at first opportunity' (the SSD model). From my perspective, I'd prefer the latter; at least then I know what's happened to data after it's been marked for deletion.
The latter can (and should) be implemented with explicit TRIM support. The operating system must have control over purging.
I'm still keen to see those realistic real-world use cases. If another poster has posted them, can you provide a link?
I'm guessing you don't consider networked computers (e.g. SMB shares, FTP sites, NFS mounts) to be storage devices either then, since the remote host will merrily overwrite deleted files with other people's data however it likes there too?
This is of course a spurious comparison. SMB, FTP, and NFS are presented to the operating system as file trees. A drive with automatic garbage collection is presented to the operating system as a block device... but it does not actually implement the correct semantics for a storage block device.
What other somethings do you have in mind?
It seems that another poster has already enumerated more ways for automatic garbage collection to break.
p.s. Just thought of another example. RAM. Where you store data in memory logically, and how it is arranged physically - including zeroing of dead pages - are completely out of your control and even out of your view. Does this mean you consider RAM not to be a storage device, since you can't reliably construct a stego side-channel using dead pages of memory?
Again, RAM is presented to the OS as logical addresses, and it does faithfully restore the data that was stored in those logical addresses.
A hard drive, like RAM, presents a logical block layer to the OS which is decoupled from the underlying physical data storage. Correct data storage and retrieval is required at that logical block layer. Automatic garbage collection violates the integrity requirements of a hard drive even at the logical layer. It imposes a secondary logical layer which assumes you are using a standard filesystem in a standard way. This introduces an additional and very scary mode of failure: the possibility now exists that the firmware might actively delete certain logical block data without the knowledge of the operating system. Of course, this could happen even with older, regular firmware, but only as an accident--by default, older firmware is programmed to store everything at the logical layer, no matter what it is. Active deletion raises the stakes considerably.
Honestly, the more I think about this, the more appalled I am that any manufacturer would actually do what you describe. I will be sure to make every possible effort to avoid such drives.
Well, if you think I (and the tech support staff on various SSD manufacturers forums) are wrong, you're welcome to buy an SSD and check for yourself. It's not quite as easy as typing 'it's impossible' a bunch of times, but it's a lot more likely to be correct.
A SSD that performs automatic garbage collection by interpreting the filesystem in firmware is not, in my opinion, a storage device.
Suppose I am a filesystem developer. Suppose I want to modify NTFS in such a way that deleted segments of an NTFS disk layout become (in my modified filesystem) a repository for meaningful data. This is not as absurd a concept as it appears. In my line of work (cryptography), storing actual meaningful data in deleted segments might be something that you want to do, for example in steganography.
If the SSD goes deleting disk sectors behind my back, then it becomes impossible for me to develop said filesystem. A storage device should store what I tell it to store. If it doesn't do that, then it's not a storage device. In this sense, it is, by definition, impossible for a valid storage device to implement automatic garbage collection at the filesystem level. Any device that does such a thing, by definition, does not meet the requirements of a storage device, the primary one of which is to retain data without alteration.
Sure, those deleted sectors are safe to erase in an NTFS volume, but how do you know that my operating system is using this NTFS volume as an NTFS volume? What if I'm doing steganography or something where those deleted sectors matter?
The same way they added GC to older models of SSD drives where it didn't already exist of course, and the same way they update features on any hardware. You flash the firmware with revised code.
Remind me never to upgrade the firmware on any hard drive ever again. I do want TRIM support, but I do not want automatic garbage collection, for the reasons outlined above.
I will concede that it is possible for a write-only device to implement automatic garbage collection at the filesystem level, but I maintain that no valid storage device can do so, since to do so violates the core requirement of a storage device in a fundamental and unfixable way.
Actually, this is no longer correct. SSDs (such as the one in this study) are quite capable of examining the filesystem stored on the drive, independently, and the concept of 'dutifully' and ignorantly maintaining deleted data goes out of the window as a result.
What you're describing is impossible. It might be possible for some of the more common filesystems, such as FAT or NTFS (although, given the difficulty of supporting NTFS in Linux, I highly doubt that embedded firmware on a drive can parse the NTFS format). It is utterly impossible in the case of new filesystems. Think about it -- if a piece of hardware predates the creation of ext4, or ext5, or whatever, then how can the hardware understand the filesystem?
That's a nice straw man, but I have made a great effort to emphasize that Watson is bypassing strategically relevant aspects of human Jeopardy, and not just irrelevant things like breathing or pumping blood.
It's not a "controversial" statement. People are questioning the parsimonious rule-lawyering. Is there even a rule written anywhere about how contestants must receive the information?
Even if you dislike the nit-picking, it is indisputable that Watson had an advantage at the buzzer. Even the mainstream media picked up on this fact.
I'm not impressed that Watson can buzz a buzzer faster and more accurately than a human. I'm not impressed that Watson won the game, because all things being equal, accurate buzzing is a huge advantage in Jeopardy. I am impressed that Watson can participate in the game well enough that buzzing becomes an issue, but all the media hype (including, sadly, the hype on slashdot) is about the fact that Watson won, not about the huge improvements in AI that rightfully should be highlighted.
Except, you know, that "Underwhelming achievement" and "I for one am not at all impressed by this feat." thing.
That was pretty denigratory, wouldn't you say?
As I have made amply clear in every single one of my posts, my objection is to the title of this article: "Watson Wins Jeopardy Contest," and to all surrounding media hype along the lines of "Watson wins Jeopardy OMG"
Given Watson's inherent advantages in timing, winning a Jeopardy game (or what remains of a Jeopardy game after modifying it to allow such computer contestants) is a decidedly underwhelming achievement, and not something that I find impressive or meritorious in any way, shape, or form. The fact that Watson won the tournament is the least important achievement out of all the things that Watson accomplishes, and I find it irritating that it is the most hyped.
None of this has anything to do with Watson itself. It is the combination of Watson and calling it a Jeopardy player that I find objectionable.
In case you did not find any of the above clear before, I hope I have now made this clear.
I just think you are focusing too much on a largely irrelevant facet of this particular show.
We'll just have to agree to disagree. The entire thrust of everything I've said is that the mechanics and strategy of buzzing is not an irrelevant facet of the game of Jeopardy.
The contrast with IBM's Deep Blue is perhaps illuminating. Deep Blue, of course, did not actually physically move chess pieces. But I have no objection to saying that Deep Blue plays chess. The act of physically moving chess pieces is strategically irrelevant to the game of chess. Contrast this with the role that input/output plays in a Jeopardy game, and you'll have to agree (I hope) that timing is a bigger part of the game of Jeopardy than it is in chess.
A gray area (for me) might be the question of whether Deep Blue can play blitz chess, where the physical movement of chess pieces does become strategically relevant to the game.
- very fast information retrieval; Google et-al have similar technology, but let's not forget that it's all pretty new.
- natural language query decoding; Watson doesn't understand NL queries, but it is able to decode them into search queries.
- precise information extraction and presentation; whereas Google produces several pages of results with the answer in there (and can often highlight the right result) Watson can pull "the answer" out and present it in the required format.
Absolutely agree.
Notice what you left out? "Winning a Jeopardy contest." Yet the media hype (and even the title of this Slashdot article) is all about just winning a Jeopardy game. That's what bothers me.
In either case, I'm confident Watson would have won. Just because Ken and Brad don't know how to process electronic signals, that's not Watson's fault. Yet you are trying to penalize Watson because he has no eyes or ears.
Nothing of the sort. I do think Watson would have won using eyes and ears anyway, but it's not a question of "penalize." Do you "penalize" a computer in a 100-meter dash because it has no legs? Of course not. A computer cannot even compete in a 100-meter dash, but that's not a "penalty" -- it's a statement of fact.
Watson won a Jeopardy-like game, but that game is not identical to Jeopardy. I don't think Watson is capable of playing Jeopardy. The game of Jeopardy contains elements which inherently make sense only in the context of humans. Again, for emphasis (because everyone else here seems to completely misunderstand my point), I'm not denigrating Watson in any way. I'm just saying that the game of Jeopardy means one thing, and Watson is not playing that game.
I bet there's nothing in the Jeopardy official rules that says that you must see or hear the clue. They have had blind contestants and deaf contestants, so clearly they are open to alternate sensory methods.
I think it's pretty safe to say that no contestant prior to Watson has ever received an electronic data feed.
Watson actually did use a trivial actuator to press the physical button.
I was aware of this from the outset, and nothing that I wrote in any of my posts contradicts this, or is in any way based on a lack of Watson pushing a button.
I couldn't imagine someone turning these differences into some sort of conspiracy, but there you are.
For the n-th time, I'm saying something very simple: Watson is playing a game not equal to the game of Jeopardy. I went out of my way to clarify that Watson's achievement is meritorious in its own right. It's just that the achievement, whatever it is, is not Jeopardy. I don't know how you infer a conspiracy theory from this.
I don't know about you but I don't think it is even a given an artificial intelligence should communicate or receive input via OCR and voice recognition. It seems far more efficient to communicate via text in the first place.
That's an entirely separate issue and one where I agree with you. Of course a computer should communicate via text.
I think you're reading far more into my comments than what I actually said. All I said is that the game of Jeopardy does not communicate with contestants via direct digital electronic signals. Therefore Watson is not playing Jeopardy. (It is doing something else, equally impressive, but that something is not Jeopardy.) The fact that this simple objective statement of indisputable fact is somehow controversial is just mind-boggling.
They miss the real point: That a computer could do a level of natural language processing that was impossible before.
On the contrary, I fully agree that this is the real point.
You can label this achievement as impressive in many different valid ways. Just don't call it Jeopardy. That is the one characterization that is objectively false.
Since the "controversy" in question is some dude on Slashdot shitting on an achievement orders of magnitude bigger than anything he will ever achieve, I sincerely doubt it made their radar.
I do not post anonymously. You can easily find me in any search engine. I am confident in my achievements and see no need to defend them. Perhaps you can reveal your own identity so that readers can fairly judge which among the two of us is "some dude on Slashdot"? In any case, such personal attacks add nothing of value to the debate.
You completely avoided my question. If it's so easy, why didn't they just do it? Explaining how easy it is does not answer my question. It just makes it more and more incongruous that they did not just do it.
Actually, you are wrong about the buzz in. It gives the humans an advantage. The reason is that Watson has a mechanical buzzer that it presses. So the only advantage would come from reacting faster. However, the rules state that you can buzz in only after the host has finished reading the clue. If you buzz in earlier then you are penalized by.25sec.
Watson has a computer clock. It never buzzes in early, and it never suffers the.25sec penalty. The humans did suffer this penalty on several occasions.
It's ridiculous beyond belief to claim that the humans had the advantage in buzzing in.
OCR on a clean image (like the Jeopardy screens) is a solved problem. A physical button can be pressed with a trivial actuator, which could be almost as fast as the direct digital connection. How would adding either of those features to the machine make it any more impressive?
As I've already pointed out, if you think this feature is trivial, then why didn't IBM just implement it to avoid controversy? Make no mistake -- IBM consciously chose to give Watson an inhuman advantage.
Slashdot Mirror
Dozens of applicants for professorships? I've applied for teaching/generalist English professorships in the last year for which there have been 500-800 applicants. No kidding. Those are extreme cases, but most searches, even in specialist areas, are netting at least 150 applications.
The GP said that the ratio of Ph.D candidates to positions was dozens to one, not that the ratio of applications to positions was dozens to one. The two numbers are not the same, unless each candidate applies to exactly one position on average.
In reality, each candidate applies to dozens of academic positions on average. (Some apply to hundreds, some apply to none; the average is probably on the order of a few dozen.) A few dozen people per position, multiplied by a few dozen applications per person, is entirely consistent with the range of 150-800 applicants per position.
You say that you are (applying to be) an English professor. I am a math professor. I have no sympathy for mathematicians who can't write. Writing is a big part of my research, and every individual on this planet is better off having rudimentary skill in communication. One can even reasonably argue that English in particular is the most important language worldwide. But, by the same token, I also consider foundational math, like English, to be a basic skill that every individual needs. Those who lack mathematics skills are bound to make the same kind of mistake that you displayed.
The solution to this absurdity is to build a time machine, go back to the 80s and define three protocols "http:", "httpe:" (encrypted) and "httpv:" (identity validated) so users don't grow up thinking https: is secure.
Well said. But why do we need a time machine? https is broken and we need to fix it.
Your whole line of posts is based on some sort of premise that we must maintain compatibility with the status quo. My whole point is that the status quo is so irretrievably broken that we must fix it, even if we need drastic steps such as eliminating compatibility with prior notions of "URL" or "https".
Firefox's hysteria against self-signed https goes in the opposite direction. It reinforces the status quo and makes https (or httpe or whatever you would want to call it in an ideal world) even more unusable.
The problem can be fixed. SSH uses no certificates whatsoever, and yet people successfully trust SSH encryption for root-level access. SSH is a far more robust and secure protocol than SSL ever will be.
There's no reason why browsers have to display "https://" or even "http://".
Except, that's a significant part of the address. "http://somesite.org" and "https://somesite.org" could, potentially, point to different content (certainly different vhosts).
Web servers already display different content to users based on their geographical location or their login cookies or any number of state variables, and these content changes are not reflected in the URL. Your point means nothing.
Sites using https generally do so because they want to exchange sensitive data, and the use of a self-signed certificate might indicate that a MiM attack is in progress, or (possibly more likely) that the site is being run by a cowboy outfit who can't be arsed to get proper certificates. So, a self-signed https connection is always slightly fishy (there are plenty of innocent explanations, but identifying those requires human judgement + technical understanding).
This is a tautology. Since today's browsers are so alarmist about self-signed certificates, the use of self-signed certificates is automatically fishy. If you remove the alarmism then the amount of legitimate usage of self-signed certificates would increase dramatically.
self-signed https = someone could be mounting a man-in-the-middle attack or you may have been spoofed/phished to the wrong website.
The same holds for regular http. Someone could be mounting a man-in-the-middle attack with regular http.
Meanwhile, there is one big difference between http and self-signed https that you omitted. With regular http (and only regular http), large-scale attacks like police surveillance and content filtering become possible. https (even self-signed) prevents large-scale passive attacks.
I'd suggest (3) is by far the best place at which to start nagging - most users will rarely encounter this situation (only sites with very small user bases, like home servers or in-development sites have a real excuse for not getting a cert) so you're not going to swamp typical users with bogus warnings. For the typical user, this does mean that something out-of-the-ordinary is happening.
Again, the fact that self-signed certificates are out-of-the-ordinary is a tautology that you helped to set up by insisting that they be treated as out-of-the-ordinary.
And remember at the end of the day, all browsers like firefox actually do is warn you, encourage you to view the certificate and decide whether you want to trust it temporarily or permanently
NO! That's not what firefox does. If firefox did in fact do what you claimed it did, then I would be happy.
In practice, firefox effectively blocks self-signed certificates entirely. It takes five (count them, five) mouse clicks to connect to a self-signed https site in firefox, compared with one mouse click in IE. A regular user is scared away after even one mouse click, much less five. Thus in practice firefox ends up blocking self-signed certificates entirely.
Regular http has no warnings whatsoever, even though every attack against self-signed https is also possible against http, and some attacks against http are not possible against self-signed https. This situation is absurd beyond belief.
Yes, a self-signed https connection can be more dangerous than a plain http one if you see the "https" or the "golden padlock" and assume you have a secure connection.
The obvious solution is: don't display "https" or the "golden padlock."
There's no reason why browsers have to display "https://" or even "http://". The average non-technical user doesn't care about the protocol; they just care about the "golden padlock." On the other hand, the average technical user already knows what's going on.
Nobody here is arguing that self-signed https connections deserve a "golden padlock." That's your own straw man.
The proposal is that we should treat self-signed https connections the same as unencrypted http connections. The same. Not worse. Not better. The same.
I have yet to see anybody articulate an even remotely coherent argument against this proposal.
2 - Defragging: similarly, if you're moving data around in dead space without safely duplicating it or having a filename pointing to the blocks in use at any given time, you're not being careful. Also, which defraggers have random 3-minute gaps in operation that would even allow GC to kick in?
I think it is time to start bringing the discussion to a close, as it appears that we do share at least some common ground.
I will comment only on this one question. Your implication that the 3-minute rule somehow makes the GC "safe" is missing my point entirely. Yes, in practice there are checks and balances such as you describe, that make the GC unlikely to screw up. But, in my view, "unlikely" is not good enough. I want, and need, perfect (logical) block storage and retrieval. This should and must be the design goal. Of course, this goal is impossible to achieve in practice. For example, if the firmware (such as older, pre-SSD firmware) is designed with the goal of providing logical block storage, but fails in this task because of some honest bug, then I can understand that. At least, in this case, the programming code was written unambiguously with the correct goal (and no other directly conflicting goals) in mind, even if this goal was not achieved in practice due to an unintentional bug.
However, when a manufacturer deliberately designs firmware with the goal of deleting logical sectors, no matter how well-intentioned or well-implemented, this design goal (by definition) must come into conflict with the original, core goal of reliable (logical) data retrieval. I do not care what happens in the underlying physical layer, but I do care very greatly about data accuracy at the logical layer. The existence of certain checks and balances to prevent data loss is better than no checks and balances, but it is not better than the REAL alternative, namely, firmware that stores and retrieves logical blocks correctly, and is designed for this and only this purpose, without any other directly contradictory design goals.
No one, not even rocket scientists, has ever succeeded in writing bug-free software. But one should make an effort to minimize the number of opportunities for data loss bugs to arise. Firmware-based logical-sector garbage collection fundamentally and irreconcilably contradicts every reliability design principle known to man. That is why I consider the idea to be so abhorrent.
This GC only works with NTFS filesystems. If you are operating an SSD device using an NTFS file tree to store data, then you (as a programmer) are not using the drive as a block device as you suggest; you're accessing it (or you should be accessing it) via the abstract file tree.
The ATA standard (parallel or serial) does mandate that the drive appear as a logical block device. A drive with an ATA connector must honor that requirement or (IMO) it violates the standard. USB Mass Storage is another example where the standard mandates logical block storage -- what happens if I put one of these defective drives into a USB enclosure?
The only difference is that people seem to take block-level access to the disk for granted;
For many non-edge-case applications, like whole-disk encryption (remember, I'm a cryptographer), block-level access is exactly what you need, and anything less is unacceptable.
I understand that the firmware is supposed to (obviously) ignore non-NTFS volumes and fall back to block storage semantics. But the mere presence of active garbage collection is unwanted to me. It adds another possible failure opportunity.
The problem, from my perspective, is that your arguments hinge on the idea that marking data as 'deleted, and the filesystem can now overwrite it at some random future point, perhaps instantly or never' (the HDD model) is in some way better than 'deleted, purge at first opportunity' (the SSD model). From my perspective, I'd prefer the latter; at least then I know what's happened to data after it's been marked for deletion.
The latter can (and should) be implemented with explicit TRIM support. The operating system must have control over purging.
I'm still keen to see those realistic real-world use cases. If another poster has posted them, can you provide a link?
Link is here. In short:
I'm guessing you don't consider networked computers (e.g. SMB shares, FTP sites, NFS mounts) to be storage devices either then, since the remote host will merrily overwrite deleted files with other people's data however it likes there too?
This is of course a spurious comparison. SMB, FTP, and NFS are presented to the operating system as file trees. A drive with automatic garbage collection is presented to the operating system as a block device ... but it does not actually implement the correct semantics for a storage block device.
What other somethings do you have in mind?
It seems that another poster has already enumerated more ways for automatic garbage collection to break.
p.s. Just thought of another example. RAM. Where you store data in memory logically, and how it is arranged physically - including zeroing of dead pages - are completely out of your control and even out of your view. Does this mean you consider RAM not to be a storage device, since you can't reliably construct a stego side-channel using dead pages of memory?
Again, RAM is presented to the OS as logical addresses, and it does faithfully restore the data that was stored in those logical addresses.
A hard drive, like RAM, presents a logical block layer to the OS which is decoupled from the underlying physical data storage. Correct data storage and retrieval is required at that logical block layer. Automatic garbage collection violates the integrity requirements of a hard drive even at the logical layer. It imposes a secondary logical layer which assumes you are using a standard filesystem in a standard way. This introduces an additional and very scary mode of failure: the possibility now exists that the firmware might actively delete certain logical block data without the knowledge of the operating system. Of course, this could happen even with older, regular firmware, but only as an accident--by default, older firmware is programmed to store everything at the logical layer, no matter what it is. Active deletion raises the stakes considerably.
Honestly, the more I think about this, the more appalled I am that any manufacturer would actually do what you describe. I will be sure to make every possible effort to avoid such drives.
Well, if you think I (and the tech support staff on various SSD manufacturers forums) are wrong, you're welcome to buy an SSD and check for yourself. It's not quite as easy as typing 'it's impossible' a bunch of times, but it's a lot more likely to be correct.
A SSD that performs automatic garbage collection by interpreting the filesystem in firmware is not, in my opinion, a storage device.
Suppose I am a filesystem developer. Suppose I want to modify NTFS in such a way that deleted segments of an NTFS disk layout become (in my modified filesystem) a repository for meaningful data. This is not as absurd a concept as it appears. In my line of work (cryptography), storing actual meaningful data in deleted segments might be something that you want to do, for example in steganography.
If the SSD goes deleting disk sectors behind my back, then it becomes impossible for me to develop said filesystem. A storage device should store what I tell it to store. If it doesn't do that, then it's not a storage device. In this sense, it is, by definition, impossible for a valid storage device to implement automatic garbage collection at the filesystem level. Any device that does such a thing, by definition, does not meet the requirements of a storage device, the primary one of which is to retain data without alteration.
Sure, those deleted sectors are safe to erase in an NTFS volume, but how do you know that my operating system is using this NTFS volume as an NTFS volume? What if I'm doing steganography or something where those deleted sectors matter?
The same way they added GC to older models of SSD drives where it didn't already exist of course, and the same way they update features on any hardware. You flash the firmware with revised code.
Remind me never to upgrade the firmware on any hard drive ever again. I do want TRIM support, but I do not want automatic garbage collection, for the reasons outlined above.
I will concede that it is possible for a write-only device to implement automatic garbage collection at the filesystem level, but I maintain that no valid storage device can do so, since to do so violates the core requirement of a storage device in a fundamental and unfixable way.
Actually, this is no longer correct. SSDs (such as the one in this study) are quite capable of examining the filesystem stored on the drive, independently, and the concept of 'dutifully' and ignorantly maintaining deleted data goes out of the window as a result.
What you're describing is impossible. It might be possible for some of the more common filesystems, such as FAT or NTFS (although, given the difficulty of supporting NTFS in Linux, I highly doubt that embedded firmware on a drive can parse the NTFS format). It is utterly impossible in the case of new filesystems. Think about it -- if a piece of hardware predates the creation of ext4, or ext5, or whatever, then how can the hardware understand the filesystem?
Pure ad hominem is a sure sign that your argument has no merit.
That's a nice straw man, but I have made a great effort to emphasize that Watson is bypassing strategically relevant aspects of human Jeopardy, and not just irrelevant things like breathing or pumping blood.
It's not a "controversial" statement. People are questioning the parsimonious rule-lawyering. Is there even a rule written anywhere about how contestants must receive the information?
Even if you dislike the nit-picking, it is indisputable that Watson had an advantage at the buzzer. Even the mainstream media picked up on this fact.
I'm not impressed that Watson can buzz a buzzer faster and more accurately than a human. I'm not impressed that Watson won the game, because all things being equal, accurate buzzing is a huge advantage in Jeopardy. I am impressed that Watson can participate in the game well enough that buzzing becomes an issue, but all the media hype (including, sadly, the hype on slashdot) is about the fact that Watson won, not about the huge improvements in AI that rightfully should be highlighted.
I'm not denigrating Watson in any way.
Except, you know, that "Underwhelming achievement" and "I for one am not at all impressed by this feat." thing. That was pretty denigratory, wouldn't you say?
As I have made amply clear in every single one of my posts, my objection is to the title of this article: "Watson Wins Jeopardy Contest," and to all surrounding media hype along the lines of "Watson wins Jeopardy OMG"
Given Watson's inherent advantages in timing, winning a Jeopardy game (or what remains of a Jeopardy game after modifying it to allow such computer contestants) is a decidedly underwhelming achievement, and not something that I find impressive or meritorious in any way, shape, or form. The fact that Watson won the tournament is the least important achievement out of all the things that Watson accomplishes, and I find it irritating that it is the most hyped.
None of this has anything to do with Watson itself. It is the combination of Watson and calling it a Jeopardy player that I find objectionable.
In case you did not find any of the above clear before, I hope I have now made this clear.
But again, while all this might be relevant to professional Jeopardy matches, this wasn't one.
Yes, this is exactly and completely my point. This wasn't a Jeopardy match.
It was simply an impressive way to show off what Watson could do, not some "humans vs machines" ultimate showdown, despite the media portrayal.
100% agree.
I just think you are focusing too much on a largely irrelevant facet of this particular show.
We'll just have to agree to disagree. The entire thrust of everything I've said is that the mechanics and strategy of buzzing is not an irrelevant facet of the game of Jeopardy.
The contrast with IBM's Deep Blue is perhaps illuminating. Deep Blue, of course, did not actually physically move chess pieces. But I have no objection to saying that Deep Blue plays chess. The act of physically moving chess pieces is strategically irrelevant to the game of chess. Contrast this with the role that input/output plays in a Jeopardy game, and you'll have to agree (I hope) that timing is a bigger part of the game of Jeopardy than it is in chess.
A gray area (for me) might be the question of whether Deep Blue can play blitz chess, where the physical movement of chess pieces does become strategically relevant to the game.
The three huge achievements are :
- very fast information retrieval; Google et-al have similar technology, but let's not forget that it's all pretty new.
- natural language query decoding; Watson doesn't understand NL queries, but it is able to decode them into search queries.
- precise information extraction and presentation; whereas Google produces several pages of results with the answer in there (and can often highlight the right result) Watson can pull "the answer" out and present it in the required format.
Absolutely agree.
Notice what you left out? "Winning a Jeopardy contest." Yet the media hype (and even the title of this Slashdot article) is all about just winning a Jeopardy game. That's what bothers me.
In either case, I'm confident Watson would have won. Just because Ken and Brad don't know how to process electronic signals, that's not Watson's fault. Yet you are trying to penalize Watson because he has no eyes or ears.
Nothing of the sort. I do think Watson would have won using eyes and ears anyway, but it's not a question of "penalize." Do you "penalize" a computer in a 100-meter dash because it has no legs? Of course not. A computer cannot even compete in a 100-meter dash, but that's not a "penalty" -- it's a statement of fact.
Watson won a Jeopardy-like game, but that game is not identical to Jeopardy. I don't think Watson is capable of playing Jeopardy. The game of Jeopardy contains elements which inherently make sense only in the context of humans. Again, for emphasis (because everyone else here seems to completely misunderstand my point), I'm not denigrating Watson in any way. I'm just saying that the game of Jeopardy means one thing, and Watson is not playing that game.
I bet there's nothing in the Jeopardy official rules that says that you must see or hear the clue. They have had blind contestants and deaf contestants, so clearly they are open to alternate sensory methods.
I think it's pretty safe to say that no contestant prior to Watson has ever received an electronic data feed.
Watson actually did use a trivial actuator to press the physical button.
I was aware of this from the outset, and nothing that I wrote in any of my posts contradicts this, or is in any way based on a lack of Watson pushing a button.
I couldn't imagine someone turning these differences into some sort of conspiracy, but there you are.
For the n-th time, I'm saying something very simple: Watson is playing a game not equal to the game of Jeopardy. I went out of my way to clarify that Watson's achievement is meritorious in its own right. It's just that the achievement, whatever it is, is not Jeopardy. I don't know how you infer a conspiracy theory from this.
I don't know about you but I don't think it is even a given an artificial intelligence should communicate or receive input via OCR and voice recognition. It seems far more efficient to communicate via text in the first place.
That's an entirely separate issue and one where I agree with you. Of course a computer should communicate via text.
I think you're reading far more into my comments than what I actually said. All I said is that the game of Jeopardy does not communicate with contestants via direct digital electronic signals. Therefore Watson is not playing Jeopardy. (It is doing something else, equally impressive, but that something is not Jeopardy.) The fact that this simple objective statement of indisputable fact is somehow controversial is just mind-boggling.
They miss the real point: That a computer could do a level of natural language processing that was impossible before.
On the contrary, I fully agree that this is the real point.
You can label this achievement as impressive in many different valid ways. Just don't call it Jeopardy. That is the one characterization that is objectively false.
Since the "controversy" in question is some dude on Slashdot shitting on an achievement orders of magnitude bigger than anything he will ever achieve, I sincerely doubt it made their radar.
I do not post anonymously. You can easily find me in any search engine. I am confident in my achievements and see no need to defend them. Perhaps you can reveal your own identity so that readers can fairly judge which among the two of us is "some dude on Slashdot"? In any case, such personal attacks add nothing of value to the debate.
You completely avoided my question. If it's so easy, why didn't they just do it? Explaining how easy it is does not answer my question. It just makes it more and more incongruous that they did not just do it.
Actually, you are wrong about the buzz in. It gives the humans an advantage. The reason is that Watson has a mechanical buzzer that it presses. So the only advantage would come from reacting faster. However, the rules state that you can buzz in only after the host has finished reading the clue. If you buzz in earlier then you are penalized by .25sec.
Watson has a computer clock. It never buzzes in early, and it never suffers the .25sec penalty. The humans did suffer this penalty on several occasions.
It's ridiculous beyond belief to claim that the humans had the advantage in buzzing in.
OCR on a clean image (like the Jeopardy screens) is a solved problem. A physical button can be pressed with a trivial actuator, which could be almost as fast as the direct digital connection. How would adding either of those features to the machine make it any more impressive?
As I've already pointed out, if you think this feature is trivial, then why didn't IBM just implement it to avoid controversy? Make no mistake -- IBM consciously chose to give Watson an inhuman advantage.