Yes you seem to not want us to read the actual news article but instead the spin from the gun sight. I'm guessing there is something wrong with the original article. If there is a link in the site, you can COPY and PASTE it HERE!!!! Type Ctrl+C to copy and Ctrl+V to paste.
Proportionalize, based on the person's total income (of any kind) vs the average income. If you made 2x the average income, you pay 2x share. If you made 1/10 the average income, you pay 1/10.
That is the same tax rate for everybody, described rather obscurely. Is that what you intended?
If you intended the *rate* to be proportional, ie the person who makes 2x the average pays 4x, and the person who makes 1/10 pays 1/100, that does not actually integrate the correct sum (except in the trivial case where everybody makes the average amount). And it has the problem that the highest incomes have to pay more than 100%.
During the time they are alive a smoker will use more medical care than a non-smoker for the same period of time. Therefore it makes perfect sense that they should pay more for insurance. They are not paying for the insurance after they die, while the non-smoker will continue to pay for insurance and thus make up for the fact that in the end they may cost more.
I think he means "a large screen that can hold more than one job at a time". Desktop has nothing to do with how powerful the machine is, and I really would not be surprised if in the end the machines on desktops and in phones are of equal power (probably because the desktop is just a screen communicating with bluetooth to a phone that is the actual computer).
Say what you want about Canonical, but Mir apparently kicked the Wayland developers in the ass and got them working again. Which is why there are so many posts about Wayland recently.
It takes a lot more of those to send a compressed image over the wire than it does to send the instructions to build an image
This is false. Have you ever worked with 3D output. We use NX here and to run VirtualGL because the bandwidth to send the image of the GL window is about 1/20th the size of the GLX requests.
As many others have pointed out this is also false for most modern 2D applications because they render locally and send images. This could be blamed on the lack of a PostScript/SVG capable rendering, but I suspect even with SVG support the image will be smaller (this is why Google Maps sends images rather than SVG).
If I've got an ssh open to a remote host, all X11 apps I run on that connection on the remote host automatically appear on my own desktop without any special treatment at all.
No they don't. The reason they appear is because the environment variable $DISPLAY has been set to a particular value, and the application you are running is linked with libX11.so, which contains code that looks at $DISPLAY and determines that it needs to open an ssh connection back to another application running on your local machine (the X server) and to package up all the xlib requests and send them over the network to your xserver, which unpacks them and interprets them the same way it interprets local ones.
There is no magic. The claim that somehow the remote machine does not contain "the desktop" is also false, there is at least libX11, and in modern applications there is the toolkit library like Qt or GTK, rendering such as pango/cairo/freetype, libpng and libjpeg, and stored on that machine are all the fonts, icons, and other images the application is going to draw.
The problem is the above complexity is being described for Wayland while X is being described as "ssh -X makes it work".
When Wayland is in use you can be pretty certain that ssh will be quickly fixed to set $WAYLAND or whatever is needed to make the remote clients work, and it will be just as "automatic".
From footage I have seen, the crowd was smaller than when the marathon was finishing. Also there was already a lot less press coverage, and nothing live any more.
I can't think of any reason a bomber who wants to cause maximum fatalities or maximum publicity would delay to this point, rather than do it when the top runners are finishing. They could possibly have even killed somebody famous. I think the bombers were just unable to get their bombs to the location before this point, possibly due to higher security, or just the crowds.
"MS" is also Missisippi, it is the stock symbol for Morgan Stanley, it means Master of Science, and there is quite a few other things it can mean which can be put in the same sentence as Microsoft. So I agree the diatribe against "M$" is stupid, it is a useful abbreviation. I also notice that if somebody says something insulting like "Microsuck" nobody comments, but if somebody says "M$" suddenly out of the woodwork comes all the "oh you are childish! Childish! Childish!" responses. I think it shows desperation by some Microsoft defenders who are not at all secure in their claim that nobody should use "M$".
It was a desktop box, not a laptop. Yes I probably could replace Ubuntu with something else, or just install xfce, but I have not gotten around to that yet, and really Unity is not that bad... (I would probably also be saying Windows 8 is not that bad if I had been using it for a month).
I've always wondered about this. I would think it is not very important whether the debt is greater or less than 1 year of GDP. In fact the debt is many hundreds larger than 1 day of GDP, so if you want to make the debt sound bad compare it to that. Or to a minute. Or if you want the debt to sound smaller, say it is smaller than 10 years of GDP.
Worry that Windows 7 installed machines would become unavailable, and worry about UEFI or whatever the booting is, got me to replace my pretty old desktop which only ran Linux because Windows stopped booting for some reason, with a $425 ASUS. I violated the warranty to put a cheap nVidia graphics card in and to repartition the disk to run Ubuntu as well (the new Unit stuff, unfortunately similar to Windows 8), and ran decrapifier on Windows. Only problem is that the sound is very quiet (in both systems) which is probably a hardware problem, and stupid Windows does not recognize my serial keyboard unless I also leave a USB keyboard plugged in (the serial keyboard works for the BIOS and for Ubuntu), and Ubuntu has an equally stupid bug where it swaps my monitors until the first time I move the mouse between them.
Any case, I wanted to say that Windows 8 actually *caused* a sale recently. I wonder if people like me, trying to upgrade to the best thing available that did not run Windows 8, caused any increase in desktop sales, slightly offsetting the overall reduction.
According to the quote, only the Iranians are on your computer. The Russians and Chinese are still trying to get in.
I think it is probably best to just wait, the Russians and Chinese will kick the Iranians out, and then fight each other, since there is no way two or three of them will fit in a little computer. You only have to deal with the final winner.
I don't have Python3.3 here but this is pretty much the failing code:
# make p be a Unicode string with an unpaired surrogate: p = u"\uDC00" # "encode" it into UTF-8: q = p.encode("utf8") # "decode" it using the surrogate escape: r = q.decode("utf8", "surrogateescape") # what is r? r u'\udced\udcb0\udc80' # did we preserve data? r == p False
I'm not trying to convert Unix filenames to Windows filenames
I want to write a Windows filename into a file that is UTF-8, and read it back. The new Python 3.3 behavior makes this impossible (unless I use non-standard "decoders"). Or are you suggesting that if I want to embed Windows filenames into a text file, I have to use UTF-16 for the text file?
It appears that the normal "decoder" for UTF-8 and also this new "surrogateescape decoder" both consider the UTF-8 encoding of an unpaired surrogate as an "error". The normal "decoder" will throw an error, the other one will replace it with 3 unpaired surrogates. In both cases I have failed to produce all possible Windows filenames, because they can contain unpaired surrogates. This is very similar to the problem on Unix where UTF-8 filenames can contain UTF-8 errors, and thus cannot use the results of the normal "encoder" into UTF-8.
I also believe that the "surrogateescape encoder" for turning UTF-32 back into UTF-8 can be fooled into producing a UTF-8 string that will decode into a different UTF-32 string, this is done by placing the surrogates in a pattern such that the resulting "errors" are actually a valid UTF-8 encoding. I have not proven this yet but it seems likely.
It is impossible to write some valid Windows NTFS filenames in UTF-8!!!! That is a serious defect.
It has nothing to do with the fact that they made UTF-8 -> Unicode -> UTF-8 lossless.
It is IMPOSSIBLE for UTF-8 -> Unicode -> **UTF-16** to produce all possible UTF-16 strings!
Therefore it is IMPOSSIBLE to store Windows filenames losslessly in a UTF-8 text file.
In addition I worry that there are security problems in that a sequence of unpaired surrogates in UTF-16, when converted to UTF-8 and back again, may turn into another unexpected character (because they were arranged so that the resulting UTF-8 instead of having N error bytes had a valid encoding). This could be prevented but it makes the UTF-8 "encoder" much much more difficult and bug prone.
They need to fix this, and the only fix that I think will work:
1. Conversion from UTF-8 to UTF-16 must translate unpaired surrogates so that all possible UTF-16 strings can be created this way. Errors can be translated as the 0xDCxx codes.
2. Conversion from UTF-16 to UTF-8 must translate unpaired surrogates to the correct 3-byte UTF-8 sequence. If you want it could carefully check that undoing the 0xDCxx errors results in an invalid UTF-8 sequence and undo it back to error bytes. I'm not sure if there is any point because it is still going to be lossy, though "less lossy" I guess if you assume the main source is UTF-8 errors.
3. Unicode strings must save the UTF-8 as long as there is no modification to the string so that if the UTF-8 is asked for the result is the identity, with no reliance on lossy conversion.
The slide for UTF-16 clearly says that UTF-16 is the result of "encoding", not "decoding": http://farmdev.com/talks/unicode/. Also I did experiments and the new encoding cannot produce unpaired surrogates, therefore it cannot produce all possible NTFS filenames. Thus they have managed to reproduce the UTF-8 problem on Unix on Windows with UTF-16 as well.
I find it odd you seem to think that the only source of filenames is os.listdir(). I know a lot of people like to put filenames in text files. It is kind of useful, in fact this is supported directly by Python when you use a filename in quotes in a.py file! Yet they have made it impossible to place all possible UTF-8 filenames in a Python script unless a bytes api is used and the programmer has to write the UTF-8 code units individually as \xNN sequences, making it unreadable.
Your suggested solutions are just like all the other ones: basically never use Unicode at all in your Python program and use byte arrays everywhere. Technically this works but it does raise questions as to why they bothered to do all this work to support a "Unicode" object. I also find it really painful to write quoted strings if the string contains any non-ASCII because I have to use \xNN sequences for the UTF-8.
Also we are back to the stupid programmer problem: when the programmer says "print bytes" and it throws an exception (because there was an invalid UTF-8 encoding in there) in 95% or so of the cases (based on my own experience with fixing stupid programmers mistakes) the "solution" is to somehow throw all bytes with the high bit set away so that in effect the strings are ASCII only (sometimes the bytes are literally thrown away, sometimes they think they are being smart by replacing them with "\xNN" or \NNN octal, and sometimes they double-UTF-8-encode them. I have NEVER seen a programmer fix the exception in any way that preserves correct UTF-8, and I have also never seen anybody who says "oh it is better that I got that exception rather than seeing my string printed out".
This is the underlying problem: the current behavior encourages ASCII-only use and is effectively destroying attempts to migrate to Unicode. They need to make it easy to write a reliable program that uses UTF-8 and UTF-16, which means it must not do something unexpected if a physically possible pattern is encountered in the data.
They seem to have screwed up badly, I tested this a bit.
I wanted to check if the translation to/from UTF-16 was lossy due to the surrogate replacements. It is not, but only because it appears that the UTF-8 encoding of 0xDC80..0xDCFF is considered "invalid" and thus turns into 3 of these codes when converted to UTF-16 rather than one. Thus it is impossible to write some valid Windows NTFS filenames in UTF-8 (actually I think it is impossible to put any of the codes 0xD800..0xDFFF into a UTF-16 filename by using "decode" as they like to call it). They also seem to have missed the problem that a malicious UTF-16 string can contain a sequence of these characters that "encodes" into a valid UTF-8 string and thus "decodes" back into an unexpected UTF-16 character.
They really, really need to think about this again and get it right. We now have 3 encodings (UTF-8, UTF-16, and UTF-32) and NONE of them can be losslessly translated to another one. They need to realize that UTF-16, including errors, can be losslessly translated to/from UTF-8 by obvious means. The opposite is NOT true!!! UTF-32 can also be losslessly translated to UTF-8 except for code points greater than 0x10FFFF (or 0x7FFFFFFF if you use 6-byte UTF-8). The opposite could be true by selecting values with the sign bit set to represent errors, though I have never seen this done.
If a "Unicode" string is built from UTF-8, it needs to keep the original UTF-8 around. There is no way they are going to get the bugs out of there mess with the filesystem if they don't figure this out!
The PEP is interesting and seems to be one of the sources of the 0xDC80..0xDCFF. I saw it earlier as a method of fixing the handling of argv without throwing exceptions, though I don't know where the documentation is.
This is still wrong in that you have to pass the special "surrogateescape" and use encode/decode. I want to be able to store a "Unicode" string that contains a UTF-8 error without an exception being thrown. The exception would be thrown if you attempt to translate the string to UTF-16 or look at the code points (though I also recommend there be ways to avoid the exception).
Their problem is that they seem to think that UTF-16 (or perhaps UTF-32) is somehow "decoded" and UTF-8 is "encoded", while in fact it is the opposite, and they seem to be thrashing around trying to hide the fact they got it wrong with this filesystem stuff. On Unix at least a filename is a stream of bytes, and changing the "locale" should not change what file it identifies. Even on Windows with NTFS a filename is UTF-16, which is not "decoded" in their terminology, despite the ridiculous claim in the PEP that "Windows got it right". Windows with the use of 16-bit characters has caused an extrondinary amount of pain and has set I18N back decades.
A UTF-8 string is an ARRAY of bytes, therefore comparing it to an ARRAY of fixed-size things like floats is correct. I have no idea why you are insisting it is somehow like adding more bits to a fixed-size object.
I want to put an arbitrary array of bytes into a "string" and DEFER the exceptions or other errors that are thrown until something actually needs to look at the Unicode code units. In particular I should be able to copy the string to another string and to a byte array without any errors and do raw I/O. I would further argue that any operation that does not have to look at the code units, such as concatenation, also does not produce errors, but that is somewhat less important.
When displaying to the user it would be a lot better if error blocks were shown for the errors and anything that does parse as UTF-8 shows the resulting Unicode glyphs. Readable Unicode for the majority of the string helps considerably for the user figuring out what is wrong.
However conversion to error blocks can be a security problem in other parts of the software, meaning that throwing exceptions, or another form of the data where errors are an object that is distinct from any possible code point, may be useful there. This is why it is really important that nothing be done to the UTF-8 data until as late as possible, since only the final step knows what type of error handling is correct.
Current handling of UTF-8 by Python is extremely error prone and strongly discourages use of Unicode. As soon as the stupid programmer decides to strip out the bytes with the high bit set because they threw an exception, we have regressed to ASCII-only, which is worse than what we had in about 1983...
The Python3.3 "hybrid" string is perhaps a solution but there needs to be some changes. It can store "ASCII" or "UCS-2" or "UTF-32". I would add the ability to store "UTF-8", reusing the ASCII pointer, but the UTF-8 is allowed to be an arbitrary byte array. It can also store "UTF-16" including errors in the UCS-2 pointer. It is also capable of storing fixed-sized characters in one of the 3 arrays (ASCII if nothing is greater that 127, UCS-2 if there is only BMP and no paired surrogates). It would figure out and convert to these fixed-sized arrays only when len() or indexing is used. I would add the ability to get "len" and "index" for UTF-8 and UTF-16 so you can work with code units. In addition there should be iterators that let you visit the code points without conversion, which is where there would be huge savings over current Python handling of Unicode. Iterators that also do different normalizations would be really useful too.
Look up IEEE NaN and then say something intelligent about arrays of floats. They can contain invalid sequences of bits and yet all systems I am aware of do not suddenly make the arrays "not arrays of floats" and do not make them unprintable and do not make them impossible to use as a source for math operations.
Yes, you can use byte arrays everywhere in every api. But that kind of defeats the purpose of Python strings, since you are no longer using them!
And a 1000-byte UTF-8 string with a single error in is 99.9% UTF-8 characters. In fact I may not be able to figure out it is this magical "not UTF-8" without a very expensive scan! That is incredibly stupid.
You know your precious UTF-16 can have errors, too? Did you notice that nothing throws exceptions on them? Think hard about why this is, rather than spouting idiocy. UTF-8 is not different than UTF-16 and is in no way less of "Unicode" despite the absolute hatred spouted by some people and the attempts to sabotage it by this insisitence that "errors" are somehow not part of it.
Slashdot Mirror
Yes you seem to not want us to read the actual news article but instead the spin from the gun sight. I'm guessing there is something wrong with the original article. If there is a link in the site, you can COPY and PASTE it HERE!!!! Type Ctrl+C to copy and Ctrl+V to paste.
Proportionalize, based on the person's total income (of any kind) vs the average income. If you made 2x the average income, you pay 2x share. If you made 1/10 the average income, you pay 1/10.
That is the same tax rate for everybody, described rather obscurely. Is that what you intended?
If you intended the *rate* to be proportional, ie the person who makes 2x the average pays 4x, and the person who makes 1/10 pays 1/100, that does not actually integrate the correct sum (except in the trivial case where everybody makes the average amount). And it has the problem that the highest incomes have to pay more than 100%.
During the time they are alive a smoker will use more medical care than a non-smoker for the same period of time. Therefore it makes perfect sense that they should pay more for insurance. They are not paying for the insurance after they die, while the non-smoker will continue to pay for insurance and thus make up for the fact that in the end they may cost more.
Post the original link, then. It should not be that hard.
I think he means "a large screen that can hold more than one job at a time". Desktop has nothing to do with how powerful the machine is, and I really would not be surprised if in the end the machines on desktops and in phones are of equal power (probably because the desktop is just a screen communicating with bluetooth to a phone that is the actual computer).
Say what you want about Canonical, but Mir apparently kicked the Wayland developers in the ass and got them working again. Which is why there are so many posts about Wayland recently.
It takes a lot more of those to send a compressed image over the wire than it does to send the instructions to build an image
This is false. Have you ever worked with 3D output. We use NX here and to run VirtualGL because the bandwidth to send the image of the GL window is about 1/20th the size of the GLX requests.
As many others have pointed out this is also false for most modern 2D applications because they render locally and send images. This could be blamed on the lack of a PostScript/SVG capable rendering, but I suspect even with SVG support the image will be smaller (this is why Google Maps sends images rather than SVG).
If I've got an ssh open to a remote host, all X11 apps I run on that connection on the remote host automatically appear on my own desktop without any special treatment at all.
No they don't. The reason they appear is because the environment variable $DISPLAY has been set to a particular value, and the application you are running is linked with libX11.so, which contains code that looks at $DISPLAY and determines that it needs to open an ssh connection back to another application running on your local machine (the X server) and to package up all the xlib requests and send them over the network to your xserver, which unpacks them and interprets them the same way it interprets local ones.
There is no magic. The claim that somehow the remote machine does not contain "the desktop" is also false, there is at least libX11, and in modern applications there is the toolkit library like Qt or GTK, rendering such as pango/cairo/freetype, libpng and libjpeg, and stored on that machine are all the fonts, icons, and other images the application is going to draw.
The problem is the above complexity is being described for Wayland while X is being described as "ssh -X makes it work".
When Wayland is in use you can be pretty certain that ssh will be quickly fixed to set $WAYLAND or whatever is needed to make the remote clients work, and it will be just as "automatic".
All the runners should have been carrying their own bombs, that would have stopped it!
From footage I have seen, the crowd was smaller than when the marathon was finishing. Also there was already a lot less press coverage, and nothing live any more.
I can't think of any reason a bomber who wants to cause maximum fatalities or maximum publicity would delay to this point, rather than do it when the top runners are finishing. They could possibly have even killed somebody famous. I think the bombers were just unable to get their bombs to the location before this point, possibly due to higher security, or just the crowds.
"MS" is also Missisippi, it is the stock symbol for Morgan Stanley, it means Master of Science, and there is quite a few other things it can mean which can be put in the same sentence as Microsoft. So I agree the diatribe against "M$" is stupid, it is a useful abbreviation. I also notice that if somebody says something insulting like "Microsuck" nobody comments, but if somebody says "M$" suddenly out of the woodwork comes all the "oh you are childish! Childish! Childish!" responses. I think it shows desperation by some Microsoft defenders who are not at all secure in their claim that nobody should use "M$".
It was a desktop box, not a laptop. Yes I probably could replace Ubuntu with something else, or just install xfce, but I have not gotten around to that yet, and really Unity is not that bad... (I would probably also be saying Windows 8 is not that bad if I had been using it for a month).
I've always wondered about this. I would think it is not very important whether the debt is greater or less than 1 year of GDP. In fact the debt
is many hundreds larger than 1 day of GDP, so if you want to make the debt sound bad compare it to that. Or to a minute. Or if you want the debt to sound smaller, say it is smaller than 10 years of GDP.
Worry that Windows 7 installed machines would become unavailable, and worry about UEFI or whatever the booting is, got me to replace my pretty old desktop which only ran Linux because Windows stopped booting for some reason, with a $425 ASUS. I violated the warranty to put a cheap nVidia graphics card in and to repartition the disk to run Ubuntu as well (the new Unit stuff, unfortunately similar to Windows 8), and ran decrapifier on Windows. Only problem is that the sound is very quiet (in both systems) which is probably a hardware problem, and stupid Windows does not recognize my serial keyboard unless I also leave a USB keyboard plugged in (the serial keyboard works for the BIOS and for Ubuntu), and Ubuntu has an equally stupid bug where it swaps my monitors until the first time I move the mouse between them.
Any case, I wanted to say that Windows 8 actually *caused* a sale recently. I wonder if people like me, trying to upgrade to the best thing available that did not run Windows 8, caused any increase in desktop sales, slightly offsetting the overall reduction.
According to the quote, only the Iranians are on your computer. The Russians and Chinese are still trying to get in.
I think it is probably best to just wait, the Russians and Chinese will kick the Iranians out, and then fight each other, since there is no way two or three of them will fit in a little computer. You only have to deal with the final winner.
I don't have Python3.3 here but this is pretty much the failing code:
# make p be a Unicode string with an unpaired surrogate:
p = u"\uDC00"
# "encode" it into UTF-8:
q = p.encode("utf8")
# "decode" it using the surrogate escape:
r = q.decode("utf8", "surrogateescape")
# what is r?
r
u'\udced\udcb0\udc80'
# did we preserve data?
r == p
False
I'm not trying to convert Unix filenames to Windows filenames
I want to write a Windows filename into a file that is UTF-8, and read it back. The new Python 3.3 behavior makes this impossible (unless I use non-standard "decoders"). Or are you suggesting that if I want to embed Windows filenames into a text file, I have to use UTF-16 for the text file?
It appears that the normal "decoder" for UTF-8 and also this new "surrogateescape decoder" both consider the UTF-8 encoding of an unpaired surrogate as an "error". The normal "decoder" will throw an error, the other one will replace it with 3 unpaired surrogates. In both cases I have failed to produce all possible Windows filenames, because they can contain unpaired surrogates. This is very similar to the problem on Unix where UTF-8 filenames can contain UTF-8 errors, and thus cannot use the results of the normal "encoder" into UTF-8.
I also believe that the "surrogateescape encoder" for turning UTF-32 back into UTF-8 can be fooled into producing a UTF-8 string that will decode into a different UTF-32 string, this is done by placing the surrogates in a pattern such that the resulting "errors" are actually a valid UTF-8 encoding. I have not proven this yet but it seems likely.
It is impossible to write some valid Windows NTFS filenames in UTF-8!!!! That is a serious defect.
It has nothing to do with the fact that they made UTF-8 -> Unicode -> UTF-8 lossless.
It is IMPOSSIBLE for UTF-8 -> Unicode -> **UTF-16** to produce all possible UTF-16 strings!
Therefore it is IMPOSSIBLE to store Windows filenames losslessly in a UTF-8 text file.
In addition I worry that there are security problems in that a sequence of unpaired surrogates in UTF-16, when converted to UTF-8 and back again, may turn into another unexpected character (because they were arranged so that the resulting UTF-8 instead of having N error bytes had a valid encoding). This could be prevented but it makes the UTF-8 "encoder" much much more difficult and bug prone.
They need to fix this, and the only fix that I think will work:
1. Conversion from UTF-8 to UTF-16 must translate unpaired surrogates so that all possible UTF-16 strings can be created this way. Errors can be translated as the 0xDCxx codes.
2. Conversion from UTF-16 to UTF-8 must translate unpaired surrogates to the correct 3-byte UTF-8 sequence. If you want it could carefully check that undoing the 0xDCxx errors results in an invalid UTF-8 sequence and undo it back to error bytes. I'm not sure if there is any point because it is still going to be lossy, though "less lossy" I guess if you assume the main source is UTF-8 errors.
3. Unicode strings must save the UTF-8 as long as there is no modification to the string so that if the UTF-8 is asked for the result is the identity, with no reliance on lossy conversion.
The slide for UTF-16 clearly says that UTF-16 is the result of "encoding", not "decoding": http://farmdev.com/talks/unicode/. Also I did experiments and the new encoding cannot produce unpaired surrogates, therefore it cannot produce all possible NTFS filenames. Thus they have managed to reproduce the UTF-8 problem on Unix on Windows with UTF-16 as well.
I find it odd you seem to think that the only source of filenames is os.listdir(). I know a lot of people like to put filenames in text files. It is kind of useful, in fact this is supported directly by Python when you use a filename in quotes in a .py file! Yet they have made it impossible to place all possible UTF-8 filenames in a Python script unless a bytes api is used and the programmer has to write the UTF-8 code units individually as \xNN sequences, making it unreadable.
Your suggested solutions are just like all the other ones: basically never use Unicode at all in your Python program and use byte arrays everywhere. Technically this works but it does raise questions as to why they bothered to do all this work to support a "Unicode" object. I also find it really painful to write quoted strings if the string contains any non-ASCII because I have to use \xNN sequences for the UTF-8.
Also we are back to the stupid programmer problem: when the programmer says "print bytes" and it throws an exception (because there was an invalid UTF-8 encoding in there) in 95% or so of the cases (based on my own experience with fixing stupid programmers mistakes) the "solution" is to somehow throw all bytes with the high bit set away so that in effect the strings are ASCII only (sometimes the bytes are literally thrown away, sometimes they think they are being smart by replacing them with "\xNN" or \NNN octal, and sometimes they double-UTF-8-encode them. I have NEVER seen a programmer fix the exception in any way that preserves correct UTF-8, and I have also never seen anybody who says "oh it is better that I got that exception rather than seeing my string printed out".
This is the underlying problem: the current behavior encourages ASCII-only use and is effectively destroying attempts to migrate to Unicode. They need to make it easy to write a reliable program that uses UTF-8 and UTF-16, which means it must not do something unexpected if a physically possible pattern is encountered in the data.
They seem to have screwed up badly, I tested this a bit.
I wanted to check if the translation to/from UTF-16 was lossy due to the surrogate replacements. It is not, but only because it appears that the UTF-8 encoding of 0xDC80..0xDCFF is considered "invalid" and thus turns into 3 of these codes when converted to UTF-16 rather than one. Thus it is impossible to write some valid Windows NTFS filenames in UTF-8 (actually I think it is impossible to put any of the codes 0xD800..0xDFFF into a UTF-16 filename by using "decode" as they like to call it). They also seem to have missed the problem that a malicious UTF-16 string can contain a sequence of these characters that "encodes" into a valid UTF-8 string and thus "decodes" back into an unexpected UTF-16 character.
They really, really need to think about this again and get it right. We now have 3 encodings (UTF-8, UTF-16, and UTF-32) and NONE of them can be losslessly translated to another one. They need to realize that UTF-16, including errors, can be losslessly translated to/from UTF-8 by obvious means. The opposite is NOT true!!! UTF-32 can also be losslessly translated to UTF-8 except for code points greater than 0x10FFFF (or 0x7FFFFFFF if you use 6-byte UTF-8). The opposite could be true by selecting values with the sign bit set to represent errors, though I have never seen this done.
If a "Unicode" string is built from UTF-8, it needs to keep the original UTF-8 around. There is no way they are going to get the bugs out of there mess with the filesystem if they don't figure this out!
The PEP is interesting and seems to be one of the sources of the 0xDC80..0xDCFF. I saw it earlier as a method of fixing the handling of argv without throwing exceptions, though I don't know where the documentation is.
This is still wrong in that you have to pass the special "surrogateescape" and use encode/decode. I want to be able to store a "Unicode" string that contains a UTF-8 error without an exception being thrown. The exception would be thrown if you attempt to translate the string to UTF-16 or look at the code points (though I also recommend there be ways to avoid the exception).
Their problem is that they seem to think that UTF-16 (or perhaps UTF-32) is somehow "decoded" and UTF-8 is "encoded", while in fact it is the opposite, and they seem to be thrashing around trying to hide the fact they got it wrong with this filesystem stuff. On Unix at least a filename is a stream of bytes, and changing the "locale" should not change what file it identifies. Even on Windows with NTFS a filename is UTF-16, which is not "decoded" in their terminology, despite the ridiculous claim in the PEP that "Windows got it right". Windows with the use of 16-bit characters has caused an extrondinary amount of pain and has set I18N back decades.
A UTF-8 string is an ARRAY of bytes, therefore comparing it to an ARRAY of fixed-size things like floats is correct. I have no idea why you are insisting it is somehow like adding more bits to a fixed-size object.
I think you may be arguing the same thing I am.
I want to put an arbitrary array of bytes into a "string" and DEFER the exceptions or other errors that are thrown until something actually needs to look at the Unicode code units. In particular I should be able to copy the string to another string and to a byte array without any errors and do raw I/O. I would further argue that any operation that does not have to look at the code units, such as concatenation, also does not produce errors, but that is somewhat less important.
When displaying to the user it would be a lot better if error blocks were shown for the errors and anything that does parse as UTF-8 shows the resulting Unicode glyphs. Readable Unicode for the majority of the string helps considerably for the user figuring out what is wrong.
However conversion to error blocks can be a security problem in other parts of the software, meaning that throwing exceptions, or another form of the data where errors are an object that is distinct from any possible code point, may be useful there. This is why it is really important that nothing be done to the UTF-8 data until as late as possible, since only the final step knows what type of error handling is correct.
Current handling of UTF-8 by Python is extremely error prone and strongly discourages use of Unicode. As soon as the stupid programmer decides to strip out the bytes with the high bit set because they threw an exception, we have regressed to ASCII-only, which is worse than what we had in about 1983...
The Python3.3 "hybrid" string is perhaps a solution but there needs to be some changes. It can store "ASCII" or "UCS-2" or "UTF-32". I would add the ability to store "UTF-8", reusing the ASCII pointer, but the UTF-8 is allowed to be an arbitrary byte array. It can also store "UTF-16" including errors in the UCS-2 pointer. It is also capable of storing fixed-sized characters in one of the 3 arrays (ASCII if nothing is greater that 127, UCS-2 if there is only BMP and no paired surrogates). It would figure out and convert to these fixed-sized arrays only when len() or indexing is used. I would add the ability to get "len" and "index" for UTF-8 and UTF-16 so you can work with code units. In addition there should be iterators that let you visit the code points without conversion, which is where there would be huge savings over current Python handling of Unicode. Iterators that also do different normalizations would be really useful too.
Look up IEEE NaN and then say something intelligent about arrays of floats. They can contain invalid sequences of bits and yet all systems I am aware of do not suddenly make the arrays "not arrays of floats" and do not make them unprintable and do not make them impossible to use as a source for math operations.
Yes, you can use byte arrays everywhere in every api. But that kind of defeats the purpose of Python strings, since you are no longer using them!
And a 1000-byte UTF-8 string with a single error in is 99.9% UTF-8 characters. In fact I may not be able to figure out it is this magical "not UTF-8" without a very expensive scan! That is incredibly stupid.
You know your precious UTF-16 can have errors, too? Did you notice that nothing throws exceptions on them? Think hard about why this is, rather than spouting idiocy. UTF-8 is not different than UTF-16 and is in no way less of "Unicode" despite the absolute hatred spouted by some people and the attempts to sabotage it by this insisitence that "errors" are somehow not part of it.