It sounds like this is designed to be compatible with the application api to X, so that is not a problem with it.
I think a great deal of even the Xlib api could be jettisoned because no modern program uses it. A simple thing to do would be to pretend to be 32-bit full color all the time, no matter what the hardware is, and eliminate all the colormap api.
I believe the functions that are being put in the kernel is the *setup* so that the user-space code in the X server has that "access to video memory and registers". That is actually a huge messy part of the current X server. Doing this in the kernel allows other programs to have an idea what X did, and also allows things other than X to get the same access to video memory and registers, and also means that a "video driver" is findable in the same lists as all your other drivers.
Note that this hardly seems to be a techincal issue. It appears to be entirely caused by bad communication and excessive complexity. Anyway here is what you do. There are some indications that Ibex fixed this but these steps fixed my Hardy version:
1. Edit/etc/laptop-mode/laptop-mode.conf and change correct line to read: CONTROL_HD_POWERMGMT=1 (this makes laptop_mode call hdparm)
2. Edit/etc/default/acpi-support and change correct line to read: ENABLE_LAPTOP_MODE=true (this makes power.sh run)
3. Edit/etc/acpi/power.sh Comment out or delete the 4 for...done loops containing $HDPARM commands. (this stops power-on from messing with the disks) And change the arguments to $LAPTOP_MODE from start/stop to "auto" in both cases. (this makes it run the laptop_mode command correctly rather than forcing the mode on and off)
4. Create/etc/pm/power.d/laptop-tools and make it read "exit 0" and then "chmod +x" it. (this stops suspend/resume from messing with hdparm settings)
5. Create/etc/pm/sleep.d/10laptop_mode_restart and make it contain the following: <pre> #!/bin/bash case $1 in hibernate) /etc/init.d/laptop-mode stop ;; suspend) /etc/init.d/laptop-mode stop ;; thaw) /etc/init.d/laptop-mode start ;; resume) /etc/init.d/laptop-mode start ;; *) echo Something is not right. ;; esac </pre>
Chmod +x this file. (this makes suspend/resume run the laptop tools)
HOW TO TEST:
This command will tell you how your disk is set:
sudo hdparm -I/dev/sda | grep "Adv"
The correct results to stop disk thrashing are 254 or 255. When laptop_mode is *really* on then the correct value is 1. If you see 128 then things are not working, this is the setting the disk resets to on suspend/sleep/power off.
I think it is just going to look in the contents of the controls. This would be really useful, for instance if you search for "Widget Model XJ123" it will now find a page by a manufacturer where the only place they list it is in a pulldown list that lets you choose the product to buy.
The idea is that you don't have to count *all* the paper ballots. Only a small subset until you are confident that the electronic total has not been tampered with.
You are right about gaming in practice. If somebody wants to tamper with the software and has any way to predict enough machines that will not be subject to hand counts then they can fix the election. This could be done by messing with the software that "randomly" chooses what districts to recount.
Unfortunately your analysis is completely backwards.
In the last two presidential elections the popular vote was clearly in favor of one candidate (Gore in the first one and Bush in the second one). A popular vote would have to be far closer than either of these to be contested. The electoral collage was why a small area with a close tie was amplified so that it made a difference.
Your idea of having a district per vote would help. An easier idea would be to have the electoral votes for each state divided fractionally by the vote in the state (this would preserve the greater weight for a voter in a small state). Unfortunatly the constitution says a state can decide how to assign their electoral votes and almost all of them say "they all go to the winning candidate" because this gives them more power in getting influence from the elected politicians.
We use an optical scan system that is fairly foolproof.
What makes you think it's foolproof. If it decided to count every 3rd McCain vote for Obama, how would you know?
Because the machine count would not match the collected paper ballots.
Only a small number of districts need a hand count to check this, as long as which districts was unknown before the election. This would make it far too risky to tamper with the software in the optical scan machine.
I have the problem that caused a huge amount of discussion here before, where the disk in the laptop repeatedly parks and then wakes a few seconds later to write something. I have used up about 1/3 of the lifetime limit on the disk for these parks. I have updated from Feisty to Gibbon and Hardy and it is still there.
I have a script to turn off the disk timeout to fix this. However I have to run it manually every time I unsleep the machine. I have followed the instructions to install it in init.d and power on/off but it appears to have no effect. Either it is not run or it's changes are undone after it is run. This is very frustrating and I see absoltely zero information about this.
I heard there was some analysis about why Windows does not have this problem, and it was determined that Windows reads the disk *continuously* until it sleeps. This led the manuafacturers to adjust the timeout very small so it is, in effect, a fast "Windows is sleeping" detector. Now it appears that Linux is using the disk far less often, but not zero, so the result is worse for the disk. It would be really cool if Linux actually stopped reading the disk at all when nothing is happening, but I realize this is difficult, so I think an acceptable solution is to replicate how Windows acts.
In any case, despite all the yakking about this earlier, I see no comments. Have they addressed this or not? Is it ever going to be fixed? Or can somebody tell me how to debug the init stuff and find out why my script has to be run manually?
I think it is enormously clever, while at the same time enormously useless.
It was pretty obvious that they had no way to know where the paper was, so they mostly demonstrated it as an x-ray into the image that was on the screen. I do not see any reason why the paper's location could not be determined by the touch detector, however.
I would think any practical use could be done by software moving a virtual "paper". One thing this would do is allow the papers to overlap.
I have to agree that FireFox's download display is pretty unusable. I did not know about "open containing folder" but even then it would be difficult to use. What I wanted was "copy the name of the downloaded file to the clipboard". I would have settled for "open the file where it was already downloaded". I could not find any way for it to tell me where it saved the downloaded copy. In the end I had to go back to the web page and do "save linked item as", which is really stupid considering I had already downloaded the whole thing (a pdf file).
Honestly, "I just saved a file and now I don't know where I put it" is more indicative of the human operating the computer, than it is of the computer apparently lacking facilities to find the files.
And this statement perfectly demonstrates why Linux is not now, and will not be for a very long time, a true Desktop OS.
Wow, and if you had some reading comprehension you would notice that the *solutions* he proposes are all for Windows:
(google desktop, etc) will all index files in ways that you can easily retrieve them beyond that base Windows will do.
I have to agree. I certainly am not voting for that airhead, but the RNC certainly can spend $150,000 on her clothes if they think it will help them get elected. This is tiny compared to other expenses. Perhaps she won't wear every outfit while campaigning, but I'm sure they spend orders of magnitude more on unused things like commercials that they decide not to air or on setting up campaign stops that are then cancelled.
And besides if you read those articles, they clearly state that this "ice age prediction" is *contrary* to prevailing opinion at that time.
I can clearly state that I was told by teachers (in very liberal Massachusetts) that CO2 would turn the planet into Venus. Whether this is true or not it is pretty obvious that the popular consensus was for warming.
Early 1980's? You are talking about "Nuclear Winter", not something caused by CO2. In fact enemies of Sagan and the Union of Concerned Scientists *attacked* Nuclear Winter by saying "everybody knows that pollution will cause the earth to get warmer".
Face it, you can make your revisionist history all you want, but I was also there. I was told for DECADES that "CO2 will make the planet like Venus".
I don't know what world you were in. I was told as a child about 100 times that "CO2 can make the planet like Venus due to a runaway greenhouse effect".
You are talking about "nuclear winter" and I clearly remember that Carl Sagan was ATTACKED because "everybody has been saying for years that pollution will make the world warmer". (this is of course because particulates have a cooling effect, as opposed to the long-term accumulation of CO2)
I also quite well remember being told in the 1960's that CO2 "could make the planet like Venus due to the greenhouse effect". This was popular perception, not from scientists. I think a lot of this was due to the Mariner probe finding the surface of Venus to be far hotter than expected.
You can believe all you want but the truth is that the consensus has been toward warming for 4 decades now. Looking earlier and all I can find is worries about the natural Ice Age cycle starting again, with no indication that humans were causing or preventing it.
There were ideas to make sending email "expensive", would it be possible to apply this here? Use a calculation that is expensive to solve but where the solution is easy to test, such as factoring a large number. The biggest problem with the scheme is that a solver has to be added to the browser somehow.
A web site that allows a user to post messages would send a random number to be factored as soon as possible. The browser would then work on this in the background, before the user even decides to post something. When the user decides to post something, the form contains a hidden field that the browser fills in with the factored value (the browser would pop up a message if the calculation is not yet done when the user tries to submit the form, and offer the user the ability to wait or cancel the submission). The web site would then immediately send a new random number for the next submission.
Any bot would have to continuously solve these things and thus would not be able to post very fast. Also hopefully the fans will turn on and make lots of noise so the user might get an idea that their machine is infected.
Another idea that would not use up your computer's battery is to have a third-party service that provides a random key, but only after a long delay. This third-party service would refuse to process more than one key at a time per host, so a bot could not do many requests in parallel.
Does any of this sound at all useful or possible?
Re:Intel e1000e bug fixed?
on
Linux 2.6.27 Out
·
· Score: 3, Informative
Nobody has figured out the bug. The patch is so that the bug (whatever it is) cannot destroy the card. It does this by setting the hardware so it ignores any attempt to overwrite it.
This should be pretty obvious from the comment you quoted.
As far as I can tell, the hardware "works". If that card did not work then probably people would not notice this bug, because they would not see the hardware fail! In fact it is strongly suspected the bug is not in the driver but in some other part of Linux.
Re:Kernel changes I wanted in as of v1.0
on
Linux 2.6.27 Out
·
· Score: 2, Interesting
I agreed at one point but I think Linux may have a better scheme possible now, and it would be nice for them to persue it.
This would be to make the "stable binary api" be only for user-space drivers. If you want a stable api then you have to write a user-space driver. In-kernel drivers would remain as they are, with a binary api that changes, and could even be changed to require GPL code.
An awful lot of the problem drivers (web cams, sound, scanners, printers, cameras, even wireless) do not need the speed of being in the kernel. Things that really need speed (disks and wired networks) actually work pretty good in Linux today. (graphics are a whole different story but I think before Linux figures out that mess, graphics will end up being so integrated into the cpu that a "graphics driver" will make as much sense as a "floating point co-processor driver", ie it just won't exist)
This will require Linux to fix and promote the user space driver api. And there certainly is no guarantee that will happen. But I feel this is the correct approach today if a "stable binary api" is wanted.
Things that provide a filesystem-like api would be the easiest (use FUSE) but this covers a lot of random devices. Things that require a lot of fast and synchronous communication (ie wireless) are harder. But this can be done in steps, so a bad API is not locked down prematurely.
The hardware manufacturers test their hardware with Windows. Whatever Windows does (whether it is correct or not), if the motherboard does not work, they will fix the motherboard. This means that whatever Windows uses gets fixed. But stuff that is not used by Windows (ie various ACPI apis or arguments to the api) is untested and thus it is a total crap shoot whether it works or not or matches the spec.
Basically if Windows uses interfaces A and D of A,B,C,D of ACPI then A & D will work, and B & C will be a random api that varies depending on manufacturer.
Linux has to reverse engineer "what parts did Windows really use". They actually have to figure out that B & C cannot be used because Windows did not call them.
Windows may very well be obeying the spec and not using "undocumented" apis. But that does not mean that there is no problem making Linux work. There is a major reverse-engineering effort to find out exactly what subset is used and thus works.
THe most obvious proof is the problem Microsoft had with Vista that you mention. Microsoft programmers decided to start using B & C and discovered that because previous versions of Windows were not using them that lots of hardware did not implement them correctly. I'll bet that despite access to their own source code, they had to do a pretty major "reverse engineering" just to get Vista to work, as the exact behavior is not that easy to figure out even with the source code.
If you want to compute the average word length, you need to locate the spaces between the words. Since this requires scanning the text anyway, you can decode the UTF-8 at the same time.
I want an explanation why "number of characters" has to be O(1) (constant time) rather than O(n) (takes time relative to the size of the string). Your example fails this test.
The file system identifies files with an array of bytes (or 16-bit words on NTFS). There is no "encoding", that is a function of the I/O routines that draw this array of bytes on the screen to present it to the user. The fact is that no matter what you do, it is possible to put an invalid "encoding" into that array on that disk. We MUST be able to specify that invalid encoding filename in the api (at least so that it can be deleted).
The only portable way to do os.listdir is to return arrays of bytes. On NTFS/Windows it should convert from UTF-16 to UTF-8. Fortunatly invalid UTF-16 can be losslessly translated to UTF-8 and then back (as long as unmatched surrogate pair halves are preserved, which is easy to do). So this will be able to present and handle invalid filenames on both Unix and Windows.
I am very afraid they will f**k it up and make os.listdir return "unicode" (which will really be on Windows whatever array of words the file is identified with, whether it is valid UTF-16 or not). They will then write some monstrous mess on Unix because they will quickly discover that outside the perfect academic world, invalid encodings do end up naming the files. Most likely they will have to put escape sequences of some sort into their UTF-16 to quote the exact bytes that were in the UTF-8.
I think the byte string should contain the literal bytes that are in the input source file, so in effect the string is in the encoding of the source file. However in most of my posts I am assuming the encoding is UTF-8.
I agree that a u"" string should convert from the encoding the compiler is using to read the file to UTF-16 (or apparently UTF-32 on Linux). However I am very worried that doing this by default (rather than the previous requirement to put a u in front) is going to cause a lot of grief, because it is no longer a 1:1 mapping. Strings are used for things other than text and this is going to cause trouble. I also very much dislike the fact that this does not force people to make their input files be UTF-8, in fact it sounds like Python may not even default to UTF-8, because of incompatability. This is very very bad IMHO.
Slashdot Mirror
It sounds like this is designed to be compatible with the application api to X, so that is not a problem with it.
I think a great deal of even the Xlib api could be jettisoned because no modern program uses it. A simple thing to do would be to pretend to be 32-bit full color all the time, no matter what the hardware is, and eliminate all the colormap api.
I believe the functions that are being put in the kernel is the *setup* so that the user-space code in the X server has that "access to video memory and registers". That is actually a huge messy part of the current X server. Doing this in the kernel allows other programs to have an idea what X did, and also allows things other than X to get the same access to video memory and registers, and also means that a "video driver" is findable in the same lists as all your other drivers.
They did kill the Windows version of Shake. They did not kill the Linux version for a few reasons:
1. Heavily used in render farms so it would kill their market if they did not support it (though that does not explain why they kept the GUI working).
2. They see Linux not as a competitor but more as an ally in fighting Windows on the desktop.
3. Worry about negative publicity.
A bigger batter will last longer no matter what the operating system is doing.
Note that this hardly seems to be a techincal issue. It appears to be entirely caused by bad communication and excessive complexity. Anyway here is what you do. There are some indications that Ibex fixed this but these steps fixed my Hardy version:
/etc/laptop-mode/laptop-mode.conf and change correct line to
/etc/default/acpi-support and change correct line to read:
/etc/acpi/power.sh
/etc/pm/power.d/laptop-tools and make it read "exit 0" and
/etc/pm/sleep.d/10laptop_mode_restart and make it contain
/etc/init.d/laptop-mode stop
;;
/etc/init.d/laptop-mode stop
;;
/etc/init.d/laptop-mode start
;;
/etc/init.d/laptop-mode start
;;
;;
/dev/sda | grep "Adv"
1. Edit
read: CONTROL_HD_POWERMGMT=1
(this makes laptop_mode call hdparm)
2. Edit
ENABLE_LAPTOP_MODE=true
(this makes power.sh run)
3. Edit
Comment out or delete the 4 for...done loops containing $HDPARM
commands.
(this stops power-on from messing with the disks)
And change the arguments to $LAPTOP_MODE from start/stop
to "auto" in both cases.
(this makes it run the laptop_mode command correctly rather than
forcing the mode on and off)
4. Create
then "chmod +x" it.
(this stops suspend/resume from messing with hdparm settings)
5. Create
the following:
<pre>
#!/bin/bash
case $1 in
hibernate)
suspend)
thaw)
resume)
*)
echo Something is not right.
esac
</pre>
Chmod +x this file.
(this makes suspend/resume run the laptop tools)
HOW TO TEST:
This command will tell you how your disk is set:
sudo hdparm -I
The correct results to stop disk thrashing are 254 or 255.
When laptop_mode is *really* on then the correct value is 1.
If you see 128 then things are not working, this is the setting
the disk resets to on suspend/sleep/power off.
I think it is just going to look in the contents of the controls. This would be really useful, for instance if you search for "Widget Model XJ123" it will now find a page by a manufacturer where the only place they list it is in a pulldown list that lets you choose the product to buy.
The idea is that you don't have to count *all* the paper ballots. Only a small subset until you are confident that the electronic total has not been tampered with.
You are right about gaming in practice. If somebody wants to tamper with the software and has any way to predict enough machines that will not be subject to hand counts then they can fix the election. This could be done by messing with the software that "randomly" chooses what districts to recount.
Unfortunately your analysis is completely backwards.
In the last two presidential elections the popular vote was clearly in favor of one candidate (Gore in the first one and Bush in the second one). A popular vote would have to be far closer than either of these to be contested. The electoral collage was why a small area with a close tie was amplified so that it made a difference.
Your idea of having a district per vote would help. An easier idea would be to have the electoral votes for each state divided fractionally by the vote in the state (this would preserve the greater weight for a voter in a small state). Unfortunatly the constitution says a state can decide how to assign their electoral votes and almost all of them say "they all go to the winning candidate" because this gives them more power in getting influence from the elected politicians.
What makes you think it's foolproof. If it decided to count every 3rd McCain vote for Obama, how would you know?
Because the machine count would not match the collected paper ballots.
Only a small number of districts need a hand count to check this, as long as which districts was unknown before the election. This would make it far too risky to tamper with the software in the optical scan machine.
I have the problem that caused a huge amount of discussion here before, where the disk in the laptop repeatedly parks and then wakes a few seconds later to write something. I have used up about 1/3 of the lifetime limit on the disk for these parks. I have updated from Feisty to Gibbon and Hardy and it is still there.
I have a script to turn off the disk timeout to fix this. However I have to run it manually every time I unsleep the machine. I have followed the instructions to install it in init.d and power on/off but it appears to have no effect. Either it is not run or it's changes are undone after it is run. This is very frustrating and I see absoltely zero information about this.
I heard there was some analysis about why Windows does not have this problem, and it was determined that Windows reads the disk *continuously* until it sleeps. This led the manuafacturers to adjust the timeout very small so it is, in effect, a fast "Windows is sleeping" detector. Now it appears that Linux is using the disk far less often, but not zero, so the result is worse for the disk. It would be really cool if Linux actually stopped reading the disk at all when nothing is happening, but I realize this is difficult, so I think an acceptable solution is to replicate how Windows acts.
In any case, despite all the yakking about this earlier, I see no comments. Have they addressed this or not? Is it ever going to be fixed? Or can somebody tell me how to debug the init stuff and find out why my script has to be run manually?
This was displayed at Siggraph.
I think it is enormously clever, while at the same time enormously useless.
It was pretty obvious that they had no way to know where the paper was, so they mostly demonstrated it as an x-ray into the image that was on the screen. I do not see any reason why the paper's location could not be determined by the touch detector, however.
I would think any practical use could be done by software moving a virtual "paper". One thing this would do is allow the papers to overlap.
I have to agree that FireFox's download display is pretty unusable. I did not know about "open containing folder" but even then it would be difficult to use. What I wanted was "copy the name of the downloaded file to the clipboard". I would have settled for "open the file where it was already downloaded". I could not find any way for it to tell me where it saved the downloaded copy. In the end I had to go back to the web page and do "save linked item as", which is really stupid considering I had already downloaded the whole thing (a pdf file).
Honestly, "I just saved a file and now I don't know where I put it" is more indicative of the human operating the computer, than it is of the computer apparently lacking facilities to find the files.
And this statement perfectly demonstrates why Linux is not now, and will not be for a very long time, a true Desktop OS.
Wow, and if you had some reading comprehension you would notice that the *solutions* he proposes are all for Windows:
(google desktop, etc) will all index files in ways that you can easily retrieve them beyond that base Windows will do.
I have to agree. I certainly am not voting for that airhead, but the RNC certainly can spend $150,000 on her clothes if they think it will help them get elected. This is tiny compared to other expenses. Perhaps she won't wear every outfit while campaigning, but I'm sure they spend orders of magnitude more on unused things like commercials that they decide not to air or on setting up campaign stops that are then cancelled.
That says 1971, not "early 1980's".
And besides if you read those articles, they clearly state that this "ice age prediction" is *contrary* to prevailing opinion at that time.
I can clearly state that I was told by teachers (in very liberal Massachusetts) that CO2 would turn the planet into Venus. Whether this is true or not it is pretty obvious that the popular consensus was for warming.
Early 1980's? You are talking about "Nuclear Winter", not something caused by CO2. In fact enemies of Sagan and the Union of Concerned Scientists *attacked* Nuclear Winter by saying "everybody knows that pollution will cause the earth to get warmer".
Face it, you can make your revisionist history all you want, but I was also there. I was told for DECADES that "CO2 will make the planet like Venus".
I don't know what world you were in. I was told as a child about 100 times that "CO2 can make the planet like Venus due to a runaway greenhouse effect".
You are talking about "nuclear winter" and I clearly remember that Carl Sagan was ATTACKED because "everybody has been saying for years that pollution will make the world warmer". (this is of course because particulates have a cooling effect, as opposed to the long-term accumulation of CO2)
I also quite well remember being told in the 1960's that CO2 "could make the planet like Venus due to the greenhouse effect". This was popular perception, not from scientists. I think a lot of this was due to the Mariner probe finding the surface of Venus to be far hotter than expected.
You can believe all you want but the truth is that the consensus has been toward warming for 4 decades now. Looking earlier and all I can find is worries about the natural Ice Age cycle starting again, with no indication that humans were causing or preventing it.
There were ideas to make sending email "expensive", would it be possible to apply this here? Use a calculation that is expensive to solve but where the solution is easy to test, such as factoring a large number. The biggest problem with the scheme is that a solver has to be added to the browser somehow.
A web site that allows a user to post messages would send a random number to be factored as soon as possible. The browser would then work on this in the background, before the user even decides to post something. When the user decides to post something, the form contains a hidden field that the browser fills in with the factored value (the browser would pop up a message if the calculation is not yet done when the user tries to submit the form, and offer the user the ability to wait or cancel the submission). The web site would then immediately send a new random number for the next submission.
Any bot would have to continuously solve these things and thus would not be able to post very fast. Also hopefully the fans will turn on and make lots of noise so the user might get an idea that their machine is infected.
Another idea that would not use up your computer's battery is to have a third-party service that provides a random key, but only after a long delay. This third-party service would refuse to process more than one key at a time per host, so a bot could not do many requests in parallel.
Does any of this sound at all useful or possible?
Nobody has figured out the bug. The patch is so that the bug (whatever it is) cannot destroy the card. It does this by setting the hardware so it ignores any attempt to overwrite it.
This should be pretty obvious from the comment you quoted.
As far as I can tell, the hardware "works". If that card did not work then probably people would not notice this bug, because they would not see the hardware fail! In fact it is strongly suspected the bug is not in the driver but in some other part of Linux.
I agreed at one point but I think Linux may have a better scheme possible now, and it would be nice for them to persue it.
This would be to make the "stable binary api" be only for user-space drivers. If you want a stable api then you have to write a user-space driver. In-kernel drivers would remain as they are, with a binary api that changes, and could even be changed to require GPL code.
An awful lot of the problem drivers (web cams, sound, scanners, printers, cameras, even wireless) do not need the speed of being in the kernel. Things that really need speed (disks and wired networks) actually work pretty good in Linux today. (graphics are a whole different story but I think before Linux figures out that mess, graphics will end up being so integrated into the cpu that a "graphics driver" will make as much sense as a "floating point co-processor driver", ie it just won't exist)
This will require Linux to fix and promote the user space driver api. And there certainly is no guarantee that will happen. But I feel this is the correct approach today if a "stable binary api" is wanted.
Things that provide a filesystem-like api would be the easiest (use FUSE) but this covers a lot of random devices. Things that require a lot of fast and synchronous communication (ie wireless) are harder. But this can be done in steps, so a bad API is not locked down prematurely.
Both of you are missing the point.
The hardware manufacturers test their hardware with Windows. Whatever Windows does (whether it is correct or not), if the motherboard does not work, they will fix the motherboard. This means that whatever Windows uses gets fixed. But stuff that is not used by Windows (ie various ACPI apis or arguments to the api) is untested and thus it is a total crap shoot whether it works or not or matches the spec.
Basically if Windows uses interfaces A and D of A,B,C,D of ACPI then A & D will work, and B & C will be a random api that varies depending on manufacturer.
Linux has to reverse engineer "what parts did Windows really use". They actually have to figure out that B & C cannot be used because Windows did not call them.
Windows may very well be obeying the spec and not using "undocumented" apis. But that does not mean that there is no problem making Linux work. There is a major reverse-engineering effort to find out exactly what subset is used and thus works.
THe most obvious proof is the problem Microsoft had with Vista that you mention. Microsoft programmers decided to start using B & C and discovered that because previous versions of Windows were not using them that lots of hardware did not implement them correctly. I'll bet that despite access to their own source code, they had to do a pretty major "reverse engineering" just to get Vista to work, as the exact behavior is not that easy to figure out even with the source code.
If you want to compute the average word length, you need to locate the spaces between the words. Since this requires scanning the text anyway, you can decode the UTF-8 at the same time.
I want an explanation why "number of characters" has to be O(1) (constant time) rather than O(n) (takes time relative to the size of the string). Your example fails this test.
The file system identifies files with an array of bytes (or 16-bit words on NTFS). There is no "encoding", that is a function of the I/O routines that draw this array of bytes on the screen to present it to the user. The fact is that no matter what you do, it is possible to put an invalid "encoding" into that array on that disk. We MUST be able to specify that invalid encoding filename in the api (at least so that it can be deleted).
The only portable way to do os.listdir is to return arrays of bytes. On NTFS/Windows it should convert from UTF-16 to UTF-8. Fortunatly invalid UTF-16 can be losslessly translated to UTF-8 and then back (as long as unmatched surrogate pair halves are preserved, which is easy to do). So this will be able to present and handle invalid filenames on both Unix and Windows.
I am very afraid they will f**k it up and make os.listdir return "unicode" (which will really be on Windows whatever array of words the file is identified with, whether it is valid UTF-16 or not). They will then write some monstrous mess on Unix because they will quickly discover that outside the perfect academic world, invalid encodings do end up naming the files. Most likely they will have to put escape sequences of some sort into their UTF-16 to quote the exact bytes that were in the UTF-8.
I think the byte string should contain the literal bytes that are in the input source file, so in effect the string is in the encoding of the source file. However in most of my posts I am assuming the encoding is UTF-8.
I agree that a u"" string should convert from the encoding the compiler is using to read the file to UTF-16 (or apparently UTF-32 on Linux). However I am very worried that doing this by default (rather than the previous requirement to put a u in front) is going to cause a lot of grief, because it is no longer a 1:1 mapping. Strings are used for things other than text and this is going to cause trouble. I also very much dislike the fact that this does not force people to make their input files be UTF-8, in fact it sounds like Python may not even default to UTF-8, because of incompatability. This is very very bad IMHO.