IE is also a bastard (in previous versions, not sure about current) in how many Kbps you were getting. They showed your download total, how much time was left, and a number you thought was how many Mb/Kb/Bps you were downloading at the time... but the download speed is actually an average too! If you stuck an Iptables drop rule in the middle of your download the download speed counter would
100KB..99KB..98KB..97KB.. count down slowly over time even though your current download speed was 0 ZERO ZIP NADA. If your download was going fast at one point you could sit there 20 minutes before you realized that number of bytes downloaded wasn't increasing but you still had a download speed!?!
PROTIP: Have your download rate show what speed you are currently downloading at in the current sampling window. That's what matters now.
>but the single thing that people most want - time remaining - remains bogus
That's because 'people' don't provide a guaranteed minimum bandwidth, throughput, and IOPS. Give me a real time OS with performance metrics and a predetermined number of operations and I'll tell you how long it's going to take. Bitching at the application when your ISP is changing the bandwidth at order of magnitude levels each sampling isn't the applications fault. It isn't the applications fault that your disk is doing between 1 and 150 IOPS. It isn't the applications fault that your disk is writing at 100MBps and drops to 400KBps seemingly randomly.
Shouldn't jump back in such scenario. Progress bars display percentage of _work_ done, not of time remaining. There might be some weighting behind the scenes to make the two correspond better (e.g. assign more relative "work" to copying files than to moving files, because moving is generally instantaneous when done within the same filesystem), but my observations are that this is rarely the case (e.g. OSX Software Update divides the progress bar equally between all pending updates, regardless of their size, and each package's part is divided equally between downloading and installing progress). The bar might need to stall for some time, or update the _time_ remaining, but the percentage-of-work-already-done won't change just because suddenly the available resources are lower.
This all depends if dependency resolution occurs before or during the install.
>but surely disk write speeds could be estimated relatively accurately, right?
No.
Lets say you are writing two files, both 1 GB in size. For the first file you have 1.1GB free of contiguous space, you'll be able to write it at the streaming speed of your hard drive (100MBps or more for a decent drive). Ok, that took just a few seconds, a few seconds more and we'll be done, right? Wrong. Now the OS has to find places to put fragments of the next 1GB file. You've moved problem domains streaming throughput to random IOPS. Now you are getting around 400KBps, over 200 times slower then before! Of course if you had an mostly empty SSD, you'd have seen no difference, but wait, your SSD is filling up, now erase cycles are messing things up. On top all that your A/V could decide to fire up and use 100% of the CPU and or disk in the meantime.
So the answer is, unless your storage gives you an guaranteed minimum throughput/IOPS attempting to figure out how long something will take is very hard.
>We assume that copying one file has an overhead equivalent to copying k bytes Unfortunately your assessment doesn't take mechanical limitations, memory caches, and filesystem layout in consideration.
You attempt to address that in your previous post, but you seem to miss the point where you think bytes are equivalent.
On spinning hard disks writing contiguous files is a pretty fast operation. If the disk doesn't have other tasks occurring you commonly see ranges between 20MBps and 150MBps. This all goes to crap when you start writing and changing small files. You then are limited by IOPS of your device. Seek times become the limiting factor, writing 1MB of random data suddenly takes longer then 50MB of streaming data. Your seek time per file can very greatly depending if the file gets fragmented, or each file is written at differing locations on the hard drive. Installers "tend" to copy a bunch of larger data files first then apply configuration changes that follow the above pattern.
An excellent demonstration of this is to take a fast hard drive that has a write speed of 120MBps and IOPS of about 130. Then take an SSD with write of around 80MBps and IOPS of 10,000. The hard drive will beat the SSD in bulk copy, when the small operations occur the SSD wins hands down.
Install an HP printer for the best example of this. Breaks down the install in steps, usually 3-5, and in each step it shows the name of the current file being installed as well as the size of that file.
Too bad that HPs printer installers suck terribly. They find more reasons to fail, or just to grind and grind forever installing god knows what. I'm not really sure why their base printer drivers are over 100MB for some models. Other manufactures for models that perform similar functions will be 20-30MB.
The problem here is that the progress bar/time remaining indicators in most programs are terrible abstractions. Developers make it worse by hiding inside black boxes.
Example 1, a person says they are 25% done filling up drinks. They are behind a curtain.
Example 2, you're watching a person fill up 30 cup sized drinks, he's completed 15, you have a pretty good idea how long it's going to take. But you look at the end of the table and there are 30 thimbles that he has to fill without spilling, and you realize this will take a bit longer.
Example 3, you're installing a program, it's at 25%. 25% of what?
Example 4, you're installing a program, it's copied 2GB of 4GB of data files and processed 0 of 1000 registry keys.
1 and 3 are the same type of problem, giving a number is pretty much meaningless as the workloads are going to change. 2 and 4 give more detail and separate work types that are processed differently.
0. Show progress of search (x% of search database completed) 1. x file completed of y total 2. x number of files completed of y OR x GB of files copied of y GB 3. Not easy, though doing some simple queries first you could say 'Updating database, X possible canidates' being if X is large that this might take a while. 4. Yea, pretty much screwed here, since it's likely you're installing something from Microsoft who is the seemly worst at calculating how long something will take (WTF RU Doing.net? as you watch the hard drive grind, then stop for long periods of time, only to start grinding and stopping again and again)
It has come to pass that I sincerely prefer the hourglass (or spinning discus or beachball, or whatever) to seeing the various permutations of horror inflicted on the progress bar.
From serial progress bars that use the same bar, to progress bars that empty again (though the empty-on-uninstall is just brain-twinging, not actually wrong) to progress bars that change function halfway through, I find that I cannot stand the abuses of user interface design that some idiots perpetrate.
For crying out loud, why cannot a simple progress bar actually display some indication of !@#$ing progress for once?!
I've had too many program stick on the spinning hourglass when they were actually deadlocked (you could stick a debugger on it and see that it was in a race condition it would never break out of).
I prefer the linux bootup method of progress determination. Keep spitting out information and lots of it. If I see that the programs stopped on 'Dittling Bits 103 of 1000' for 5 minutes I'll have a good idea that something is wrong, and what it could possibly be.
Sometimes a progress bar isn't the right solution. If the amount of data is indeterminate a counter showing how many bytes have been processed, or how many records/lines/entries have been processed is even better. If the user knows he has 1 million records and the program is at 500k, AND the counter is continuing to go up, that the work may complete before the heat death of the universe.
>At half the tasks done, the bar should be at 50% > but they won't affect the amount of work that has been already completed
You assume that the program knows how much it will have to do before it completes. Many programs have to call in 3rd party libraries that install in the background of what you are doing, they don't know how long the other program will take (think.net libraries). Dependency chains suck in these cases.
Also, installing a large application is not a constant rate event, especially on spinning disk. A fair number of installers work like this. 1. Install large data files, this goes fast because disks have pretty good streaming throughput. 2. check and write ini files OR check and write registry keys. Unless you have an SSD this is slow as crap (maybe 100 IOPS if you're lucky). You could have wrote 1GB of data before then at 100MBps, then because of fragmentation and random seeks suddenly drop below 1MBps.
Windows 7 and 8 give you messages that you are running low on memory. I'm not 100 percent sure, but I think they kill the largest userspace program (though this just might be the program dying from lack of ram). Running out of (disk) space is generally the bigger problem, linux doesn't like to log you in if it cannot syslog the attempt to disk.
>Who in their right mind opens a Word doc from and unknown source
The idiot secretary in the next office over, or the next floor down.
Then the payload mines her email addresses and sends you "Minutes from meeting" or some similar crap. So now instead of having an email from an unknown person you get an email from someone you'd expect to get word documents from. Hopefully you are in a company that has decent A/V on incoming mail, most small businesses don't.
After all, what do you think 130,000 CIA employees do all day, sit around and stare at the walls?
No, most sit and do documentation. Some deal all day with bureacratic nonsense. While there is a field operations division, most of them are a bit busy on other continents to worry about some random guy leaking confidential State Department cables.
Yes, other continents like Europe, you know, where Assange was.
You seem to miss the attack that targets the encrypted authentication database.
If an attacker can grab a copy of the database and cover his tracks he has a practically unlimited amount of time to crack at all the passwords at the same time.
You are also missing that distributed GPU cracking can eat every combination of 8 chars or less in a relatively short period of time even when the hashes are well salted.
To steal the user/pass table, much of the time you only need read access. A simple SQL injection can get you all the information you need. If you have write access you can have the plaintext passwords written to a txt file on the server for later retrieval, but that doesn't give you access to existing accounts, unless you change their password. Also, the authentication systems don't always have the data you really want, they are jumping point to 'do' something with another persons account.
>Other then these scenarios, vast majority of "your password is too short and as a result not secure" is scaremongering bullshit.
>..some shitty forum that has old version of BBS software
Like the Play Station Network, or whatever. The issue is, you don't know what will happen with the user/pass database on those sites. As far as I know, battle.net has never lost the encrypted passwords and usernames. If they did, your short random password would have been outed in a day on a fast multi-gpu cracker.
That's what this article is about. If the service provider loses their shit, anything 8 char's and under containing [a-z][0-9]-!@#$%^&*()_=+ will be cracked in a few days unless they were using a hash designed to be slow. We are not talking about badly encrypted (md5+salt is bad these days though) the versions of sha-??? can be pretty quickly cracked on GPUs.
>There are far more profitable and easier methods, that actually work. It's called SQL injection. That's how they get the list of passwords in the first place. It's a shame that systems are designed that allow it to happen though.
Slashdot Mirror
IE is also a bastard (in previous versions, not sure about current) in how many Kbps you were getting. They showed your download total, how much time was left, and a number you thought was how many Mb/Kb/Bps you were downloading at the time... but the download speed is actually an average too! If you stuck an Iptables drop rule in the middle of your download the download speed counter would
100KB..99KB..98KB..97KB.. count down slowly over time even though your current download speed was 0 ZERO ZIP NADA. If your download was going fast at one point you could sit there 20 minutes before you realized that number of bytes downloaded wasn't increasing but you still had a download speed!?!
PROTIP: Have your download rate show what speed you are currently downloading at in the current sampling window. That's what matters now.
>but the single thing that people most want - time remaining - remains bogus
That's because 'people' don't provide a guaranteed minimum bandwidth, throughput, and IOPS. Give me a real time OS with performance metrics and a predetermined number of operations and I'll tell you how long it's going to take. Bitching at the application when your ISP is changing the bandwidth at order of magnitude levels each sampling isn't the applications fault. It isn't the applications fault that your disk is doing between 1 and 150 IOPS. It isn't the applications fault that your disk is writing at 100MBps and drops to 400KBps seemingly randomly.
Shouldn't jump back in such scenario. Progress bars display percentage of _work_ done, not of time remaining. There might be some weighting behind the scenes to make the two correspond better (e.g. assign more relative "work" to copying files than to moving files, because moving is generally instantaneous when done within the same filesystem), but my observations are that this is rarely the case (e.g. OSX Software Update divides the progress bar equally between all pending updates, regardless of their size, and each package's part is divided equally between downloading and installing progress). The bar might need to stall for some time, or update the _time_ remaining, but the percentage-of-work-already-done won't change just because suddenly the available resources are lower.
This all depends if dependency resolution occurs before or during the install.
So the progress is being unmade ?
Possibly, if an optional component fails do you cancel everything, or just roll back the option and finish the install?
I can predict the future...
"What Intel givith, Microsoft taketh away."
>but surely disk write speeds could be estimated relatively accurately, right?
No.
Lets say you are writing two files, both 1 GB in size. For the first file you have 1.1GB free of contiguous space, you'll be able to write it at the streaming speed of your hard drive (100MBps or more for a decent drive). Ok, that took just a few seconds, a few seconds more and we'll be done, right? Wrong. Now the OS has to find places to put fragments of the next 1GB file. You've moved problem domains streaming throughput to random IOPS. Now you are getting around 400KBps, over 200 times slower then before! Of course if you had an mostly empty SSD, you'd have seen no difference, but wait, your SSD is filling up, now erase cycles are messing things up. On top all that your A/V could decide to fire up and use 100% of the CPU and or disk in the meantime.
So the answer is, unless your storage gives you an guaranteed minimum throughput/IOPS attempting to figure out how long something will take is very hard.
>We assume that copying one file has an overhead equivalent to copying k bytes
Unfortunately your assessment doesn't take mechanical limitations, memory caches, and filesystem layout in consideration.
You attempt to address that in your previous post, but you seem to miss the point where you think bytes are equivalent.
On spinning hard disks writing contiguous files is a pretty fast operation. If the disk doesn't have other tasks occurring you commonly see ranges between 20MBps and 150MBps. This all goes to crap when you start writing and changing small files. You then are limited by IOPS of your device. Seek times become the limiting factor, writing 1MB of random data suddenly takes longer then 50MB of streaming data. Your seek time per file can very greatly depending if the file gets fragmented, or each file is written at differing locations on the hard drive. Installers "tend" to copy a bunch of larger data files first then apply configuration changes that follow the above pattern.
An excellent demonstration of this is to take a fast hard drive that has a write speed of 120MBps and IOPS of about 130. Then take an SSD with write of around 80MBps and IOPS of 10,000. The hard drive will beat the SSD in bulk copy, when the small operations occur the SSD wins hands down.
Install an HP printer for the best example of this. Breaks down the install in steps, usually 3-5, and in each step it shows the name of the current file being installed as well as the size of that file.
Too bad that HPs printer installers suck terribly. They find more reasons to fail, or just to grind and grind forever installing god knows what. I'm not really sure why their base printer drivers are over 100MB for some models. Other manufactures for models that perform similar functions will be 20-30MB.
The problem here is that the progress bar/time remaining indicators in most programs are terrible abstractions. Developers make it worse by hiding inside black boxes.
Example 1, a person says they are 25% done filling up drinks. They are behind a curtain.
Example 2, you're watching a person fill up 30 cup sized drinks, he's completed 15, you have a pretty good idea how long it's going to take. But you look at the end of the table and there are 30 thimbles that he has to fill without spilling, and you realize this will take a bit longer.
Example 3, you're installing a program, it's at 25%. 25% of what?
Example 4, you're installing a program, it's copied 2GB of 4GB of data files and processed 0 of 1000 registry keys.
1 and 3 are the same type of problem, giving a number is pretty much meaningless as the workloads are going to change.
2 and 4 give more detail and separate work types that are processed differently.
We have a winner.
That said you can do something like this.
0. Show progress of search (x% of search database completed) .net? as you watch the hard drive grind, then stop for long periods of time, only to start grinding and stopping again and again)
1. x file completed of y total
2. x number of files completed of y OR x GB of files copied of y GB
3. Not easy, though doing some simple queries first you could say 'Updating database, X possible canidates' being if X is large that this might take a while.
4. Yea, pretty much screwed here, since it's likely you're installing something from Microsoft who is the seemly worst at calculating how long something will take (WTF RU Doing
It has come to pass that I sincerely prefer the hourglass (or spinning discus or beachball, or whatever) to seeing the various permutations of horror inflicted on the progress bar.
From serial progress bars that use the same bar, to progress bars that empty again (though the empty-on-uninstall is just brain-twinging, not actually wrong) to progress bars that change function halfway through, I find that I cannot stand the abuses of user interface design that some idiots perpetrate.
For crying out loud, why cannot a simple progress bar actually display some indication of !@#$ing progress for once?!
I've had too many program stick on the spinning hourglass when they were actually deadlocked (you could stick a debugger on it and see that it was in a race condition it would never break out of).
I prefer the linux bootup method of progress determination. Keep spitting out information and lots of it. If I see that the programs stopped on 'Dittling Bits 103 of 1000' for 5 minutes I'll have a good idea that something is wrong, and what it could possibly be.
Sometimes a progress bar isn't the right solution. If the amount of data is indeterminate a counter showing how many bytes have been processed, or how many records/lines/entries have been processed is even better. If the user knows he has 1 million records and the program is at 500k, AND the counter is continuing to go up, that the work may complete before the heat death of the universe.
The Windows8/Server 2012 dialog is much better in this case.
http://encosia.com/blog/wp-content/uploads/2012/09/windows-8-file-copy-dialog.png
It draws a graph showing you the current rate, in which you can see the average over time.
>At half the tasks done, the bar should be at 50% > but they won't affect the amount of work that has been already completed
You assume that the program knows how much it will have to do before it completes. Many programs have to call in 3rd party libraries that install in the background of what you are doing, they don't know how long the other program will take (think .net libraries). Dependency chains suck in these cases.
Also, installing a large application is not a constant rate event, especially on spinning disk. A fair number of installers work like this. 1. Install large data files, this goes fast because disks have pretty good streaming throughput. 2. check and write ini files OR check and write registry keys. Unless you have an SSD this is slow as crap (maybe 100 IOPS if you're lucky). You could have wrote 1GB of data before then at 100MBps, then because of fragmentation and random seeks suddenly drop below 1MBps.
nmap -v -sS -T 5 ($isp/network)
Do you eat hamburgers every day? For breakfast, lunch, and dinner?
Hopefully you don't. A varied diet of activities is just as healthy for the mind as a varied diet of food is for the body.
>whenever I tried to pull a file remotely the connection would grind to a halt
That's bufferbloat or a poor queuing discipline.
fq_codel - http://lwn.net/Articles/504005/ would make the connection behave better.
Sectors on a hard drive are still 512B or 4096B http://en.wikipedia.org/wiki/Disk_sector
You write files in clusters, "1,301 byte file. Size on disk: 4.00 KB (4,096 bytes)" http://en.wikipedia.org/wiki/Data_cluster (or allocation units)
So, the underlying harddrive using ^2, the file system uses ^2, memory uses ^2, marketing uses something different, hmmmm.
Hard drive manufactures are just dicks and the only thing that doesn't fit nice what what they are doing to their customers.
Other storage media may store data differently and the above may not apply.
Windows 7 and 8 give you messages that you are running low on memory. I'm not 100 percent sure, but I think they kill the largest userspace program (though this just might be the program dying from lack of ram). Running out of (disk) space is generally the bigger problem, linux doesn't like to log you in if it cannot syslog the attempt to disk.
>Who in their right mind opens a Word doc from and unknown source
The idiot secretary in the next office over, or the next floor down.
Then the payload mines her email addresses and sends you "Minutes from meeting" or some similar crap. So now instead of having an email from an unknown person you get an email from someone you'd expect to get word documents from. Hopefully you are in a company that has decent A/V on incoming mail, most small businesses don't.
After all, what do you think 130,000 CIA employees do all day, sit around and stare at the walls?
No, most sit and do documentation. Some deal all day with bureacratic nonsense. While there is a field operations division, most of them are a bit busy on other continents to worry about some random guy leaking confidential State Department cables.
Yes, other continents like Europe, you know, where Assange was.
unless your phone gets snatched, then they have your email and your mobile.
You seem to miss the attack that targets the encrypted authentication database.
If an attacker can grab a copy of the database and cover his tracks he has a practically unlimited amount of time to crack at all the passwords at the same time.
You are also missing that distributed GPU cracking can eat every combination of 8 chars or less in a relatively short period of time even when the hashes are well salted.
To steal the user/pass table, much of the time you only need read access. A simple SQL injection can get you all the information you need. If you have write access you can have the plaintext passwords written to a txt file on the server for later retrieval, but that doesn't give you access to existing accounts, unless you change their password. Also, the authentication systems don't always have the data you really want, they are jumping point to 'do' something with another persons account.
>Other then these scenarios, vast majority of "your password is too short and as a result not secure" is scaremongering bullshit.
>..some shitty forum that has old version of BBS software
Like the Play Station Network, or whatever. The issue is, you don't know what will happen with the user/pass database on those sites. As far as I know, battle.net has never lost the encrypted passwords and usernames. If they did, your short random password would have been outed in a day on a fast multi-gpu cracker.
That's what this article is about. If the service provider loses their shit, anything 8 char's and under containing [a-z][0-9]-!@#$%^&*()_=+ will be cracked in a few days unless they were using a hash designed to be slow. We are not talking about badly encrypted (md5+salt is bad these days though) the versions of sha-??? can be pretty quickly cracked on GPUs.
>There are far more profitable and easier methods, that actually work.
It's called SQL injection. That's how they get the list of passwords in the first place. It's a shame that systems are designed that allow it to happen though.