The Perception of 'Random' on the iPod

Robaato writes "Stephen Levy writes in the Guardian about the perception of randomness, or the lack thereof, on an iPod set to shuffle." From the article: "My first iPod loved Steely Dan. So do I. But not as much as my iPod did.... I didn't keep track of every song that played every time I shuffled my tunes, but after a while I would keep a sharp ear out for what I came to call the LTBSD (Length of Time Before Steely Dan) Factor. The LTBSD Factor was always perplexingly short." My first iPod shuffle refused to let me delete (sigh) Weird Al's Polkamon off of the flash memory.

Comments

Bias

[Anonymous Coward]

http://en.wikipedia.org/wiki/Confirmation_bias

Re:Bias

[IntelEmployee]

Blah Blah Blah........ Clusters are a common occurrence in randomness. If it played the same song every time you would have one extreme. If it played all the songs evenly a part, you would have the other extreme. Try something in the middle? That's where you will have clusters of a few songs played a lot, and clusters of some songs not played often or even at all. Want me draw you a picture? i iii iiiiii iii i It's all pseudo random anywho, and I don't think your clock cycle has a thing for steely dan, else the whole world is in trouble!

Re:Bias

[pchan-]

This is exactly what is going on. Good call, AC.

Some years ago, I worked on an mp3 playing device (no, not Apple). Our users were quite often complaining that our random was not truly random, and seems to be clustering, favoring, disliking some thing or another. Some would swear that there was some intelligence to it, picking particular songs. I've seen the shuffle code, it's a simple array swap. I ran a numerical simulation on the output and found that the distribution of the array elements from their original position equal throughout. Further, there seemed to be no specific clustering, as the probability that any item would end up next to any of its peers was again equally distributed throughout. We had some of the customers submit their own ideas and tried them out in code. In general, we found that we never outperformed the simple array swap in terms of randomness, though most results were about the same.

The conclusion that we reached: If you have a lot of Jimmy Buffet, you're going to hear a lot of Jimmy Buffet. And on that one occassion that two Buffet songs play back to back, you're going to think to yourself "this random sucks". But it is, in fact, all in your head.

*I'm sure someone will want to bring up the seed issue. Let's just say that we had it covered.

Re:Bias

[Stooshie]

Just to expand on what pchan said, the chance of getting all the songs ordered by artist, is just the same as getting any other particular combination of songs.

Re:Bias

[bratseth]

I can't believe people really don't know this? iPods do not play (pseudo)randomly.

They have a hard disk and a memory cache. Each time they have to spin the disk to read songs not in the cache, lots of power is consumed. To increase the battery time, the iPod will always prefer to play whatever's already in the cache.

I'm sure they meant well when they did this, but the result is truly devastating for many use cases, because it will cause the iPod to always prefer what you have heard recently over anything else when you switch back and forth between different random selections. It is especially bad that this optimization is also in play when the iPod is running on net power. There should at least be an option to switch to true (pseudo)randomness, right now the only workaround I know is to blow the cache by letting it play random songs from the entire library over a very long period.

To verify that this is true, try the following experiment: Play one song from an album you haven't listened to for a while, then choose to play all the songs from that album in random order. Guess which one comes first.

Re:Bias

[Yer Mom]

Finally, iTunes doesn't make a note in its database if I've skipped a song before it finished or early on.

iTunes 7 does.

Mind, it doesn't do anything particularly useful with it, but you have Last Skipped and Skip Count available for Smart Playlists, so you could probably get some way towards what you want...

Re:And Zonk dupes himself... again...

[strider44]

I think that the slashdot editors can be forgiven for posting a link to an article on a similar topic a year and a half later...

That article is btw referenced in this one.

Old News.

[mh101]

I recall The Steve personally addressing this issue in one of his keynotes (although not with iPods, but iTunes). People thought iTunes' shuffle feature wasn't random enough. Steve assured everyone that it indeed was completely random, but then announced that iTunes had a new "Smart Shuffle" option. The description in iTunes is "Smart shuffle allows you to control how likely you are to hear multiple songs in a row by the same artist or from the same album." There's a slider with "More Likely" on one end, "Less likely" on the other end, and "Random" right in the middle. Although this feature is in iTunes, it has not yet made it onto iPods.

I personally have had it happen where my iPod is in shuffle mode and I've heard not just two songs in a row by the same artist, but a song plays and then the next song from that album follows it. And that's with a library of over 5,000 songs. Naturally it's more likely to happen on a much smaller Shuffle with a fraction of the songs.

Re:OCD

[monoqlith]

Not OCD. Sub-clinical schizophrenia. . Read all the way to the end of the article(I'm new here). If you don't want to, I'll summarize: It's more about randomness than the iPod. He eventually realizes his suspicions of programmer malfeasance are in fact an expression of his own favoritism, not the iPod's. In other words, its all in his head. So a worthwhile, interesting article, and even if he could have benefitted by experimenting himself, that wouldn't have made for a very fun read, or an interesting question to ask Steve Jobs while he had the chance.

It's because of the birthday paradox

[neomage86]

You can get the technical details here: http://en.wikipedia.org/wiki/Birthday_paradox/


The basic gist is that their are far more possible pairs than we'd intuitively imagine. For example, with 20 albums of 20 songs each, the chance of two songs in a row being from the same album is actually:
400/400 * 20/400 = 1/20
Which makes a lot of sense once you sit down and think about it, but is a lot higher than an uneducated guess.

This is the same reason that collision/timing attacks are feasible.

RTFA

[mh101]

The article actually goes into more depth than people seem to be getting from the summary. The author is not complaining about problems with the shuffle. Rather, he starts by making note of his early observations, then describing his research into music shuffling, and how we perceive patterns where there are none.

FTFA:

First of all, note what it doesn't do - it's not like mixing all the songs in the equivalent of a big bucket of lottery balls and picking out the next one. Instead, as the name implies, it shuffles the entire library so as to reorder them, just as a blackjack dealer shuffles a deck of cards. If you listen to the entire library all through, you will hear every song once and once only. What is important, then, is not whether a song is included but how evenly an artist's songs are distributed throughout the list. When I say that Steely Dan is over-represented, it means that the band's songs show up early in the run - it would be like a blackjack dealer whose first hand had aces in it.

Or

We perceive trends when there are none. Poker players invariably believe they can lock into streaks. Backgammon champions swear that dice can go hot or cold. Likewise, people think they can cosmically predict what song will come next on their shuffle. The blogger Kapgar, who claimed this power, remembers vividly the times when he predicted a song and the iPod amazingly delivered it. But there may have been a thousand times when his iPod played songs he didn't guess - non-memorable circumstances that, not surprisingly, didn't make an impression.

Check the play count

[Lumpish Scholar]

I didn't keep track of every song that played every time I shuffled my tunes....

Yeah, you did; or rather, iTunes kept track for you, as the "play count". Take a look.

(I looked at mine; it was closer to uniform than I'd perceived. There's also a "Skip Count", but it's blank for all my songs.)

Re:And Zonk dupes himself... again...

[GarfBond]

Not to mention it looks like Steven Levy essentially duped his own article for the Guardian, with the added benefit of time and history with the shuffle to make a conclusion.

Similar to radio stations

[dodongo]

I know there will be snarky +5 Funny comments underneath this (as well there should be), but this system to decrease the perceived randomness is actually really similar to the algorithm most radio stations use when programming their music.

There's a simple parameter that's set to control, to within one minute, the amount of temporal separation there must be between playing two songs from the same artist, or the same song twice. The radio algorithm is a little more complicated, since songs aren't in just one big batch like the iTunes library, but in different categories, based generally on the perceived desire of target listeners to hear a given new song, or like and identify with a given older song.

The system is built off the (once literal, now metaphorical) use of index cards: The format clocks say, e.g., at the top of the hour, play a P category song, followed by a B category song, then a G, then an A, etc. You'd have a set of rules, like "don't play the same artist within 45 minutes" or "don't play the same current song within 3 hours", and you'd take the first card in the category that fit all the rules, play it, and move the card to the back of the stack.

Basically, what Apple is doing with that slider is enabling artist separation control, which is completely one of the illusions radio stations (used to) use to convince you they had every song under the sun available to them.