Using Computers for Sophisticated Music Analysis

Tom Avril writes "Need an accompaniment for your melody? Seeking a virtual dancer to try out your new choreography? Or perhaps you're making a new TV commercial, and you need a snippet of music that sounds something like Radiohead, but a bit more mellow. Increasingly, sophisticated software can help with these sorts of tasks. We got a look at the latest from the nascent field of Music Information Retrieval, after its conference in Philadelphia: 'A key part of the conference each year is the announcement of results from a sort of software shoot-out — a competition in which various universities pit their music-analysis algorithms against one another. Entrants from more than a dozen countries competed in 18 tasks, using their computers to "listen" to selections of music, then identify such things as the genre, mood, composer or title. The eventual goal: to help people search for music they might like by combing through millions of audio files in a database. ... In another task, the computer had to identify tunes that someone hummed. "The idea is, you go into the karaoke bar and start humming, and the computer retrieves your song," Downie said.'"

Comments

Results

[Anonymous Coward]

Artist: Britney Spears
Song: Hit Me Baby
Rating: Shit
Conclusion: Humans are weak and stupid

Action Plan: Terminate Britney Spears

The real idea...

[Anonymous Coward]

You start humming and the RIAA deducts the money from your account for your reproduction.

Re:The real idea...

[jacquesm]

I don't think the RIAA has stooped quite as low yet as to interfere with your ability to reproduce

'Fake Plastic Trees' for the masses

[digitaldc]

Or perhaps you're making a new TV commercial, and you need a snippet of music that sounds something like Radiohead, but a bit more mellow.

MORE mellow than 'Fake Plastic Trees?'
This actually may work, especially if you are selling some sort of sleeping aid or anti-anxiety medication.

Neat stuff.

[pushing-robot]

On the one hand, I'd love it if my home stereo could determine what song I was humming and start playing along.

On the other hand, my family would kill me.

Re:Neat stuff.

[truthsearch]

That reminded me of a funny Married with Children episode. Oh, Al...

Not new tech

[Ethanol-fueled]

While this technology is very neat, programs which convert sound(wav/mp3) into Midi data have existed for many years. The programs featured in the competition are the next logical step. It's simply data-mining applied to music.

Music is math, but math is not necessarily music. Much of the computerized music based on mathematics alone sounds like atonal shit. Mathematical algorithms can be great for accompaniment but are not meant to replace a human composer.

Re:Not new tech

[ahankinson]

If you think this technology is like a midi->wav converter only better, you're off by orders of magnitude.

"Simply" data mining for music is a significant problem. What data do you mine? The audio signal does not contain all of the perceptual cues we understand as humans, and so things like "rhythm" and "tempo"; i.e. the things in music that get us to dance or tap our feet to it, are hard to pinpoint and even harder to extract.

Other problems, such as the Query-by-humming problem, are further complicated by two intractable problems: 1. People can't sing well out of their head, and 2. What they do sing may or may not bear any resemblance to the actual song they're remembering.

This research uses the latest advances in signal processing, machine learning, psychoacoustics, computer vision and pattern recognition. To compare it to a midi to wave converter is like comparing a paper airplane to the space shuttle.

Re:Not new tech

[cybin]

music is not math, man. sorry.

"Music is a communicative signal
comprised of patterns whose
performance and perception are
governed by combinatorial rules, or
a sort of musical grammar" -John Sloboda

that being said, music is not language either, and the brain is not a computer.

if you don't like algorithmic music, or atonal music, don't listen to it. and certainly don't rely on a computer to tell you what art you should absorb and experience...

and, i have to say, i appreciate the reverence for human composers, but mathematics has no bearing on my music apart from the incidental involvement of things like acoustics. to say otherwise is to turn the human composer into a mere algorithm machine.

Re:Not new tech

[mcscooter]

If you've looked at many waveforms in a recording program you'd know that rhythm and tempo are pretty easy to identify. I've studied algorithmic analysis of music and tempo. Rhythm and pitch are all reasonably analyzed because they're easily identified by math (what computers are made for). It's identifying things like timbre that set the mood based off cultural experience, etc that are hard to nail down with a computer. A performer can play 1 note many different ways to set many different moods and computers have trouble analyzing why it sounds different to us because it is hard to compute a lifetime of musical experience and expectation.

Re:Not new tech

[cybin]

guess we're just gonna have to agree to disagree, man.

you are speaking in analogies to try to define something that is, in itself, abstract. i actually prefer to tell my students that music is "sound organized in time", it's a much better definition.

by the same token i could tell you that language is a specific form of music, since they share certain features, but language communicates specific semantic concepts whereas music does not (in the absence of lyrics).

the difference between art and science is the difference between imagination and reason. if you want to listen to music and think about numbers, go ahead -- but not all music is like that, nor is it intended to be.

"Reason is the enumeration of quantities already known; Imagination is the perception of the value of those quantities, both seperately and as a whole. Reason respects the differences, and Imagination the similitudes of things. Reason is to Imagination as the instrument to the agent, as the body to the spirit, as the shadow to the substance." -Percy Shelley

Math is the language of reason, music is (one) language of imagination. They are compatible, sure. But I'd like to see a computer able to see beyond quantities and tell me the nature of the imagination behind a piece of music...

Recognising tunes from a simple rendition

[HuguesT]

The tune recognition task is easier than it sounds (ha). In fact it's enough to hum the *contour* of the music, i.e. whether it simply goes up or down, for a couple of bars, ignoring the rythm even.

This way of indexing and recognising music is called the Parson Code and is quite effective.

Re:Recognising tunes from a simple rendition

[thedonger]

I'm not a Luddite. But...
Is there some value to being able to recall a song, or at least to using your brain to perform the exercise of recalling from memory? This can effectively replace our need to perform this task. (New iPhone app: Hum into it and it will ID the song!) Extend that to all such tasks, which we generally regard as getting in the way of us doing what we really need to.

Example: Calculators very effectively replaced log tables, and we are all grateful for that. But they have also replaced valuable manual math skills, effectively robbing young people of a certain amount of conceptualization we now reserve for mathematicians, if for anyone.

We may be creating technology which will gradually make us a non-contemplative people, living only in the moment. And if you live in the moment, you forget the past, allowing those in control to make you repeat it.

Re:Recognising tunes from a simple rendition

[Anonymous Coward]

One of my senior projects for college used a very similar but more detailed schema in recognizing musical patterns.

In musical terms, a step is the amount of change from note-to-note. The Parson Code is limited in which it only indicates the direction of the pitch, and not the amount. I simply took account the actual half-steps used between each pitch. Like the Parson Code, it would ignore the rhythm, and easily account for identical melodies that are in different keys.

Minuet in G would look something like this:
-7 2 2 1 2 -7 0 9 -4 2 2 2 1 -12 0 5 2 -2 -1 -2 2 1 -1 -2 -2 -1 1 2 2 -4 4 -2

It was fairly easy for me to find an exact match using that encoding, or match to a certain %, since more information is provided than using the Parson Code method.

I feel that this is not far off from how the human brain recognizes melodies as most people do not have perfect pitch, but relative pitch in which we can recognize a certain melody by the difference in pitch changes even when the melody is using a different rhythm, or is in a different key than the original.

Re:Recognising tunes from a simple rendition

[cleatsupkeep]

The iPhone app is already done, Midomi.

http://www.topiphoneresource.info/midomi-iphone-app-sing-hum-or-whistle-to-find-music/

Re:Recognising tunes from a simple rendition

[digitig]

Minuet in G would look something like this:
-7 2 2 1 2 -7 0 9 -4 2 2 2 1 -12 0 5 2 -2 -1 -2 2 1 -1 -2 -2 -1 1 2 2 -4 4 -2

Anyone in particular's Minuet in G (Bach? Mozart? Beethoven? Me?), or all of them? And I thought the key didn't matter, so that would be all minuets?

Re:Recognising tunes from a simple rendition

[Petrushka]

It appears to be the one by Bach from the Anna Magdalena notebook -- d4 g,8 a b c | d4 g, g | e' c8 d e fis | g4 g, g etc -- and the numbers indicate a pitch change measured in semitones, but not indicating rhythm, harmony, or counterpoint.

Re:Recognising tunes from a simple rendition

[jhfry]

We may be creating technology which will gradually make us a non-contemplative people, living only in the moment. And if you live in the moment, you forget the past, allowing those in control to make you repeat it.

Your quote shows that we are not... as I assume you are pretty well immersed in technology since your posting on slashdot.

I would argue that technology has actually made us a more contemplative people.

Visit a farm someday, someplace rural and 'backward', and you will find that as a whole the people there are much simpler and much less likely to question the nature of our existence.

On the flip side, some of the most profound things I have heard said in recent times have come from the mouths of some text messaging, youtubing, myspace obsessed teens. Apparently there is something gained by having all of this technology at our disposal, and much of it is in the area of deeper thought.

It is true that something is lost when we allow computers to simplify some of the old, character building, exercises in patience that were done in the past... but I believe that the losses are more than made up by a greater amount of creativity, ability, and openness.

The one thing that no amount of technology can replace is the human mind's ability question the world around us... technology has only allowed us to move past much of the mundane stuff in pursuit of higher goals.

I hope that someday mathematics would be a conceptual course like most science courses. With computers and calculators able to do the math, lets let non-mathematicians focus on understanding what math does, where it's used, and how to apply it and then give them a calculator to actually do the math. Perhaps an association of Mathmaticians could be formed that would design, test, and certify software or calculating devices as being accurate... so that an aspiring engineer could focus instead on learning the principals of engineering.

I made it through an engineering degree with a great calculator and a piss poor command of Trig and Calculus. I wish the 8 math classes I had to take had been less about solving equations and more about forming equations. As a working adult I find that what I really need to know is how to look at what I do know and use it to find what I don't. Sure I can solve the equation, but little good it does me if I don't know the equation in the first place. Had I spent more time on math application theory and less on memorizing rules and tricks to solve the problem I would be far better off now because I could let the computer solve the problem for me once I formed it.

Great Idea

[pete-classic]

Self-regulating karaoke. If the computer can't tell what the hell you're singing it's probably best for you to stay off the stage.

-Peter

unnecessary

[JustSee12]

Or perhaps you're making a new TV commercial, and you need a snippet of music that sounds something like Radiohead, but a bit more mellow.

You don't need software algorithms for that, just go download a Coldplay album. Except maybe replace "mellow" with "soulless."

To help people search for music they might like ?

[eulernet]

The problem with this technology, and what Pandora or LastFm applied, is that the programs tend to choose always the same kind of music, and it's boring.
When I listen to music, I like to have some variety, not always playing the same thing, again and again, in different forms.

I like listening to one genre, and then switch to another genre, and the programs are unable to provide that.

Re:I was about to say...

[S-100]

Last.fm may be good, but here's the Pandora summary for why it played a particular song (James Taylor's Handy Man):

the song features pop rock qualities, folk influences, a subtle use of vocal harmony, use of string ensemble, major key tonality, a vocal-centric aesthetic, a good dose of acoustic guitar pickin' (sic) a dynamic male vocalist, electric pianos, acoustic rhythm guitars, romantic lyrics...

While a sophisticated computer may be able to detect some of these characteristics, I stand by my comment.

Re:I was about to say...

[Geoff]

The Music Genome Project is definitely tracking things that (at this point) take a human to notice. Features like "great trumpet solo" or "ambiguous lyrics" are quite a bit beyond the sorts of musical features being extracted by the tools described in the papers at the conference, based on the few I looked at.

Humans are fantastic musical processors. Computers not so much. Which is what makes the problem so fascinating, I think.