Is Time Moving Forward Or Backward? Computers Learn To Spot the Difference

sciencehabit (1205606) writes For the first time, scientists have taught computers to figure out the direction of time in videos, a result that could help researchers better understand our own perception of time. Regardless of any possible applications, "we just thought it was a great problem," says one of the study's authors. Teaching computers to see the arrow of time combines computer science, physics, and human perception to get at the heart of the question, "How do we understand the visual world?" The researchers "broke down 180 YouTube videos into square patches of a few hundred pixels, which they further divided into four-by-four grids. Combining standard techniques for discovering objects in still photographs with motion detection algorithms, the researchers identified 4000 typical patterns of motion, or 'flow words,' across a grid’s 16 cells. ... When they tested their program on the remaining 60 videos, the trained computers could correctly determine whether a video ran forward or backward 80% of the time."

Comments

Easy

[ZombieBraintrust]

Beginning

hey guys..

End

Click Like To Subscribe. Follow Hashtag blah on twitter, blah blah reddit. Hand waves to bottom of screen.

Re:Easy

[ZahrGnosis]

Actually, if you guessed that a randomly selected set of youtube videos were being played, you know... FORWARD, you'd probably be correct more than 80% of the time without having to actually think at all. I assume their 80% result was based on something more difficult, but it's still kind of a silly sounding number without context.

Re:Easy

[ZombieBraintrust]

I assumed they took youtube vids and then played them backwards half the time. Then they asked the computer to guess forward or backward. I think it would be easier to guess based on the audio than the vidio. You could just put the audio through speach regonition. If the sound makes sense as words in some language then it is playing forwards. But good for them to look at the harder problem.

Re:Easy

[Dcnjoe60]

But the point is not to tell if sound is moving forward or backwards.

The point is to detect if a video (possibly even without sound) is moving forward or backwards.

But you could easily mess up this program by showing it a video of Michael Jackson doing the Moonwalk.

However, whether detecting if sound or video is moving backwards or not has nothing to do with detecting if time is moving backwards or not. Playing a file backwards still occurs with time moving forwards.

Re:Easy

[gladish]

http://en.wikipedia.org/wiki/B...

Re:Easy

[Gallefray]

http://en.wikipedia.org/wiki/B... [wikipedia.org]

Except the headline *doesn't* end in a question mark. It clearly finishes it with the letter `e`, ergo Betteridge's Law of Headlines doesn't apply.

Re:Easy

[michelcolman]

If the sound makes sense as words in some language then it is playing forwards.

Except for certain Prince songs.

Re:Easy

[michelcolman]

Yes, that's a different law: if a headline ends in a question mark, the author of the article got everything backwards.

Re:Easy

[Jane Q. Public]

I assume their 80% result was based on something more difficult, but it's still kind of a silly sounding number without context.

On our next episode: Scientists Use Computers To Determine Which Direction The Earth Is Turning.

Re:Easy

[david_thornley]

Consider Weird Al Yankovic's "Amish Paradise" video. There's one scene in which everything but Al seems to be moving in reverse. (In reality, it was shot with Al walking and singing backwards, then reversed.)

How'd it do on Top Secret?

[Overzeetop]

https://www.youtube.com/watch?...

(and..."Europe on 5 Quaaludes a Day," forward version: https://www.youtube.com/watch?...

I know

[nitehawk214]

I will call you yesterday and let you know.

Re:I know

[geekoid]

I was in yesterday, and you never called.

Re:I know

[nitehawk214]

How about the second Thursday of last week?

time or not

[buswolley]

This means nothing. It is not detecting time per se. It is detecting things violations like objects don't fall up, or other such experienced pattern that is the result of time.

Re:time or not

[rogoshen1]

Hi gnick, this is your boss. Step into my office please.

Re:time or not

[Your.Master]

What is the difference?

How is detecting a violation like objects falling up different from detecting time?

I can only imagine you think the real test is some different scenario?

ok

[hurfy]

If I close my eyes and say "forward" what percentage do I get right?

Re:ok

[TheCreeep]

Depends on the data set used. In this case 50%

Re:ok

[Carnildo]

More significantly, if you see left-to-right motion and say "forward", what percentage do you get right? I suspect there's a bias in videos towards left-to-right motion of subjects (or conversely, right-to-left motion of backgrounds), and I don't see anything in the paper about controlling for it.

Time isn't moving at all

[kruach aum]

Events unfold in time, but time itself doesn't move. Substitute space for time to make the absurdity clearer: "Is space moving forwards or backwards?" Space isn't moving, we move through space.

Re:Time isn't moving at all

[TheCreeep]

You're imposing an arbitrary abstraction on a complicated natural phenomenon.
Space *is* not doing anything. We observe things in the real world and we construct useful models that contain notions such as "time" and "space".

In this particular case, time is analogous to the frame index of the video. If that index is increasing, time is moving forward. If it is decreasing, time is moving backwards in this video. Everybody understands what it means so stop being pedantic.

Re:Time isn't moving at all

[sjames]

If you prefer, now moves through time. At the quantum level there is no directional preference yet at the macro level, there is a distinct preference. Why that is so is an open question.

Re:Time isn't moving at all

[kruach aum]

I'm not doing anything of the sort. I'm clarifying that the title is incoherent. If time could move, growing older would make no sense; you would age because time would move past you, rather than you move through time. And because everyone has a different age, it would mean that time moves differently past everyone. Everyone (and every thing) would have their own personal time bubble, rather than time being just a dimension of a shared world.

Re:Time isn't moving at all

[Charliemopps]

Events unfold in time, but time itself doesn't move. Substitute space for time to make the absurdity clearer: "Is space moving forwards or backwards?" Space isn't moving, we move through space.

Since you're being pedantic, so can I. Space can travel. It moves all the time. It warps and contorts, and acts very much like a "thing" Time also warps and contorts. So if you want to get all picky, the article is still correct :-p

Re:Time isn't moving at all

[kruach aum]

For space to be able to travel there needs to be some frame of reference against which it can be judged to have travelled. As the frame of reference for travel is space itself, if space could travel we wouldn't be able to tell.

Re:Time isn't moving at all

[Ultra64]

http://en.wikipedia.org/wiki/M...

Sounds like bad methology

[imsabbel]

You tube videos? And analysation on block level?

Won't the fact that the video codec has a direct timeline (with predictive frames, etc) override the rsults?

Re:Sounds like bad methology

[Minwee]

Be quiet! You'll jeopardize our funding.

Re:Sounds like bad methology

[TheCreeep]

The only reason that the codec would influence the outcome is if it would generate artifacts that are somehow informative of the direction of time. In practice, codecs aim to keep as much information as possible while reducing the required space by finding regularities in the video. If the quality is sufficiently high, there would be next to no artifacts. Only in cases where the quality is extremely low there are tell-tale signs of the direction of time. Everyone has seen damaged Xvid movies I assume.

Re:Sounds like bad methology

[mrbobjoe]

Good question about the codec, they took that into account somewhat by including a dataset that used a codec with only intra-frames (I imagine this was MJPEG).

From section 3.3:

We also filmed a small number of video clips using a camera which could record in a video codec which used only intra-frame, rather than inter-frame coding, meaning that there was no possibility of compression artefacts holding any time-direction information. This dataset comprises 13 HD videos of tennis balls being rolled along a floor and colliding with other rolling or static balls.

The algorithms tested did well on this dataset as well, 12/13 classified correctly.

Re:Sounds like bad methology

[geekoid]

No.

Maybe your opinion should be with held until you get more data then a /. summary.

Re:Sounds like bad methology

[mikael]

I'd imagine they decompress the video into it's constituent frames. That's easy to do with various Linux command line tools. Now you have to determine whether each adjacent pair of images are moving forwards or backwards in time. You can split this task up into small tiles to make use of parallel processing. Now you've got various sorts of movement; no change (eg. blue sky), upwards movement (smoke, clouds, rockets), sideways movement (cars, people), downwards movement (stuff falling, parachutists). Each of those will have it's own pattern of pixel movement and colors.

If you can understand what an object is, you can impose some sort of expectations on how it will move. You just need to look at some of those early comedy movies where the directors discovered how to play a film reel backwards. A tractor/trailer going backwards wasn't unusual, but someone lying on the ground, rolling backwards then jumping back into a standing position on the trailer was. Another one would be paratroopers receive an order to retreat, standing in a field, inflating their parachutes and jumping upwards into the back of an aircraft. So some rules are: human figures don't jump higher than 2 or 3 feet without help of a trampoline or unless they are a super hero. Smoke doesn't concentrate itself back into a small tube. Liquids don't fall upwards into the ceiling. If the system can understand those rules, it can tell when a video is being played backwards or forwards.

Re:Sounds like bad methology

[saccade.com]

+1. It seems like the results are perhaps keying off the compression artifacts introduced rather than any fundamental image data. Moreover, the compression artifacts are consistent from video to video, forming a consistent training set.

I know how we can REALLY put it to the test...

[Anonymous Coward]

Hard Mode: Test it on Back to the Future.

Re:I know how we can REALLY put it to the test...

[danbert8]

pfft, test it on any Ace Ventura movie where Jim Carrey does his own reverse taking sequence!

Try the videos slashdotters link to...

[damn_registrars]

The political videos that people like to link to from here often praise the past, and demonize the present. While they leave me wishing I could have my time back I would love to know if the comptuers could recognize time as moving forward.

(That said, as they are mostly political speeches with no significant moving object the identification could be done by speech pattern recognition)

Re:Try the videos slashdotters link to...

[david_thornley]

Praising the past and demonizing the present goes back as far as we have records. Not that they do it right nowadays.

Re:Sooo....

[clam666]

Is that the only measure of success?

...and this is why the terrorists are winning. If you're in 'murica, get some Jesus. If you aren't, stop takin' meh jobs!

Re:Lazy AI

[TheCreeep]

Can you stop the lazy commenting experiments that require zero effort? This was by no means lazy, it was a complicated computer vision task that required a lot of preparation and experimentation.

Oh really?

[ArcadeMan]

Have they tried their software on the music video Amish Paradise by "Weird Al" Yankovic?

Hitchhiker's Guide prototype

[Gestahl]

Behold, the first piece of the the trans-dimensional Hitchhiker's Guide. One of the first things it has to do is figure out is which way time runs in whatever reality it finds itself in.

Re:Sooo....

[Dishevel]

This saves us from the terrorists how?

Is that the only measure of success?

Not. It is not the only measure. We also measure success by thinking of the children. Any other questions?

Shameless Ad

[lorinc]

Seriously, the poster was presented yesterday at CVPR and ends up on /. today. There is nothing sensational about it. I'm getting sick of science turning into PR stunts all the time.

Is entropy decreasing?

[jeffasselin]

We can't even make machines that can figure out if entropy is increasing or decreasing in a video recording (something most humans can do unsciously). But we'll have human-equivalent AI is in our grasp within 10 years. And the singularity is coming within 20.

Sure.

Memento

[Baby Duck]

But would it recognize the time flow of a movie like Memento? Every sequence moves forward, but each subsequent sequence predates the previous one, with overlap. There are also flashback sequences.

Obviously time is moving "forward"

[Applehu Akbar]

That being the label we apply to the direction we observe time to be moving in. If there is some other direction time could be moving in, please demonstrate it so we could label it appropriately.

Probably cheating

[penguinoid]

Odds are it's probably cheating somehow, eg discovered which direction based on text fading/scrolling, or backwards voice, or something. Also, 80% success rate is rather poor, though I suppose some sort of things would be hard to tell (if they have little change in entropy).

Nothing new here.

[Dcnjoe60]

It would seem that before you could make a computer detect time moving backwards, you would first have to devise a way to actually make time move backwards. Running a video in reverse still occurs with time moving forward. That's nothing new. VCRs did this with a little LED. DVDs do it with an on-screen display.

Now, if they found a way to actually make time move backwards, that would be something.

Chinese retirees walking backwards

[magarity]

To really confuse it, point it out the window of an apartment complex anywhere in China. At any given time at least one retiree is walking around the grounds backwards as a form of exercise and or coordination boost (I haven't figured out which yet).

Abrupt transitions in optical flow?

[RandCraw]

Dr Freeman spoke about this work at CVPR this week. In the videos I saw he identified small markers of temporal transition as indicative of moving forward or backward. Those they labeled as backward appeared to recognize asymmetric movement -- as in gradual acceleration followed by sudden deceleration as uniquely forward flow (as when a hand swings down and strickes a table top) -- an asymmetry that cannot occur in reverse (as in sudden acceleration followed by gradual deceleration).

Dr Freeman did not propose this as the causal phenomenon in question, but that made the most sense to me in light of the motions he identified as evidence for backward motion.

Re:Abrupt transitions in optical flow?

[RandCraw]

I suspect most events like you describe probably would occur too quickly for conventional cameras to capture, but I see your point. It seems to me that kind of mition would have to take the form of a percussive force that arises without visible warning -- like the launch of an explosive powered bullet, and unlike the launch of a golfball being struck by a moving golf club that rapidly approached the stationary ball before making contact. And as you suggest, I doubt the firing of a bullet is the kind of motion seen most often in videos.

I suspect the vast majority of conventional motion sequences follow a path of 1) slow accel followed by slow decel (providing little clue as to directon of time), or 2) slow accel followed by fast decel (something that occurs often in forward moving time sequences, as a moving object is stopped suddenly by an impact). Thus path #2 is probably frequent enough and visible enough to be the anomaly that lets Freeman's group recognize the backward passage of time.

Forward and Backward are just definitions

[YoungManKlaus]

it only depends on the way you look at things