Testing AI Methods With FlightGear

mikejuk writes "The open source flight simulator Flight Gear is great fun but it can also be used for serious research. Suppose you want to develop a drone that can roam the seas and spot debris so that ships can be directed to it and pick it up. It's a good idea, but how do you test your methods? The obvious way is to take to the sea and fly a drone over real debris and see what happens. It uses a lot of fuel and generates a lot of sea sickness. Why not just fly a simulated drone over a simulated sea and save the sea sickness? This is what Curtis Olson, project manager at FlightGear and he explains how to get OpenCV to use the simulator as if it was a camera."

Comments

Tacos

[SCPRedMage]

But can it simulate taco delivery drones?

Re:Tacos

[Anonymous Coward]

Right now only hard shell tacos can be delivered. We are hoping to roll out support for softshell tacos by late summer and support for tamales by Christmas.

Yawn...

[Anonymous Coward]

And this is news ? Man, where has slashdot gone ?

Re:Yawn...

[Anonymous Coward]

To mars. You should join us. We have punch.

Re:Yawn...

[Anonymous Coward]

And girls. Well, some. Speaking of punch, could you bring some cases with you? Thanks.

Re:Yawn...

[Anonymous Coward]

Also, some girls?

Drones get seasick?

[Dynedain]

The obvious way is to take to the sea and fly a drone over real debris and see what happens. It uses a lot of fuel and generates a lot of sea sickness.

If it's a drone, how is there sea sickness?

Re:Drones get seasick?

[kidgenius]

They are implying that you are controlling the drone from a boat on the sea. Thus you would get sea-sick.

Re:Drones get seasick?

[dcw3]

They are implying that you are controlling the drone from a boat on the sea. Thus you would get sea-sick.

Interesting, but that would be a bad assumption considering the ability of many drones to be controlled at great distance (from land) for many hours. And, even if they were controlled from sea, why would you want to skip part of that, making your simulation unrealistic?

Re:Drones get seasick?

[Richard_at_work]

You would still have to "seed" the search area with identifiable items so that your test is proper, and clear the search area for the "no results" outcome, rather than simply relying on whatever is drifting around when you got there.

You know, actually engineer a proper test environment...

Re:Drones get seasick?

[clolson]

There are really really expensive drones that can fly for days on end, but the folks that typically are interested in environmental issues and searching for marine debris (even NOAA.gov) have trouble affording to buy them. From a practical and economic standpoint it makes much more sense to focus on smaller, less expensive drones that operate near the ship (and there for would be launched from the ship and recovered back to the ship.) Also consider a ship that travels at most about 10 kts. If you fly more than 20 nm from the ship and spot something, that's two hours of ship time to drive over and check it out.

What debris?

[Mattygfunk1]

What are we looking for here? Remnants of an aircraft or ship? What other debris is out there?

Am I wrong in thinking that leaving a busted up craft to sink isn't the worst crime? It's not much pollution. Is this to help looking for bodies?

Re:What debris?

[DrData99]

There is a LOT of debris from the tsunami in Japan, and some of it is quite large and not sinking. http://www.torontosun.com/2012/03/26/ghost-ship-off-bc-heralds-arrival-of-tsunami-debris

Re:What debris?

[tomhath]

Japanese fishing trawlers? Search and Rescue is probably the most obvious. Pirates also come to mind.

Re:What debris?

[gman003]

Overboard shipping containers. Search-and-rescue (aka find the bodies). Security (if it can spot debris, it can spot actual ships). Oil slicks, possibly.

Also, if there's something *continually* spitting out debris or something similar, tracking it down to stop it would be important.

Re:What debris?

[Nyder]

...

Also, if there's something *continually* spitting out debris or something similar, tracking it down to stop it would be important.

That "something" is called man.

Re:What debris?

[Anonymous Coward]

Rogue dolphins bent on dominating the world... who knows...

Re:What debris?

[dcw3]

...

Also, if there's something *continually* spitting out debris or something similar, tracking it down to stop it would be important.

That "something" is called man.

What, fish don't shit in the sea?

Re:What debris?

[gman003]

HEY! That's an unfair generalization.

Women can pollute as well.

Re:What debris?

[Anonymous Coward]

Does the pope shit in the woods?

Re:What debris?

[Anonymous Coward]

That's mostly noise pollution.

Re:What debris?

[irussel]

they don't shit plastic materials like we dump into the sea.

Cool, but...

[jmDev]

Surely they can think of a better reason to research this other than flying around looking for debris.

Re:Cool, but...

[fredrated]

Surely they can think of a better reason to research this other than flying around looking for debris.

We all know the 'better' reason: to look for people to kill. However, looking for people to kill would upset some people, so we call it 'looking for debris'.

Re:Cool, but...

[Baloroth]

We all know the 'better' reason: to look for people to kill. However, looking for people to kill would upset some people, so we call it 'looking for debris'.

Exactly. "Debris". The fact that they aren't debris yet is just semantics.

Re:Cool, but...

[Anonymous Coward]

I think for that you just mount a flir on a reaper and call it a day.

Unless there are lots of al qaeda floating in the ocean using something to mask their heat signatures.

Re:Cool, but...

[clolson]

A) do you own a reaper? (a global hawk would be ok too) B) do you own a flir? C) if the answer to both A+B is yes! then do you mind dunking your whole setup in salt water and flying it 1000 miles from the nearest point of land? D) do you enjoy staring at a flir image for hours on end trying to pick out something that isn't endless water? If you answered yes, yes, yes, and yes to all of the above, then please call me. :-)

Simulation

[MisterMidi]

Let me get this right. They're using a simulated drone flying in simulated air over a simulated ocean to develop algorithms to look for simulated debris? I can see this work within the limits of the software models, but I don't think the real world cares about those limits.

Re:Simulation

[Anonymous Coward]

Boy I'll bet you will be shocked to know that commercial airplane pilots do all their training on simulators and their first real flight is with paying passengers onboard.

Re:Simulation

[Anonymous Coward]

I hate to break it to you, but to get a commercial pilot's license, you need quite a bit of time flying a plane before you can even apply. That isn't even sufficient to fly passengers, that's the minimum to accept money for flying. So we're talking skywriters and folks who fly those signs around beaches. Then, to get an ATP license, you need a commercial pilot's license and at least 1500 hours of fly time. So, while commercial airplane pilots train on simulators, their first real flight was years before they stepped in their first airliner cockpit.

Re:Simulation

[pitchingchris]

The world does not care about those limits, but at least you can test against trivial cases. You may need to tweak it in the field to fit the "real world" limits, but using the simulator should get you closer to a working model. Otherwise you are wasting money on fuel, possible crashes, and data processing. There's many steps that need to be tested before hitting the real world tests to make a complete system capable.

Re:Simulation

[Anonymous Coward]

News flash. You don't need your ATP to fly passengers in an airline. And, yes, the commercial certificate *IS* the minimum, save for a few exceptions, to fly passengers for money. Granted, you are still *heavily* restricted as to what you can do until you are on-board with a part 135 or 121 carrier. But with 150 total hours you can be damn sure I was able to legally fly with paying passengers. Note, that's for rotorcraft helicopter. Airplane Single-engine, land is 250 hours.

Re:Simulation

[clolson]

pitchingchris: exactly. No simulation is perfect, but is the simulation useful? If the simulation allows you to develop and test the bulk of your code in a comfortable environment, it may just be useful. No one wants to be in the hot seat chasing down a segfault while the ship is costing $20k per day just to idle and drift, and then you have a crew of 20-30 folks sitting around twiddling their thumbs waiting for your code to compile -- you are supposed to be flying and working. The other question to ask is how well do the simulation results translate to the real world. Here is where savvy engineering enters the picture. A savvy engineer will use the simulation as a *tool*. They will know and understand what aspects apply to the real world and what aspects don't. They will probably have already done some validation testing to help them determine how well the simulation does match up with the real world -- which parts they can trust and which parts they should ignore. You obviously wouldn't want to optimize your computer vision algorithm for FlightGear computer generated imagery, but you can see your algorithm running, you can test things like saving and loading video, communication with other hardware and software components, user interfaces -- there is a ton of stuff that you can do when you have a plausible simulation on your desk that you really don't want to be wasting your time doing at sea when you are borderline seasick and everything is 10x more difficult.

Re:Simulation

[ceoyoyo]

News flash for you - 150 and 250 are both greater than zero.

Re:Simulation

[Anonymous Coward]

Wow, your math skills are amazing. The point was that the talking BS about the ATP and commercial reqs was not even close. And the sub-parent was semi-correct. The first time a pilot flies a passenger jet it is with passengers. Prior to that they've got zero actual flight time with that aircraft. It's all sim time.

Re:Simulation

[Hognoxious]

Nothing to do with the question in hand.

Modelling something that originated inside a sim is the kind of thing a sim ought to be good at. That's an entirely different problem to perceiving something that exists in the real world outside the sim.

Excellent!

[Rostin]

I find this article very satisfying because every time I see a story about robotics research where the main objective doesn't seem to be building robots but developing algorithms, I wonder why they wasted time and money building robots. Like, for instance, that research that made rounds in popular science articles a couple of years ago about robots that evolved the ability to lie. Why bother building and programming little robots to physically carry out the task of gathering "food", when the whole thing obviously could have been simulated?

Re:Excellent!

[clolson]

Ultimately the point of robotics and AI is to accomplish some useful real world task. The question to ask is what is the best, fastest, most economical way to build a system? A UAV mishap could set a small program back by months and 10's of thousands or 100's of thousands of $$$ (or a whole lot more if you look at the flag ship military drones.) The point of the original flightgear.org article is to show an example of how it is possible to construct a simulated environment and then leverage that to accelerate the development of a complicated and tricky collection of software.

Re:Excellent!

[ceoyoyo]

Because a simulation is only as good as the person who made it. You never know whether your simulated robots are actually doing something new/useful/whatever, or just exploiting some flaw in your simulation.

Simulations are great for quick development, but at some point you need to move into the real world.

Re:Excellent!

[clolson]

Simulations are great for quick development, but at some point you need to move into the real world.

Absolutely. The ideal situation would be to develop with simulated and real world testing together, comparing and validating your simulated results against your real world results. The original article isn't advocating for 100% in-a-vacuum simulation-based development, but rather trying to show how much you can do with carefully applied simulation -- using a cool, real-world project as an example. For what it's worth, this is just one small slice of a much larger marine debris effort that involves modeling and prediction based on satellite imagery, hours of close-in imagery captured from "manned" aircraft, as well as several at-sea tests of UAV's and UAV imagery. http://highseasghost.net/

Re:Excellent!

[ceoyoyo]

Yes, this project seems like a great use of an off the shelf simulation. I was replying to the GP who wrote:

"Why bother building and programming little robots to physically carry out the task of gathering "food", when the whole thing obviously could have been simulated?"

Checking something with pure simulation is a great way of checking that your assumptions are indeed correct given... your assumptions. As you point out, if you take the opportunity to test (and improve) your simulation against the real world as you're testing your robot against the real world, you end up with a robot that works AND a simulation that is much better.

Re:Excellent!

[Rostin]

That might be true in some circumstances. For example, if the goal of the research is to create new algorithms for self-driving or piloting, then at some point, you definitely want to actually build the car or the plane and let the software drive/fly it around. It would be impossible to incorporate every situation that a robot like that might encounter in the "real world" into a simulation. On the other hand, if you are interested in robots that scoot around on a smooth 2D surface and the only part of their behavior that interests you is how their communications with one another evolve, it's much more difficult to understand what you gain by actually building them. In fact, I can think of many advantages of a pure simulation in that case: It would (probably) run much faster than in real time, which would permit investigation of a larger range of parameters and also provide more information (is the emergence of dishonesty a fluke, or does it happen often? Let's run the simulation 1,000,000 times from slightly different starting states and see). Nothing would break down. The experiment would be repeatable. It would be much easier to instrument the behavior of the robots. Etc.

My suspicion is that in many cases, these things are physically built because the researchers enjoy goofing off (and have access to money and cheap labor) and it makes for good press releases.

Re:Excellent!

[ceoyoyo]

"On the other hand, if you are interested in robots that scoot around on a smooth 2D surface and the only part of their behavior that interests you is how their communications with one another evolve, it's much more difficult to understand what you gain by actually building them."

How did you model their communications? Did you adequately model their movement? Do the assumptions you made modelling their movement affect the behaviour you're interested in? What else have you assumed without realizing it?

If you're only interested in how simulated robots behave, then go ahead and simulate them. If you want to say something that's applicable to the real world, which most people do, then you need to, at some point, make contact with that real world. I've done scientific simulations, and I've reviewed lots of other people's work involving them. The things people assume, usually without realizing it, in their simulations are incredible.

Using the MIT lying robots as an example, a simulation might easily ignore occultation, or incorrectly model the optical effect of several robots swarming together flashing their lights.

"My suspicion is that in many cases, these things are physically built because the researchers enjoy goofing off (and have access to money and cheap labor) and it makes for good press releases."

I guess you were never in grad school. You've got two to four years to finish your thesis, and your supervisor is on your ass to produce some papers. You'd LOVE to just spend the weekend writing some simulation code, type it up and submit, then go drinking, but the reviewers reject your paper because they've already seen a dozen shoddily built simulations that morning. So your supervisor tells you to get off your lazy ass, write some code to run the swarm of mainly undocumented robots the last grad student built out of spare parts lying around the lab, and see if your simulation results survive first contact with the enemy (reality).

Re:Excellent!

[Rostin]

If you're only interested in how simulated robots behave, then go ahead and simulate them.

Yes, that's my point. I'm glad that you agree.

I guess you were never in grad school.

Actually, it's because I have been that I've seen many, many instances of, "Look at the eyecatching but basically unnecessary and useless thing we did just because it looks cool!" Presentation and salesmanship is an important part of doing successful (funded) research.

Oops

[ilsaloving]

Now they need to develop an AI that can detect and incoming stampede of slashdot readers so it has time to brace for impact!

This isn't exactly new...

[Ogi_UnixNut]

Flightgear has been used for ages for testing drone software .For example, the Paparazzi drone project has interfaces for flightgear to allow you to simulate a drone flying.

I used just such a setup for testing out the software and seeing how it works before I actually get round to building a drone. I know some people use it for development as well, so unless I'm missing something, this isn't really new?

Re:This isn't exactly new...

[clolson]

"There is nothing new under the sun." :-) However, within the context of FlightGear, the ability to draw realistic seascapes with waves, swells, sunglint, seafoam, ship wakes, etc. is relatively new. This graphical capability is what enables this particular use of FlightGear -- aiding the development and testing of AI/Computer Vision software for the ultimate purpose of automatically locating ocean debris. Paparazzi is good stuff, their hardware/software seems to often do well at UAV competitions. :-)

Re:This isn't exactly new...

[Ogi_UnixNut]

Aaaaah, I see, that makes more sense, thanks for the explanation! :-)

And yeah I am very impressed with Paparazzi, I am actually surprised it doesn't get as much mention as other (arduino-based) drone systems. I was looking at developing a simple loitering drone with cameras, at the request of some companies and individuals who are interested in securing their compounds a bit better (getting a "birds-eye" view), and Paparazzi seems to be a good match for it. Now I just have to get off my arse and develop the hardware/flying machine, which is the hard part tbh.

Turns out getting a drone flying is easy, getting it to take stable pictures on the other hand, is a hell of a lot harder :-) I guess in this case the bigger the drone the less it will be affected by wind changes and turbulence

Mod Parent Up

[getto man d]

One of the best comments on this article so far. Algorithmic development and optimization is not a trivial subset of problems in active robotics research.

On this episode of FlightGear...

[milbournosphere]

James wears a hat, Jeremy plays "a nice game of chess", and the Stig flies a drone! *cue intro music*

Only terrorists..

[nurb432]

Well, you know the drill... How long before someone like the HSD goes out and tries to ban simulation software that has 'performance' beyond a certain level.

Much as they did with encryption decades ago, classifying them as munitions and legally limiting access to 'high quality encryption'. My old copy of windows 1.0 still has the export restriction sticker, due to encryption.

Gratz to Curt Olson, from friends at GE Healthcare

[djh101010]

Off-topic shout out, sorry. I got my first job as a "real" Unix sysadmin in 1992 or so, when Curt left GE Medical Systems to go work on FGFS and other projects. Really cool to see it's going well and he's getting some top recognition. Good on ya, Curt! (I worked for Paul O. at your old desk after you left, we've talked a few times).

next up - OpenCV Aimbots for all of your games.

[citizenr]

Computing power is finally getting there.
Just imagine a computer with a screen grabber playing the game. We could even make tourneys where AIs fight each other :o

Re:Gratz to Curt Olson, from friends at GE Healthc

[clolson]

Hey! Good to hear from you, hope all is going well.

SkyNet VS The Matrix

[DarthVain]

Head 'sploded....

OK, so you have these drones that fly in the sky (re-purposed military drone perhaps?), and now you propose to give them AI? Why not just start the machine revolution right now?

So rather than use these things in "real life", you are going to simulate debris, in a simulated ocean, on a simulated earth? Don't forget to at least put the women in red in there as well...

All I know is that we burned the sky...

Re:SkyNet VS The Matrix

[clolson]

Simulation is useful because the "Great Pacific Garbage patch" is about 1000 nm miles from the nearest point of land (draw a line straight north from HI and straight west from LA/San Fran and you'll be right about in the middle of it.) But you can't just pick a nice afternoon and run out there for a quick test flight. :-)

Re:SkyNet VS The Matrix

[Kryptonut]

You mean "Stealth"?

Re:SkyNet VS The Matrix

[DarthVain]

Does ANY drone have a range of 2000nm? I doubt it, which makes it all a bit implausible anyway. I suppose you could make a little drone aircraft carrier (heck make that automated also)... Of course then you would have to make autonomous destroyers and subs to defend it... :) Of course you would need some autonomous satellites to control the whole thing... SkyNet wasn't built in a day!

This really just sounds like cheapo software testing, where they simulate the environment rather than using real testing... Of course you are now testing two variables, your software and the simulation, and any bug now has a chance of being either one, rather than just your software. Seems likes like shortcut to me.

It does make me smile a bit about the possible failure scenarios, as this is basically navigation software with terrain identification. "Well sir, we sent it out and it never came back..." Realistically these things would all use GPS, and have flotation devices, so really this is just about the expense of playing fetch.

Re:SkyNet VS The Matrix

[DarthVain]

Are you using that as an example of a simulated movie?

If you are, it wasn't a very good simulation. I know I didn't find it believable as a movie anyway... :)

The aerospace blockset for simulink

[Anonymous Coward]

The aerospace blockset for simulink has used flight gear for years. Nothing new.

http://www.mathworks.com/help/toolbox/aeroblks/f3-19546.html

Why Not?

[Anonymous Coward]

because its licensed under the GPL and you have to release publicly any code that you add.

Change it to LGPL and it will be used.

Sorry to be a Grammar Nazi...

[Anonymous Coward]

Hate to be a Grammar Nazi, but reading the summary and seeing sentences like this always trip me up.

"This is what Curtis Olson, project manager at FlightGear and he explains how to get OpenCV to use the simulator as if it was a camera."

This is what Curtis Olson.....

This is what he does?
This is what he is?
This is what...what!?
What exactly is this supposed to mean?

Nothing to see here...

[Anonymous Coward]

People in industry have been doing this with flight sim and xplane for some time now. If you have a simulator with a plugin architecture, this kind of thing is easily done.

Shameless plug

[Trogre]

FlightGear is a wonderful flight simulator. I have been tracking its progress for the past eight or so years and it's great to see how it has progressed.

If there's something that it doesn't do well, you can always make it better yourself.