Clojure and Heroku Predict Flight Delays

murphee writes "Flight delayed again? Should have asked FlightCaster, a new site using statistical analysis to predict the delay of your flight in real-time. What's even better, the services is fully buzzword compliant: it's built with Clojure, distributed with Hadoop, served with Rails, and hosted on Heroku. This interview with one of the FlightCaster developers gives the gory details on architecture, Clojure tips, and your boss a reason to let you have all the multimethods and macros you can eat. Seems like now that O'Reilly's publishing a LISP book, the Age of Parenthesus has come..."

Comments

Unnecessary

[JohnPetrucci00]

This is unnecessary because they tell you at the airport if your flight is delayed.

Re:Unnecessary

[Wuhao]

They tell you at the airport if your flight is delayed, when the airline actually posts that your flight is delayed. For various reasons, this does not necessarily happen promptly. For example, airlines may hold off announcing a delay until very close to departure, because they haven't ruled out using a different plane than what was scheduled, or because they think it'll be a close enough delay that it's worth keeping everyone at the gate ready to board, or because the information just didn't get posted. Anyone who's ever waited at the gate 10 minutes past departure next to a sign that says "ON TIME" with no plane in sight knows what I'm talking about here.

Supposedly, this software tries to analyze airport traffic across airlines to try and figure out which flights are going to get delayed by ATC. So, its aim is to predict certain delays before they happen, much less before they make it onto the airport departure and arrivals screens, or the airline websites.

How well it works, I couldn't tell you.

Re:Unnecessary

[Em Emalb]

This is unnecessary because they tell you at the airport if your flight is delayed.

Back in my day you had to WALK over to a big sign and try to track down your flight information, or, even worse, you had to talk to the person standing behind the counter. And we all know that that ain't diet Dr Pepper they're drinking in that cup on their counter, no sir, that's 100% pureed baby souls. One time, I was taking a flight to Santa Barbara and I couldn't remember the airport code...well, that's not true. I could remember it, but I'd forgotten my tri-focals, so I couldn't read the blasted sign, and I had to go to that counter to find out about my flight. Well, after standing in line for close to 3 months, I finally get to the counter and the "lady" behind the counter shoots a 4 foot flame from her anus that burned my ticket to little cinders.

Where was I? Oh yeah, I'm stoked for something like this. It beats havin' ta walk, sonny-Jim.

Re:Unnecessary

[ajs]

This is unnecessary because they tell you at the airport if your flight is delayed.

Oh dear my, no.

The airlines actually make it a strict policy to lie about delays. They don't release that information until many minutes and often hours until after they know about it. I used to work in the air traffic industry, and the data that I had access to at the time would show me delays that were scheduled by the FAA up to a couple of days in advance, but the airlines kept strict control over that information because leaking it would mean that competitors could offer to pick up passengers from delayed flights.

Re:Unnecessary

[Fnkmaster]

Last 2 times I have flown afternoon flights on Continental from Newark to Charleston, SC I have experienced delays on both legs of the trip there and back. The flights back from Charleston to Newark have been delayed by at least 2 hours both times. I fly this route regularly and it's very frequently delayed (>6 of the 12 times I've flown it in the last year). In fact, last time (2 weeks ago), my 6pm flight took off at 8pm, and the 4pm flight still hadn't taken off yet.

I checked this route in their system and it says it is 85% on-time, 11% delayed less than an hour, and 4% delayed an hour or more.

I am highly skeptical of this service, based on that initial result. I'll try it out the next few times I fly though.

Re:Unnecessary

[Anonymous Coward]

Most delays are NOT ATC based delays. The airlines simply tell you they are so you dont complain as much.

Most delayrs are the airlines fault or weather reasons. For example, ATC has no controll over a crew not prepping a plane in time for takeoff (maybe due to mechanical issues etc). ATC deals fairly with all planes in the system.

Your delayed plane probably lost its takeoff slot and had to wait to be reinserted into the system at the next available time. NOT ATC's fault at all your plane couldnt take off on time. They fit you in as best they could.

Lack of story comments so far

[loteck]

Everyone is busy Googling 75% of the terms used in the summary trying to figure out what it even says. Someone in the know care to interpret?

Re:Lack of story comments so far

[EraserMouseMan]

Exactly. This project is simply a case study on how to build a web application on top of the most obscure infrastructure imaginable. That must be why there aren't many posts yet. All the bleeding-edge early-adopter \. flame-stokers were recruited for FlightCaster and are under NDAs.

Works like a charm

[Fahrvergnuugen]

Southwest's site says my flight is delayed 45 minutes. Flightcaster says its right on time! (for those interested, its southwest flight 2978)

Re:Works like a charm

[Fahrvergnuugen]

Okay, I take it back - flightcaster is now updated and shows my flight being delayed.

Heroku link broken

[Mr. Sketch]

The link for Heroku is broken, it points to http://slashdot.org/heroku.com.
Here is a working link:
http://heroku.com/

Re:Heroku link broken

[camperdave]

Do we have to code
in seventeen syllables
using Heroku?

What's it good for?

[GameMaster]

Can someone fill me in on what this would be good for? Since it's statistics based and not, officially, affiliated with the airlines, you can't really expect to use it to arrive at the airport later than officially scheduled (I suppose you could, but there's a good chance that your, particular, flight will buck the statistical averages and will take off sooner than predicted without you). If they worked, directly, with the airlines and got the airlines to guarantee that they wont take off earlier than the statistical model predicts then I could see it being useful but that's never gonna happen because they only care about finishing as many flights per hour as possible.

Re:What's it good for?

[mcmonkey]

Absolutely nothing, huh! Say it again...

Well, no. You would not use this to decide when to go to the airport for your flight. In that case, you're not interested in likelihoods, you're interested in the specific condition of your flight.

When you would use this is in the flight booking stage. For example, your itinerary involves connecting flights. You don't to be stuck with hours to kill in the airport between flights, but you also don't want to miss the connection.

If the schedule says I'll have 30 minutes between flights, what are the odds I miss my connection?

Not only is the Service Buzzword compliant

[mandark1967]

So is the Summary!

GG K-dawg!

Re:speaking of buzzwords...

[FlyingBishop]

What exactly do you need clarified? The first sentence says exactly what it does, it predicts flight delay times. The rest talks about the tools used to do it. And though the tools are perhaps overhyped, referring to the tools isn't really a buzzword, just a statement of fact. Now, if the summary had said something along the lines of

"Flight delayed again? Shoulda asked FlightCaster, a new site leveraging the power of the cloud to provide real-time statistical analysis of flight delays. It uses a combination of cutting-edge tools to do its magic. The filesystem uses the infinitely scalable redundant Hadoop cloud storage filesystem. The software itself is written in the dynamic AI language Clojure, which is a dialect of LISP. The web front-end combines magic of the Ruby on Rails framework with the Web 2.0 grandeur of Heroku.

Now that is so laden with buzzwords I can't even understand what it says, and I wrote it. The summary is fairly good. The Buzzword-compliant bit was just poking fun at the fact that Rails and Hadoop have buzzwords thrown around them all the time.

Re:What is the "Age of Parenthesus"?

[irenaeous]

Oh! Lisp. Yeah. I get it now. You misspelled "Parenthesis". A Lisp book author of all people!

Holy schnikes

[reginaldo]

From the article: 'There are only two of us that have been working on the research side of things...'

So there are 2 guys that built this machine learning process, distributed using cascade and hadoop, and they built and distributed an app to show the results using rails and heroku?

These guys probably eat my code as a breakfast snack. Seriously, how do I become that badass?

Clojure and Heroku?

[Chris Mattern]

I caught their act in Vegas! They were absolutely amazing!

We need other predictors too

[rinoid]

I want it to predict whether the doctor will see the patient on time, early, or 10/20/30/40+ minutes late.

Re:Define "International" flights not supported

[omnichad]

Oh, come on. I'm american, and I agree with GP. If it doesn't support "International," it should at least define the perspective of what defines "International."

Really good ETA info is available, for a price.

[Animats]

They're scraping free data from the FAA web site and FlightStats, then pumping it out into an iPhone app feed.

But they're not using a really good data source. The high quality system is PASSUR RightETA. This system uses hundreds of radar receivers near airports to pick up the transponder signals from aircraft. It doesn't transmit. Any radar in the area that triggers an aircraft transponder causes the transponder to emit, and the PASSUR receivers pick that up. Using multiple receivers and time of flight calculations, the aircraft can be located very precisely. In fact, this is more accurate than single-point radar. You can buy a feed of this data, but it's not free.

One Book Per Language?

[DragonWriter]

[WIth Practical Common Lisp free & from Apress] O'Reilly might not want to bother...

Right, because if there is one book available free that addresses a language, there's no reason to bother having any more books covering that language.