Slashdot Mirror
Zap2It Labs Discontinuing Free TV Guide Service
QuijiboIsAWord writes "Zap2It Labs, which provides free TV listing data for personal use, has long been the main source of program guide information for users in the US and beyond. They've announced via their webpage that,
due to abuse of the service, data will no longer be available after September 1st. There is no other direct source, and no option to pay for the service even if the users wanted to. Without a data feed of this type, users will be reduced to scraping websites at best. Is this going to be a killing blow for MythTV?"
I don't use MythTV, and so I was surprised to see that it relies on a private third-party source for TV listings. Isn't there any way to obtain this information in an "open-source" manner?
We all know what to do, but we don't know how to get re-elected once we have done it
Their website does not explain. Is just using the data in MythTV, "abuse"?
I first became aware of MythTV some years ago from a developer that was excitedly working on the project. With all the effort that was going on at the time, nobody seemed to have a clear-cut idea of a long-term, stable way of getting TV listings. "Scraping web pages" was the only plan.
Looks like five years later, it's still the only plan.
Best Windows Freeware
I've been a Zap2it subscriber for at least three years for my MythTV.
At first, they made me fill out a big online survey as "payment" for the service. The first time it was about 30 questions.
The third time (this is like every 3-6 months) they only asked one question.
For the last year, the survey has been "click here to renew."
What's with that? I'm willing to give up some personal time and info to pay for this service, and they can't even think of a way to leverage that?
What's wrong with just "scraping" web pages (I assume that means writing code which automatically downloads the webpages and captures the data of interest, rather than requiring a human to do it. I do this all the time with Perl code.)?
There's multiple sites out there with TV listings: Yahoo TV, Zap2It, MeeTV (the one I use), etc. Just write perl scripts to capture the listing information from these sites, and modify MythTV to allow the user to choose any service he wishes. Of course, some of these sites may (stupidly) screw with their HTML in order to throw off these scripts, but that's easily worked around with regular updates. So we just need to have a "myth-scripts" distribution site where your Myth box automatically checks for updates to the perl scripts every day and downloads them if necessary, just like we already do with many other things.
No, it's not quite as reliable and efficient as a static interface to this data, but if these companies are stupid enough to remove static interfaces, thinking we're just going to go back to doing everything manually and looking at all the ads, this seems like a reasonable solution. There's no way of preventing automated scripts from downloading webpages.
Tribune put new management in Zap2it. They have been unresponsiveand treated the Data Direct service like a redheaded stepchild for a year now. The data has been bad, with long outages on it from time to time for a while now. Many of us that have used myth and other xmltv systems have tried to pay for a subscription for a couple of years now and they refuse.
it's the new management, they hate that OSS people are getting access to the data and want to stop it.
Do not look at laser with remaining good eye.
What if I was to write a web service that exposed the data garnered from website-scraping? You could just write a standard XML request, wrap it in SOAP tags and send it to the web service, and you'd be returned whatever information you requested- by channel, time, or show name...
Any takers?
Without a DVR, there is zero chance that I'll see any show that doesn't air between 7-10PM on weekdays. For that matter, the odds of me seeing shows that ARE between 7-10PM is slim, since I'm often doing other stuff too. From their POV, they'd probably be better off with me seeing bits of pieces of commercials at FF speed than with me seeing nothing at all.
Slow down, cowboy! It has been 4 hours since you last posted. You must wait another few hours.
I think the solution is to pressure TV stations to provide listings in something like an RSS feed. Then all you need is a database of RSS feed URL's. Any idea's on how to sell TV stations on how this will benefit them?
I was happy to find your clear, concise, comment down here all by itself. It makes it easy for a clean response.
Let's say that MythTV implemented your paid service plan and began charging the princely sum of $2 per month for the data.
I would give it all of 7 days before that paid for data became available for free. Someone, somewhere, would buy the data for $2 per month and load it up for others to have free of charge. It would be a daily torrent that you could pull, or a streamed RSS feed, a static layout site with a downloadable screen scraper, or any one of a dozen other ways I can think.
So now instead of a million dollar revenue stream you'd get a thousand dollar revenue stream coming from the 500 users who would actually be wiling to pay when a free source is available.
If you can answer the question of how to prevent the above scenario from happening I can put you in touch with some content providers who will pay REAL money for your idea. The kind of money that allows people to retire for life...at the age of twenty.
MythTV doesn't fast forward, it does commercial skip. Automatically. It hits a commercial break and just jumps past.
Tivo used to allow a really nice FF feature to skip commercials. Now they overlay advertisements on top of the advertisements you are fast forwarding through. Not to mention the advertising in the rest of the UI. If you own one, you should know what I'm talking about. If not, Google found me someone's blog with pictures
Every time a feed is lost and a new one is found, every time the guide feed changes, every user has to manually upgrade the software, and it's sure to happen many times in the future..
That is inefficient.
What would be ideal would be if myth boxes could connect to each other to form a peer-to-peer network and share some of their guide data with each other.
Let the authenticity of an entry update (in case of conflicting stories) be determined by the reported source of the update, a vote, and the "age" and reputation of the mythtv installations reporting the listing.
In that manner, when one feed dies, any mythtv user who can would be able to scrape or find listings data and provide it for the benefit of all mythtv users.
Also, no one mythtv node would necessarily need the full listings, they would only need to submit updates randomly based on newly acquired data, and to submit queries based on listings they're interested in.
I.E. channels the user is currently trying to lookup listings for, or channels in the channel lineup for the next few days, and searches for program names the user is interested in or wants recorded every time.
Basically, what they're saying is that they will be changing their database structure, and can't be bothered to re-code the Labs.Zap2It part. They'd rather spend the time on other things, like increasing ad revenue.
Bear in mind that TMS charges each station (Somewhere in the region of $75/day) to collect their data, and charges their commercial end-users (somewhere in the region of $500/market/month) to provide the data. Quite a lucrative trade, wouldn't you say? Add to that advertising revenue from their site, and subtract bandwidth fees, and they still make a tidy profit.
It has been theorised that the main reason labs.zap2it was set up was to save bandwidth costs after XMLTV scrapers started hitting their public site. As the data was served up in chunks in the midst of a sea of other information (Links, advertisements, commentary, navigation etc) it cost considerably more in bandwidth, so it made more fiscal sense to offer the data for free, so they could track who was using it and where, along with only having to serve the data itself, with compression and selectivity. This is only a theory, mind, but considering how the Myth community (On it's own, without any of the other projects that were/are using Zap2It feeds) has grown, I think removing this option will drastically increase Zap2It's bandwidth bills without adding any ad revenue back into the pot (A scraper doesn't see ads, and doesn't care about them).
In other words, this could be a costly mistake for TMS. Here's hoping they see sense, and work out a way to work with MythTV and others.