Slashdot Mirror
Zap2It Labs Discontinuing Free TV Guide Service
QuijiboIsAWord writes "Zap2It Labs, which provides free TV listing data for personal use, has long been the main source of program guide information for users in the US and beyond. They've announced via their webpage that,
due to abuse of the service, data will no longer be available after September 1st. There is no other direct source, and no option to pay for the service even if the users wanted to. Without a data feed of this type, users will be reduced to scraping websites at best. Is this going to be a killing blow for MythTV?"
I don't use MythTV, and so I was surprised to see that it relies on a private third-party source for TV listings. Isn't there any way to obtain this information in an "open-source" manner?
We all know what to do, but we don't know how to get re-elected once we have done it
Their website does not explain. Is just using the data in MythTV, "abuse"?
... Without a data feed of this type, users will be reduced to scraping websites at best. Is this going to be a killing blow for MythTV?" I fail to see how, unless the screenscrapers specifically mentioned don't work. I wrote a scraper for my MythTV box to pull PPV information from somewhere, since it wasn't coming from Zap2it. It's not that hard, really, and took me all of an hour to do.I first became aware of MythTV some years ago from a developer that was excitedly working on the project. With all the effort that was going on at the time, nobody seemed to have a clear-cut idea of a long-term, stable way of getting TV listings. "Scraping web pages" was the only plan.
Looks like five years later, it's still the only plan.
Best Windows Freeware
I use a Windows based PVR that works with the Web site titantv.com Clicking on the 'record' button on the Web site sends a small file that a helper app converts to a Windows scheduler entry. I can't believe it would be that hard to adapt to Linux.
The Titan TV web site includes advertising and also does tracking. While personally, I don't care if anyone knows I watch StarGate and Myth Busters, privacy issues may be a concern for some.
[Insert pithy quote here]
I've been a Zap2it subscriber for at least three years for my MythTV.
At first, they made me fill out a big online survey as "payment" for the service. The first time it was about 30 questions.
The third time (this is like every 3-6 months) they only asked one question.
For the last year, the survey has been "click here to renew."
What's with that? I'm willing to give up some personal time and info to pay for this service, and they can't even think of a way to leverage that?
It doesn't appear that this will affect their main listings on their website, just the programs that tap into their database.
I've always just setup an easy shortcut to their web page to get listings, so I really don't think this will affect my usage in the least.
This will be the killing blow for MythTV and other open source DVRs? I think not. MythTV predates Zap2It and managed to do okay. Yes, it relied on screen scraping, but it worked. Furthermore, I know I'm perfectly prepared to pay a small monthly fee for a good data source. Maybe $5 a month? Since a company offering such a service doesn't need to recoup costs for selling hardware below cost (as Tivo does), such a price should be feasible. Since Zap2It was free, there wasn't much incentive for someone to offer the service, but now there is. I'm hoping the free market will see the opportunity and we can work something else.
Search 2010 Gen Con events
What's wrong with just "scraping" web pages (I assume that means writing code which automatically downloads the webpages and captures the data of interest, rather than requiring a human to do it. I do this all the time with Perl code.)?
There's multiple sites out there with TV listings: Yahoo TV, Zap2It, MeeTV (the one I use), etc. Just write perl scripts to capture the listing information from these sites, and modify MythTV to allow the user to choose any service he wishes. Of course, some of these sites may (stupidly) screw with their HTML in order to throw off these scripts, but that's easily worked around with regular updates. So we just need to have a "myth-scripts" distribution site where your Myth box automatically checks for updates to the perl scripts every day and downloads them if necessary, just like we already do with many other things.
No, it's not quite as reliable and efficient as a static interface to this data, but if these companies are stupid enough to remove static interfaces, thinking we're just going to go back to doing everything manually and looking at all the ads, this seems like a reasonable solution. There's no way of preventing automated scripts from downloading webpages.
http://www.gossamer-threads.com/lists/mythtv/users /275533#275533
And it isn't just MythTV that uses the guide data provided...
When the source is open, the possibilities are endless.
Tribune put new management in Zap2it. They have been unresponsiveand treated the Data Direct service like a redheaded stepchild for a year now. The data has been bad, with long outages on it from time to time for a while now. Many of us that have used myth and other xmltv systems have tried to pay for a subscription for a couple of years now and they refuse.
it's the new management, they hate that OSS people are getting access to the data and want to stop it.
Do not look at laser with remaining good eye.
Shouldn't Google or Yahoo or Microsoft or any of the other big media / tech companies of the 21st century provide this information as a free WebService? Seems like something Google would jump on top of since their mission statement is to organize the worlds information. Well, TV listings is information.... get on that Google!
MythTV can read the broadcast schedules on the airwaves - see EIT. At least that's what I use in the UK. I can also still scrape the Radio Times website in XMLtv as well.
Not having to pay a monthly fee to automatically record shows you watched previously
Not having to deal with commercials inserted when you fast forward 30 seconds.
Now, if they could only find a way to strip thos FREAKIN' RUDE graphics that certain vile channels overlay on top of the movie/show. I can deal with a tiny little station identification, but those rude obnoxious ads obscure vital parts of the show about one out of every 3 times. I swear my Blood pressure goes through the roof.excitingthingstodo.blogspot.com
Isn't there any way to obtain this information in an "open-source" manner?
The best way to get the information is from the stations and cable operators.
Unfortunately, MythTV and other PVR users are in the game of cutting out ads; TV programming is purely to sell ad space, and always has been, save when programs were entirely paid for by one company and the show was branded in their name. What motivation do TV stations have to assist people who are purposefully going out of their way to cut out the ads?
Please help metamoderate.
What if I was to write a web service that exposed the data garnered from website-scraping? You could just write a standard XML request, wrap it in SOAP tags and send it to the web service, and you'd be returned whatever information you requested- by channel, time, or show name...
Any takers?
Now there's just the question of who? Who is expert at spidering the web? Who likes to provide new cheap-to-free services in their quest to take over new markets. Who would love to put yet another spike into Microsoft's side by removing yet another possible revenue source for them? Who doesn't have to worry about financing such a small, cheap service alongside their already multitudes of underutilized servers and bandwidth?
Google?
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
For those folks with DirecTV, that's not an option. In fact, at this point, we have no options at all other than writing a guide scraper for TVGuide.com. Fortunately, their website data is fetched in Javascript using xmlHttpRequest, so it is probably straight XML in some dialect that could be converted into the same format as Zap2It uses with very little effort. If done carefully (request the entire schedule exactly once, then only refetch the current day each day and fetch any new days added to the schedule), it might not add enough server hits for them to even care.
Check out my sci-fi/humor trilogy at PatriotsBooks.
"MYTH TV offers everything TIVO does"
With a TiVo, you can plug in three cables and press power, at which point you are done but for the watching. You're claiming MYTH can do this too?
So no, it does NOT offer everything TiVo does.
That's a good thought, but I think you miss the mark. Data Direct was created as a response to the various screen scrapers that existed before it - scrapers which had to pull down and parse entire HTML documents instead of simply using compact, per-user xml feeds. This service was intended to reduce total bandwidth use, and as far as I'm aware it succeeded at doing so.
It's possible that there has been a shift in management and that this history lesson was forgotten, but if their intention is to save bandwidth it seems doubtful that this is a good way to do so.
You forgot being able to watch a recording on your laptop while on the toilet. I'm puzzled as to why MythTV doesn't advertise this feature a bit more, since it's one of my favorites.
That being said, you can't honestly suggest that MythTV is always a better choice than Tivo, unless you've conveniently forgotten about the teeming masses of people who couldn't install a capture card if their life depended on it. Ease is the killer feature for Tivo and bundled PVRs.
It's nothing but crumpled porno and Ayn Rand.
I think the solution is to pressure TV stations to provide listings in something like an RSS feed. Then all you need is a database of RSS feed URL's. Any idea's on how to sell TV stations on how this will benefit them?
I was happy to find your clear, concise, comment down here all by itself. It makes it easy for a clean response.
Let's say that MythTV implemented your paid service plan and began charging the princely sum of $2 per month for the data.
I would give it all of 7 days before that paid for data became available for free. Someone, somewhere, would buy the data for $2 per month and load it up for others to have free of charge. It would be a daily torrent that you could pull, or a streamed RSS feed, a static layout site with a downloadable screen scraper, or any one of a dozen other ways I can think.
So now instead of a million dollar revenue stream you'd get a thousand dollar revenue stream coming from the 500 users who would actually be wiling to pay when a free source is available.
If you can answer the question of how to prevent the above scenario from happening I can put you in touch with some content providers who will pay REAL money for your idea. The kind of money that allows people to retire for life...at the age of twenty.
I've seen some of the newer LCD HDTVs sold here in the U.S. which have the ability to display program guide data, no doubt culled from available EIT data, but this is really intended to allow viewers to see what's on right now or in the immediate future without resorting to channel flipping. I bought a Sharp Aquos one generation before this feature became standard, so I missed out, but I really don't feel like I'm missing too much because of the described limitations.
Unlike printed guides, which have to limit show descriptions to one or two lines at best, digital guides can be as detailed as you wish. By providing your own RSS feed of your schedule, you could increase the information given, which should increase the number of people willing to watch your channel. Even though I have a DVR I still watch (interesting) commercials, so don't completely write off my viewership.
And here's how you can directly make money:
By providing your own guide information, you can insert sponsorship lines and charge for them.
"Tonight on Lost, sponsored by Coca-Cola, Jack and Kate have more awkward sexual tension, while Hurley tries to cheer everyone up, Sawyer acts rudely, and Sayid kills someone with his feet."
Even us DVR users will very often look at the guide information. Voila, you've just sold an ad!
It doesn't hurt to be nice.
Thank you for being the first to thank them.
It sucks that it' shutting down (I use it for GB-PVR)- but they did do it for free for years. Let's not demonize them, no matter the reason, and start looking for alternatives.
Get your Unix fortune now!
MythTV doesn't fast forward, it does commercial skip. Automatically. It hits a commercial break and just jumps past.
Tivo used to allow a really nice FF feature to skip commercials. Now they overlay advertisements on top of the advertisements you are fast forwarding through. Not to mention the advertising in the rest of the UI. If you own one, you should know what I'm talking about. If not, Google found me someone's blog with pictures
I think you are on to something. I've been keeping tabs on the NZ Mythtv forums & mailing list (am about to move there) and EPG seems an ongoing issue for them.. I've often thought what you've suggested would be their answer.
People further down in the thread suggested that TV channel EPG is right up Google's alley, and I agree... just how do you persuade them???? Hopefully there are a few Mythtv users in the Googleplex that have the sway to pull this off. Google have the funds to purchase the EPG data, AND the customer base to insert ads in the stream (however ads would not be click-able).
Google... please buy ZAP2IT LABS!!! please!!!
Every time a feed is lost and a new one is found, every time the guide feed changes, every user has to manually upgrade the software, and it's sure to happen many times in the future..
That is inefficient.
What would be ideal would be if myth boxes could connect to each other to form a peer-to-peer network and share some of their guide data with each other.
Let the authenticity of an entry update (in case of conflicting stories) be determined by the reported source of the update, a vote, and the "age" and reputation of the mythtv installations reporting the listing.
In that manner, when one feed dies, any mythtv user who can would be able to scrape or find listings data and provide it for the benefit of all mythtv users.
Also, no one mythtv node would necessarily need the full listings, they would only need to submit updates randomly based on newly acquired data, and to submit queries based on listings they're interested in.
I.E. channels the user is currently trying to lookup listings for, or channels in the channel lineup for the next few days, and searches for program names the user is interested in or wants recorded every time.
Basically, what they're saying is that they will be changing their database structure, and can't be bothered to re-code the Labs.Zap2It part. They'd rather spend the time on other things, like increasing ad revenue.
Bear in mind that TMS charges each station (Somewhere in the region of $75/day) to collect their data, and charges their commercial end-users (somewhere in the region of $500/market/month) to provide the data. Quite a lucrative trade, wouldn't you say? Add to that advertising revenue from their site, and subtract bandwidth fees, and they still make a tidy profit.
It has been theorised that the main reason labs.zap2it was set up was to save bandwidth costs after XMLTV scrapers started hitting their public site. As the data was served up in chunks in the midst of a sea of other information (Links, advertisements, commentary, navigation etc) it cost considerably more in bandwidth, so it made more fiscal sense to offer the data for free, so they could track who was using it and where, along with only having to serve the data itself, with compression and selectivity. This is only a theory, mind, but considering how the Myth community (On it's own, without any of the other projects that were/are using Zap2It feeds) has grown, I think removing this option will drastically increase Zap2It's bandwidth bills without adding any ad revenue back into the pot (A scraper doesn't see ads, and doesn't care about them).
In other words, this could be a costly mistake for TMS. Here's hoping they see sense, and work out a way to work with MythTV and others.