Zap2It Labs Discontinuing Free TV Guide Service

QuijiboIsAWord writes "Zap2It Labs, which provides free TV listing data for personal use, has long been the main source of program guide information for users in the US and beyond. They've announced via their webpage that, due to abuse of the service, data will no longer be available after September 1st. There is no other direct source, and no option to pay for the service even if the users wanted to. Without a data feed of this type, users will be reduced to scraping websites at best. Is this going to be a killing blow for MythTV?"

Comments

This is troubling

[quanticle]

I don't use MythTV, and so I was surprised to see that it relies on a private third-party source for TV listings. Isn't there any way to obtain this information in an "open-source" manner?

Re:This is troubling

[ajs]

I don't use MythTV, and so I was surprised to see that it relies on a private third-party source for TV listings. Isn't there any way to obtain this information in an "open-source" manner? I can't imagine how that would work. Ultimately, you need information from the studios, and that's going to require a business relationship. TV Guide has such a relationship, as did these folks it seems (or perhaps TV Guide and these folks have a common feed).

I'm a little shocked that these guys didn't just go commercial, though, and build a MythTV add-on that allows you to subscribe to their product.

Re:This is troubling

[QuijiboIsAWord]

The Zap2It Labs service receives all the data from the networks, typically directly, and aggregates it all into a customizable feed which users set up by signing up for an account and selecting the cable provider or broadcast area of their choice.
Many different applications, besides just MythTV, use this data. Some people have individually negotiated with their local stations to get the data they need, but that's extremely hit or miss, and about as convenient as attempting to negotiate with your Cable company to get them to carry broadcast networks in HD unencrypted - Not very, and most people in most areas won't bother.

Re:This is troubling

[elrous0]

Not unless someone volunteers to compile the voluminous amount of information themselves, then set up a distribution method.

This was always MythTV's achilles heel, more than even HDTV. For all the talk about "Unlike Tivo, MythTV can NEVER be shut down or crippled," MythTV always had this dependency on a third party, for profit service. It's possible someone could replace them, but they're going to want SOME form of revenue (and since no one is going to tolerate ads on their MythTV, or pay for the service, this is unlikely).

Re:This is troubling

[SEAL]

and since no one is going to tolerate ads on their MythTV, or pay for the service, this is unlikely

Given the choice between:

1. paying Comcast's fees, DVR service, etc
2. paying Tivo or
3. paying for a subscription to an XML TV Listing service, and keeping my MythTV box

I'll take #3.

People will pay for it as long as no free alternatives are out there.

Re:This is troubling

[vux984]

They rely on advertising to make money and aren't going to freely help people with a device that most people are going to use to skip advertising.

They seem to forget that a substantial chunk of those viewers wouldn't be watching at ALL, if it weren't for DVRs. I like a show that's on when I'm usually playing with my kids. If I didn't have a DVR, I wouldn't watch that show period. Yes, I skip through many of the commericials during playback, but not all of them, and not if the commercial catches my eye, or is for a product I'm interested in. I even rewind to watch a commercial from the start (like if I skip into the middle of a Mac/PC ad I haven't already seen) etc.

Before I had a DVR I hit mute and/or pipped the commercials while I browsed the channel guide, or checked on the hockey game, or something. Its not like I was sitting there 'attentively watching' all the ads before.

I expect advertisers are probably losing eyeballs overall as people adopt DVRs, but its probably not nearly the issue they think it is.

Re:This is troubling

[One Louder]

That may be true, the the broadcasters don't give a crap whether or not you watch or care about their programs. The programs are only there to make you watch the ads - it's the only way they make money. The eyeballs watching ads are the only eyeballs they care about, not the ones watching shows. Programs *cost* them money.

Re:This is troubling

[Reverend528]

Isn't there any way to obtain this information in an "open-source" manner?

The best possible "open source" solution would be to create some sort of wiki for tv listing data.

Re:This is troubling

[Ucklak]

Some people might find this hard to belive but on the business side of things, the commercials are considered content and the programming is considered filler.

Re:This is troubling

[Line_Fault]

They are commercial!!

Zap2It is "Tribune Media Services, Inc", they provide "bulk" guide data to large TV services, such as Telco's(IPTV) and Satellite providers.
http://www.tms.tribune.com/products.html

Re:This is troubling

[Paradise Pete]

I'm a little shocked that these guys didn't just go commercial, though

Maybe they are. If they had instead announced that they would start charging then people would rebel and complain about them being greedy pigs. This way they just take it away, then respond to the outcry by saying "OK, OK, we'll bring it back as a small subscription service."

Re:This is troubling

[Vengeance_au]

In Australia, I've been using reliably for over 2 years (it has been around longer) the Oztivo TV Guide data, which is community created, supported and maintained. It was originally set up by a group of people who imported Tivo devices to Australia, where there is no Tivo data available. This data has been extended so there are now Mythtv and MCE scripts to allow access to the data. I've found the data to be better than some of the Aussie pay-services, purely because the guys who maintain the data are using it as well, so have a vested interest in making it as useful as possible. This service supports TV across all the states and regional centers in Australia, as well as the cable and satellite channels.

I'm guessing it's down to a need - in the US, there has not been a need for this kind of community data source, as Zap2It has been a free source. Now that it's gone, I'd be guessing there will be multiple US community based solutions springing up within weeks, and within 6-12 months most will fall away or aggrigate, leaving one or two that everyone uses.

Re:This is troubling

[NoMaster]

As the OP said, he's in .au. Here, due to some legal oddities involving telephone numbers, it is copyrighted information. There's an ongoing case between Nine (who now own HWW, the TV guide aggregators) and IceTV over IceTV's provision of a downloadable guide service for PVRs. IceTV claim that their guide is based on history, common knowledge, and the network's own advertisements (i.e. facts & public knowledge); Nine claim that it's not - and even if it was, that doesn't matter, because it's the facts, not the collation of them, that's copyrighted. And strictly speaking, under Australian law, they're right.

As an aside, one of the other networks here has announced the introduction of Tivo. Many people seem to think that this will guarantee a free, hopefully EIT-based, EPG for everybody, based on conditions in the Broadcasting Services Act. To them, I'd say "go and read the Act - particularly, focus on the differences between the requirements for commercial broadcasters and national broadcasters (i.e. the ABC & SBS)". You'll see the Act was modified last year, and doesn't require commercial broadcasters to share guide information at all...

I haven't got it on hand, and it's been a while, but I seem to remember it being Sections 2 & 3, subsection 20-something?

Re:This is troubling

[Yer Mom]

That is the first time I've ever seen "PIP" used as a verb.

Clearly, you've never used CP/M...

Anybody know what the "abuse" was?

[Palmyst]

Their website does not explain. Is just using the data in MythTV, "abuse"?

Re:Anybody know what the "abuse" was?

[ajs]

I think the "abuse" they're talking about is programs that are pre-configured to hit their service so that everyone on the Net who tries them out hammers their servers.

It seems fair to start charging, but odd that they're just shutting it off. They say they're willing to license to other companies, so presumably they're hoping someone will come along and offer a package to the MythTV folks by licensing the data and re-selling subscriptions.

Re:Anybody know what the "abuse" was?

[ChaosDiscord]

Based on their previous complaints and this message, I think the problem was people were using the free data set, then redistributing it, probably for profit, possibly indirectly (say, selling devices that used Zap2It's free service). Zap2It makes money selling their data set and they were very generously offering it for free to individuals. But you weren't allow to redistribute it.

Re:Anybody know what the "abuse" was?

[ajayrockrock]

No. For us, all you had to do was register with labs.zap2it.com and then you can get the listings for private non-commercial use. So we (mythtv users) weren't abusing the system. I imagine that the abuse they are talking about is people signing up for the service and then reselling their data (that they themselves aggregate from other sources).

--Ajay

Re:Anybody know what the "abuse" was?

[Minwee]

According to a posting on mythtv.org...

Some reasons and other comments given for the scheduled discontinuence, copied from the forum, copied from the mailing list:

• Continued use of the service to support commercial products, in violation of the agreement.
• Commercial products continued to grow despite previous appeals that this activity stop.
• There are significant changes to the supporting data structure forthcoming and we could not devote resources to the continued upkeep and enhancements of the service.
• Maintenance of the service is impacting our resource pool for other projects.
• We sought alternative options but were unable to find a solution.
• We recognize the hardship this creates for the user community. We are open to alternative solutions and would consider proposals that met the needs of the user community and our company.
• We looked into options to turn this into a paid service however we do not have the infrastructure at this time.

Re:Anybody know what the "abuse" was?

[Applekid]

While no longer linked from the front page, this link still works.

Basically, the content was provided free provided it remained for non-commercial use. After all, commercial ventures have to pay for those listings and if they could get it for free, nobody would pay.

I hope they at least tried to weed out the abusers before just cutting the cable.

Re:Anybody know what the "abuse" was?

[Lijemo]

"It seems fair to start charging, but odd that they're just shutting it off. They say they're willing to license to other companies,..."

The infrastructure required to sell to businesses/institutions and the infrastructure required to sell to individuals is completely different.

In one model, salespeople develop personal relationships with contacts at the customer locations, discuss their needs with them, put together a custom package, and then make a small number of large transactions. Also, in the B2B model you don't need to provide the front-line "my computer's coffee holder broke" level of support to end-users: the customer deals with their users first, and only passes on to you the customer support they can't solve easily. In this model you advertise in trade journals, trade shows, and relevant conferences.

In another model you are advertising via mass media, processing a very large number of much smaller transactions. You have to keep track of a lot more customer data-- and if you offer online purchasing, then you don't have the easy answer of keeping customer data secure by just not connecting it to the internet. You have to offer a much higher volume of customer service calls from a much less technical variety of enduser.

Switching from B2B to B2C (or vice-versa for that matter) is a huge expense in terms of money, time, and energy, but generally not much chance of reward.If you're good at one of these two models, then you have no reason to switch to the other; if you're bad at the one you're doing now, then you aren't very likely to survive the costs and turmoil of the switch.

Thus, as someone that works for a company that sells to institutions but not to individuals, it doesn't surprise me in the least that they aren't changing their entire business model.

Re:Anybody know what the "abuse" was?

[ChaosDiscord]

I think you're trying to suggest that Slashdot readers are, generally, hypocrites because they don't care if new technology or business models drive old ones out of business. Of course, that's utter nonsense. This isn't about Zap2It's business model being harmed by new technology or new business models. This is about Zap2It offering something to individuals for free out of simple generosity. This generosity has been abused and does threaten their business model, so they going to stop being generous. Zap2It doesn't suffer in the slightest, only those who benefited from their generosity do.

Now, if a third party started giving away television schedules without relying on Zap2It's data, harming Zap2It's business model, then we'd be on the same ground. Of course, if that happened and Zap2It whined about it, I don't think you'd see much sympathy here.

How is MythTV dead?

[PolyDwarf]

... Without a data feed of this type, users will be reduced to scraping websites at best. Is this going to be a killing blow for MythTV?" I fail to see how, unless the screenscrapers specifically mentioned don't work. I wrote a scraper for my MythTV box to pull PPV information from somewhere, since it wasn't coming from Zap2it. It's not that hard, really, and took me all of an hour to do.

Achilles Heel

[L. VeGas]

I first became aware of MythTV some years ago from a developer that was excitedly working on the project. With all the effort that was going on at the time, nobody seemed to have a clear-cut idea of a long-term, stable way of getting TV listings. "Scraping web pages" was the only plan.

Looks like five years later, it's still the only plan.

This is quite bad :(

[kalpol]

This was an invaluable service - makes me wonder who's putting the pressure on them :(

Re:This is quite bad :(

[Lumpy]

Tribune put new management in Zap2it. They have been unresponsiveand treated the Data Direct service like a redheaded stepchild for a year now. The data has been bad, with long outages on it from time to time for a while now. Many of us that have used myth and other xmltv systems have tried to pay for a subscription for a couple of years now and they refuse.

it's the new management, they hate that OSS people are getting access to the data and want to stop it.

Re:This is quite bad :(

[Etherized]

That's a good thought, but I think you miss the mark. Data Direct was created as a response to the various screen scrapers that existed before it - scrapers which had to pull down and parse entire HTML documents instead of simply using compact, per-user xml feeds. This service was intended to reduce total bandwidth use, and as far as I'm aware it succeeded at doing so.

It's possible that there has been a shift in management and that this history lesson was forgotten, but if their intention is to save bandwidth it seems doubtful that this is a good way to do so.

Re:This is quite bad :(

[The-Ixian]

You must have a lot of whiney friends. I have been a member of the MythTV mailing list for years and I have never seen anyone use that argument as a major reason why MythTV is better than Tivo.
 
The top reasons MythTV is better than Tivo (and other commercial DVRs) are: 1. Commercial flagging 2. Editing and Exporting utils 3. media center addons (mythdvd, mythvideo, mythmusic, etc).

Misuse?

[Cerberus7]

How, exactly, does one go about "misusing" TV schedule listings? Is this really because Zap2It was making all other forms of TV listings obsolete while not making any money at it? The announcement is quite vague. Does anybody have details on what's going on?

TitanTV

[rlp]

I use a Windows based PVR that works with the Web site titantv.com Clicking on the 'record' button on the Web site sends a small file that a helper app converts to a Windows scheduler entry. I can't believe it would be that hard to adapt to Linux.

The Titan TV web site includes advertising and also does tracking. While personally, I don't care if anyone knows I watch StarGate and Myth Busters, privacy issues may be a concern for some.

Going, going, gone?

[scribblej]

I've been a Zap2it subscriber for at least three years for my MythTV.

At first, they made me fill out a big online survey as "payment" for the service. The first time it was about 30 questions.

The third time (this is like every 3-6 months) they only asked one question.

For the last year, the survey has been "click here to renew."

What's with that? I'm willing to give up some personal time and info to pay for this service, and they can't even think of a way to leverage that?

The main listings

[DaveWick79]

It doesn't appear that this will affect their main listings on their website, just the programs that tap into their database.
I've always just setup an easy shortcut to their web page to get listings, so I really don't think this will affect my usage in the least.

The killing blow? I think not.

[ChaosDiscord]

This will be the killing blow for MythTV and other open source DVRs? I think not. MythTV predates Zap2It and managed to do okay. Yes, it relied on screen scraping, but it worked. Furthermore, I know I'm perfectly prepared to pay a small monthly fee for a good data source. Maybe $5 a month? Since a company offering such a service doesn't need to recoup costs for selling hardware below cost (as Tivo does), such a price should be feasible. Since Zap2It was free, there wasn't much incentive for someone to offer the service, but now there is. I'm hoping the free market will see the opportunity and we can work something else.

Re:The killing blow? I think not.

[Cramer]

How is writing the tool in the first place mooching off everyone else?

Writing it isn't -- and I didn't say so. Throwing it to the community hoping someone else will fix the problem before you have to invest any more of your expensive time. In your own words:

If I wrote one and threw it out to the community to maintain it, the cumulative cost to me for helping to maintain it when I notice something wrong is nearly zero, as chances are, someone will already have fixed it before I notice the problem 9 times out of 10.

The "$50" figure was an estimate. The cheapest selection is $39.99, or so. Add in tax, tag, and the other BS and 50 is good round number. Plus, it's a good bet your subscribed to more than the absolute dirt cheapest minimum package. I see I'm wrong.

Billing is simple if you only do it once a year...

Hah. For someone who makes as much as you claim, you should have better sense. You do realize how businesses actually work? You're going to be picking up new customers one at a time spread out over time. Do you plan to stack up everyone's bill and process them all on one day a year? And presumablly not allow them access until then? See. Laughable.

The service will be charging people the instant they sign up. And there must be confirmation that a bill has been paid. There has to a channel ("CUSTOMER SUPPORT") for dealing with billing errors -- there will be errors... even the teenager taking your order at McDonald's messes up occassionally and they're 2 f'ing feet in front of you. If you think you can use paypal without any such support, and a "no refund" policy, I will warn you not to hang your entire business on it. Paypal is a very dangerous way to fund a business -- paypal has and will continue to close accounts for unspecified reasons (and you may never get your money out of the account.)

Customer service is only mandatory if you're trying to make money or if you're charging enough that you might reasonably get sued if things don't work.

Making money: Check. Charging for something: Check. You are vastly underestimating people's willingness to sue, esp. when a $60 trip to small claims court can put you on the hook for much more than that. (and ruin your credit rating. etc. etc. If you won't spend 5$ for guide data, why would you waste a day for to defend yourself in small claims court?) There's a great deal of satisfaction in screwing the one who screwed you.

As for servers and bandwidth, ... would probably be pretty minimal if you designed the distribution protocol correctly...

A bittorrent inspired push model is doomed to fail, for a miriade of reasons. Not the least of which is the port forwarding for an inbound connection from nearly random outside hosts. Granted, MythTV users tend not to be complete idiots, so the expectation of the users getting the port forwarding setup correctly is better than average, but still not 100%. (btw, the spider web distribution you appear to be suggesting is patented. it's been around as long at bittorrent.)

The simplest method with the least complexity that will Just Work(tm), is a simple http request. Each user asks for the data of interest to them. That's how tivo does it; it's how everybody does it. Tivo's been doing this for a decade. If there were a better way to do it reliablly, others would already be doing it.

As an expensive hobby, it might be doable for a single DMA. I know of torrent sites that cost as much (and more) per month to operate. But even that can be a non-trivial amount of work with the variations in cable lineups. But I still think this is a perfect project for Google!

I don't see the problem

[Grishnakh]

What's wrong with just "scraping" web pages (I assume that means writing code which automatically downloads the webpages and captures the data of interest, rather than requiring a human to do it. I do this all the time with Perl code.)?

There's multiple sites out there with TV listings: Yahoo TV, Zap2It, MeeTV (the one I use), etc. Just write perl scripts to capture the listing information from these sites, and modify MythTV to allow the user to choose any service he wishes. Of course, some of these sites may (stupidly) screw with their HTML in order to throw off these scripts, but that's easily worked around with regular updates. So we just need to have a "myth-scripts" distribution site where your Myth box automatically checks for updates to the perl scripts every day and downloads them if necessary, just like we already do with many other things.

No, it's not quite as reliable and efficient as a static interface to this data, but if these companies are stupid enough to remove static interfaces, thinking we're just going to go back to doing everything manually and looking at all the ads, this seems like a reasonable solution. There's no way of preventing automated scripts from downloading webpages.

Re:I don't see the problem

[MS-06FZ]

What's wrong with just "scraping" web pages (I assume that means writing code which automatically downloads the webpages and captures the data of interest, rather than requiring a human to do it. I do this all the time with Perl code.)?

There's multiple sites out there with TV listings: Yahoo TV, Zap2It, MeeTV (the one I use), etc. Just write perl scripts to capture the listing information from these sites, and modify MythTV to allow the user to choose any service he wishes. Of course, some of these sites may (stupidly) screw with their HTML in order to throw off these scripts, but that's easily worked around with regular updates. So we just need to have a "myth-scripts" distribution site where your Myth box automatically checks for updates to the perl scripts every day and downloads them if necessary, just like we already do with many other things.

No, it's not quite as reliable and efficient as a static interface to this data, but if these companies are stupid enough to remove static interfaces, thinking we're just going to go back to doing everything manually and looking at all the ads, this seems like a reasonable solution. There's no way of preventing automated scripts from downloading webpages. They can't prevent it - but if they decide they don't like you doing it, they can come up with all kinds of ways to make it hard.

For instance, they could replace their nice little HTML table with some flash code - that's not going to be impossible for you to read with scripts, but it'll be a lot harder.
Another option would be to use images instead of text - possibly even breaking up the images into smaller images, to make it harder for automated tools to pull it and OCR it.
They could load their page with bogus invisible text.
They could provide the listing data in a funky character encoding.

So if these TV listing sites decide that the impact from web scrapers is becoming significant, they have options that can make it very difficult for you.

Killing blow?

[grasshoppa]

Me thinks that someone doesn't quite grasp the strength of OSS: That strength not just being free. Ultimately, it's the dissemination of data. Which is precisely why this problem is uniquely suited to being solved by OSS.

This will be a minor set back at worse. But, like any set back, it will make the overall product stronger.

That said..."NNOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO OO!!!!!!!!!!!!!". Zap2it was damn handy. Thank you Zap2it, for a great service.

MythTV devs are working on this...

[cesman]

http://www.gossamer-threads.com/lists/mythtv/users /275533#275533
And it isn't just MythTV that uses the guide data provided...

Google should provide a WebService

[Anonymous Coward]

Shouldn't Google or Yahoo or Microsoft or any of the other big media / tech companies of the 21st century provide this information as a free WebService? Seems like something Google would jump on top of since their mission statement is to organize the worlds information. Well, TV listings is information.... get on that Google!

Re:Google should provide a WebService

[Frenchy_2001]

Actually, Microsoft already provides such a service for their Media Center Editions users.
The media center shell will get that data automatically, with a 2 weeks forward visibility.

I suppose that microsoft either pays for that service or just provides it from a known good stream and you pay for it in the cost of the OS (either Win XP MCE or some of the Vistas).

As all those streams, they are not always correct (programming DOES change unexpectedly), but usually, works just fine.

It has another source

[TinheadNed]

MythTV can read the broadcast schedules on the airwaves - see EIT. At least that's what I use in the UK. I can also still scrape the Radio Times website in XMLtv as well.

Re:Myth will survive

[gurps_npc]

MYTH TV offers everything TIVO does plus:

Not having to pay a monthly fee to automatically record shows you watched previously

Not having to deal with commercials inserted when you fast forward 30 seconds.

Now, if they could only find a way to strip thos FREAKIN' RUDE graphics that certain vile channels overlay on top of the movie/show. I can deal with a tiny little station identification, but those rude obnoxious ads obscure vital parts of the show about one out of every 3 times. I swear my Blood pressure goes through the roof.

TV stations are hardly interested in helping...

[SuperBanana]

Isn't there any way to obtain this information in an "open-source" manner?

The best way to get the information is from the stations and cable operators.

Unfortunately, MythTV and other PVR users are in the game of cutting out ads; TV programming is purely to sell ad space, and always has been, save when programs were entirely paid for by one company and the show was branded in their name. What motivation do TV stations have to assist people who are purposefully going out of their way to cut out the ads?

Re:TV stations are hardly interested in helping...

[Cro Magnon]

Without a DVR, there is zero chance that I'll see any show that doesn't air between 7-10PM on weekdays. For that matter, the odds of me seeing shows that ARE between 7-10PM is slim, since I'm often doing other stuff too. From their POV, they'd probably be better off with me seeing bits of pieces of commercials at FF speed than with me seeing nothing at all.

Re:TV stations are hardly interested in helping...

[Perp Atuitie]

If they want people to watch their shows they have to let people know what's on. Simple as that. If They think anybody's going to go to scifi.com and then nbc.com through all 99 or 155 channels, they're nuts. Pretty much every program guide in print, on tv, or on the Net seems to depend on Zap2It except maybe Yahoo. The cable companies are going to have a hell of a time selling their DVRs and digital packages if there's no way to know what's on. But the "entertainment" industry is run by folks who run about as dumb as it gets, so maybe they will destroy the TV business as we know it. Not such a bad idea.

What if I was to write a web service?

[rehtonAesoohC]

What if I was to write a web service that exposed the data garnered from website-scraping? You could just write a standard XML request, wrap it in SOAP tags and send it to the web service, and you'd be returned whatever information you requested- by channel, time, or show name...

Any takers?

Re:What if I was to write a web service?

[soxos]

The hard part isn't writing the code to make the data available, the hard part is getting the hardware, securing a data center location, and paying for bandwidth, all without a revenue stream.

The difference comes down to OSS and Free Services. The same rules don't really apply.

However... we could create a project that would allow for smaller ad-hoc communities and not have to have one site serving the entire internet, just 100 or so users per site...

I'm on board if you want to discuss more (and you got the skillz to pay the billz).

Re:What if I was to write a web service?

[soxos]

No, that seems like a pretty good start. Like I was talking about, if we don't try to create a Zap2It replacement, but instead, small communities that can run the same codebase, then this could work without having to have a humongous server. Check this blog entry where we can create a discussion about creating a new project for this Pat F.

Re:Myth will survive

[Anonymous Coward]

I can give you TiVo's occasional recording of repeats, although a little working with the Season Passes will avoid most of that, but WTF are you talking about with inserted commercials and fast-forwarding?

Rather than a Million Screen Scrapers...

[Nom du Keyboard]

Rather than a million screen scrapers each hitting various listing sites, better one screen scraper sharing the resulting data with a million video rebels. This solution would probably leave everyone happier.

Now there's just the question of who? Who is expert at spidering the web? Who likes to provide new cheap-to-free services in their quest to take over new markets. Who would love to put yet another spike into Microsoft's side by removing yet another possible revenue source for them? Who doesn't have to worry about financing such a small, cheap service alongside their already multitudes of underutilized servers and bandwidth?

Google?

Re:Rather than a Million Screen Scrapers...

[cpt kangarooski]

Even if a compilation is not creative, it can still be subject to contract or trade secret law.

True. However, in the case of tv listings which have been published online, they're clearly not trade secrets. A contract is possible, but AFAIK none of the sites that have the listings bother with them. Specht v. Netscape would be instructive in such a case.

We need GoogleTVListings (tm)

[Blasphemy]

Google needs a TV Listings aggregator!

Of course it would be the best ever... With version numbers for each day and diffs available that only contain the changes. The whole thing would be available as an rss feed and would be free, of course.

Why not use Guide+?

[maillemaker]

My TV automatically downloads, somehow (over the air? cable?) channel lineup listings through the Guide+ system.

Could a computer not do the same thing?

Re:Oh no!

[dgatwood]

For those folks with DirecTV, that's not an option. In fact, at this point, we have no options at all other than writing a guide scraper for TVGuide.com. Fortunately, their website data is fetched in Javascript using xmlHttpRequest, so it is probably straight XML in some dialect that could be converted into the same format as Zap2It uses with very little effort. If done carefully (request the entire schedule exactly once, then only refetch the current day each day and fetch any new days added to the schedule), it might not add enough server hits for them to even care.

No it doesn't

[dharbee]

"MYTH TV offers everything TIVO does"

With a TiVo, you can plug in three cables and press power, at which point you are done but for the watching. You're claiming MYTH can do this too?

So no, it does NOT offer everything TiVo does.

Re:No it doesn't

[drakaan]

Sure it does...you just have to buy it from someone who preconfigured it...just like they did to your TiVo.

Re:No it doesn't

[Reapman]

Average joe doesn't look at it like that. He says a service. One service "just works" the other requires work. For the majority of people, they'd rather pay for a finished product then build it themselves. The apples / oranges would be each of these products are designed for different audiences.

Re:No it doesn't

[norton_I]

Yes. Also, myth has content-based commercial detection using blank frames, audio dynamic range, and scene cut rate. For most programs, a single button jumps an entire block of commercials regardless of the length. You can set myth to automatically cut commercials, and literally never see another commercial at any speed. It doesn't do very well catching very short segments such as the "moment of zen"/credits at the end of the daily show, so I don't do that, but the auto-skip is far better than tivo's FF overshoot compensation.

Re:No it doesn't

[garcia]

Average joe doesn't look at it like that.

I consider myself far more computer savvy than the typical "Average Joe", running Linux for over 10 years, yet I cannot be bothered with purchasing, building, installing, and dealing with a MythTV setup when my DirecTivo does everything I need just fine.

Re:No it doesn't

[jZnat]

I don't suppose you expect to get your TiVo service for free, do you? You're still paying a monthly fee to use the hardware, and just like with mobile phone companies in the US, the hardware price is subsidised so that you'll sign up to a multi-year contract that nets them more profit in the long term. With MythTV, once you buy a box or set one up, you don't pay anything ever again with regards to the hardware or any sort of subscription. I'd like to see TiVo do that (they used to, but not anymore; can't get a lifetime subscription thing from them anymore). TCO, my friend.

Re:No it doesn't

[sfbiker]

To be fair, your Tivo costs $100 + $179 = $279 if you keep it for a year. If you keep it for 2 years, then it's at least $100 + $299 = $399. Prices are higher if you pay monthly.

Subscription rates are here: http://dynamic.tivo.com/2.0.plans.step.2.asp?box=s eries280hrDTDVR

I doubt you can build an equivalent MythTV box for $279, but perhaps for $399? A quick google search couldn't find any for sale -- does anyone sell preconfigured MythTV boxes?

Re:No it doesn't

[jedidiah]

...for small values of "just works".

I can tweak my recording rules in ways on the Myth that can't be matched on my Tivo. Really easy and cheap to add storage too. Plus I can compress using MODERN compression algorithms and avoid the blockiness of Basic quality.

Lack of an mpeg4 unit is another instance of that laurel resting I mentioned earlier.

Re:No it doesn't

[toleraen]

I don't suppose you expect to get your TiVo service for free, do you?

You might want to read the summary, or even the title. From what our fine /. editors have implied, you won't be getting your MythTV feeds free either.

TCO indeed...when you figure in TCO for MythTV, you are taking into account the risk you take in your reliance on a free service that you have absolutely no control over, right? Because that's kind of necessary.

Re:No it doesn't

[pakar]

http://www.xyzcomputing.com/index.php?option=conte nt&task=view&id=589&Itemid=0&limit=1&limitstart=6

There you have a short story about someone that built a box... and about the prices..

VIA EPIA board - $100 (these are a bit expensive, so it might even be cheaper to go with some low-end motherboard with built-in gfx card and a AMD-Sempron..)
128Mb RAM - $30 (128Mb is quite enough for a pure mythtv box)
HD 160Gb - $40
TV-Card (DVB-T) - $80 (With a CI-Slot)
Case - $50

hmm... so that's $300, and with this you have full control...... Want to have a HDTV capable box, then just go with a bit more expensive EPIA board with DVI output... Want to have more than one PCI card, maybe 2 hybrid tuners, then go with a case that supports a raiser card.

So there you go... with $300 you can have a simple tv-box, add a few more $ and get dual-tuners, add a few more and have a HDTV capable box... add a few more and control you X10 home-automation system..
Don't like the remote? Well, get a bluetooth dongle for the system and use your cellphone, or just buy some other remote...

This is the charm with mythtv.. You can do just about anything if you want to.

And a tip if you are going to build a system like this. Do keep in mind that you might want upgrade the system in the future so try to think of what you might want to add in the future.. And if you are going to have it as the main entertainment system try to keep everything on the system at stable-versions and no SVN checkouts :)

Re:No it doesn't

[AeroIllini]

With a TiVo, you can plug in three cables and press power, at which point you are done but for the watching. You're claiming MYTH can do this too? "With an Aibo, you can charge it up and press power, at which point you are done and have a fully functional robot dog at your disposal. You're claiming that LEGO Mindstorms can do this too?"

MythTV is not, on its own, a plug-and-play solution. It is a tool used to accomplish a task, and in its current form right now, it is designed for people who want to build their own system and configure it just the way they like. It does not include hardware, and it does not benefit from the economy of scale in manufacturing that TiVo does. If you want the ease of use of TiVo and are willing to put up with the shenanigans of a company who is slowly reducing what you are allowed to do with the downloaded content (which, by the way, is reducing your fair use rights) then I encourage you to buy a TiVo, and stop trolling in a MythTV thread.

Do not confuse "I want it to work RIGHT NOW" with "I want to build it myself". These are very different and, in some cases, mutually exclusive concepts, and they require different tools. We want to build our own boxes. Go away and let us play.

Make it a paid service

[pz]

Like many others, I'm a little surprised that they aren't moving to a subscription model. Clearly they know better what their available resources are, and what they are and are not capable of handling, but it seems like a missed opportunity to walk away from a situation where their servers are getting hammered, and start charging a small fee. Many, many of the MythTV users would happily pay a few dollars a month to have a steady stream of information. Sounds like it could be a million dollar annual income right there, and that's got to be very hard to walk away from for any company.

As for commercial abuse, if they know it's happening, they presumably are taking steps to quash it as well, without much luck. Probably like playing whack-a-mole.

Let's all hope Google comes to the rescue.

Re:Make it a paid service

[Buelldozer]

I was happy to find your clear, concise, comment down here all by itself. It makes it easy for a clean response.

Let's say that MythTV implemented your paid service plan and began charging the princely sum of $2 per month for the data.

I would give it all of 7 days before that paid for data became available for free. Someone, somewhere, would buy the data for $2 per month and load it up for others to have free of charge. It would be a daily torrent that you could pull, or a streamed RSS feed, a static layout site with a downloadable screen scraper, or any one of a dozen other ways I can think.

So now instead of a million dollar revenue stream you'd get a thousand dollar revenue stream coming from the 500 users who would actually be wiling to pay when a free source is available.

If you can answer the question of how to prevent the above scenario from happening I can put you in touch with some content providers who will pay REAL money for your idea. The kind of money that allows people to retire for life...at the age of twenty.

Re:Make it a paid service

[pavera]

well, sure, but lets say you want to provide this "free" service, to even be marginally popular/useful, you would need at least the 50 largest markets, and the top 3 providers in those 50 markets, so you're going to have to cover at least OTA/local, incumbent cable, dish, and directv. So that is 4 accounts for 1 market, times 50. That is 200 accounts. You really think there are tons of people out there just dieing to pay $400/month to give away free tv guide data? I really don't think so. At first I thought the original reply was spot on with the criticism of the idea of trying to make money from this, but your parent made me rethink it. The market is entirely too fragmented, different localities, different channels, different providers. Plus for sports you've got market blackouts, stuff that is available in Nevada might not be available in California...

There is no way to efficiently pirate this data. It only makes sense for a centralized provider to charge a small fee and distribute the information directly from there.

Re:Myth will survive

[IthnkImParanoid]

You forgot being able to watch a recording on your laptop while on the toilet. I'm puzzled as to why MythTV doesn't advertise this feature a bit more, since it's one of my favorites.

That being said, you can't honestly suggest that MythTV is always a better choice than Tivo, unless you've conveniently forgotten about the teeming masses of people who couldn't install a capture card if their life depended on it. Ease is the killer feature for Tivo and bundled PVRs.

What happened to Mythtv's paid service, TechnoVera

[digitalderbs]

A year ago it was announced that an alternative paid service through TechnoVera was available to replace Zap2It with part of the funds going back to OSS and mythtv -- and no periodic surveys. Couldn't users switch to this service? -- or is it no longer available? (I've never used this service myself.. any users care to respond?)

Re:Myth will survive

[teknotus]

I think the solution is to pressure TV stations to provide listings in something like an RSS feed. Then all you need is a database of RSS feed URL's. Any idea's on how to sell TV stations on how this will benefit them?

But EIT has limitations

[LionMage]

Excellent link. However, using EIT has at least one drawback:

Unlike DVB systems, an EIT in ATSC covers a limited time span. [...] It's possible to transmit up to 128 EIT tables for any virtual channel, but it's very rare that so many are transmitted - terrestrial systems have to transmit EITs covering at least the current time slot and the next three time slots.

So at least in the U.S., the EIT would be of limited utility to MythTV users, since it is unlikely that one would be able to use this data to schedule multiple days in advance. Also, the article you linked only indicates that EIT is a standard part of ATSC (the terrestrial digital TV broadcast standard), and wouldn't necessarily be present in analog broadcasts (which are going to be around for a little while longer) or in digital cable transmissions (which use QAM here in the U.S.). It's entirely possible that a broadcaster might supply EIT data, and a cable operator might strip it out to save some bandwidth.

I've seen some of the newer LCD HDTVs sold here in the U.S. which have the ability to display program guide data, no doubt culled from available EIT data, but this is really intended to allow viewers to see what's on right now or in the immediate future without resorting to channel flipping. I bought a Sharp Aquos one generation before this feature became standard, so I missed out, but I really don't feel like I'm missing too much because of the described limitations.

Re:But EIT has limitations

[LionMage]

Whoops, scratch what I said about QAM. After more careful reading, it looks as though EIT is available for cable systems in the U.S., although the article cited doesn't address the issue of encrypted QAM (which most open hardware and software can't decrypt, making the QAM capabilities of most HD tuner cards limited at best). So even though EIT might be technically available for digital cable users in the U.S., it might not be practically accessible to MythTV. And there are no guarantees for how far in the future scheduling data will be provided.

Re:But EIT has limitations

[martin_b1sh0p]

Actually, if you use TV Guide On Screen, it's 8 days worth of info here in the states (not just the next three time slots). Not as good as Tivo's 2-3 weeks but still better than nothing. I know this because I have an LG LCD TV with a DVR built in downstairs which has the TV Guide as well as a Tivo in the bedroom upstairs.

http://en.wikipedia.org/wiki/TV_Guide_On_Screen

http://www.tvgos.com/

I use DVR but what is Zap2It

[Sciros]

I have an HD cable box at home with built-in DVR that functions like Tivo... yet I haven't even HEARD of Zap2It, let alone needed to use it ever. Warner's digital cable boxes have great TV guides, so as long as cable provides have that, why need another online one?

Common sentiment

[UnknowingFool]

I'm sure I represent most other MythTV users when I say: "FSCK!"

Re:Myth will survive

[raw-sewage]

You forgot being able to watch a recording on your laptop while on the toilet. I'm puzzled as to why MythTV doesn't advertise this feature a bit more, since it's one of my favorites.

Yes, and more generally, MythTV's client-server architecture opens up a lot of possibilities. If you have a busy household (e.g. big family), build yourself a monster MythTV backend, and stash in in your basement. Then install MythTV thin clients everywhere in your house, and everyone can watch what they want when they want.

I also use the MythVideo plugin. I've copied all the DVDs I own to a big file server (so I have all my movies "on-demand" via MythTV). My girlfriend said it would be cool to install another MythTV system in a spare bedroom. Then when we had guests, they could watch a movie before going to bed (or when they get up in the morning, whatever).

Yeah, it's definitely not trivial to setup. But I always thought someone could make a business out of setting MythTV up for people. Charge a big upfront fee, and it's free after that. (Of course, you expose yourself to risk like this Zap2It debacle!)

Re:So what?

[grommit]

The TV market will be transformed by IPTV in 10 years? That's cute. Really. I have no doubt that there will be more people using IPTV in 10 years than there are now (all dozen of them) but to say that it will transform the market is stupidly optimistic. You seem to forget that for better or worse, media companies control the TV market currently and just like they control the music market (hear of RIAA?) they are not willing to change how they do business. Yes, I know there are plenty of places to get indie music but unless you're one of those people that think so highly of themselves that they refuse to listen to any music that becomes mainstream, you are missing out on a lot of music.

It'll be the same way with IPTV for a very long time to come.

P2P Listings

[Doc Ruby]

TV listings seems like a great P2P app for MythTV. Listings for future broadcasts can be read by users from their local media.

In small entries (enforced by the GUI) that don't trip the copyright of the original publisher. Or, like most P2P users, they'll ignore the copyrights.

It's like a Napster index that doesn't point to copyright violated media, but to fair use of one's own media subscriptions.

So you can automate scanning the data.

[DrHow]

A number of the comments here seem to take the view that one source of listings is as good as any other. However, the power of getting the listings data (as opposed to a look at formatted listings) is that you can then use software to read that data and find all the programs that might interest you. The XMLTV package itself offers a number of tools for scanning the data and selecting programs based on rules you specify. There also exist other application programs which have sophisticated search capabilities for treating XML program data. The number of different TV stations I receive now is staggering. I would waste much more time finding the programs I want to watch if I did not have software that automatically weeds out most of what I would not care to consider. Furthermore, there are programs, which would interest me and which I would enjoy watching, but which I would miss if I did not have the tool to spot them for me because it would just take too much time for me to scan for such shows in the listings for all the stations which I rarely watch.

Re:Open Source != Free as in Beer

[EvanED]

Open Source doesn't mean that a company can't charge something. Look at Redhat. They sure as hell don't give everything away for free.

No, but CentOS does.

Open source means a change from a product-based market to a service-based market, at least to a FAR larger extent than is acknowledged by the typical "look at Red Hat" line. Remember, Red Hat makes most of its money because people want support. That's fine if you want to be in that business and can come up with some value to add to the product that only you can provide.

If you have something like TV schedules, there's no service there, unless you want to talk about providing the bandwidth to you to get it. If Bob generates the schedules and Fred buys them from Bob and starts giving them out, what's the incentive to go to Fred and pay for them? I certainly can't think of any added value Bob can give. So if Bob spends a good bit of money compiling the listings and the only people who actually buy them from him are those who want to actually support Bob while the people who just want the listings go to Fred, that could very easily be a losing proposition for Bob.

Re:Myth will survive

[SydShamino]

Unlike printed guides, which have to limit show descriptions to one or two lines at best, digital guides can be as detailed as you wish. By providing your own RSS feed of your schedule, you could increase the information given, which should increase the number of people willing to watch your channel. Even though I have a DVR I still watch (interesting) commercials, so don't completely write off my viewership.

And here's how you can directly make money:
By providing your own guide information, you can insert sponsorship lines and charge for them.

"Tonight on Lost, sponsored by Coca-Cola, Jack and Kate have more awkward sexual tension, while Hurley tries to cheer everyone up, Sawyer acts rudely, and Sayid kills someone with his feet."

Even us DVR users will very often look at the guide information. Voila, you've just sold an ad!

Thanks

[ImaLamer]

Thank you for being the first to thank them.

It sucks that it' shutting down (I use it for GB-PVR)- but they did do it for free for years. Let's not demonize them, no matter the reason, and start looking for alternatives.

Re:Myth will survive

[BobPaul]

MythTV doesn't fast forward, it does commercial skip. Automatically. It hits a commercial break and just jumps past.

Tivo used to allow a really nice FF feature to skip commercials. Now they overlay advertisements on top of the advertisements you are fast forwarding through. Not to mention the advertising in the rest of the UI. If you own one, you should know what I'm talking about. If not, Google found me someone's blog with pictures

Re:Myth will survive

[roye]

I never see ads when I use Tivo's 30 sec skip. That you have to redo the hack after a reboot (admittedly almost never) is bullshit. But I can take the 5 seconds to press Select, Play, Select, 3, 0, Select. Instant 30 second skip anytime you want. This is also good to use during a boring minute or so of a show.

Re:Myth will survive

[russ1337]

I think you are on to something. I've been keeping tabs on the NZ Mythtv forums & mailing list (am about to move there) and EPG seems an ongoing issue for them.. I've often thought what you've suggested would be their answer.

People further down in the thread suggested that TV channel EPG is right up Google's alley, and I agree... just how do you persuade them???? Hopefully there are a few Mythtv users in the Googleplex that have the sway to pull this off. Google have the funds to purchase the EPG data, AND the customer base to insert ads in the stream (however ads would not be click-able).

Google... please buy ZAP2IT LABS!!! please!!!

Re:Myth will survive

[mysidia]

Every time a feed is lost and a new one is found, every time the guide feed changes, every user has to manually upgrade the software, and it's sure to happen many times in the future..

That is inefficient.

What would be ideal would be if myth boxes could connect to each other to form a peer-to-peer network and share some of their guide data with each other.

Let the authenticity of an entry update (in case of conflicting stories) be determined by the reported source of the update, a vote, and the "age" and reputation of the mythtv installations reporting the listing.

In that manner, when one feed dies, any mythtv user who can would be able to scrape or find listings data and provide it for the benefit of all mythtv users.

Also, no one mythtv node would necessarily need the full listings, they would only need to submit updates randomly based on newly acquired data, and to submit queries based on listings they're interested in.

I.E. channels the user is currently trying to lookup listings for, or channels in the channel lineup for the next few days, and searches for program names the user is interested in or wants recorded every time.

The real reason why TMS is killing Zap2It

[Anaerin]

Taken from a post above:

Some reasons and other comments given for the scheduled discontinuence, copied from the forum, copied from the mailing list:

• Continued use of the service to support commercial products, in violation of the agreement.
• Commercial products continued to grow despite previous appeals that this activity stop.
• There are significant changes to the supporting data structure forthcoming and we could not devote resources to the continued upkeep and enhancements of the service.
• Maintenance of the service is impacting our resource pool for other projects.
• We sought alternative options but were unable to find a solution.
• We recognize the hardship this creates for the user community. We are open to alternative solutions and would consider proposals that met the needs of the user community and our company.
• We looked into options to turn this into a paid service however we do not have the infrastructure at this time.

(NB: Emphasis mine)

Basically, what they're saying is that they will be changing their database structure, and can't be bothered to re-code the Labs.Zap2It part. They'd rather spend the time on other things, like increasing ad revenue.

Bear in mind that TMS charges each station (Somewhere in the region of $75/day) to collect their data, and charges their commercial end-users (somewhere in the region of $500/market/month) to provide the data. Quite a lucrative trade, wouldn't you say? Add to that advertising revenue from their site, and subtract bandwidth fees, and they still make a tidy profit.

It has been theorised that the main reason labs.zap2it was set up was to save bandwidth costs after XMLTV scrapers started hitting their public site. As the data was served up in chunks in the midst of a sea of other information (Links, advertisements, commentary, navigation etc) it cost considerably more in bandwidth, so it made more fiscal sense to offer the data for free, so they could track who was using it and where, along with only having to serve the data itself, with compression and selectivity. This is only a theory, mind, but considering how the Myth community (On it's own, without any of the other projects that were/are using Zap2It feeds) has grown, I think removing this option will drastically increase Zap2It's bandwidth bills without adding any ad revenue back into the pot (A scraper doesn't see ads, and doesn't care about them).

In other words, this could be a costly mistake for TMS. Here's hoping they see sense, and work out a way to work with MythTV and others.