Slashdot Mirror
The Raid-Proof Hosting Technology Behind 'The Pirate Bay'
HughPickens.com writes Ernesto reports at TorrentFreak that despite its massive presence the Pirate Bay doesn't have a giant server park but operates from the cloud, on virtual machines that can be quickly moved if needed. The site uses 21 "virtual machines" (VMs) hosted at different providers, up four machines from two years ago, in part due to the steady increase in traffic. Eight of the VMs are used for serving the web pages, searches take up another six machines, and the site's database currently runs on two VMs. The remaining five virtual machines are used for load balancing, statistics, the proxy site on port 80, torrent storage and for the controller. In total the VMs use 182 GB of RAM and 94 CPU cores. The total storage capacity is 620 GB. One interesting aspect of The Pirate Bay is that all virtual machines are hosted with commercial cloud hosting providers, who have no clue that The Pirate Bay is among their customers. "Moving to the cloud lets TPB move from country to country, crossing borders seamlessly without downtime. All the servers don't even have to be hosted with the same provider, or even on the same continent." All traffic goes through the load balancer, which masks what the other VMs are doing. This also means that none of the IP-addresses of the cloud hosting providers are publicly linked to TPB. For now, the most vulnerable spot appears to be the site's domain. Just last year TPB burnt through five separate domain names due to takedown threats from registrars. But then again, this doesn't appear to be much of a concern for TPB as the operators have dozens of alternative domain names standing by.
I mean, 620 GB of storage isn't much, but I'm sure some people would want to RAID it anyway. Although I've heard that Police RAID only works with write-only storage...
Ezekiel 23:20
So, you're like the last person in the world to understand that TPB holds no content, just pointers to content?
Watch this Heartland Institute video
Raids only make sites become raid-proof. Just as monitoring creates encryption and oppression creates rebellion.
But of course one cannot fight the core problem when the core problem is oneself.
(RIAA, MPAA and various law enforcement agencies not counting as "people").
Watch this Heartland Institute video
Are they making $ off this or just doing it for the lulz?
The Pirate Bay definitely deserves praise for staying up, despite being famous and constantly attacked by the media mafia. They bring hope that one day we may live in a world where sharing of knowledge, art and data is encouraged rather that prosecuted, and that some of today's files will survive until then, as well.
It will require a lot of work until we get there in the social realm (fighting the abusive law). It may help if technical solutions exist (decentralization, anonymity, security) that allow everyone to ignore the nonsensical law, to make the case even more obvious and to get by with our files in the meanwhile.
Their traffic is up that much?!
I imagine it has a lot to do with more and more countries coming online with broadband in recent years--countries where people often don't have any legal options to purchase movies, or the money to purchase them even if they did.
SJW's don't eliminate discrimination. They just expropriate it for themselves.
(RIAA, MPAA and various law enforcement agencies not counting as "people").
They do know this! And their tactic is to let you think that they don't.
The "keys to the kingdom" point to virtual machines that can be rehosted faster than the raider can work the legal system in multiple countries to get to the next level of servers after raiding the load balancer. The point is not that they can prevent raids but that any raids will be ineffective at shutting them down for more than a few minutes. That effectively discourages raids as a strategy as they are expensive and ineffective.
Single points of failure aren't that much of a factor in virtual machines. If the hardware goes down, the VM is restarted elsewhere. If the machine dies, you copy it from backup... or have a copy on standby (which can be and often is the same thing). After all, there aren't many changes on a worker machine like that.
Not to forget, nothing stops you from clustering that shit from Microsoft's to Amazon's cloud.
Considering that I've never found TPB to be down when I needed it and haven't heard many complaints in that direction, I'd say their system works pretty well for them.
That's perhaps part of it, but the main reason is that TPB is still the best place to get new stuff. I always download all new TV-series there, except netflix-series that I can get on... netflix.
As for games, I recently wanted to buy a few games on stream, but they demanded a copy of my id because my card was issued in another country than the one I live in at the moment. Fine, goodbye. Downloaded the games at TPB instead.
It should not only be possible but also convenient to buy media. Netflix and GOG are great and get my money, other places that are less great and makes it difficult can keep their digital copies while I go browsing at the Pirate Bay.
went legit half a decade ago?
According to many people, they have always been just as legit as Google or Bing.
They just provide more accurate search results for certain types of searches.
" but they demanded a copy of my id because my card was issued in another country than the one I live in at the moment"
Sounds like Steam was trying to prevent credit card fraud, most people think that is a good thing and it is not exactly a privacy concern since your id is tied to your credit card to start with.
It doesn't matter who it's for, TPB is doing nothing wrong by simply providing directions to content. It would be like me telling you that you can buy drugs from random people in a certain area of the city. I'm not selling you drugs, I'm only telling you where you can get them.
To be fair he didn't say it was a privacy concern. I have had to deal with similar processes for different reasons and, only did so because there was something I wanted and couldn't get another way on the other side of that process. Any way you slice it, its more work, frustration, and waiting around.
any barrier thrown up, even for the best of reasons, is going to dissuade some amount of people you were not intending to.
"I opened my eyes, and everything went dark again"
but they demanded a copy of my id because my card was issued in another country than the one I live in at the moment.
You need to give them your name and address anyway for a credit card transaction, and you were being subject to fraud prevention. That's an excuse to pirate, not a reason.
I would say sort of the opposite. It became the best place when pretty much every single one of its competitors was raided or threatened out of the business. I burned through a few I liked more before TPB was the only left.
Troll is not a replacement for I disagree.
I was surprised it was so humongous. I think Wikipedia is like a tenth that size and it is based on hosting its own content.
Troll is not a replacement for I disagree.
Well, don't hold out. Where can I get some weed already?
You are welcome on my lawn.
I agree, "exclusive distribution rights" should be illegal unless you are the content creator. Capitalism is supposed to be about competition.
Get free satoshi (Bitcoin) and Dogecoins
Most musicians I know make money doing gigs (i.e. working for a living). Movies are generally profitable or not based on theatrical sales - a time when there are no quality online versions; sales after a theatrical run is complete rarely changes a flop to profitability.
Interestingly, there are troupes of actors travelling all over the country and world who make money night after night performing in venues all over the country side. It's called theater, and - interestingly - when you put a "star" in a show you don't even have to travel. Have you seen the sellouts for Neil Patrick Harris, or Patrick Stewart on Broadway? Even if you ignore the fact that people can still make money performing live, the top movies, since 1920 have *the theatrical receipts* often exceeding the production cost by a factor of 4. That's a margin even the stingiest of capitalists drools over. In fact, the top 50 theatrically grossing movies (which are mostly from the last 20 years) grossed no less than 775 Million dollars EACH, and only 7 of them cost more than 200 Million to make, with none more than 300 Million. It's probably okay not to worry too much about being able to feed the families of the poor movie executives, even if by some strange change in the copyright law they lost all rights to their films at the close of the production run.
Is it just my observation, or are there way too many stupid people in the world?
Privacy isn't the only reason to not provide a copy of ID. Consider how often we hear of payment data being stolen. When the legitimate company asks for a copy of your ID, they are trying to protect themselves, not you. If someone stole your payment card info and copy of your ID, they have everything needed to "prove" to someone else that they're you. It's not like the old days where you could fax in a low resolution copy of your ID and rest assured that the piece of paper lives in a file that will never see the light of day.
TPB will get sued in a favorable location for the plaintiff. The plaintiff will use the judgment to go after TPB bank accounts. The back accounts are much harder to hide than the servers because TPB wants to get paid for the ads it displays.
Interesting that registrars will threaten sites that assist in obtaining illegal copies of software or media, but will do nothing whatsoever when they are shown that their customers are selling kiddie porn, illegal / counterfeit drugs, counterfeit anything else, etc...
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
My reason not to give my ID, (even if it means paying for a service which can be taken free by showing an ID Pardus anyone???) is simply that I do not know when the database they use to store my ID will be either hacked by a script kiddie or raided by a foreign government. My domestic government already has full access to my person and data, there is no need to increase my accessibility (!)....
They are basically providing meta-data (even meta-meta-data). Aside from RIAA and clueless judges, I was assuming most people are seeing their service as legit as it gets.
should be illegal unless you are the content creator.
A small problem with your idea. Who is "you" ? So if an author enters into an agreement to have his book made into a movie by say, The Disney Company who is the "content creator"? Is it the author? The screen writer?, The shareholders in Disney Stock? The CEO of Disney?
Lets say for instance you arcademan come up with a really cool video game that seems to be an absolute hit. Do you have the resources to put up a server farm that can distribute that content to the masses? Let's say I do and I put my considerable resources to work on your behalf for some fee per distributed unit. I really have no sure way of knowing if you will sell 100, 1000, 10000 or even a million units. So I want an exclusive agreement because I have made those resources available to you and therefor have tied those up.
Hey KID! Yeah you, get the fuck off my lawn!
There are some on the wikipedia page but I suspect some of them are outdated given what's said in the article. I was interested in reading about their VM setup and how they communicate with each other and what platform they're using, etc, but I can't find any details anywhere. I went through their blog, their forums, the affiliated articles, etc. Does anyone know where one might find more details of their infrastructure?
That soon a DEA SWAT team will attack VMWare development facilities and smash everything up, using trumped-up drug charges.
With some VM architectures, the VM may not even need to be restarted. With IBM's POWERVM and vSphere's FT, it actually runs multiple VMs at the same time executing the same instructions in lockstep. That way, if the primary fails, it is only seconds before the secondary takes over.
Colorado. Washington. (You're welcome.)
I have sometimes said that the torrent file (or magnet link) is a virtualization of the actual data files, kind of a seed which you plant and from which the data grows. Even when the torrent does not contain the actual data files, you "see" the files through the torrent.
Personally I'm more impressed by something that's small and efficient in these days, rather than something massive.
As a content creator, I would prefer spending time creating content.
And, if I sell the rights, like Notch sold minecraft, he would be the only rights enforcer under your plan, and he is only interested in making cool little games. No sale would be possible.
Care to reconsider?
"The remaining five virtual machines are used for load balancing...."
Sanity is the trademark of a weak mind. -- Mark Harrold
You need to give them your name and address anyway for a credit card transaction, and you were being subject to fraud prevention. That's an excuse to pirate, not a reason.
So? It's still inconvenient because now you're stuck in a manual process that they will eventually get around to when you want to play right now. I've done something similar when a game without warning refused to activate - granted, I'd been playing with WINE settings and uninstalled/reinstalled quite a few times but this was Friday afternoon. A few hours later and no reply, I said fuck it and downloaded a cracked version off TPB. Support came back to me on Monday and started asking questions about why I'd used so many activations, I just sent back a reply basically saying I've found a permanent solution so go fish. Okay so fraud prevention is a bit more valid reason but it still doesn't fix the immediate problem.
We've had this discussion many times before here on /. with regards to Linux, no matter how many valid reasons there is for "CANTFIX" problems ranging from crap Linux support, undocumented formats and hardware, "embrace extend extinguish" incompatibility and lockout users don't care. This doesn't work, give me something that works. I must admit my tolerance has grown extremely slim, when you know that there's a not-so-legal alternative that always works flawlessly it really doesn't take much before I say "screw this, I'll get it from TPB. Heck, I still download GoT even though I pay for HBO Nordic.
Live today, because you never know what tomorrow brings
Ernesto reports at TorrentFreak that despite its massive presence the Pirate Bay doesn't have a giant server park but operates from my butt, on virtual machines that can be quickly moved if needed.
[...]
One interesting aspect of The Pirate Bay is that all virtual machines are hosted with commercial butt hosting providers, who have no clue that The Pirate Bay is among their customers. "Moving to my butt lets TPB move from country to country, crossing borders seamlessly without downtime. [...]"
Moments like this remind me why I installed that firefox extension.
It does seem like a lot. In 2012 someone ran a scaper on tpb to grab all of their magnet links, it came in at under 100MB compressed. Of course this didn't include the comments or the .torrent files.
https://torrentfreak.com/downl...
The torrent is available at https://thepiratebay.se/torren...
The technology listed is not raid-proof, only raid-resistant.
It is still vulnerable to legal attack IF the governments in the countries where the servers are located are willing to use subpeonas or other means to "quietly" (i.e. without TPB finding out) determine what the next "downstream" server is until they have a full list, then do a coordinated takedown.
All it takes to stop this is to make sure that at least some key servers are in countries in which such court orders could not be legally issued.
The summary didn't say it, but I would think that after all that they have been through, TPB also has recent-enough "disconnected" backups of all of their key servers that they could bring it all back up within a matter of days if their servers were all seized at the same time. I would also think that they have a "shadow staff" who can take over in the event that the people currently running the show are arrested or ordered by a court to not participate in the project.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
44GB. Surprising. http://en.wikipedia.org/wiki/W...
Current revisions only, no talk or user pages. (This is probably the one you want. The size of the 13 February 2014 dump is approximately 9.85 GB compressed, 44 GB uncompressed).
So, you're like the last person in the world to understand that TPB holds no content, just pointers to content?
With TPB mainly running on magnet links, it's not even that it's a hash of pointers to content these days. Even the actual pointers have gone off-site, which reduces the bandwidth by 99%. My guess is TPB actually serves up more ads than content, if you count bytes.
Live today, because you never know what tomorrow brings
Yeah you run it on a VM. Everyone's using VMs all the time now, right?
Have you tried running Battlefield 4 inside a VM?
- For the complete works of Shakespeare: cat
"The remaining five virtual machines are used for load balancing...."
There's one controller though (there's ALWAYS one controller).
Nope. TPB is the only moderately trustworthy public tracker through which to obtain anything, other than kickass.to. Most downloaders that I know use a private tracker site where there is a history of quality and community, but for everything else TPB is really the only solution.
Interesting. I wonder if the EULA was left in the pirated material whether that could be used to insist on MBA in the case of being caught for said piracy.
We have laws regarding processing of personal data in my country, specifically to prevent their misuse, and this process completely circumvents the protection of my person by stringent local regulations, since I don't think that foreign companies without any legal presence here just happen to be compliant. Yet you claim that by giving up my rights, I actually gain them? That doesn't sound quite right to me.
Ezekiel 23:20
That's what I thought had happened. I remember TPB selling themselves to a software company for like ten million bucks with plans to turn into a "legitimate tracker of licensed/contracted content". Everyone went nuts over it. Then they all switched to private trackers.
I've actually always been highly suspect of TPB. Not because of those behind it, but because it is such a high value target, compared to other trackers that you could use (especially private, obviously -- though then there are situations like Demonoid and others that become really iffy due to certain events).
Wait, what? Since when are retailers supposed to ask for your identity when using a credit card? My understanding was that they were actively discouraged from doing so by credit card companies. In fact, I remember they used to have a toll free number you could call to report a retailer if they refused to accept your VISA without giving them ID.
You reason is entirely valid IMO - you bought the game, and it didn't work under WINE. This guy is saying straight up he doesn't want to buy it because the sellers asking for basic information, and specifically, that they're doing a job that protects both the credit card company (they're liable for fraud, at least here) and himself (since if they're liable for fraud, they need to raise rates to compensate). Those are two entirely different leagues, and comparing them as if they're equal is disingenuous.
Billing addresses are a thing, you know. They use it as verification.
The practical reality in Washington at least is still:
A) Medical dispensary
B) Some guy in a certain area of the city
Because there are so few legal recreational stores, and the few legit growers are having trouble keeping up with demand.
There a few (illegal) delivery services that operate and anyone who smoked weed already has a connection.
Why would I leave my great connection to buy from a store? My dude is never out, has a few varieties.
Growers make more money selling to their normal connections then going legit. IMO they government got a little too greedy with all the taxes on it...
Be seeing you...
As for games, I recently wanted to buy a few games on stream, but they demanded a copy of my id because my card was issued in another country than the one I live in at the moment. Fine, goodbye. Downloaded the games at TPB instead.
There are two other very good reasons to use TPB, Very recently, I downloaded Civ V from TPB. I have not enjoyed any of the series since Civ 2 but this one seemed to have enduring good feedback so I wanted to try it out. I downloaded it and started playing it. I looked at my clock and realized I was 20 hours into my first session. My how time flies. I went ahead and bought it on Steam. By the by, I do not currently reside in the country my card is issued from. apparently, if you hit cancel (back?) and submit enough, it becomes confused and allows the transaction anyways.
The other reason I use the TPB is to have a copy of a game that I already own that does not have to "install". I just zip up the directory along with a .reg file and whenever I move to another computer, I unzip it and run the .reg file if necessary and WHAM. Instantly playable. Screw installing StarForce and trying to authenticate against a server which no longer exists. I paid for the game and I will play it. On my terms. My access to the game does not stop just because someone wants to shut down their authentication servers. That is fraud on their part. They should release a patch to not require authentication if they do not feel like running the servers anymore.
"Someone needs to talk to the tree of liberty about its ghoulish drinking problem." by ohnocitizen
No need for a controller, you can always do a round-robin DNS server list.
Well, all the free publicity as of late....
Examples of such countries?
From what I was seeing in Gabon last week (last time I'll be there for several years, 3rd or 4th visit in the last year, IIRC), they have no shortage of legal options for buying movies, and many more for buying discs which are commercially stamped from probably illegal copies of discs. I fail to think of any other reason for a Francophone country with an Anglophone minority to be selling DVDs in Chinese (originals imported by migrant workers, probably) and Russian language (source unknown).
Bandwidth on the other hand? Downloading a standard-resolution DVD would take several days if you could steal all the bandwidth of a major (200+ bed) hotel. That would cost ... well, several days of wages, plus several months of income for the downloading machine.
Remember when you had dial-up. Remember web-surfing on 33.6bps? Actually, no, I suspect that you don't. You're taking for granted bandwidth which for much of the world simply does not exist. You should try being on my work vessel for a time : a 4Mbps satellite link with 2Mbps allocated to one particular service, 1Mbps for phone lines in and out (about 1 in 5 phone calls fails due to no line available - retry in 30 seconds), and the remaining 1Mbps allocated for all business and personal use between 180 people on board. (And we're a hundred kilometres from any mobile phone coverage, so forget that. Not that mobile phones are allowed outside the Faraday cage when we go into radio silence for explosives/ flammables operations.)
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"