Slashdot Mirror
TinyDisk, A File System on Someone Else's Web App
Psy writes "I attended Phreaknic this weekend where Acidus released TinyDisk, a shared file system that runs on top of TinyURL or his own implementation NanoURL. TinyDisk compresses a file, encrypts it, and dices it into clusters. Each cluster is submitted to TinyURL as if it were a url. This clusters can be read back out of the database, making TinyDisk a global file system anyone can use. There are safeguards in the default config to prevent people from dumping gigs of MP3s into TinyURL. While file-system-on-web-applications are nothing new (GMail file system anyone?) this hack shows how easy it is to accidentally design a web application insecurely despite the default PHP protections. See his presentation for more info"
Nifty little program all the same and a nice hack ,
Having it running on his NanoURL implementation locally , could allow for a cool little service . Though there are better ways to offer web based file systems in the real world .
He does state in the FAQ that its not intended to pollute TinURL in any way
Perhaps it will give TinyURL a nudge to tighten up their security though .
The only things certain in war are Propaganda and Death. You can never be sure which is which though
TinyURL might become not so tiny anymore...
One that hath name thou can not otter
I got a BigDisk - I dont know about you guys.
I love to upload DNA strands using it.
I saw this a few hours ago, and from what I understand the process goes as follows:
:)
1- Open a meta file
2- Retrieve and concatenate all the clusters from TinyURL in the order
specified in the meta file.
2- Base64 decode the file
3- Decrypt the file with the algorithm and key in the meta file
4- Decompress the file with the algorithm in the meta file.
5 - Verify the file size given in the meta file is correct for the
decoded/decrypted/decompressed file
6- Verify the checksum with the algorithm and value in the meta file matches
for the decoded/decrypted/decompressed file
7- Set the filename of the decoded/decrypted/decompressed file to the
filename specified in the meta file.
Hope that helps somebody
46487 466780 252994 376409 96920 39622 205366 244315 622115 512361 668040 63608 259203 955314 811176 652718 166330 23922
Insecure? Rancid tabloid hyperbole more like.
I like NanoURL but it scratches real easily.
Pretty soon you'll see someone trying to use this as their backup system for 30gb of pr0n. Will large files kill TinyURL? What kind of latency is this going to introduce? If nothing else, this might constitute a DoS attack on TinyURL.com (which would be illegal.
It's still interesting work.
$nice = $webHosting + $domainNames + $sslCerts
Bwahahahahaha.
is it another way to backup more porn/mp3s online?
Manojar - pronounced like Manager
Gmail Filesystem
That's pretty much what I understood.
I adore the ingenuity (correct spelling?) of the hack but... I can't really find a problem this hack is a solution for.
As a way to distribute files, it's probably too slow. The pro's I see here: the file is not stored as one single file but it's stored as a distributed file (a set of Base-64 encoded clusters), making removal of the file hard. On the other hand, if one single segment drops out, the file will be destroyed (except if some redundancy exists, of which I did not find evidence).
If you want to send attachments in an e-mail, this is a very complicated way to do it. Every receiver must have the decoder program to re-assemble the file.
Moreover, if tinyURL builds in a check to see whether the submitted URL exists (not just some 404 page), the whole concept would probably break.
Anyways, very clever hack!
--Use ant to make
But overall 'WHY?' must be the question? Al Quaeda or The Real IRA? They still have their old working communication channels. Also who needs space like this? Space of this amount could be made redundant and available by using GoogleMail, Yahoo and Hotmail in synchrony. If none of those are available, presumably you'd have it on USB key as well.
[% slash_sig_val.text %]
I noticed that the whole of Alice in Wonderland is compressed to just 20 clusters and each cluster is represented by the five-letter keys used by TinyURL. So is it not possible, using the same method, to reduce the entire metafile (which is merely a textfile of less than 1kB) into a single-line URL? Then you can have the program retrieve the metafile from the URL and the actual file from the metafile. So instead of sending people a metafile, you can just copy and paste them one line of URL.
If you want your online app to not be used by scripts such as this, implement a CAPTCHA. Sure, people could still use it if they wanted to input a bunch of letters for every single chunk of their file...
It's better to vote for what you want and not get it than to vote for what you don't want and get it.
- E. Debs
Encryption, clusters, nano.. Why don't they encode the tiny clusters to DivX while they're at it and embed it in the silences in movies... :)
This sounds like a very cool conference, are they going to distribute a conference program in pdf format, or is Phreaknic too underground for that, and require you to get it off torrent ??
~jennifer.k~
That's what alt.binaries.* is for.
Deleted
Sure, well all know PHP is far from synonymous with security, but this seems to be a case of exploiting a web app using only the mechanics intentionally made available, just in a novel way. Seems like some unfounded (though not necessarily undeserved) PHP bashing.
It is a nifty hack, but let's not kid ourselves and pretend this is anything new, or that it's even a good thing.
At its core, Tinyurl is just a write-once database. You add data and get back a key/pointer to said data. As with typical databases, the size of the pointer is logarithmic in the size of the input (* number of keys stored, not bytes; however, the number of bytes/key is bounded under some constant, so it's effectively the number of bytes).
This gives us a logarithmic compression scheme, where our compression ratio (N-logN)/N approaches 100% as N gets large. This kind of "infinite compression" is what makes the method attractive: you put in say a kilobyte of data and get out a (currently) 5 byte key. All you have to do is keep an index of the keys.
TinyDisk doesn't seem to do this, but you could then turn around and store the index as a key. Take 1000/5 = 200 keys and get back one key. Lather. Rinse. Repeat. In the end, you have a single key that points to the backup of your mp3 collection, all in one TinyUrl! Not too shabby. After all, it's free storage, right? Wrong. Someone ends up paying for the infinite compression. In this case, it's Tinyurl. If this kid had stopped to think for a few minutes before publishing his hack, he would have realized that he's actually doing a malicious, antisocial thing. I suspect there will be a dozen copycats in the wild before the end of the day.
Farewell TinyUrl, we knew ye well.
The end of TinyURL is in sight. Yes, this is (probably) a clever hack.
But this is a misuse of a really useful service.
When TinyURL's administrator has to either go out and buy his
second 2Terabyte disk array in a week or shutdown, which do
you think he will pick?
Here is a video of Acidus's presentation. If you haven't seen him present before (At Hope, O'Reilly's E-Tech, Toorcon, Phreaknic, Interz0ne, etc, etc) he puts on a good show.
The presentation was called: Layer 7 Fun: Extending web applications in interesting ways. He discusses how traditional web applications work -vs- "new" web ppas that use AJAX. He talks about writing extensions to web apps using an API supplied (ala Housingmaps.com, or chicagocrime.org). Finally he talks about writing an extension to a web app where you don't have access to an API. TinyDisk was a case study for writes these so-called "non-sanctioned" extensions. He has a funny little slide he goes back to about how to properly implement a web app (which TinyRUL fails to do). Things like "don't wallow users to uploaded arbitrary amounts of data directly into your database."
Funny Stuff. His upcoming talk at Shmoocon seems pretty cool too.
In the Recommended reading section this is stated:
There are definitive works in certain fields that online guides and HOWTOs cannot even approach in terms of detail or quality. It's a class of books that are so familiar people refer to them by nicknames instead of by full title.
Well maybe so, but I did not know them all, and in the interest of helping people along the path here they are:
Books like:
K&R, The C Programming Language by Brian W. Kernighan and Dennis M. Ritchie
The Dinosaur Book, Operating System Concepts by Abraham Silberschatz
Knuth's never-ending story, The Art of Computer Programming, but Donald Knuth
The White Book, Introduction To Algorithms by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Cliff Stein
P&H, Computer Organization and Design The Hardware/Software Interface David Patterson John Hennessy
The Illustrated's. TCP/IP Illustrated Series (The Illustrated's) - W. Richard Stevens
The Rainbow series. U.S. DOD Computer Security Series
Have you Meta Moderated t
Sorry to be Mr. Obvious this morning, but I take issue with submitter's conclusion that TinyDisk illustrates a security issue on the part of tinyurl.com. It rather illustrates the ease of creating a leachable web app that resource pirates can abuse. Yes, I have a negative opinion of those using such a creative hack against others who provide services to the general public in good faith.
-- @rjamestaylor on Ello
The underlying message is that web application development is inherently difficult to secure, despite PHP's valiant attempts to protect programmers from themselves. This is the opposite of PHP bashing. It's PHP apologetics.
I disagree with the article's premise. It seems to me the same sort of mindset that attributes to "pilot error" aviation incidents that would better be attributed to poorly designed instrumentation.
Take the list of cluster URLs. Concatenate them into a single URL. Submit it again. Thus compressing literally ANY file to five characters.
At least, as long as the possibility space of five-character URLs isn't exhausted. It's very much first come, first served.
qntm.org
-Eric
SJW: Someone who has run out of real oppression, and has to fake it.
Way to go Acidus!
I guess once this goes down, I'll have to go back to posting UUencoded files in peoples blogs.
PayPal UK used to use the same technique to validate you were the owner of a credit card: make a small reverse payment and ask you how much it was. Then they got proper integration with the banks and stopped doing it like that.
And WTF is this modded 'offtopic'?
looks like an implementation of Michael Zalewski's Juggling With Packets concept, the storing of data in buffers of publicly available services for use as a filesystem.
News for merdes. Shit that matters.
Ask me about my sig.
I wonder how much longer tinyurl will last once people start dumping gigs of porn and pirated movies onto it, broken into millions upon millions of fake urls.
Sounds like fun! Where do I sign up?!
Where's the security issue?
I could make a similar "file system" with any website that allows users to create accounts and post some information online. I could split a file in some packets, create a Slashdot account for each, and post the packet's content and some meta information in that account's journal.
It's not a file system, because it doesn't even closely implement any POSIX semantics. It's more like P2P, without the P2P :)
Also, I don't really get what's so special about this. I mean, you could also UUEncode files and post them to a forum (or even different forums), and find some way to reassemble these files.
Spread it across a bunch of servers, and as soon as you have enough parts rebuild the file. Maybe use a meta search engine to find the parts.
"Can there be a Klein bottle that is an efficient and effective beer pitcher?"
Only corporations should act malicious and antisocial? That doesn't seem very fair. Why are you picking on the little sociopaths? At least it's possible to do something about them. Once you get a large corporation interested in a new form of plundering, it's all over.
I don't really see this as abuse as much as the fundamental flaw in providing free services. If the cost to support the service is higher than the cost to the recipient, it's just a matter of time before someone finds a way to cut their costs by increasing someone else's.
Someone sets up a nice public service, and some asshole has to come along to abuse it. This isn't clever, it's no different from any other act of mindless vandalism. Society is held together by an implicit contract that people not act like selfish pigs. The guy who releases this software is a selfish pig.
Yes, I RTFA, and looked at how things work, the fact that PHP is being used is immaterial.
The basic functionality of TinyURL, NanoURL or any other service is to accept a string (presumably a URI) and return a shorter string that will serve as a pointer to it. If you want your application to accomplish that it doesn't matter what it was written in, people can store things other than URLs in your database. The protections against this sort of use/abuse suggested in the article are also language independent.
paul reinheimer
He's acting antisocial by creating an untimely demise of a free service that others have thus-far been able to use cooperatively. Sure, TinyUrl could go to a CAPTCHA system without affecting its intended users, but will they? Or will they just shut down when they get a few GB of new URLs today? If his true intent was to get them to plug the hole, he should have gone to the admins of TinyUrl and explained to them the problem. Demonstrating it on a global scale is not a way to win friends. It's antisocial.
My slashdot CAPTCHA word is sadness. How apropos.
From the TinyDisk FAQ:
Q: This damn thing doesn't work on large files! #@%& You!
A: Did you not read the manual? Man I wish I could punch you in the face over TCP/IP! Change the config file's MaxSize line. By default the limit is 2 megs.
Sure, but I think it's a pretty dumb idea because of the large overhead (in time and data) of actually retrieving that data.. http request and response, encoding, etc. And the fact that tinyurl will (rightly) kick your ass off the service once he's on to you.
oooooh TinyDISK..... nevermind
Is there an open source "human filter" for web aps (like the kind that makes you type what is in an image of noisy and/or distorted numbers and letters?
I had a similar idea a number of years back basically masquerading uuencoded files inside of bogus html files that get crawled by Google's caching bot... the idea being that if you knew the names of the files you could query the cache and retrieve the UUEncoded bits.
At the time, no one else had written about such things. I just never got around to automating the process, so it never really materialized. Maybe some brave and time-rich soul would like to give it a go?
"This above all, to thine own self be true"
SDFS010101M5&AI2!$3D$@8G)O861C87-T3:6YG('-YW1E;2X@ ($1U8VLN"FAE`
Writing captchas is super easy in PHP. Well, depending on how complex you want the captcha to be, of course. You can make a basic one in 5 minutes, though. Those things don't really stop people, but they do make things much more difficult.
why don't you just go listen to the presentation. First of all, he's not just a kid... he does review of web app security as a job. Thanks for bringing in that honored tradition of ageism. Secondly, He beleives he already found someone else doing this because he stumbled across several random blocks of data during his tests. THIRD, storing large ammounts of data this way is highly impracticle because of the lantency of retrieving files like this.
;P
This was the only presentation I got to catch most of because I was running my ASS off doing AV for the con this year
He doesn't want to hurt tinyurl, but I doubt that there will be long term issues for that service because of it. It's simply too much of a pain in the ass to use for anything other than data hiding.
Can't TinyURL just put a filter on input requiring a URL to be in the form of http://stuff.tld/stuff/stuff?
-- There's only one replacement for displacement.....
If TinyURL didnt like this, seems like they could trivially add a check to see if URL's they are given actually exist by accessing them. If they get a 404, dont accept. For those sites that give a redirect instead of a 404, follow the redirects until they get to a page that actually answers, and use its URL instead.
Perhaps you should have read the sentence immediately following the one you quoted:
After all, it's free storage, right? Wrong.
He wasn't suggesting it was a good idea to do it - he was giving a sample mindset of someone who would use TinyDisk to do stupid/malicious things.
In my analogy the programmer is the pilot and the programming language is the instrumentation. The goal is not to create an environment from which you cannot build an insecure application any more than instrumentation's goal is to make it impossible to crash the plane. The idea is to make it easy to see if you're at low altitude, or in the case of web apps, to see if you're not checking your inputs, for example.
http://www.yahoo.com/?weruiosahfsroiwehrfsafayewri uweiuryasuhfiudsahfiuyrfyeqwiuryiuwqyfdiusahdfsadh fguyyertqewrywiquyeriahsfhdsifgetryeqrriugsidfhgsa dgfhiuywiuegrfwtegfyags
This would have been a straightforward feature to copy/adapt into PHP if anyone were interested in making it a decent server-side web language.
Don't say "just like in any other language" when you're unaware of languages out there that are more suited to web applications.
If TinyURL didnt like this, seems like they could trivially add a check to see if URL's they are given actually exist by accessing them. If they get a 404, dont accept. For those sites that give a redirect instead of a 404, follow the redirects until they get to a page that actually answers, and use its URL instead.
T AGOESHERE
Rather than a 404
Two flaws. (1) It is possible to create a loop of redirects. Of course, the solution on TinyURL's end would be to follow an arbitrary number of redirects, and declare anything that redirects more than that an invalid URL
(2) There are probably thousands of webapps that will give a valid response to URLs like the following: http://www.example.com/cgi-bin/script?ARBITRARYDA
(2a) Barring that, why not just set up your own web server that returns a HTTP 200 for any URL? Hell, you don't even need to go that far, you could probably write a single line of perl that listens on port 80, and returns an HTTP 200 along with a small document in response to any HTTP request.
Simple solution, encode all your information as google searches, liken d+seven+years+agob rought+forth+on+this+continent
http://www.google.com/search?hl=en&q=four+score+a
http://www.google.com/search?hl=en&q=our+fathers+
My amazing wife - Artist, Author, Philosopher - Laurie M
Slashdot has started automatically parsing URLs! Time for me to advance my calendar to 1999.
My amazing wife - Artist, Author, Philosopher - Laurie M
This pisses me off. Yeah, it's cute, but so what? Goodbye, TinyURL, I knew you well. Too bad you're about to be shitflooded into oblivion.
it appears that tiny url has gone DOWN. Well, that was fun while it lasted...
Meta-mod to the rescue!