Slashdot Mirror
Storing Very Large Files On Amazon's Unlimited Cloud Photo Storage
AmiMoJo writes: Last year Amazon started offering unlimited cloud storage for photos to customers who subscribed to its "Prime" service. Japanese user YDKK has developed a tool to store arbitrary data inside a .bmp file, which can then be uploaded to Amazon's service. A 1.44GB test image containing an executable file uploaded at over 250Mb/sec, far faster than typical cloud storage services that are rate limited and don't allow extremely large files.
This is why we can't have nice things.
Now they'll be changing it
almost a Japanese Zipper: YKK
Thanks for the post, I'll have to check out.
This is blatant misuse of the service. Amazon will change its terms and restrict or eliminate this service to counter this. Thanks for fucking it up for everyone else.
My org had dozens of videos housed at Viddler.com's "free hosting" while it lasted. Viddler had trouble being free a couple of years ago and sent a big bill we couldn't pay. When we asked were our videos deleted, Viddler tech support said they existed... somewhere... in Amazon.
Gently reply
Awesome, thanks. This is really informative.
A better tool would be to split the data among smaller files. A 1.44 GB BMP is sure to attract attention. 1440 one MB jpegs isnt. Am I right? Peeps?
I'll get right on that learning to read Japanese..
First the article with the luser asking help desk question and now this with the link in Japaneses.
I think that with the new overlords Timothy has gone full honey badger on us.
-- I have a private email server in my basement.
That's great. And exactly how long do you think Amazon will allow this to go on before:
a. Amazon runs a script to test that file magic matches extension or delete?
b. A limit of 20MB per file is established?
c. The free service gets a 5GB cap; want more then pay?
d. Amazon shutters the service entirely?
This sort of crappy hack has already been done with other services.. A proof of concept is no longer needed. At this point you are just abusing a service to the likely detriment of everyone.
Fuck YDKK!
Here is my research...
Steganography & Amazon Cloud Drive:
http://bsmuir.kinja.com/stegan...
Seems quite complicated.
If Amazon doesn't convert the images, he could just upload a PNG file with a lot of information stored in ancillary chunks... the png specification even allows creating custom/developer chunks which should be ignored by any parser that doesn't understand them (for compatibility with future versions of the standard)
For example, just abuse the hell out of iTXt or zTXt chunks in the format : http://www.libpng.org/pub/png/...
For private chunks, see this bit : http://www.libpng.org/pub/png/...
For fucking it up for the rest of us!
Back in the day, when I worked as a dev at a social networking site, we would resample old photos that hadn't been accessed in over some threshold (let's say it was 1 year, for the sake of argument). Anything older than the threshold would get re-encoded in JPEG to a poorer representation in order to save storage space.
So what stops Amazon from doing the same thing? Do their TOS say they won't?
Non-image data under those circumstances become pretty much useless, even if packaged so that they appear to be an image of off-station TV reception. Once you include a lossy recompression, your data are no longer data, but noise for real.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
Do you want new terms of service? Because that's how you get new terms of service.
If you take the trouble to read through Amazon's TOS, and click to their actual rates, you can buy unlimited storage for photos, videos, AND ARBITRARY FILES for only $60 per year. Not only that, but Prime gets you 5 GB of videos and non-photo files for free.
Going through all the hassle of specially encoding your data files so that they masquerade as photos seems like a heapload of time better spent earning $60 so that you don't have the long-term headaches and potential for being banned from Amazon's service that such abuses flirt with. You want a real backup service? Buy it, it isn't expensive.
Backblaze, a darling of Slashdot, is only $50 per year. It isn't worth the hassle or time to beat the system for such low prices. Amazon Glacier is $0.007/GB/month. Both systems offer encrypted storage. Why work hard when someone else has done the figuring out for you?
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
They want us to lose data. Want us to lose data.
That is how they be.
Working for him makes me want to die, but I have to support my family. If this country had proper safety nets I could flee.
Y mother was raped to death at Denny Park not far from Amazon so you just know it was one of Bezos's employee.
My sister died there. The SPD is crooked and lets people use drugs in that park.
This remembers me old times with aol servers and kamaleon http://www.abcdatos.com/programa/particionar-archivos.html
This software cuts (and joins) files masking them with jpg files
After he decided to shutdown our AWS account without wanting to put nearly 200 people out of work for no reason than his own personal pettiness since our CTO was quoted in the NYT wrt an AWS problem, I'll believe anything about that evil, hateful man.
Downloading 13GB compressed backup files from Amazon S3 is incredibly slow at only 938 kilobytes per second using their AWS CLI tools... even slower at 214.5 kilobytes per second using their AWS Powershell tools.
Reply to your own AC posts on Slashdot much? Also FYI, no one cares.
He hates us. It's as simple as that.
I have a fast PC and connection, and the load time for Amazon Cloud sites is VERY slow.
TIFF/AIFF are much simpler atom-based formats than BMP. Nothing stops you using your own atom type so as to avoid breaking existing TIFF software, either.
They do nothing about drug use there. People are dying, but they refuse to make arrests because they don't want the number to tell the whole truth about how dangerous it is to work at Amazon.
Rather than scramble data into a BMP, why not use the built-in expandability of a format like GIF? We put that scalability in there in the late Eighties for a reason. Not necessarily this specific one, but we did try to be forward-thinking.
Source: a 7000x-series PPN
From the soon-to-be-banned dept.
Those days are dead and gone and the eulogy was delivered by Perl. --Rob Pike
And Bezos doesn't want the true crime numbers here real eased.
1.44gb? And no-one got the reference?
There was a similar tool, featured on a site that shall not be named, that did this last year: https://github.com/tylerpitchf... . It chopped files and directories into user defined bitmaps according to the write up. Made for Google, but worked for Amazon too apparently according to the write up from the article. https://www.linkedin.com/pulse...
GmailFS wrote blocks as individual emails. You could mount it through FUSE. https://en.m.wikipedia.org/wiki/GmailFS
With Unlimited files, you could vary your block size up to 1.44GB, mount it, and write anything you want infinite in size. Not to hard at all. I/o would wrap/unwrap blocks in your choice of image formats. Compressed file system: no prob with JPEG or others.
Just wait until they turn on their automatic convert-to-low-quality-JPEG functionality. :) All your .BMP files will be converted to 400 KiB .JPG files. Hope your executable is OK with lossy compression.
http://www.petitcolas.net/steg...
"MP3Stego will hide information in MP3 files during the compression process. The data is first compressed, encrypted and then hidden in the MP3 bit stream. Although MP3Stego has been written with steganographic applications in mind it might be used as a copyright marking system for MP3 files (weak but still much better than the MPEG copyright flag defined by the standard). Any opponent can uncompress the bit stream and recompress it; this will delete the hidden information â" actually this is the only attack we know yet â" but at the expense of severe quality loss.
The hiding process takes place at the heart of the Layer III encoding process namely in the inner_loop. The inner loop quantizes the input data and increases the quantiser step size until the quantized data can be coded with the available number of bits. Another loop checks that the distortions introduced by the quantization do not exceed the threshold defined by the psycho acoustic model. The part2_3_length variable contains the number of main_data bits used for scalefactors and Huffman code data in the MP3 bit stream. We encode the bits as its parity by changing the end loop condition of the inner loop. Only randomly chosen part2_3_length values are modified; the selection is done using a pseudo random bit generator based on SHA-1.
We have discussed earlier the power of parity for information hiding. MP3Stego is a practical example of it. There is still space for improvement but I thought that some people might be interested to have a look at it."
If true, this is the kind of thing that would generally put a decent dent in customer trust, AWS-wise. NYT link?
I mean, I love hating on rich people too, especially those used to puff the "self-made" or "class mobility" propaganda, but let's get some data here.
It's one thing to abuse the service, and use it differently than intended to personal gain.
But 1.44GB is not even what I'd consider very large. I'm yet again the victim of click bait headlines. I was expecting sizes in the TBs.
Back in school one of our tasks in our informatics course was to analyze and explain an algorithm which hides arbitrary data in the lowest value bits of 24 bit bitmap files. We did that in Turbo Pascal (o; It was very interesting to see this is possible. Of course, it were only text messages we hid there, as storage space was rather limited, to put it mildly.
Link or it didn't happen
Why are people griping about what this guy did? So he cheated the system. All of us here have cheated the system in one way or another. The real issue is that Amazon will now go back on it's work with the Unlimited photo storage. That's going to be the real problem, instead of them finding a way to prevent this and punish those that abuse the system. They just punish everyone instead. That seems like a useless learning lesson. This is just a vicious circle, as it happened with the unlimited 3G data until people tethered to it and ran up the data. Why not just find a solution to the problem instead of punishing everyone?
I personally wrote a steganography tool for JPEG-2000 files for a graduate school project - it just stored data in the least damaging sections of the file. The resultant files were still perfectly legal image files, lossy compressed, and minimally visually damaged.
Kudos for the hands-on. I was fascinated some years ago with progressive GIF overlays and coded some stuff to produce them, not so concerned with stenography and hiding the presence of a message, but more with novel ways of presentation.
One example was embedding a public key into a GIF image. Starting with a standard base image and palette that was the same for everybody, like a shiny golden key floating over a smooth blue gradient... the key bits encoded as a series of overlays that when displayed, made the key sparkle and the background vary in color, all happening over ~10 seconds. The idea was that while most people didn't stand a chance memorizing much "BEGIN PGP PUBLIC KEY BLOCK" gobblegook, we'd be better equipped to remember the distinct "sparkle" of an image. More of a style thing than a useful crypto concept.
I also experimented with things like encoding process/memory access and toyed with the idea of filesystem journals rendered as displayable GIFs. It was a fascinating foray into the realm of data structures and helped me to become the person I am today. I presently jet sewers for a living.
Wouldn't it be strange to see some future Slashdot shocker headline, "Bit Rot Discovered In Cloud, All Data Will Be Reduced to Gaussian Noise By 2030". And like the proverbial boiling frog we deny the problem or postpone dealing with it as everything progressively (but slowly) dissolves into static. People who try to raise consciousness and alarm are booed off Slashdot with comments like, "I can read it. What's wrong with you? posted by folks who are also having trouble reading things but enjoy sniping at others more. Then as it reaches the final stages all electronic mediums are projecting mostly static but people are pretending they see and understand the messages perfectly. And most oddly, when we hit Peak Gaussian something resembling a modern society continues to function. Then unfettered by structure society literally melts into phantasmagorical goo. Something... like... THIS.
<blink>down the rabbit hole</blink>
Tivo used to distribute some data at night on a TV channel. I caught it one night in a fit of insomnia, it looked like a video stream comprised of QR codes. I'm guessing the Tivo box recorded it and then decoded the full frames and then stored whatever the data stream was.
Like QR codes, the "data" would seem fairly impervious to scaling and resampling provided that the "bits" or white/black blocks were large enough to survive downsampling. You wouldn't really care if they converted them to compressed image data because the image was the data but represented at a low enough practical resolution that downsampling or format conversion wouldn't change the image enough to inhibit decoding.
You could even do something like the color-enhanced HCC2D code "extension" of QR codes for greater image data density.
Each image file could then be a rough equivalent of a disk block or sector, allowing the client side to manage a file system of sorts.
to get around an overbearing corporate firewall that forbade not only executables, but archives containing executables as well. In order to be able to e-mail new versions of a program that the overbearing company had bought, he wrote a program that packed the .exe code in a BMP file.
I thought we could already do this? I remember hiding .rar archives in .jpg images. Is Amazon able to detect this magic?
I teach graduate CS courses at a university, and we get the occasional cheater. Sometimes, the cheating is blatant three students just turned in exactly the same work. However, there are occasions where we suspect cheating, but they did a good job of disguising it. Of course, they do poorly on exams. If those students would spend their time and energy on learning the material, they would learn something and get a good grade.
Yup $50 a year is a great & easy solution - ....And didn't folks try doing this with GMail back in the day? Google offered unlimited email so somebody figured out a method to "uuencode" their harddrive backup and email it to themselves? Kind of like porn back in the nntp news-group days?
People are having fun building Rube Goldberg machines. Let us all doubt that this is a serious commercial solution - and just admit it is a run "built it on a Raspberry Pi" toy solution.
Amazon knows how much storage each account is using up, and what sort of access pattern it uses. Only a small number of accounts will take advantage of this to the point where it's problematic, in the context of AWS as a whole. For those accounts, the system can just alert the team that someone is probably gaming the system, and then an actual human can look at what's going on and whether it's violating the TOS, then make a decision. They probably don't even need to be able to see the actual file content to make reasonable decisions.
A BMP consists of basically a simple header describing the file and the raw contents. I have done this several times to show i.e. mistakes in encryption usage concepts (for example, to see the startled face of students when showing them the effect of using ECB when encrypting an image with repeated patterns). Where's the novelty in that?
I could not read TFA since it was in Japanese. From Amazon.com's pages:
About Prime Photos: http://www.amazon.com/gp/help/...
In addition to the unlimited photo storage, you will also receive 5 GB of free storage space that can be used to store videos and files we canâ(TM)t recognize as photos.
Certain photo formats are excluded. For more information, go to Cloud Drive Photos & Videos File Requirements.
So apparently they decide what is a photo. Myself I'd not trust a third party to not degrade the quality; I'd opt for encrypted container with photos INSIDE it. The same page also restricts this to "personal use":
Prime Photos is for your personal, non-commercial use only. You may not use it in connection with a professional photography business or other commercial service.
Personally I think that sucks. By comparision, my VPS provider gives me a very cheap VPS which I can use for whatever purpose I want as long as I do not break any laws or disrupt other users. They price based on performance and bandwidth; not arbitrarily created market segmentations.
Cloud Drive Photos & Videos File Requirements: http://www.amazon.com/gp/help/...
Photos and videos you upload through your web browser on the Cloud Drive website must be 2GB in size or less.
File and folder names must contain less than 255 characters, and cannot include the incompatible characters listed below.
They list common supported formats; this includes RAW. And they do mention encryption:
For photos: JPEG, BMP, PNG and most TIFF files (these files typically have the .jpg, .jpeg, .bmp, .png or .tiff extensions). In addition, some RAW format photos can also be viewed. For more information, go to About RAW Photo Files.
For videos: MP4, Quicktime, AVI, MTS, MPG, ASF, WMV, Flash and OGG.
Note: The unlimited photos storage benefit for Prime members only applies to files recognized as photo files. Photo files that have been encrypted before they're uploaded will count against your storage quota.
About RAW Photo Files: http://www.amazon.com/gp/help/...
Nikon (NEF files) - Nikon D1, Nikon D1X, Nikon D4, Nikon Coolpix A, Nikon E5700, Nikon AW1, Nikon D800, Nikon D50, Nikon D610
Canon (CR2 Files**) - Canon 5D, Canon 1D, Canon 1D MarkIIN, Canon Rebel SL1, Canon 60D, Canon 5D MarkIII, Canon 1D MarkIV
**While Cloud Drive recognizes these files as photos, some of the information associated the file (like the time and date the photo was taken) may not be recognized.
Sony (ARW files) - Sony A7, Sony A7R, Sony A6000, Sony NEX-5T, Sony NEX-3N, Sony NEX-6
I doubt RAW format pictures can be compressed lossily? Does anyone know this for a fact?
No lossless formats, images will be slightly recompressed. Free is free, what do you expect.
Liability is limited to $50, so after you spend months transferring your files over your limited upstream bandwidth, they can delete your files for whatever reason, including that they just don't want to encourage people "misusing" their service in this way. You complain, they hand you $50, and they're done. Arguably, they can also do that for the Unlimited Everything service, so even if you pay, they can terminate service for anyone they're not making money with. Ultimately, your files are only as safely stored so long as its cheaper to keep 'em than to delete 'em. That's the free market golden rule.
This is hilarious.
Ahem. I uploaded about 1 TB of files to Amazon Cloud Drive and it only took a few days. Not sure where you get the idea it takes "months transferring your files."
Kriston
Ahem. I uploaded about 1 TB of files to Amazon Cloud Drive and it only took a few days. Not sure where you get the idea it takes "months transferring your files."
1TB is hardly stretching the definition of unlimited storage. How about 100TB?