Slashdot Mirror

← Back to Stories (view on slashdot.org)

Ask Slashdot: Cloud Service On a Budget?

Posted by samzenpus on from the throw-a-little-money-at-it dept.

First time accepted submitter MadC0der writes "We just signed a project with a very large company. We are a computer vision based company and our project gathers images from a facility from PA. Our company is located in TN. The company we're gather images from is on a very high speed fiber optic network. However, being a small company of 11 developers, and 1 systems engineer, we're on a business class 100mb cable connection which works well for us but not in this situation. The information gathered from the client in PA is s 1½mb .bmp image, along with a 3mb Depth map file, making each snapshot a little under 5 megs. This may sound small, but images are taken every 3-5 seconds. This can lead to a very large amount of data captured and transferred each day. Our facility is incapable of handling such large transfers without effecting internal network performance. We've come to the conclusion that a cloud service would be the best solution for our problem. We're now thinking the customer's workstation will sync the data with the cloud, and we can automate pulling the data during off hours so we won't encounter congestion for analysis. Can anyone help suggest a stable, fairly price cloud solution that will sync large amounts for offsite data for retrieval at our convenience (nightly Rsync script should handle this process)?

8 of 121 comments (clear)

  1. I'll be the one to say it... by atari2600a · · Score: 4, Insightful

    ...WHY are you using BMP in the first place? Does whatever you're generating these on not have the processing capability to compress to PNG before transferring? I mean it SOUNDS like it'd save 10-20% off the total transfer...Anyways, what I'd do is I'd simply plop a server rack at the source that takes all the images for a given hour or whatever, tar.gz.bz2.whatevers them & send them over. Otherwise, I mean, Amazon wouldn't be TERRIBLE?

    1. Re:I'll be the one to say it... by FishOuttaWater · · Score: 4, Interesting

      Yes, your first line of defense is to examine what they need as far as these images, and that will tell you how far you can go in reducing their size for transmission and storage. Can they be scaled down? Can they be lossy? Can you take some time to run a more effective lossless algorithm on them? Is there redundancy between images? Secondly, do you have to move the whole image? Can you do your work on a lower quality image to define the series of steps required and then apply those steps remotely at their location? Just think real hard about what the requirements are, and don't rush yourself. You may come up with your best ideas in the shower on this when you have time to think outside the box.

  2. Sounds like... by cultiv8 · · Score: 3, Interesting

    the sales guy oversold your capabilities. Instead of asking about cloud options, why don't you just pick a server host with a good reputation (Amazon and Rackspace come to mind) and pass the costs onto the client?

    --
    sysadmins and parents of newborns get the same amount of sleep.
  3. Snail Mail and a hardrive by duckgod · · Score: 4, Informative

    Assuming you don't need real time analysis(doesn't look like it from problem description). Send a couple 500gb hard drives and have someone mail you the daily load of images each day with overnight shipping.

  4. Re:BYOS by BobC · · Score: 3, Funny

    Is the data being generated 24/7? If so, that's 432 GB/day, pretty much exactly 12 hours worth of your 100 Mbps bandwidth. So some spooling is needed, but why in the cloud? The main goal would seem to be avoiding paying twice to move the data, so you'd want to avoid using through a 3rd party if at all possible.

    1. The simplest solution would appear to be to put a laptop with a 500+ GB HD at their facility. A laptop because it essentially has a built-in UPS, and the CPU can sleep much of the time.

    2. Develop a relationship with whoever provides their bandwidth. Find the nearest peering point. Put a laptop there.

    3. Get the NSA as a client, do some sysadmin work for them. Your data will be RIGHT THERE!

  5. Re:BYOS by Chrisq · · Score: 4, Funny

    [I would use SSDs in a metal padded case knowing Fedex].

    Fedex is like UDP, an unreliable delivery service. In fact there is only one fault of UDP it does not duplicate. Things can arrive broken, out of order, delayed, or not at all but I have never heard of Fedex delivering multiple copies!

  6. Re:BYOS by Anonymous Coward · · Score: 4, Funny

    I have. Sometimes Amazon messes up. This is how I have a copy of XCOM. :)

  7. 5mbytes every 3seconds is only 13.333 mbits/s. by millisa · · Score: 4, Informative

    we're on a business class 100mb cable connection
    100mbps = 12mbyte/s (give up 15-20% for the packet overhead, 10megabytes/sec).

    Distilling that summary into the data that mattered:
    1.5mb image, 3mb file each under 5 megs.
    and
    images every 3-5 seconds

    The files are 5megabytes total.
    In a perfect world, they'd transfer in 0.5 seconds.

    Leaving 2.5 - 4.5 seconds for the porn.

    Let's assume they are the bigger size, 5megabytes, and they transfer in the more frequent number, every 3 seconds.
    5MBytes/3s = 1.66667 Mbytes/s = 13.33333 mbits/s.

    Why is a facility with a 100mb/s line incapable of handling this?
    How did a problem where a 100mb/s line can't handle 13.3333mb/s come to a conclusion of "Fix it with the cloud?"

    In any case, if you want to do a cloud setup, just about all of them will handle small 13.3mb/s constant rates and you'll pay for it more than if you figured out why your line isn't keeping up.