Slashdot Mirror

To see the book pages of ULIB, please dowload free TIFF plugin or DjVu plugin

Watermarking is a process that happens when imagery is pulled from the image server, likely JPEG 2000, and occurs regardless whether the imagery was requested with the Google Earth application, the Google Earth API, or any other API that knows how to request imagery from the image server. The imaging server application is likely either written using ER Mapper's ECW JPEG 2000 SDK (a GPL-like open source license for images under 500mb - i.e., Google Earth images) or LizardTech's MrSID proprietary server. I'm guessing unless Google wrote their own imaging server, they chose ER Mapper's (I would), it's more flexible and written to JPEG 2000 standards.

"What formats allow an easy mix of image and text data (without formatting)?"

http://www.lizardtech.com/

DjVu is what some libraries use and there's some free software out their.

BTW there are legal size scanners out their. Shop around, plus don't forget to check manufacturers websites for refurbished and discontinued models.

(Of course, you will still need to spend lots of time scanning, naming and classifying those pages. The ADF and 10yo nephew suggested in another post might be useful for that.)

DjVu offers very compact representation without the need to OCR the document (I've converted a 13 megs scanned PDF into a 600K DjVu which was much faster and easier to read), and optionally a "hidden text layer" if you want to OCR it to make it searchable.

Nadaou is on the right train of thought. If I can offer the following:

Hardware
Stick with consumer Garmin, etrex units are small (easy to conceal), easy to use and affordable.

Software
Consider ESRI's free ArcExplorer. It supports MANY data format, very simply to use, and is cross platform.
http://esri.com/software/arcexplorer/index.html

Data
I highly recommend Landsat satellite imagery, specifically Landsat GeoCover 2000. It is free, global coverage at 30 metre resolution, compressed, yada yada, trust me, it is exceptional!
Get it here:
https://zulu.ssc.nasa.gov/mrsid/
http://glcf.umiacs.umd.edu/portal/geocover/earthsa t.shtml

These MrSID wavelet compressed images can also be viewed using a free geoviewer from LizardTech:
http://www.lizardtech.com/download/dl_options.php? page=viewers

Do as much prep work at home before going over there. I would suggest printing maps with UTM grid on them (this is the grid that the GPS can tell your position in) - I've always found this more useful then Lat-Long (how far away is 7 seconds?!?). Essentially build one map at a good logical scale like 1:10,000, then PAN & PRINT, PAN & PRINT, repeat as necessary.

They have another imaging technology that they purchased from AT&T called DjVu. They've Open Sourced the viewer for that technology under the GPL.

I believe an encoder/decoder is also available under a GPL license, though LizardTech doesn't appear to be happy with the GPL because they are pro software patents, and the GPL is not. The encoder/decoder may or may not be a fractal engine, someone more knowledgeable will have to answer that question.

LizardTech may be involved in a squable over the JPEG2000 technology. Something to do with patent litigation.

= 9J =

They have another imaging technology that they purchased from AT&T called DjVu. They've Open Sourced the viewer for that technology under the GPL.

I believe an encoder/decoder is also available under a GPL license, though LizardTech doesn't appear to be happy with the GPL because they are pro software patents, and the GPL is not. The encoder/decoder may or may not be a fractal engine, someone more knowledgeable will have to answer that question.

LizardTech may be involved in a squable over the JPEG2000 technology. Something to do with patent litigation.

= 9J =

Does anyone know what happened to fractal image format files (.fif) and why they never took off?

Hard to implement. Patent mess. CPU requirements. No better fidelity [they have just as many artifacts as JPEG, but the artifacts are 'nicer looking']. Massive increases in available bandwidth. Fractal wierdness with editing, you always have to convert back to raster anyway.

The tech has found niche applications though, such as image scaling : Lizardtech's Genuine Fractals is pitched as an image rescaling tool, but it's basically the same tech as you're talking about. In this case, it's just a save/load plugin for photoshop. You save any file as a fractal and then when you reopen it it asks you how big you want it. 200%? 500%? Whatever, it scales it up and looks much the same, but with a strange kind of fractal painting-like effect.

The strange thing is that if you zoom in on one of these scaled images you start to see shapes that weren't in the original -- that weren't in the original scene, though at normal zoom the image might look perfectly normal. It's kind of creepy.

There is a photoshop plugin that works with images for resizing and it seems to be really good. Genuine Fractals 3.5 seems like it would be able to zoom any picture without loss of data. I'm not sure, however, if any text data would be legible, however.

LizardTech's DjVu plugin is avaialable for Windows, OS X, and OS 9 here. The Windows version does work on Firefox, although not as well as it does in IE, unfortunately.

The UNIX versions are all based off of DjVuLibre, and are not made by LizardTech. Those can be found at sourceforge.

The system isn't 800TB, but will scale to 800TB, according to this EDS press release. In fact, given that they've spent a mere $2.5M (powerpoint!) there's not a hope in hell that they've got 800TB! The powerpoint says it's a 5TB EMC SAN & an ADIC tape library for backup.

An interesting point is that they're delivering the documents using DjVu by Lizardtech, which is GPLd, and developed by the creators of DjVu in conjuction with LizardTech (after a period of LT not-getting-it). The DjVuLibre home page is here. LizardTech still have the best encoders for the format.

I'm using Firefox (from Windows sadly) and I can access the content just fine.

As for OSX and Linux users, there is a plug in for viewing the content needed. But they report to support OSX and "UNIX". The plug-in is called DjVu and has an open source equivalent at sourceforge (with RPMs, OS/2 and even Cygwin support).

Well, since LizardTech likes to sue people too

Well, since LizardTech likes to sue people too

Well, since LizardTech likes to sue people too (see http://www.lizardtech.com/company/news/ermsuit/), I don't quite see how DjVu would be an improvement over jpeg.

Sorry, I linked to a DjVu community page and not the official one.
DjVu is a trademark of LizardTech Inc (which also provides browser plugins for MacOS etc etc).

I am not a professional photographer (I know few though) but rather take photos as a hobby (and as the unofficial photographer for family events). I haven't moved to the digital side yet, but I am looking (I'm looking at a Minolta that is still in the $1000 range). For all the research that I have done, the megapixels is not the most important point, but rather the features of the camera, namely the ability to use the various lenses I already have for my SLR.

Depending on what a person is planning to do with the picture, the higher MP may create an extra step. If you take a full resolution picture, your not going to be sending a bunch of these around as email attachments. If you are posting to the web though, you could look at creating thumbnails or working with various image compression and/or "pyramid" files (check out Lizard Tech's MrSID image format, although this is more for often seen in GIS or document management environments or consider JPEG 2000).

Whle most of the market will be happy with the 2 - 3 MP camera, as long as prices continue to drop you'll find more buying the higher end cameras (which is a good situation to drive prices down even more).

DjVu also supports real text embedded together with the image, plus their position in tscanned bitmap. So you can search for text within a DjVu file. Works really neat.

And [Ff]ree DjVu viewers can be obtained for multiple platforms.

Disclaimer: I work for the company the sells the commercial version of DjVu, LizardTech

DjVu is licensed from AT&T labs, and has both a commercial component and an open source component called DjVuLibre. The technology works by analyzing documents, particularly scanned color documents, for hard edges. Hard edges typically indicate text, while smooth, continuous tones indicate background images. DjVu then "segments" the two types of imagery on the page into different layers and compresses them using different formats for optimal compression and quality.

Okay, enough marketing. While it does have some warts, it's a pretty cool technology to work with. That, of course, and I'm happy to have any job these days.

• There are other formats, like DjVu, that have certain advantages over .PDF, but think carefully before using them. Will you be able to read your files 10, 20 years from now? In .PDF's case, the answer is an unequivocal 'yes' because of widespread government, military, and commercial standardization around it. I hate to see people spend hours scanning manuals in DjVu or another nonstandard format, because I'm 95% sure I won't be able to read them years down the road on a completely different platform.
  
• To make the document searchable, use an OCR package like FineReader if possible... but expect to spend even more time babysitting the process.
  
• Experiment with your scanner resolution settings to minimize the resulting .PDF file size. There's a big difference in size between 200 dpi and 300 dpi, and between a B&W and color scan.
  
• For some mysterious, forehead-slapping reason, flatbed scanners often use glossy-white backing material in the lid. This encourages bleedthrough of text on the reverse side of double-sided material, making your scanned documents look sloppy and compress poorly. Placing a sheet of black paper, plastic, or cardboard material between your document and the scanner lid will make a big difference.

LizardTech make some other awesome products such as Genuine Fractals for Photoshop. Incidentally this is also used by DSLR photography enthusiasts to sharpen their images better than most traditional techniques are able to achieve.

It is more significant to know the image resolution than the file size. This is represented by the number of ground units a single pixel takes up. 5 meter resolution refers to each pixel being 5 square meters (obviously). For mapping grade imagery, you want at least 5 meters. The standard is moving much closer to 1 meter as companies like Space Imaging start to dominate the market.

The performance is TERRIBLE on single large images, if you plan to use it at multiple zoom scales. It is much more efficient to have a spatially indexed series of seamless smaller images. This index, known as an image catalog in the GIS world, is basically a database with image coordinates and file names. When you zoom in to a given area on the map, the geographic coordinates are used to determine which images to display.

It is possible to compress the file size using such means as jpeg, but this is not thought of very highly in the mapping world. The ideal is to have a georeferenced (has a *.tfw header file that contains the coordinates for one of the image corners) tiff file, and to add compression using a package like MrSID. People who use air photos frequently will have two datasets: a compressed one and a standard tiff. It is much easier to change a map projection on an uncompressed image.

If you have some $, there is a software package called SDE that enables high performance raster display for mapping purposes. It works really well, but you are getting locked into a highly proprietary and expensive format.

Having worked on GIS imagery before (from the Australian government), I'm assuming it's in MrSid format. The 100:1 kind of compression ratios are completely legit, it's really incredible. I believe it uses fractal compression of some sort. It's produced by LizardTech.

JPEG and JPEG2000 are great for compressing images where you care more about how it looks than preserving the actual data since they use psychovisual enhancements. MrSid does a much better job at preserving the integrity of the data at much much higher compression rates.

Hi molo. Print them with a professional digital lab. Do not take your CF cards to Wal-Mart or something. We often use Reedy or Miller's. They both do a great job, and have free customized FTP software that let you select what size prints you want, what crops you want for off sizes (i.e., a 5x7 print from a 2:3 aspect ratio file), additional features like type of paper, etc, and then upload. Your prints arrive on your doorstep a few days later. You can either have them just print the files as is, (in which case a 4x6 is about $0.35) or have them color correct the files, in which case a 4x6 is something like $1.25. We print "as is", because we've got a Gretag Macbetch Eye-One Photo calibrator, which I know is what Reedy uses to calibrate their monitors, but I don't know about Miller's. Regardless, when calibrated, the color I see on the screen is the color that comes back in my prints. There are less expensive calibrators out there you can buy, or if you can find a friend who has one, have him stop by with it can calibrate your monitor.

Also, I'm planning to buy an Epson 2200 printer which will let you print up to an 8x10. I know pros who swear by them. However, it's probably about the same cost if not more expensive per print, given the high ink and paper costs, than having your images printed at a pro lab. I just want one for rush orders.

Big prints like 20x30 can certainly be made from a 10D JPEG, but we usually rasterize them first with Genuine Fractals. So long as it's a full-frame image and well exposed, you really shouldn't have a problem printing a 20x30. For anything bigger, I'd switch to RAW and definitely rasterize.

Anyway, long story short, find a pro lab. Don't print at Wal-Mart. You can make big prints, too.

sorry.. this is a better link for document examples.. let's see an open source version of this ... drool...
DJVU samples

The quality of the image could be better.
I mean attempting to actually study this
(granted you have the need/desire to, and have polished latin abilities) you couldn't. This should be done in a methond similar to that of the MILLION BOOK PROJECT, using something like DJVU, which yes I know is proprietary, but has true readable digital document capabilites and is highly optimized (file size) for web based documents.

The flip side to your comment...

>It's *nix only
That isn't fully correct. LizardTech provides encoders for Windows, although at a price. You're right that it doesn't have much support (notably a free Windows encoder), but JPEG2000 isn't widely supported either; no new format ever is. DjVu could go somewhere, although just where cannot be known yet.

DjVu format is Wavlet Based with OCR
I have tried the Solo version, file sizes were incredibly small. A nice feature is that the Tiff is compressed to a wavlet compression technology, and if you buy the full version you can add an OCR layer which means you can text search. The sample content demonstrates the systems capabilities.

Many libraries are converting rare books and manuscripts because it preserves the original image so well.

DjVu format is Wavlet Based with OCR
I have tried the Solo version, file sizes were incredibly small. A nice feature is that the Tiff is compressed to a wavlet compression technology, and if you buy the full version you can add an OCR layer which means you can text search. The sample content demonstrates the systems capabilities.

Many libraries are converting rare books and manuscripts because it preserves the original image so well.

GIS is a vast field even if you dont throw in web access to data. narrow down your interest or hire someone to do the GIS side and you do the web access side.

web server..
MapServer - works and is mature and stable
ESRI's GIS data server is super expensive but the new version runs on Linux
PDFMap - combine this with mapserver so that your users can download maps they make
SVG - ive seen some cools things happening with SVG
MRSID - for image compression, costs for the compressor but i have built some cool stuff with thier free server.

desktop...
ESRI has a free viewer which reads a XML file. its works ok. there is a Linux version but i havnt used it yet. all of the free (not open source) GIS viewers suck in one way or another cause the companys have some "real" version they really want you to buy (ala ESRI).
Free GIS .. if its open source and GIS its at freegis.org but frankly there isnt allot built out yet thats simple yet modular and will grow with GIS users as they begin to get specialized (image interp, business siteing, habitat annalysis, etc).

theres GRASS and some others but nothing close to the commercial products (unfortuantely). furthermore all of the commercial products are over priced. you have to spend 3,000$ US (single licence) to get anything at ESRI that works. If your going to manage a GIS data collection you should get ArcView but try to find a free solution for end users on the desktop. look at combining MapServer and some of the Content Management systems that are out there for developing GIS access online.

This article was not intended to be read by experienced webmasters with knowledge in the area of JPG compression, but webmasters who don't know how to tweak images by compressing them...

/. is for experienced webmasters (and maybe even web developers, too). This article, on the other hand, is for newbies hacking their first pages in Front Page, complete with flaming titles, blinking text, and loud backgrounds. These people are not web developers (let alone web masters, god forbid), no more than my mother's a computer scientist cuz she programmed her VCR to record Murder She Wrote reruns. These people sure as hell don't read /. This article does not belong anywhere on /., especially not the front page. It's insulting to the readership.

Additionally, basic JPEG sucks and I don't condone singing its praises. It introduces artifacts and destroys the image. It might be fine for pretty graphics that GIF won't do justice to, or thumbnails in a photo gallery. But, to deliver high quality images (photos, wallpapers, or scientific imagery) it's a lousy choice, but the only one if you want your users to view the data in their browsers.

As for JPEG 2000, wavelet technology is already in use (such as LizardTech's MrSid software, which is not unimpressive). Certainly a fun and interesting idea, but it takes longer to decode and doesn't compress near as well as PNG (90k PDF). Considering how long it's taken PNG to gain acceptance (I'm still not convinced there's enough browser support to use pure PNG on my sites), I'm not holding my breath for JPEG 2000 as a web medium, and looking at the numbers, I'll take PNG over JPEG in a heartbeat.

In the real world, companies don't want to either GPL their software (required if they use this GPL library), or reinvent all the code from scratch based on the spec, unless there's huge demand for it (which there won't be due to chicken & egg scenerio)... So, don't expect to see any support for DjVu anytime soon.

I've seen a demo of their software in action. An entire CD-ROM (or more) of overlapping images (aerial photography in this case) that were geographically positioned could be viewed using their software. You could pan & zoom in near real time, and this was about 4 years ago! Their software uses wavelet compression.

The software was only on Windows at the time, but according to their download page they have both a browser plug-in and standalone viewer for Linux.

While this topic is well outside my usual area of (in)expertise, I think these guys products might be able to help you out... That is, if you were not absolutely tied into the TIFF format for your viewable files. As others have pointed out, TIFFs are a poor format for very large images.

Anyway, LizardTech's technology is used by plces like the USGS and Library of Congress to allow instant access to very large scanned maps and other documents. In particular, you might want to check out their software aimed directly at working with big photos.

While this topic is well outside my usual area of (in)expertise, I think these guys products might be able to help you out... That is, if you were not absolutely tied into the TIFF format for your viewable files. As others have pointed out, TIFFs are a poor format for very large images.

Anyway, LizardTech's technology is used by plces like the USGS and Library of Congress to allow instant access to very large scanned maps and other documents. In particular, you might want to check out their software aimed directly at working with big photos.

it's not simply a matter of getting more pictures on a card. jpeg2000 is a superior compression format in that the picture quality is far better at a given level of compression when compared to jpeg. there are far fewer artifacts introduced at given levels of compression. plus, at roughly half the space of a tiff file, it can store a LOSSLESS compressed file. in addition, it supports metatag data and is scalable, making it very useful in making webpages, archiving, e-mailing - and all with just one file. at the hospital where i am working, we are currently evaluating its usage for our digital pathology archives.

i agree, SOMEONE needs to start pushing jpeg2000. jpeg is simply not enough anymore. there is work being done on jpeg2000. unfortunately, a lot of it seems to be companies trying to proprietize the format.
luratech
lizardtech

Why don't you take a look at this? Apparently they use some sort of wavelet compression on the background. =)

Something CmdrTaco could have found for himself before sticking his foot up his ass(again)

I am one of the four persons who created DjVu in the first place. The events took place in AT&T-Labs Research between 1997 and 1999.

1. There is Linux support. Just go to the download page and select the Linux platform. Most of DjVu was first coded under Linux.
2. After the Lizardtech deal, we set up a "non commercial site" named DjVuZone. It contains general information, benchmarks. links, a searchable digital library, etc.
3. There is source code. Lizardtech recently had the good idea to relicense version 2 of the DjVu reference library under the GPL. We have the corresponding online documentation on DjVuZone. We are just waiting for the release of version 3 to redo that part of the site.
4. DjVu combines several new technologies including new approaches to arithmetic coding (Z'-coder), new compression methods for textual images (Soft pattern matching, JB2), new wavelet method (IW44), and new ways of combining them together. The current implementation is geared toward compressing scanned document images in 24 bit colors around 300 dpi (raw size is 25MB) and typically packs them into 50-60KB. Neither TIFF, nor JPEG, nor JPEG-2000 nor Fax-G4 can do that. None of these technologies will let you realistically view such documents over the web. DjVu can.

Hope this helps :-).

- Leon Bottou, AT&T-Labs Research.

I am one of the four persons who created DjVu in the first place. The events took place in AT&T-Labs Research between 1997 and 1999.

1. There is Linux support. Just go to the download page and select the Linux platform. Most of DjVu was first coded under Linux.
2. After the Lizardtech deal, we set up a "non commercial site" named DjVuZone. It contains general information, benchmarks. links, a searchable digital library, etc.
3. There is source code. Lizardtech recently had the good idea to relicense version 2 of the DjVu reference library under the GPL. We have the corresponding online documentation on DjVuZone. We are just waiting for the release of version 3 to redo that part of the site.
4. DjVu combines several new technologies including new approaches to arithmetic coding (Z'-coder), new compression methods for textual images (Soft pattern matching, JB2), new wavelet method (IW44), and new ways of combining them together. The current implementation is geared toward compressing scanned document images in 24 bit colors around 300 dpi (raw size is 25MB) and typically packs them into 50-60KB. Neither TIFF, nor JPEG, nor JPEG-2000 nor Fax-G4 can do that. None of these technologies will let you realistically view such documents over the web. DjVu can.

Hope this helps :-).

- Leon Bottou, AT&T-Labs Research.

It is heavily used in Remote Sensing, online mapping, and Web based-Geographic Information Systems.

It is very slick!

In general, with 'free' software vs 'paid' software, I think the corporate world has the advantage of advertising their product. But this corp produces high-quality, useful products!

Corporate digital asset collections

Real estate sites

Online catalogs/retail companies

Auction sites

Libraries

Medical sites

Geospatial imagery/government agencies

Corporations are storing everything digitally now: pictures, instructions, etc and need are searching for a way to manage all of this. This product is attempt to fill that void.

In a nutshell, this will be a specialized format that we will see for businesses that need to pass digital assets to the user.

They have a browser plugin for Linux/x86/glibc2 available for download here

Lizardtech is another company doing similar research that is seems luratech is doing. My company has done some alpha/beta testing for lizardtech (we're in the graphic arts industry). The cool thing is that they have a web server application (interfaced in perl) that allows you to zoom in on images in a web site.

Another company that has done probably the most ground breaking work in wavelets is Iterated Systems who we also helped out in the past. They are not as adept at bringing tools to market but have done a lot of original research into wavelet compression.

Jason

Iterated Systems- http://www.iterated.com
Lizardtech- http://www.lizardtech.com

The cool thing about this isn't so much the compression of an image (visually lossless compression to 40:1 or better), but the scaling of an image. You can scan an image at optimum setting (maybe 8x10, 1200dpi, etc) and compress it. But the cool part is later on when you want a poster size of this image, or a web thumbnail, you can scale the image up or down with great clarity from a single source. Extremely cool in this industry.

Jason

More information on JPEG 2000 can be found at http://www.jpeg.org.

I am wondering if the IJG is planning to (and has sufficient resources to) implement JPEG 2000 support quickly as the specification finalizes. Does anyone know? I asked Tom Lane about this indirectly a while ago, and he just pointed me to the www.jpeg.org web page.

The IJG did a great job on the current library, and I hope that they can do JPEG2000. I also think that if they need support (manpower/money) it would behoove the OSS comunity to provide it.

If for one agree that wavelet based approaches to compression are the future off lossy continuous tone compression. The MrSID technology for instance is great, but they hold a very tight hold on their proprietary technology. I think it is important to establish a popular, and public format and technology to fill this void or proprietary interests will damage OSS efforts.