Slashdot Mirror
A New Web Image Format
MrP- writes: "BetaNews is reporting that a company called LizardTech has developed a new image format for the Web called DjVu." Apparently, it differentiates between forground and background components of an image, and compresses each appropriately. Good idea, but I'm skeptical of improvements (especially because they say it's "20 times faster then gifs" -- which measure compression in terms of speed? And they also say it compresses faster then pdf, but pdf isn't really an
image format). No Linux support. And I don't see any source code on the format, so don't expect it to get a lot of support on any major Web sites, regardless of the compression.
Nude: girls naked. Naked: guys nude. hehe
Open technology.
Even if this company ports its format on a variety of OS, I won't be able to use it in my own products, I won't be able to use it with my favorite browser, etc. See what happened with GIF.
New Image Compression Algorithm claims 1000:1 ratio
Talk about posting the same story twice...
I doubt, therefore I may be.
DjVu is almost three years old. It was developed by AT&T not LizardTech. LizardTech just bought it about 8 months ago. It is not, was not, and will never be designed to replace PDF, GIF, JPEG, or PNG. I have been using DjVu as the core of a web based document management system for over a year and a half. It is absolutely bar none the best, fastest, and most cross platform way to go from paper->web out there.
Look at the ways to get a scanned document on the web:
1) GIF, PNG, JPEG: Large filesize or bad rendering. If I need to send a 300dpi page to a web browser, the browser isn't going to let a user pan and zoom on it and it certainly won't print it correctly. JPEG is the only one of the three formats that actually has a place to store the document DPI regardless.
2) PDF: Creating a pdf from a scanned image means either encapsulating a lossy or losless image in a file or doing OCR and risking unreliable information.
DjVu regularly achieves compression ratios of 1200:1 or more at very very acceptible quality. There is a IW44 fractal compressed background layer and a loslessly compressed foreground layer. The information is progressive also. As the file downloads the foreground shows, then the background, then the color information loads. Example documents on the DjVu website have shown entire 300 dpi full color sharper image catalogs compressed to fit on a floppy disk.
Btw not only are djvu plugins available for windows, macintosh, linux, and solaris. Let's not forget HP-UX and IRIX. How's that for covering the bases? If youre not supported, you can write your own for your particular flavor of UNIX.
Geez get it straight.
~GoRK
in the main post, it claims no linux support, and not that i support this format, they do have a linux plug-in viewer, just no authoring/encoding software for linux :o)
No, it isn't. Think about how handy it would be to let someone dial in their own image quality, or do it for them of course. Some poor bastard on a 28.8kbit/sec modem would be able to set it to super low quality, which would look somewhat lousy but be quickly navigable.
This would be handy in a system with CPU to spare but limited storage, like a 200mhz+ embedded webserver using flash memory to store images. Then again, you'd have to store them compressed there, as well, but at least you wouldn't have to store N different versions of the file for users with various different quantities of bandwidth.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Did anybody even follow the LizardTech link? Right on the front page is a link to a page describing DjVu. The whole product ("image format") seems geared towards scanning in Real Life documents and presenting them online. If you *read* the page it explains why it claims it is faster (first downloads high contrast data, then photographs and graphics, clarifying the image as it goes) and smaller (some wavelet kung foo). I don't see anywhere where they are pitching this as competition for gif or png, so everybody put down the flamethrowers. This is a very small niche product for digitizing and presenting real life mundane documents.
It's 10 PM. Do you know if you're un-American?
This should do a good job of a lot of the things that JPEG (even at high-quality q) is currently failing on for us: photographs that contain a lot of bluescreen / greenscreen; pencil / pen artwork, etc. With JPEG, I get nasty haloes around foreground elements; these are very visible on a flat background such bluescreen or paper. JPEG also forces a mottled effect on the bluescreen due to color quantization I believe. DjVu appears to reduce or eliminate these undesireable side effects. I will be following its progress with interest (as long as it doesn't get the kiss of death from proprietary interests...)
Charlie Ledogar, Media Software Engr., Lucasfilm Ltd.
Compression speed DOES matter. First of all, many things on the web are compressed on the fly - you just don't see it. gzip is used, as far as I know, for most of it. Partly due to its modest processor usage. You try serving up a million pages all compressed using something like bzip2 on a rackmount server with only two processors - you'll quickly melt the metal.
Dave
'Round the firewall,
Out the modem,
Through the router,
Down the wire,
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
The website clearly indicates that this software is designed for images that come from scanning papers documents that contain both text and graphics. The algorithm is supposed to recognize the text and store it as a separate layer (but still as an image) from the rest of the image. Furthermore, the image can be transmitted in such as way that the text and significant features of the image are transmitted first, followed by the bits representing less important features later - a technique known as progressive image transmission (among other names).
I certainly believe that they use wavelets to do this. In fact, the hot wavelet method of the 90's, EZW (Embedded-Zero tree Wavelet) compression allows for exactly that: to compress an image in such a way that the more significant bits come first, followed by the less significant bits in order of significance. Picking out text that is layered on top of a background image is relatively easy with wavelets: just pick out the really large coefficients in the wavelet expansion, since those most likely correspond to parts of the image where large jump discontinuities occur. This can all be done automatically, of course.
All is Number -Pythagoras.
Some of the members of the team include Leon Bottou, Yann LeCun (the guy who was one of the few inventors of Backprop), Yoshua Bengio, Patrick Haffner, Patrice Simard, and Larry Rabiner. I know those guys and they're very good. Don't know why ATT ultimately sold this product, though.
As far as the text compression goes - it works by clustering individual characters into subgroups and using the latter as a highlevel compression scheme plus encoding the differences. So you could even 'edit' the text (they did it but didn't release it for some reason). Anyway, it's pretty cool stuff.
On an off note, its nice to see that PNGs are becoming more widespread, probably 12%-20% of sites that I've seen recently are using PNGs for all the images
Here's my own image compressor. It's just like jpeg but I had to trim the source a bit to fit inside the teeny-tiny submission box. I hope you can still understand it.
./en 50 | bzip2 > image.4a
./de > image2.pgm
8 ]-23.5)*LQ:1.0)*co[B*u+C*x]*\
) ;O=0.0; } } M+=\
N Zao~}mUehkvlok";
The codec emits a byte stream considerably larger than the raw image (4x) but this stream is easily compressed by bzip2 or gzip. It reads and writes portable gray map (.pgm) files (both ascii and binary). If you want color I suggest you transform the image into YUV colorspace and compress each plane seperately. The color planes should be downsampled 4:1 (2x2 blocks) and compressed at very low quality (less than 15).
Build instructions:
Save the code as 4a.c
gcc -O2 -ansi -Wall -o en -DMAGICK=1.0 4a.c -lm
cp en de
Running the program:
convert image.gif image.pgm
cat image.pgm |
cat image.4a | bunzip2 |
convert image2.pgm image2.gif
To make a pgm from a jpeg use:
djpeg -grayscale -pnm image.jpg > image.pgm
Hints:
Try quality settings from 15 to 200.
Bad input causes core dumps.
The image is cropped if the dimensions aren't divisible by 8.
====================
/* 4a by Ryan Salsbury */
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#define D(a,i) a=(i); while (a--) {
#define i int
#define d double
#define szi sizeof(i)
#define szd sizeof(d)
#define gc() getchar()
#define pc(c) putchar(c)
#define w while
#define v void
#define L goto
#define _9 2+7
#define _27 3+24
#define di(F,A,B,C,G,E) \
v F(i j,i h,G* M,E* S){ d O = 0.0; i x,y,u,q,s,t; D(t,h/8)D(s,j/8)D(y,8)D(x,8)\
D(q,8)D(u,8)O+=M[u+q*j]*(A?ca[u]*ca[q]*(lQ[u+q*
co[B*q+C*y]; } } S[x+y*j]=O*(A?1.0:ca[x]*ca[y]/(lQ[x+y*8]-23.5)/LQ
8; S+=8; } M+=j*7; S+=j*7; } }
i nm()
{
i c = gc(), a = 0;
L0: if (c > 32) L L1;
c = gc();
L L0;
L1: if (c - 35) L L2;
L3: if (c == 10) L L0;
c = gc();
L L3;
L2: if (c <= _9*_27 || c >= _27*_9) return a;
a=a*10+c-48;
c = gc();
L L2;
}
v rp(i* x, i* y, d** X)
{
d K, *V;
i h, m, p, G, H, I = gc()*gc();
*x = (p=nm())&~7;
*y = nm()&~7;
m = nm();
*X = V = malloc(256 + szd **x **y);
K = MAGICK;
D(G,*y) D(H,p)
if (I==4240) h = gc();
else h = nm();
*V++ = h*K - 128.0;
} V-=p-*x;
}
}
v wp(i x, i y, d* V)
{
i n;
printf("P2\n%i %i\n255\n", x, y);
D(n,x*y)
i l = *V+++128.0;
l = l&~0xff ? l>>31?0:255 : l;
printf((n%20)?"%i ":"%i\n", l);
}
pc(10);
}
d co[64], ca[8], LQ;
char* lQ = "%! %,9CK\"\"#(.IJF##%,9HRG#&*0Ca[L'*7GQtoY,5FN\\pweA
di(fw,0,1,8,d,i)
di(rv,1,8,1,i,d)
i main(i argc, char* argv[])
{
char T[32];
i x, y, *pi;
d* ui;
d p = 0.1963495408494;
d pp = 0.3535533905933;
D(y,8) D(x,8)
co[x+y*8] = cos((2*x+1)*y*p); }
ca[y] = y?0.5:pp; }
if (argv[0][strlen(argv[0])-1] == 'n')
{
LQ = 100.0 / ((argc-2) ? 70 : atoi(argv[1]));
rp(&x, &y, &ui);
pi = malloc(x*y*szi);
fw(x, y, ui, pi);
printf("%i\n%i\n%f\n", x, y, LQ);
fwrite(pi,szi,x*y,stdout);
}
else
{
x = nm();
y = nm();
LQ = atof(fgets(T,32,stdin));
ui = malloc(x*y*szd);
pi = malloc(x*y*szi);
fread(pi,szi,x*y,stdin);
rv(x, y, pi, ui);
wp(x, y, ui);
}
return 0;
}
Sorry, I did take a brief look at it, didn't see that part and did let myself be goaded by our glorious leaders.
I don't know if "dumbass" is neccessary though. I was refering to the fact that the W3C seems to have a preference for PNG so perhaps we should stick to their standards.
I work for a bunch of architects, engineers, and designers who could probably use a good image compression scheme. But I'm not going to tell them about this. Why? Because the proliferation of formats can become a huge pain for little sysadmins like me, and enough is enough.I am so tired of troubleshooting "but I can't convert myfile.bozoimage into myfile.gonzoimage". I realize standardization is a pipe dream, but I will not encourage this kind of thing. (Same for document formats - Screw Corel, Microsoft, and everybody who has to make a proprietary document format)
It's nota my planet, monkey-boy - Dr Lizardo.
I'm willing to bet those weren't in fact FIF files, but FITS files. FITS (flexible image transport system) is the de facto standard for astronomical images these days.
The problem is compressing them. The ideas behind decompressing fractals are easy; use the self similarity inherent in pictures (a tree looks much like another tree, so build a picture of a forest by layering trees and then correcting the errors).
But figuring this out ammounts to exhaustive search. Apparently IFS had decent heuristics that got decent compression in only a few minutes. I think this is what was licenced, and that the file-format was open? This was a while ago, so I'm filling in the gaps in memory by guessing.
Perhaps I'm just ignorant, but I thought PNG was supposed to replace GIF when Unisys started seeking royalties? Slashdot advertised the "convert your web page's GIFs to PNG" contest or whatever it was. (Ironically, Slashdot still has MANY Gifs!)
Secondly, isn't compression becoming less of a concern since more and more people are stepping into the broadband arena?
It seems to me like the product makes sense, but it would have been much better received 10 years ago or a year ago when Unisys was going insane.
Yes, we have no spoon.
The List of Grievances with Slashdot.
Can it detect the difference between art and pornography?
I don't make the rules. I just make fun of them.
The comapny was Iterated Systems. The technology is still being developed, and highly advanced - they even have a fractal moving image compressor that can, [according to a workmate who did a thesis on compression - take with a grain of salt if you like, because I can't back it up] fit eight minuted of better-than-VHS-quality film on a floppy disk. He saw it at a trade show, allegedy. But believe what you will.
The licensing fees for FIF and other iterated technologies are huge. And the weird thing is, if they were open source, Iterated would be in much better straights than they are now. We'd all be using infinitely zoomable, highly comrpessed images [both GIF, JPG, and PNG are web standards due to seeming freeness], and despiute the fact they'd loose the revenue, they'd be known as the number one player in town for fractal image creation tools. And people would but their software over the other utilities which use the same Open Source image format, because they had the fastest algorithms, extra features, or other competitive advantages.
Just a thought.
I have to address a point in the betanews article:
The Internet has become a place where images rule, and visually pleasing Web sites reign over the simpler, more text-based sites.
The man has obviously never been to asciiboner.com. Anne Marie should check this site out too, so as to get an idea of what to expect on our wedding night.
--Shoeboy
The BetaNews article is actually very misleading - it is confusing two of LizardTech's products.
DjVu is a document format (like PDF), not an image format, and the techniques mentioned in the article refer to compression of documents.
LizardTech do have a compressed image format. This is called MrSID, and uses completely different techniques for compression.
I would be interested to see an independent comparison between MrSID and PNG - unless there are huge advantages of using the proprietry MrSID format over the OS PNG format, I don't predict much of a future for MrSID on the web (although it would seem that LizardTech are touting it more for internal use rather than for general distribution anyway).
[Happosai]
I don't know why the summary on Slashdot says "No Linux support". LizardTech has both decoders and encoders available for Linux.
Also the summary picks on LizardTech's use of speed as a feature. While this isn't a standard measurement, it is a way to tell people that you will get your images faster because the files are smaller. That's not a big crime. They also do talk more specifically about their format producing smaller files, so they do understand real measurements. BTW, while it is possible that they say elsewhere that DjVu compresses faster than pdf, what I saw was that the documents download faster, not compress faster.
The Slashdot write-up complains about LizardTech's comparison of DjVu with pdf, pointing out that pdf isn't an image format. True, but the LizardTech description refers to DjVu as "DjVu for Documents", and their web page describes why DjVu is good for documents. Images seem to be just part of the data they need to handle.
Finally, I haven't seen any source for Acrobat either, but it is very popular on the Internet, so lack of source won't necessarily keep LizardTech from succeeding with DjVu.
Is DjVu actually any good? I have no idea. Slamming the product with incorrect and misleading comments doesn't help one decide, though.
My algorithm was lossless and what strikes me the most is the speed - my algorithm was notable for the speed of decompression, and we used it in particular just because it was so much faster than GIF, which was a significant advantage when you were scanning through lots of images on CDROM. While I only implemented it for 8-bit indexed images I felt it would likely work fine for any bit depth or color space.
Medior was later purchased by AOL and renamed to AOL Productions. I think AOL Productions isn't around anymore but lots of old Medior people still work for AOL, for example former Medior President Barry Shuler is a high-level exec at AOL.
Basically, my invention worked by dividing the image up into lots of little subregions and encoding each pixel in a given subregion in the minimum number of bits required to encode the number of colors that occurred in that region.
For example, in an image that was black on top and white on the bottom, I'd have two subreqions with zero bits each and a single element color table that was either black or white.
If it was snow - black and white pixels randomly intermixed - the whole image would be one bit with a two-element color table containing black and white.
The big trick was to divide the image in a good way, in such a way that the whole image was reduced the most and the size of the data required to reconstruct the regions wasn't too big. In practice I found it worked OK to start with lots of small fixed-size squares then merge adjacent squares that had similar color schemes.
I wrote a document for Medior that described what I invented in great detail and what I predicted this could be made to do. What I actually got it to do in practice was not nearly what it was capable of, but this was because of the limited time available to implement it.
Michael D. Crawford
GoingWare Inc
-- Could you use my software consulting serv
PNG's are good, BUT:
Software for creating/reading them (end-user software as well as API's) still isn't as common and widespread as it *should* be. These things take time, yes, but they are slowed by factors like popular browsers not supporting PNG's properly anyway.
Don't get me wrong though, I would like to see PNG becoming the dominant format. But PNG seems to forever be stuck in the "early adopter" phase ..
http://www.djvu.com/cgi-bin/products/products.pl
lists DjVu Solo 3.0 (encoder for single pages only) as a free download. Could you use that?
http://www.djvu.att.com still has source for the reference library for people wanting to write viewers, but an encoder would be harder - I guess there might be patent issues too.
rant
Like so many other factors, speed is one that does matter if in the extreme, but once it's good enough, it has very little importance. Size is arguably more important, because every byte saved is saved every time you use the file, while time saved is only saved during compression, which is often batched.
So for time, order of magnitude seems to be the differentiator, while size has a linear importance.
Obviously, this person is not very well informed.
c elibrary/DjVuRefLib.html
Linux has always been the primary development platform for DjVu products. Typically the release cycle is Linux and other unix platforms, then Windows, and finally Mac.
You'll also find the 2.2 DjVu Reference Library is available with GPL licence. The 3.0 Reference Library is scheduled to be released shortly...
http://www.lizardtech.com/products/djvu/referen
I hope that the GIF thing doesnt happen again.. ;)
Wait till it's a "standart" then ask for license fees.
The idea sounds pretty nice, but I'd like to see quality / compression ratio comparisions with hard numbers
Before you email me, remember: "There is no god!"
http://slashdot.org/article.pl?sid=older/980628231 4218&mode=nested
That was almost 2.5 years ago. The reason they compare against pdf is that the target application seems to be text; compress the black and white text with one-bit RLE, and the background with JPG sort of deal.
First time I used PNG, I thought the same thing.
Actually, it is just that PNG has a far more
complicated interphase for selecting options.
Once you select the right options, PNG will compress on the average about 10-20% smaller than
a GIF. I've seen as high as 50% smaller. The
only thing I've seen that is better for lossless
compression hasn't been released publicly yet.
When it is, I'll be happy to tell people about it.
Come on people, let's stick to open standards here...
A few years ago a company came up with a compression which was actually rather good, using fractals. It was called FIF. They made the mistake of greed ovtaking common sense and tried to charge for a license to write compressors for it... The result - when is the last time you saw an FIF file?
If these guys don't have an open format they will simply go the same way.
And how the heck does it figure out what the "foreground" and "background" components are, anyway? If I have to go in and mask out the background before saving, then forget it.
If it ain't broke, it doesn't have enough features yet.
See DjVu "non-commercial" site.
--Daniel
I am one of the four persons who created DjVu in the first place. The events took place in AT&T-Labs Research between 1997 and 1999.
Hope this helps :-).
- Leon Bottou, AT&T-Labs Research.
...unlike the original story commentary above stated. It's on AT&Ts site Right Here and you can download and tinker with it. But read the AT&T Source Code License first; you can only distribute patches to the distribution if you change anything. Also, they don't give you all the encoding code, omitting the background/foreground image seperation and the lossy JB2 wavelet encoder that handles bitonal images (some of their best examples of compression!). They actually suggest someone should make a GIMP plug-in for all this. And because they give you the rest of the JB2 back-end, they're practically begging for someone who's read the literature to bridge that gap.
That was real nice of them.
Black holes are where the Matrix raised SIGFPE
The article has it about right. Png is vastly superior - excellent lossless compression, sometimes better than lossy methods (plus technical features like a full alpha channel), and, most importantly for its dominance over gif, it is unencumbered by patents or closed source algorithms.
Speed of compression is not a factor in compression - otherwise we would use bmps or xpms, which have zero compression time - because they're uncompressed. Size matters. Speed doesn't.
I really can't see much market, and very little application for this compression. On-the-fly compression of images for web download would be redundant, since a png would be smaller than this format, so the speedy on-the-fly compression of uncompressed images is pointless.
And in any case, modern PCs are more than powerful enough to almost transparently display well compressed images, so a simpler format is about 10 years out of time.
If it was open source, it could perhaps have a market in replacing things like xpms, which are used in games for processing speed, but even it was, the benefit would be marginal, since hard disk space is, relative to image size, almost infinite, so compressing them slightly wouldn't make much difference - and for download those images would be gzipped anyway.
Free Anne Tomlinson!!
Not so fast - they have a plugin available for Netscape and IE at:
. pl?tsb=443224
http://www.lizardtech.com/cgi-bin/products/desc
Cheers,
Tim
It's official. Most of you are morons.
Comment removed based on user account deletion
I hope it'll reduce it to zero byte
____________________
Ni!
Flash? Hell, no! check out my home page and see how many images and plugins I've used (hint: none).
DjVu is applicable for any raster image. It just happens to be good at seperating high constrast regions (text) out of the image. And they may be referencing the name AT&T gave to the whole technology, while LizardTech chose to split it up into two distinct groups.
Black holes are where the Matrix raised SIGFPE
I'll second that MrSid is pretty cool. (I found it by way of some old railroad maps at the library of congress. So, if you are looking for some free content, there you go.)
--
Business. Numbers. Money. People. Computer World.
I have to question if CmdrTaco even went to the web site. There are SDK's for linux/win/solaris, and they measure the speed in terms of PDF because DjVu is a contender to PDF: you can add hyperlinks to documents you scan in.
As I understand it, it is possible to set different compression factors (from lossless to very lossy) on different polygonal regions of a JPEG file.
Since JPEG is already fairly-well supported, why not use it? The only thing that needs to be changed is the image-creation software; it needs to know how to detect the difference between background and foreground content. Image-creation software also has a huge advantage here; layers (and their relative z-Indexes) give great hints as to what is, and what isn't, "background"
--
Do daemons dream of electric sleep()?
This post looks legit to me despite the AC status. I.e., you can believe it. Cheers, Ed. (Only people from AT&T Labs-Research write "AT&T Labs-Research" :-)
I wish people would get off the "if it's not OSS it must suck" bullshit attitude. LizardTech has some really well-designed software, and if it bothers you that they're proprietary (and IMHO, grossly overpriced), then bigod go out and clone it. Having an OSS equivalent of MrSID and DjVu, especially in a form that would be usable to the point-and-drool crowd, would be a very strong selling point for getting Linux into the MS/Adobe-dominated document archiving market.
--
Proud member of the Weirdo-American community.
Not only you are an Anonymous Coward (if you have something to say come out and speak up !) but you also had to post this off-topic bullshit 2 times to get it right. Next time don't write anything and use the Preview button to make sure nothing actually spilled in the Comment box 8)
Thank you
If this graphics format is so great, why aren't they using it on their own site? Everything I saw was in .gif or .jpg.
DOH!
I say we take off and nuke it from orbit. It's the only way to be sure...
Have a look at:
http://cgi.netscape.com/cgi-bin/pi_moreinfo.cgi?P
Will
per mere, per terras
It's AT&T's technology.
;) ).
6 220&mode=thread
2 314218&mode=thread
http://djvu.research.att.com/
Even mentioned in slashdot LONG ago (and we know how timely slashdot is
http://slashdot.org/article.pl?sid=99/03/29/031
http://slashdot.org/article.pl?sid=older/980628
Do a search on slashdot for djvu for more links.
Link.
Um, you can't arbitrarily change a license for software (that you have already licensed.)
If you have a free open-source encoder, you can continue using it for no cost. No $2000 fee.
retrorocket.o not found, launch anyway?
I didnt read through all the comments here, so this has probably been said already...
But wasn't DjVu developed a GOOD while back by AT&T or Bell Labs or something? I could have sworn I took a look at this 2 or 3 years ago.
--
Dave Brooks (db@amorphous.org)
http://www.amorphous.org
It was really just a proof of concept anyway....so it's not costing me enough bandwidth to matter having switched away from DjVu.
Werd.
This scheme has the advantage that it keeps high-frequency content (the edges of the text) out of the background, enabling it to compress better (wavelets and JPEG don't handle high frequencies well: they either smear them, add ringing, or require a lot of bits). Another advantage is that there are specialised algorithms for compressing binary (1 and 0) images containing text. The one used by DjVu is called 'SPM' (Soft Pattern Matching). It works (roughly) by breaking the text image up into isolated characters, then deciding which characters look similar to each other. On most pages, there will be a lot of repeated characters. You can then get improved compression by storing only one instance of each unique character shape. SPM does something fancier than that but that's the basic idea.
This three-layer model is known as MRC (Mixed Raster Content). It's used outside DjVu - most notably in a file format called TIFF-FX. This is an extension of TIFF intended for Internet Fax (including colour fax). TIFF-FX is, if I remember right, RFC2301. There is an extension to TIFF-FX in the works to add support for JBIG2. What's JBIG2? It's a format standardised by ISO for doing bilevel image compression, using the same concept of improving text compression by identifying repeated shapes. (Actually, there's a lot more in JBIG2 but I'll leave that aside). The concepts used in DjVu's SPM were incorporated into the JBIG2 design (AT&T participated in the design of JBIG2), so JBIG2 can do anything SPM can do. JBIG2 was approved by ISO and ITU this past spring.
Another file format that includes a lot of the same concepts as DjVu (MRC, repeated shape compression) is ScanSoft's XIFF - it's yet another TIFF extension, and is used in ScanSoft's Pagis line of products. In fact, much of the stuff in TIFF-FX is a standardised version of features that appeared in XIFF. MRC is also going to be in one of the later parts of JPEG-2000.
I've been involved in the work on XIFF and TIFF-FX and I was the editor of the JBIG2 standard. So what do I think of DjVu? One problem is that AT&T abandoned the standards process - they felt it was moving too slowly. This means that they opted for a proprietary solution over an open standard one. Yes, the standards process is slow, but when it works it produces things like JPEG and PNG (a W3C standard) - formats that everyone can use, and where you have a choice of encoders, a choice of decoders, and few worries about interoperability. Another problem is its reliance on arithmetic coding - while that gets you good compression, it can have speed problems. JBIG2 offers the choice of arithmetic compression (for applications where size is the most important issue) and Huffman-based compression, for applications where speed is the most important issue. I've personally seen a JBIG2 decoder decompress at over 1 gigapixel per second...
I have long given up on slashdot reporting the whole truth and nothing but the truth...
-Chris
This was on slashdot like two years ago, when it was AT&T pushing it instead of this LizardTech place that apparently picked it up.
One of the many things I hate. thingsihate.org
He posted this two years ago: DjVu plug-in available on Linux/Irix/Solaris/Mac. First post by sengan:
:-)
New Image Compression Algorithm claims 1000:1 ratio
Hasdi
More over that, the first demo compressor / netscape plugin that AT&T released was for Linux... And this was two years ago... :)
Not only is there a Linux x86 binary available, the source code is also online here.
It seems that web designer omit the fact that there is always slow computers and slow connections. My work computer can't play flash animations (too slow), my home computer can't load big pages (56k winmodem). If a better compression could exist, it could leverage my poor home connection and if this compression is fast it can leverage my work computer. In the two case i am a winner. :-)
I agree with you that this reply seems to be a flame, it's nearly one because i am in anger to those who thinks that everybody have a T1 at home...
> Who makes pdf's that way?
you have a printed document, with color. Now, you want to archive/distribute it. What are you going to do ?
When going the PDF way, you blindly scan the color document and don't do any extra job (for instance, take all those re-editions of Sierra/Lucas Art old adventure games. The original manula is included as a huge PDF file).
So their stuff makes half-sense to me. But, clearly, there is a lot of hype on their site...
Cheers,
--fred
1 reply beneath your current threshold.
IMO - they can all go and stick it up their hole where their head is. :)
Regarding their "Convert your paper assets into the fastest, smallest, clearest digital format" there was something like: "any paperless office will wind up consuming more paper than a simple - paper-based office."
Conclusion? DIE AT&T - DIE! and take your bleeding proprietary format with you - it's allright DejaVu already!
--
Disclaimer: I used to work at AT&T Labs as an outside contractor in their library.
It is heavily used in Remote Sensing, online mapping, and Web based-Geographic Information Systems.
It is very slick!
In general, with 'free' software vs 'paid' software, I think the corporate world has the advantage of advertising their product. But this corp produces high-quality, useful products!
----- "Computers will never catch on, why don't you go outside and play!"
I'm really interested to know where JPEG 2000 support is for the GIMP and the rest of the free software world. Last I read, GIMP wasn't planning on supporting JPEG 2000 because of licensing issues, which would be a real shame, since wavelet compression is an awesome concept.
---
"He may look like an idiot, and talk like an idiot, but don't let that fool you. He really is an idiot." - Duck Soup
It all depends how you make your PDFs. The bulk of the size of a typical PDF is in the images, and the images may be embedded within the PDF in various formats and with various levels of compression. A comparison without detailing what format was used is meaningless.
For example, if you scan a typical one page black and white text document at 300dpi and save the result as a TIFF with CCITT Group 4 compression it will be 20-50kB. If you make a PDF by printing to Postscript then using the PDF generation tool within the GhostScript Viewer you will end up with a PDF that is several MB. That's because embedded within the PDF will be an uncompressed bitmap. If you make the PDF using the c42pdf utility your PDF will be the same size as the TIFF, 20-50kB. That's because embedded within the PDF will be the compressed bitmap.
If you use the Adobe Acrobat tools to produce your PDFs, you will find lots of options for controlling how embedded images are compressed, what the resolution is and what formats are used. You can select different options for B&W, greyscale and colour images.
I suspect the DjVu comparison is not comparing like with like. And in any case, as has been noted, anyone who generates PDFs of documents like annual reports with a scanner is missing the point.
Even for black and white images, PDF is the wrong way to go when you're dealing with scanned input. You either use their OCR which (like all OCR) introduces error, or you store the scanned image. The compression they use for the scanned image is quite poor. Modern compression methods like JBIG2 and the SPM compression used in DjVu can reduce a scanned document to a tiny fraction of its original size - 100:1 is not uncommon.
Group 3 and Group 4 fax do really badly on halftoned images; JBIG-like compressors can do quite a bit better. JBIG2 includes a special mode to compress halftoned images.
TIFF6 doesn't support the MRC model used in DjVu - you have to go to TIFF-FX to get that.
Something CmdrTaco could have found for himself before sticking his foot up his ass(again)
Stating on Slashdot that I like cheese since 1997.
I think you are right about it not being open. Notice they have two doors for visitors? One is for commercial use! I smell PDF revenue stream. Give away free viewers and sell the encoders.
The truth shall set you free!
The big problem I have with this article is that DJVu isn't a "new image format". It doesn't even display things inline (like GIF, PNG and JPEG). It is however an excellent alternative to PDF if size of file is your main concern.
The extensive references to "speed" when compared to GIFs and PDFs could be one of two things. They could be talking about Download speed (my personal experiences show DjVu files to be about 10 times smaller than GIFs and even more when compared to PDFs. Or, they might be speaking of encoding speed which DjVu seems to excel in
Here is a problem however: the command line encoder used to be free for non-commercial use. I was using DjVu for encoding swim-team documents for a small non-scholarship collegiate swim team. Certainly this counts as non-commercial. HOwever, the new version from Lizard Tech would cost me $2,000 USD to run. That is absurd by comparison. So I'm abandoning DjVu since I can no longer afford the encoder.
Incidentally, if you want to see how it worked for me, I used it on nearly every swim meet results page for a few years. Here is an example, just click on the links next to the word "Splits" in each event: http://www.k-swimming.org/cgi-bin/swimming/results /meet_view.pl?8
Werd.
I see what you mean.
I'm on 256kbps DSL right now, but even if I were in a modem, things wouldn't be that problematic. I only see an advantage to DSL when downloading large movies and game demos. Of course getting slashdot in 3-4 seconds versus 10 seconds is comfortable but IMHO it doesn't warrant for better image compression. I'm used to browsing with 3-4 netscape windows open and downloading at the same time and load times certainly don't bother me when I'm with a 33.6 modem.
While I don't assume everyone has a T1 at home, I usually assume everyone has a half-decent machine. Thinking otherwise reminds me of Andy Tanenbaum arguing with Linus Torvalds and reasoning that it was a bad idea to make Linux run only on 386 machines, thus breaking compatibility with older x86 processors.
Flavio
So is JPEG 2000. So, who needs this?
Now, if it were a fractal compressor, then it might actually matter.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
First you have to have a good format, then it has to be accessible and affordable, then it has to be accepted. For the life of me, I can't figure out why PNG hasn't replaced GIF.
Actually, though, that's part of the final hurdle -- a Catch-22. No one will adopt it until web pages and browsers support it. Web pages won't support it until browsers do and browsers have no reason to until web page creators demand it.
And why exactly is "fractal compression" so much better than wavelet-based?
.|` Clouds cross the black moonlight,
~Tim
--
~Tim
--
Rushing on down to the circle of the turn
They have a browser plugin for Linux/x86/glibc2 available for download here
Yes, I know that link is broken becauseBe careful. People in masks cannot be trusted.
DjVu has been around for 2 years, and isn't anything new. In fact, it wasn't actually designed by Lizardtech - it was developed as an Open Source technology in the Olivetti and Oracle Researtch labs in Camridge, UK, and was sold when US telco AT&T purcahsed the labs.
Hence the Open Source products generally only seem to be there to satisfy existing licensing requirements from prior to Lizardtech's purchase. It's doubtful Lizardtech tend to encorage that aspect of the technology, and they're only promoting the closed source stuff.
However, the compression is indeed very real and the cross platform nature makes it quite useful for archiving stuff that won't be modified frequently in the future - remeber, that text ain't vectorized, it's just another layered image, AFAICT.
IMHO there's very little need for highly compressed images on the web right now. With so much bandwidth available and the notion that images aren't always mandatory, what difference does 100kB of data make to a web page?
Of course the concept's interesting, and if it's really innovative it could be used in conjunction with other techniques to improve video compression, but considering the format's not open [(don't even get me started on this, because we're considering the web here)] and perhaps not all that fantastic, why bother?
Flavio
This is not an image format, but a new PDF format.
How is a GPL reference lib proprietary?
Stating on Slashdot that I like cheese since 1997.
It's an old technology developed at AT&T a few years ago. See : What is DjVu.
YMMV but it seems to work quite well.
I was able to get a look at this at some Press thing at Comex. What they were actually showing off was their MrSID program, which is what's needed to view the images. Aside from thr fact that they were pulling from a server in Seattle off of a T1, it was still pretty impressive as to what they could do with their images. You could "zoom in" on small sections of images and check them out with amazing clarity. While impressive, the need for such things isn't all that broad. I discussed possible uses for their new format and they told me they were targeting e-store types of places. Say you wanna buy a blanket from a store online, but you're not too sure about the quality, since you can't get an upclose of the blanket...just zoom in. I thought it was fairly impressive.
Jpeg 2000
>faster then gifs" -- who measures compression
>in terms of speed?
Well, what LizardTech's site says is:
>Download a 50-page color catalog in DjVu format
>in the time it takes to download a single page
>of that same catalog in PDF format.
So it looks like when they say "faster", they mean "smaller" (and thus faster to download).
BTW, I so much hope that if this goes anywhere, the format will be made open. (Fat chance :( ) Someone made a comparison on the BetaNews story with the problems with GIF; but that's not comparing like with like; we have a new format which requires a plug-in. It's more like pdf, only worse, because the viewers won't be as easily available.
GROGGS: alive and well and living in
GIF = "jif"
JPEG = "JAY-peg"
PNG = "ping"
DjVu = "duh-ju-voo"
So you have the choice between drooling on yourself or saying "deja vu," which has more syllables, tone changes, and stops than its nearest competitors. Plus the danger of the illiterate calling it "day-JAH-voo."
-----
Go ahead, blame me... I voted for Nader!
-----
Go ahead, blame me... I voted for Nader!
There's another competitor to MrSID by Iterated Systems called MediaBin. We were helping Iterated out from an actual field use (we're a prepress/graphic arts shop) but they were more interested in the science and not implementation. Looks like MediaBin is their offering.
Lately we've been much more interested in MrSID and have been using it a little bit, but are hoping to include it in our home brewed media management system.
Jason
DjVu has both image layers and a data layer. The high-contrast b&w layer is OCR'd and the recognized text is stored in the data layer. The image can thus be indexed by any search engine.
It appears that they've simply filtered the documents into 2 layers.
One for bilevel compression, say using CCITT group 4 compression (Faxes use group 3, group 4 is better ratio, but bigger disaster if you drop a bit), or maybe JBIG (IBM have a patent on the statistical model of the Q boxes, not on the actual compression algorithm, so all you need are your own Q boxes). Both of the above compare the current line with the previous line, and barf horribly when given dithered bilevel images, such as newspaper photos, or noisy scans.
The second layer is the full colour one, and can be implimented as a JPEG.
Tiff 6 supports all of the above.
The 3GB Tiff document they talk about on their site was probably 3GB using the "uncompressed" setting. Apples and Oranges, as they say.
The world doesn't need any new de-facto "standards" when there are perfectly good present non-proprietory standards which can do the same.
FP
Also FatPhil on SoylentNews, id 863
At the Cobb County (Georgia) Court Website, I can't remember the address..but they compress all their court documents using this method, it doesn't seem too bad.
- nick
Code commentary is like sex.
If it's good, it's VERY good.
My office has been taken over by iPod people.
The web page talks about the 90% of documents in the world that are still on paper only. Now I really hope that everybody understand that using DjVu alone to webify these documents will be of little value. How do you expect to find those documents? Will Google be able to index images?
Scanned documents on the web will have to be included in web pages that contain reasonable keywords and extracts from the texts. Scanning, compressing and commenting may actually be more work than keying in the entire document, so most people will probably just skip the commenting.
But heck, nobody seem to remember that the web was about content, not layout, so I'm sure DjVu will be warmly welcomed.
I watched that last night too but didn't know those were FIFs... I was actually fascinated that they could discern anything in those close-up pixelated shots!
/."
Here's to the Ever-Expanding Universe!
"I'm not a bitch, I just play one on
The House Between - Original Sci-Fi Series
After installing the plugin and checking out the samples, I saw that what most people are saying is true, it is a PDF clone, not an image compressor format. If you look at some of the sample files, you will also see they are MUCH bigger than standard PDF files, the majority of them range from 120,000 to 190,000 per page. It appears to initally load faster than Acrobat, but I think that's because it's more of a streamlined plugin rather than a whole system. The images themselvces however seem to be much bigger than any PDF I have ever encountered. Anyone else go through their sample directory?
-Sternn
A little late in posting, but anybody browsing "old" news might want to know that LuraTech also has wavelet compression.
---
do we really need another image format? do we even want another image format. you can claim all the nice enhancements / speed improvements / compression improvements you want - its still yet another damn file format that is not built into what we've already got.
sometimes it seems that people just like to try to invent something for the sake of inventing it, not on whether or not it fills some sort of real need.
Hi! This is the Sig, blatantly attached to the end of this comment.
"DjVu" makes me think of french "Deja Vu" which means "already seen"
The world belongs to those who get up early. - I'm far from being the king of Earth then
Corporate digital asset collections
Real estate sites
Online catalogs/retail companies
Auction sites
Libraries
Medical sites
Geospatial imagery/government agencies
Corporations are storing everything digitally now: pictures, instructions, etc and need are searching for a way to manage all of this. This product is attempt to fill that void.
In a nutshell, this will be a specialized format that we will see for businesses that need to pass digital assets to the user.
I have a positive modifier on Troll. When I mod someone Troll their karma should go UP!
Its more a document archiving format than a new web image format [although it happens to be viewable over the web] - as the article states although its non vectorized, it uses layered bitmaps to create more efficiently encodable data chunks.
:-)
And, actually, there is Linux support, and source code available. Just Lizardtech aren't going out of the way to tell anybody about it - see my above post
Why bother posting the article if it's such a non-event?
This graphic format has been around for at least 3 years now... How can they tout it as new?
Now the claims on the document format bother me. First, they compare it against PDF which we all know is large and bloated to begin with. Sure, if I took any document and separated out the text and formatting and images, compressed it down I'd probably have a "new" revolutionary document format. Doesn't this just sound like HTML? I've written server side scripts and client applets that will compress HTML the same way, and I think my results would be about the same as this (and perhaps faster?). You'll still need their plug-in to view their documents. They also say that a 2.5gb TIFF is compressed down to 3mb. Wow. I can do that now if I convert the TIFF to JPEG with little loss of quality. I really don't see what the big deal is about.
I wish the /. reporters would do a little research before they go posting messages that send the readers into a frenzy of clicking and sending off emails to friends about the next wave sweeping the internet.
liB
This is not news, it's already been on Slashdot I guess. This codec is from AT&T's labs. For a really new image codec, check Ulli's website: http://ulli.linuxave.net/fiasco/. It would be nice to add FIASCO! support to the GIMP and gdk-pixbuf for example...
If there is no linux support, how come I have a plugin for it on my slackware box? It's called nsdejavu.so (maybe its not the same, but it sure looks like it to me).
The main attraction of DjVu is that your scanned documents are tiny (typically less than 50KB) which makes it feasible for putting them on the web. Just about every other format results in files too big for easy distribution on the web. Interestingly, you can convert a *.ps.gz file into a DjVu file, and see a dramatic improvement in file size while preserving almost all of the detail. I am not talking about simple pages here, by very complex ones with a mixture or real images / artwork, and text.
Apologies for any mistakes, but I think that I got most of it right.
-- GWF
The Computational Beauty of Nature
We tested the new image format some months ago (in the sense that we tested its usage with the plugin) and I must admit that decompression is quite fast and compression ratio is quite noteworthy.
I seem to remember that the compression process is a bit slow but I'n really sure about it.
Do not ask me wether the plugin is strongly optimized or its the format in itself which is smart, I'm just talking about results.
Put on by AT&T. Got a "compressed" T-shirt for my trouble (shrink-wrapped, about the size of 3 CD jewel cases stacked).
Anyway, things not mentioned elsewhere. The main "cool" things that the exhibitors mentioned was that DjVu would take an image that had graphics and text on it, distinguish between the two and on the same image compress the graphics part with wavelets and the text part with a fax-like compression whose name I don't remember.
The text compression worked by taking all the characters and deciding which ones were similar, store a small image for the character and just remember the placement of that character. For example "hi there" would store h, i, t, e, and r once then slap the images down in the correct places when decompressing. The nice part about this is that it works just as well on Chinese characters or other pictograph languages as it does on Western languages. Those of you wanting to scan and store Asian languages should give this a look.
As for PDF, if you're generating the PDF file yourself you're not going to do any better with DjVu since it just treats the PDF as a big image. I got the 1040 PDF off of the IRS site and ran a DjVu compressor on it and it came out about the same size since PDF's just have strings of characters for text.
Here are just a few of the things that DjVu can do that PDF cannot:
For my money, DjVu is a superior file format that has been nicely tuned for the web. PDF was optimized for print pre-press, and it shows.
LizardTech had a booth at Seybold this year, and let me tell you this technology is very, very impressive. They demonstrated extremely high-res files at various zoom points--and explained that the files were very small. This thing also worked lightning fast. Except for a just-visible delay, zooming happens almost instantly. Another thing: I'm fairly sure it was doing raster, not vector. In any case, it was obvious it went way beyond PNG.
Remember when PNG was supposed to be The Next Big Thing (TM)? PNG was billed as the greatest image format for the Internet, since it was compact, high-quality, supported interlacing, and (I think) animation.
When was the last time you saw a PNG image? I've never seen them, except when I've made a few. The fact is, the image formats that are ingrained into people's sites and minds are GIF and JPEG. Nowadays, even digital cameras store images in JPEG. I remember when they used to have proprietary formats, like KDC on the Kodak DC40. That thing was $1000 when we got one, now you can get better cameras that store JPEGs for a couple hundred at most. Amazing.
Just as well, about DjVu. What kind of stupid name is that for a file format? How do you even pronounce it? I also fail to see the advantage of compressing foreground and background separately, but hey, I'm no imaging expert.
New image format? I'll believe it when I see it. Everywhere.
I do not belong in the spam.redirect.de domain.
If I invented some crazy cool new image compression format... I'd charge admission too. Isn't that sorta what the wavelet compression plugin publishers are doing? Getting your personal info in lieu of money, i guess... $.02
> Apparently, it differentiates between forground and background components of an image, and compresses each appropriately.
Yeah, but can it detect the difference between nude and naked?
Sheesh, evil *and* a jerk. -- Jade
Part of the betanews article is misleading : DjVu does not compress 'faster' than PDF/GIF/JPG but more efficiently, which enable faster transmission of compressed documents.
Note to the editor : please read the article more carefully next time. Thanks.