Slashdot Mirror


User: billstewart

billstewart's activity in the archive.

Stories
0
Comments
7,948
First seen
Last seen
Profile
(view on slashdot.org)

Comments · 7,948

  1. Re:FreeS/WAN on Open VPNs On Unix That Support Windows Clients? · · Score: 1
    OK, so it's mostly developed "Across the Great Lakes" rather than strictly overseas :-) (Early parts were from Greece, and various parts are written in Germany and a few other places, but the bulk of the effort is Canadian.)

    The web site www.freeswan.org has a pointer to several web archives of the mailing list, where there are reports and discussions of compatibility issues.

  2. Re:Security of PPTP on Open VPNs On Unix That Support Windows Clients? · · Score: 2

    Mudge and Schneier analyzed PPTP and found half a dozen major things wrong with it. The symmetric encryption algorithm it uses, RC4, is quite strong, but has a one rule aboutusing it safely - Never use the same key twice. PPTP violates this two or three different ways, and has some leftover Microsoft Wimpy Password Algorithm cracks that provide a couple of other ways in, and there may be some other holes as well. (Then there's the usual security principle of "Never plug anything sensitive into a Windows machine":-)

  3. Re:FreeS/WAN compatible with various packages on Open VPNs On Unix That Support Windows Clients? · · Score: 3
    The FreeS/WAN project www.freeswan.org is the Free Linux implementation, developed outside the US to avoid export restriction problems. They've done a lot of work on compatibility with a large number of other packages, including Nortel Contivity and PGPnet. Typical compatibility situations are that IPSEC/IKE have zillions of different options for keying, and any two products will have some subset that work, usually manual keying. The other typical issue is that for policy reasons, FreeS/WAN only does 3DES, and some commercial products only do single-DES. (John Gilmore, who funds FreeS/WAN, spent a lot of money and time developing the DES cracker to convince people that single DES is dead...)

    Nortel has a policy of Freeswan compatibility, so you should be able to use their server or client to talk to a FreeSWAN linux box. Nortel's client software runs on Win95, Win98, and NT, and is free if you buy the Nortel hardware (formerly Bay, formerly New Oak.) I don't know if it's free if you don't buy a box from them. So far I've used the Nortel client only with Nortel servers, but it works quite well and has multiple options for keying, including SecureID.

  4. Re:This was predicted some time ago on Encryption Market Opening Up · · Score: 3
    PGP's real importance was that you _could_ use it, and you could get it easily, and everybody rubbed the government's face in the fact that the laws against crypto publishing directly violated the First Amendment, plus Phil had the guts to stand up to them publicly, with good lawyers backing him. This not only had excellent PR value, but got a lot of people interested in crypto. From a more practical standpoint, it was Netscape's decision to include SSL, even with wimpy crypto, that moved the public perception of crypto from "something only spies and paranoids and occasional political activists use" to "of course I use it, how else would I send my credit card number across the Internet without getting ripped off!" Even now most of the public doesn't send much encrypted email (even the cypherpunks don't send huge amounts of encrypted email), but everybody knows you can, and everybody knows you should always use encrypted form for your credit cards and other sensitive personal information, because otherwise Hakk3rZ will steal it, and that's a good start. (Sigh - if you can't get the public to use a term correctly, at least you should exploit the heck out of their misuse :-)

    The real place that cryptography has been left out has been the Voice-over-IP telephony world. The de facto standard H.323 doesn't do it, though some of the newer protocols like SIP and MGCP provide hooks or full mechanisms for it, and most of the proprietary Internet telephony programs don't appear to support it either. This means that we're building an easily wiretapped infrastructure for international calls, and starting to build one for US domestic calls as well (and at least in the UK, wiretapping ISPs is easier legally than wiretapping telephones.) On the other hand, H.323 is somewhat of a lowest-common-denominator protocol, and the newer protocols will probably be adopted because of increased functionality; until then we'll need to get IP telephony services to adopt IPSEC.

    IPSEC is still only marginally ready for prime time, but capabilities and compatibility of free and commercial implementations are improving, and there's substantial business demand pulling them. The automotive industry ANX network jumpstarted it, but the cost advantages of dial internet compared to running your own modem pools are one of the big drivers, and for some industries, the ability to use the internet instead of private frame or ATM networks for corporate traffic is also a big economic win, though that's more dependent on communication patterns.

    I suspect end-to-end encryption for cellphones will be a small niche market for a long time, as opposed to encrypting the airlink from the phone to the cell site. What may change it is the obvious interconnection between voice over IP and cellphones merging into internet telephony to the cellphone. Cellphones already digitize and compress voice, which is one of the hard parts, but cellphones take a telephony-centered view of mobile connectivity which will take some work to merge with the still-evolving mobile IP technology. The obvious first level of integration is gateways between the cellphone carriers and the internet voice carriers, which makes it easy to still charge by the minute for cellphones. In countries that use handiphone service (mostly Asia - it's the "you can use the phone anywhere but we don't switch cells, so you can't move very far" dumb cheap technology), it wouldn't be too hard to integrate a handiphone base station with DSL so anybody could run their own microcell and get their cut of the cellphone charges, which has viral marketing possibilities that are harder to implement in a usable-while-moving true cellular system.

  5. It's a policy to accept applications, $$, comments on ICANN Has Approved New TLDs · · Score: 1
    The icann.org page will probably be slashdotted soon:-), so I've included it below (reformatted a bit for slashdot.) Basically, it'll cost $50K to apply to propose a new TLD, apply by October 1, public comments closed by October 15th, winners announced Nov 20, haggle details with the winners till end of the year.

    ========== Section from ICANN minutes ===================

    New Top-Level Domains

    Whereas, the Domain Name Supporting Organization
    (DNSO) has conducted a consensus-development process on the introduction
    of new TLDs and the issues concerning the protection of famous trademarks
    in the context of introduction of new TLDs;

    Whereas, the Names Council of the DNSO made a
    set of recommendations to the Board on 18/19 April 2000, including
    the recommendation that the Board establish a policy for the introduction
    of new gTLDs in a measured and responsible manner;

    Whereas, the Names Council of the DNSO made a
    second set of recommendations to the Board on 19 May 2000, which concerned
    Famous Trademarks and the Operation of the DNS;

    Whereas, the ICANN staff has posted a document
    entitled "ICANN Yokohama Meeting Topic: Introduction of New Top-Level
    Domains" on 13 June 2000 and sought public comment on the web
    site concerning the Names Council recommendations and related issues;

    Whereas, over 1,300 comments were received on
    the ICANN web site in response to the staff posting;

    Whereas, on 15 July 2000 a public forum was held
    in Yokohama concerning the issues discussed in the staff paper;

    Whereas, the Names Council recommendations were
    transmitted to the Protocol Supporting Organization and the Address
    Supporting Organization for their comment regarding the implications
    on activities within their scopes of primary responsibility;

    Whereas, no negative comment was received from
    either Supporting Organization;

    Resolved [00.46], that the Board hereby adopts
    the Names Council's recommendation that a policy be established for
    the introduction of new TLDs in a measured and responsible manner.

    Resolved [00.47], that the President is authorized
    to implement this policy according to the following schedule, which
    the President may adjust if necessary to accommodate circumstances
    that arise:

    • 1 August 2000 - ICANN to issue a formal call
      for proposals by those seeking to sponsor or operate one or more
      new TLDs, accompanied by a New TLD Registry Application Form, instructions
      for filling out the application, and a statement of criteria for
      the Board's eventual decision.
    • 1 October 2000 - Deadline for ICANN's receipt
      of applications. Portions of these applications deemed appropriate
      for publication for purposes of public comment or otherwise will
      be posted on ICANN's web site.
    • 15 October 2000 - Close of period for public comments on proposals.
    • 20 November 2000 - After approval by the Board,
      ICANN to announce selections for negotiations toward entry of agreements
      with registry sponsors and operators.
    • 31 December 2000 - Target date for completion of negotiations.

    Resolved [00.48], the President is authorized
    to establish a non-refundable fee of USD $50,000 for the submission
    of an application to become a sponsor or operator of a registry, which
    the Board finds is a reasonable estimate of ICANN's costs likely to
    be associated with receipt and evaluation of such applications, and
    follow-up.

    Resolved [00.49], in connection with applications,
    the President should seek information that he determines is appropriate.
    Without limiting the information that may be sought, the Board commends
    to the President's consideration the data elements described in section
    IV of the staff paper, and also notes that the data elements should
    include:

    • full information about the technical, business,
      management, and financial capabilities of the proposed operator
      of the registry;
    • a detailed description of the policies contemplated
      to promote orderly registration of names in the initial phases of
      introduction of the TLD;
    • full details concerning arrangements proposed
      to protect users in the event of registry failure; and
    • measures proposed for minimizing use of the
      TLD to carry out infringements or other abuses of intellectual property
      rights.

    Resolved [00.50], that the President is authorized
    to establish guidelines for assessing which proposals to select for
    negotiations toward entry of agreements with registry sponsors and
    operators. The Board commends the following topics to the President
    for inclusion in the guidelines:

    • The need to maintain the Internet's stability,
      and especially the protection of domain-name holders from the effects
      of registry or registration-system failure.
    • The extent to which selection of the proposal
      would lead to an effective "proof of concept" concerning
      the introduction of top-level domains in the future, including the
      diversity the proposal would bring to the program, such as fully
      open top level domains, restricted and chartered domains with limited
      scope, noncommercial domains, and personal domains; and a variety
      of business models and geographic locations.
    • The enhancement of competition for registration services at the registry and registrar level.
    • The enhancement of the utility of the DNS.
    • The evaluation of delegation of policy-formulation functions for special-purpose TLDs to appropriate organizations.
    • The extent to which the proposal would meet previously unmet types of needs.
    • The importance of appropriate protections of rights of others, including intellectual property rights, in connection with the operation of the TLD, especially during the start-up phases.

    Resolved [00.51], that the President is authorized to seek technical advice from appropriate individuals or organizations to assist the evaluation of proposals.

    ========================================

  6. Music and LISP; Alice's Lisp Machine on What About Functional Languages? · · Score: 1
    Bob Kanefsky's Songworm Filk record has Eternal Flame, the "God wrote in LISP code" parody of Julia Eklar's God Lives On Terra, sung by Julia.

    Also, there's a cached version of Alice's Lisp Machine, which has pointers to open-source figure Richard Stallman and lots of AI-Lab people and in-Jokes.

  7. Event listeners were hard in procedural langs also on What About Functional Languages? · · Score: 2
    Remember learning to program for X Windows or other graphics systems back when you were a C programmer? It was a major step - instead of writing code where all the subroutines belonged to you, except library stuff you could call to do things for you and return with the answers, you needed to learn to write code that
    • Had functions with arguments in a format that event-meisters could call
    • Covered all the reasons an event-meister would try to call you
    • Used C's appalling syntaxes for asking the event-meister to call your routines if something happened instead of writing your own event loop, because your program was no longer in charge of what got called when.
    Maybe that's as unnatural to do in a functional language as in a procedural language, but you have to give the functional folks as much slack as you've used. My friends who write Smalltalk swear by the stuff. A friend of mine used to write window-app prototypes in "winterp", an Xlisp windowing package that ran on X Windows, and said that the prototypes generally took much less time to write than the product C code, but also did more.

    The real issues are the ability of teachers to teach the stuff, the quality of books to teach or learn from, and the availability of programming environments and tools for not only learning but also production. The latter used to be a hard problem, but now that you can include a CD in a book or put a few tens of MB on a web site, and everybody has access to Windows, it's less difficult, and we return to the previous problem of bootstrapping the learning process. Java pulled it off, but it was in the right place at the right time, with "fast web application development" as the hook when that was needed. Academics aren't always as good at self-promotion as commercial marketing departments.

  8. Competing Spammer Rating Services on MAPS RBL Challenged In Court Case · · Score: 2
    MAPS is one of several services that rate spammers and spam-supporting ISPs. They're one of the best-known, as is ORBS, who are in my opinion too aggressive and not flexible enough, but that's a matter of taste. But there are other rating services out there. Many email services have anti-spam options - some run their own, some outsource to MAPS, and some don't say. Does anybody know services that do an especially good or bad job?

    I use pobox.com as my stable email address, and their anti-spam service has a choice of marking or discarding spam. I use marking, and find it's 90-95% accurate.

  9. Content-filtering services and libertarianism on MAPS RBL Challenged In Court Case · · Score: 2
    There's a difference between saying "People shouldn't do X" and saying "The government should ban people from doing X". The former is a moral or aesthetic judgement, and libertarians can make any such judgements they like. The latter is an unlibertarian call for government action, which wasn't what prolixity said. Libertarianism can include a wide spectrum of opinions, from "People should/shouldn't watch TV televangelists" to "People should/shouldn't watch porn on TV", as long as you're not calling for the government to ban those options.

    In this case, MAPS RBL is a content-rating service, and you might or might not like their ratings. If you like them, you might use an ISP that provides MAPS as a blocking service; if you don't like them, you should pick a different ISP. I use pobox.com's optional spam rating service, with the "mark but don't block spam" option, and it's right about 90% of the time.

  10. My mailing list is being harassed this way on MAPS RBL Challenged In Court Case · · Score: 2

    I'm on a large mailing list that will remain nameless. It has a policy of allowing anybody to post to the list, whether they're subscribed or not, because we believe in openness and tolerance of anonymity, and are willing to tolerate a certain level of spam as the cost of open access. Unfortunately, some harasser has recently forged subscriptions to lots of non-confirmation mailing lists, so the list is getting hit with them. Even with confirmation-based lists it's a problem, because the harasser (who's presumably subscribed to the list) can receive the confirmation requests and reply to them, but it does cut down on the harassment, and it makes it easier for other people to do unsubscribes.

  11. AOL's delay in responding makes it interesting on Interview With Mike Sklut · · Score: 2

    That and the fact that the security hole was found by an 11-year-old kid. But three years is a long time for that simple a hole to stick around.

  12. Promiscuity isn't enough on Earthlink Refuses To Install Carnivore · · Score: 2
    Cringely blew it. A couple of posts (250 and 112) have touched on the issue. Promiscuity isn't enough to receive everything, because there is no central point that it all goes by to receive it at. Any medium or large ISP or colo center, or even most small ISPs that have multiple locations, have a bunch of routers and switches that are designed to keep traffic flowing on the LAN or WAN segments where it's needed and not flowing on segments where it's not needed, because you have to do that to make things scale. A colo center might put your host on a 100 Mbps Ethernet with a dozen other hosts, and you can sniff their traffic, but the data switches that get the traffic from their 155 Mbps OC3 or 2.4Gbps OC48 to the Internet aren't going to shove the traffic from all the other 100Mbps Ethernets in the building onto yours - it won't fit. Each one gets only their own local traffic. Buying one host at an Exodus location isn't going to snoop all the OC48s coming into the building, nor will it snoop all the traffic going between servers in the same building (big hosting centers get a lot of traffic like that.) If you know that usualsuspects.com has a web server there, and you asked really nicely, you might get put on the same Ethernet segment, but that's not the one that whitehouse.gov or gambino.org are on, so it doesn't do you much good.

    Some ISPs might put all their mail servers on one big fast Ethernet so everything routes there, which makes it easier to do centralized management and some security, but traffic that isn't going to those mail servers doesn't go to that segment. This means that if you dial in to ISP A, and use your web client to access a web server at ISP B, or your POP client to access a mail server at ISP C, or your email sender to send mail to an SMTP server at ISP D, you're probably not going through ISP A's POP server Ethernet, you're just going through the LAN connections that get you to the routers going to those other ISPs. If it's all in one building, the carnivores might hang a bunch of promiscuous taps on every segment there and go into some big hacked multiprocessor router-thing, but anything less won't cut it.

  13. Floating point is actually very standard on Future Of Internet-Based Distributed Computing · · Score: 1
    Just about everybody uses the IEEE floating point standards these days. It's a bit complex, but people bought into it. If there are exceptions, they're mainly in DSP chips, not floating point support on CPUs. That doesn't mean that you don't have to pay a lot of attention to numerical behaviour if you're designing algorithms, because it can matter a lot what order you do calculations in and how the errors propagate.


    Also, while SETI is highly float-oriented, and nuclear engineering and oil-company problems may also be, crypto and big primes and similar problems are purely integers - you can do just fine on a Celeron.

  14. "Hands On San Francisco" and Related Organizations on Where Can One Find Computer Related Charity Work? · · Score: 2

    Hands On San Francisco is an organization that coordinates volunteers for community service projects. There are related organizations in New York, Atlanta, Los Angeles, and Washington DC. They're not a place to find a high-computer-skills job, but they do need volunteers, and high-tech companies in the city are welcome.

  15. Re: Habitat for Humanity on Where Can One Find Computer Related Charity Work? · · Score: 2

    However, if you're not the type to get off your ass (:-), Habitat for Humanity folks say that it's always easy to get people to show up for the glamorous parts of building a house - there's other hard boring work to be done preparing for it, and there's a lot of need for money. Send them a check, or Donate Online using credit cards.

  16. Medical Research / Biotech Companies on Where Can One Find Computer Related Charity Work? · · Score: 2
    OK, so you want to help humanity while continuing to do geek work at geek salaries rather than charity salaries. There are a lot of companies in the medical and biotech research area that have similar cultures to the computer business - anywhere from stuffy corporate to fast-moving startups - but instead of creating the next cool computer game that prevents teenagers from getting exercise (:-), you'll be creating the next life extension technique or disease cure. Those companies are increasingly computerized, so you don't need to be a biochemist to work there, though that sort of background can help. It'd be cool to cure AIDS, but even small steps like a better arthritis painkiller or a less dangerous chemotherapy drug can also be a major win for humanity. Or you could help your company develop ways to get drug testing through the FDA faster - amazing numbers of people die who could be helped by medicines that are stalled in the approval processes.

    Your stock options are just as likely to turn to wallpaper as in the computer industry, of course, but you get to feel extremely self-satisfied if it pays off....

  17. Re:Donated CPU Time on Future Of Internet-Based Distributed Computing · · Score: 2

    Typically, no, you can't write off your donated CPU time, because you weren't making any money off it so there's nothing to write it off against. It's similar to not being able to write off your labor for time you donate to charity. If your PC belongs to a business, rather than being your personal PC, you might be able to do Fancy Accountant Tricks to write off some of the depreciation on the PC, but you've got to have a good way to audit how much of the resources got used for work vs. charity, and even then it'd probably be pretty dodgy. (Neat trick if you can do it, since you're probably using 95-99% of the CPU time for the background CPU-burner, but that depreciation was already an expense so you already got to write it off; it's not likely to get you anything.)

  18. Great Internet Mersenne Prime Search mersenne.org on Future Of Internet-Based Distributed Computing · · Score: 2

    GIMPS, the Great Internet Mersenne Prime Search still needs your CPU cycles. It's good math, and can use all the CPU it can get, and it's found four of them already. It runs quietly in the background, and cooperates will with firewalls and with full-time or part-time internet connections. I don't recommend running it on laptops using batteries, since it eats power, but it's fine for any machine that's plugged in.

  19. Security for Participants and Projects on Future Of Internet-Based Distributed Computing · · Score: 3
    CNN does mention security, but only in the context of security of the project itself - nobody minds if information about a public project leaks out, e.g. finding big primes or little green men, but a commercial customer would probably be more concerned about their information leaking out if they're using a distributed processing service for solving their problems. Some of this can be helped by encrypting communications to the coordination server, but that doesn't protect you from the people running the PCs doing the calculations.

    What CNN doesn't talk about is security for the participants' machines. Open source is helpful, because you can see what you're running, and people can find bugs in it, but that's really most effective for the first few special projects like GIMPS, distributed.net and SETI than it will be for running arbitrary code in a large distributed-processing industry. The worst case would be malicious distributed-processing code (either viruses or simple DDOS applications), but even non-malicious code with buffer overflow bugs could be a real disaster, both to the PC users and to whoever their machines might be used to target. It's possible to be somewhat safer by using sandboxed computing environments, such as Java, so everybody knows their machine will be safe, but they tend to be much slower than running compiled native applications. This can be improved somewhat by using standard compiled libraries, e.g. bignum calculations, but it's still a wide open problem.

    Are there any environments you know about that are safer, or safe enough and faster?

  20. Reinventing Onion Routing / Pipenet on Fling:Anonymous Protocol Suite · · Score: 3
    A first readthrough of the documentation tells me that it needs better documentation :-) No surprise - new projects are often that way, but doing a good architectural description is a critical factor in making a project like this succeed, because people can understand where it needs work and where to help, and because security models need to address a lot of issues to be able to meet their goals.

    I had trouble telling what the technical goals of the project were - are they addressing traffic analysis, or only protecting content? They're describing a bunch of complex shuffling, but don't indicate why they chose those methods and what attacks they're trying to protect against. Some of the earlier projects like Pipenet and Onion Routing found that there are theoretical weaknesses if you only send traffic when you have real traffic, or if you do anything that makes it possible for an eavesdropper to tell what the boundaries between messages are, because the eavesdropper can do enough correlation to identify reasonably accurately where the traffic is going. The alternative is to build connections between sites that always have constant traffic levels, using filler traffic when there's no real traffic. This has a major cost/performance impact that affects the willingness of servers to support this kind of application. By contrast, IPSEC gives you all the privacy you need by encrypting, but doesn't try very hard to block the user identification.

    Privacy servers like this also depend on having lots of users - if there are only two people using it, it's easy to tell who's communicating with whom. It's nice to do technology, but you also need to work on a social or business model that encourages lots of people to run the client, and if it's got separate servers, to run servers as well. That's one of the cool things about Zero Knowledge - they've got a model that they hope will achieve this, though whether they succeed will depend on whether they implement it well enough for users to accept it and whether they can market it well enough to really take off. Some things are overnight successes - Hotmail, Napster - while others limp along at a low level for a long time, like the current remailer networks, mainly because they're annoying to administer and responding to complaints when they're abused is annoying. I wish the Fling folks good luck - but there's a lot of work they've got ahead of them to make it working and accepted.

  21. Lunch Time. Pay Phone. The old days. on CNet On Online Freedom · · Score: 2

    (All you young folks - why when I was your age, we had to walk to work uphill both ways in the snow....) So 20 years ago, most people did this sort of thing on their lunch break or coffee break, if they worked at a job that wasn't flexible about that sort of thing. It was especially true for factories, but office jobs were often that way as well. Bell Labs was still part of The Phone Company, and there were pay phones in the building you could use for long-distance non-business calls, which still cost actually money back then instead of being basically free like they are today - and the PBX or Centrex had call detail records, so if you made personal calls from your desk you could identify which they were and how much they cost. But at least some businesses you needed to deal with would talk to you on the phone - if you needed to deal with the Motor Vehicle Department, you had to go in person, either at lunchtime with everybody else, or during work hours if your boss didn't mind you talking the time, or the one evening a week they'd be open late (late meaning about 7-8pm.) Banks were much the same way, though ATMs were starting to emerge.

  22. DNS and Fancier Load Balancers Already Do This on Will BXXP Replace HTTP? · · Score: 2
    Already been done. The standard way people accomplish this is to use URLs like http://www.foo.com/image.gif, and have their DNS server at foo.com rotate what IP addresses it gives out, pointing to machines that are also called www1.foo.com, www2.foo.com, etc.. The simple round-robin model just rotates between them; fancier load balancing servers from a variety of vendors check which servers are least busy.

    Another approach that some fancy load-balancers use is to always give the same IP address, but fake out the packet requests using NAT or other routing tricks, or having the web servers themselves change their IP addresses dynamically. It's a bit uglier, but browsers and DNS resolvers often cache their results, so a web server might die or get overloaded but users are still requesting pages from it because they've cached its IP address - this way you spread out the load more evenly at the cost of more dynamic address-shuffling translation work by the load balancing server.

  23. Selling Eyeballs to Advertisers, of course on Yahoo Will Use Google Instead Of Inktomi · · Score: 2
    Yahoo sells eyeballs to advertisers. I'd guess that with the new Google search engine in place,
    the display the user sees will still be a Yahoo-formatted page, with Google search results but Yahoo-managed banner ads, hyperlinks, and other decorations. I'm more interested in what Yahoo does to link the banner ads with the search terms or search results, though perhaps Google also gets to sell that information to advertisers or information aggregators, even though they're not providing the banner ad themselves.


    Remember when the Internet term "IP" meant "Internet Protocol" instead of "Intellectual Property"?

  24. Big RAM in Database, Web servers on IBM Promises More Memory In The Same Space · · Score: 1

    No, a file server for your department doesn't need moby RAM. But a large web server or a enterprise-sized database server both win by using lots of RAM to cache data, because RAM really has a few orders of magnitude faster throughput than disk drives, and the latency is much lower which makes a lot of difference for databases. Even a few years ago, it made sense to toss a GB of RAM into a database server, and now that that's only $1000 or so, it usually makes sense to buy more.

  25. Preventing memory expansion; cache block sizes on IBM Promises More Memory In The Same Space · · Score: 2
    A standard hack that on-the-fly compressors use is to keep a flag bit indicating whether a given chunk is compressed or uncompressed. That way you can avoid the problem that recompressing already-compressed data often grows, at the cost of a flag. On the other hand, if most of your data is compressed anyway, like web servers caching lots of JPEGs, you still don't win, and you're hauling it uncompressed on most of the busses anyway.

    I am interested in how big a block of memory this stuff operates on. Compressing a few K at a time, such as a disk block, may be big enough to win, but compressing a 128-bit cache line almost certainly loses. Where's the breakeven point? The hope of this technology is that if you stick it way out in L3 cache, you're usually hauling big enough chunks at a time that the decompression latency for getting the first byte is made up for by the bandwidth of getting the rest of the bytes from a smaller amount of memory.


    Help, help, I'm being compressed!