Slashdot Mirror
Open Source Library Card-Catalog Apps?
dmd writes: "Does there exist Open Source software for maintaining a small to medium sized library card-catalog? It seems all the tools are available:
a perl module for working with
MARC records, several for working with Z39.50 and XML, and even a web site apparently devoted to nearly this exact topic. An actual, working, catalog, however, seems to be missing. Is this something that would be valuable? I, for one, have nearly 5k volumes in my collection, and they're begging for some discipline." I'm sure cash-strapped public libraries and schools would like to be able to use free / Free tools for this, since paper books aren't going away anytime soon. Not to mention for CDs, videos, charts, museum holdings ... any ideas out there? Turnkey solutions?
Actually, this is something I have been looking into for months. The reason being that most elementary/middle/high schools can't afford a decent catalogue system. Several that I have seen have been using software with (no kidding) CGA graphics interfaces, and searching by title only.
Those schools that do have money move to software like Eloquent -- systems that are way more complex than a school library typically needs. Most schools don't need that much power/customisation, and can't afford it anyway. What seems to be needed is a basic system that offers searching on author/title/subject/keyword, and possibly uses MARC records (though for a school library this is not essential).
It would have to be easy to set up, and low maintenance (ie. a basic linux box shoved under a desk somewhere with a UPS and a tape backup). You need to keep in mind that libraries -- and school libraries in particular -- are likely to have a multitude of machines running different OSes, so something like a web interface would be perfect.
Considering the fact that most schools are getting networked these days, it's feasible to have a linux box sitting under a desk somewhere running a database, some library software, and Apache, and a bunch of Mac/PC clients running MacOS and Windows and interfacing to this thing via a web server. The checkout could be the same idea. This could be extended to have non-web clients running on various platforms and talking to the server via CORBA.
In talking with librarians, I've found that you can't just say "dump MacOS/Windows and put Linux on all your machines" because they don't just use them for searching. They use them to run all sorts of stuff -- CD-ROM based educational software, etc. In other words, it's important to remember that for software like this, you can't just get a bunch of developers together and make decisions and write code. There are a ton of assumptions you just can't make when you're dealing with libraries and schools. There's a bunch of research into what people really want that's required. That makes it a little trickier a project than, say, a mahjongg game -- no offense to mahjongg hackers...
Anyway, this is a fantastic opportunity for development, and one that I have been very interested in for a while now. It's also been on the GNU project's list of stuff to do for years now. Contributing a GPLed library system would be great not only for Free Software, but also for schools everywhere who can't afford decent software in their libraries.
And I don't mean this to sound like a slam against MySql. No SQL database could do it in a way that a librarian would be completely happy with, primarily because of the wonderful MARC format.
The MARC format is the standard format used to store biliographic information. It was originally created in the early 60's, with the idea that the primary means of transmission would be on tape. It supports well over 300 different major fields, ranging from simple ones that anyone would understand (auther, title, publisher) to arcana that only a trained librarian could love (is the item a festschrift, unusual pagination comments, magazine run dates, and on and on.) Most of the major fields have "sub-fields", where the data is broken into different elements (i.e. an author field field will have a name sub-field, a dates sub-field, a title-subfield, and possibly others.)
Fields in the MARC format have a theoretical maximum length of 10,000 characters. Many of the fields can be repeated any number of times (co-authors, variant titles, subject headings). I've seen several attempts to model the MARC format in a relational model, and while it can be done, it's a royal pain in the ass and it inevitably winds up with trade offs.
For a simple catalog, where you aren't worried about working with the MARC format, a relational database (including MySql) will be perfectly adequate. But librarians love the MARC format, and it is such a basic element of modern librarianship that any system that couldn't import and export it would be considered unacceptable - like a car with a crank starter.
And I should know. I worked as a librarian for several years; I even have the MLIS to prove it.
Commercially available library software that is actually used by libraries is much more than just a cataloging/look-up system to replace those old 3*5 cards.
You need an acquisitions module that has the ability to do electronic ordering and approval plan processing.
The search and report capabilities on the staff interface for these things is amazing. I can collect a list of all item records belonging to location X and created within [ range of dates ] that are attached to bibliographic records for [ material type ] within a [ call number range ], sort the records according to my criteria, then output selected fields from either the bib. or the item, or both, in the order I choose to the device of my choice (including print to e-mail or fax) and I haven't even begun to make the system sweat. Yes, this is a fairly straight forward thing to do (selecting records based on data spread across multiple related/linked records) in SQL but, you also need a front end that the end user can comprehend.
If you're going to code it, it will need to be able to interact with all of the prevailing vendors... Ebsco, Baker & Taylor, Basil Blackwell, Swets & Zeitlinger, Matthews, etc... You will want tech contacts from each of these vendors to fine tune the ordering/receiving/approval interfaces.
Finally, the amount of fiscal reporting done in libraries can boggle your mind. You would never suspect that something so seemingly simple could be so complicated. If you don't have the ability to generate financial reports you might as well go back to index cards and hand written ledgers.
My office has been taken over by iPod people.
Almost every library, from the Podunk Public Library to the Podunk University Library, doesn't use tapes anymore. Libraries use the internet to connect to the Library of Congress via Z39.50 and get their MARC records free of charge from LC's Z39.50 server. OR if LC doesn't have it, they can connect to the hundreds of thousands of other library databases hosting Z39.50 servers and get the records from them, for free. OCLC is not the only game in town anymore.
FLS uses mysql and java to do it's work. I once emailed the guy who wrote it and he said there wasn't much interest in it, so he stopped developing it. Maybe he/someone else would like to take the 'code' and run with it. http://freshmeat.net/news/1999/03/28/922680974.htm l
Yet another case of the government giving away the farm to some pack of rich assholes. But how many times in the past have they granted exclusive rights to something that the goverment spends million$ of taxpayer dollars to produce (oil rights, grazing rights, timber, mining, etc)?
It's also another example of the LoC getting drunk on technology. There was a great article in the New Yorker about the LoC getting a bee in its bonnet about microfilming. In the process of microfilming newspapers, warehouses full of bound copies of DECADES of original newspapers were tossed/sold/given away only to be replaced by incomplete, blurry and often illegible microfilmed copies. Original sources? Gone *forever*.
The article said that the internet and the web are the next technologies that libraries will sink millions into all while pitching their original materials. We're litterally throwing away our history as we generate it.
>Incidentally, the "go get MySQL, you dumbass"
> posters are missing an important point:
> libraries use the MARC data standard for catalog
> records, and
> SQL doesn't cope well with the kind of
> tricks MARC can do.
Several people have said that, but I was hoping for an example or two to support the statement.
Not that I'm interested in MARC, but that I'm curious about where people percieve that SQL has such limitations.
-fb Everything not expressly forbidden is now mandatory.
You obviously know nothing about libraries, nor databases. Creating a good library system would take years of work. It's much better to buy a canned system, or hopefully a OSS system designed for the job.
This is a fascinating programming task and I was glad to discover the references Slashdot linked to on this subject.
I've never been happy with card catalogs and their electronic descendants or with web based search engines. Knowledge exploration needs a lot of work, and these kinds of tools will lead to interesting results.
If I was young and had lots of time, this would be a really challenging project to work on. Designing a schema to organize knowledge is a really cool task.
Micro$oft(R) Windoze NT(TM)
(C) Copyright 1985-1996 Micro$oft Corp.
C:\>uptime
if it can speak Z39.51, and just search the LOC for full entries based on ISBN. (Unfortunately, I don't know of such an OSS product.)
I understand the Library of Congress made a deal with a database company, which now has exclusive rights (and a compilation copyright?) to distribute the catalog information in electronic form.
Want it? Buy it from them. Complete with their software and a limited license.
I'll try to get the person I heard flaming about this ("Flagrant misappropriation of public records/product of tax dollars", etc.) to post with the details and whether it's still current.
The patent office aldo cut a similar deal at one point. It was still in effect as of maybe four years ago (when I interviewed at what turned out to be the database company that had the USPTO's sweatheart deal).
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
You can't just code a database -- that's almost entirely useless; there's also the matter of controlling circulation, tracking books out/returned/requested/held/sent to bindery, etc.
Plus import & export from vendors, billing, accepting bill payments, cross-referencing, all kinds of freaky subject indexing, mondo-bizarro file formats from a zillion years ago (MARC), etc. etc. etc.
There's a reason library systems tend to be proprietary -- it's because nobody else in their right mind wants to get involved with things like MARC and Z39.50.
. . . but then again I could be wrong.
I have no
As stated in other posts, SQL can have serious problems with some of the tricks that go on with MARC format.
My office has been taken over by iPod people.
Second is that half of the pieces that go into a big library management system (including the catalog part) are really generic business systems: EDI, invoicing, accounting, etc., but they haven't been abstracted out of the realm of our systems vendors. So the level of standards followed there is minimal so those modules generally don't interoperate with our trading partners (i.e. internal payment systems and external suppliers). Lots of redundant keying and more crappy systems to maintain there, all of which is typically deeply and proprietarily tied into the catalog data.
All that said -- and to our vendors' credit they are tending to get better these days -- we've been sharing catalog data like hackers are sharing code for over 100 years. We've been doing it online for about 35 years, but the way we do it now is pretty much the same way we've been doing it for those 35 years. i.e. largely dependent on one of two .orgs/vendors to be a clearinghouse for sharing catalog data. But those folks disappear if they can't sell the data back to us after we create it for them. So nobody running a library wants them to disappear. Especially because we've got to handle one-of-a-kind rare items in big research libraries as well as unusual local items in public libraries and so on.
Imho the solution is to first outsource all the standard business stuff to vendors+free software that can do the same job with existing standards-based tools. Then abstract away as much as possible of the catalog data into free references sources shared and maintained by the library community (think: you could run your own amazon.com recommendations site, etc.). This is what we're trying to do (shameless plug alert) with the jake project for journals. Same thing applies for books, although there are probably >=100M records to normalize.
If we can get that done, then anybody could hack up a gtk+ front end to the free, shared catalog, and pick and choose the items you have yourselves. It would work sorta like dict.org or jake. Just imagine how much easier it will be to search for ebooks in gnutella once this is done... :)
. . . nevertheless, a database is (by itself) almost entirely useless. It's like saying "hey, you asked for an airplane, so here's some bulk aircraft-grade aluminum. That's good enough, right?"
Unless, of course, you're talking about a "library" system that can't talk to any other library system, doesn't understand any standard library data format, cannot respond to queries in a standardized way, cannot do billing, or payment, or MeSH indexing, or anything else that library systems do.
The database is the raw material, not the finished system. There's a reason there aren't very many library software packages -- nobody in their right mind wants to write one. It's a nightmare.
Go back to writing online shopping carts.
I have no
Another example is nautical charts. All the data collection, etc. is done by NOAA, but Maptech, Inc. has been granted exclusive distribution rights. The result is that it is impossible to get nautical chart data without paying Maptech's outrageous fees. And this is data which taxpayers have ALREADY paid for!
Anyone looking for an open source application for keeping data about their CD's, MC's, LP's etc. may look at Project CrossRec http://www.crossrec.org which works with developing an open source (BSD license) front-end and sentral repository for all data regarding audio recording on any medium. The apps are RDBMS based (PostgreSQL and others), and supports data and relations between data not seen in commercial systems.
While most people would tend to think this is a fairly simple endeavor, it's not. Our consortium runs a library automation system on top of VMS on an Alpha cluster. It happens to be one of the best systems out there as far as high availability is concerned. However, it still doesn't take user's needs (Ref. librarians and patrons) into account.
If something like this is going to work, you have to anticipate what your users are going to be asking for. A simple cataloging system could probably be done with MySQL or Postgres, but it will lack any of the functionality that is in existing library catalog products.
In addition to the MARC records, you would eventually need to maintain a patron database, an acquisitions database, a record maintenance interface, circulation records and policies, etc... AND it would all have to be easy to use for a non-technical person. Even the system we use is not "easy" for the average user, it's jst reliable.
Personally, I would love to see an open source and/or free software catalog system that outshines systems like Dynix and DRA. Especially if it brings user interfaces into the year 2000. (There is plenty of talk about new client server interfaces, but nothing has come to fruition as of yet.)
I think the biggest challenge a project like this would face is that programmers are not librarians and vice-versa. They come from very separate worlds and have very little understanding about what the other discipline finds important in an automation system.
Peace,
D.B.
Don't complain about my web page. It's mine. ALL MINE.
MARC is an extremely flexible data structure. Relational databases tend to view the world as a set of linked tables with data elements. Of course as anyone who has ever survived a CS data structures course can tell you, you can use almost any data structure to store any other data structure. Its mostly a matter of efficiency. Yes, people have stored MARC data in a relational database. It worked, sorta of. Think of a table where any column could be repeated any number of times, or omitted, or be of any length, or .... A book can have no authors, or 1 author or lots of authors, or some authors are really corporations, and other authors are really pseudonyms, and some authors have different names in different languages.
Not an insurmountable problem if all you have
is a relational database. The usual method is breaking the MARC record up into lots of itty-bitty pieces. You have the author table, the title table, the uniform title table, the serial title table, etc. The problem with breaking it up is you have to put it all back together again. Think about doing a join across 900 different tables. At this point CJ Date starts in with denormalizing your data....
Its a case of when you have a hammer everything looks like a nail. Relational databases have their place, but they aren't always the best way to store and retrieve all types of data. There are databases which aren't relational, and sometimes they outperform a relational database on specific types of data.
Presumably people have looked at PICK (allows multivalued fields) and object oriented databases for this sort of thing? OODBMSs are very good at handling data with this sort of complex structure - see www.exceloncorp.com (Excelon, formerly Object Design), www.objectivity.com, and www.versant.com for some commercial examples. There are also some open source near-equivalents but I don't know the URLs - one is called Texas Persistent Store, I think.
As with RDBMSs, there are some arguments for going with commercial ODBMSs if you have very demanding requirements. The commercial ones also have lots of extra tools, many of them dealing with XML, which involves complex nested data structures and is also suited to ODBMSs.
...the app you're looking for is called a DATABASE!!!
Code it. There is a major customization problem with such a product for as you have mentioned each library would need to add a module based on what they currently possess/maintain (cd's, videos, microfiche). A simple database in SQL might also do the job with a PHP4 front web interface. Dunno...its up to the person that needs the sucker done.
You are a unique individual, just like everyone else.
-*-*-*-*-*-*-*-*-*-*-*-*-*-*
Sig it.
There are applications out there (i.e. GnomeCard) that could easily be adapted to this purpose. Think about it: Catalogue cards are basically "book business cards" -- they tell you how to get a hold of a particular book. So why not adapt a contact manager to this purpose? It surely would be easy.
--
We may not imagine how our lives could be more frustrating and complex—but Congress can. – Cullen Hightower
I believe you can get all the authority tables from the LOC if you subscribe to their cataloging service... there's got to be a better way. FOIA, maybe?
As part of my undergraduate curriculum at WMU, we had to work in small groups on a reasonable size project to complete the baccalaureate writing requirement. Although most projects were for other departments in the University, my group and some others worked for outside nonprofit groups. In particular, my group worked on a database system for the Kiwanis Club of Kalamazoo. Perhaps you could contact your local university and see if they don't have a similar course? Given the right students you could have yourself a great program.
- Zebra information server. Eats Marc (UsMarc, other local variants) as well as XML, mails, newsgroups, etc. You can add more input filters. Talks Z39.50
- Yaz Z39.50 toolkit for client and server side
- Zap web gateway and a PHP module for building easy search gateways to anything that understands Z39.50, for example our own Zebra
- and more. Even more to come later...
I am of course biased, but these tools are designed for library applications. All open source, at Index Data.
In Murphy We Turst
Instead of state/local/federal governments spending money so that specific schools or libraries (etc.) can purchase proprietary software, why not just roll it all together, total up all the amount, and then spend maybe 50% of that on funding some open source development?
From there on, each school can spend a small amount of money paying someone to customize it to their needs, but we could have interoperable, open, inexpensive software where it's really needed.
---
Open Source Digital Library System
Note: Those of you simply suggesting ordinary databases don't have a clue as to what is actually involved. Yes, you need a database. But that is only one of MANY pieces that make up and automated library system. Commercial software for this stuff can cost tens and tens of thousands of dollars.
An open source system would be welcome, indeed.
What restrictions, if any, are there on the use of Library of Congress data? The quality and quantity of data available on their Z39.50 port looks quite impressive.
I sent them an email about this a couple of weeks ago but did not receive an answer. Any insight would be appreciated.
LILO boot: linux init=/usr/bin/emacs
for more on how even librarians (who might be expected to have more archival appreciation) are "throwing away our history as we generate it", check this excerpt from Nicholson Baker's well-researched and insightful "Discards":
- And abruptly you realize, looking at these expressive dirt bands [caused by patron handling of cards], that even the libraries, like Harvard and the New York Public Library and Cornell, who microfilmed or digitized some of their cards prior to destroying them, have - by failing to capture any information at all about the relative reflectivity of the edge of each card - lost something of real interest, something eminently studiable. Who knows what a diligent researcher who photographed (from above, on a tripod) each close-packed drawer of Harvard's Widener catalog with a high-contrast camera might find out, were he to correlate his spectrographic dirt-band records with the authors that, as distinct clumps, exhibited some darkening? Of course the "Kinsey" cards would be thoroughly dirt-banded - but which others? This is, or was, a cumulative set of scholarly Nielsen ratings for topics at twentieth-century Harvard that is perhaps more representative than any other means of surveying we have. Instead of tossing its catalog out, Harvard ought to have persuaded a rich alumnus to endow a chair for dirt-banded studies.
it's an excellent read for anyone interested in the "books meet bytes" situation.---
the problem with teens is they're looking for certainties.
Hollywood, Television, has become the dream machine. We need to take that back; each of us is a Dream Machine
Koha is probably the most complete system out there. 2 other projects that are attempting this (and also trying to work together) are the Open Source Digital Library System, which in fact maps MARC into a relational database, and the Avanti project is a project that is building an API for a library circulation system. These two projects are now looking at how they can interact together, so as not to duplicate effort. Both also could use help from interested developers, btw :-).
"What we have here, is a failure to communicate." - Cool Hand Luke
How many sites can we slashdot at once?
But here is a better one - take your idea, but put it on one of those credit card shaped/sized CD-R disks (they are like a 3 inch CD-R, but with straight "sides" - the ends are rounded, though). They only hold a 50-100 meg or so. They were originally designed as a "promo" style item - instead of exchanging business cards, you could exchange a whole ton of data (with business card info printed on one side, of course)...
I support the EFF - do you?
Reason is the Path to God - Anon
The major problem I ran into with writing something of the sort is that there's lots of information that you really want to have that isn't on the web. Cataloging rules, the full description of the MARC fields, some of the lists (organization, I think, is one example). I could get some of those from a library, but strangely enough although I'm sure most libraries have them, they aren't necessarily on the stacks, but in people's offices. Even then, I'd have to keep them checked out for long enough that I'd rather buy a copy.
But, if anyone wants to work on it I'd be glad to help. My ideal app would have to
It seems like the poster wanted something mainly for home use, for a large collection of media. I could see such a system set up for small libraries as well (non-profit, schools, other small libraries). For these uses, all the problems inherent in a large library management system probably go away.
I would love to see such a system - I have a large collection of books myself that I would love to catalog (I also have a ton of other media that I would love to catalog as well). Such a system would be worth looking into, if it existed. I actually started to design one for my home use, but never got further than data layouts and screen layouts (mainly due to the fact that such an application is, to me at least, while practical, not very sexy or exciting to keep me motivated)...
I support the EFF - do you?
Reason is the Path to God - Anon
The Kansas City Public Libraries bought a commercial product from DRA years ago, but skimped on the professional support. They worked with it for years using only their own programmers.
The result was so unusable that the technique used by the Librarians to find a book was:
Now the catalog is so full of bad entries that even after years of "clean up" to work with the new, fully-supported DRA web-capable interface (See
http://www.kclibrary.org) it's still a painful experience to try to find something. Multiple spellings for the same author are unlinked, some books under one, some under another. Some books in a series might be under one call number, while another book is somewhere else. (For instance, the second book of the Gormenghast trilogy is catalogged as the trilogy itself, and it's therefore impossible to retrieve volume 1 or 3.)
This is probably more a result of stingy city budgets than any inherent problem with the job, but it shows how badly it can be done.
Meanwhile, a reliable Open Source system that's not too cryptic to implement could save libraries a lot of (your) money and might even help keep the most cash-strapped public libs from being shut down due to "budget constraints." Something to think about.
Isn't this what the oh so popular web browser lynx was originally developed to do?? I'm fairly sure this is the case.
CmdrChalupa
(who cannot, for the life of him remember how to change his sig)
CmdrChalupa, who finally changed his sig (drop -FlogSpammersNow- for my real address)
Your tax dollars paid for all the cataloging at the Library of Congress.
Unfortunately, some years back the firm that records these records in the MARC formats legally got control not only of their formatted tapes, but of any use of the information used after extraction from these tapes. In other words, they not only own the format, but the government funded information contained in the format.
This is critical because these MARC tapes are the primary source of library cataloging information for most libraries. There are some other independent networks, primarily of educational institutions in the western US, but most libraries depend on the Library of Congress OCLC tapes.
The whole thing stinks, and is ridiculous. As a former librarian, who also holds a BSCS, I was outraged at this theft of public assets. The worst part was dealing with my moronic former colleagues who screamed that of course this company should own this information - it was "intellectual property." Thousands of librarians wrote letters in supporting this company's "intellectual property rights" to work created at tax payer expense.
This happened because most librarians think that putting information into a data format is some mystical arcana mastered only by brilliant wizards. They do not realize that the far more difficult part of the operation was the original cataloging done by the awesome catalogers at the Library of Congress.
So, libraries pay for the nose for software. First, the fee that the vendor has to pay for using the MARC tapes, the royalties for the actual use of the data contained on the tapes, and then for the library software itself. BTW, most library software is so atrocious, buggy, and difficult to use that it's writers would receive a failing grade if it had been turned in as a senior project at any half way reputable college.
Public libraries, unfortunately, are too often dependent on fiercely proprietary-minded vendors for their daily operations.
Incidentally, the "go get MySQL, you dumbass" posters are missing an important point: libraries use the MARC data standard for catalog records, and SQL doesn't cope well with the kind of tricks MARC can do.
Well, people are already working on the XML spec. (More than one, I think.) There is, I believe, a MARC XML spec based on the normal MARC format, but that's not the most user friendly format around. (It's designed to be converted back and forth to MARC.) I also think Dublin Core can be expressed in XML, but I'm not sure.
From what I have seen there aren't a lot of amazing library systems anywhere, OS or proprietary.
Since I don't know where you looked for a system I might be completely off base but this seems like a project a group of CS grads would have picked up somewhere along the line. Check with some universities, somebody has to have replaced those ancient mainframe systems somewhere. If not there are a few very good OS databases (seems like more pop up every day). If you are interested in building a library system they would be an excellent place to begin.
Icebox