Slashdot Mirror
F/OSS Flat-File Database?
Leemeng writes "I'm looking for a simple, free, and F/OSS flat-file database program. I'm storing info about Wi-Fi access points that I come across, maybe 8-9 fields per entry. I've outgrown Notepad. This info is for my own reference only; it is not going on a Web server. Googling was unhelpful, with results skewed towards SQL, Access (MS), and Oracle, all of which would be overkill for my purposes. My criteria are: it must be simple, F/OSS, must work in Windows Vista, preferably use a portable format, must not be an online app, and must not require Java. Does such a beast exist?"
Can't be Java... well, how about Python?
Here is a completely free (PD, not GPL-style "you're free to do as we tell you") database engine that will do what you have described thus far.
The database engine is about 19k bytes (not a typo), has no dependencies (other than Python itself), supports a useful subset of SQL so you can actually create flexible queries that produce well-sorted results from your database, and it works everywhere Python does, which is to say, it works pretty much everywhere. It's just as happy operating on a command line as it is on a web server. The results (the actual databases) are 100% portable from OS to OS. I use it on various linuxes, OS X, and Windows for tasks very similar to yours.
Comes with tutorial examples, sample databases and extensive docs. In a 13k (not a typo) archive.
I've fallen off your lawn, and I can't get up.
I would have recommended HSQL, but you don't want Java. Frankly, usually, when we're talking databases I won't say "use a spreadsheet", but with 10 fields, you might as well use a spreadsheet. Of course OpenOffice.org Base is out, because it uses HSQL.
Something like CSQL might fit, but I have no experience with it.
Ahhh...the great dumpster continuum. Many a free computer will be found there. -- sowth (748135)
Comma Separated Variable Text Files, as exported and imported by Excel. You can get libraries to read and write these, and search these in most languages.
Otherwise what's wrong with a simple database like MySQL or PostgreSQL on your computer?
No comprende? Let me type that a little slower for you...
Sqlite is used in many apps (including Firefox), it's small, and it's efficient. It also has bindings to just about every imaginable language.
I find it amazing that you didn't come across it in Googling...
Get it http://www.sqlite.org/ here.
There are GUI clients that work fine for this sort of thing, SQL is simple for doing basic things. One file, one database.
Obligatory blog plug: http://www.caseybanner.ca/
Have you looked at SQLite?
C based, no client-server. Very small, quite fast. And a very permissive license.
a comma delimited file?
Umm... just write a few one-liner perl scripts to get info out.
findstr, the windows equivalent of grep.
I think this would be perfect, I found it when I was looking for a DB which met similar criteria. I don't care about windows, but it does work under windows AFAIK.
http://www.sqlite.org/
Use it all the time for everthing from trivial databases to several 10s of megabytes. Since query's can be entered on the command line it's quite simple to write an ascii results grabber.
Absolute statements are never true
Since the majority of the comments so far have pointed at SQLite, I'm kind of surprised that the post didn't come "from the sqlite dept." :)
Use perl with a tied hash like BerkeleyDB http://search.cpan.org/~pmqs/BerkeleyDB-0.34/
Easy, fast, simple.
Trying to solve such an issue as a flat file is a poor design characteristic. This says "I don't want to learn SQL" or "I want the output human readable." If you just want to store information for programmatic use, use SQLite and quit worrying about data storage format. If you want it human readable, grab libxml2 (works on Windows too) and store it as XML. Decide which problem you are solving because if you don't need (i.e. it's not helpful) it human-readable and/or want the ability to search it quickly without loading the whole thing into memory, then SQLite is probably a better solution.
Support my political activism on Patreon.
In addition to Calc, there is also Base, the alternative to MS Access. Check it out here: http://www.openoffice.org/product/base.html It's gotta be way easier to work in OpenOffice than a whole database system (making up accounts, tables, reports, and all).
Open office should do the trick.
BDB is *not* a relational database though, it's just a storage/indexing engine, which is used most notably by MySQL as a backend. SQLite on the other hand is a full file-based RDBMS with a small runtime, so it might be a bit of overkill for you in this particular scenario.
But both of them run on Windows, Linux and the BSDs, so you won't have portability problems. And most languages have bindings for them.
The twitter monologues. Click on my homepage and be amazed.
The only thing that comes to mind is Mac only. Bento (which from the write up is exactly what you want). I really think the only thing on Windows that really comes close is Microsoft Access and even that isn't what you're looking for.
If you are just doing tabulated data in a piece meal form, Excel with an Access back-end will do the trick, I'd be willing to be you're going to be able to find templates that help you start. You could also use File-maker (which is overkill, but easy to use) and go to the user community for starting templates.
Burn Hollywood Burn
Can be used by all spreadsheet programs (Excel,Gnumeric,OOCalc probably even Google's online offerings) for complete portability
Depends what you want to be able to do with your data? If it's just quick searches to find local wifi points then it seems overkill to use a multi-platform/FOSS database
---- There are 10 types of people in the world. Those that understand binary and those that don't
If that doesn't satisfy your need, take a look at Berkley DB. It offers a more sophisticated interface than DBM.
A database seems overkill.
Read the XML records into an STL container for easy access.
For a small number of fields like that, why re-invent the wheel? Grab OpenOffice and just create a spreadsheet. Easily searchable, sortable, extendable, and all with zero maintenance on your part.
I recently moved my collection of serial numbers out of a defunct proprietary program and into a spreadsheet - couldn't be happier.
I don't know what kind of crack I was on, but I suspect it was decaf.
Have you used OO Base?
I was using it about a year ago and I found it buggy and hard to use. I was using it to access
PostgreSQL.
I like the other parts of OpenOffice that I use, Write, Calc, Draw.
Of course, I have not had time to look at it since then.
"We can't solve problems by using the same kind of thinking we used when we created them." -- Albert Einstein
Just get Python, and use the version of SQLite that comes with it:
import sqlite3
mydb = sqlite3.connect('sample.db')
mydb.execute("create table contacts (fname text, lname text, email text)")
mydb.execute("insert into contacts values('Spooky','Monster','spook@spammity.spam')")
mydb.commit()
mydb.close()
You can then use the free and open SQLite database browser to browse, edit, and print your table.
You may think you're keeping it simple by using a flat file, but you're really not. It may be somewhat easier to manually edit, but it's also easier to screw up, and I've never heard of one with the ability to undo changes.
- Despite popular opinion, I am not perfect.
Metakit is a small footprint database that might fit your needs. Metakit
echo "badger, badger, badger, badger, snake" >> my_file
Technology -- No Place For Wimps! Grateful Dead and Jerry Garcia Chatroom -- http://www.wemissjerry.org
If it's just for your reference, you don't need a database at all. Databases don't become technically worth-while until you get to indexing. And until you have fifty-thousand records, or complex queries, you don't need indexing.
So why not simply write out a tab-delimited file? Retrieval is as simple as reading every line, splitting by the delimeter, and regexp'ing whichever field you're examining. 15'000 records takes about two seconds on a modern machine. Of course, for anything more complicated, that same file gets easily imported into your favourite spreadsheet application.
You don't need to worry about locking because you're the only one using it. And otherwise, your application simply locks the file handle, or creates & destroys a traditional lock-file.
It's a thirty-line perl script (of legible perl). You can do it in JScript as a local HTA if you want the benefit of html etc. interfaces.
There are a number of XML databases, several free and open source, that will rely only on "flat-files". You could probably get by with Microsoft's xml libraries, though there are a number of ways to manipulate and query a set of xml documents. Several of these XML databases implement XQuery which may help if your dataset grows beyond effective queries by visual inspection in Notepad.
eXist is one alternative; while I haven't personally used it the home page indicates it's a fairly capable project.
Sedna also appears to be feature-rich.
There was a similar discussion on Slashdot specifically with reference to XML databases, here.
Happy hunting -
If the Government becomes a lawbreaker, it breeds contempt for law;
Even better, look at Tiddly Wiki. No engine required - the entire wiki is run in javascript. Just one .html file to carry around with you.
Advice: on VPS providers
As the original poster had been utilizing Notepad previously, vi/vim should both be considered significant upgrades.
It would help if the OP had posted the current number of records and projected growth. Size of the database and how it will be utilized goes a long way to determining a 'best' solution.
Class dbtxt includes the ability to undo changes, and features human-readbility for the flat database files. It's trivial and not error-prone at all to mod a database file to delete, or undelete a record, and of course you can do it through the database engine as well.
I've fallen off your lawn, and I can't get up.
Why mess about with filty batchfiles, bashscripts and clunky sqlite, mysql. How about a full Oracle Database 11g Enterprise? o;)
1993 called to recommend PCFILE!
http://www.umich.edu/~archive/msdos/database/pcfile/
Mind you, you'll have to toss Jim Button $10 as it's shareware!
AT&ROFLMAO
I'm guessing he wants a FOSS version of FileMaker Pro; something that is an application with a GUI. All you guys are suggesting various frameworks he can use to program his own. No, I don't have an answer either, other than to suggest the spreadsheet in OpenOffice.
set first "Spooky"
set last "Monster"
set email "spook@spammity.spam"
# In practice it's really easy to put values into variables
mydb eval {insert into contacts values (:first,:last,:email)}
The advantage? That code is now totally armour-plated against SQL injection attacks as well as being fast. Which is nice, really really nice.
"Little does he know, but there is no 'I' in 'Idiot'!"
Nobody has yet mentioned that MySQL has a CSV storage engine - just create a table with type CSV and away you go. It does require the MySQL engine though.
Otherwise, I agree with most posters who say just use a simple text and/or xml file if the data volume is relatively small. That should be more than enough!
SixD
The poster didn't want SQL, so I'm guessing they ruled out SQLite on those grounds. QDBM and Oracle/Sleepycat/Berkley DB-4 would be the smallest, fastest "pure" flat-file databases that are also cross-platform - at least, that I know of.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Kexi should be able to hande that pretty easily, and can even create a nice form for the data entry:
http://www.kexi-project.org/
Erik Dalén
I'm still using it for a customer database, it's two main problems are it's buggy and hard to use, and slow.
...
So really it's 3 main problems are being buggy, hard to use, slow* and totally bloated
Actually it's not half as buggy as it was a few releases back - I can actually use it now to fill in a form without it crashing. Just tried opening a windows XP made odb file in Ubuntu (OOo 2.4) and it's slow as hell and bugged to death, so you might add portability to that list of main problems.
[Yes I've filed bug reports].
---
* slow for searching, takes about 2 mins to do a straight text search on 500 records.
He doesn't need python. He just needs a database. He can download a precompiled binary for windows that allows one to work with the database at the command line. Python is not necessary.
And if the command line is too much, as others have noted there is already a convenient firefox extension for graphically interacting with a sqlite database.
Your problem is that you are looking for a "database". This is a very complex kind of program that is total overkill for your purpose. Will you really be storing hundreds of thousands of records and wanting to do some complex querries? Ifnot why not just use a speadsheet? If you want a free spreadsheet look at OpenOffice.
Why one earth would you use a database to store one table on information? Save youself some trouble and use a spreadsheet.
Why not the pickle/cPickle module? Not a flat-file but easier to manipulate since your just serializing native Python objects.
e.g.
import cPickle
class Record:
def __init__(self, fname, lname, email):
self.fname = fname
self.lname = lname
self.email = email
# ---
# Write out the data
records = list()
records.append(Record("John", "Doe", "jd@email.com"))
output = open("records.db", "wb")
cPickle.dump(records, output)
output.close()
# ---
# Read back the data
inputFile = open("records.db", "rb")
data = cPickle.load(inputFile)
for record in data:
print record.fname, record.lname, record.email
inputFile.close()
Berkely DB?
Firebird (http://www.firebirdsql.org/) works with many programming languages so it is not tied to java, works fine with windows vista and operates without an installed server in its "embedded server" mode.
...because he wants to hack his own system...
The sqlite3 module as shipped with Leopard says that it supports question mark substitution, so his example would be better written like:
BTW, the same could be written for PostgreSQL like:
Hopefully sqlite3 will get the same standard substitution in the future, but at least it is possible to do it safely today.
Dewey, what part of this looks like authorities should be involved?
Been searching flat files with this open source VB5 program since 1999. Searches flat files at 20,000,000 cps displaying hi-lited hits in matching lines only or full context mode. Results can be exported to a results file. Append txt files together. Search and replace option. Run multiple versions with various default settings. I have been yapping and yapping about flat files, only to get great amounts of resistance from the Geeks. The search uses lots of defaults and quick keys (for typists). From startup to shutdown nothing is faster. Channel9 and thedailyWTF.com have the best threads. Check it out. It's the only program you'll ever need
Challenge: I have better access to my Video, Music, Pics and Text than anyone on Earth.
1) My example was done to show how quick and easy it would be to set up and populate a table in Python.
2) He's explicitly mentioned several times that this is for his own private non-web use.
3) Your fly is open.
- Despite popular opinion, I am not perfect.
Actually, the Python SQLite adapter supports the standard Python DB API (PEP 249), and so you can just follow its recommendations (which includes using placeholders).
I know that feeling.
TextDB: If you don't have access to a database but you do have access to PHP, and you want your web data stored in a database, then this is what you're looking for! Full sql compatibility is planned as well as a perl port. This is OS-independent and is licensed under GNU GPL, has a web-based interface and is developed using PHP. (check out: http://sourceforge.net/projects/textdb). Berkeley DB (BDB): This is a computer software library that provides a high-performance embedded database, with bindings in C, C++, Java, Perl, Python, Ruby, Tcl, Smalltalk, and many other programming languages. Berkeley DB is redistributed under the Sleepycat Public License, an OSI and FSF approved license and runs on a variety of operating systems including most Unix-like and Windows systems, and real-time operating systems. It comes in three different editions: Berkeley DB(originally written in C), Berkeley DB Java Edition, Berkeley XML DB (check out: http://en.wikipedia.org/wiki/Berkeley_DB)
Take a look at Metakit. http://www.equi4.com/metakit/index.html. Its single file, small, fast and it has proven itself over many years. It is written in C++ and bindings for Python and Tcl are available. Instead of tables, rows and fields it uses similar concepts called view, index and property. Interestingly, a property may also be a subview thus allowing a mix of a relational database (flexible) and a hierarchical database (fast).
Do you always ignore requirements when designing a solution, or are you making a special exception in this case?
If you were blocking sigs, you wouldn't have to read this.
Yes, one can create work-arounds with GPL'd library code, like compiling the library separately and building a socket or messaging interface to it (which interface of course has to be GPL'd as well), but this can be a PITA as you point out. And, as I'm sure you'd agree, workarounds like that are not always possible for performance and other reasons.
Yes, authors who use the GPL have decided that we can use their code in a certain way, and that's their privilege. However, it can be a dumb choice, because as the grandparent poster points out, those of us who do rely on proprietary licensing models for our revenue (and that would be the majority of us) are often unable to give anything back to the community as a result. I'd love to be able to use some GPL'd libraries in our stuff, and I'd love to be able to piss in those libraries to contribute bug fixes and help out in general. But the GPL forces me in another direction, usually to reinvent the wheel or to use something that's closed source and potentially less capable.
If you author a library, think hard about the licensing model. The GPL may not be the best choice -- unless, of course, you have a religious belief that all software should be GPL, which is your decision to make. I don't get into religious arguments, so that would be the end of the discussion.
I use hsql. It has many options: flat file, memory just to name 2.
I have also heard good things about H2 and apache derby. These have been especially valuable for embedded apps
I like notetab a lot, wish he'd port it over to Linux...
waiting for ad.doubleclick.net