Slashdot Mirror

SQLite is controlled by a corporation, and that corporation requires adherence to a religious code in order to contribute patches to what is (in theory) an open source project.

Open source, absolutely, open contribution, absolutely not:

But SQLite is not open-contribution. In order to keep SQLite in the public domain and ensure that the code does not become contaminated with proprietary or licensed content, the project does not accept patches from unknown persons. All of the code in SQLite is original, having been written specifically for use by SQLite. No code has been copied from unknown sources on the internet.

But even if they were open contribution, they make it very clear they only insist your stick to the spirit of the specifics of the code of conduct to be a part of the community. Maybe go to the effort of reading its overview? See also the background story on its adaptation.

Forking is a poor alternative because they've got extensive test suites that are not open.

So no, I don't care what the maintainers believe in their own little heads, but if they require contributors to hold those same beliefs, it's going to cause problems.

"In their own little heads" shows you don't have the slightest bit of grace required to conform to the spirit of this 1,500 years tested "Code of Conduct", and but it will only cause problems in your mind, they most certainly can do without your non-existent to date contributions to the community. In that, it's functioning as useful filter to keep griefers like yourself away from the project.

I'd also predict this will put a damper on their revenue.

As the KJV puts in in Mark 8:36:

For what shall it profit a man, if he shall gain the whole world, and lose his own soul?

There's no more widely used piece of FOSS in existence, I'm sure there quite willing to take the very small hit they're going to get from intolerant bigots who will stop paying them money because they're Christian. Which would have happened anyway sooner or later, whether or not they'd adopted this code.

It's all around a bad idea for a for-profit corporation to start proselytizing,

And why is this?

if I was their customer (or my company was) I'd certainly drop the contract at the next opportunity.

Empty words from someone who was never going to be one of their customers.

SQLite is controlled by a corporation, and that corporation requires adherence to a religious code in order to contribute patches to what is (in theory) an open source project.

Open source, absolutely, open contribution, absolutely not:

But SQLite is not open-contribution. In order to keep SQLite in the public domain and ensure that the code does not become contaminated with proprietary or licensed content, the project does not accept patches from unknown persons. All of the code in SQLite is original, having been written specifically for use by SQLite. No code has been copied from unknown sources on the internet.

But even if they were open contribution, they make it very clear they only insist your stick to the spirit of the specifics of the code of conduct to be a part of the community. Maybe go to the effort of reading its overview? See also the background story on its adaptation.

Forking is a poor alternative because they've got extensive test suites that are not open.

So no, I don't care what the maintainers believe in their own little heads, but if they require contributors to hold those same beliefs, it's going to cause problems.

"In their own little heads" shows you don't have the slightest bit of grace required to conform to the spirit of this 1,500 years tested "Code of Conduct", and but it will only cause problems in your mind, they most certainly can do without your non-existent to date contributions to the community. In that, it's functioning as useful filter to keep griefers like yourself away from the project.

I'd also predict this will put a damper on their revenue.

As the KJV puts in in Mark 8:36:

For what shall it profit a man, if he shall gain the whole world, and lose his own soul?

There's no more widely used piece of FOSS in existence, I'm sure there quite willing to take the very small hit they're going to get from intolerant bigots who will stop paying them money because they're Christian. Which would have happened anyway sooner or later, whether or not they'd adopted this code.

It's all around a bad idea for a for-profit corporation to start proselytizing,

And why is this?

if I was their customer (or my company was) I'd certainly drop the contract at the next opportunity.

Empty words from someone who was never going to be one of their customers.

Well.... SQLite Is Public Domain, we fool! :) The code of conduct is also basically completely useless because of what is stated in https://www.sqlite.org/copyrig... : "Open-Source, not Open-Contribution SQLite is open-source, meaning that you can make as many copies of it as you want and do whatever you want with those copies, without limitation. But SQLite is not open-contribution. In order to keep SQLite in the public domain and ensure that the code does not become contaminated with proprietary or licensed content, the project does not accept patches from unknown persons. "

The text is on SQLite's website: Code of Conduct.

It's actually a pretty decent code of conduct; if you ask only that developers govern their interactions with others according to the code.

I am guessing the objections are that the code contains religions admonitions such as:

Rule 1. First of all, love the Lord God with your whole heart, your whole soul, and your whole strength.

A majority of developers will likely be of some religion that can identify with that statement; However, the statement could disturb
atheists who might participate in the project, when the CoC references particular individual practices they disagree with.

On the other hand it's also true that the code doesn't mention any consequences for failing to follow specific rules on individual conduct.
it particularly says: .... the SQLite developers elected to govern their interactions with each other, with their clients, and with the larger SQLite user community in accordance with the "instruments of good works" .... This rule is strict, and none are able to comply perfectly. Grace is readily granted for minor transgressions. All are encouraged to follow this rule closely, as in so doing they may expect to live happier, healthier, and more productive lives. The entire rule is good and wholesome, and yet we make no enforcement of the more introspective aspects. ....

The link to the CoC is in the summary, but here it is again: SQLite Code of Conduct.

It is OBVIOUSLY a joke. I don't find it particularly funny, but I don't see any harm either. It is clearly ridiculing some of the over-the-top CoCs, and in many cases that ridicule is well deserved.

The people taking this seriously need to eat more fish or, if they are vegan, some omega-3 supplements, to help their brains work better.

Here's the copy from SQLite: https://www.sqlite.org/codeofconduct.html

Why are you using a spreadsheet when you have that much data?

Because we don't have a license for an SQL server or an IT department prepared to support a free one. Duh.

So you can afford a license for MS Excel but not pay nothing for SQLite or PostgreSQL?

Oh, I "forgot" you said you had no IT department to help you with the free one... But you have an IT department helping you with all your Excel problems?

Or you don't? How the **** are you handling all the Excel problems then? If you don't have any, great, your workplace seems to be a place full of Excel wizards. But if they are, they should be able to learn how to use a SQL-engine/server without much trouble.

Learn to use the right tool for the job instead of using shitty tools. After the initial period of learning you will wonder why you were such complete and utter idiots for all those years, when you could have done real work instead in a fraction of the time and with much less headache from trying to debug poorly designed spreadsheets.

SQL doesn't mean having to store it on a server somewhere.

Yes, it does. If the SQL server software happens to be running on your desktop then it's a server, especially if you expect other people to access the data.

No, SQLite is "server-less", so there are options if you don't want "servers".

But servers aren't something evil, why are you so obsessed with not having servers? You are right in that any computer serving something to clients are technically servers. But in reality when you talk about servers you mostly mean dedicated servers, servers which is not used as a workstation for someone else.

My computer at my old work was always turned on, so for a while I ran FileMaker to share data with my coworkers, and later my PostgreSQL server which replaced FileMaker on it. When everyone realized it was really helpful to share date we got it a new home on a "real server". But those can be pretty much anything with a little computing power. Heck, you could run PostgreSQL on a Raspberry Pi if you wanted something that is out of the way, doesn't draw much power and doesn't generates noise. I wouldn't recommend it, but there are all kinds of solutions, and I bet you have some computers unused somewhere that could be used.

If you have machines capable of running Excel, you can sure as heck let them run some kind of SQL-server.

Why are you using a spreadsheet when you have that much data?

Because we don't have a license for an SQL server or an IT department prepared to support a free one. Duh.

So you can afford a license for MS Excel but not pay nothing for SQLite or PostgreSQL?

Oh, I "forgot" you said you had no IT department to help you with the free one... But you have an IT department helping you with all your Excel problems?

Or you don't? How the **** are you handling all the Excel problems then? If you don't have any, great, your workplace seems to be a place full of Excel wizards. But if they are, they should be able to learn how to use a SQL-engine/server without much trouble.

Learn to use the right tool for the job instead of using shitty tools. After the initial period of learning you will wonder why you were such complete and utter idiots for all those years, when you could have done real work instead in a fraction of the time and with much less headache from trying to debug poorly designed spreadsheets.

SQL doesn't mean having to store it on a server somewhere.

Yes, it does. If the SQL server software happens to be running on your desktop then it's a server, especially if you expect other people to access the data.

No, SQLite is "server-less", so there are options if you don't want "servers".

But servers aren't something evil, why are you so obsessed with not having servers? You are right in that any computer serving something to clients are technically servers. But in reality when you talk about servers you mostly mean dedicated servers, servers which is not used as a workstation for someone else.

My computer at my old work was always turned on, so for a while I ran FileMaker to share data with my coworkers, and later my PostgreSQL server which replaced FileMaker on it. When everyone realized it was really helpful to share date we got it a new home on a "real server". But those can be pretty much anything with a little computing power. Heck, you could run PostgreSQL on a Raspberry Pi if you wanted something that is out of the way, doesn't draw much power and doesn't generates noise. I wouldn't recommend it, but there are all kinds of solutions, and I bet you have some computers unused somewhere that could be used.

If you have machines capable of running Excel, you can sure as heck let them run some kind of SQL-server.

That's why I specifically mentioned the "goto fail" issue. That tiny bug completely broke SSL/TLS. How could they not be testing basic functionality like that before it's released?

I'll grant that this particular situation might not have been tested, although to me, testing with root and a blank password seems fairly obvious. But this seems like a more widespread problem for Apple and how they test (or don't test) basic functionality. And I'm not talking about using human testers. This should be 100% automated. And that means it happens for every official build for the rest of time.

Check out how in-depth SQLite's testing procedures are, for example: https://www.sqlite.org/testing... This sort of comprehensive testing doesn't eliminate all bugs, of course, but it's highly unlikely that any obvious bugs are going to slip by this test suite. And whenever a bug is found, it's not considered fixed until a regression test is added to the suite that will catch any future incidents of that sort of bug. This is how you build long-term stability into your software.

HTML is already a "safe subset" of itself; HTML is merely a markup language. You don't need to implement the DOM or Javascript, especially in an email client.

The problem is, as it has always been, that there are bugs in software that renders HTML.

There have been exploitable bugs in JPEG decoders, PNG decoders, TIFF decoders, GIF decoders, even BMP decoders, BMP being the most braindead simple of image formats: would you say that we need a "safe subset" of image formats?

There have been exploitable bugs text editors. Would you say that we need a "safe subset" of plain text?

It's possible to have bugs in any language, for software decoding any format.

Features don't make for security holes; bad implementations make for security holes, as demonstrated there are bugs in even the simplest, most featureless formats. What you need is better testing.

Sure, every feature multiplies the number of tests, but it's possible to write those tests, if you insist on it, and it's also possible to blow off testing entirely, even if what you've written is simple and featureless, and still have exploitable security holes.

Features, complexity, are not the root cause of security flaws, failing to bring a commensurate amount of testing and rigor is the root cause. Going back to the good old days of less featureful software is not going to fix a damn thing.

Extended family members converged from around the southeast to the parking lot of the First Baptist Church of Cross Hill, SC, which is about 100 yards from the center line of totality. We brought a picnic lunch including eclipse-themed items such as Sunkist cola and Moon-Pies. We choose Cross Hill since it is not near any major cities or highways, thereby avoiding crowds and gridlock.

I brought a telescope and a white sheet to spread on the ground in order to see the shadow bands. The weather was partly cloudy, but the sun was clear of clouds for totality.

Photo of totality: https://sqlite.org/tmp/total-e...

I had previously been at the center line of the annular eclipse of 1984 as it traversed the campus of Georgia Tech. A total eclipse is much better. To be able to look up and see what appears to be a hole in the sky is something you do not want to miss. If you have never witnessed a total solar eclipse before, I encourage you to add this to your "bucket list".

Fuzzing is essentially harnessing the power of our modern computational power in a brute force fashion, combined with the knowledge that many errors (especially crashes), by nature, can be leveraged into an exploit.

In my own scripting language project, I have two fuzz tests I perform - I first fuzz a set of source scripts, and in another test, I fuzz a set of compiled bytecode, which exercises both the lexer/parser and runtime interpreter phases. I didn't even bother with a library either, just a small routine that randomly swaps and corrupts source from the original. It's really amazing how simple something like that will catch so many bugs.

Honestly, I was implementing this just for the sake of robustness. No one but me is using the library yet, and it's just for local use in my game engine. But if you're connected to the internet in any way, there's really no excuse these days for not having a set of fuzzing tests you regularly run during your normal regression testing, and there are some great libraries available to help do this. You can even leverage Google's massive computational resources for testing for free (perhaps even get paid a small bounty) if your project is important enough, which OpenVPN certainly is. Hopefully the OpenVPM devs/maintainers take note of this and make this happen, and we'll all be more secure for it.

BTW, if you ever want to read about an incredibly comprehensive test and regression suite, check out SQLite's description of their testing methodology: https://www.sqlite.org/testing...

The SQLite developers were also surprised by how many bugs OSS-Fuzz (and American Fuzzy Lop) have found in SQLite.

The best explanation I have is that OSS-Fuzz and AFL are exploring extreme corner-cases of the code where human-generated tests would never think to go. Fuzzing is great for finding bugs that involve totally unreasonable inputs that never happen in actual practice and which can only appear as part of a deliberate attack. Fuzzing has not found any bugs that would impact the day-to-day use of SQLite.

In other words, fuzzing finds an entirely different class of bugs from what the mountains of other test cases for SQLite are designed to find. This is a good thing. We encourage testing diversity.

Here is a list of issues found in SQLite by OSS-Fuzz (and now fixed): https://www.sqlite.org/src/sea...

There are a few cases of NULL pointer dereferences or other crashes that come about while unwinding the stack following an Out-Of-Memory error. Those kinds of errors are real, and we are grateful to OSS-Fuzz for finding them, even if they are seldom seen in the wild. Other issues were assertion faults that probably would not have resulting in a crash if assert() has been disabled (which is the case for all default builds of SQLite). And then there are things like https://www.sqlite.org/src/tim... which is not really a bug at all - OSS-Fuzz was submitting a funky recursive VIEW query that after unwinding all the nested views resulted in a very larger prepared statement, which took too long to process and so OSS-Fuzz timed out. SQLite was getting the correct answer, it was just taking too long. Since the submitted SQL was of no practical use, we "fixed" that problem by limiting the size of prepared statements to be about 100 times larger than any real SQL statement needs to be, rather than the default limit of about a 10 million times larger.

The SQLite developers were also surprised by how many bugs OSS-Fuzz (and American Fuzzy Lop) have found in SQLite.

The best explanation I have is that OSS-Fuzz and AFL are exploring extreme corner-cases of the code where human-generated tests would never think to go. Fuzzing is great for finding bugs that involve totally unreasonable inputs that never happen in actual practice and which can only appear as part of a deliberate attack. Fuzzing has not found any bugs that would impact the day-to-day use of SQLite.

In other words, fuzzing finds an entirely different class of bugs from what the mountains of other test cases for SQLite are designed to find. This is a good thing. We encourage testing diversity.

Here is a list of issues found in SQLite by OSS-Fuzz (and now fixed): https://www.sqlite.org/src/sea...

There are a few cases of NULL pointer dereferences or other crashes that come about while unwinding the stack following an Out-Of-Memory error. Those kinds of errors are real, and we are grateful to OSS-Fuzz for finding them, even if they are seldom seen in the wild. Other issues were assertion faults that probably would not have resulting in a crash if assert() has been disabled (which is the case for all default builds of SQLite). And then there are things like https://www.sqlite.org/src/tim... which is not really a bug at all - OSS-Fuzz was submitting a funky recursive VIEW query that after unwinding all the nested views resulted in a very larger prepared statement, which took too long to process and so OSS-Fuzz timed out. SQLite was getting the correct answer, it was just taking too long. Since the submitted SQL was of no practical use, we "fixed" that problem by limiting the size of prepared statements to be about 100 times larger than any real SQL statement needs to be, rather than the default limit of about a 10 million times larger.

The SQLite developers were also surprised by how many bugs OSS-Fuzz (and American Fuzzy Lop) have found in SQLite.

The best explanation I have is that OSS-Fuzz and AFL are exploring extreme corner-cases of the code where human-generated tests would never think to go. Fuzzing is great for finding bugs that involve totally unreasonable inputs that never happen in actual practice and which can only appear as part of a deliberate attack. Fuzzing has not found any bugs that would impact the day-to-day use of SQLite.

In other words, fuzzing finds an entirely different class of bugs from what the mountains of other test cases for SQLite are designed to find. This is a good thing. We encourage testing diversity.

Here is a list of issues found in SQLite by OSS-Fuzz (and now fixed): https://www.sqlite.org/src/sea...

There are a few cases of NULL pointer dereferences or other crashes that come about while unwinding the stack following an Out-Of-Memory error. Those kinds of errors are real, and we are grateful to OSS-Fuzz for finding them, even if they are seldom seen in the wild. Other issues were assertion faults that probably would not have resulting in a crash if assert() has been disabled (which is the case for all default builds of SQLite). And then there are things like https://www.sqlite.org/src/tim... which is not really a bug at all - OSS-Fuzz was submitting a funky recursive VIEW query that after unwinding all the nested views resulted in a very larger prepared statement, which took too long to process and so OSS-Fuzz timed out. SQLite was getting the correct answer, it was just taking too long. Since the submitted SQL was of no practical use, we "fixed" that problem by limiting the size of prepared statements to be about 100 times larger than any real SQL statement needs to be, rather than the default limit of about a 10 million times larger.

> Fossil was meant to be a 'lite' DVCS

Fossil was meant to support the needs of SQLite, one of the most popular and actively-developed code bases in the world. If Fossil can meet its needs, chances are good that it can meet your project's needs, too. There are very few NetBSD-scale projects out there, compared to the number that are plenty fast under Fossil.

And yes, I'm aware that you could list hundreds of projects at that scale, but I believe I could find millions software projects smaller than that. If you try to argue against this point, you'd be arguing a 0.01% type of position. I'm happy with Fossil going after the other 99.99%.

> Hell, it uses SQLite as its storage backend.

This is a problem how?

The biggest single problem with SQLite from a performance standpoint is that it doesn't have row-level locking, limiting its use in concurrent systems. SQLite has multiple-reader, single-writer concurrency, but multiple writers to a single DB file serialize their writes to a single table. SQLite will let multiple writers can modify separate tables at the same time, but I'm going to assume Fossil has some usage pattern that requires that all commits to hit at least one common table.

According to NetBSD's source-changes mailing list there are only about forty commits per day, so even at that scale, concurrency is simply not an issue.

Projects like NetBSD don't get to be behemoths over night. It takes 40 commits per day for 25 years to make that happen.

SQLite's other big limitation is that it isn't a client-server DBMS, so when you need multiple clients over the network to access the DB, you need some intermediary to provide that access. The naive approach is to use a networked file system, but this is likely to cause problems. But we don't have that problem with Fossil: we have fossil server, which exposes the DB over HTTP. Since a single process is manipulating the central DB, which is a local file to it, there can be no locking problem. If by some small chance two users need to commit at the same time, fossil server will serialize the commits.

> It's a lovely DVCS otherwise.

I agree. The vast majority of the users of Git today could run just as well on Fossil.

Nobody has mentioned the temp fix yet so

On OS X, Open /Applications/Spotify.app/Contents/MacOS/Spotify in a hex editor.
Search for "VACUUM;" Replace with "xxxxxx;"

Once you apply that fix you can manually vacuum with

sqlite3 mercury.db vacuum;

"~/Library/Application Support/Spotify/PersistentCache/mercury.db"
download sqlite3 from https://sqlite.org/download.ht...

https://sqlite.org/random-pass... shows example output with a link to the source code.

I've used a variety of hosting providers, but I always keep coming back to Linode. Their product is competitively priced, they provide exceptional service and support, and they are very simple to use. And, unlike AWS, you don't need a calculator and 2 hours spent parsing fine print in the documentation to figure out how much a given level of service will end up costing you. I highly recommend Linode for your cloud computing needs. I hope they are able to resolve their DDoS problems quickly.

What are alternatives to all the relational databases that use SQL and provide the whole ACID package?

The true answer to your question is on wikipedia.

However, since your question isn't quite relevant to Oracle, I'll join everyone else in answering the slightly different question "how can my data be reasonably safe even if I don't use Oracle". For the majority of applications where SQL is used ACID isn't honestly needed for safe data. Just keep adding it to a DBM and back it up regularly. Even where it is the promises of SQLite will be fine. Even better, if you every magically got to the stage where you needed Oracle later in your project and for some magic reason Postgres isn't a better fit for your needs (being simpler and normally perfectly performant) then the great thing about SQLite is that, if you can just read a bit, it uses the same SQL language so you can just change over to Oracle later.

N.B. I did not say "for the majority of data". I said for "the majority of applications". 90% of Oracle applications seem to hold a small table of users and a tiny bit of data that could just as well be listed in a file and read into memory. The corporate standard says "Oracle" so in it goes.

Turning on all warnings and forcing them to errors certainly would have caught the bug in Apple's SSL code. Anyone who just lets warnings fly by in C code is an idiot. Even if the warning is mildly silly, getting it out of the way lets the important warnings stand out. Sensible warnings from C compilers are the very reason we don't use lint anymore. Even then you still have to watch out, because some warnings won't appear at low optimization levels, and I recall hearing that there are a few obscure warnings not turned on by -Wall.

Let me quote from one of the best-tested and most widely used projects out there, SQLite, from http://www.sqlite.org/testing....

Static analysis has not proven to be especially helpful in finding bugs in SQLite. Static analysis has found a few bugs in SQLite, but those are the exceptions. More bugs have been introduced into SQLite while trying to get it to compile without warnings than have been found by static analysis.

The bolded part has been my experience unfortunately. Static analysis is nearly useless.

An appropriate test for something like an SSL stack is a separate test harness that "fuzzes" the stack by exploring large random combinations of values, some with known good certificates and others with randomly generated (and thus broken) ones. These days one can spin up thousands of VMs, run a massive suite of billions of test cases in parallel over a few hours, then spin them down and spend a relatively small sum of money.

And yes, the test harness for something like this is probably going to exceed the # of lines of code of the actual implementation by an order of magnitude. For really important security-critical stuff like cryptography, SSL/TLS, keychain management, etc it is well worth the effort.

"Think of SQLite not as a replacement for Oracle but as a replacement for fopen()" --- About

the more risk-averse would be very, very, jumpy about taking 'anonymous coward' at his word that they are authorized to use a given piece of code under the terms of whatever license, that he is even the author, and so forth. That might hinder adoption.

That's why SQLite and other programs have so many problems with adopting a "public domain"
license: For all intents and purposes the public domain can not contain new works. Even if I say my code is in the public domain, I can change my mind at any later point and sue you. MIT, GPL, etc. is needed because you must expressly permit use to assure the users they won't be sued.

From SQLite copyright:

Even though SQLite is in the public domain and does not require a license, some users want to obtain a license anyway. Some reasons for obtaining a license include:

    You are using SQLite in a jurisdiction that does not recognize the public domain.
    You are using SQLite in a jurisdiction that does not recognize the right of an author to dedicate their work to the public domain.
    You want to hold a tangible legal document as evidence that you have the legal right to use and distribute SQLite.
    Your legal department tells you that you have to purchase a license.

If you feel like you really have to purchase a license for SQLite, Hwaci, the company that employs the architect and principal developers of SQLite, will sell you one.

Yay! Open source you need to pay a license fee for to cover your ass! Public Domain? No thanks. Note that to contribute code you have to jump through a bunch of legal hoops too. It's just dumb.

GP:

Is it legally possible to author and licence an opensource project without disclosing your identity?

Talk about fixing the wrong fucking problem. Gah. Source code by itself is language. Punishing folks for language is evil. I agree we need a way to drop rights to works, and to anonymously publish -- You can publish code on a paste-bin fairly anonymously through TOR. Promoting and Maintaining it is another ball of wax.

mysql is of historical curiosity. At best.

I'd be willing to bet there are more deployments of MySQL than of all other standalone RDBMSs combined.

Which you will loose, as that spot is taken by SQLite.

Go install some SQL database on your desktop machine and play with it. MySQL, MariaDB, and Postgres are all free and will work on Linux or Windows desktops.

Or, experiment with SQLite. You can download a self-contained standalone precompiled binary that you run as an ordinary command-line program. (ex: "sqlite3 mynewdatabase.db") In fact, sqlite3 is already installed by default on your Mac and probably also on your Linux desktop, so you might not need to install anything at all. There are no servers to set up and maintain and no access permissions and user accounts and passwords to configure. And the database you create is just an ordinary disk file that can delete once you finish experimenting.

All of the databases on your Android and iPhone are SQLite databases, so if you want to look at some real-world data, just upload them and look at them using the sqlite3 command-line tool. You might find other SQLite databases to look at already on your workstation from programs like Firefox, Skype, iTunes, Dropbox, etc.

MySQL, MariaDB, and PosgreSQL are all fine products. But if all you want to do is experiment with the SQL language, they are way, way more complication than you need.

For those interested, SQLite also has a 2nd generation statistical optimizer for query plans in the works: http://www.sqlite.org/draft/queryplanner-ng.html And it already has index-only scans too.

Are you looking to contribute in the actual development of the database software or just looking at what database to use? I see lots of comments seem to imply the latter and focus on what is wrong with database x from a user viewpoint (a DBA is a user, an application developer is a user) - if you want to make your hands dirty you should really look into the community and how to get involved. MySQL has not been that great on Oracle days on external community so you might be better off looking at some of the numerous forks and how they treat their developers.

If you are looking for a way to get employed through knowing the insides of a database engine in the Open Source world I would suggest SQLite (it is really used everywhere nowadays) or as a little bit more niche product, Firebird - Firebird is a really nice database which can be used on embedded projects easily + has robust set of features and has a permissive license (I am a bit biased because the company I work for uses it as a core database engine...).

http://www.sqlite.org/apple-and-sqlite.gif ;-)

Perhaps you should be looking at Sqlite, which is a "a self-contained, serverless, zero-configuration, transactional SQL database engine" (as it says on their webpage).
 
You can run it interactively (or through a bash script or something) with the sqlite3 command line shell, or (most efficiently) hook it into your own programs and use it to do all kinds of clever SQL stuff directly within your program.
 
Oh yeah, it's also explicitly public domain, so you can use it for any purpose and in any application whatsoever.

Some of the tests are included with the code, but others are proprietary and only available under license. It bugged me and I've stayed with Berkeley DB. Info here (see "TH3 test harness").

I'm a developer at an ISV. Personally, I am waiting for XP to go. Microsoft has some great technology (WWSAPI, SQL Server 2012 LocalDB) that looks like it will solve some of the problems we need to solve with our application, but it's not available on XP.

I'm really intrigued by why you couldn't use SQLite3 instead of SQL Server 2012 LocalDB or any API other than WWSAPI for web services.

GPL isn't copyright

If it was not a copyright licence then it would have no force under the law and would be fucking pointless. Yes the license has generous terms for the user but it's still a copyright license that has the full protection of copyright law. As a proffesional developer I cannot legally use GPL software unless I comply with it's terms. The only difference between a GPL license and the more traditional ones is that the GPL terms are entirely focused on the "improving society" rationale for granting a monopoly.

Nothing is restricted under the GPL

Incorrect. The user has to comply with redistribution restrictions because the aim of the license is to benifit scoiety as a whole rather than compensating the creator.

Copyright as a concept is neither inherently good or evil, it was originally a rather nobel intention to support the arts and sciences for everyone's benifit that has over the centuries devolved into a legal minefield that implements property rights for trivial "creative works" such as merging 2 button clicks on an order form into one button click. Most OSS licenses reflect the noble intentions of copyright law, the MAFIAA licenses ignore all that "social responsibilty" babble in favour of personal gain that is far beyond the social value of their distribution networks.

Having said that, anyone who is genuinely committed to the idea that social benifit is more important than personal control over their creations would not bother creating a license like the GPL, they would release their work into the public domain and be done with it (eg: the world's most popular database software).

So now we have gone from "unit tests have no value and are just double the work LOL" to "unit tests don't solve every problem".

I agree: unit tests don't solve every problem. I said I'm happy every time an assert catches a bug; I didn't say I never use the debugger anymore.

things most often break in the integration of all those pieces...not just isolated individual pieces you wrote the unit tests for.

Don't you have unit tests that exercise the whole system? I do.

I work with audio, and I have unit tests that run reference files through the system and then verify that the output is close to what was expected. This is an end-to-end unit test, which will catch bugs in the I/O layer, bugs in the processing code, etc. etc.

Of course you'll never get 100% coverage. (Although SQLite claims 100% branch test coverage!)

steveha

I've been running http://www.sqlite.org/ on Linode since 2004. They've been great. Highly recommended.

I looked at this problem once, in the context of distributed file systems. I'd divide files into the following categories, with slightly different semantics for each. This can be fitted into the standard UNIX/DOS/Windows model, but it resolves some issues that result in programs doing elaborate workarounds to get correct file semantics.

• 
  Unit files A unit file is only meaningful once it has been completely written and closed. Once closed, it will never be rewritten, only replaced. Most files are unit files. The file system should guarantee that 1) unit file replacement by a new version is an atomic operation, 2) unit files not properly closed do not replace old versions, so that when a program fails while writing a unit file, the old version remains undamaged, and 3) a unit file being written is not visible to other processes until closed.. This was a feature of some mainframe operating systems. Because the UNIX file system concept doesn't have this, there's much fussing around with ".part" files and renaming strategies to achieve unit file semantics.
• 
  Managed files Managed files are written and read in a random access fashion, but only by programs which understand their format. Managed files typically contain databases of some type. The file system should guarantee that 1) managed files can be opened for full or partial exclusive use, and 2) additional functions for insuring that file writes are flushed from cache to disk are available.

  Managed files have the most complex semantics, but not many programs use them. The ones that do typically go to a lot of trouble to get the file semantics right, to maintain database integrity. Read what SQLite needs from a file system to get a sense of how managed files need to work.

• Scratch files Scratch files do not outlive the process group that created them, and are invisible outside that process group. They can be read and written freely, but do not have permanent existence in the file system. The file system should guarantee that when the process group goes away, so does its scratch files, so as not to clutter up the file system. This would stop the accumulation of abandoned temporary files.

We have everything or nearly everything we need to implement this - though admittedly it could be a bit cumbersome...

"Scratch files" can be created by creating a file, opening it, and unlinking (removing) it. The file continues to exist until the last open filehandle to it goes away. But since it's unlinked, it has no permissions and no one else can open it. But if you want to grant a filehandle to that file to another process, you can either fork() and the new process will get a copy of the filehandle, or you can send an open filehandle to another process via... a Unix Domain socket? (IIRC. I know there's a mechanism, I don't remember if Unix Domain Sockets are it. The dbus system uses filehandle passing for certain things, I believe.)

"Unit files" would be more or less the same thing: create the new version of the "unit file" with a temporary name and unlink it from the filesystem. Processes opening the file will see the old copy - until you're ready to commit the new version, at which point you unlink the old version and link the new version to the old filename. Anyone who opened the old version will continue to see it until the last open filehandle to it is closed. The main bit that's missing, I guess, is that this isn't an atomic operation. If you unlink the old file and then somehow fail to link the new version to the old filename, the file momentarily doesn't exist (until you link the old one back again...) And you don't get the "commit all or revert all" behavior you describe for a batch operation.

It's worth noting that "Unit files" take the same amount of resources as the current scheme but you lose the ability to peek at the new version before it's committed...

I'm not

I looked at this problem once, in the context of distributed file systems. I'd divide files into the following categories, with slightly different semantics for each. This can be fitted into the standard UNIX/DOS/Windows model, but it resolves some issues that result in programs doing elaborate workarounds to get correct file semantics.

• Unit files A unit file is only meaningful once it has been completely written and closed. Once closed, it will never be rewritten, only replaced. Most files are unit files. The file system should guarantee that 1) unit file replacement by a new version is an atomic operation, 2) unit files not properly closed do not replace old versions, so that when a program fails while writing a unit file, the old version remains undamaged, and 3) a unit file being written is not visible to other processes until closed.. This was a feature of some mainframe operating systems. Because the UNIX file system concept doesn't have this, there's much fussing around with ".part" files and renaming strategies to achieve unit file semantics.

  The file system could unduplicate unit files based on their content. One approach would be that the real name of a unit file is its cryptographic hash; any other name it has is an alias. Backup programs can usefully use such information.

  The ability to close and commit a group of unit files as an atomic operation would be useful. This should be the last step of an "install", so that if anything goes wrong, the install is automatically backed out, like a database rollback.

  This is the default form of file.

• Log files Log files are written sequentially. Writing can only be done at the end. The file system should guarantee that 1) writing must be sequential, 2) in the event of a crash and restart, the file may not be complete but will be correct up to the end position of the file, which will be at the end of a single write operation, and 3) multiple processes can write to one log file, but all processes append, never overwriting each other.

  This is how UNIX/Linux files ought to work in "append" mode.

• Managed files Managed files are written and read in a random access fashion, but only by programs which understand their format. Managed files typically contain databases of some type. The file system should guarantee that 1) managed files can be opened for full or partial exclusive use, and 2) additional functions for insuring that file writes are flushed from cache to disk are available.

  Managed files have the most complex semantics, but not many programs use them. The ones that do typically go to a lot of trouble to get the file semantics right, to maintain database integrity. Read what SQLite needs from a file system to get a sense of how managed files need to work.

• Scratch files Scratch files do not outlive the process group that created them, and are invisible outside that process group. They can be read and written freely, but do not have permanent existence in the file system. The file system should guarantee that when the process group goes away, so does its scratch files, so as not to clutter up the file system. This would stop the accumulation of abandoned temporary files.

This is how UNIX and Linux should have worked. Today, programs struggle to get those semantics across platforms, but don't always succeed, leaving behind truncated files, partial failed installations, junk files, and database disasters where two programs accessed the same managed file.

As for metadata, the original MacOS "resource fork" concept was a good one. But the original implementation was botched. The resource fork was a badly implemented tree-type database store, one that was corrupted if a program failed to close the file properly. If the resource fork had been implemented so that at the end of each write, the resource fork was guaranteed to be in a usable state, the whole concept might have been more successful. It took a long time for Apple to fix this; the phrase "damaged resource fork" appears tens of thousands of times in Google until 2006.

What does this have to do with "being predictable"? Compiler optimizations, by definition, don't change semantics of code so long as it's written right. If you're relying on something that's undefined behavior according to the language (or your compiler) spec, you're doing it wrong.

There's no line between compiler optimizations and bad organization of code - these two are completely orthogonal. As GGGGGP noted, a common way to optimize code real well, regardless of how it's written, is to concat all .c/.cpp files together, and then compile the result. Doing so will inevitably take a lot of resources, no matter how well it's organized.

What happens if keys point to different types of data (images, text, movies, urls, other tables)?

In that case, you should be using something like SQLite. See this page for more information.

Look at SQLite. It's used _everywhere_ these days. Their own list is only a small segment of it's pervasiveness. I'm a programmer for a large company, and I use it in internal software I develop as part of my job.

Has SQLite made the world a better place? Definitely.
Moreso than a GPL equivalent would have? Of course.
Have they got the full credit they deserve? Perhaps not.

Of course, BSD vs GPL is a matter of personal preference, but after all, how is using SQLite in a proprietary program any different from using Linux in proprietary hardware? I don't see a philosophical difference. It's not as if anyone is taking SQLite, modifying it, and selling their modified copy. Who'd buy it when SQLite is so good?

I work daily with PostgreSQL with great pleasure.

I would add to your list the excellent SQLite : serverless, zero configuration. It's used as storage engine by Thunderbird, among others.

http://www.sqlite.org/index.html

Strangely, Oracle is one of the consortium members.

http://www.sqlite.org/cvstrac/wiki?p=ExperimentalMailUserAgent

You mean SQLite ?

http://www.sqlite.org/ a "replacement for fopen()" -- http://www.sqlite.org/about.html

http://www.sqlite.org/ a "replacement for fopen()" -- http://www.sqlite.org/about.html

Two orders of magnitude is not 20x, it's 100x.

And for non-intensive applications, that's still fine.

And SQLite isn't actually that slow anyway. It's comparable.

There will never be a war between Open Source and Public Domain. It would be nonsensical.

For all of you trivia freaks, SQLite is an example of high quality source code that is widely used and is in the Public Domain.

Richard Hipp (the main auther of SQLite) has said that his entirely open licence http://www.sqlite.org/copyright.html [sqlite.org] has caused problems to some companies, so he also has a commercial option http://www.hwaci.com/cgi-bin/license-step1 [hwaci.com]

as do some other storage engines available for MySQL (such as PBXT).

MyISAM does not, of course; it is more comparable to SQLite, although the latter does have transactions.

I think it should be replaced with SQLite

Your queries do not produce the same result. SELECT from x left join y on x.c is an outer join whereas SELECT from x, y where x.c = y.c is an inner join. Ok, it is a cross join, but most databases will transform it to inner join. I would like to know what are those SQL engines which do a cross join when using the select from x, y where x.c = y.c syntax. Even SQLite converts comma joins to inner joins if it is more efficient to do so. Also, using left joins when inner join is the thing you want is not good for performance. You can look the above link for explanation why.