Continuous VACUUM actually is the solution. It just should be automatic instead of manual.
Databases inevitably have some point in a transaction where they require 2 versions of the same row to be present in persistent storage (on disk). That obviously means that the old version (or the new version in case of a rollback) has to be removed at some point in time. Some databases choose to do this on transaction commit, adding a little bit of overhead to each transaction. Some databases choose to do this in a separate process at scheduled intervals, reducing the commit overhead but adding the overhead of having more versions on disk. PostgreSQL has choosen the second path, and VACUUM is the cleanup process.
Which solution is the best depends on the requirements. As you have discovered, tables with a high turnover get easily bloated when the cleanup is not done frequently enough. The solution for that is to cleanup more often, with cleanup at commit of each transaction as the higher limit. But quite likely it is sufficient if cleanup occurs every X transactions or every Y seconds.
The intention was to have the pg_autovacuum utility integrated in the backend to manage the vacuum process for all databases in a cluster. If enabled, it would allow for automatic vacuum and analyze on tables, with some logic to learn if tables are high-turnover or fairly static. Unfortunately, the patch for that didn't make it into the 8.0 beta.
PostgreSQL has buildin recovery. Upon every boot the system verifies if it was correctly shut down and if not it replays the Write Ahead Log (WAL) from the last checkpoint (which occur every X seconds and force all data to be written to disk) until the last committed transaction. That is all recovery you need in a database, and it has been in PostgreSQL for years.
And by all standards I know, PostgreSQL is a true multiuser database. It supports concurrent access and acls (at many levels, the cluster, the database, the schema, the table etc. you can tell exactly who can see and touch what). As a matter of fact, we happily run dozens of PostgreSQL databases for our webhosted customers.
A universal ID is simple. Just piggyback it on top of some unique identifier we already have. The article mentions using a SIM, but we could also easily piggyback it on top of DNS. Every customer gets a unique identifier as in "username@realm" from his ISP, where realm is a fully qualified domain name of the ISP.
In fact, that is exactly what RFC 2486, The Network Access Identifier does. No need for anything new there.
If you can't do this now already, there is something wrong with the setup of either your accesspoints or your device. Walking around while listening to webradio on your laptop without even a hickup when you switch from one access point to another is everyday reality here.
Granted, without the billing (because they feel that internet access should be free for their community), but many Dutch universities and research institutions together with SURFnet (the National Research and Education Network) have developed a roaming solution already. Based on IEEE 802.1x, EAP-TTLS and RADIUS it allows for seemless roaming between the participants.
This WiFi roaming has recently been extended and now institutions in Portugal and Croatia are joining as wel.
European NRENs are already doing this on an international scale. With credentials from participating local university students and employees can already log on to the WiFi hotspots of other participating universities and research institutions in both their own and other countries.
The authentication mechanism is based on IEEE 802.1x and uses a RADIUS backend to enable cross domain authentication. Currently this is in operation (machine translation) between the Netherlands, Portugal and England, and Croatia is next.
SURFnet is also working (machine translation) with local hotspot operators in the Netherlands to make sure Dutch students can roam their networks as well. Initially this will offered as a free service fro students.
First, it is not made for the users. It is made for the developers that want a nice email client for themselves and graceously let us use that email client too. Second, after the choices have been made for what functionality is going to be in it, everything that is in the codebase that supports unwanted functionality is indeed unneeded code.
Yes, you are wrong, as of PostgreSQL 7.2 VACUUM can run without locking the table completely.
Garbage collection is a problem every database faces. Due to ACID requirements it is pretty much (absolutely?) impossible to run a database that updates rows without having multiple versions of the same row on disk at some time during the operation. So at some point in time you have to get rid of that duplicate. You can choose to do that after commit of a transaction (or the last transaction for which the row is still visible), but that would potentially make every transaction slower. So in PostgreSQL the choice was made to do this at an administrator determined moment (and I presume that choice also was the easy one). In older versions of PostgreSQL VACUUM would lock the entire table and physically force all the valid rows to be rewritten consecutively and then reclaim the space at the end. This mode is still available as VACUUM FULL, but nowadays there is a new mode (sometimes called lazy vacuum) that only marks space safe to be overwritten. Subsequent updates/inserts will overwrite it eventually. Regular running of this command will eventually lead to some steady state where there is some x% of bloat in the table, but there is no significant amount of locking required.
XS4ALL, one of the largest Dutch ISP's, has made a system where users can specify their own RBL settings. They can choose for up to 13 RBL's to be used to either reject or tag email messages with custom headers which can easily be processed further. I think that is about as good as it gets, and since it is the user himself that has to set his own preferences, the default is to allow everything to pass, all the arguments about free speach are also pretty moot. It currently serves over 140.000 users, so I think that it is pretty scalable (although it doesn't do content scanning).
Like screen readers are the solution to all problems. How are better screenreaders going to help deaf people? How are screen readers going to help people with a physical dysability that prevents them from using a mouse? How are screen readers going to help those with ADHD? How usefull is a screenreader if the navigation is on the bottom of 200 lines of text?
Accessibility is not about reading webpages aloud to the blind.
You are linking to something that counts security advisories for Linux and compares them with security advisories for other OS'es. You seem to have misunderstood that security advisories for a certain OS are something entirely different from security advisories for a RDBMS. The 2 are completely unrelated.
In fact, if you want to you can run PostgreSQL on Windows if receiving less security advisories is more important than the time-to-fix. Or if you feel really paranoid, just run PostgreSQL on OpenBSD. That gives you an excellent safety record, with only 1 remotely exploitable security problem in nearly 6 year.
Nonsense. UT used to be the largest user in the Netherlands until about 14 months ago. After that, they were surpassed by Delft University of Technology, which until the fire today used about 50% more bandwidth as the UT. And even they aren't the biggest anymore, because the University of Utrecht recently became larger than them. In total, about 20.000 rooms for individual students in the Netherlands are responsible for over 1 Gbps of traffic. This is non-stop (i.e. during the evening it is closer to 2 Gbps), and over 90+ % is outgoing traffic from 'unknown ports'.
I guess forcing the RIAA and the MPAA to change their business models the hard way counts as innovation nowadays:)
I recently tried quite a few databases to get a free one with full JDBC and Unicode support. Let's just say that even with first installing Cygwin PostgreSQL was a lot easier to get running on Windows, let alone on Unix. FireBird idem. I don't see SAPdb overcoming this hurddle on the entry level easily. On the top level, it is probably issues like the (lack of a) reputation.
I think that the main threat these systems pose to privacy is not how much data becomes accessible to the government, but how much data becomes accessible to private entities. I remember seeing a story (on CNN?) about a bar somewhere in the US that required everybody who entered to have their drivers license scanned. Through some system the owner could then extract details such as name, gender and address to profile his customers.
I think that these systems pose a far bigger threat to the privacy of the average citizen as the desire of some government to give everybody a unique number to be able to normalize some databases that are only for internal use.
The MBone is alive and kicking. I am having 79 channels in my IP/TV listing as we speak (while watching an MPEG-1 real-time transmition of a TV channel).
Best site to get more info and downloads is http://videolabs.uoregon.edu/
Why do you assume this is about Flash when the person submitting the email to the discusiion list is working on CFML Language Development?
Do you know anything the general public does not know, or are you just jumping to conclusions? (Understandable, the wish is the father of the thought.)
I don't see the problem. All it says is that if they have released product X under the Macromedia Open Source License and take part of product X and put it in product Y they do not need to tell you they copied the code from product X and where to get the source of product X. But it only applies to their own contributions, so if somebody else contributed a little bit to X as well and that part is to be included in product Y, they still need to include it. It is just an assertion that their own licence is not used to force them to opensource more as they want.
To me, it looks like they just want to make sure their licence is not viral. And that is not a problem for OSI certification, judging by the fact that the BSD licence is OSI certified.
Slashdot Mirror
Continuous VACUUM actually is the solution. It just should be automatic instead of manual.
Databases inevitably have some point in a transaction where they require 2 versions of the same row to be present in persistent storage (on disk). That obviously means that the old version (or the new version in case of a rollback) has to be removed at some point in time. Some databases choose to do this on transaction commit, adding a little bit of overhead to each transaction. Some databases choose to do this in a separate process at scheduled intervals, reducing the commit overhead but adding the overhead of having more versions on disk. PostgreSQL has choosen the second path, and VACUUM is the cleanup process.
Which solution is the best depends on the requirements. As you have discovered, tables with a high turnover get easily bloated when the cleanup is not done frequently enough. The solution for that is to cleanup more often, with cleanup at commit of each transaction as the higher limit. But quite likely it is sufficient if cleanup occurs every X transactions or every Y seconds.
The intention was to have the pg_autovacuum utility integrated in the backend to manage the vacuum process for all databases in a cluster. If enabled, it would allow for automatic vacuum and analyze on tables, with some logic to learn if tables are high-turnover or fairly static. Unfortunately, the patch for that didn't make it into the 8.0 beta.
PostgreSQL has buildin recovery. Upon every boot the system verifies if it was correctly shut down and if not it replays the Write Ahead Log (WAL) from the last checkpoint (which occur every X seconds and force all data to be written to disk) until the last committed transaction. That is all recovery you need in a database, and it has been in PostgreSQL for years.
And by all standards I know, PostgreSQL is a true multiuser database. It supports concurrent access and acls (at many levels, the cluster, the database, the schema, the table etc. you can tell exactly who can see and touch what). As a matter of fact, we happily run dozens of PostgreSQL databases for our webhosted customers.
A universal ID is simple. Just piggyback it on top of some unique identifier we already have. The article mentions using a SIM, but we could also easily piggyback it on top of DNS. Every customer gets a unique identifier as in "username@realm" from his ISP, where realm is a fully qualified domain name of the ISP.
In fact, that is exactly what RFC 2486, The Network Access Identifier does. No need for anything new there.
If you can't do this now already, there is something wrong with the setup of either your accesspoints or your device. Walking around while listening to webradio on your laptop without even a hickup when you switch from one access point to another is everyday reality here.
Granted, without the billing (because they feel that internet access should be free for their community), but many Dutch universities and research institutions together with SURFnet (the National Research and Education Network) have developed a roaming solution already. Based on IEEE 802.1x, EAP-TTLS and RADIUS it allows for seemless roaming between the participants.
This WiFi roaming has recently been extended and now institutions in Portugal and Croatia are joining as wel.
You should be able to figure out how it is supposed to work from the following PowerPoint presentation:
Enterprise Campus Security: Addressing the Imploding Perimeter (especially slides 23-27)
European NRENs are already doing this on an international scale. With credentials from participating local university students and employees can already log on to the WiFi hotspots of other participating universities and research institutions in both their own and other countries.
The authentication mechanism is based on IEEE 802.1x and uses a RADIUS backend to enable cross domain authentication. Currently this is in operation (machine translation) between the Netherlands, Portugal and England, and Croatia is next.
SURFnet is also working (machine translation) with local hotspot operators in the Netherlands to make sure Dutch students can roam their networks as well. Initially this will offered as a free service fro students.
XHTML is 3 years old.
And neither standard appeared overnight.CSS 1 is 7 years old.
First, it is not made for the users. It is made for the developers that want a nice email client for themselves and graceously let us use that email client too.
Second, after the choices have been made for what functionality is going to be in it, everything that is in the codebase that supports unwanted functionality is indeed unneeded code.
And if you don't like it, fork.
Well, the built process might be a bit tedious, but you can download the source for a native version of PostgreSQL 7.2.1 for Windows.
Yes, you are wrong, as of PostgreSQL 7.2 VACUUM can run without locking the table completely.
Garbage collection is a problem every database faces. Due to ACID requirements it is pretty much (absolutely?) impossible to run a database that updates rows without having multiple versions of the same row on disk at some time during the operation. So at some point in time you have to get rid of that duplicate. You can choose to do that after commit of a transaction (or the last transaction for which the row is still visible), but that would potentially make every transaction slower. So in PostgreSQL the choice was made to do this at an administrator determined moment (and I presume that choice also was the easy one).
In older versions of PostgreSQL VACUUM would lock the entire table and physically force all the valid rows to be rewritten consecutively and then reclaim the space at the end. This mode is still available as VACUUM FULL, but nowadays there is a new mode (sometimes called lazy vacuum) that only marks space safe to be overwritten. Subsequent updates/inserts will overwrite it eventually.
Regular running of this command will eventually lead to some steady state where there is some x% of bloat in the table, but there is no significant amount of locking required.
If you use AOL you don't care about your email anyway.
XS4ALL, one of the largest Dutch ISP's, has made a system where users can specify their own RBL settings. They can choose for up to 13 RBL's to be used to either reject or tag email messages with custom headers which can easily be processed further. I think that is about as good as it gets, and since it is the user himself that has to set his own preferences, the default is to allow everything to pass, all the arguments about free speach are also pretty moot.
It currently serves over 140.000 users, so I think that it is pretty scalable (although it doesn't do content scanning).
Like screen readers are the solution to all problems. How are better screenreaders going to help deaf people? How are screen readers going to help people with a physical dysability that prevents them from using a mouse? How are screen readers going to help those with ADHD? How usefull is a screenreader if the navigation is on the bottom of 200 lines of text?
Accessibility is not about reading webpages aloud to the blind.
You are linking to something that counts security advisories for Linux and compares them with security advisories for other OS'es. You seem to have misunderstood that security advisories for a certain OS are something entirely different from security advisories for a RDBMS. The 2 are completely unrelated.
p orted-platforms.html
In fact, if you want to you can run PostgreSQL on Windows if receiving less security advisories is more important than the time-to-fix. Or if you feel really paranoid, just run PostgreSQL on OpenBSD. That gives you an excellent safety record, with only 1 remotely exploitable security problem in nearly 6 year.
For a complete list of supported platforms see http://developer.postgresql.org/docs/postgres/sup
Phroenix
Nonsense. UT used to be the largest user in the Netherlands until about 14 months ago. After that, they were surpassed by Delft University of Technology, which until the fire today used about 50% more bandwidth as the UT. And even they aren't the biggest anymore, because the University of Utrecht recently became larger than them.
:)
In total, about 20.000 rooms for individual students in the Netherlands are responsible for over 1 Gbps of traffic. This is non-stop (i.e. during the evening it is closer to 2 Gbps), and over 90+ % is outgoing traffic from 'unknown ports'.
I guess forcing the RIAA and the MPAA to change their business models the hard way counts as innovation nowadays
I recently tried quite a few databases to get a free one with full JDBC and Unicode support. Let's just say that even with first installing Cygwin PostgreSQL was a lot easier to get running on Windows, let alone on Unix. FireBird idem.
I don't see SAPdb overcoming this hurddle on the entry level easily. On the top level, it is probably issues like the (lack of a) reputation.
I think that the main threat these systems pose to privacy is not how much data becomes accessible to the government, but how much data becomes accessible to private entities. I remember seeing a story (on CNN?) about a bar somewhere in the US that required everybody who entered to have their drivers license scanned. Through some system the owner could then extract details such as name, gender and address to profile his customers.
I think that these systems pose a far bigger threat to the privacy of the average citizen as the desire of some government to give everybody a unique number to be able to normalize some databases that are only for internal use.
The MBone is alive and kicking. I am having 79 channels in my IP/TV listing as we speak (while watching an MPEG-1 real-time transmition of a TV channel).
Best site to get more info and downloads is http://videolabs.uoregon.edu/
Why develop propietary switches and fileformats when all that is required is a full implementation of multicast (which is just a part of IP)?
Why do you assume this is about Flash when the person submitting the email to the discusiion list is working on CFML Language Development?
Do you know anything the general public does not know, or are you just jumping to conclusions? (Understandable, the wish is the father of the thought.)
I don't see the problem. All it says is that if they have released product X under the Macromedia Open Source License and take part of product X and put it in product Y they do not need to tell you they copied the code from product X and where to get the source of product X. But it only applies to their own contributions, so if somebody else contributed a little bit to X as well and that part is to be included in product Y, they still need to include it.
It is just an assertion that their own licence is not used to force them to opensource more as they want.
To me, it looks like they just want to make sure their licence is not viral. And that is not a problem for OSI certification, judging by the fact that the BSD licence is OSI certified.
I have one word for you: multicast.
Imagine a Beowulf cluster of PDA's.