Real simple.. You don't have a compiler on the target machine
And no scripting languages? Even then, the hacker can just pre-compile a binary.
allow no file transfer services for incoming files.
With root, any access restrictions on the machine itself can easily be defeated. If the access restrictions are outside of the machine (e.g. firewall), an HTTP POST request is a completely legitimate request that can be used to upload files.
Your web server can easily be compiled with a hard-coded config file, and disallow the command-line switch.
If you have root, you can unmount the partition that stores the config files and create a RAM disk with the same path but different content.
No, it's not the end-all-be-all solution, but it is a neat idea, and would add another layer of complexity that most script-kiddies would not easily get around.
Seems like an expensive gimick to me. Besides, most security problems are caused by lazy/incompetent administrations -- if you have the time and knowledge to invest in stuff like this, you're unlikely to be successfuly hacked anyway.
You just make sure that the web server is also on the write-only drive
A "web server" is a process -- it runs in RAM, not on disk. As I said, it would be trivial to kill off the existing web server and write a 50 line C app that dished out hacked pages on port 80.
It'd be easy enough to set up a server where everything (OS, server software, data) is on a read-only hard drive, and the only writable partitions would be tmp and var (for logging).
Not to mention the fact that (a) many web servers have a configurable path to their config file (e.g. through a command-line arg), so you could change that to point to a hacked conf file in/tmp (b) you can easily compile and install your own web server in/tmp.
And you can limit the size of those partitions sufficiently that they could not do a lot with them.. Even make them network partitions.
Not sure why making them "network partitions" would make the slightest difference. Re: size limiting, the source for a trivial HTTP daemon is tiny, much smaller than the smallest practical/tmp partition.
And if you have root, it's easy enough to make a RAM disk and stick your hacked content on that.
I don't see how this provides more than a trivial layer of security. Once someone has root on a box, it's trivial to replace the httpd's root directory with a writeable one (/tmp, even) -- unless the box is locked down very tightly, there will be at least a few directories which the hacker can write data to, which is all that is needed. Or the hacker could modify the web server itself, if it's an OSS one. Or you could even stop the site's HTTP daemon and write your own one, which returns hacked content stored in RAM -- that would be particularly trivial to do.
Also, this scheme would make it difficult to update static web content without rebooting.
Row level locking is a win for a data warehouse, with little writes and many reads, but in a heavily updated transactional environment hits a brick wall pretty quick.
Why would row level locking be a win (compared to MVCC) in that environment? A read doesn't generate a nother version of the tuple, so you don't suffer any MVCC overhead.
Transactions? You must be thinking of another database.
Sorry -- I meant "atomic operation" (which in most databases is a transaction, I suppose it's a single query in MySQL).
The "Data" I've lost was usually still in the middle of being written.
Ah, ok. If the "atomic operation" hadn't completed yet, there is no way the database can (or should) store the results of the operation -- everything should be discarded.
We're too far gone. See my other comments
Ah, I see. Good luck!:-)
Re:From using MySQL/PostgreSQL and researching SAP
on
PostgreSQL vs. SAP?
·
· Score: 1
You mention that PostgreSQL is clean compared to MySQL. I suppose that could mean many things. After studying the database syntax, I didn't feel PostgreSQL was nearly as "clean" as MySQL.
It's certainly much more compliant with the SQL92/99 standards.
One token example is escaping data. Depending on the situation, you may need to escape a character with one, two, or even four backslashes.
The syntax for string literals indicates that you only need to use a single backslash to escape an input value. If the input value you're trying to create has an escape sequence of its own (e.g. a regular expression), you'll need to do both SQL escape and regular expression escapes. I'm not sure what the problem with this behavior is, or how to implement it any better...
With most other databases, there is a prepared statement feature that directly transfers data as a separate blob. With PostgreSQL, good luck getting your binary data in.
If you need to load binary data, the easiest method would probably be to use PQescapeBytea(), or the equivalent function provided by your language interface.
I've never lost data due to a crash (well, nothing that hadn't been written within a few seconds of the crash.)
So in other words, you've lost data due to a crash:-)
So does MySQL not flush disk writes to disk before a transaction commits? Your reply seems to suggest that MySQL waits for the kernel to flush disk buffers to disk -- if that's the case, it definately sounds like there is a potential for data loss.
I've had to deal with quite a few crashes recently.
for starters, take a look here: http://lab.applinet.nl/postgresql-jdbc/ those are the results of the jdbc compliance tests, ringing in at around 43%
Although that page refers to the JDBC driver included with PostgreSQL 7.1 -- I'd be interested to see the results of running those tests against the JDBC driver included with PostgreSQL 7.2 or CVS HEAD.
though i can see from your other posts you're aware of these issues, it's still a bummer.
Yes, our JDBC implementation has a number of deficencies, as you've pointed out -- I think your criticisms are justified. I'm a C programmer, so using Java more than necessary is a bit of a grating experience:-) Nevertheless, I've just started working on getting the latest source to compile with JDK 1.4. Beyond that, I'll see what else I can do...
While MySQL adds features one by one, Postgresql seems to absorb lots of stuff at each releases. While this is great feature-wise, it can lead to pretty nasty bugs and instabilities.
I have to disagree with you there. The long release cycle of PostgreSQL has the goal of integrating features and giving them a long time to stablize and be tested in the tree, before they are released. The beta periods last for a long time, and a release is only made when all the developers are satisfied that no more bugs can be found. So while PostgreSQL only makes releases on an occaisonal basis, every effort is made to ensure that those releases are as stable as possible.
That bug is really critical (can you imagine the state of your datas when counters go backward ?), but it took 2 months to be discovered and fixed.
Yes, that was a bad bug. On the other hand, it had been present for a long time -- IIRC, the bug was present in 7.1.0, 7.1.1, 7.1.2, and 7.2.0, at which point someone reported it and it was fixed. So it's fairly safe to say that it only occurred under very, very rare circumstances, and likely effected very few users.
Actually the MySQL developpers make the effort of documenting every petty bug fix, which is a very professional thing.
Every PostgreSQL release includes the full CVS changelogs from between the previous release and the current one -- so every single change made to the source (including typo fixes, etc.) is listed. A more concise set of release notes are also made available, which highlight the major user-visible changes or bug fixes.
Re:From using MySQL/PostgreSQL and researching SAP
on
PostgreSQL vs. SAP?
·
· Score: 1
Firebird might be a PostgreSQL killer.
I'm curious -- what features does Firebird have that PostgreSQL does not?
I have also used the C++ interface, which seems to work O.K, but it is much less powerfull than the jdbc implementation could be if it were to implement some of the cool features such as writable result sets and prepared statements that the java interfaces declare (but which PostgreSQL doesn't yet implement)
For the C++ interface, you might want to try libpqxx -- it's much nicer (IMHO) than the default libpq++. It's been integrated into the CVS tree and will be in the next release of PostgreSQL.
Updateable result sets for JDBC have been implemented in CVS, and will be in 7.3. I implemented prepared statements and that should be in CVS soon, but I'm not sure what the status of JDBC support for that is.
Downloading the latest PostgreSQL from cvs (a few months ago) did appear to include the source for this, however they wouldn't build with jdk1.4 due to many new methods having been added to the interfaces since the implementation was last updated (and even many of those that were there will just throw an org.postgresql.Driver.notImplemented exception).
Hmmm... this will definately need to be fixed before the 7.3 release (if the 1.4 methods aren't implemented, they should at least be declared and throw exceptions so that JDBC will compile out of the box with JDK 1.4).
psql-jdbc doesn't support prepared statements. well, it doesn't actually prepare them.
I implemented prepareable statements for PostgreSQL a couple weeks ago -- the patch is on pgsql-patches. It should be in CVS in a couple days, and will be in the 7.3 release.
you can't return multiple rowsets from a stored procedure.
This has also been implemented and will be in 7.3 (at least, table functions for functions defined in C -- not sure if there's support for PL/PgSQL table functions yet).
Can you elaborate on the deficiencies you've found in the JDBC implementation?
if this is an "enterprise" level application, i see little choice but to dive into SAP and figure it all out - otherwise i think you'll run into bigger problems down the road.
I think that's an unfair assessment. The only specific criticism you've made of PostgreSQL is the quality of the JDBC driver -- I'm not sure why that has a negative impact on "enterprise level" applications in general (i.e. non JDBC stuff)...
I don't see how this is a good idea. Chastise the RIAA all you like, but the fact is that they have the right to distribute their music in any form they choose. For example, it would be perfectly legal for the RIAA to decide that they want to distribute all their music in vinyl in the future, so as to cut down on piracy. There's nothing wrong with that -- sure, it would hurt their business, and sure, it would be a bad idea, but they are entirely within their rights to do it. And BTW, it's (effectively) their music -- the artists involved have willingly signed distribution deals which give the labels the rights to distribute the music as they see fit. Maybe that's not a good idea on the artist's part, but again -- that's no really the point.
Similarly, there is nothing wrong with copy-protected CDs from a legal standpoint (IANAL). I don't think it's a good idea, and I think it will definately hurt the RIAA's sales, but that doesn't change the fact that the RIAA are entirely justified in choosing the formats in which they want to distribute.
If I have a propriatary app that uses MySQL or PostgreSQL - is the database server part of my app as far as the GPL is concerned?
Not sure about the rest of your question -- but I can tell you that it doesn't apply to PostgreSQL. Since PostgreSQL is released under a BSD-derived license (no advertising clause), you can do what you like with it, pretty much.
Currently, Red Hat Linux offers Sendmail and Postfix as two Mail Transport Agent (MTA) alternatives. For print daemon alternatives, the choices are LPRng and CUPS. The configurations for LPRng and CUPS are completely separate. If you switch from one printing system to another, you will have to reconfigure your printers.
If an employee wants to complete a project using QNX + WN + Python, give him the opportunity to do so. Don't judge him according to the tools he's using. Just wait for the result.
The problem here is that "the result" isn't just the visible product that was produced, but the code that makes it work. Whether you're working on a dynamic website, shrink-wrapped app, or in-house app, the reality is that the code you write is likely to be maintained by a cast of hundreds after you're finished with it.
So yeah, it would be cool to use the latest interesting language -- but standards exist for a reason. My department is moving all our code from a mix of crufty Perl CGIs to Java servlets, JSPs and beans. Does Java suck? Not really, but somewhat. Are there days or projects where I'd rather be using Common Lisp, AxKit, or Zope? Sure -- but that's not the point. If I left an eclectic mess of code behind me like that, it would be a maintainence (and admistrative) nightmare.
I'm not disagreeing with anything you said, in fact you all but reiterated everything I said.
No I didn't, read my post again.
The 7.1 vacuum analyze required table locks.
PostgreSQL has lots of different types of locks of varying granularities. Saying "table locks" doesn't mean a whole lot.
Doesn't matter which phase of it required locks
It does though -- in 7.1, splitting vacuum and analyze internally reduced the time that an exclusive lock needs to be held.
By breaking that into a separate commands the need for downtime is reduced drastically
This is where you're wrong. The reduction in downtime has nothing to do with allowing ANALYZE to be executed separately. It is entirely the result of the new vacuum code (which is "lazy", unlike a VACUUM FULL -- which does a 7.1-style VACUUM). In 7.2, running VACUUM (with or without ANALYZE) is fast, and doesn't require an exclusive lock -- so your database can continue serving clients while a VACUUM is executing. Whether you choose to run ANALYZE at the same time or separately is really irrelevant.
I'm just happy that I don't have to bring a production server to its knees once a week (or for that matter once a day) just to do some table maintenance.
I should probably clarify that - the full locking vacuum was separated out into two parts one which analyzes statistics and doesn't lock the tables and another which does what the old vacuum did by reordering data blocks to shrink the size on disk....
This is incorrect (I believe it describes the situation after 7.1: the VACUUM ANALYZE command only needed to lock the table exclusively when VACUUMing, only a read lock was needed for ANALYZE).
In 7.2, the ANALYZE command can now be used separately, as you say. However, there are other (more important) improvements: ANALYZE only takes a look at a statistical sampling of the rows in the table. This means that collecting statistics on even enormous tables is very fast. Furthermore, VACUUM has been made "lazy" by default: this means that it doesn't attempt to reclaim space as aggressively as before, but it no longer requires an exclusive lock on the database (instead, it cooperates with other DB clients). The old behavior is available as "VACUUM FULL", and it is suggested whenever you need to reclaim a lot of diskspace (e.g. you delete hundreds of thousands of rows of data and need the space).
you might run vacuum once a week or so
It was (and is) suggested that you run VACUUM once per day.
it's the statistics that help the query planner turn SQL into faster queries.
As far as I know, you only really need to update your planner stats when you change the statistical distribution of your data. Of course, running ANALYZE's reasonably often won't do any harm, and is a relatively cheap operation (performance-wise).
As a related aside, I'm gonna stop using Debian if it continues to have such long release cycles. I eventually got suitable openssl (0.9.6), postgres (7.0) and java (1.3) installed, but it took days and a lot of pain because of the length of the "to do A you must first do B, to do B you must first do C, to do... chain.
Run Debian unstable -- my unstable boxes have Linux 2.4.9ac10, XFree86 4.1.0+, Java 1.4b2, Postgres 7.1.3 (or 7.2-dev), OpenSSL 0.96b, etc. Plus, don't be afraid to compile and install stuff by hand -- for instance, I never used pre-packages versions of Apache because I'll be recompiling it frequently anyway.
I've been testing Evo for a while (with an eye on getting that Palm VII), and I'm wondering where I am WRT to the other bug reporters. I've reported 20 unique bugs -- what have other people got?
Slashdot Mirror
And if you have root, it's easy enough to make a RAM disk and stick your hacked content on that.
Bottom line -- this technique is pointless.
I don't see how this provides more than a trivial layer of security. Once someone has root on a box, it's trivial to replace the httpd's root directory with a writeable one (/tmp, even) -- unless the box is locked down very tightly, there will be at least a few directories which the hacker can write data to, which is all that is needed. Or the hacker could modify the web server itself, if it's an OSS one. Or you could even stop the site's HTTP daemon and write your own one, which returns hacked content stored in RAM -- that would be particularly trivial to do.
Also, this scheme would make it difficult to update static web content without rebooting.
So does MySQL not flush disk writes to disk before a transaction commits? Your reply seems to suggest that MySQL waits for the kernel to flush disk buffers to disk -- if that's the case, it definately sounds like there is a potential for data loss. Have you considered using PostgreSQL?
Updateable result sets for JDBC have been implemented in CVS, and will be in 7.3. I implemented prepared statements and that should be in CVS soon, but I'm not sure what the status of JDBC support for that is. Hmmm... this will definately need to be fixed before the 7.3 release (if the 1.4 methods aren't implemented, they should at least be declared and throw exceptions so that JDBC will compile out of the box with JDK 1.4).
Similarly, there is nothing wrong with copy-protected CDs from a legal standpoint (IANAL). I don't think it's a good idea, and I think it will definately hurt the RIAA's sales, but that doesn't change the fact that the RIAA are entirely justified in choosing the formats in which they want to distribute.
Not sure about the rest of your question -- but I can tell you that it doesn't apply to PostgreSQL. Since PostgreSQL is released under a BSD-derived license (no advertising clause), you can do what you like with it, pretty much.
AFAIK the entire PostgreSQL source tree is BSD'd (with the exception of a few optional modules in contrib/, but that is being fixed).
From the release notes linked to in the article:
Currently, Red Hat Linux offers Sendmail and Postfix as two Mail Transport Agent (MTA) alternatives. For print daemon alternatives, the choices are LPRng and CUPS. The configurations for LPRng and CUPS are completely separate. If you switch from one printing system to another, you will have to reconfigure your printers.
That's an interesting school...
The problem here is that "the result" isn't just the visible product that was produced, but the code that makes it work. Whether you're working on a dynamic website, shrink-wrapped app, or in-house app, the reality is that the code you write is likely to be maintained by a cast of hundreds after you're finished with it.
So yeah, it would be cool to use the latest interesting language -- but standards exist for a reason. My department is moving all our code from a mix of crufty Perl CGIs to Java servlets, JSPs and beans. Does Java suck? Not really, but somewhat. Are there days or projects where I'd rather be using Common Lisp, AxKit, or Zope? Sure -- but that's not the point. If I left an eclectic mess of code behind me like that, it would be a maintainence (and admistrative) nightmare.
No I didn't, read my post again.
PostgreSQL has lots of different types of locks of varying granularities. Saying "table locks" doesn't mean a whole lot.
It does though -- in 7.1, splitting vacuum and analyze internally reduced the time that an exclusive lock needs to be held.
This is where you're wrong. The reduction in downtime has nothing to do with allowing ANALYZE to be executed separately. It is entirely the result of the new vacuum code (which is "lazy", unlike a VACUUM FULL -- which does a 7.1-style VACUUM). In 7.2, running VACUUM (with or without ANALYZE) is fast, and doesn't require an exclusive lock -- so your database can continue serving clients while a VACUUM is executing. Whether you choose to run ANALYZE at the same time or separately is really irrelevant.
On that, we agree
You do realize that is a trivial workload, right? Maybe you meant "10 inserts per second"? (or 100x as much)
This is incorrect (I believe it describes the situation after 7.1: the VACUUM ANALYZE command only needed to lock the table exclusively when VACUUMing, only a read lock was needed for ANALYZE).
In 7.2, the ANALYZE command can now be used separately, as you say. However, there are other (more important) improvements: ANALYZE only takes a look at a statistical sampling of the rows in the table. This means that collecting statistics on even enormous tables is very fast. Furthermore, VACUUM has been made "lazy" by default: this means that it doesn't attempt to reclaim space as aggressively as before, but it no longer requires an exclusive lock on the database (instead, it cooperates with other DB clients). The old behavior is available as "VACUUM FULL", and it is suggested whenever you need to reclaim a lot of diskspace (e.g. you delete hundreds of thousands of rows of data and need the space).
It was (and is) suggested that you run VACUUM once per day.
As far as I know, you only really need to update your planner stats when you change the statistical distribution of your data. Of course, running ANALYZE's reasonably often won't do any harm, and is a relatively cheap operation (performance-wise).
Why not '/usr/local' ?
Run Debian unstable -- my unstable boxes have Linux 2.4.9ac10, XFree86 4.1.0+, Java 1.4b2, Postgres 7.1.3 (or 7.2-dev), OpenSSL 0.96b, etc. Plus, don't be afraid to compile and install stuff by hand -- for instance, I never used pre-packages versions of Apache because I'll be recompiling it frequently anyway.
I've been testing Evo for a while (with an eye on getting that Palm VII), and I'm wondering where I am WRT to the other bug reporters. I've reported 20 unique bugs -- what have other people got?