Before doing this you may want to check what the local laws are. Police States do not like privacy. Encryption is not always legal. If you find it's illegal you will probably also want to check what the penalties are.
As any veteran of website Content Management Systems (CMS) will tell you, the problem is that custom coded sites are a nightmare in waiting for most customers. Unless you have a mega budget, and a team of developers, and the guarantee of the ongoing budget and team, you are much better off using a existing CMS. The security is likely to be better than with a custom CMS especially if there are lots of people contributing, the opportunities to grow the site through new functionality will be better, especially if it's open source, and the cost is likely to be less for more.
Behind the scenes most of CMS there is a database. Often, but not always, it's coupled to a scripting language, and usually it's organized in such a way that it's modular. This modularity is what gives non-custom CMSs their teeth, but compared to custom CMSs, the cost is that the queries on the database are non-optimal. Often data is read twice or more by different parts of the CMS because they aren't able to directly work together to retrieve the data in the way that a custom CMS can. It's this explosion of database queries which is usually responsible for any drop in comparative speed, and not the cost of executing scripts.
The revolutionary approach is not to waste time redeveloping stuff that other people have already built, but to reuse at low or no cost through the power of open source. It's almost always quicker, easier and cheaper to buy faster hardware and use a scripting language than it is to develop software in C++.
Some games and operating systems may be written in C++, but they have different needs and requirements, and operate in different environments; a formula 1 car is optimized for traveling around a race circuit, but it is useless when there are speed bumps.
Drupal represents tens of millions of dollars investment and is built by some of the world's best CMS developers. It takes a lot for a custom CMS to compete with that.
Disclosure: I'm involved in the Drupal project. I develop for Drupal. I've been developing for Drupal for 6 years. I have programmed in C++ and scripting languages. I have built custom CMSs. I've built sites in other CMSs. I threw away my CMS framework when I discovered Drupal, not because it was better, but because, even where it wasn't, the rate of growth of Drupal meant that it soon would be. The rate at which Drupal is developed far outstrips anything that all but the very largest companies can sustain. I would say that I'm a CMS veteran of contemporary CMSs. Whatever that means.
Critical is perhaps the wrong word
on
Drupal 7
·
· Score: 2, Informative
The issues are critical because they are blockers to releasing a polished product.
There will be no fundamental API changes. And chances are that most of these won't affect you in any way if you build a site using Drupal.
Drupal 7 is already being used on production sites - some of them quite major - but we can't yet recommend you use it.
The list of issues at the moment is:
"Do not enable the management menu by default". This is about removing some on-screen duplication that wastes pixels.
"Cannot install on PHP 5.3.2". I have a working version on 5.3.2. This is a false positive - something to do with the reporters set up. D7 needs PHP PDO extension, which is part of the default PHP. Most likely not enabled for the person who reported the bug.
"Links are needlessly unable to fully participate in D7 AJAX framework" Making sure that the Ajax framework is orthogonal with respect to content types, I think.
"Security harden stream wrappers by defaulting them as remote" We worry about security. This would subject streams to the level of scrutiny that remote data is subject to. If allow_url_include in php.ini is false, then Drupal should not allow remote streams as content.
"Skip to main content" link doesn't work correctly in the overlay" We are screen-reader friendly by default. We also have a new mode of operation - overlaying a screen with things that need to be done before you can continue with your original task. This should work for screen readers too, so the link needs to point to the 'topmost' content.
"All fields are hidden after the administrator newly configures a view mode to not use 'default' display" Oops! It shouldn't be possible to get into a state where critical stuff seemingly disappears. A feature that's central to D7 is fields - that is, all content is extensible with new fields. It should always be possible to set fields to display correctly.
"Some schema code incorrectly rely on the generic type instead of the engine-specific type" D7 supports storing data in different storage formats, including several database engines. There's some small inconsistency; the schema (and things that manipulate it) should use the column types specific to a particular engine - for instance it's AUTO_INCREMENT in MySQL, but Serial in PostgreSQL, and different rules apply to how each is used.
"Clean-up the upgrade path: UPGRADE.txt" If you are upgrading from Drupal 6, we'd like it to be a smooth ride.
"Screen reader users need a clear, quick way to disable the overlay". Not all the screen readers will like the overlay system, so there should be a ridiculously easy way to stop using overlays, which is immediately accessible from a screen reader.
"Allow dashboard to limit available blocks" Users can configure their own dashboard, but not everything that is displayable makes sense on the dashboard page, so it would be sensible to limit what can be displayed. The User Interface for this is being tested.
And I'd like to take this opportunity to advertise - Drupalcon Chicago will be in March 2011. And if you are in Europe, I have it on reasonably good authority that you can look forward to the announcement of the location of the Autumn conference either later this week or next week.
And this week we moved the Drupal.org site redesign live.
I think I can claim to be a real programmer having designed & written shrinkwrap software, built in-house corporate software, developed embedded control software, made web apps used by thousands of users daily, written several frameworks, all for software running on a diverse range of hardware, from microcontrollers through to mainframes and most things in between, in a pretty wide variety of languages, from systems languages like C, through to dynamic languages like PHP and Ruby, and all the usual suspects in between.
In my opinion Drupal core is well written. Tons of thought goes into it, with lots of smart, experienced people weighing in on the architectures and reviewing the code. It's (far) better than most closed source apps I've had the pleasure of working with.
This is also true of the top, say 10 to 20% of non-core modules. The ones that no serious site is without. The quality of the rest varies hugely, from bad, through to awesomely good.
Drupal is not alone here. This is a problem with almost all except the simplest CMSs.
The problem is one of complexity: when you are doing one thing, it's easy to optimize it into one SQL query. But when you have a modular system, you want each module to work independently as much as possible because you want to avoid dependency hell, and you want to minimize module interaction because that will lead to side-effect bugs which are difficult to track down. The architecture choice that most CMSs make in order to foster sustainable growth is that each module gets to handle its own data. But that means that you don't get optmized database queries - in fact you are almost certainly guaranteed that you will have multiple reads of the same data, as different modules work on that data.
Which is why all mature CMSs also have caching strategies. They typically cache the output of whole pages if pages do not differ by user, and partial pages where they do, constructing the page from the appropriate parts. They also usually have in-memory caching strategies to avoid repeated db reads.
Better CMS's have variable caching strategies, allowing you to store pages in memory, in files and in database as appropriate. Drupal has modules for all of these strategies and also works well with Varnish, a caching proxy/accelerator.
My 4.5 modules worked with 4.6 with no changes. My 4.6 modules worked with 4.7 with no changes. My 4.7 modules worked with 5 with very minor changes. My 5 modules worked with 6 with very small changes. I haven't tried porting any 6 modules to 7 yet, but looking at the list of modules that already support 7, I'm not quivering with fright at the prospect.
The API is not guaranteed not to change, but a lot of people work hard to make sure that the change is evolutionary and the thousands of automated tests help to make sure that
I seriously object to my comments being marked as troll! This was not trolling. It's a serious point. It's hardly reasonable to say that there's one rule if it happens in the US, and another for everywhere else. Yes, I know that there were/are wars in both places, but there _have_ been deaths that were murder rather than battlefield deaths, and US citizens have not been held to account in courts of the countries where the actions took place.
Except that there was a recent report that terahertz radiation was resonating DNA and partially unzipping it, creating 'bubbles' of unzipped regions. I haven't seen any corroboration of this, so it may not be proven, but it's certainly worth a pause because it's not an effect that had been predicted. Now mm waves are not sub-millimeter waves, but they are adjacent in the spectrum, and it's this sort of thing that we should worry about.
Probably there's no cause for concern, but again - when the potential impact is very high, though the risk is low, we should be taking great care.
The health effects of millimeter wave scanning are what we should be worried about - there's an unknown risk but a high possible impact: imagine if in 10 years time millions of people start developing melanomas as a result of being scanned.
The x-ray backscatter machines are much less worrying; we've had 100(?) years of experience with X-rays and we understand what x-rays can do to DNA.
But we have very little experience of mm level radiation.
What I've seen in the press is cheerleading. 'Experts say there is little cause to worry' with unknown experts talking in vague generalities. I've seen articles saying that the energy involved is less than the energy emitted by a cell-phone. That may well be the case, but it's not in the cell-phone spectrum, and even a little energy in the wrong place can do a lot of damage. Just see what a match can do to a pile of paper*.
Of course it's impossible to completely prove something is safe. But I don't think we don't have empirical evidence that it's safe. Or at least - I've not seen it.
I absolutely have not made an exhaustive search for literature on the health effects of mm radiation, and I'm not an expert. In the brief searches I have made I have seen that there's a lot of scare-noise based on what seems like only a few sources which imo are not applicable. What I don't see is a long list of studies. And even more striking is that that non-existent list of studies is not full of papers saying 'we found no observable effects'**.
But I have found that it's possible to cook bacteria with mm waves! Maybe this is a hint that there's a potential problem. And in the diagrams of atmospheric absorption of radiation that I've looked at, for example, it looks like mm waves are mostly absorbed, which suggests that we'd have little evolved defense to mm wave damage.
What I'd really like to see is a series of mm wave experts saying things like 'we've studied the health effects of mm wave scanners for 10 years now, and I'd have no qualms about subjecting my three month old baby to a scan because I'm confident that there's no health risk associated.'
* Pedants: yes, I know the energies involved are different orders of magnitude - it's a metaphor.
** It's quite possible I was looking in the wrong place. I'd be very pleased to see a detailed response with a link to this list.
Didn't American's fight a war of independence because of this? Maybe American politicians have forgotten, Someone seems to think that it's ok to make law without reference to the people it affect.
For the people who are helped by music when programming:
I listen to a pretty wide range of music, from classical through to current pop, with a lot of stuff inbetween. I find that some music that I like helps me to program, but other music, which I also like hinders me.
I work better with rhythmically strong music - a heavy beat without deep complexity. Music which encourages me to pay attention to lyrics is bad, but that might just be that familiarity is a requirement so as not to get sidetracked into _listening_ to the music.
The best music I know of for taking me to the zone is Talking Heads. And their best programming album of all for me is Stop Making Sense, which I must have listened to thousands of times.
It's starting to get repetitive. I would like some new programming music: what music helps you most?
Mail it to Linus, Alan Cox and the maintainers of subsystems which it abuses. Include clear notes of how it works, and what can be done to protect the systems. If you can't trust these people with it, then you should not trust Linux with your data at all. Even better, since you understand the tricks it uses, if you can write some patches, and submit them, together with your proof of exploit.
On a personal note - I also want to say thank you for doing this work. I use Linux both on servers, and as my normal desktop, and I'm immensely pleased that people are looking at making it safer: thank you.
If you are reading this, then you are probably under 45 years old.
I think the average life expectancy is rising - currently 75 years old or so in most Western countries - for men.
But more importantly medicine is going through an exponential increase in its abilities. It hasn't yet reached microprocessor rates of doubling every 18 months, but we certainly have learned more in about the last 20 or 30 years about how the human body works than we learned in all of history before that.
This strongly suggests that the average life expectancy will start rising exponentially too, with a lag to account for the development of applications from the basic discoveries.
So it is not unreasonable to expect some of you reading this to live to 150, barring any nasty accidents. And that means that climate change will have a direct impact on your life. You may have to live through mass starvation, wars fought over resources like water, extinction of once common species and the rise of diseases once only found in the tropics.
So, whether or not it's a totally proven phenomenon is irrelevant - _you_ personally can't afford to act as if it is unproven: you need to take personal action now, in the hope that it will help to mitigate the consequences.
We as a species need to learn the lessons of all the previous civilisations that have managed to extinguish themselves through resource starvation; don't spend time arguing that it may not be happening - act as if it is while hoping that it's not. Definitely carrying on with research at the same time. Maybe we will show that it's not happening, and that would be fantastic, but we can't wait for that.
I can't answer most of your questions because I haven't read the book, but I can tell you that a node is the smallest unit of addressable content that Drupal deals in. What's that in English, you ask? By addressable, I mean it has a URL, and can be displayed provided you have the rights to view it. By content I mean that it's something that someone entered into Drupal, and it's stored in the database. By smallest I mean that as a reader of the site, while you often view a node by itself, nodes can also be combined to produce a page - for instance by the Views module or the Panels module. Usually this combination is a node itself.
Nodes can also be differentiated from one another by their Content Type, for instance one node may be a Story or Article node, while another might be a House Listing node. This helps in making lists of nodes, and in searching them.
From a programming point of view, a node is an interface abstraction - if you write a module for dealing with a new sort of content, and want all the benefits of the framework (for example, making any data held in your new sort of content searchable), then you implement a few well-described interfaces, and voila!, Drupal is able to manipulate your new sort of content. This interface is in the form of hooks that you implement for your new sort of content.
You are also free to ignore these hooks, putting your data outside of the range of what Drupal considers to be content. An example of a module that does this is Webform - which is used to collect forms that site users fill in. This data does not appear in the Drupal search index, or benefit from any of the automatic rights that nodes enjoy.
Sorry about the state of the documentation: it's hard to find the right level for everyone. Perhaps it needs more signposting.
Contrariwise: it's very well structured. Each module has a defined structure. The core is modular too, with functionality being in predictable places. Functions are given predictable names, thanks to the hook mechanism.
If you are going to contend that it could have been laid out better, give firm examples.
Isolation. PHP4 does not have namespacing. PHP4 does not support interfaces. Until recently PHP4 was supported by Drupal. The point of isolation is to enhance security and attempt to limit the effect of bugs. Given that the language makes few attempts to support programming-in-the-large, it's a bit much to expect software developed using it to do so.
I suspect that by 'interfaces are pretty much defined by giving modules access to the entire core', you meant that you would like to sprinkle getters and setters liberally, then you are right. Drupal eschews a layer of make-work that prettifies without adding any value. PHP is not an especially speedy language, and adding them would have little benefit in return for a unhealthy slowdown. If you mean that the there is an agreed structure to key variables, and they are passed between procedures without limiting access to subparts, then yes, you are right. You might expect this to cause problems, but in my experience, it doesn't. Partly this is because of the test layer. Partly because the system depends upon modules obeying implicit rules.
I see no cop-out at all. Modules that respect the rules of interaction work fine together. Where modules are known not to work together, this is documented.
'If the module API isolated internals...' This isn't possible in PHP without using objects. PHP4 objects did not support the hook mechanism. A smart a object-based hook mechanism for PHP5, that didn't require registration of each hook, only became possible with magic methods. Only supported since PHP5. And last time I looked the performance of magic methods was abysmal.
'The code is garbage all the way through.' Another sweeping statement. Clearly all the tens of thousands of lines of code are rubbish.
Globals. The point of reducing the use of globals is to minimise the impact of a change in one place impacting unexpectedly in another. Drupal6 has 45 globals. Most of these are sensible as globals: for instance $language - an object containing information for the active language. Of these 45 only 1 is in wide use: $user which represents the current user, storing their preferences.
View, logic & database access. Database - Drupal contains a database independent data layer and has since Drupal5. All access is through the database layer. Views layer: Drupal has a strong theming layer, with more than 1 theming engine to choose from. All presentation is through themes. The data passed to themes is a limited structured subset of the data held by the core. Hooking into output requires you to work through the theme layer. Logic is separate from the theme layer and is usually event driven. Ownership of these tasks is implicit in the names of the functions - for instance - mymodule_block_theme is the callback for theming the block output by mymodule.
Methods are long. Yes, sometimes. Sometimes not. It depends on what is sensible. The same chunks of code: I'm sure there are repeats. I've not seen whatever it is that you mean - perhaps some examples.
I don't contend that Drupal is perfect. It's not. But what I see in your list is a bunch of accusations, with no proof.
Yes, each version is an evolutionary jump on the previous version. Not surprisingly, D6 has got better caching than D5.
Boost: It replaces pages with static html. This static cache is updated on a schedule. But if you have someone who is logged in, because it completely bypasses PHP, you have to replace all the page elements that are different for logged in people with Javascript driven widgets which load the block data dynamically. For a site with lots of elements this is a lot of work. Plus you are depending on Javascript being switched on. So if you have lots of logged in users, and lots of blocks just for logged in users, and you don't have much control over their environment the site is viewed in, then this isn't really a starter.
Global cache clear whacks everything, obviously, but it's quite surprising to me that item/page cache clear does this. I've never had a close look at caching on D5, but knowing that there's a potential problem here is useful. Thanks.
I'm guessing that you already track the groups.drupal.org that deal with high volume sites and the distributions intended for your situation like PressFlow, Mercury, OpenPublish, and http://groups.drupal.org/high-performance. Pressflow in particular has spent lots of effort on optimisation, and there's a D5 version. Pressflow also only supports MySQL, which answers one of your concerns. Maybe you've already trawled through it, looking at SQL optimisations they've done.
InnoDB: I can't comment, but I'm pretty sure that someone in the high-performance group has experience to offer. I think most high-performance sites are using InnoDB, and I'm pretty sure it's the Pressflow default. I know there's an issue with node counts on InnoDB, and I think that Pressflow changes the way that this is handled.
BTW - check out http://groups.drupal.org/node/25617 not so much for the comparison of performance stats on D6, because it's concerned with non-logged in users, but because the discussion below the article will probably interest you.
Once more, thanks for your very good posts; it's always useful to learn other people's experiences.
Thanks for the detailed reply. Sorry, I wasn't meaning to teach you to suck eggs. That part of my reply was for readers who don't know much about Drupal beyond it having a strange name.
Interesting: I didn't know about the table lock on cache! Are you using Drupal 6? I've just done a search, but didn't find anything. I did find a status report I didn't know about! At admin/reports/status/sql which shows lock hits amongst other things. Maybe you are on Drupal 5? The cache updates I found where all straight inserts/updates or deletes. But whatever - I'd say that you were right in moving caching to memcached if you have to support millions of pageviews.
Yes - I agree the cache layer needs PHP execution, and that's not so great. Actually, IMO it's not the PHP, but that it's a hit against the database, even though it is a single query. A hit against the file system would avoid the whole DB connection and query. Drupal Boost caching module wouldn't bring huge benefits for you because you have lots of logged in users, but it uses static files and doesn't hit the database.
Block cache can be efficient at cutting down database hits and code execution. And if you are using Views2, it will also use block caching for HTML and queries. It has options for lots of different caching strategies - from one for everyone through to per user per block. Block caching not well documented though.
One thing that I'm not sure of is why one website would interfere with the cache of another if you are not sharing content. [Readers: you can configure Drupal to use a single database for all websites, or multiple databases. If you have a single database, then you can configure each installation to use a different set of tables, through setting a prefix to the table name. It's very unusual to share any of the cache tables across sites except in the shared-content scenario, where the same content is being presented on more than one website.] Maybe I've misunderstood. It might be that the sites are running on a cluster against a db shared across the cluster. In this scenario, I can imagine one machine locking others out, though I'm not sure where in the code a table lock is.
Multisite: Good points and bad points. The management of update is the bad point, when you put in a new module with database upgrades. You have global downtime unless you do progressive updates by putting the module first in the specific directory for each of the particular sites and running update for that site. But it does lower the admin cost.
SQL rewriting works in some circumstances. See hook_db_rewrite_sql. [Readers: The base problem is that it's difficult with modularity to avoid explosion of queries. This is true of all modular systems with database interactivity. You can avoid it if one module is intimately aware of another, and then the two can co-operate, though this is context dependent. For instance - if the first module loads the user details (name, user account, email, etc), and a second, optional, module loads a user profile, then you could cut down on the queries by including the user profile columns as part of the user table, and loading the two at the same time. Modularity, however, is not your friend here - to make the management of a profile module easy, it's much better to have it in its own table. But now you are doing table joins or loading one and then the other. If you have a heavily customised system, then you can make the joined table by hand, and modify the queries. You can do this either by changing the core code, or by rewriting the SQL on-the-fly, by writing a new module. With the second option you can leave the core alone, and your changes are overlayed, at the cost of a small drop in performance of the PHP (since there's an extra function to run). The PHP drop is almost certainly negligible compared to the database speed-up, except where the database is already in memory.]
I absolutely agree - look at the source code - one of the reasons that we standardised on Drupal was that the code is well structured and well thought out. It's based on a call-back system with hooks triggered by naming convention. It is well documented, and has a wide series of tests that are automatically run against it, for regression tests of changes. The API is documented at api.drupal.org, and includes examples of how you can extend the system.
All the above applies to the core code, and the widely used modules just outside core - like Views, which is a module to help non-programmers build structured queries and present the results.
I agree that amongst the third-tier modules, the quality varies hugely, as you see in much open-source software. Some of them are very well written, but others are not.
Each module has an issue queue, which is publicly visible, and usage statistics tell you how widely it's used, which allows you to make some decisions even if you aren't a coder.
The only criticism I have with the core code is a technical one that doesn't matter to end-users; most of it is not object-orientated. This is not to say that it's unstructured: it is very modular. And there's a good reason it's not object oriented: it's because PHP4 made it very difficult to build a name triggered callback system with objects. However, I expect the core will become more and more object oriented over time since PHP5 objects are much more robust with reasonable performance characteristics.
What is large-ish in your context? I'm not sure that readers will realise that you are coming from a particular point of view. I maintain a Drupal installation with 1.5m nodes, which I wouldn't call large, because it has a limited purpose and does not use hundreds of extra modules. We have not had to patch or customise. And it runs on a VM on commodity hardware.
I assume that you have a site with a great deal of functionality. Not every reader will realise that a CMS that is modular is naturally going to result in an explosion of queries; every module controls its own queries, each with its own tables. There are several ways of combating this: you can cache content and results, you can lazy-load daa, or you can reduce modularity, combining tables.
Drupal already uses the first two strategies, and occasionally prticular modules are designed to work together and implement the third. For anonymous users, pages are delivered from cache. For logged in users pages can be put together from cached blocks, if your site is set up to do this. (There are some restrictions on this; it's not available in every circumstance.) It sounds like you may have shared content which is an unusual installation. Under normal installations, one site of a multisite install does not ineract with another site. The caching is not designed with shared content multisites in mind; I've also had problems with this, but I think that this is an edge case. It would be great if your organisation would contribute your experience, by posting your rewritten cache engine.
I don't however agree with your suggestion of a focus on a single database. Yes, specialising to a single database will allow the use of features available on that database only, but at the cost of a portion of the audience. Similarly, we could say only a particular version of a particular database was supported, to allow us to use the optimisations available for that engine, but again this would cut the audience dramatically. Instead what is needed is a structured way to optimise. To a small extent this is already available through the SQL rewite hook, but this isn't especially safe. Drupal 7 deals with this by abstracting the SQL further, but the SQL rewrite is much safer, and more suited to rewrites for optimising.
I'd be really interested in where you've had full table locking problems. Do you have an issue reference on drupal.org?
Regarding "... if you're looking for something to build a high-traffic site with plenty of customization and member-only data, you're better off building your own framework from scratch..." With any CMS there's a trade-off: you can build your own, bearing the full cost yourself, or you can use one that's built already, whether closed or open source, amortizing the cost of the build over many organisations and individuals. By my estimation, Drupal core represents many millions of dollars of work. It's possible that your organisation can afford to invest that sort of amount in your own custom CMS, but the experience of companies that have built their own is that the cost of maintenance is too high, and many of them are migrating to open source software, which does more than they could hope to do, at a cheaper price, but which comes with its own problems. Organisations can use the money saved on the build to either make larger sites, more sites, or specialise the system, which is what your organisation appears to be doing.
It would be fantastic if others could learn from your DBA experience. Would you consider posting a case study to drupal.org?
I do see a lack of civility in your posts, but not much information.
Drupal does claim to be modular. It doesn't often claim to be simple, unless you are only using a few core modules, perhaps in the default install, for instance, to make a blog. In the base install it _is_ less polished than, say, Wordpress. But it does more. I'm not sure how you measure intuitivity - what's intuitive for one person isn't necessarily, for another.
I don't know what you mean by 'Behavours are inconsistent across themes'. Isn't the point of themes to bring individualism to your Drupal installation? Doesn't that by definition mean that different themes will do different things?
Regarding 'Half the available themes are broken'. I have no idea where you get this statistic. I do know that if you install a Drupal 5 theme on a Drupal 6 installation, it will most likely not work. Perhaps that's what you did?
Regarding 'Not supporting things off the bat'. Again, I'm not sure what you want? By default Drupal 6 supports posting stories, pages, blogging, aggregating content from feeds, hierarchically structured collections of pages, comments on any content, contact forms, multilingual sites, forums, login with OpenID, polls, clean URLs, content categorisation, registered user profiles, searching, basic statistics tracking, and user uploads. Amongst other things. I'm not sure what else you would like a CMS to do as in the basic installation.
Regarding 'Enabling the modules requires manual downloads and dependency hells.'. Yes, in most installations you have to manually download and enable modules. This makes it more secure. Modules list their requirements and will not allow you to enable them unless their dependencies are also available. I'm not sure how this is 'hell'? Surely it's sensible not to allow a module to be enabled if it's guaranteed to break?
Regarding 'The notion of using the site as you build it is shit,' On the other hand, eating your own dogfood is a very good way of being sure that your users will enjoy the site you are building.Usually one would put the site into maintenance mode, but this isn't required. But again, if you don't like to use what your users use, you ARE able to use an admin theme. If you are building a larger site, it's not recommended practice to work on the live site. I have no idea what you mean about 'just writing your own CSS'.
'The only people who are willing to waste the time to understand Drupal enough...are those...chasing buzzwords...' Drupal is not for everyone. It's definitely not perfect; it's a large, rapidly growing system, which is being improved as it grows. Quality outside of the core is variable, as you would expect. It doesn't solve every problem, despite having around 5,000 modules available, spread across Drupal 5 and 6; clearly it didn't suit you, although it is fine for hundreds of thousands of other users. I don't know what problem you were trying to solve, but there are many CMS out there, and I'd recommend you start by investigating Sharepoint.
Are you complaining that the security team takes time to go through the 2000+ components, find problems and notify you?
You can unsubscribe from the list, and rely only upon the status subsystem, which if you have not switched it off, will notify you on a regular basis about upgrades and security fixes for the only modules you are using.
In contrast to your assertion: Drupal has an _excellent_ security history, and the fact that you are alerted about updates serves to highlight this.
You may wish to switch to a CMS which has no security warnings, but I would not feel comforted by lack of warnings.
It would appear that your experience doesn't stretch terribly far; off the top of my head I can name several much less secure systems. Finding, fixing and announcing vulnerabilities is a good thing: by your measure a hugely exploited CMS with no fixes would be better!
Regarding you assertion that the rewrite engine cannot be disabled; this is just plain wrong. The Apache rewrite engine can be disabled without any problem. If you do this, then you won't enjoy clean URLs, instead you'll have URLs like www.somesite.com/index.php?q=some/path instead of www.somesite.com/some/path. Internally Drupal always works with the first form. However, the rewrite engine is a widely used Apache module - with perhaps millions(?) of sites using it. It may very well have exploits - just as any software may - but it is trusted by lots of users.
Followsymlinks can be disabled too. It's required for rewriting and for one form of upload. Drupal works without problems without it. However, there's nothing inherently insecure in symlinks, and the default Drupal directory layout does not symlink to outside of the install tree.
Database load. I note that your assertion about load is without any reference to figures. I'm not certain which CMS you think is well written. However I'll note that there is a general problem with CMSs which are designed to be easily extensible: tightly integrated system usually use a single SQL statement to retrieve data - the designer knows all the constraints at design-time. A loosely coupled system is usually not able to do this: the designer has little idea of what will be present at run time. So it's in the nature of most loosely coupled system to run one query or more for each additional module. Drupal uses a loosely coupled callback orientated architecture. This means its very easy to extend. However the downside is that each module will usually include extra tables. Drupal is fairly smart about loading this extra data, but beyond that, to counteract the tendency for growth in queries, Drupal has a caching subsystem that is active in several layers. For anonymous users, Drupal only runs a few queries which determine where in the cache the data sits, and returns it.
Perhaps you'd like to elaborate with some firm figures and an example of a CMS that in your opinion does it right.
Regarding PHP security. Again - have you any firm facts to show that PHP is inherently less secure than any other language? The consensus in security circles is that openness is better for security. *You* are able to download the PHP source code and contribute patches. If you know of a security issue, I'd urge you to help fix it. Or is this opinion without facts to back it up?
Again, I'd be interested to know which CMS you do recommend to the person in the street. I would not at the moment recommend Drupal for most brochureware sites, though it is capable of brochureware, however for sites in excess of about 100 pages, for sites where there is a heavy community aspect, and for sites which hope to change and grow, Drupal is an excellent choice.
I think you are misinformed. Morpheus seemed to be targeted at a range of software, including Joomla, but not Drupal: as far as I can see, none of the URL's it scanned are Drupal-based. See http://zeroq.kulando.de/post/2008/08/20/morfeus-fucking-scanner for example, but there are others out there.
In fact, Drupal has an excellent history of security. We find holes, fix them and issue patches. There is a security mailing list that anyone can sign up to. You will receive mail on the latest security fixes. Your Drupal installation will tell you when components are out of date, and when there are security updates. It will also email you on a regular basis if you don't care to look at your status, or ignore the status message at the top of the page when you log in as an administrator. Drupal will not download and install components without human intervention: components require manual installation.
Just like any software, I'm certain that Drupal has as yet undiscovered exploits. What's important is whether they are found and fixed, and we have a good track record of doing this.
Microsoft have derived a stack of publicity from the Sensecam and lifeblogging - it's made them look like a terrific company. I think this PR needs some counter-balance: Microsoft made Lyndsey Williams, the inventor of the Sensecam, redundant. Possibly not the best way to reward someone who was responsible for millions of dollars of positive PR; you don't get rid of the people who are doing brilliant work if you plan on delivering brilliant products in the future. But this has probably been a good thing for the rest of us; Ms Williams is prolific in bringing new devices to prototype and beyond. Her site shows her recent work, including the Sensebulb - a device for non-intrusive monitoring of elderly people who live alone. It can detect unusual situations and alert friends and relatives. This would have saved the lives of two people I knew. She has a stack of other interesting projects on the go too. Her site is well worth reading.
Disclosure: I know Ms Williams and take the opportunity to promote her work whenever I can. I'm not paid for this: I'm not in PR.
Slashdot Mirror
Before doing this you may want to check what the local laws are. Police States do not like privacy. Encryption is not always legal. If you find it's illegal you will probably also want to check what the penalties are.
As any veteran of website Content Management Systems (CMS) will tell you, the problem is that custom coded sites are a nightmare in waiting for most customers. Unless you have a mega budget, and a team of developers, and the guarantee of the ongoing budget and team, you are much better off using a existing CMS. The security is likely to be better than with a custom CMS especially if there are lots of people contributing, the opportunities to grow the site through new functionality will be better, especially if it's open source, and the cost is likely to be less for more.
Behind the scenes most of CMS there is a database. Often, but not always, it's coupled to a scripting language, and usually it's organized in such a way that it's modular. This modularity is what gives non-custom CMSs their teeth, but compared to custom CMSs, the cost is that the queries on the database are non-optimal. Often data is read twice or more by different parts of the CMS because they aren't able to directly work together to retrieve the data in the way that a custom CMS can. It's this explosion of database queries which is usually responsible for any drop in comparative speed, and not the cost of executing scripts.
The revolutionary approach is not to waste time redeveloping stuff that other people have already built, but to reuse at low or no cost through the power of open source. It's almost always quicker, easier and cheaper to buy faster hardware and use a scripting language than it is to develop software in C++.
Some games and operating systems may be written in C++, but they have different needs and requirements, and operate in different environments; a formula 1 car is optimized for traveling around a race circuit, but it is useless when there are speed bumps.
Drupal represents tens of millions of dollars investment and is built by some of the world's best CMS developers. It takes a lot for a custom CMS to compete with that.
Disclosure: I'm involved in the Drupal project. I develop for Drupal. I've been developing for Drupal for 6 years. I have programmed in C++ and scripting languages. I have built custom CMSs. I've built sites in other CMSs. I threw away my CMS framework when I discovered Drupal, not because it was better, but because, even where it wasn't, the rate of growth of Drupal meant that it soon would be. The rate at which Drupal is developed far outstrips anything that all but the very largest companies can sustain. I would say that I'm a CMS veteran of contemporary CMSs. Whatever that means.
And I'd like to take this opportunity to advertise - Drupalcon Chicago will be in March 2011. And if you are in Europe, I have it on reasonably good authority that you can look forward to the announcement of the location of the Autumn conference either later this week or next week. And this week we moved the Drupal.org site redesign live.
I think I can claim to be a real programmer having designed & written shrinkwrap software, built in-house corporate software, developed embedded control software, made web apps used by thousands of users daily, written several frameworks, all for software running on a diverse range of hardware, from microcontrollers through to mainframes and most things in between, in a pretty wide variety of languages, from systems languages like C, through to dynamic languages like PHP and Ruby, and all the usual suspects in between.
In my opinion Drupal core is well written. Tons of thought goes into it, with lots of smart, experienced people weighing in on the architectures and reviewing the code. It's (far) better than most closed source apps I've had the pleasure of working with.
This is also true of the top, say 10 to 20% of non-core modules. The ones that no serious site is without. The quality of the rest varies hugely, from bad, through to awesomely good.
Drupal is not alone here. This is a problem with almost all except the simplest CMSs.
The problem is one of complexity: when you are doing one thing, it's easy to optimize it into one SQL query. But when you have a modular system, you want each module to work independently as much as possible because you want to avoid dependency hell, and you want to minimize module interaction because that will lead to side-effect bugs which are difficult to track down. The architecture choice that most CMSs make in order to foster sustainable growth is that each module gets to handle its own data. But that means that you don't get optmized database queries - in fact you are almost certainly guaranteed that you will have multiple reads of the same data, as different modules work on that data.
Which is why all mature CMSs also have caching strategies. They typically cache the output of whole pages if pages do not differ by user, and partial pages where they do, constructing the page from the appropriate parts. They also usually have in-memory caching strategies to avoid repeated db reads.
Better CMS's have variable caching strategies, allowing you to store pages in memory, in files and in database as appropriate. Drupal has modules for all of these strategies and also works well with Varnish, a caching proxy/accelerator.
My 4.5 modules worked with 4.6 with no changes. My 4.6 modules worked with 4.7 with no changes. My 4.7 modules worked with 5 with very minor changes. My 5 modules worked with 6 with very small changes. I haven't tried porting any 6 modules to 7 yet, but looking at the list of modules that already support 7, I'm not quivering with fright at the prospect.
The API is not guaranteed not to change, but a lot of people work hard to make sure that the change is evolutionary and the thousands of automated tests help to make sure that
I seriously object to my comments being marked as troll! This was not trolling. It's a serious point. It's hardly reasonable to say that there's one rule if it happens in the US, and another for everywhere else. Yes, I know that there were/are wars in both places, but there _have_ been deaths that were murder rather than battlefield deaths, and US citizens have not been held to account in courts of the countries where the actions took place.
http://www.huffingtonpost.com/2010/04/05/wikileaks-exposes-video-o_n_525569.html
http://edition.cnn.com/2009/WORLD/meast/05/12/iraq.soldiers.killed/
http://www.nytimes.com/2010/01/08/world/asia/08blackwater.html
Which is to say that there is no absolute rule that a person should be tried in the jurisdiction of the place under which the action had effect.
US citizens have been guilty of murder in Iraq and Afghanistan, and yet we do not see them on trial there. Why not?
The lesson I take from this is that it's not a rule that a crime is tried in the place in which it was committed.
You are absolutely right.
Except that there was a recent report that terahertz radiation was resonating DNA and partially unzipping it, creating 'bubbles' of unzipped regions. I haven't seen any corroboration of this, so it may not be proven, but it's certainly worth a pause because it's not an effect that had been predicted. Now mm waves are not sub-millimeter waves, but they are adjacent in the spectrum, and it's this sort of thing that we should worry about.
Probably there's no cause for concern, but again - when the potential impact is very high, though the risk is low, we should be taking great care.
The health effects of millimeter wave scanning are what we should be worried about - there's an unknown risk but a high possible impact: imagine if in 10 years time millions of people start developing melanomas as a result of being scanned.
The x-ray backscatter machines are much less worrying; we've had 100(?) years of experience with X-rays and we understand what x-rays can do to DNA.
But we have very little experience of mm level radiation.
What I've seen in the press is cheerleading. 'Experts say there is little cause to worry' with unknown experts talking in vague generalities. I've seen articles saying that the energy involved is less than the energy emitted by a cell-phone. That may well be the case, but it's not in the cell-phone spectrum, and even a little energy in the wrong place can do a lot of damage. Just see what a match can do to a pile of paper*.
Of course it's impossible to completely prove something is safe. But I don't think we don't have empirical evidence that it's safe. Or at least - I've not seen it.
I absolutely have not made an exhaustive search for literature on the health effects of mm radiation, and I'm not an expert. In the brief searches I have made I have seen that there's a lot of scare-noise based on what seems like only a few sources which imo are not applicable. What I don't see is a long list of studies. And even more striking is that that non-existent list of studies is not full of papers saying 'we found no observable effects'**.
But I have found that it's possible to cook bacteria with mm waves! Maybe this is a hint that there's a potential problem. And in the diagrams of atmospheric absorption of radiation that I've looked at, for example, it looks like mm waves are mostly absorbed, which suggests that we'd have little evolved defense to mm wave damage.
What I'd really like to see is a series of mm wave experts saying things like 'we've studied the health effects of mm wave scanners for 10 years now, and I'd have no qualms about subjecting my three month old baby to a scan because I'm confident that there's no health risk associated.'
* Pedants: yes, I know the energies involved are different orders of magnitude - it's a metaphor.
** It's quite possible I was looking in the wrong place. I'd be very pleased to see a detailed response with a link to this list.
Didn't American's fight a war of independence because of this? Maybe American politicians have forgotten, Someone seems to think that it's ok to make law without reference to the people it affect.
For the people who are helped by music when programming:
I listen to a pretty wide range of music, from classical through to current pop, with a lot of stuff inbetween. I find that some music that I like helps me to program, but other music, which I also like hinders me.
I work better with rhythmically strong music - a heavy beat without deep complexity. Music which encourages me to pay attention to lyrics is bad, but that might just be that familiarity is a requirement so as not to get sidetracked into _listening_ to the music.
The best music I know of for taking me to the zone is Talking Heads. And their best programming album of all for me is Stop Making Sense, which I must have listened to thousands of times.
It's starting to get repetitive. I would like some new programming music: what music helps you most?
Mail it to Linus, Alan Cox and the maintainers of subsystems which it abuses. Include clear notes of how it works, and what can be done to protect the systems. If you can't trust these people with it, then you should not trust Linux with your data at all. Even better, since you understand the tricks it uses, if you can write some patches, and submit them, together with your proof of exploit.
On a personal note - I also want to say thank you for doing this work. I use Linux both on servers, and as my normal desktop, and I'm immensely pleased that people are looking at making it safer: thank you.
If you are reading this, then you are probably under 45 years old.
I think the average life expectancy is rising - currently 75 years old or so in most Western countries - for men.
But more importantly medicine is going through an exponential increase in its abilities. It hasn't yet reached microprocessor rates of doubling every 18 months, but we certainly have learned more in about the last 20 or 30 years about how the human body works than we learned in all of history before that.
This strongly suggests that the average life expectancy will start rising exponentially too, with a lag to account for the development of applications from the basic discoveries.
So it is not unreasonable to expect some of you reading this to live to 150, barring any nasty accidents. And that means that climate change will have a direct impact on your life. You may have to live through mass starvation, wars fought over resources like water, extinction of once common species and the rise of diseases once only found in the tropics.
So, whether or not it's a totally proven phenomenon is irrelevant - _you_ personally can't afford to act as if it is unproven: you need to take personal action now, in the hope that it will help to mitigate the consequences.
We as a species need to learn the lessons of all the previous civilisations that have managed to extinguish themselves through resource starvation; don't spend time arguing that it may not be happening - act as if it is while hoping that it's not. Definitely carrying on with research at the same time. Maybe we will show that it's not happening, and that would be fantastic, but we can't wait for that.
I can't answer most of your questions because I haven't read the book, but I can tell you that a node is the smallest unit of addressable content that Drupal deals in. What's that in English, you ask? By addressable, I mean it has a URL, and can be displayed provided you have the rights to view it. By content I mean that it's something that someone entered into Drupal, and it's stored in the database. By smallest I mean that as a reader of the site, while you often view a node by itself, nodes can also be combined to produce a page - for instance by the Views module or the Panels module. Usually this combination is a node itself.
Nodes can also be differentiated from one another by their Content Type, for instance one node may be a Story or Article node, while another might be a House Listing node. This helps in making lists of nodes, and in searching them.
From a programming point of view, a node is an interface abstraction - if you write a module for dealing with a new sort of content, and want all the benefits of the framework (for example, making any data held in your new sort of content searchable), then you implement a few well-described interfaces, and voila!, Drupal is able to manipulate your new sort of content. This interface is in the form of hooks that you implement for your new sort of content.
You are also free to ignore these hooks, putting your data outside of the range of what Drupal considers to be content. An example of a module that does this is Webform - which is used to collect forms that site users fill in. This data does not appear in the Drupal search index, or benefit from any of the automatic rights that nodes enjoy.
Sorry about the state of the documentation: it's hard to find the right level for everyone. Perhaps it needs more signposting.
Contrariwise: it's very well structured. Each module has a defined structure. The core is modular too, with functionality being in predictable places. Functions are given predictable names, thanks to the hook mechanism.
If you are going to contend that it could have been laid out better, give firm examples.
Isolation. PHP4 does not have namespacing. PHP4 does not support interfaces. Until recently PHP4 was supported by Drupal. The point of isolation is to enhance security and attempt to limit the effect of bugs. Given that the language makes few attempts to support programming-in-the-large, it's a bit much to expect software developed using it to do so.
I suspect that by 'interfaces are pretty much defined by giving modules access to the entire core', you meant that you would like to sprinkle getters and setters liberally, then you are right. Drupal eschews a layer of make-work that prettifies without adding any value. PHP is not an especially speedy language, and adding them would have little benefit in return for a unhealthy slowdown. If you mean that the there is an agreed structure to key variables, and they are passed between procedures without limiting access to subparts, then yes, you are right. You might expect this to cause problems, but in my experience, it doesn't. Partly this is because of the test layer. Partly because the system depends upon modules obeying implicit rules.
I see no cop-out at all. Modules that respect the rules of interaction work fine together. Where modules are known not to work together, this is documented.
'If the module API isolated internals...' This isn't possible in PHP without using objects. PHP4 objects did not support the hook mechanism. A smart a object-based hook mechanism for PHP5, that didn't require registration of each hook, only became possible with magic methods. Only supported since PHP5. And last time I looked the performance of magic methods was abysmal.
'The code is garbage all the way through.' Another sweeping statement. Clearly all the tens of thousands of lines of code are rubbish.
Globals. The point of reducing the use of globals is to minimise the impact of a change in one place impacting unexpectedly in another. Drupal6 has 45 globals. Most of these are sensible as globals: for instance $language - an object containing information for the active language. Of these 45 only 1 is in wide use: $user which represents the current user, storing their preferences.
View, logic & database access. Database - Drupal contains a database independent data layer and has since Drupal5. All access is through the database layer. Views layer: Drupal has a strong theming layer, with more than 1 theming engine to choose from. All presentation is through themes. The data passed to themes is a limited structured subset of the data held by the core. Hooking into output requires you to work through the theme layer. Logic is separate from the theme layer and is usually event driven. Ownership of these tasks is implicit in the names of the functions - for instance - mymodule_block_theme is the callback for theming the block output by mymodule.
Methods are long. Yes, sometimes. Sometimes not. It depends on what is sensible. The same chunks of code: I'm sure there are repeats. I've not seen whatever it is that you mean - perhaps some examples.
I don't contend that Drupal is perfect. It's not. But what I see in your list is a bunch of accusations, with no proof.
Yes, each version is an evolutionary jump on the previous version. Not surprisingly, D6 has got better caching than D5.
Boost: It replaces pages with static html. This static cache is updated on a schedule. But if you have someone who is logged in, because it completely bypasses PHP, you have to replace all the page elements that are different for logged in people with Javascript driven widgets which load the block data dynamically. For a site with lots of elements this is a lot of work. Plus you are depending on Javascript being switched on. So if you have lots of logged in users, and lots of blocks just for logged in users, and you don't have much control over their environment the site is viewed in, then this isn't really a starter.
Global cache clear whacks everything, obviously, but it's quite surprising to me that item/page cache clear does this. I've never had a close look at caching on D5, but knowing that there's a potential problem here is useful. Thanks.
I'm guessing that you already track the groups.drupal.org that deal with high volume sites and the distributions intended for your situation like PressFlow, Mercury, OpenPublish, and http://groups.drupal.org/high-performance. Pressflow in particular has spent lots of effort on optimisation, and there's a D5 version. Pressflow also only supports MySQL, which answers one of your concerns. Maybe you've already trawled through it, looking at SQL optimisations they've done.
InnoDB: I can't comment, but I'm pretty sure that someone in the high-performance group has experience to offer. I think most high-performance sites are using InnoDB, and I'm pretty sure it's the Pressflow default. I know there's an issue with node counts on InnoDB, and I think that Pressflow changes the way that this is handled.
BTW - check out http://groups.drupal.org/node/25617 not so much for the comparison of performance stats on D6, because it's concerned with non-logged in users, but because the discussion below the article will probably interest you.
Once more, thanks for your very good posts; it's always useful to learn other people's experiences.
Thanks for the detailed reply. Sorry, I wasn't meaning to teach you to suck eggs. That part of my reply was for readers who don't know much about Drupal beyond it having a strange name.
Interesting: I didn't know about the table lock on cache! Are you using Drupal 6? I've just done a search, but didn't find anything. I did find a status report I didn't know about! At admin/reports/status/sql which shows lock hits amongst other things. Maybe you are on Drupal 5? The cache updates I found where all straight inserts/updates or deletes. But whatever - I'd say that you were right in moving caching to memcached if you have to support millions of pageviews.
Yes - I agree the cache layer needs PHP execution, and that's not so great. Actually, IMO it's not the PHP, but that it's a hit against the database, even though it is a single query. A hit against the file system would avoid the whole DB connection and query. Drupal Boost caching module wouldn't bring huge benefits for you because you have lots of logged in users, but it uses static files and doesn't hit the database.
Block cache can be efficient at cutting down database hits and code execution. And if you are using Views2, it will also use block caching for HTML and queries. It has options for lots of different caching strategies - from one for everyone through to per user per block. Block caching not well documented though.
One thing that I'm not sure of is why one website would interfere with the cache of another if you are not sharing content. [Readers: you can configure Drupal to use a single database for all websites, or multiple databases. If you have a single database, then you can configure each installation to use a different set of tables, through setting a prefix to the table name. It's very unusual to share any of the cache tables across sites except in the shared-content scenario, where the same content is being presented on more than one website.] Maybe I've misunderstood. It might be that the sites are running on a cluster against a db shared across the cluster. In this scenario, I can imagine one machine locking others out, though I'm not sure where in the code a table lock is.
Multisite: Good points and bad points. The management of update is the bad point, when you put in a new module with database upgrades. You have global downtime unless you do progressive updates by putting the module first in the specific directory for each of the particular sites and running update for that site. But it does lower the admin cost.
SQL rewriting works in some circumstances. See hook_db_rewrite_sql. [Readers: The base problem is that it's difficult with modularity to avoid explosion of queries. This is true of all modular systems with database interactivity. You can avoid it if one module is intimately aware of another, and then the two can co-operate, though this is context dependent. For instance - if the first module loads the user details (name, user account, email, etc), and a second, optional, module loads a user profile, then you could cut down on the queries by including the user profile columns as part of the user table, and loading the two at the same time. Modularity, however, is not your friend here - to make the management of a profile module easy, it's much better to have it in its own table. But now you are doing table joins or loading one and then the other. If you have a heavily customised system, then you can make the joined table by hand, and modify the queries. You can do this either by changing the core code, or by rewriting the SQL on-the-fly, by writing a new module. With the second option you can leave the core alone, and your changes are overlayed, at the cost of a small drop in performance of the PHP (since there's an extra function to run). The PHP drop is almost certainly negligible compared to the database speed-up, except where the database is already in memory.]
I absolutely agree - look at the source code - one of the reasons that we standardised on Drupal was that the code is well structured and well thought out. It's based on a call-back system with hooks triggered by naming convention. It is well documented, and has a wide series of tests that are automatically run against it, for regression tests of changes. The API is documented at api.drupal.org, and includes examples of how you can extend the system.
All the above applies to the core code, and the widely used modules just outside core - like Views, which is a module to help non-programmers build structured queries and present the results.
I agree that amongst the third-tier modules, the quality varies hugely, as you see in much open-source software. Some of them are very well written, but others are not.
Each module has an issue queue, which is publicly visible, and usage statistics tell you how widely it's used, which allows you to make some decisions even if you aren't a coder.
The only criticism I have with the core code is a technical one that doesn't matter to end-users; most of it is not object-orientated. This is not to say that it's unstructured: it is very modular. And there's a good reason it's not object oriented: it's because PHP4 made it very difficult to build a name triggered callback system with objects. However, I expect the core will become more and more object oriented over time since PHP5 objects are much more robust with reasonable performance characteristics.
What is large-ish in your context? I'm not sure that readers will realise that you are coming from a particular point of view. I maintain a Drupal installation with 1.5m nodes, which I wouldn't call large, because it has a limited purpose and does not use hundreds of extra modules. We have not had to patch or customise. And it runs on a VM on commodity hardware.
I assume that you have a site with a great deal of functionality. Not every reader will realise that a CMS that is modular is naturally going to result in an explosion of queries; every module controls its own queries, each with its own tables. There are several ways of combating this: you can cache content and results, you can lazy-load daa, or you can reduce modularity, combining tables.
Drupal already uses the first two strategies, and occasionally prticular modules are designed to work together and implement the third. For anonymous users, pages are delivered from cache. For logged in users pages can be put together from cached blocks, if your site is set up to do this. (There are some restrictions on this; it's not available in every circumstance.) It sounds like you may have shared content which is an unusual installation. Under normal installations, one site of a multisite install does not ineract with another site. The caching is not designed with shared content multisites in mind; I've also had problems with this, but I think that this is an edge case. It would be great if your organisation would contribute your experience, by posting your rewritten cache engine.
I don't however agree with your suggestion of a focus on a single database. Yes, specialising to a single database will allow the use of features available on that database only, but at the cost of a portion of the audience. Similarly, we could say only a particular version of a particular database was supported, to allow us to use the optimisations available for that engine, but again this would cut the audience dramatically. Instead what is needed is a structured way to optimise. To a small extent this is already available through the SQL rewite hook, but this isn't especially safe. Drupal 7 deals with this by abstracting the SQL further, but the SQL rewrite is much safer, and more suited to rewrites for optimising.
I'd be really interested in where you've had full table locking problems. Do you have an issue reference on drupal.org?
Regarding "... if you're looking for something to build a high-traffic site with plenty of customization and member-only data, you're better off building your own framework from scratch..." With any CMS there's a trade-off: you can build your own, bearing the full cost yourself, or you can use one that's built already, whether closed or open source, amortizing the cost of the build over many organisations and individuals. By my estimation, Drupal core represents many millions of dollars of work. It's possible that your organisation can afford to invest that sort of amount in your own custom CMS, but the experience of companies that have built their own is that the cost of maintenance is too high, and many of them are migrating to open source software, which does more than they could hope to do, at a cheaper price, but which comes with its own problems. Organisations can use the money saved on the build to either make larger sites, more sites, or specialise the system, which is what your organisation appears to be doing.
It would be fantastic if others could learn from your DBA experience. Would you consider posting a case study to drupal.org?
I do see a lack of civility in your posts, but not much information.
Drupal does claim to be modular. It doesn't often claim to be simple, unless you are only using a few core modules, perhaps in the default install, for instance, to make a blog. In the base install it _is_ less polished than, say, Wordpress. But it does more. I'm not sure how you measure intuitivity - what's intuitive for one person isn't necessarily, for another.
I don't know what you mean by 'Behavours are inconsistent across themes'. Isn't the point of themes to bring individualism to your Drupal installation? Doesn't that by definition mean that different themes will do different things?
Regarding 'Half the available themes are broken'. I have no idea where you get this statistic. I do know that if you install a Drupal 5 theme on a Drupal 6 installation, it will most likely not work. Perhaps that's what you did?
Regarding 'Not supporting things off the bat'. Again, I'm not sure what you want? By default Drupal 6 supports posting stories, pages, blogging, aggregating content from feeds, hierarchically structured collections of pages, comments on any content, contact forms, multilingual sites, forums, login with OpenID, polls, clean URLs, content categorisation, registered user profiles, searching, basic statistics tracking, and user uploads. Amongst other things. I'm not sure what else you would like a CMS to do as in the basic installation.
Regarding 'Enabling the modules requires manual downloads and dependency hells.'. Yes, in most installations you have to manually download and enable modules. This makes it more secure. Modules list their requirements and will not allow you to enable them unless their dependencies are also available. I'm not sure how this is 'hell'? Surely it's sensible not to allow a module to be enabled if it's guaranteed to break?
Regarding 'The notion of using the site as you build it is shit,' On the other hand, eating your own dogfood is a very good way of being sure that your users will enjoy the site you are building.Usually one would put the site into maintenance mode, but this isn't required. But again, if you don't like to use what your users use, you ARE able to use an admin theme. If you are building a larger site, it's not recommended practice to work on the live site. I have no idea what you mean about 'just writing your own CSS'.
'The only people who are willing to waste the time to understand Drupal enough...are those...chasing buzzwords...' Drupal is not for everyone. It's definitely not perfect; it's a large, rapidly growing system, which is being improved as it grows. Quality outside of the core is variable, as you would expect. It doesn't solve every problem, despite having around 5,000 modules available, spread across Drupal 5 and 6; clearly it didn't suit you, although it is fine for hundreds of thousands of other users. I don't know what problem you were trying to solve, but there are many CMS out there, and I'd recommend you start by investigating Sharepoint.
Are you complaining that the security team takes time to go through the 2000+ components, find problems and notify you?
You can unsubscribe from the list, and rely only upon the status subsystem, which if you have not switched it off, will notify you on a regular basis about upgrades and security fixes for the only modules you are using.
In contrast to your assertion: Drupal has an _excellent_ security history, and the fact that you are alerted about updates serves to highlight this.
You may wish to switch to a CMS which has no security warnings, but I would not feel comforted by lack of warnings.
It would appear that your experience doesn't stretch terribly far; off the top of my head I can name several much less secure systems. Finding, fixing and announcing vulnerabilities is a good thing: by your measure a hugely exploited CMS with no fixes would be better!
Regarding you assertion that the rewrite engine cannot be disabled; this is just plain wrong. The Apache rewrite engine can be disabled without any problem. If you do this, then you won't enjoy clean URLs, instead you'll have URLs like www.somesite.com/index.php?q=some/path instead of www.somesite.com/some/path. Internally Drupal always works with the first form. However, the rewrite engine is a widely used Apache module - with perhaps millions(?) of sites using it. It may very well have exploits - just as any software may - but it is trusted by lots of users.
Followsymlinks can be disabled too. It's required for rewriting and for one form of upload. Drupal works without problems without it. However, there's nothing inherently insecure in symlinks, and the default Drupal directory layout does not symlink to outside of the install tree.
Database load. I note that your assertion about load is without any reference to figures. I'm not certain which CMS you think is well written. However I'll note that there is a general problem with CMSs which are designed to be easily extensible: tightly integrated system usually use a single SQL statement to retrieve data - the designer knows all the constraints at design-time. A loosely coupled system is usually not able to do this: the designer has little idea of what will be present at run time. So it's in the nature of most loosely coupled system to run one query or more for each additional module. Drupal uses a loosely coupled callback orientated architecture. This means its very easy to extend. However the downside is that each module will usually include extra tables. Drupal is fairly smart about loading this extra data, but beyond that, to counteract the tendency for growth in queries, Drupal has a caching subsystem that is active in several layers. For anonymous users, Drupal only runs a few queries which determine where in the cache the data sits, and returns it.
Perhaps you'd like to elaborate with some firm figures and an example of a CMS that in your opinion does it right.
Regarding PHP security. Again - have you any firm facts to show that PHP is inherently less secure than any other language? The consensus in security circles is that openness is better for security. *You* are able to download the PHP source code and contribute patches. If you know of a security issue, I'd urge you to help fix it. Or is this opinion without facts to back it up?
Again, I'd be interested to know which CMS you do recommend to the person in the street. I would not at the moment recommend Drupal for most brochureware sites, though it is capable of brochureware, however for sites in excess of about 100 pages, for sites where there is a heavy community aspect, and for sites which hope to change and grow, Drupal is an excellent choice.
I think you are misinformed. Morpheus seemed to be targeted at a range of software, including Joomla, but not Drupal: as far as I can see, none of the URL's it scanned are Drupal-based. See http://zeroq.kulando.de/post/2008/08/20/morfeus-fucking-scanner for example, but there are others out there.
In fact, Drupal has an excellent history of security. We find holes, fix them and issue patches. There is a security mailing list that anyone can sign up to. You will receive mail on the latest security fixes. Your Drupal installation will tell you when components are out of date, and when there are security updates. It will also email you on a regular basis if you don't care to look at your status, or ignore the status message at the top of the page when you log in as an administrator. Drupal will not download and install components without human intervention: components require manual installation.
Just like any software, I'm certain that Drupal has as yet undiscovered exploits. What's important is whether they are found and fixed, and we have a good track record of doing this.
Microsoft have derived a stack of publicity from the Sensecam and lifeblogging - it's made them look like a terrific company. I think this PR needs some counter-balance: Microsoft made Lyndsey Williams, the inventor of the Sensecam, redundant. Possibly not the best way to reward someone who was responsible for millions of dollars of positive PR; you don't get rid of the people who are doing brilliant work if you plan on delivering brilliant products in the future. But this has probably been a good thing for the rest of us; Ms Williams is prolific in bringing new devices to prototype and beyond. Her site shows her recent work, including the Sensebulb - a device for non-intrusive monitoring of elderly people who live alone. It can detect unusual situations and alert friends and relatives. This would have saved the lives of two people I knew. She has a stack of other interesting projects on the go too. Her site is well worth reading.
Disclosure: I know Ms Williams and take the opportunity to promote her work whenever I can. I'm not paid for this: I'm not in PR.