Sorry for the double-post... I did think of something else regarding Erlang.
In Java, you get high-level primitives that let you do crazy things with threads, so long as you remember to synchronize everything -- if you want to avoid locking, you have to build something else yourself. In Erlang, you get processes which are arguably higher-level, but isolated from each other -- if you want to use any sort of shared state in Erlang, you have to build it yourself.
That's more or less what I'm saying here -- if you want to join, or transact across every record in your database, SQL is the way to go. But you almost never want to do that, and even if you did, it'd be a means to an end which is probably as easily realized in a fundamentally different way.
Then I should rephrase: Are you really trying to say that a relational database is the only possible way, or the highest possible way, to abstract database operations?
I think you misunderstood my application of "abstract". They are on the lower levels of conceptual abstraction - they expose the user to concepts that are more primitive than relational ones.
No, I understood what you meant, I just disagree that RDBMS is higher-level in any absolute sense. For example:
in a proper RDBMS (SQL to not), transaction are magic - they "just work"... Or, say, joins - again, in a RDBMS, you specify the join, no matter how complicated in terms of what you want to get, and let the query analyzer figure out how to get there...
...if you're on a single machine with zero hardware issues.
Beyond that, you have to think about it, and plan, and do things like shard your data -- which kills both joins and transactions across data on multiple shards.
As an example, Google App Engine does support transactions. They're different, sure, but they exist -- they optimistically-lock, and they apply only to a single entity group at a time, but they are fully ACID within those limitations.
Now, let's talk about joins. You're in the traditional RDBMS mindset, which means joins are important to you, all the time. It's pretty much the only way you can get the database to talk about any structure more complex than a single record. Say I want to add tagging to a given record type -- in an RDBMS, the obvious way to do that is to have at least two tables, one table full of tags, and another representing the relationship between a given tag and a given record. I'm also going to need some deep voodoo to query that properly.
Contrast to, say, App Engine again -- arrays are natively supported, and can be queried on. I can simply create a 'tags' field on the record and fill it with a list of tags, and then I can query for which records have a given tag -- and the database natively understands this.
Or say I want to store a complex data structure -- let's say a cached RSS feed. This is going to be either an absurdly complex set of tables in SQL, or maybe I'll just dump the raw XML as a text field -- but then how would I query it? Or I could simply convert that XML to JSON (or, hell, leave it as a string) and dump it into CouchDB, in which any and all queries beyond get-by-key are map/reduce views. In this case, if I wanted to query by something like author, I could create a JavaScript map function which pulls that out. And Riak is shaping up to be better at this kind of thing than CouchDB -- in Riak, I'd probably store the raw XML, unmodified, and query it using any language I want.
You can sort of emulate this in SQL -- create a table with a text field to store the raw RSS dump, then anytime you come up with a new kind of query, you add a column, then manually go through every record in your database to initialize the default value of that column, since there's no way you're going to be parsing and manipulating XML in a SQL UPDATE statement.
But the key word there? Emulate. In other words, CouchDB or Riak give you higher-level concepts which natively support map/reduce, which have it Just Work, even scaling out to a cluster, while SQL is going to make you build it yourself.
It's different primitives, but they aren't automatically lower-level just because they're not the ones you're used to.
So again, SQL has its place. It certainly makes sense if you have a ton of data to log and analyze later, and you know it's in a fixed schema, you don't know how you're going to query it, but you do want those queries to run quickly. It also makes sense if you just need a quick place to store stuff on one machine -- SQLite is amazing. But neither SQL itself nor RDBMS in general is the end-all of databases, and they certainly aren't the highest-level databases.
Some of these browsers in their most modern form still can't pass CSS ACID tests.
Few of them, and few tests. It seems to be mostly IE that's the problem here.
Also, keep in mind that the ACID tests are deliberately designed to expose known bugs in browsers and embarrass those browser manufacturers into fixing them. As long as there's a new ACID test, there's going to be at least one browser that doesn't score 100% on it, and that's by design of the ACID tests. Yes, every browser should try to score 100%, but if a browser doesn't, that just means they have work to do, it doesn't mean the standard is suddenly broken.
After all, does Adobe Flash follow what their docs say 100% of the time? If so, that really would be extraordinary. That's one of the advantages of competing implementations -- you can show an example of something implementing the spec correctly to show that it can be done, and if the implementation is open source, parts of it may be absorbed into other implementations.
It is currently the case that you can build a web app which will run well on Firefox, Safari, Opera, Konqueror, Epiphany, Camino, Chrome, iCab, basically everything except IE. If we could ditch IE and Flash, we'd pretty much have what we want from HTML right now.
In contrast, when you're comparing SQL to "NoSQL", the latter is less abstracted.
Well, depends how you define NoSQL. The idea behind it is to use anything but SQL -- do you seriously mean to say that SQL is the only possible way to abstract database operations?
Many of the new NoSQL databases use REST APIs. Some of them use Map/Reduce. Some of them use common abstraction layers like Lucene. I would argue that in most cases, they are more abstract than SQL. When we're talking about a system where I can design my application without knowledge of where data will physically be stored, yes, that's more abstract than a system where I must manually shard my data and send the SQL query to an appropriate server.
So no, I would argue it's much more like going from Java to Erlang, or vice-versa. Erlang is more abstract with respect to threading, and less abstract with respect to string manipulation.
ActiveRecord? Definitely. Rails as a whole? You might consider replacing it with another Ruby framework, but the same ideas are going to apply. Remember how Rails and Merb are merging? Merb tends to be ORM-agnostic, but the recommended Merb stack suggested DataMapper, which does support a few NoSQL databases.
Even if you needed a different ORM per NoSQL database, it wouldn't marginalize Rails as a whole, but that simply isn't the case. Just use DataMapper, then plug in the flavor of the day.
I think some developers keep looking for the holy grail. Some magical solution that will turn development from punching in code, to Star Trek: "Computer do my job for me please".
While true, that doesn't invalidate actual progress.
deep down, with all the frameworks and generators, if you want your code to do what you want it to do, you are still writing out if statements a lot.
Sure, no one claimed otherwise. The point is that I'd much rather be writing the if statements that have something to do with what I want my code to do, rather than the if statements that have to do with how to manage a session, or how to handle one URL vs another, or how to sanitize this particular query for the database.
As an example, you bash Rails -- ActiveRecord is hardly the only ORM capable of this, but it's possible to build a Rails app without writing a single line of SQL, or opening yourself up to a hint of a possibility of a SQL injection attack. Similarly, writing in any language higher-level than, say, C or C++, will generally save you from the possibility of a buffer overrun.
Now, is that going to take out all the tedium? No, it might not even make my job less tedious at all. What it will do is mean I'm more productive -- I'm doing more in the same amount of time. That's usually valuable.
yes, OO and such also belong to this. Not the concept themselves, but the way most people talk about. OO means code re-use right?
It's a way of organizing your program, and it actually does have a lot to do with code re-use. For example:
Apparently you can't re-use functions. No way, no how. NEXT!
One core goal of OO is to encapsulate and hide away the details of a given concept and expose only a simple, re-usable interface. Now, anything you can do in one Turing-complete language, you can do in another, but it can be a considerably different level of pain and complexity. Consider basic patterns like iterators in a language like C -- yes, you can do them, but does it really make sense?
Just check, how many times do you get one of those managers wannabe introducing something they read in a magazine because it promises that you don't need to write another line of code ever!
Well, I tend to advocate Rails, web apps, NoSQL, REST, higher-level languages, metaprogramming, and so on. I do this partly because of what I can do with those, and partly because they make my job easier, and thus more productive.
I never do them because I expect to stop having to write code.
SQL isn't the problem, it's a tool. Bad programmers are the problem.
You could say the same about assembly language. You could also say the same about threads, and dismiss things like functional programming and the actor model as fads.
I'll give you a simple example: Given a big transactional SQL database, if you want it to scale to more than a few machines, you're going to want to shard it. That's going to be a ton of manual work, figuring out what you can shard, what keys to shard it on, adjusting it later on the fly to ensure that each DB server has exactly what it can handle in terms of data and load, and so on. You might be able to write software to do this for you, but that software is going to be fairly tightly coupled to your data model and your app.
It's possible I'm missing something there, and it's possible there's an easier way to do it, but it seems like every way to scale SQL has similar tradeoffs. Put a proxy in front of your DB cluster, giving the impression of a single database out of those shards? Your app is now not talking directly to the database, and certain queries won't be supported, and certain other queries will be slow or unreliable.
The database I'm working with now is Google AppEngine. It's pretty much natively sharded, and the tradeoffs are understood up front -- you can only transact over entities in the same group, but if your app is built up front to define entity groups appropriately, Google can physically shard them for you. It's a similar advantage to using Erlang for concurrency -- you probably won't be running your Erlang app on a machine with several thousand cores, but if you've got several thousand concurrent actors, it will trivially scale to anything in between.
Like Erlang, it's also not a magic bullet. I still use SQL in things like SQLite, because it's the best tool for the job.
If you'd mentioned the fact that this appears to be written in Java, you might have a point. But despite this, and the fact that it's in userland, they seem to be getting pretty decent performance out of it.
And keep in mind, all of this is to support reducing the amount of storage required on a hard disk, and it's a fairly large programming effort to do so. Seems like this entire project is just the opposite of what you claim -- it's software types doing extra work so they can spend less on storage.
...even the amount of space "wasted" by a Word doc is insignificant, especially if the university is willing to outsource their email. Mine has a Gmail option. Despite the number of random.docs, PDFs, and other crap that gets sent around, not to mention all the HTML mail, I've projected that if I stay on as a grad student, I'd still barely use half the storage I have there if I preserve every single message, including most of the spam.
If you make your living exchanging your talents for money, so are you. So what is your point?
I suppose it's the difference between a Companion and a whore.
I tend to choose my employers, and I certainly don't express opinions simply because I'm paid to. That doesn't mean I'll never express an opinion my employer tells me to, only that one of the following is true:
It's actually my opinion, too.
I'll present clear disclaimers that these are the opinions of my employer, and explain where mine differ.
I'll refuse to do it, and if it becomes an issue, I'll look for a new job.
Now, I don't know which De Icaza is at this point -- whether he's a Mono developer because it's his passion, and whether he really believes it's a viable platform (especially given Microsoft's patent minefield), or whether he's doing all this simply because he's got a fat paycheck from Microsoft. But that's the difference.
HDD's are heavier and more fragile then LTO 4 tapes, also more vulnerable to static.
So not as portable, I know, but nonetheless portable. I have an external hard drive which sat in my backpack for several months, and I carried that backpack everywhere -- the drive is still good.
(we are talking about your entire business if your office burns down, you need to take care).
Yes, that's true, but it's also something you'd presumably be monitoring. The idea of a backup is to have at least two copies, so that if one copy fails, you can reconstruct from the other. If the "portable" hard drives fail, you buy a new one and copy the data from the original source -- the live system.
hosting is cheap.
For 6 TB?
Depends how you do it. With a colo, I imagine it would be -- just make sure it fits in the box.
You assert that tapes require a enterprise level service to be properly stored (well actually it can be done for A$20 a week for a fireproof lock box).
I suppose I'm making assumptions about the physical size of the tapes, and the need to have them organized.
I'm not assuming "enterprise-level" -- those are your words, not mine. I'm factoring the labor of actually physically carrying the tapes and storing them, even if it is one person carrying them to a lock box.
Then you blatantly state that enterprise level hardware for your scenario is overkill.
For the drives? Yes. You'd want to put them in some rented corner of an enterprise-level datacenter, but you don't necessarily need enterprise-level disks and controllers, even to move a terabyte (or six) a week.
Linux is free. So is Solaris, for that matter.
My time however is not. Compared to my time a Windows license is almost free.
Irrelevant, unless you're claiming it's going to necessarily take less of your time to do it with Windows.
A$2000 for a box, with controller and 1 disk. Add another A$150 per 1.5 TB disk. Unless your suggesting we should use the receptionists machine as a backup system. This is where your system becomes highly unreliable. Not enterprise ready at all.
"Enterprise-ready" is a bit of a buzzword. Google runs largely on consumer hardware. I'm not suggesting you literally take the receptionist's machine...
And even with the exchange rate, sorry, 1.5 TB is not A$150. Try more like $110. Not "enterprise-ready", you say? That's another reason to use something like Solaris and ZFS -- it doesn't have to be. Even RAID stands for "Redundant Array of Inexpensive Disks."
Here is the problem, you're treating your backups as a hot system, not a cold system.
I'm not sure what you mean by that.
I've lost count of how many times someone has asked me for a file they deleted off the server months ago
So use hard disks and something like... oh... last time I set anything like this up, I used BackupPC. At this point, though, even a straight copy would benefit from block-level dedup. Either way, the result is that you can store many versions -- daily backups for a week, weekly backups for a month, monthly backups for a year...
If they deleted it off the server months ago, there's probably a version of it somewhere. Maybe not the most recent version, but a version.
do you intend to keep adding disks to this array?
Possibly, but that's not required by the above. If it were (assuming your massive numbers of changes), there's always the possibility of taking a snapshot of the system to a disk or five (using something simple, like a tarball split across those disks) and unplug those disks for awhile.
LTO-4 are 800 GB uncompressed, 1600 GB compressed, I get a compression ratio of about 1:1.2 - 1:1.4 So I fit between 1 and 1.2 TB on each tape, some data compresses up to 1:1.8. But that's not the advantage of tapes.
Um... duh?
In fact, I would much sooner turn off the hardware compression (assuming that's what they're doing) and do it in software -- probably get a better ratio, or tune it based on how much CPU I feel like burning on it.
Hard drives are extremely fragile and not portable.
More fragile than a tape, maybe, but they're certainly portable.
With a tape you can destroy the casing, even cut the actual tape and you can still recover from it.
You can do the same with a hard drive, it just costs more.
Maintaining a large RAID array on-site doesn't help if your building burns down and maintaining one off site is quite a bit more expensive then a 24 LTO-4 tape loader.
The former is obvious, but I'm not sure about the latter -- hosting is cheap. How long would it take for the tape to pay for itself? And that's assuming it's cheaper to hire someone to physically move the tapes offsite.
You fail to take into account the cost of actually buying and maintaining a redundant RAID array.
Not really, I just assumed it would be done intelligently. In other words:
An EMC SAN will cost upwards of US10K, an IBM ServerRAID 8i PCI-X card will cost about US$800,
Overkill.
even software raid requires a box with an OS to run it
Linux is free. So is Solaris, for that matter. And a box can be had for less than $800 -- assuming you need a dedicated box that will do nothing else, which again seems like the dumb way to do it.
Even if you're mad enough to run a critical business system on consumer hardware a decent PC will set you back A$1500 for a 6 disk system, A$2000 for a 7 disk system as most consumer boards only have 6 SATA slots.
Are you factoring in the cost of the drives themselves? If so, I don't see how a drive and an extra controller (even PCI will do) is an extra $500.
Aside from the risk of running consumer disks (3yr warranty) compared to enterprise level tapes (lifetime warranty from IBM and Tandberg)
Hence the RAID -- and chances are, by the time any of them fail, you'll be able to replace them with something much larger for the same price. If you're running ZFS, it doesn't even have to be the same size.
Then again, why are we doing RAID? This is backup. Do checksumming and call it a day -- if a file is lost in the backup, we presumably have a live copy anyway.
there is little difference in the set up costs
Yes -- though it gets much cheaper if you pay someone else to do that. For example, Amazon S3.
On-line backups are difficult when your data upload rate is limited, I back up 6 TB weekly plus daily differentials which are about 1 TB per week, one set of tapes goes off-site each month.
Data upload rate, and in Australia -- ah. Now it does start to make sense -- so you're right, you'd need to build the offsite facility yourself, and you'd probably need to physically transport anyway. Given that, tapes start to make sense.
I am a bit skeptical of your daily diffs, though. Where does that amount of data churn come from?
In other words, 800 gig, so they assume 1.6 terabytes will fit once compressed. Hey, I can do the same thing for a hard drive, count that 1.5 TB as 3 instead. But let's stick to actual physical size...
1.5 TB hard drives run about $99 on Newegg, or about an extra 3 cents per gig, or about $2.35 per drive. Unless my math is off, that's well over a thousand tapes to pay for the drive.
In other words, this starts to pay for itself when you have to store over 800 terabytes (and probably closer to 1 PB) of data. At that size, yes, transportation and storage starts to matter, not to mention maintenance and basic operations -- maybe you're right, but at that point, I have no idea whether the human to swap tapes and physically move them (or the human in the remote tape library to handle them after sending data over the Internet) or to run that many extra servers (where do all those hard disks go? Do they spin down when not in use?)
Encrypt it to begin with, especially if you've got a home media server. Store the key on a separate device -- say, a USB key. If you need to nuke everything, simply take some thermite to that USB key.
Probably overkill, but the point is, it's a lot easier to destroy a key than it is to destroy the data.
Hard drives are a terabyte for under $100. Tapes even close to a terabyte seem to require drives in the multiple thousands of dollars, and I can't find prices on the media. Even if you factor in some sort of hot-swap chassis and a server to run it on, I think hard drives win.
Tell me that's not more "maintainable" than hundreds of lines of XML config files.
Now that I think about it, those hundreds of lines of XML config files are going to have to be interpreted by something to turn them into method calls -- so again, it's going to result in metaprogramming, unless you define your schema again by repeating all those methods and properties inside your model source as method and property definitions.
That might help motivation... I'm not sure it would help actual productivity.
For example, I agree that people shouldn't micromanage programmers, and should examine the output based on how effective, readable, maintainable it is, not based on how well it conforms to the way things have been done in the past. On the other hand, coding standards are there for a reason -- not everything needs to be unit tested, but if you've got a new guy who just doesn't like writing tests, maybe even has some philosophical objections to them, that could be a problem.
I mean, it'd be nice if you could just give him a problem to solve on his own, but chances are, you're all working on a big system together, and his lack of tests affects everyone else's ability to work with the entire codebase, not just his part.
So it's not that I disagree with ever impinging on someone's autonomy, but I definitely disagree with doing it for the wrong reasons. In this case, it simply does not result in better code to ban metaprogramming, and if anything, will result in driving away the better programmers.
I'd say similar things about complexity. I like working on complex, intricate, and elegant systems, but in the real world, what's often needed is much more mundane. In fact, metaprogramming can make things much simpler, and that's a win. I don't think it's a huge problem for motivation, in that if you can solve the same problem with a tenth of the complexity in a tenth of the time, you can solve ten problems instead of just one, which is probably a lot more satisfying. (Imagine trying to defeat one huge insanely complex boss for an entire game -- not fun.)
In fact, that probably helps with feedback -- and I do agree, feedback is good.
You might have implied that with "if it can be avoided", but it can always be avoided -- the question is whether it leads to a better or worse result.
What do you mean #2 is unlikely? Quite clearly, since MY religion is the Great Holy Truth and the One True Way it is completely inconceivable that aliens could believe anything else!
I realize you're being sarcastic, but I'm amazed that people think this way, and don't take the two seconds of critical thinking required (or maybe thirty seconds of research) to discover that most people on this planet don't share their religion.
Re:These techniques are horrid for maintainability
on
Metaprogramming Ruby
·
· Score: 5, Insightful
These "metaprogramming" techniques, regardless of whether they're done in Ruby or JavaScript or Lua or some other language, cause nothing but problems when it comes to maintainability.
If done poorly, sure.
Sure, they might help the programmer write the code in slightly less time, but then they make it damn near impossible to debug later on.
Then your debugging tools suck, or the metaprogramming was abused. Banning it isn't like banning goto, it's like banning exception handling. Just because you can abuse it and ignore all errors doesn't mean you should never be able to intercept an exception.
Errors are missed because they only happen in obscure cases at runtime,
Can you give an example of such an "obscure" case?
I can see errors being missed because they only happen at runtime, but guess what? Them's the breaks. If you don't like it, you need to stop using Ruby or JavaScript or Lua, and stick to languages like Java, and make sure you always used checked exceptions, and you still won't catch all of them.
Runtime errors happen. That's what unit tests are for.
Even working code becomes unnecessarily complex,
Metaprogramming can make code much shorter, more readable, and more maintainable. If there's some additional complexity at runtime, so be it.
When I catch my developers using these techniques, we have a nice little chat about writing maintainable code, and then their code gets reworked.
If I was working for you (and I'm glad I'm not), I'd look forward to educating you on maintainability.
Let's look at an example: attr_accessor. It wouldn't be Ruby without it. You probably allow it, and it probably makes sense to you to allow it. It's also an example of metaprogramming -- while it's built into the language, and probably hand-tweaked in C for speed, it really makes much more sense once you understand that logically, attr_accessor itself could have been written using define_method.
How about another example: Any decent ORM. Either it's going to use metaprogramming to define your schema (like DataMapper does with 'property'), or it's going to use metaprogramming to expose your existing schema (like Rails does by reading your database). Tell me that's not more "maintainable" than hundreds of lines of XML config files.
Or I'll use a real-world example from the appengine-jruby project:
.
def parent_property(name)
define_method("#{name}_id") do
k = key.first
k.kind_of?(AppEngine::Datastore::Key) && k.parent
end
belongs_to_entity(name, false)
end
And in actual code:
class Foo ...
parent_property:parent end
That defines a method 'parent_id' which returns the current id of the 'parent' of the current record (an Appengine-specific concept), and a method 'parent' which returns the actual entity of the parent. But notice I passed the symbol:parent explicitly -- I could just as easily have passed something else. (Why would I want to? Because maybe I need a parent method for some other reason, but I don't want to lose this functionality.)
Metaprogramming techniques can be abused. So can anything. Goto is harmful, but sometimes it's useful to be able to return a value from the middle of several nested loops without having to terminate each explicitly. Perl can look like line noise, and while it won't ever really be beautiful, you can write maintainable, readable Perl. You can write COBOL in any language.
So yes, it can be abused, but it's also an extremely powerful technique, and you are crippling your programmers and taking a hit on both productivity and maintainability by banning it.
People who believe in tarot cards, or any other form of superstition or mysticism, are not atheists.
That's a semantic argument, and not one I'm particularly looking forward to, but technically speaking, yes they are. Again, the word "atheist" simply means, not a theist -- that is, not one who believes in a god.
Now, you'd be right to say they're probably not skeptics or rationalists, and they are by definition not naturalists, but to say they're "not atheist" is akin to a protestant claiming catholics are "not Christian" or vice versa.
I don't know where you get that definition from, but it isn't accurate.
That's not correct, either, as atheist simply means one who doesn't believe.
Atheism can be either the rejection of theism, or the position that deities do not exist.
That's about right.
atheism - the doctrine or belief that there is no God
Again, I reject this definition. However, the same site also offers the correct one:
atheistic - rejecting any belief in gods
And again, note that in either case, we're still talking about gods specifically.
One who rejects or is ignorant of theism.
And so on.
The only two definitions I see there, and the only two I've heard in general, can be summarized like this:
One who denies that any gods exist.
One who does not believe that any gods exist.
Tarot cards do not require deities, nor do ghosts, UFOs, or for that matter, perpetual motion machines. There are any number of things which are unscientific, possibly paranoid, dogmatic, and foolish, without ever claiming to be a religion, and certainly without being concerned with a deity in particular.
I'll agree that atheism, by itself, is not necessarily rational. But most people who admit to being atheist are rational.
Most, yes. However, if you want any meaningful statistics, you have to also include agnostics, and some of them include those who have no particular religion. Not all atheists are "out of the closet", so to speak, which is one reason I'm as vocal as I am -- to reassure those who aren't that it's ok to be an atheist, and that there are others who think the way they do -- some seem to live with the idea that there's something wrong with them, or that they should be quiet and say they're just not particularly religious...
If it comes down to talking to aliens, tho, I suspect that scientific rationalism - and probably anarchic scientific rationalism - will win the day.
I would emphasize the science and the rationalism, and while I don't know that anarchy is best, I would say someone without religious or political affiliation would be best.
Slashdot Mirror
Sorry for the double-post... I did think of something else regarding Erlang.
In Java, you get high-level primitives that let you do crazy things with threads, so long as you remember to synchronize everything -- if you want to avoid locking, you have to build something else yourself. In Erlang, you get processes which are arguably higher-level, but isolated from each other -- if you want to use any sort of shared state in Erlang, you have to build it yourself.
That's more or less what I'm saying here -- if you want to join, or transact across every record in your database, SQL is the way to go. But you almost never want to do that, and even if you did, it'd be a means to an end which is probably as easily realized in a fundamentally different way.
The idea is to "use anything but relational DB".
Then I should rephrase: Are you really trying to say that a relational database is the only possible way, or the highest possible way, to abstract database operations?
I think you misunderstood my application of "abstract". They are on the lower levels of conceptual abstraction - they expose the user to concepts that are more primitive than relational ones.
No, I understood what you meant, I just disagree that RDBMS is higher-level in any absolute sense. For example:
in a proper RDBMS (SQL to not), transaction are magic - they "just work"... Or, say, joins - again, in a RDBMS, you specify the join, no matter how complicated in terms of what you want to get, and let the query analyzer figure out how to get there...
...if you're on a single machine with zero hardware issues.
Beyond that, you have to think about it, and plan, and do things like shard your data -- which kills both joins and transactions across data on multiple shards.
As an example, Google App Engine does support transactions. They're different, sure, but they exist -- they optimistically-lock, and they apply only to a single entity group at a time, but they are fully ACID within those limitations.
Now, let's talk about joins. You're in the traditional RDBMS mindset, which means joins are important to you, all the time. It's pretty much the only way you can get the database to talk about any structure more complex than a single record. Say I want to add tagging to a given record type -- in an RDBMS, the obvious way to do that is to have at least two tables, one table full of tags, and another representing the relationship between a given tag and a given record. I'm also going to need some deep voodoo to query that properly.
Contrast to, say, App Engine again -- arrays are natively supported, and can be queried on. I can simply create a 'tags' field on the record and fill it with a list of tags, and then I can query for which records have a given tag -- and the database natively understands this.
Or say I want to store a complex data structure -- let's say a cached RSS feed. This is going to be either an absurdly complex set of tables in SQL, or maybe I'll just dump the raw XML as a text field -- but then how would I query it? Or I could simply convert that XML to JSON (or, hell, leave it as a string) and dump it into CouchDB, in which any and all queries beyond get-by-key are map/reduce views. In this case, if I wanted to query by something like author, I could create a JavaScript map function which pulls that out. And Riak is shaping up to be better at this kind of thing than CouchDB -- in Riak, I'd probably store the raw XML, unmodified, and query it using any language I want.
You can sort of emulate this in SQL -- create a table with a text field to store the raw RSS dump, then anytime you come up with a new kind of query, you add a column, then manually go through every record in your database to initialize the default value of that column, since there's no way you're going to be parsing and manipulating XML in a SQL UPDATE statement.
But the key word there? Emulate. In other words, CouchDB or Riak give you higher-level concepts which natively support map/reduce, which have it Just Work, even scaling out to a cluster, while SQL is going to make you build it yourself.
It's different primitives, but they aren't automatically lower-level just because they're not the ones you're used to.
So again, SQL has its place. It certainly makes sense if you have a ton of data to log and analyze later, and you know it's in a fixed schema, you don't know how you're going to query it, but you do want those queries to run quickly. It also makes sense if you just need a quick place to store stuff on one machine -- SQLite is amazing. But neither SQL itself nor RDBMS in general is the end-all of databases, and they certainly aren't the highest-level databases.
Some of these browsers in their most modern form still can't pass CSS ACID tests.
Few of them, and few tests. It seems to be mostly IE that's the problem here.
Also, keep in mind that the ACID tests are deliberately designed to expose known bugs in browsers and embarrass those browser manufacturers into fixing them. As long as there's a new ACID test, there's going to be at least one browser that doesn't score 100% on it, and that's by design of the ACID tests. Yes, every browser should try to score 100%, but if a browser doesn't, that just means they have work to do, it doesn't mean the standard is suddenly broken.
After all, does Adobe Flash follow what their docs say 100% of the time? If so, that really would be extraordinary. That's one of the advantages of competing implementations -- you can show an example of something implementing the spec correctly to show that it can be done, and if the implementation is open source, parts of it may be absorbed into other implementations.
It is currently the case that you can build a web app which will run well on Firefox, Safari, Opera, Konqueror, Epiphany, Camino, Chrome, iCab, basically everything except IE. If we could ditch IE and Flash, we'd pretty much have what we want from HTML right now.
In contrast, when you're comparing SQL to "NoSQL", the latter is less abstracted.
Well, depends how you define NoSQL. The idea behind it is to use anything but SQL -- do you seriously mean to say that SQL is the only possible way to abstract database operations?
Many of the new NoSQL databases use REST APIs. Some of them use Map/Reduce. Some of them use common abstraction layers like Lucene. I would argue that in most cases, they are more abstract than SQL. When we're talking about a system where I can design my application without knowledge of where data will physically be stored, yes, that's more abstract than a system where I must manually shard my data and send the SQL query to an appropriate server.
So no, I would argue it's much more like going from Java to Erlang, or vice-versa. Erlang is more abstract with respect to threading, and less abstract with respect to string manipulation.
Bullshit.
ActiveRecord? Definitely. Rails as a whole? You might consider replacing it with another Ruby framework, but the same ideas are going to apply. Remember how Rails and Merb are merging? Merb tends to be ORM-agnostic, but the recommended Merb stack suggested DataMapper, which does support a few NoSQL databases.
Even if you needed a different ORM per NoSQL database, it wouldn't marginalize Rails as a whole, but that simply isn't the case. Just use DataMapper, then plug in the flavor of the day.
As an example, Rails (and DataMapper) run on Google App Engine.
I think some developers keep looking for the holy grail. Some magical solution that will turn development from punching in code, to Star Trek: "Computer do my job for me please".
While true, that doesn't invalidate actual progress.
deep down, with all the frameworks and generators, if you want your code to do what you want it to do, you are still writing out if statements a lot.
Sure, no one claimed otherwise. The point is that I'd much rather be writing the if statements that have something to do with what I want my code to do, rather than the if statements that have to do with how to manage a session, or how to handle one URL vs another, or how to sanitize this particular query for the database.
As an example, you bash Rails -- ActiveRecord is hardly the only ORM capable of this, but it's possible to build a Rails app without writing a single line of SQL, or opening yourself up to a hint of a possibility of a SQL injection attack. Similarly, writing in any language higher-level than, say, C or C++, will generally save you from the possibility of a buffer overrun.
Now, is that going to take out all the tedium? No, it might not even make my job less tedious at all. What it will do is mean I'm more productive -- I'm doing more in the same amount of time. That's usually valuable.
yes, OO and such also belong to this. Not the concept themselves, but the way most people talk about. OO means code re-use right?
It's a way of organizing your program, and it actually does have a lot to do with code re-use. For example:
Apparently you can't re-use functions. No way, no how. NEXT!
One core goal of OO is to encapsulate and hide away the details of a given concept and expose only a simple, re-usable interface. Now, anything you can do in one Turing-complete language, you can do in another, but it can be a considerably different level of pain and complexity. Consider basic patterns like iterators in a language like C -- yes, you can do them, but does it really make sense?
Just check, how many times do you get one of those managers wannabe introducing something they read in a magazine because it promises that you don't need to write another line of code ever!
Well, I tend to advocate Rails, web apps, NoSQL, REST, higher-level languages, metaprogramming, and so on. I do this partly because of what I can do with those, and partly because they make my job easier, and thus more productive.
I never do them because I expect to stop having to write code.
I run a site of 1M pages daily (1/3-slashdot according to Alexa) with just a single system with 2x Xeon E5420, Django/PostgreSQL at 10% load.
Good for you -- that says nothing about how much you're actually doing for each page.
just stick with a standard ORM
As a rule, I do. I use DataMapper in Ruby. It's just that DataMapper has pluggable backends, some for SQL databases, some for more exotic things.
SQL isn't the problem, it's a tool. Bad programmers are the problem.
You could say the same about assembly language. You could also say the same about threads, and dismiss things like functional programming and the actor model as fads.
I'll give you a simple example: Given a big transactional SQL database, if you want it to scale to more than a few machines, you're going to want to shard it. That's going to be a ton of manual work, figuring out what you can shard, what keys to shard it on, adjusting it later on the fly to ensure that each DB server has exactly what it can handle in terms of data and load, and so on. You might be able to write software to do this for you, but that software is going to be fairly tightly coupled to your data model and your app.
It's possible I'm missing something there, and it's possible there's an easier way to do it, but it seems like every way to scale SQL has similar tradeoffs. Put a proxy in front of your DB cluster, giving the impression of a single database out of those shards? Your app is now not talking directly to the database, and certain queries won't be supported, and certain other queries will be slow or unreliable.
The database I'm working with now is Google AppEngine. It's pretty much natively sharded, and the tradeoffs are understood up front -- you can only transact over entities in the same group, but if your app is built up front to define entity groups appropriately, Google can physically shard them for you. It's a similar advantage to using Erlang for concurrency -- you probably won't be running your Erlang app on a machine with several thousand cores, but if you've got several thousand concurrent actors, it will trivially scale to anything in between.
Like Erlang, it's also not a magic bullet. I still use SQL in things like SQLite, because it's the best tool for the job.
If you'd mentioned the fact that this appears to be written in Java, you might have a point. But despite this, and the fact that it's in userland, they seem to be getting pretty decent performance out of it.
And keep in mind, all of this is to support reducing the amount of storage required on a hard disk, and it's a fairly large programming effort to do so. Seems like this entire project is just the opposite of what you claim -- it's software types doing extra work so they can spend less on storage.
Well, just how repetitive is your porn collection?
...even the amount of space "wasted" by a Word doc is insignificant, especially if the university is willing to outsource their email. Mine has a Gmail option. Despite the number of random .docs, PDFs, and other crap that gets sent around, not to mention all the HTML mail, I've projected that if I stay on as a grad student, I'd still barely use half the storage I have there if I preserve every single message, including most of the spam.
Yeah, it'll be so nice having to travel around the world to deliver that scrap of "paper" to someone, instead of just hitting "send".
Good news everyone. Icaza is still a whore.
If you make your living exchanging your talents for money, so are you. So what is your point?
I suppose it's the difference between a Companion and a whore.
I tend to choose my employers, and I certainly don't express opinions simply because I'm paid to. That doesn't mean I'll never express an opinion my employer tells me to, only that one of the following is true:
Now, I don't know which De Icaza is at this point -- whether he's a Mono developer because it's his passion, and whether he really believes it's a viable platform (especially given Microsoft's patent minefield), or whether he's doing all this simply because he's got a fat paycheck from Microsoft. But that's the difference.
HDD's are heavier and more fragile then LTO 4 tapes, also more vulnerable to static.
So not as portable, I know, but nonetheless portable. I have an external hard drive which sat in my backpack for several months, and I carried that backpack everywhere -- the drive is still good.
(we are talking about your entire business if your office burns down, you need to take care).
Yes, that's true, but it's also something you'd presumably be monitoring. The idea of a backup is to have at least two copies, so that if one copy fails, you can reconstruct from the other. If the "portable" hard drives fail, you buy a new one and copy the data from the original source -- the live system.
hosting is cheap.
For 6 TB?
Depends how you do it. With a colo, I imagine it would be -- just make sure it fits in the box.
You assert that tapes require a enterprise level service to be properly stored (well actually it can be done for A$20 a week for a fireproof lock box).
I suppose I'm making assumptions about the physical size of the tapes, and the need to have them organized.
I'm not assuming "enterprise-level" -- those are your words, not mine. I'm factoring the labor of actually physically carrying the tapes and storing them, even if it is one person carrying them to a lock box.
Then you blatantly state that enterprise level hardware for your scenario is overkill.
For the drives? Yes. You'd want to put them in some rented corner of an enterprise-level datacenter, but you don't necessarily need enterprise-level disks and controllers, even to move a terabyte (or six) a week.
Linux is free. So is Solaris, for that matter.
My time however is not. Compared to my time a Windows license is almost free.
Irrelevant, unless you're claiming it's going to necessarily take less of your time to do it with Windows.
A$2000 for a box, with controller and 1 disk. Add another A$150 per 1.5 TB disk. Unless your suggesting we should use the receptionists machine as a backup system. This is where your system becomes highly unreliable. Not enterprise ready at all.
"Enterprise-ready" is a bit of a buzzword. Google runs largely on consumer hardware. I'm not suggesting you literally take the receptionist's machine...
And even with the exchange rate, sorry, 1.5 TB is not A$150. Try more like $110. Not "enterprise-ready", you say? That's another reason to use something like Solaris and ZFS -- it doesn't have to be. Even RAID stands for "Redundant Array of Inexpensive Disks."
Here is the problem, you're treating your backups as a hot system, not a cold system.
I'm not sure what you mean by that.
I've lost count of how many times someone has asked me for a file they deleted off the server months ago
So use hard disks and something like... oh... last time I set anything like this up, I used BackupPC. At this point, though, even a straight copy would benefit from block-level dedup. Either way, the result is that you can store many versions -- daily backups for a week, weekly backups for a month, monthly backups for a year...
If they deleted it off the server months ago, there's probably a version of it somewhere. Maybe not the most recent version, but a version.
do you intend to keep adding disks to this array?
Possibly, but that's not required by the above. If it were (assuming your massive numbers of changes), there's always the possibility of taking a snapshot of the system to a disk or five (using something simple, like a tarball split across those disks) and unplug those disks for awhile.
Further more, your ide
Comments about her being hot -- fine, not sexist.
Suggesting she should've been a model? I'm not sure if it's sexist, but seems a bit obnoxious. Maybe she did what she wanted -- followed her passion?
LTO-4 are 800 GB uncompressed, 1600 GB compressed, I get a compression ratio of about 1:1.2 - 1:1.4 So I fit between 1 and 1.2 TB on each tape, some data compresses up to 1:1.8. But that's not the advantage of tapes.
Um... duh?
In fact, I would much sooner turn off the hardware compression (assuming that's what they're doing) and do it in software -- probably get a better ratio, or tune it based on how much CPU I feel like burning on it.
Hard drives are extremely fragile and not portable.
More fragile than a tape, maybe, but they're certainly portable.
With a tape you can destroy the casing, even cut the actual tape and you can still recover from it.
You can do the same with a hard drive, it just costs more.
Maintaining a large RAID array on-site doesn't help if your building burns down and maintaining one off site is quite a bit more expensive then a 24 LTO-4 tape loader.
The former is obvious, but I'm not sure about the latter -- hosting is cheap. How long would it take for the tape to pay for itself? And that's assuming it's cheaper to hire someone to physically move the tapes offsite.
You fail to take into account the cost of actually buying and maintaining a redundant RAID array.
Not really, I just assumed it would be done intelligently. In other words:
An EMC SAN will cost upwards of US10K, an IBM ServerRAID 8i PCI-X card will cost about US$800,
Overkill.
even software raid requires a box with an OS to run it
Linux is free. So is Solaris, for that matter. And a box can be had for less than $800 -- assuming you need a dedicated box that will do nothing else, which again seems like the dumb way to do it.
Even if you're mad enough to run a critical business system on consumer hardware a decent PC will set you back A$1500 for a 6 disk system, A$2000 for a 7 disk system as most consumer boards only have 6 SATA slots.
Are you factoring in the cost of the drives themselves? If so, I don't see how a drive and an extra controller (even PCI will do) is an extra $500.
Aside from the risk of running consumer disks (3yr warranty) compared to enterprise level tapes (lifetime warranty from IBM and Tandberg)
Hence the RAID -- and chances are, by the time any of them fail, you'll be able to replace them with something much larger for the same price. If you're running ZFS, it doesn't even have to be the same size.
Then again, why are we doing RAID? This is backup. Do checksumming and call it a day -- if a file is lost in the backup, we presumably have a live copy anyway.
there is little difference in the set up costs
Yes -- though it gets much cheaper if you pay someone else to do that. For example, Amazon S3.
On-line backups are difficult when your data upload rate is limited, I back up 6 TB weekly plus daily differentials which are about 1 TB per week, one set of tapes goes off-site each month.
Data upload rate, and in Australia -- ah. Now it does start to make sense -- so you're right, you'd need to build the offsite facility yourself, and you'd probably need to physically transport anyway. Given that, tapes start to make sense.
I am a bit skeptical of your daily diffs, though. Where does that amount of data churn come from?
Citation needed.
Dell sells LTO-4 (800GB/1.6TB)
In other words, 800 gig, so they assume 1.6 terabytes will fit once compressed. Hey, I can do the same thing for a hard drive, count that 1.5 TB as 3 instead. But let's stick to actual physical size...
1.5 TB hard drives run about $99 on Newegg, or about an extra 3 cents per gig, or about $2.35 per drive. Unless my math is off, that's well over a thousand tapes to pay for the drive.
In other words, this starts to pay for itself when you have to store over 800 terabytes (and probably closer to 1 PB) of data. At that size, yes, transportation and storage starts to matter, not to mention maintenance and basic operations -- maybe you're right, but at that point, I have no idea whether the human to swap tapes and physically move them (or the human in the remote tape library to handle them after sending data over the Internet) or to run that many extra servers (where do all those hard disks go? Do they spin down when not in use?)
Encrypt it to begin with, especially if you've got a home media server. Store the key on a separate device -- say, a USB key. If you need to nuke everything, simply take some thermite to that USB key.
Probably overkill, but the point is, it's a lot easier to destroy a key than it is to destroy the data.
Hard drives are a terabyte for under $100. Tapes even close to a terabyte seem to require drives in the multiple thousands of dollars, and I can't find prices on the media. Even if you factor in some sort of hot-swap chassis and a server to run it on, I think hard drives win.
Tell me that's not more "maintainable" than hundreds of lines of XML config files.
Now that I think about it, those hundreds of lines of XML config files are going to have to be interpreted by something to turn them into method calls -- so again, it's going to result in metaprogramming, unless you define your schema again by repeating all those methods and properties inside your model source as method and property definitions.
That might help motivation... I'm not sure it would help actual productivity.
For example, I agree that people shouldn't micromanage programmers, and should examine the output based on how effective, readable, maintainable it is, not based on how well it conforms to the way things have been done in the past. On the other hand, coding standards are there for a reason -- not everything needs to be unit tested, but if you've got a new guy who just doesn't like writing tests, maybe even has some philosophical objections to them, that could be a problem.
I mean, it'd be nice if you could just give him a problem to solve on his own, but chances are, you're all working on a big system together, and his lack of tests affects everyone else's ability to work with the entire codebase, not just his part.
So it's not that I disagree with ever impinging on someone's autonomy, but I definitely disagree with doing it for the wrong reasons. In this case, it simply does not result in better code to ban metaprogramming, and if anything, will result in driving away the better programmers.
I'd say similar things about complexity. I like working on complex, intricate, and elegant systems, but in the real world, what's often needed is much more mundane. In fact, metaprogramming can make things much simpler, and that's a win. I don't think it's a huge problem for motivation, in that if you can solve the same problem with a tenth of the complexity in a tenth of the time, you can solve ten problems instead of just one, which is probably a lot more satisfying. (Imagine trying to defeat one huge insanely complex boss for an entire game -- not fun.)
In fact, that probably helps with feedback -- and I do agree, feedback is good.
You might have implied that with "if it can be avoided", but it can always be avoided -- the question is whether it leads to a better or worse result.
What do you mean #2 is unlikely? Quite clearly, since MY religion is the Great Holy Truth and the One True Way it is completely inconceivable that aliens could believe anything else!
I realize you're being sarcastic, but I'm amazed that people think this way, and don't take the two seconds of critical thinking required (or maybe thirty seconds of research) to discover that most people on this planet don't share their religion.
These "metaprogramming" techniques, regardless of whether they're done in Ruby or JavaScript or Lua or some other language, cause nothing but problems when it comes to maintainability.
If done poorly, sure.
Sure, they might help the programmer write the code in slightly less time, but then they make it damn near impossible to debug later on.
Then your debugging tools suck, or the metaprogramming was abused. Banning it isn't like banning goto, it's like banning exception handling. Just because you can abuse it and ignore all errors doesn't mean you should never be able to intercept an exception.
Errors are missed because they only happen in obscure cases at runtime,
Can you give an example of such an "obscure" case?
I can see errors being missed because they only happen at runtime, but guess what? Them's the breaks. If you don't like it, you need to stop using Ruby or JavaScript or Lua, and stick to languages like Java, and make sure you always used checked exceptions, and you still won't catch all of them.
Runtime errors happen. That's what unit tests are for.
Even working code becomes unnecessarily complex,
Metaprogramming can make code much shorter, more readable, and more maintainable. If there's some additional complexity at runtime, so be it.
When I catch my developers using these techniques, we have a nice little chat about writing maintainable code, and then their code gets reworked.
If I was working for you (and I'm glad I'm not), I'd look forward to educating you on maintainability.
Let's look at an example: attr_accessor. It wouldn't be Ruby without it. You probably allow it, and it probably makes sense to you to allow it. It's also an example of metaprogramming -- while it's built into the language, and probably hand-tweaked in C for speed, it really makes much more sense once you understand that logically, attr_accessor itself could have been written using define_method.
How about another example: Any decent ORM. Either it's going to use metaprogramming to define your schema (like DataMapper does with 'property'), or it's going to use metaprogramming to expose your existing schema (like Rails does by reading your database). Tell me that's not more "maintainable" than hundreds of lines of XML config files.
Or I'll use a real-world example from the appengine-jruby project:
And in actual code:
That defines a method 'parent_id' which returns the current id of the 'parent' of the current record (an Appengine-specific concept), and a method 'parent' which returns the actual entity of the parent. But notice I passed the symbol :parent explicitly -- I could just as easily have passed something else. (Why would I want to? Because maybe I need a parent method for some other reason, but I don't want to lose this functionality.)
Metaprogramming techniques can be abused. So can anything. Goto is harmful, but sometimes it's useful to be able to return a value from the middle of several nested loops without having to terminate each explicitly. Perl can look like line noise, and while it won't ever really be beautiful, you can write maintainable, readable Perl. You can write COBOL in any language.
So yes, it can be abused, but it's also an extremely powerful technique, and you are crippling your programmers and taking a hit on both productivity and maintainability by banning it.
People who believe in tarot cards, or any other form of superstition or mysticism, are not atheists.
That's a semantic argument, and not one I'm particularly looking forward to, but technically speaking, yes they are. Again, the word "atheist" simply means, not a theist -- that is, not one who believes in a god.
Now, you'd be right to say they're probably not skeptics or rationalists, and they are by definition not naturalists, but to say they're "not atheist" is akin to a protestant claiming catholics are "not Christian" or vice versa.
I don't know where you get that definition from, but it isn't accurate.
Here's a start:
someone who denies the existence of god
That's not correct, either, as atheist simply means one who doesn't believe.
Atheism can be either the rejection of theism, or the position that deities do not exist.
That's about right.
atheism - the doctrine or belief that there is no God
Again, I reject this definition. However, the same site also offers the correct one:
atheistic - rejecting any belief in gods
And again, note that in either case, we're still talking about gods specifically.
One who rejects or is ignorant of theism.
And so on.
The only two definitions I see there, and the only two I've heard in general, can be summarized like this:
Tarot cards do not require deities, nor do ghosts, UFOs, or for that matter, perpetual motion machines. There are any number of things which are unscientific, possibly paranoid, dogmatic, and foolish, without ever claiming to be a religion, and certainly without being concerned with a deity in particular.
I'll agree that atheism, by itself, is not necessarily rational. But most people who admit to being atheist are rational.
Most, yes. However, if you want any meaningful statistics, you have to also include agnostics, and some of them include those who have no particular religion. Not all atheists are "out of the closet", so to speak, which is one reason I'm as vocal as I am -- to reassure those who aren't that it's ok to be an atheist, and that there are others who think the way they do -- some seem to live with the idea that there's something wrong with them, or that they should be quiet and say they're just not particularly religious...
If it comes down to talking to aliens, tho, I suspect that scientific rationalism - and probably anarchic scientific rationalism - will win the day.
I would emphasize the science and the rationalism, and while I don't know that anarchy is best, I would say someone without religious or political affiliation would be best.