Slashdot Mirror
Linux Kernel Archives Struggles With Git
NewsFiend writes "In May, Slashdot discussed Kerneltrap's interesting feature about the Linux Kernel Archives, which had recently upgraded to multiple 4-way dual-core Opterons with 24 gigabytes of RAM and 10 terabytes of disk space. KernelTrap has now followed up with kernel.org to learn how the new hardware has been working. Evidently the new servers have been performing flawlessly, but the addition of Linus Torvalds' new source control system, git, is causing some heartache by having increased the number of files being archived sevenfold."
GIT is focused on trading more filespace for less bandwith. This is important for a lot of scattered developers who can afford 1-2 GB more on a harddrive, but 200-300 mb more would suck on a dsl or dialup connection.
It takes a man to suffer ignorance and smile
Be yourself no matter what they say
Then he would be able to Git-R-Done
Overrated / Underrated : Moderation
Remind me again why switching from free bitkeeper was such a great idea??
`grep -r`ing source code under Subversion takes much longer than with CVS, due to all the .svn files.
I actually decided to go ahead and read the article.
Their two problems are:
(1) rsync takes a long-ass time to run when it has to compare a crapload of files. The solution they're working on is to build a better rsync that saves its state.
(2) The i386 architecture sucks. FTFA: "master.kernel.org is still an i386 machine. It's constantly hurting for lowmem since the dentry and inode caches can only live in lowmem." The solution for that is to upgrade master.kernel.org to a 64bit machine.
(sightly) offtopic. wasn't reiser4 supposed to have 'plugin' support, so things like version control could be built directly into the file system? the prospect of being able to say type:
touch bar
echo 'foo' > bar
revisions bar
output of revision history
cp bar/revision/1 bar-version-1.0.backup
granted yes, the storage requirements and cpu usaged might be horrific, but i think something like this is inevitable in file systems, and certainly i welcome the day it becomes a reality.
- tristan
I've been struggling with stupid gits for years now. (Da-dum-dum). Thank you! I'll be here all week.
Don't just game, Dungeoneer
Aren't file system scalability issues why people start using databases?
Sounds like a software engineering issue.
The whole point of a database is to isolate you from the actual representation of the data on disk and to make querying for data easy, so you don't have to parse those files at all! For disaster recovery, I pity you if you prefer to try and extract the data manually from the files on disk themselves when you could use one of the (many) tools (that are part of the DBMS!) designed for exactly this purpose.
Game! - Where the stick is mightier than the sword!
On the flip-side, if kernel.org is using XFS, JFS, Reiserfs (I doubt they'd risk Reiser4 yet) or any other very high-performance filesystem, then maybe the problem is one of organization.
It is rare that you actually need large numbers of files holding very small amounts of data or metadata. What is probably wanted is a virtual layer that allows the software to see those many small files, but where the files are bundled together to be more efficient to access in this kind of a setting.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Bow before my might, l1nux l00s3rs!
This remark 'i386 sucks because there is little lowmem' is stupid ... let them improve linux to need less lowmem. Why would you always want dentry and inode's in lowmem anyway; it think they could be swapped into highmem just a well. That will still be faster then not having them in mem...
using a 64bit arch is a workaround, a solution would be to fix their memory management. I you don't need huge user processes, just increase kernel address range to 3Gb, and userspace to 1Gb. No more problems with lowmem ...
Maybe kernel.org should finally consider moving to a more appropriate filesystem than ext3, preferably reiserfs for it beeing optimized to handle a lot of small files. Tail packing not only saves disk space but more important a lot of memory in block cache.
Kernel sources take up, what, only a handful of gigabytes?
So the rate of files being archived was multiplied by 128?
(wouldn't it be cool to store data from your SQL tables in easy-to-parse flat files for instance? That would make recovery and manipulation a lot simpler)
*snicker*
*laugh*
*great rolling peals of laughter*
*sigh*
*wipes tear from eye*
You haven't done much work that actually required databases (or that would massively benefit from a relational programming model). The whole point of moving from flat files to a database is so that the data is stored already parsed, recovery is done by a tool provided by the db vendor, and manipulation is done within rules (constraints) that prevent "programming accidents" (bugs) or "pilot error" (users) from breaking relationships between parts of your data. That eliminates most of the need for recovery right there.
CM systems get much more powerful and IMHO, simpler, when you start using a decent database as the backend. As for distributed work, there are plenty of good databases that inexpensively and easily fit onto any modern workstation (PostgreSQL is my personal favorite) that can act as a local backing store, giving you fully detached functionality and the benefits of a relationally organized system.
Regards,
Ross
"*snicker*
*laugh*
*great rolling peals of laughter*
*sigh*
*wipes tear from eye*"
*yawn...*
"sqlfs"
(that's developer-speech for a boring but feasible project that would make you shove your buzzword db-admin-speech up your arse)
End of story
http://kerneltrap.org/node/5070
this interview with the maintainers has a comment from sombody who claims he asked by email and got the reply that ext3 is used
if thats not a good enough perhaps guessing that as "At this time, the servers run Fedora Core and use the 2.6 kernel provided by RedHat." they might be using ext3 that is the default.
"sqlfs"
(that's developer-speech for a boring but feasible project that would make you shove your buzzword db-admin-speech up your arse)
Point #1:
A quick google search yielded a few links for "sqlfs".
Now, are you really talking about a filesystem implemented in a relational database? You're pretty confused if you think you contradicted what I wrote. That's exactly what I'm advocating, except that I'm advocating that this database-backed filesystem also be CM-aware.
For this particular file-centric application, I *love* filesystems implemented as front-ends to databases.
Point #2:
You're really confused if you think I'm a db-admin or that I'm in favor of a CM tool that requires a DB-admin to install or use. I'm in favor of an ACID database as a core part of an effective CM system. Whether end users know it's on their machine is completely irrelevant (in fact, I think the user shouldn't have to know it's there).
Point #3:
Why would that project be boring? Sounds like it would be pretty neat if you ask me. I'm actually a little disappointed that none of the links yielded a project with recent activity.
Regards,
Ross
Why do you make an assumption that a database backend must be relational one? A lot of systems designed to, a) store vast amount of data b) fast retrieve data, do not use RDBMS as the backend as it tends to suck for tasks common in archival, digital asset management and maybe even configuration management. They use file/directory based indices. It really kills performance when tree-like data (read: directory structure and branched source) is forced into relational model. Add some metadata tagging to that and you'll see some horrible performance when the server is spending it's time matching data via crossref tables.
There certainly are problem sets for which the relational model is inappropriate. However, there is no large data management problem for which a flat file is more appropriate than a relational database (assertions by the Prevayler group nonwithstanding).
As for your assertion that tree-like data is a poor fit for relational programming, it's an issue of having a deeper understanding of relational programming (a "kind" of programming parallel to procedural or object-oriented programming). Trees fit perfectly well into relational models, you just need to understand how a relational system should manage them. If you're ever interested in learning more, a great book to read would be: SQL For Smarties.
Even more germane to the specifics of trees in databases, the same author has a newer book (that I haven't read) Trees and Hierarchies in SQL for Smarties. Only one person has reviewed it and they weren't overwhelmed, so who knows.
Tags are only a problem if it wasn't designed into the schema from the start (i.e. CVS style tags strapped into RCS files). If a tag is actually an association of deltas as in Subversion (much more amenable to a relational model), asking for "the state of the system at 'Release 1.1' (changeset 1134)", becomes fairly simple to implement and much simpler to optimize.
Regards,
Ross
Are you reading this man?
You're responsible for all the world's problems! The linux kernel, bitrot on my cds, war in Iraq, Guantanamo Bay, and now git!
Come on Linus, clean up your act!
(Sorry if this offends *anyone*)
Anything is possible, except skiing through revolving doors.
We did some tests comparing reiser3, xfs, and ext3 with the dir_index option on 2.6 kernels. We were writing thousands (ok tens of thousands) of small files into a couple of directories (specialized app, you don't want to know.)
When directories got large, ext3 with the hash lookups (between 800 and 1500 creations per second on newish hardware) ran much faster than xfs, oh and several orders of magnitude faster than ext3 without the directory hashing. reiser3 was slower than xfs.
We were thinking of going with xfs anyways, because it was so attractive that the directories would shrink when files were deleted (whereas ext3 directories stay big, with a hole in it.) but xfs would crash on us after a couple of days. So In March we chose ext3. We have approximately 9 million files in a single file system at the moment, it seems to work ok, but the system crashes every three weeks or so. We think we might have tortured it too much, and can reasonably keep only about 2 million files on-line, so we'll see if that helps.
of course, ymmv.
you don't want to know
But I _do_ want to know, waaahh!
I wonder if JFS would work, maybe it is too slow - I found that some xfs filesystems that I had got corrupted but even after power outages, etc my jfs partitions just keep plugging away.