Linux Kernel Archives Struggles With Git

NewsFiend writes "In May, Slashdot discussed Kerneltrap's interesting feature about the Linux Kernel Archives, which had recently upgraded to multiple 4-way dual-core Opterons with 24 gigabytes of RAM and 10 terabytes of disk space. KernelTrap has now followed up with kernel.org to learn how the new hardware has been working. Evidently the new servers have been performing flawlessly, but the addition of Linus Torvalds' new source control system, git, is causing some heartache by having increased the number of files being archived sevenfold."

Comments

Re:same reason I dislike Subversion

[Anonymous Coward]

Just exclude .svn from your grepped files. For further details, look at 'man grep' or 'info grep'.

-d ACTION, --directories=ACTION If an input file is a directory, use ACTION to process it. By default, ACTION is read, which means that directories are read just as if they were ordinary files. If ACTION is skip, directories are silently skipped. If ACTION is recurse, grep reads all files under each directory, recursively; this is equivalent to the -r option.

Re:Filesystem?

[Yenya]

Disclaimer: I run one of the kernel.org mirrors.

Ext3 vs. Reiser is not an issue here. FWIW, I use XFS on my mirror volume, and I have also noticed how the git repository increases load on my server. See the CPU usage graph of ftp.linux.cz - look especially at the yearly graph and see how the CPU system time has been increasing for last two months.

The problem is in rsync - when mirroring the remote repository it has to stat(2) every local and remote file. So the directory trees have to be read to RAM. Hashed or tree-based directories (reiserfs or xfs) can even be slower than plain linear ext3 directories, because you have to read the whole directory anyway, so linear read is faster.