Slashdot Mirror

But it's still too much money for me to be the one to go make all the first-timer mistakes and discover all the hidden costs. I guess that's precisely the reason most DIYers would buy a magazine like this.

IMO, these guides are useful for general Linux users who want a guide to various tools on their desktop.

Slackware users, on the other hand, tend to prefer a more terminal/console-centric view, so the usefulness of this guide to anyone using Slackware for, as I've usually seen it, a server of some kind [printer, file, FTP, web], would probably do better to read some other documentation.

Just my $0.25.

I believe you mean distcc -- distributed C compiler. It acts as a front-end for the local compiler which does load-sharing for compiling on multiple machines.

Especially for Gentoo, it's invaluable.

samba, rsync, ccache and much much more, but no babies as far as I know. He does have a puppy.

samba, rsync, ccache and much much more, but no babies as far as I know. He does have a puppy.

I'd like to nominate Tridge, but not for his work on Samba specifically.

I was as Linux.conf.au and saw him sit down with a complete newbie who would have been maybe 17 years old. This guy was interested in writing some code for the AI competition that was running at the conference, but didn't seem to have much of a clue about how to start it off, let alone coding anything competetive.

Tridge, out of the blue began an informal AI 101 lesson in the courtyard. I sat in on it and can honestly say I learnt more about AI in 10 minutes with Tridge then I did in four years of a CS degree. His presentation of the material came so fluently, confidently and with a warm manner.

So, I nominate him for his *general* donation to the OS movement, and not for his Samba developments (though they are fantastic). It's because of people like him that others want to get involved and make a difference too, and that's the biggest influence we could ever want.

Andrew Tridgell because rsync 0wnz0rs! Then there's that little "Samba" thing, and ccache.

- RustyTaco

Samba, rsync, ccache(? - possibly wrong on that). I love him and I want his babies.

Why the Samba Team of course. Where would we be without it?

... I could see how it'd be easy for a malicious user to force a huge number of updates by manipulating their files to force a re-patch operation, quickly taking up an ungodly amount of disk space on the server.

A strong cryptographic hash like SHA-1 would be necessary.

Patching between multiple version steps would probably make more sense, but it'd be better to adapt the algorithm to patch several times in a row instead.

One reason could be that gzip can be made to produce rsync friendlier archives.

Often, changing even a small part of an archive will change it sufficently that rsync is forced to retransfer the entire lot. However, there is a patch for gzip (which I believe is in Fedora Core 2 and is scheduled for mainline gzip inclusion too) that produces rsync friendly archives at the cost of slightly larger archives.

As somebody pointed out before in this article, there is rsync which minimizes transmitted data using some xdelta-like algorithm. This is not really new, and some sites offer anonymous rsync downloads for exact this reason.

(Rumours were that some people actually use rsync in the following way to get the latest Debian ISOs from a collection of old, already downlod packages: They cat'ed all their packages together to one huge binary file and then ran rsync against the remote ISO image and that local file. Since most data was already in that file, only transmission of a few megabytes of new data and some data arranging had to happen....)

Here I uttered a few quick&dirty thoughts (which most certainly somebody else has had before, as usual) on how rsync could help in mass patching, don't know if they are worth reading for you... .)

Or even better, rzip.

You can set it to have a buffer of up to 900 megs, as opposed to bzip2's 900k. So instead of looking for redundant information in small blocks of 900k, it looks for it in everything you compress (up to 900 megs).

And surprisingly, I haven't found it to be noticeably slower than bzip2, even on my ancient hardware (the only thing is that if you want to use it to it's full potential, you need a lot of ram, but it'll work anyway without that.. just slower).

I did XP in extreme C++ for about a year (by extreme C++, I mean boostified, Alexandrescu'd C++). We used CppUnit for our test framework.

I'm not especially satisfied with the currently available C++ unit testing frameworks. CppUnit and Boost's both have trade-offs. I suspect they'll both get better, though.

The Ant-Contrib project's cc task works pretty well, in my limited experience. I was playing around with it just this weekend. I've yet to set up a tinderbox build process, but I don't see why this wouldn't be easy with either Cruise Control or good ol' cron.

In my mind, the two biggest hurdles with doing XP in C++ are build speed and developer prejudice. You can tackle build speed with a combination of ccache, distcc, good programming principles, and cash. Tackling developer prejudice is harder. A lot of C++ programmers like to write low-level, unsafe, old school C++ code. Modern C++ mostly lets you discard unsafe coding practices without sacrificing efficiency. Whether you can convince an old C++ programmer of this is another matter; it depends on the person. I've had decent success taking Java programmers and teaching them modern C++ via pair programming.

Using Boost helps, indirectly. You write safer code, which gives you a faster development cycle. All in all, I think the basic tools are there. It might be a little harder to get fancy lava lamp integration going with C++, but there's no reason why you can't have a good build process. It's just that a lot of C++ projects haven't evolved (I think this is due, in part, to the fact that the C++ community is late to the internet; a lot of C++ programmers just don't know what's, out there.

cheers,

Jon

I did XP in extreme C++ for about a year (by extreme C++, I mean boostified, Alexandrescu'd C++). We used CppUnit for our test framework.

I'm not especially satisfied with the currently available C++ unit testing frameworks. CppUnit and Boost's both have trade-offs. I suspect they'll both get better, though.

The Ant-Contrib project's cc task works pretty well, in my limited experience. I was playing around with it just this weekend. I've yet to set up a tinderbox build process, but I don't see why this wouldn't be easy with either Cruise Control or good ol' cron.

In my mind, the two biggest hurdles with doing XP in C++ are build speed and developer prejudice. You can tackle build speed with a combination of ccache, distcc, good programming principles, and cash. Tackling developer prejudice is harder. A lot of C++ programmers like to write low-level, unsafe, old school C++ code. Modern C++ mostly lets you discard unsafe coding practices without sacrificing efficiency. Whether you can convince an old C++ programmer of this is another matter; it depends on the person. I've had decent success taking Java programmers and teaching them modern C++ via pair programming.

Using Boost helps, indirectly. You write safer code, which gives you a faster development cycle. All in all, I think the basic tools are there. It might be a little harder to get fancy lava lamp integration going with C++, but there's no reason why you can't have a good build process. It's just that a lot of C++ projects haven't evolved (I think this is due, in part, to the fact that the C++ community is late to the internet; a lot of C++ programmers just don't know what's, out there.

cheers,

Jon

Wow, you think somebody would have come up with a solution for this by now...

Yup. To quote the guy who wrote Samba, "The Free Trade Agreement between Australia and the United States is a disaster for many sections of the Australian computer industry, and for Australian computer users."

Well, for one, Dr. Andrew Tridgell is an Aussie. Several key OSS devs are as well. A major thorn in the side of Microsoft et. al. is that when they win in the US through buying laws, someone in another country does what those in the US no longer can.

One can see that the Closed Source MegaCorps have realised that unless they get all lawmakers, not just those in the US, on thier side OSS will march on with out missing a beat. This could be the thin edge of the wedge, as it were.

Soko

We go through great lengths in Samba to be as portable as possible. Our build farm runs the most popular unices on all sorts of architectures (you'd be amazed how different Linux on x86 can be than Linux on say a s390). We support a ton of platforms including some as obscure as the Amiga.

What I'd do if I were you is to just grep the Samba source code before you use a function. You'll likely find a list of platforms that it doesn't work on, or that simply doesn't have that particular function. You may also find workarounds for bugs in particular implementations.

This is unspeakably wrong! A company spends MILLIONS, perhaps even BILLIONS to come up with their own proprietary mechanism for exchanging files between their OWN services, and some damned upstart comes along, reverse-engineers it, and has the AUDACITY to make their OWN service interoperate with it...WITHOUT PERMISSION?!?!?

All of that money that the company spent, down the drain, because some "hackers" figured out how their carefully crafted system works! This is wrong and unfair!

Our course is clear! We must NOT support the evildoers who have committed this foul act of hacking! BOYCOTT! BOYCOTT!

We must NEVER use SAMBA AGAIN!!!!!

Wait...who were we talking about again?....

Because its international?

Australia has adopted the perverted USA intelectual property manners just recently, May 18 2004 (http://www.dfat.gov.au/trade/negotiations/us.html , http://samba.org/~tridge/fta_statement.html). Even now, if Samba were under attack, the development (if not all the developers themselves) might shift to Europe: perversion of the european patent system is still under construction, after the Episode I---The Patent Office Menace, Episode II---The Attack of the Parliament, and Episode III---The Revenge of the Council, the Episode IV---New Hope came, when the Netherlands, and afterwards also others, considered withdrawing their support to the prepared perverted directive, but the European Commission and the European Patent Office are undoubtedly going to strike back (http://ffii.org/). May be that Microsoft waits just for the European Union, may be it even postpones its attack untill the world for free software development gets yet smaller.

The real test of the global community power is yet to come, after the global power of the USA succedes in making the USA laws actually global.

I didn't ask people "not to recommend SAMBA" - the point is, SAMBA is an already obvious example of one such replacement. It does not cover every Microsoft service, however. I just didn't think it'd be helpful for anyone to see 500+ posts that say "Dude, do some research, SAMBA already does this". What I'm asking is, how feasible is it to do something similar for the rest of MS's services?

The MS-SQL example wasn't just a question of whether or not it's possible to use ODBC. Yes, I know what ODBC is. And, yes, I've already found UnixODBC. What I want to know is, for example, if I have a commercial package that claims to require MS-SQL server, and I try pointing it at a MySQL server in "MS-SQL Compatibility" mode, how likely is it to work? I EXPECT that the answer is probably "only if it's using ODBC to connect", but in that case, has anyone tried it, and if so, does it turn out that it occasionally still doesn't work due to special quirks or capabilities provided by MS-SQL's ODBC driver that MySQL doesn't replicate?

If I, for some reason, don't WANT an entire Windows authentication framework but just want a directory server that MS Outlook will think is "ActiveDirectory", can it be done with the right schema in OpenLDAP, or is there special missing functionality?

SharePoint's WebDAV server implementation is yet another issue, as is Microsoft's quirks in their WebDAV client driver.

I suggested this as a discussion question, not a "hey, I'm trying to do a single specific task, can somebody do a google search for me and give me detailed step-by-step instructions?" which seems to show up in the "Ask Slashdot" section from time to time lately. The subjects of making Microsoft clients somehow work with non-MS servers, and making MS servers interoperate with non-MS clients just happens to interest me, and I thought it might interest other people here too, so I asked...

No, you cited an example, not the actual definition.

The "writing your own program" part is not part of the definition.

And no, it does not neccesarily mean decompiling. Decompiling is only one method of reverse-engineering. Examining the program by comparing input/output is another one.
Perhaps you'd never use a decompiler to reverse-engineer anything, but I'm guessing that you haven't either. There are many cases in which you can't use that method, or that method is simply to difficult.

But the actual code-writing is not part of the reverse-engineering process. Writing code is engineering. You're not reversing anything!

It's called 'reverse-engineering' because engineering is the process of going from an idea to a concrete thing, like an actual program or a bridge or a car.
Reverse-engineering is the study of the engineering product in order to figure out which ideas were behind it, reversing the process. Hence the name.

Some real-life examples:
Samba is an implementation of the Windows SMB protocl based on reverse-engineering. This was done through packet analysis, not through decompilation. It is therefore also 'clean room', because they haven't seen any actual code.

Hatari, an Atari ST emulator I've contributed to, has emulation of the Atari hard drive hardware which I coded after reverse-engineering the hard-drive interface protocol. Not having an actual Atari hard-drive to test on, or a full protocol spec, I did this through disassembling and studying the driver software. It is not clean-room, since I saw the other guy's code.

(However this is not a legal problem, because A) I didn't implement a driver, rather I implemented the 'hardware' part in software. and B) I didn't use any of their code and C) Nobody cares anymore anyway.)

Classpath, an reimplementation of the Java class libraries, is 'clean room'. Sun distributes its sources to these, but noone is allowed to contribute to Classpath if they have seen them, or if they've decompiled them. The Classpath project itself is not reverse-engineering. Although some reverse-engineering is probably done by means of writing small test programs to clarify omissions from Suns Java Specification.

Decompiling is not in itself cracking. "Cracking" depending on what meaning you chose, can mean many things, among others the removal of copy protection from software.
I've did so myself back in my Atari years. Cracking is a form of reverse-engineering, and usually (but not always) entails using a disassembler. The only difference between cracking and traditional reverse-engineering is the intent.

And in the USA this is also a legal difference: The DMCA prohibits reverse-engineering with the intent to circumvent a copyright-protection device. It expressly allows it in order to achieve interoperability.

If you're European, you can look at directive 91-250-EEC of the EEC (now EU), Article 6. This legislation has been implemented in laws of all member states, and it expressly allows decompilation for interoperability purposes.

Here is how to GET internet connectivity on your box using an ISA adapter

telnet + listening connection = root

At $79, you can hardly afford the harddrive to roll your own PVR.

What about the subscription cost? I'd still like to see someone build a quality homebrew PVR for $399. Unless there time is worth nothing and they have access to really cheap hardware, you can't beat the price of a ReplayTV or TiVo.

The problem comes when someone hacks these to pull their content updates from another source. The The Hacking the TiVo FAQ hosted by samba.org is against it... but what will happen to TiVo's business model when someone hacks together support for XMLTV or RSS enclosures?

Well they are hackable, just not fully supported by Tivo Inc.

distcc compared to Cook.

The biggest point is probably that Cook requires you to do your builds through Cook, but distcc can fit in with any build system: Make, SCons, Cons, or probably even Cook.

It's a bit bad that the article doesn't mention security, but security has been considered in distcc.

If you don't trust your network, you can run distcc over ssh, just like cvs, svn, rsync, ... That gives you good protection against network atacks, and doesn't increase the exposure of the machine at all. Just set DISTCC_HOSTS="me@otherhost"

Of course confidentiality and integrity protection takes cycles, so if you *do* trust your network, you don't need to use it.

Using SSH is a far better approach than a handcrafted authentication system, which would probably be broken in various ways, if other protocols are any indication.

gcc will never be secure against untrusted input, it's just far too complex. Even trying would be an enormous waste of time.

While it is slightly different in concept, check out ccache [samba.org].

If you have common filesystems (not necessarily even mounted in the same place, though this requires some unattractive transformation rules in the nc config file), and a synchronized clock (anyone heard of ntp?), nc will either definitely be faster than distcc, or will bring your NFS server to its knees (occasionally both).

I personally liked working with NetApp support explaining how 1 user doing 1 make could swamp a NetApp F840 (pushing about 1.6Gbit/sec over load balanced gig links to a cluster of 10 4-processor ultrasparcs). Our build, which would normally take ~1.5 hours (with -j4 on one machine) could finish in only about 12 minutes with nc.

When used across a workgroup, with distributed ccache, CCACHE_HARDLINK, scons and content signatures, it still rocks with a much more lame fileserver (it spreads the ccache jobs over the cluster, and most writes are just creating hardlinks on the server).

This may not help people trying to build random packages they picked up, but if you work on a large project in a workgroup environment, all of these tools:

• ccache
• scons
• distcc
• nc

-se

(My apologies for the shameless self-promotion)

If you have common filesystems (not necessarily even mounted in the same place, though this requires some unattractive transformation rules in the nc config file), and a synchronized clock (anyone heard of ntp?), nc will either definitely be faster than distcc, or will bring your NFS server to its knees (occasionally both).

I personally liked working with NetApp support explaining how 1 user doing 1 make could swamp a NetApp F840 (pushing about 1.6Gbit/sec over load balanced gig links to a cluster of 10 4-processor ultrasparcs). Our build, which would normally take ~1.5 hours (with -j4 on one machine) could finish in only about 12 minutes with nc.

When used across a workgroup, with distributed ccache, CCACHE_HARDLINK, scons and content signatures, it still rocks with a much more lame fileserver (it spreads the ccache jobs over the cluster, and most writes are just creating hardlinks on the server).

This may not help people trying to build random packages they picked up, but if you work on a large project in a workgroup environment, all of these tools:

• ccache
• scons
• distcc
• nc

-se

(My apologies for the shameless self-promotion)

If you have common filesystems (not necessarily even mounted in the same place, though this requires some unattractive transformation rules in the nc config file), and a synchronized clock (anyone heard of ntp?), nc will either definitely be faster than distcc, or will bring your NFS server to its knees (occasionally both).

I personally liked working with NetApp support explaining how 1 user doing 1 make could swamp a NetApp F840 (pushing about 1.6Gbit/sec over load balanced gig links to a cluster of 10 4-processor ultrasparcs). Our build, which would normally take ~1.5 hours (with -j4 on one machine) could finish in only about 12 minutes with nc.

When used across a workgroup, with distributed ccache, CCACHE_HARDLINK, scons and content signatures, it still rocks with a much more lame fileserver (it spreads the ccache jobs over the cluster, and most writes are just creating hardlinks on the server).

This may not help people trying to build random packages they picked up, but if you work on a large project in a workgroup environment, all of these tools:

• ccache
• scons
• distcc
• nc

-se

(My apologies for the shameless self-promotion)

Absolutely, I use it at home all the time. It's great for sofa computing: sit on the sofa with a modest laptop, and send your compile jobs across a wireless network to a faster machine in the study.

If you use the LZO compression option then it's quite useful even on 5Mbps wireless.

You can also tell distcc to run as many jobs remotely as possible to keep the laptop from scorching your lap.

It's really nice to be able to build the kernel from source in a reasonable time on a 800MHz machine.

Absolutely, I use it at home all the time. It's great for sofa computing: sit on the sofa with a modest laptop, and send your compile jobs across a wireless network to a faster machine in the study.

If you use the LZO compression option then it's quite useful even on 5Mbps wireless.

You can also tell distcc to run as many jobs remotely as possible to keep the laptop from scorching your lap.

It's really nice to be able to build the kernel from source in a reasonable time on a 800MHz machine.

See the distcc FAQ:

nc requires the machines to have a shared filesystem mounted at the same point, synchronized clocks and exactly the same headers, libraries and tools installed. (Similarly for all the dozens of other times this idea has been invented: Sun dmake, pvmake, doozer, dbuild, ....)

If you have that setup and you're sure everything will always be in sync, fine, you can distribute many jobs. Although distcc requires the compilers be the same, it is far more forgiving of all the variables. It may also be faster, depending on the setup: NFS is pretty awfully slow on modern machines.

The whole point of distcc is that you can install it without needing root privileges, without needing to use NFS, and without needing to reconfigure your machines.

Both of them are likely to break on buggy makefiles that can't handle -j. So fix your makefile, or, as you say, use SCons. SCons rocks,
and rocks even harder with distcc.

See the distcc FAQ:

nc requires the machines to have a shared filesystem mounted at the same point, synchronized clocks and exactly the same headers, libraries and tools installed. (Similarly for all the dozens of other times this idea has been invented: Sun dmake, pvmake, doozer, dbuild, ....)

If you have that setup and you're sure everything will always be in sync, fine, you can distribute many jobs. Although distcc requires the compilers be the same, it is far more forgiving of all the variables. It may also be faster, depending on the setup: NFS is pretty awfully slow on modern machines.

The whole point of distcc is that you can install it without needing root privileges, without needing to use NFS, and without needing to reconfigure your machines.

Both of them are likely to break on buggy makefiles that can't handle -j. So fix your makefile, or, as you say, use SCons. SCons rocks,
and rocks even harder with distcc.

ccache is a compiler cache. It acts as a caching pre-processor to C/C++ compilers, using the -E compiler switch and a hash to detect when a compilation can be satisfied from cache. This often results in a 5 to 10 times speedup in common compilations.

This might be a better fit for your situation.

While it is slightly different in concept, check out ccache. It only uses a single computer but it can significantly speed up your compiles. It works by caching the results of each compilation; it will only help if you compile the same code over and over.

It's also been been discussed here on Slashdot (two years ago!) in "A Distributed Front-end for GCC" and earlier this year in "Optimizing distcc."

Distcc is great for installing Gentoo on an older computer because you can have other (faster) computers help with the compile, and if you like distcc, you may also like ccache.

XCode uses distcc for it's distributed compiling, distcc isn't rendezvous aware by default though, so you'll have to configure it yourself.

Lustre, I don't know anything about it except it exists. SMB, yes it's the spawn of satan, but if you need to share with windows machines there's really no choice.

> can take hours to build.

Ah, yes, there's the key. You're right - if a daily build means that the checkin window is only a few hours, that's a problem. Something else needs to be fixed - maybe the build can be distributed or some such.

We do hourly builds on my current project - but it's in Java, and the whole build only takes about 20 minutes, so that works fine.

> perhaps they have developed better methodologies

Yup, maybe distcc. One would hope so, anyhow.

> the right interval for builds

I'd say - as often as possible without being a pain :-)

Xgrid is proprietary, closed-source software

Actually, that is completely false:

Xcode uses distcc to manage distributed builds. The distcc client manages the setup and distribution of build tasks. The server process (distccd) manages communication with the remote computers hosting the build tasks. The server process runs on both the local, or client, computer and on the remote computer.

As 30 seconds of Googling will tell you, distcc [is] a fast, free distributed C/C++ compiler.

As they have done with KDE's KHTML engine in Safari, so is Samba's distcc engine being used in XCode.

Care to try again ?

:-)

Xgrid is proprietary, closed-source software

Actually, that is completely false:

Xcode uses distcc to manage distributed builds. The distcc client manages the setup and distribution of build tasks. The server process (distccd) manages communication with the remote computers hosting the build tasks. The server process runs on both the local, or client, computer and on the remote computer.

As 30 seconds of Googling will tell you, distcc [is] a fast, free distributed C/C++ compiler.

As they have done with KDE's KHTML engine in Safari, so is Samba's distcc engine being used in XCode.

Care to try again ?

:-)

[global]
workgroup = MIDEARTH
netbios name = HOBBIT
security = SHARE
encrypt passwords = yes

[data]
comment = Data
path = <Your_share_path>
force user = <your_username>
force group = users
read only = No
guest ok = Yes

Just rzip (better than bzip2 with large files) the email archive and burn it to DVDrs. So in case of real legal necessity, it's possible to access it and the whole setup (DVD-burner + media) probably cost around 100$.

No. Clusters are only good for doing tasks that "parallelize". Compiling is too linear.

Compiling a single file is. But most software has more than one file, and you can compile different files on different machines. Check out distcc.

Of course, even using distcc, a single low-end Athlon will be 10x faster than all of those boxes combined, and use up a lot less electricity as well. So it's not exactly a useful excercise. Interesting, perhaps.

If you are using Windows 2000/2003, an easy redundant file serving solution is to setup DFS (distributed file system). Just a tip, don't setup a domain-wide share for a file server that gets a lot of updates. Using DFS like that can create an administrative nightmare (last writer wins situation). You would want to use a domain-wide share if you have a lot of read-only files (like installation files, PDF image archives, etc) and you need a high-availability solution. You would be restoring files from tape a lot. Anyhoo, if your first server crashes, temporarily redirect your users to the second server either via DNS or just renaming the servers. DFS doesn't replicate printers, so you would have to install a new printer two times, once on the first server and a second time on your second server. Shouldn't be too much a problem if you only have 15 users.

If you are using Linux/UNIX/*BSD, you could use Rsync. There was a great article explaining Rsync usage in the June '04 print edition of SysAdmin.

I've done it. Its not pretty, but certainly doable with the help of openmosix, and ccache. Bootstrapping Gentoo with a fairly fast machine onto DVD-RAM media however, was much much slower. Yes I'm a masochist.

hehe
http://www.samba.org 'nuff said.