Slashdot Mirror

I used symlinks (or whatever the Windows name is) before Steam introduced multiple libraries so I could have some games installed on the SSD and others on an HDD. There's a great shell extension here that lets you create them. Since then I've reformatted and so I no longer need them for Steam, but they do come in handy for splitting my home folder across drives. The tool I've linked to also does junctions but I've never had to use those.

You might want to take a look at Link Shell Extension

I use the free command line tool dupemerge.exe to do file level dedupe on ntfs and I have found it to be pretty fast with lots of options.

See http://schinagl.priv.at/nt/dupemerge/dupemerge.html for full details.
"Introduction
Most hard disks contain quite a lot of completely identical files, which consume a lot of disk space. This waste of space can be drastically reduced by using the NTFS file system hardlink functionality to link the identical files ("dupes") together.
  Dupemerge searches for identical files on a logical drive and creates hardlinks among those files, thus saving lots of hard disk space.

Backgrounders
Dupemerge creates a cryptological hashsum for each file found below the given paths and compares those hashes to each other to find the dupes. There is no file date comparison involved in detecting dupes, only the size and content of the files.

  To speed up comparison, only files with the same size get compared to each other. Furthermore the hashsums for equal sized files get calculated incrementally, which means that during the first pass only the first 4 kilobyte are hashed and compared, and during the next rounds more and more data are hashed and compared.

  Due to long run time on large disks, a file which has already been hashsummed might change before all dupes to that file are found. To prevent false hardlink creation due to intermediate changes, dupemerge saves the file write time of a file when it hashsums the file and checks back if this time changed when it tries to hardlink dupes.

  If dupemerge is run once, hardlinks among identical files are created. To save time during a second run on the same locations, dupemerge checks if a file is already a hardlink, and tries to find the other hardlinks by comparing the unique NTFS file-id. This saves a lot of time, because checksums for large files need not be created twice.

  Dupemerge has a dupe-find algorithm which is tuned to perform especially well on large server disks, where it has been tested in depth to guarantee data integrity."

The problem with a lot of file duplication tools is that they only consider files individually and not their location or the type of file. Often we have a lot of rules about what we'd like to keep and delete - such as keeping an mp3 in an album folder but deleting the one from the 'random mp3s' folder, or always keeping duplicate DLL files to avoid breaking backups of certain programs.

With a large and varied enough collection of files it would take more time to automate that than you would want to spend. There are a couple of options though:

You could get some software to replace duplicate files with hard links. This will save you space but not make things any neater - DupeMerge looks like it would do it on NTFS but I haven't tried it myself.

Another alternative would be to move your data to a file system that has built in de-duplication such as ZFS and let that handle everything.

Finally when I was looking at this myself what I found was that the problem was not individual duplicate files but that certain trees of files occurred identically in multiple places (adhoc backups of systems were a big culprit here). What you could do with but which I couldn't find and didn't get round to finishing writing was something that would CRC not individual files but entire trees of files/folders and report back the matches. If something does already exist to do that I'd be quite interested myself.

No version of Windows Explorer lets you create Symlinks. A .lnk does what most users want to do (create a link to a file or folder) whithout adding the extra complications involved in adding links at the file system level. Hardlinks/junctions/symlinks are handled at the file system driver level which means they are seen by higher level drivers and applications as regular files or directories. They must explicitely check to determine if they are actually file system links, if they do not do this then there are all kinds of problems with things like parent references and recursion. Some examples:

Under older versions of Windows if you deleted a Junction with Explorer it wouid recursively delete the contents of the folder which the junction pointed to, because it appeared to Explorer like any other directory on the filesystem. Under Vista and 7 Explorer was updated to be aware of file system links and will only delete the symlink or junction reference. There are still uses in Vista which you can see when you check size of the "Windows" directory under it's properties. The WinSxS directory contains a lot of hardlinks to files in system32. Windows Explorer incorrectly adds the size of the linked files into it's total calculation. This means that the space of a single file on the file system is counted multiple times, resulting in a massive directory size.

NTFS file permissions that are inherited from parent objects will be applied differently when opened from the file system links, because the path is completely different (and will have a different set if parent objects to inherit from). This means a user can create a symlink to the C:\Wimdows\system32 folder in a their Documents folder, and when it is opened it will ingerit permissions from the user folder and not from the Windows folder (which requires Administrative privileges to change).

So you can only create file system links through the command line using fsutil or mklink. Third party tools can be used if you need a GUI interface. LinkShellExtension is freeware and integreates handling of the file system links into Explorer so you can work with them properly and easily.

GUI CLI

I currently use this program:
http://schinagl.priv.at/nt/hardlinkshellext/hardli nkshellext.html
though I am also looking for a better way to navigate my content. DRM as it is currently, only makes the process harder.

http://schinagl.priv.at/nt/hardlinkshellext/hardli nkshellext.html
Problem solveed, though it is neither a Microsoft-produced nor open source solution.

Dictionary:
http://wordweb.info/

Search to replace Windows crippled search, though Desktop search may be better:
http://www.mythicsoft.com/agentransack/

New to me, but have been wanting since before XP:
http://schinagl.priv.at/nt/hardlinkshellext/hardli nkshellext.html Video editing:
http://www.virtualdub.org/
Notepad replacement:
http://notepad-plus.sourceforge.net/

And of course Firefox and OpenOffice, but those are available for other systems, oh and VideoLan Client (VLC)
For programming: (also available for Linux)
http://www.freebasic.net/

You also may want to check out my learning japanese apps post:
http://ask.slashdot.org/comments.pl?sid=182351&cid =15073257

vulture capitalist

- A.P.

--
* CmdrTaco is an idiot.

http://members.home.com/pnevares/jargo n.html

List of Jargon Resources Mirror Sites USA:

• http://www.akrotech.com/~darkstar/jargon
• http://memes.org/jargon
• http://www.journalism.wisc.edu/jargon/
• http://www.mindspring.com/~li mbert/hacking/jargon.htm
• http://www.iscvt.org/jargon/jargon.html
• http://www.babcom.com/jargon/index.html
• http://www.hackboy.com/jargon
• http://www.pulhas.org/
• http://www2.netdoor.com/~lhand
• http://avatar.deva.net/
• http://www.blee.net/jargon
• http://www.fortuneci ty.com/skyscraper/jolt/15/jargonindex.html
• http://www.jargon.8hz.com/
• http://culture.0wnz-u.org/
• http://www.houseofhack.com/jargon
• http://jollyrogers.com/jargon/
• http://handel.math.psu.edu/jargon

• http://celestrion.totalaccess.net/do cs/jargon/
• http://www.pir.net/pir/jargon/
• http://www.technozen.com/tetsuo/jargon/
• http://ude.org/jargon
• http://web.chad.org/usr/doc/jargon-file/

• http://karnak.nmc.siu.edu/jargon/

Australia:

• http://ecco.bsee.swin.edu.au/text/jargon/

• http://ecco.bsee.swin.edu.au/text/jargon

Czechoslovakia: ttp://www.instinct.org/texts/jargon-file/

Finland: http://zone.pspt.fi/jargon/

Germany:

• http://www.ude.org/jargon
• http://www.ghks.de/computer/jargon/
• http://www.math.fu-berlin.de/~rene/jargo n/
• http://hex.rz.ruhr-uni-bochum.de/jargon/
• http://www.informatik.hu-berlin.de /~bergt/jargon

Gret Britain: http://jargon.strugglers.net

Greece: http://www.hack.gr/jargon

Italy: http://beatles.cselt.stet.it/mirrors/jargon

Japan: http://www.vacia.is.tohoku.ac.jp/jargon/

Norway: http://www.pvv.ntnu.no/misc/jargon/ Poland: http://www.uci.agh.edu.pl/jargon/

Spain: http://www.undersec.com/jargon

Sweden: http://ftp.sunet.se/jargon/

U.K.:

• http://www.new.ox.ac.uk/admin/jargon/
• http://g.pet.cam.ac.uk/g/jargon/

1. Also spelled "/. effect"; what is said to have happened when a website being virtually unreachable because too many people are hitting it after the site was mentioned in an interesting article on the popular Slashdot news service. The term is quite widely used by /. readers, including variants like "That site has been slashdotted again!" 2. In a perhaps inevitable generation, the term is being used to describe any similar effect from being listed on a popular site.

• http://www.snafu.priv.at/jargon/jargon. html
• http://www.tuxedo.org/~esr/jargon/jar gon.html