blog: how do I delete the duplicates?

[nikos]

here's the comment area for today's blog post found at
http://zabkat.com/blog/delete-duplicate-files.htm

[devilishstyl3]

did you try DuplicateFilesDeleter? it is really working for me !

[Tuxman]

The only problem is that we'd still use an "image duplicate checker" - actually, that's the only duplicate check I ever used.

[RightPaddock]

I think I'm right in saying that X2 regards hardlinks (multiple links to the SAME data) as duplicates.

IMO it shouldn't. If FMT (free) can detect and discount them when calculating folder sizes, then X2 (pay4me) should be capable of doing the same.

RP

[Kilmatead]

IMO it shouldn't.

That's more of a semantic thing, isn't it? Considering that hard-links rather literally are the same data, the user may not recognise this at first glance and discount the connection. But, I know not.

This distinction may not have legs for much longer anyway, as hard-links are deprecated in ReFS...

[nikos]

you can turn on the LINKS column then check if a duplicate group is all links among themselves

[RightPaddock]

IMO it shouldn't.

That's more of a semantic thing, isn't it? Considering that hard-links rather literally are the same data, the user may not recognise this at first glance and discount the connection. But, I know not.

This distinction may not have legs for much longer anyway, as hard-links are deprecated in ReFS...

@Kilmatead - It's not 'semantic' if I knowingly created them for my perverted purposes, or if they were created by the OS for its nefarious purposes. They are not the sort of thing that turn up by accident - I have files that have reached the link limit

MS have been singularly quiet regarding ReFS in recent times, this is the most recent item I could find on MSDN January 17, 2012 - Building the next generation file system for Windows: ReFS. Its obviously targeted at the high end e.g. data centres & cloud storage. AFAIK it's not in Windows 9, maybe it's in Windows Server 2015. Maybe Ballmer gave to Sinofsky as a leaving present, or maybe it's joined WinFS in the dead bit locker. Whatever - the chances of me ever running ReFS on my computers is zilch. One can say what one likes about MS, but it always protects its legacy - sometimes to a fault, a'la HPFS path limits


@nikos, I always show the links column, indeed I often sort on it, I also decorate the file icon with a red arrow overlay - belt and braces. But neither you nor K will ever convince me that a hardlink of file A is a duplicate of file A. AFAIK if you delete all the extra hard links for a given file, you will not recover a single nanobit of diskspace - from MS Technet

After you create and format an NTFS volume, NTFS metafiles are created. One of these metafiles is called the "Master File Table" (MFT). This file is very small when it is created (approximately 16 KB), but it grows as files and folders are created on the volume. When a file is created, it is entered into the MFT as a file record segment, which is always 1024 bytes (1 KB) in size. As files are added to the volume, the MFT grows as required. However, when you delete files, the associated file record segments are marked as free to be reused, but the total file record segments and associated MFT allocation remains the same. This behavior explains why you do not regain the space that is used by the MFT after you delete a large number of files.

AFAIK that applies to Win 7, not sure about Win 8. It used to be that unused MFT space could only be recovered with 3rd party defraggers; not sure about today, but who cares given current disk prices.


@makinero - I use a gadget called DupDetector from Prismatic Software it finds similar images based on edge patterns, colours etc, it will find mirrors, flips, resizes, monochrome renditions, and transforms (eg jpg and png) etc - make sure you get the right one, there are several DupDetectors.

RP

[RightPaddock]

RightPaddock - Most of these programs compares Visual only file 1:1
It is a boring job comparing only 1:1 and so with each file
what if such duplicates is more?

Example:
http://video.informer.com/s1/win/dupdet ... torIAR.mp4

I've used it several times to reduce collections of ~50.000+ photos to ~5,000, yes it's intensive work, typically takes ~3-4 hours

RP

[RightPaddock]

But neither you nor K will ever convince me that a hardlink of file A is a duplicate of file A. AFAIK if you delete all the extra hard links for a given file, you will not recover a single nanobit of diskspace

I forgot to mention this. If you turn duplicates into hardlinks then you will save diskspace, see ==>> DupeMerge

RP