Seemingly duplicate files but x2 says they're not

[appyface]

I have two small XML files I checked for duplicates using only "size" and "content". x2 says they are not duplicates... but they seem to be.

I have compared everything I can think of... I've used other duplicate finder programs, the files are reported as duplicates. I've visually compared myself using hex contents, text contents, file and document properties tools. Again, duplicates.



ZIP file of the actual files:

https://uploadnow.io/f/vXjXwk8

Please let me know why x2 thinks they are not duplicates? What am I overlooking?

Kind regards.

[nikos]

you got caught up with these very small files being below the limit:

- registry setting nMinDupeSize lets you ignore very small files (those that are smaller than this limit, default=1024 bytes). Many times your hard disk contains tons of small files that make the comparison harder, but with little benefit, as these files take next to no space. If you set it to 0 (use the advanced options editor) then you disable the small file check and examine all files, as in past versions.

for a different class of duplicate files experience, check out declone

[appyface]

Thank you I will try setting to 0.

An honest rant: My biggest frustration with X2, when a function is not 100% reliable to do what it says on the surface, it can be difficult or impossible to stumble across settings that are interfering. I've wasted countless hours on various tasks only to end up writing off the feature as something x2 cannot be trusted to do and never use it for that again. I don't believe that is your intention but if you look at my past posts I've had data loss and wrong answers plaguing my experience. Some are actual bugs that weren't/aren't fixed and others are some setting I had no way to know about.

When it is settings that renders the function not to work 100% of the time as expected, why isn't it in the Help document? I could not find any mention of this one, as example.

[nikos]

you can always ask!

[appyface]

What a strange answer.

There is a Help document. A setting affecting the trustworthiness of a function should be in there, no need to bother anyone or waste time figuring it out or finding a workaround and crossing off another x2 function from the list of trustworthy functions.

Sure, documentation can be overlooked when a change is made, but once it's come to light what's the excuse for not adding it?

A status message at the bottom of the window, "Some files not compared by default; see Help for Checking Duplicates" can point out the possible issues with the command and how to find out.

Another approach: A status message at the bottom of the window "some files are not compared by default, see advanced options>ignore small duplicate files". At least it would be a bread crumb to solving the problem instead of hidden in settings no one might know about or understand if they were looking right at them.

[Kilmatead]

why isn't it in the Help document? I could not find any mention of this one

Note that this subject and option is included in the <F1> Quickstart/Help under "Detecting duplicate files", using much the same language as Nikos' original answer (which he copied directly from changes.txt instead of the newhelp.htm file itself).

I do agree, however, that it could be better communicated, at the very least the suggestion of adding "some files are not compared by default" to the results is a good idea. I believe <Ctrl+F> search includes an unforced prompt along the lines of "Click here to peek into the girl's locker room" or something like that...

The help file's assumption that checking for duplicates is only predicated upon the user's wish to "save space by their elimination" is erroneous. There could be many reasons people seek or suspect duplicates existing, so ignore-by-default is... odd.

On a lighter note, I also agree that the response of "You can always ask" is indeed a strange answer. It omits the ever-present notion of memento mori, which no astute philosophe can afford to do. Nikos vs. Bus = <Splat> (And, yes, I know you meant "strange answer" to mean "rude answer", which it was, but I always forgive a misanthrope his occasional oversights of the social contract.)

[appyface]

No I did not mean rude answer, I did mean strange answer.

It's a strange thought process to put forth that coming here and asking, *after* spending time and effort to find out what went wrong (instead of just believing that the results are 100% accurate, which could be wrong as well but who would know that without awareness of some setting influencing the result) is somehow better than a clear indication in the Help Document.

Boggles my mind.

[Kilmatead]

Sorry, I thought you were just being polite. I read Nikos' retort as a veiled RTFM given that there actually is a clear indication in the Help Document. Personally, I'd just lop his head off for answering like that, but that's me.

But you do have a point, hence my +1 of adding a "some files were not compared" notification if nMinDupeSize is not 0. I understand the design of favouring speed and efficiency, but it shouldn't be at the expense of clarity.

[appyface]

I mentioned I looked at the Help - but missed finding. A suitable answer to my question would be

"It's here <URL link> fourth paragraph" or somesuch. And I would have said oops and thanks.


This is where I looked:

https://www.zabkat.com/x2h_5.htm#xa_dup - started here and read up to "Discover similar and bad quality photos".

OK I didn't *read* slowly because it reads like a novel and I am familiar with the overall function. Somewhere between a read/skim.

And I did read this: The advanced option called "Ignore small duplicate files" defines the limit of what is considered "small file" (default 1KB and under).

Call me stupid but that did not suggest to me it was ***ON*** BY DEFAULT as it says it is an OPTION.

Unless I am aware of such a setting or aware of a setting being ON by default that influences a 100% function, then I expect 100% of a function to happen in the absence of any information it will not.

Each time I find out x2 can't be trusted to do the job 100% as presented (in the absence of information otherwise) then I no longer use x2 for that function. At least this time there is a setting that hopefully renders the duplicate check function 100% going forward for me.

And yes, delivering a less than 100% result for any function (in the absence of calling attention to that) seems to be a very strange choice for a default.

[Kilmatead]

Call me stupid but that did not suggest to me it was ***ON*** BY DEFAULT as it says it is an OPTION.

To be fair, the mention of the option is preceded by:

To speed-up content comparisons xplorer² uses the same statistical content trick as in synchronize command, where only a portion of huge files is examined instead of each and every byte. Also small files are excluded...

So, while it does (eventually) explicitly state that "small files are excluded" (not "can be", but "are"), it doesn't lead with that, so while I wouldn't "call you stupid", I might make a joke about goldfish and their apocryphal memories regarding how long it takes to get from the start of a sentence to its end. But yeah, "skimming" - we're all guilty of it.

Come to think of it, that quote suggests that we can't trust the "synchronize" command by default either... hmm...

[appyface]

I have also been bit by synchronize and never use it either. Same reason, no indication it won't deliver what on surface says it does.

I admit I much and greatly prefer one line bullets that spit it out, to a few lines of prose I may have to read meaning into. I

For all of the above on duplicates, as example, I'd rather see a stand out sentence instead of those lines, or - better yet - a section right at the top before launching into the explanation of the function.

***** NOTE - The following settings [in case of more than one] affect the result:

Tools -> Advanced Options -> Layout Settings (tab) -> Ignore small duplicate files: Default value ignores comparing files of 1KB in size or less. Change this value to 0 to compare all files, or any desired value to exclude some files.


We've beat this to death. I've asked for explicit, quick, direct bullet points about things that alter a function, to be featured very prominently in the Help many times in the past and Nikos does not want to do it. Same with any helper text to point out results may not be as expected. Nikos does not want to do it. It's his program, end of story. I just wonder how many people trusted the results of various operations and didn't know they didn't get a factual result on surface or that they lost data (like the times I had data loss for believing what it said it did - thank heavens I verified results and used my backups to restore.)

[nikos]

it is a fact of life that whatever you show or add to the documentation or explain to people, some (most?) will not see it or understand it or find it. It depends on the person, their knowledge, etc (not to mention their station in the menstruation cycle ). You cannot write a perfect manual, so I don't. IMHO the quality of the documentation is much better nowadays -- and I have dumbened down the "average mastery" expected from the typical reader.

to prove a point, "small files" exception is mentioned in the documentation for duplicate files

ps. to compare 2 small text files for identity, it's much easier to use a diff tool. Windiff is included with xplorer2 so you can either do WINDOFF $A (if both files are in the same folder) or WINDIFF $N $I (if they are in different folders -- one version in the active and one in the inactive pane). CUSTOMIZE > USER COMMANDS should have the latter command as sample

[appyface]

I'm not sure what the point of your insults are or why you bothered to respond. Now you are just being rude.

[nikos]

!?