Page 1 of 1

Never Completed

Posted: 2021 Sep 03, 15:22
by MikeA01730
Tried it twice to find duplicates in ~22,500 photos (Mostly jpg & cr2 & XMP, some tif and psd). Both times it started progressing through the folders using 7% CPU time at the beginning to 16% later and eventually stabilized at ~1.2 GB of memory. Did that for maybe 1/2 hr., then stopped advancing through folders. Stalled at a different place the second time.

Was this too many files to throw at it?

Re: Never Completed

Posted: 2021 Sep 04, 05:02
by nikos
declone shouldn't have a problem with lots of files (although it does consume a lot of memory for the time being).
what kind of search parameters did you use? did you leave on the default options or have you tweaked them?
it's hard to say what went wrong.... did the progressbar appear "working"?

Re: Never Completed

Posted: 2021 Sep 04, 23:06
by MikeA01730
Search for duplicate: Photos
Extensions: bmp,ico,jpg,gif,img,pcx,png,tif,xcf,raw,emf,tga,heic,jpeg

Scan options
Similarity level: 90 %
Ignore system files: True
Compare same types: True
Ignore small files: True

Folders to search
[+] My Pictures (that's "E:\Users\Mike\Pictures\Personal", an internal HD)

Compare these properties
Photo.DateTaken
Check file content: True

Also I checked and it should have scanned 43,800 files.

I just ran it again. At "phase 1: gathering files" progress bar advanced and the "searching in" folder at the bottom advanced.

About 9 min. later the progress bar stopped, the folder searching stopped advancing (in a folder containing 756 files), but CPU use continued at ~18%, Memory use ~1.1 GB, and disk ~ 0.1 MBs.

I waited another 30 min. after it stalled but nothing changed so I cancelled it.

Re: Never Completed

Posted: 2021 Sep 05, 06:23
by nikos
ok so you are using similarity photo search and I suspect that declone had a problem getting the pic preview somewhere and it got stuck
do you think you can use this procdump tool and send me the report when declone is stuck?
www.zabkat.com/blog/procdump-debug-hung-processes.htm
thanks

ps. also try to see if you can narrow down the problem, instead of checking your entire pics collection, do them one folder at a time? Your hint is the folder name where declone got stuck in the first place

Re: Never Completed

Posted: 2021 Sep 06, 16:17
by MikeA01730
Started declone this morning, 6 min. later it had stalled at a different place than before. Let it run for 31 min. after.

I then ran procdump64 declone. The dump is here: https://www.dropbox.com/sh/re6yful6y5w7 ... giySa?dl=0. When I was looking for the dump file I ran Everything file search and got a messed up display. Also once before I got a messed up display on the desk top instead of a window. I included an image of the Everything file search: it looked to me like the window didn't refresh as I dragged the lower right corner out. That problem stopped as soon as I cancelled declone.

After clicked cancel for the first time I got output! 862 groups, up to year/folder 2009. The photos start in 2002 and go up to now. At a glance (don't have more time to look now) the output looked correct. Maybe it was working all the time but just the folder and progress display was stuck?

Don't really have time to try to narrow it down to a folder, and since it stalls at a different place each time not sure that's useful.

Re: Never Completed

Posted: 2021 Sep 07, 05:45
by nikos
thanks for the DMP file, i will look into it
however if declone responded to cancel button, then it wasn't stuck to begin with

Re: Never Completed

Posted: 2021 Sep 08, 06:35
by nikos
i see from the DMP file that the problem could be related to 32/64 bit (the beta is 32 bit) so it may go away with the final version which is going to be x64

Re: Never Completed

Posted: 2021 Sep 08, 15:36
by MikeA01730
If you do make a 64 bit version I'd love to try it. I'm actually trying to solve a problem now (organize years of photos from different cameras and sources with lots of file name collisions) and declone could be a big help.

Re: Never Completed

Posted: 2021 Sep 09, 06:09
by nikos
if you open declone installer with something like 7zip, you will find inside an 64 bit version, but it is in trial mode, so it won't let you delete more than 100 files. But please extract and try it, I want to see if it will crash on detection or not?

Re: Never Completed

Posted: 2021 Sep 09, 19:28
by MikeA01730
I tried the 64 bit version. After 4 min. it stalled, i.e. folder searching didn't change and progress bar froze. An hour later I looked at it and the progress bar was gone, declone was using 0% CPU and 120 MB of memory, the search folder at the bottom was partially clipped off and otherwise unchanged, and the list of files on the screen was empty. I closed it. I never saw any results.

Re: Never Completed

Posted: 2021 Sep 10, 08:20
by nikos
just to be certain, are you sure you were running the 64 bit version? Did it say x64 in HELP > ABOUT?

Re: Never Completed

Posted: 2021 Sep 10, 20:58
by MikeA01730
In 7z I tried to unpack everything in declone_setup_beta.exe into a new folder but it didn't work because somehow in the 7z plugins folder you have multiple files with the same name. What's with that? I thought that wasn't possible and didn't know what to do with it so I just copied declone64.exe only and tried it. Perhaps not my best moment: did that mess things up? Anyway Help doesn't work without some of the other files but I know I'm executing declone64.exe. If you want me to install the other files tell me what to do about the duplicate names.

Re: Never Completed

Posted: 2021 Sep 11, 04:42
by nikos
ok if you were running declone64 then you did the right thing. You can always double check using HELP > ABOUT menu command (next to the toolbar button), which says if the program is 64 bit or not

Re: Never Completed

Posted: 2021 Sep 14, 09:53
by nikos
there's definitely a problem with the thumbnail cache once you read a few thousands of photos. I will try to avoid the cache and see how it goes

Re: Never Completed

Posted: 2021 Sep 15, 12:31
by nikos
I believe I fixed the problem, see the other post for the download link
I tested it on 30K photos, and although it takes some time to generate the thumbnails, it works without crashing