Never Completed

Discussion and support for Desktop Rules and i-DeClone

Moderator: nikos

Post Reply
MikeA01730
Member
Member
Posts: 12
Joined: 2014 Oct 22, 23:25

Never Completed

Post by MikeA01730 »

Tried it twice to find duplicates in ~22,500 photos (Mostly jpg & cr2 & XMP, some tif and psd). Both times it started progressing through the folders using 7% CPU time at the beginning to 16% later and eventually stabilized at ~1.2 GB of memory. Did that for maybe 1/2 hr., then stopped advancing through folders. Stalled at a different place the second time.

Was this too many files to throw at it?
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

declone shouldn't have a problem with lots of files (although it does consume a lot of memory for the time being).
what kind of search parameters did you use? did you leave on the default options or have you tweaked them?
it's hard to say what went wrong.... did the progressbar appear "working"?
MikeA01730
Member
Member
Posts: 12
Joined: 2014 Oct 22, 23:25

Re: Never Completed

Post by MikeA01730 »

Search for duplicate: Photos
Extensions: bmp,ico,jpg,gif,img,pcx,png,tif,xcf,raw,emf,tga,heic,jpeg

Scan options
Similarity level: 90 %
Ignore system files: True
Compare same types: True
Ignore small files: True

Folders to search
[+] My Pictures (that's "E:\Users\Mike\Pictures\Personal", an internal HD)

Compare these properties
Photo.DateTaken
Check file content: True

Also I checked and it should have scanned 43,800 files.

I just ran it again. At "phase 1: gathering files" progress bar advanced and the "searching in" folder at the bottom advanced.

About 9 min. later the progress bar stopped, the folder searching stopped advancing (in a folder containing 756 files), but CPU use continued at ~18%, Memory use ~1.1 GB, and disk ~ 0.1 MBs.

I waited another 30 min. after it stalled but nothing changed so I cancelled it.
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

ok so you are using similarity photo search and I suspect that declone had a problem getting the pic preview somewhere and it got stuck
do you think you can use this procdump tool and send me the report when declone is stuck?
www.zabkat.com/blog/procdump-debug-hung-processes.htm
thanks

ps. also try to see if you can narrow down the problem, instead of checking your entire pics collection, do them one folder at a time? Your hint is the folder name where declone got stuck in the first place
MikeA01730
Member
Member
Posts: 12
Joined: 2014 Oct 22, 23:25

Re: Never Completed

Post by MikeA01730 »

Started declone this morning, 6 min. later it had stalled at a different place than before. Let it run for 31 min. after.

I then ran procdump64 declone. The dump is here: https://www.dropbox.com/sh/re6yful6y5w7 ... giySa?dl=0. When I was looking for the dump file I ran Everything file search and got a messed up display. Also once before I got a messed up display on the desk top instead of a window. I included an image of the Everything file search: it looked to me like the window didn't refresh as I dragged the lower right corner out. That problem stopped as soon as I cancelled declone.

After clicked cancel for the first time I got output! 862 groups, up to year/folder 2009. The photos start in 2002 and go up to now. At a glance (don't have more time to look now) the output looked correct. Maybe it was working all the time but just the folder and progress display was stuck?

Don't really have time to try to narrow it down to a folder, and since it stalls at a different place each time not sure that's useful.
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

thanks for the DMP file, i will look into it
however if declone responded to cancel button, then it wasn't stuck to begin with
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

i see from the DMP file that the problem could be related to 32/64 bit (the beta is 32 bit) so it may go away with the final version which is going to be x64
MikeA01730
Member
Member
Posts: 12
Joined: 2014 Oct 22, 23:25

Re: Never Completed

Post by MikeA01730 »

If you do make a 64 bit version I'd love to try it. I'm actually trying to solve a problem now (organize years of photos from different cameras and sources with lots of file name collisions) and declone could be a big help.
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

if you open declone installer with something like 7zip, you will find inside an 64 bit version, but it is in trial mode, so it won't let you delete more than 100 files. But please extract and try it, I want to see if it will crash on detection or not?
MikeA01730
Member
Member
Posts: 12
Joined: 2014 Oct 22, 23:25

Re: Never Completed

Post by MikeA01730 »

I tried the 64 bit version. After 4 min. it stalled, i.e. folder searching didn't change and progress bar froze. An hour later I looked at it and the progress bar was gone, declone was using 0% CPU and 120 MB of memory, the search folder at the bottom was partially clipped off and otherwise unchanged, and the list of files on the screen was empty. I closed it. I never saw any results.
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

just to be certain, are you sure you were running the 64 bit version? Did it say x64 in HELP > ABOUT?
MikeA01730
Member
Member
Posts: 12
Joined: 2014 Oct 22, 23:25

Re: Never Completed

Post by MikeA01730 »

In 7z I tried to unpack everything in declone_setup_beta.exe into a new folder but it didn't work because somehow in the 7z plugins folder you have multiple files with the same name. What's with that? I thought that wasn't possible and didn't know what to do with it so I just copied declone64.exe only and tried it. Perhaps not my best moment: did that mess things up? Anyway Help doesn't work without some of the other files but I know I'm executing declone64.exe. If you want me to install the other files tell me what to do about the duplicate names.
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

ok if you were running declone64 then you did the right thing. You can always double check using HELP > ABOUT menu command (next to the toolbar button), which says if the program is 64 bit or not
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

there's definitely a problem with the thumbnail cache once you read a few thousands of photos. I will try to avoid the cache and see how it goes
User avatar
nikos
Site Admin
Site Admin
Posts: 15760
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: Never Completed

Post by nikos »

I believe I fixed the problem, see the other post for the download link
I tested it on 30K photos, and although it takes some time to generate the thumbnails, it works without crashing
Post Reply