Syncing folders that have renamed files and subfolders

[richardk]

This question is about syncing and backing up files in general. I'm not sure xplorer2 is at all meant to do what I need but people here seem like they'd know about this issue.

The problem is how to synchronize folders of thousands of photos between my computer and external HD. I am constantly renaming photos, adding and deleting child and parent folders, moving photos or folders to new folders etc. on the computer. So if I did a sync, I can't see any way xplorer2 or any program anywhere could correctly deal with files and subfolders that have been moved to different folders. If two folders are compared, the program couldn't tell the difference between a file that was missing because I wanted to delete it permanently and one that was missing because I moved it to another folder or renamed it - correct? It seems the only solution would be delete all 88 Gb of photos from the external HD and re-copy them every time.

[nikos]

I was thinking about this too the other day about my music collection. Just moving a folder around is enough to screw up any synchronization. So xplorer2 and no other program I can think of can tackle this problem, not even moving folders (never mind renaming as you do)

a rough check is to count the total files in one collection and then for the other, and see if they are close.

[richardk]

So then how do you yourself back up your music? As a programmer is this not also a problem for your work?

Theoretically one could use continuous live backup to cloud storage but in my experience that slows down the computer enormously - completely unacceptable. This is on a Thinkpad T520 (their 'business' line), Intel i5, 4 GB, Win 7 Pro. I know it's not due to a slow computer because the slowing down was happening from the first day I got the computer, and whenever I turned off the live backup the speed went back to normal. Even then I was only backing up maybe 1 Gb of text docs and not music or photos at all!

[nikos]

I suffer the same fate as the rest of the non-programming lot
until some programming super star finds a solution, an acceptable policy is to backup your folders willy-nilly, with whatever are the latest names and folders, and from time to time do a duplicates check to get rid of the extra files. The duplicates checker can also work without filenames so it is rename-tolerant

[richardk]

duplicates checker can also work without filenames so it is rename-tolerant

My impression is that an mp3's size can change when you edit the tag. It certainly must change if you add an album cover. If that wasn't the case, size would be a good way to check duplicates since mp3s will almost never have identical sizes. Date and time won't work because (1) Microsoft's created/accessed/modified dates are outrageously dysfunctional with little relation to the real world date/time the file was created/accessed/modified and (2) even if they were correct, unrelated files will end up with identical date/time if you do a bulk rename.

So is there any other trait to use for duplicates checking?

Likewise for photos. I'm less sure but I think maybe their size can change by simply renaming.

[longfellow]

Have you considered using a dedicated synchronization tool, such as Beyond Compare? I use this to mirror various folders from my main drive to an external drive for backup. Works fine on my mp3s; catches additions, deletions, tag edits, cover art changes, etc. Might be what you need.

[richardk]

Have you considered using a dedicated synchronization tool, such as Beyond Compare?

No, I had never heard of dedicated synchronization tools, so thanks. I've spent many hours researching around for backup tools and never saw those mentioned!

The main thing is, can it deal with a file that you moved to another folder? On first glance the comparison features look good but AFAIK renaming or altering a file is a completely different thing from moving it to another folder.

[longfellow]

The main thing is, can it deal with a file that you moved to another folder? On first glance the comparison features look good but AFAIK renaming or altering a file is a completely different thing from moving it to another folder.

Yes, indeed, it will recognize all the changes you mention. Before syncing, the program shows a list of all the changes it will make so you can verify that nothing is amiss. There is a trial version, so it is easy to run some tests and see if it does what you want.

If you are unfamiliar with file syncing software, you might also want to check out SyncBack or FreeFileSync.

[nikos]

Likewise for photos. I'm less sure but I think maybe their size can change by simply renaming.

no it won't. But adding tags will probably do

as for the synchronization tools, I will be very impressed if they can detect folder reordering and renaming, so if you test these tools and you find they do work like that, please let us know

[Kilmatead]

I will be very impressed if they can detect folder reordering and renaming...

While I'm the biggest fan (and endless proponent) of the Beyond Compare programme, there are caveats to be taken into consideration.

The first is that it can't save you from yourself. That's just the way it is. There's no way any utility on Earth can track and record all your personal foibles, failures, and character flaws. That's what your wife is for.

The second is that Beyond Compare (as with any pseudo-syncro utility) is very complex and can compare pretty much anything to anything across multiple bird's-eye-esque hierarchies (especially strong in the "orphan"-display category), but it can't perform miracles, and nor does it exhibit its full potential on default settings. I've been using it religiously for years, but I still struggle when I need to re-configure it for something wonky.

That being said, there's very few things I've found it not capable of, but one of the benefits of a lugubriously pessimistic outlook is that nothing beats a good and well-thought out backup solution in the first place - and not just relying on a "what made sense at the time" sort of way.

Nikos, of all people, should not have unrealistic expectations that any single piece of software (given the vacuousness of the human mind) could ever hope to cope with the detritus of the human condition - never mind the mucky mess slovenly geeks are wont to create when left to themselves.

[RightPaddock]

I use a product called GoodSync. It detects renames and moves on source device/partition, so on syncing it can replicate such things at the target. I don't think it claims to detect everything, so it will do some deletes and copies; but for me it gets at least 90% of them without need to do a copy and probably more.

Don't ask me how it works, I no longer have any interest in knowing that sort of thing. But it runs a service task, so I would guess its monitoring the file system.

Available for Windows, OSX, Linux (including Raspberry Pi and some NAS boxes), Android and IOS (latter two are free). In a P2P setup it does block level synchronisation. And it can inter-op with the likes of Dropbox, Google Drive, Amazon, OneDrive etc. It's not free (USD30 single user) - there is a demo version that has very limited capacity.

I have it because at the time I was looking (circa 2007) for something of its ilk, it was one of the few (maybe the only) product that handled symlinks, junctions and hardlinks 'properly'

Kilmatead regarding the virtues of Beyond Compare as a file compare tool.

On Swiss Army Knives : If Jack wants something to pick stones from his horses' hooves and Jill wants something to sharpen her calligraphy quills, then its unlikely that either will be satisfied with a Swiss Army Knife, let alone one that they must share. The scissors, corkscrews, nail files, eyelash curlers etc tend to get in the way of the pick and the knife

RP

[nikos]

I wonder if we are talking about the same thing here. Let's say that at the beginning, the hierarchies are synchronized with 3 sibling folders as such
A - A
B - B
C - C

then one day I decide to move the folder B on my computer UNDER A

A
> B
C

I expect the next syncrhonization to detect a "loss" of folder and a creation of a new folder, so after it applies its synchronization we end up like this

A - A
> B - B
B - B
C - C

so syncing "detects" reordering but it creates duplicates. That's what I think these programs resolve rearrangements. They just blindly copy differences

am I mistaken?

[Kilmatead]

I wonder if we are talking about the same thing here [...] They just blindly copy differences [...]

The thing RightPaddock is talking about (by virtue of its services) would seem quite capable of making a real stab at it (as long as you've had it running for awhile). Other than that, these programmes should never be trusted to just "blindly copy" anything - where they excel is in informing the user that they should "take a closer look" before making rash actions.

And for taking that "closer look" there are none better than those discussed.

(You seem to be insisting on miracles here; since when did you become such an unreasonable pedestrian user?)

[nikos]

a guy further up claimed that these tools can do such amazing feats. I am trying to establish whether it is true or false...

[Kilmatead]

Yeah, longfellow might have spoken a bit broadly there (that's why I followed with tempering caveats), but even you know it's a common-sense thing. Short of running services to keep an (apparently) eternal record of SHChangeNotifyRegister reports, there's not many ways to keep your knickers from failing the "sniff test".