SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

A collection of especially useful xplorer² topics and ideas. New users may find it helpful to look here before searching the other forums for information. >>>>>> Please post new material in the relevant forum. (New stuff posted here will be removed.) Thanks. -fg-

Moderators: fgagnon, nikos

Post Reply
Kilmatead
Platinum Member
Platinum Member
Posts: 4580
Joined: 2008 Sep 30, 06:52
Location: Dublin

SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by Kilmatead »

So there I was, riding through Paris in a sportscar with the warm wind in my hair, when Thanos snapped his fingers... and... poof...

Anyway...
nikos wrote:I saw that other FMs are embedding this Everything tool, I might consider it but wouldn't know how to go about it.
Which is why you dragged me back from the dead as usual to show you how it's done. :wink:

Well, to be fair, while this is a fully functional proof-of-concept for one small (but useful) aspect of integration, I'm not actually that much in favour of embedding Everything into x2 full-scale simply because it's not necessary. I will expound on that later on.

So, what is this and why would I want it?

The x2 option to "Calculate subfolders size automatically" seems like a good idea on the surface, but in practice has a few drawbacks... as the tooltip itself notes, it's "a major resource hog if enabled", slowing things down quite a bit when browsing around. Not to mention, in Nikos' "infinite wisdom" he disabled it on drive root folders, so it's not very practical - you have to hit <Ctrl+D> twice to force it, and then wait forever for it to tell you that your C:\Windows folder is 27GB in size. Yippee.

It's also not persistent - if you restart your computer (or even x2 sometimes) the foldersizes that it computed ten minutes earlier all have to be calculated again... and again... every day... when you get up in the morning and crawl out of bed... again... and again...

"But what's to be done?", you cry, wringing your hands with the false modesty that only mock-desperation and first-world problems can provide, "And what's that Everything Search tool got to do with it? And isn't there already a 'Foldersize Shell Extension' which does the same thing?"

The Foldersize Shell Extension has a couple of faults - it is not persistent, it's accuracy is questionable in situations where contents change rapidly, and there's just no user control about it. Sometimes it stubbornly shows what you know to be incorrect information, and just ignores refreshes. It's also been out of development for 20 years.

However, all is not lost. One of the useful things Everything Search can do when it's idling away its dull-days in the background is to index filesizes and foldersizes for quick reference later on. This index is both remarkably light-weight and accurate for being updated in real-time, and, best of all, it's completely persistent so closing x2 or restarting your system won't cause any delays the next time you want to see a folder-size immediately.

For example, x2 tells me that my music-folder's details are: F:\Musicae "Musicae" [407.8GB]: Sub-Folders=4980, Files=37958. Yeah, that's 5,000 subfolders. x2 is not happy about that when it comes to size-calculation - never mind that's already on a drive where other folders (in root) are commonly 1.4 to 2 terabytes in size. "Calculate this!" is what x2 says when confronted thusly - and it would no doubt accompany it with a rude hand gesture if x2 had hands. Or fingers. You get the idea.

Would you want your file-manager to flip you the bird? Probably not. So, I offer this small solution. Remember, this is more of a proof-of-concept for Nikos to turn into a proper column (without the limitations of plugins), but you get the idea. The plugin itself is quite robust and has a few options (for those more curious souls) to play with.

Installation:

This is a WDX "Content" Plugin; all you need is the x2 Plugin Manager (no installation required), and, of course, the plugin itself: SizeES v0.0.0.1

Extract the archive, run the plugin manager, then drag-&-drop 64-bit (WDX64) plugin into the window and click "Apply".

Once x2 has restarted, use <Alt+K> to enter the column selection dialog, and scroll to the bottom of the available columns list - there you'll find a new entry named "Size.ES [X]" - just double-click it and you're almost done.

Everything Search must be running in the background, recommended version 1.5.0.1371a or later. While the plugin will actually work with the older v1.4 version, there's no reason not to use 1.5 - don't let the "alpha" tag scare you away, it's quite stable and has been in development for a few years. The plugin will work whether Everything is "installed" or portable - it doesn't matter.

If running as a standard user account, the Everything Service must be installed, though this is not necessary if you are an Admin user.

If using the alpha-version, a preliminary step is necessary: Open Everything Search, and in the main search bar, type:

/alpha_instance=0 <Press Enter>
/restart <Press Enter>

Then, in the (Everything) menu-bar go to:

Tools -> Options -> Indexes -> Check both "Index Files" and "Index Folders", then click "Force Rebuild"

That's it.

Troubleshooting:

Possible error messages (displayed in the plugin column):

"<No IPC:n>" - One of the installation steps was done incorrectly (most likely Everything isn't running)
"<Lost Link>" - The target of a reparse-point or shell-link is unavailable (on a missing drive, was deleted, etc.)
"<No Result>" - ES was successfully called but returned nothing, "Index Folders" probably unchecked, or attempting FRN on FAT
"<?>" - Your divorce is final; you're free; is this what you really wanted after all?

When you start x2 there will be a temporary message displayed on the statusbar saying which version of Everything you're using, and/or if it's communicating with x2 correctly.

Keep in mind that obviously this isn't going to be as instantly fast to display as a normal x2 column, but it's quite fast nevertheless, and consistent, and won't suffer long delays like x2's column does when computing in the background.

There's probably some bad stuff too - come on, out with it! Tell the truth!

The main fault (as is true with any Total Commander Plugin column), is that you can't use it for sorting. Yes, this will put-off the power users, but again, I intend this as something that Nikos could incorporate "properly" into x2 later, so that problem can be solved only by him.

Two smaller faults are that the text is left-justified, not right-justified as one would expect with numerical columns. Again, ask Nikos. Also, the foldersizes won't be included in x2's "summary" of selected-objects "size" (on the status-bar), which is a pity. But - again - ask Nikos.

You said there were options? What are they?

Just open the SizeES.ini file (automatically created wherever you placed the plugin itself). A restart of x2 will be required for changes to take effect.

Code: Select all

[SizeES]
ColumnName=Size
SizeTags=" TB| GB| MB| KB| B"
BytesOnly=0 // Display all sizes only in bytes.  If you can convert TB and GB in your head, have fun!
UpdateFolderIndexes=0 // Not recommended - initiates an Everything update when x2 is started
ShowLinkTargetSizes=1 // The size displayed is the size of the reparse/shell-link destination object
StaticAllocation=1 // A debug option for those who wonder how the code works
ShowTickCount=0 // Again, a debug option that displays the milliseconds it took to referece ES
SearchFRN=0 // ES doesn't search by paths and names, instead (on NTFS only) search by FRN (more precise)
StartupNotification=1 // Show the statusbar message in x2 or not
Enjoy.

Wait just a minute - what is that SearchFRN option? WTF? Are you mad?

This is more an inside joke for Nikos (who doesn't fully appreciate just what Everything Search can really do). On NFTS each filesystem object has an index which is referenced by the combination of the 8-digit hex VolumeSerialNumber and the file-index itself. This ensures there are no duplicate objects, so Everything always returns the size of the folder the plugin actually asked for, and won't get confused by two folders named "Pokémon" and "Pokemon". This is why the ordinary Everything Search prefix is more awkwardly "exact:folders:diacritics:\x0022%ls\x0022".

Well, I did say it was an inside joke. It's funny if you're a programmer who's into filesystem stuff. You know who you are.

Random thoughts about integrating Everything Search and x2:

As I said originally, I don't see complete integration as being useful - for one thing, sticking a toolbar button on x2 to open Everything Search works just fine, and Nikos could (and should) never re-create the custom-built interface that already exists in ES. It would be nice to "automatically" transfer search results into a scrap-container for use in x2, but frankly, copy and paste (into a scrap) does the job - if not being very elegant.

Curiously, ES and file-managers share a common observation amongst their users: Most people only use about 1% of what the programme can actually do. Certainly, when it comes to Everything, people are always impressed with the speed - and it's true - at least when it comes to searching by name. But if you look behind that simple superficial stuff, ES is an amazingly well thought-out and powerful tool which is essentially hidden behind a command-line interface.

Much like Regular-Expressions... if you know how to use them, they are tremendously powerful - but if you don't, it's all just noise to a common user. Everything Search is a command-line paradigm's wet-dream of concision, accuracy, and elegance. But only to the user who takes the time to read the voluminous documentation of how to use it. For everyone else it's just a flashy and superficial "wow, look at that" sort of experience, absent a true understanding of what's really possible.

Except for this folder size thingy. This bit's an obvious candidate to integration. At least, I think so.

Technical Stuff:

Ordinarily any integration with Everything's SDK would go through the distributed DLL the author provides as part of the kit, as the IPC (Inter-Process Communication) code depends on it. However, alternatively (with a little bit of editing) it can all just be monolithically compiled into your own program. Thus, that's what I did, so this plugin is stand-alone (as far as not needing the DLL). It does, however, require ES to be always resident in the background, but that's a normal enough part of ES's installation process to sit in the tray anyway and is completely harmless. You don't even have to use it if you prefer other search programmes for whatever reason. Even though this plugin is constantly spamming ES as you browse, it won't interfere with anything you're also concurrently doing in ES itself. At least in ES v1.5 - not sure about v1.4.

Also, the plugin is x64 only, mainly because I haven't tested for x86. And does anyone really use 32 bit stuff anymore? The source-code is included, and it's easy enough to compile if desired. Coincidentally, ES is written in C too, so take that you Bjarne Stroustrup fan-boy commies!
User avatar
nikos
Site Admin
Site Admin
Posts: 15806
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by nikos »

:shock:
User avatar
nikos
Site Admin
Site Admin
Posts: 15806
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by nikos »

as we're closing easter it's only expected to witness resurrections, so Kilmatead welcome back from the dead! :D

yesterday I didn't realize you were talking about folder sizes -- I thought you somehow integrated ES with xplorer2 via tc plugins...
but obviously that's not possible.
Kilmatead
Platinum Member
Platinum Member
Posts: 4580
Joined: 2008 Sep 30, 06:52
Location: Dublin

Re: SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by Kilmatead »

Wikipedia wrote:According to tradition, Lazarus never smiled during the thirty years after his resurrection, worried by the sight of unredeemed souls he had seen during his four-day stay in Hell. The only exception was, when he saw someone stealing a pot, he smilingly said: "the clay steals the clay."
Would that Lazarus of Bethany enjoyed such an attentive audience upon his recommence.
nikos wrote: 2024 Mar 21, 08:46 yesterday I didn't realize you were talking about folder sizes -- I thought you somehow integrated ES with xplorer2 via tc plugins...
but obviously that's not possible.
It's not like the word foldersize isn't, like, you know, in the frick-frakkin' title or anything!

Actual integration isn't really that difficult - for example I think TC itself did it in such a way that if you type into its addressbar and prefix your text with "es:" the rest of the line would just be issued to ES through the API, and the results integrated back into the window. This isn't that hard to do (the API is fairly expansive), but like I said, most people wouldn't know how to "make the most" of that typing experience, given ES's command filtering, and without the "instant feedback" that its own interface provides, the ergonomics would be... lackluster.

Anyway, mine is all about the folder sizes which (given how x2 chokes on drives) is a far more important issue - at least for a certain discerning and gentlemanly clientele of refinement that is. Heathens like you are happy with table scraps, as t'was always thus. :roll:

If you'd just try the bleedin' thing, you'd see my point. I'm sure you had a third-grade teacher at some time in ancient history say the same thing to you, but oh no, little Nikos just kept racing around the room banging into all the furniture and wondering why his head was lumpy and all the grown-ups in the room had that look of exasperated refinement on their faces.

Larzarus, indeed.

[Yes! THAT look! The one you're wearing right now as you read this! Quick, find a mirror, and witness doom itself!]

Hey, there's a lot to be said for this religious ranting and raving stuff. :D
User avatar
nikos
Site Admin
Site Admin
Posts: 15806
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by nikos »

I suppose you have much bottled up after ... what, 5-6 years in radio silence, so go on, blabber on ;)
you know about the other "official" foldersize solution for this purpose, right? But no problem reinventing the wheel!
Kilmatead
Platinum Member
Platinum Member
Posts: 4580
Joined: 2008 Sep 30, 06:52
Location: Dublin

Re: SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by Kilmatead »

nikos wrote: 2024 Mar 21, 16:07 you know about the other "official" foldersize solution for this purpose, right? But no problem reinventing the wheel!
<Sigh>

One day you will actually read my posts (which are trying to help you - you ungrateful wretch!), and you'd see that I already acknowledged its pallid existence...
Kilmatead wrote:The Foldersize Shell Extension has a couple of faults - it is not persistent, it's accuracy is questionable in situations where contents change rapidly, and there's just no user control about it. Sometimes it stubbornly shows what you know to be incorrect information, and just ignores refreshes. It's also been out of development for 20 years.
Ok, it's only been out of development for 14 years, however it was left not only unfinished, but flawed - which you would know, if you ever tried to actually use the things you purport to recommend! So, naturally, like any inquisitive cookie-monster, I've been looking for a better solution.

It turns out that ES does a far better job at indexing accurate, reliable, and perpetual foldersizes than other half-baked solutions. So, in fact, I'm not reinventing the wheel, I'm keeping the bloody thing turning, like Sisyphus and Atlas all rolled into one. And, worse for you, I'm trying (oh so desperately trying) to get you to see the forest for the trees. I'm even helping! I did the work to create a reasonable proof-of-concept, flawed only by your own imperfect implementation of plugins! Weirdly, it's you, you grumpy tyrannodidaskalos, it is you who make me do this to correct the mistakes and neglect you leave in your wake. In this world, as Machiavelli himself once said, "A man attains an elevated position only when his mediocrity prevents him from being a threat to others" - I am trying to be the breakwater here! For you, dear sir! For you!

<Ok, yes, I'm getting carried away. :D This is fun.>

Wasn't it Eric Satie who once said "Everything I undertake misfires immediately. I produce dirty rubbish and that will accomplish nothing." Now there was a decent bit of humility!

Do the Americans still pay people to be crazy evangelical ranters and ravers? I could do this for a living. :D

Is it that much to ask for you to implement accurate, reliable, and perpetual foldersizes when the work's already been done for you? Well, mostly. ES does the heavy lifting, and I just throw popcorn at you. We'll get there eventually.

Accurate.

Reliable.

Perpetual.

And fast.

Come on, daddy-o, dance to the groove! Make your middle-age count for something! Leave a legacy we users can bask luxuriously in!
User avatar
nikos
Site Admin
Site Admin
Posts: 15806
Joined: 2002 Feb 07, 15:57
Location: UK
Contact:

Re: SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by nikos »

I stand corrected but you know I'm busy writing that v6 thingy so if I read all your posts to the last word, I will never finish :)
Kilmatead
Platinum Member
Platinum Member
Posts: 4580
Joined: 2008 Sep 30, 06:52
Location: Dublin

Re: SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by Kilmatead »

Just remember that v6 needs substance more than a "facelift". Bah!

We'll also note that when those other FM's incorporated ES, they didn't just use it for search, they used it for this. Just sayin'. :wink:
User avatar
johngalt
Gold Member
Gold Member
Posts: 561
Joined: 2008 Feb 10, 19:41
Location: 3rd Rock
Contact:

Re: SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

Post by johngalt »

nikos wrote: 2024 Mar 21, 08:46 as we're closing easter it's only expected to witness resurrections, so Kilmatead welcome back from the dead! :D
Ye, gods, this!

I'm sure all the hardcore users are going to come out of hiatus as well and start asking for updates to old tools / scripts that you've generously provided over the years to hone X² as you have, @Kilmatead, as more than a few have lamented being unable to ask for updates.

Very glad to see you back!

As for the foldersize thing - excellent stuff, as always!
Image

Image
Post Reply