SizeES: A Plugin for Fast, Persistent FolderSizes in x2 via Everything Search

[johngalt]

First of all, my apologies for such a late, late reply. My years played catch up with me in a not-nice manner.

Does this make this version the bastard halfling that is still better at everything than the pure stock version?

When I was young we used to steal those old fashioned Mercedes-Benz bonnet-emblems off the streets and use them as a poor-man's brass-knuckles, or at least we pretended as much. This version of the plugin is what the kids from your neighbourhood will use to adorn their keychains and mantelpieces for years to come. And if, by chance, one of those scallywag hoodlums should find a way to use it to become the street's newest discount dentist, then, yeah, this version is indeed the bastard halfling I would want guarding the integer representations of my folders.

Noted. I'll be sure to keep an eye on it then!

Or is that a perennially 56 years old birthday that you've settled on celebrating for the rest of your life?

Around 1.77 trillion milliseconds ago back in nineteen sixty-and-eight during the Ides of March, little Kilmatead came into the world with a dangling Gauloises off his lips and a bedraggled (yet still stylish) Fedora perched atop his head.

Wait, what? You preceded me 3 years, exactly, to the day?

I have friends who are Ides babies from '69, '67, and '73, but never have met a '68 Ides baby. Until now.

You'll find this part hilarious, in that my sister was born some years later on the anniversary of the Kennedy assassination. My only sibling, and we were both born on dates of infamy, both being assassinations, no less.

At least in my imagination he did. In reality I should think he was spanked, he spit, he cried, and promptly sought out the nearest teat. That in later years he misremembered that teat as belonging to La Lupa Capitolina is, of course, purely the product of whimsy, conjecture, and youthful exuberance. Don't all men reinvent the past to fit the form they are most pleased to inhabit?

That I raised Eric Hoffer to a hero's pedestal was not only to honour a stevedore philosopher's contrition to the hard labour of hard men, but also to draw attention to the last line of his wikipedia entry, which happily spits in the face of the Ivory Tower biographies of lesser-intellectuals:

To this day, no one ever has claimed to have known Hoffer in his youth, and no records apparently exist of his parents, nor indeed of Hoffer himself until he was about forty, when his name appeared in a census.

My continuous surprise at the many people who desperately cling to the identities with which they were assigned and burdened at the whim of officialdom, is always balanced by the multitudes whose imaginary claims of "ages not their own" parade quietly through their own biographies. I've never been bothered by getting older - I neither embrace it nor fear it - but I do like to study it. From the daily morning pains engendered by my workaday 8-tonne excursions, to the entropy which inhibits a willingness of the eyes to see unaided, the process is probably not one to be challenged or retarded by chemicals from, and ablutions to, the "miracles of modern medicine". Rather, I'd like to strive to a mindful stasis, remaining acutely aware of the diminishing sunsets and the lessening grip of mental disciplines pervading my days... not with regret, or sad refrain, but instead with the child's surprise and curiosity as to why it is he will never see "that" blade of grass again, or coax the same cicada up the same tree as he remembers he may have done in his yesterdays.

So yeah, in 3 years time I'll have 16-tonne of gravel delivered to your door, and I'll pit my Hoffer to your Rand, and we'll see which of our dead geriatric idealogues can fling the dark stones of their unrepatriated youths the furthest in mind and space.

If nothing else, you'll get a new driveway out of it, and I'll just have another twinge in my back to complain about the next day. Such is life.

Brilliant exposition on both the burden of identity that some, if you'll pardon the pun, burden themselves with, but also on the the part about chemicals, in that, other than my on-going love of the 'eins' - caffeine and nicotine (yes, we spell things boorishly sometimes), I cannot agree with you more about the massive need I have to avoid overt ingestion of all things branded as 'medications'. It's getting harder to do so, but I'm happy to say that I still am not on any 'prescribed' medications, and as my last physical investigator mentioned, I'm as healthy as a horse. It was touch and go for a bit last year, but I suspect it was more latent effects of COVID paired with a violent resurgence of my allergies, than anything else, and yet I still received a nice bill of health, my last few weeks notwithstanding.

As for the gravel - if you could see my so-called front yard, you'd note I have no need for it, sirrah, as I have carefully crafted my very own gravel-laced planting - taking over my entire front 'yard', as it were (with 147 plantings, last I counted).

But the visit itself will be cherished, for sure. We'll see if Rand is all she is cracked up to be, for sure. I've less doubt about Hoffer than her, to be frank. Or, perhaps, more appropriately, how her ideas are being (mis)used in modern society by fools that don't actually understand what she stood for in the first place.

[Kilmatead]

• Are you tired of living your life a quarter-mile at a time?

• Are you overwhelmed with dread, doom, and fatigue every time you hear someone use using old-fashioned phrases like "in the blink of an eye" or "faster than snapping your fingers"?

• Do you have a nagging feeling that you should toss out your old George Winston records and embrace your inner speed-metal demons?

• Do milliseconds just feel too slow - and indeed, passé even - the older you get as you uncomfortably reflect upon the fewer you have left?

• Does your significant other constantly inspire you to wonder, really, just what is an acceptable amount of time to waste furiously waiting for foldersizes?

• Do you conclude, as surely you must, when you regularly glance longingly at that yellowed portrait of Madame Curie you secretly keep in your night-table, that perhaps the half-life of polonium-214 would be a more appropriate amount of time to wait?

If you answered yes (or heck, even no?) to any of the above questions, you might want to rethink your life's ambitions.

In the meantime, we have v3 of the SizeES plugin!

Code: Select all

Changelog for 3.0.0.0

Added: Everything SDK3 (aka 1.5 SDK) integration: IPC uses Pipes (if available) instead of Messages
	- Reduces average folder query/response time from under 3ms to ~164μs
	- To reduce bloat, the SDK source included is a highly edited subset of the original (only code necessary for this plugin)
	- Suggested use of Everything Search version 1.5.0.1387a or later: https://www.voidtools.com/forum/viewtopic.php?t=9787
	- If the previous install set /alpha_instance=0 (for this plugin), this must now be reversed [=1] to take advantage of pipes
	- If Pipes are not available (ES 1.4 and older 1.5 builds) IPC will fallback to using messages

Added: INI option 'ForceCopyData[=0]' to force the use of Messages and ignore Pipes (mainly for debug testing)
Added: The Status.ES [X] column shows current IPC-type: e.g. '<Ok> (Pipe)' or '<Ok> (Msg)'

Fixed: Zero-size folders could not be distinguished between "not indexed" and "legitimately empty" (now they are)
Fixed: By default, errors on Root-folders were not deferred while non-Root were (long boring explanation would normally ensue...)
Fixed: Regression where disabling the plugin via the INI setting didn't always defer folders

Moved to GCC 14.2 LLVM 19.1.5 MinGW-w64 12.0.0 MSVCRT toolchain: https://github.com/mstorsjo/llvm-mingw
Refactored for -std=c23

TLDR: Reduces average response time to ~164μs.

Those aren't your grandfather's milliseconds, those are micro-bleedin'-seconds, or millionths of a second. In other words, yes, the half-life of polonium-214!

I can't take much credit for this - Everything Search now uses pipes under the bonnet for IPC instead of (the slower) WM_COPYDATA, so it's just a wee bit faster than your average explosive diarrhoea. Anyway, I added this to the plugin (the piping, not the plumbing), so... well, bon appétit!

As with all ironies of computer-type-stuff, if you had installed a previous version of SizeES and went through the steps to adapt it to the newer alpha-version of Everything Search, that must now be reversed (to benefit from pipes), so:

Open Everything Search (be sure to update it), and in the main search-bar, type:

/alpha_instance=1 <Press Enter>
/restart <Press Enter>

...and you're good to go. The recommended version of Everything has been bumped to 1.5.0.1387a or later. The plugin will still work with older versions (including the much older 1.4), but it will "fall back" to using the slower IPC instead of the nicer newer betterer stuff if it can't detect a pipe-server. Use the Status.ES [X] column to inform you which one it's using. If you don't see "<Ok> (Pipe)" writ large in the mirror of your skull as you brush your teeth in the morning, you're just not one of the cool kidz and should hang your head in shame and go back to bed.

I don't anticipate any further major updates to this plugin, barring pertinent changes to the (still-evolving) ES SDK source-code, so this can be considered a final version, more or less. Point-updates might ensue as needs or inspiration strikes. The download link in the original post has been updated, or just use this one.

Enjoy.

Incidentally, in the interests of stability I implemented the SDK in such a way that the client is created and destroyed for each individual folder-query, which sounds slow and impractical... while it is possible to leave a connected pipe "open" between calls to halve the times to around 60μs (yes, I tested it), but that leads to instability in reclaiming a link if either x2 or ES is closed and/or restarted during use. Given that possibility, I went for the more robust option, since most humans wouldn't be able to sense the delta unless they were on cocaine, which would also most likely be rather detrimental, hindering their ability to use x2 productively. The disappointment of living with sobriety asks, "Which is the lesser evil?"

(If there actually are a multitude of cocaine-obsessed productivity-based x2 users out there, we would love to hear from you. This is a safe space. We don't judge. We celebrate. They say sharing is the first step.)

Incidentally (Part II), The folder-queries are adapted to WSL (Windows Subsystem for Linux), where case-sensitivity matters. I don't know if x2 itself adapts to that or not, but at least we're future-proofed for when the end-times commence with all due heralding and pomp as we anticipate the crushing fall and inevitable overthrow of the current order.

Hang the rich.

[Tuxman]

this can be considered a final version, more or less.

I nominate you for the Nobel Prize. Finished software has been a myth for so long!

(If there actually are a multitude of cocaine-obsessed

I'm all ears!

productivity-based

Oh, ok.

The folder-queries are adapted to WSL (Windows Subsystem for Linux), where case-sensitivity matters.

Does it work with NTFS case sensitivity?

[Kilmatead]

Does it work with NTFS case sensitivity?

Interestingly enough, according to the lord high executioner himself...

The new Everything3_GetFolderSizeW API will lookup folders by case first.
A case insensitive search is performed if a folder is not found.
Diacritics are always matched.

In a largely case-insensitive environment like Windows, it would seem an odd choice to explicitly state that the initial search is actually conducted case-first... which suggests he knows something about the secretive high-cabal to which you worship that I do not. Therefore I can only assume it applies to all disc environments where such distinctions are possible.

Even more curiously, from my culling of the API source code, it appears that Everything handles all internal data explicitly and exclusively as UTF-8 (going out of its way to convert wide-chars into it), which would also be an odd choice if you didn't anticipate dealing often with the non-UTF-16 (wchar_t) world.

Again, I can't speak for whether x2 takes these things seriously or not ("nikos is as nikos does"), so your successes and failures may well be his epitaph etched in some of that leftover Notre-Dame stone.

Perhaps these drugs have a larger market-availability than previously thought. One can only hope.

[Kilmatead]

<Post Edited out of Existence>

[Tuxman]

Everything handles all internal data explicitly and exclusively as UTF-8 (going out of its way to convert wide-chars into it), which would also be an odd choice if you didn't anticipate dealing often with the non-UTF-16 (wchar_t) world.

One of the countless joys of developing software that is orientated towards compiler specifics via low-level development is the manifold differences that sometimes even occur between different versions of the same compiler. A wchar_t has at least 8 bits, but perhaps more. I would now have to reverse-engineer everything in order to obtain reasonably reliable information about which compiler is used there. So in short: If you squeeze EVERYTHING (hehe, you understand?) into UTF-8, then it should be possible to work compiler-agnostically with a handful of #ifdefs.

Thankfully, the days of 7 and 18 bit architectures (outside of banks, universities and the military) are over.

[Kilmatead]

A wchar_t has at least 8 bits, but perhaps more.

Well, except in WinAPI where it's guaranteed to be 2-bytes - mayhap only 8-bits of the 16 available are actually used, but Windows will kick your caboose to the next track over if you confuse variable-byte unicode for UTF-16. I don't think he's targeting compilers (based on his API providing named functions for ANSI/WCHAR/UTF8 explicitly), I think he's targeting a database format standard, probably in the interests of size-compression. What makes it interesting (to me) is that he actually wrote his own bidirectional wchar_t <-> UTF-8 <-> ANSI conversion functions (surrogates, et al) rather than trusting library functions to do it, so he must have had some elaborate grand scheme in mind.

I know he's using the VS compiler as he used some VS/MS-only extension-notations in the API code which I had to work-around to be able to compile it via GCC/LLVM in the first place. In the end I needn't have bothered as I just decimated 95% of the code anyway (including the stuff I "fixed"), so it was an afternoon of two steps forward one step back. If he was considering compilers other than MS's, he wouldn't have done that.

That being said, I have on more than one occasion used non-standard extensions in code just to frustrate and alienate VS users - but I'm a cruel goose that way. He actually wants people to easily use and adapt his code.

[Tuxman]

I know he's using the VS compiler as he used some VS/MS-only extension-notations in the API code which I had to work-around to be able to compile it via GCC/LLVM in the first place.

As far as my (albeit aging) memory does not deceive me, the VS compiler has been (a version of) LLVM for a few years now and is therefore reasonably standard-compliant. Of course, the same cannot be said of GCC, which, to put it with the necessary restraint, has been conspicuous for about two decades, above all with overzealous ‘optimisation’ (i.e. error generation). Are there any subtle nuances in the depths of the obvious nastiness with which Microsoft usually does its business that I should be aware of?

[Kilmatead]

Are there any subtle nuances in the depths of the obvious nastiness with which Microsoft usually does its business that I should be aware of?

Nothing so egregious that the usual sins can't be cleansed with time - though MS was always the last in line when it came to standards compliance - at least where the C-family was concerned - their users dragging them kicking and screaming into the light that GCC shone long before. (Can you really use a link from 2007 to support anything other than a curmudgeonly prejudice of a dying era?)

That being said, this is the R-rated part of the film where the camera pulls in from the sweeping vista of dystopian landscape ruin out the window to reveal the hopeful young lovers in bed, and the audience sees GCC and LLVM arms and legs all akimbo, sweat glistening, firm youthful body-parts entwined, staring longingly into one another's souls, completely unconcerned with the darker tones of the swelling orchestra, intimating, foreboding, already suggesting the next dystopian wind which is not to be perfumed with hope, but with the sodden salt of their future daughter's tears.

Intentionally bad scriptwriting aside (only Hollywood would use the word dystopian twice in the same sentence), progress is... progress?

[Tuxman]

(Can you really use a link from 2007 to support anything other than a curmudgeonly prejudice of a dying era?)

If the self-imposed task is to prove that GCC was the cause of interesting problems many years ago, which it was, then it makes perfect sense not to necessarily cite evidence from last week. I had taken it for granted that the situation has not improved since then and that increasingly clever optimisations lead to increasingly clever errors.

[Kilmatead]

I had taken it for granted that the situation has not improved since then...

The very definition of Curmudgeonly, indeed.

...and that increasingly clever optimisations lead to increasingly clever errors.

Betwixt v4.2 and v14.2, one might hope that they've discovered a whole lost civilisation of new and especially clever errors to inflict upon ye! And maybe fixed a few things along the way. I'm just in it to see what fun goodies c23 brings to the potpourri.

[Tuxman]

I'm just in it to see what fun goodies c23 brings to the potpourri.

If you are brave enough to specify C23 when compiling, you will at least be in for an entertaining series of surprises, including (but not limited to) white space being relevant, the consequences of which, of course, being implementation-defined. The joys of not using Python are less abundant than they once were, it seems.

[Kilmatead]

If you are brave enough to specify C23 when compiling, you will at least be in for an entertaining series of surprises...

It's not as exciting as it seems, and bravery has little to do with it. The current GCC and Clang versions are fully compliant to c23 (or at least as "fully" to meaning "nothing breaks"), and everything I've tested works fine. The relevancy of white-space is a non-issue because the only reason anyone would have non-compliant code is if they somehow turned off all compiler warnings for the last 7 years (c17), if not longer. Anyone who would have used white-space in such a contravening manner (placing spaces between hex/unicode prefixes/suffixes, etc.) deserves what they get.

Almost all the additions (and subtractions) are mostly just certifying things which compiler extensions have been doing for years anyway (standardising attributes, preprocessor-tricks, etc), or just common-sense stuff like finally removing triglyphs, making bool an integral type, and standardising what a nullptr_t actually is, instead of allowing each implementation to make up its own (is it zero, or void-zero, or void-pointer-zero?) - now it's nullptr, which (while nice) is a ridiculous number of characters to type for such a common thing. More things to blame on the C++ dweebs.

The most imaginative complaint I've seen (of people who have code which will genuinely break), are those who enjoyed using realloc "tricks" (0-size request == acts like free) which admittedly are exceptionally useful in loops), but despite these tricks being previously tolerated for years, they were never fully sanctioned, so the "complaints" are only superficially valid. When I was a kid we used to go to the cinema after school, and there was an unofficial "rule" which said if you don't actually leave the theatre when the film ends, you can just wait and see the film for a second or third time, and no one will kick you out. As kids often like to see films (or hear stories, etc) multiple times without tiring of them, this was a useful "rule". So, one day when a manager does finally say "lads, out" it triggers a distinctive (and irrational) entitlement issue. The realloc thing reminds me of that, so I can't share their grief (though I can see why they liked the "trick" in the first place - but at the end of the day it's nothing a simple wrapper couldn't cure).

Anyway, I've been compiling with it for months now and the world hasn't ended yet, so apparently the wacky-boffins at CERN got away with it again.

Postscruptum:

Curiously the (decidedly whiney) article about c23 I linked above contains a reference you might find familiar... and which may have led you years ago to start seeing demons where there are only CERN engineers...

...is a licence for the compiler to undertake aggressive opimisations that are completely legal by the committee's rules, but make hash of apparently safe programs; the confused attempt to improve optimisation ... spoils the language.

Beware the prejudice of knowledge.

[Tuxman]

Almost all the additions (and subtractions) are mostly just certifying things which compiler extensions have been doing for years anyway (...).

Let's assume, for ostensibly valuable reasons, that writing down lived practice for the purpose of standardising this practice in IT is a good thing. This assumption already breaks down with the second example that comes to mind, namely with regard to the web (or, as rather unpleasant contemporaries oversimplify it, "the internet"). Or do you think it's a sustainably good idea to declare the lived practice of Google, a company that earns most of its money with marketability, to be the standard and at the same time to impose DRM and comparable, sit venia verbo, crap not only on every web user, but also (and of all people) on web browser creators as an absolute necessity? I do not.

I admit that this example doesn't seem so obvious in relation to the C language. On the other hand, I'm afraid it's deceptive.

Curiously the (decidedly whiney) article about c23 I linked above contains a reference you might find familiar... and which may have led you years ago to start seeing demons where there are only CERN engineers...

...is a licence for the compiler to undertake aggressive opimisations that are completely legal by the committee's rules, but make hash of apparently safe programs; the confused attempt to improve optimisation ... spoils the language.

Beware the prejudice of knowledge.

The ironic subtext of this quote is even funnier than simply looking at it from today's perspective, because the whole truth is that C was already a confused attempt to improve optimisation. Do I hear the wolf warning of the hunter?