Anyway...
Which is why you dragged me back from the dead as usual to show you how it's done.nikos wrote:I saw that other FMs are embedding this Everything tool, I might consider it but wouldn't know how to go about it.
Well, to be fair, while this is a fully functional, if small, (but useful) aspect of integration, I'm not actually that much in favour of embedding Everything into x2 full-scale simply because it's not necessary. I will expound on that later on.
So, what is this and why would I want it?
The x2 option to "Calculate subfolders size automatically" seems like a good idea on the surface, but in practice has a few drawbacks... as the tooltip itself notes, it's "a major resource hog if enabled", slowing things down quite a bit when browsing around. Not to mention, in Nikos' "infinite wisdom" he disabled it on drive root folders, so it's not very practical - you have to hit <Ctrl+D> twice to force it, and then wait forever for it to tell you that your C:\Windows folder is 27GB in size. Yippee.
It's also not persistent - if you restart your computer (or even x2 sometimes) the foldersizes that it computed ten minutes earlier all have to be calculated again... and again... when you get up in the morning and crawl out of bed... again... and again...
"But what's to be done?", you cry, wringing your hands with the false modesty that only mock-desperation and first-world problems can provide, "And what's that Everything Search tool got to do with it? And isn't there already a 'Foldersize Shell Extension' which does the same thing?"
The Foldersize Shell Extension has a couple of faults - it is not persistent, its accuracy is questionable in situations where contents change rapidly, and there's just no user control about it. Sometimes it stubbornly shows what you know to be incorrect information, and just ignores refreshes. It's also been out of development for 20 years.
However, all is not lost. One of the useful things Everything Search can do when it's idling away its days in the background is to index foldersizes for quick reference later on. This index is both remarkably light-weight and quickly accurate for being updated in real-time, and, best of all, it's completely persistent so closing x2 or restarting your system won't cause any delays, it'll all reappear instantly..
For example, x2 tells me that my music-folder is 407.8 GB with 4980 subfolders and 37958 files. Yeah, that's 5,000 subfolders. x2 is a little grumpy about that when it comes to size-calculation - never mind that's already on a drive where other folders (in root) are commonly 1.4 to 2 terabytes in size. "Calculate this!" is what x2 says when confronted thusly - and it would no doubt accompany it with a rude hand gesture if x2 had hands. Or fingers. You get the idea.
Would you want your file-manager to flip you the bird? Probably not. So, I offer this small solution. The plugin itself is quite robust and has a few options (for those more curious souls) to play with.
Installation:
All you need is the x2 Plugin Manager (no installation required), and, of course, the plugin itself: SizeES v3.0.1.1
Please note that xplorer2 version 6.0.0.1 or above is required. If installed on older versions it will still load, but it won't actually do anything.
Due to significant changes, the older version of SizeES v0.0.0.3 is now officially legacy, but is still available to download here for anyone interested (it will still work properly for older x2 versions).
Instructions:
Extract the archive, run the plugin manager, then drag-&-drop either the 64-bit (WDX64) or 32-bit (WDX) plugin into the window and click "Apply".
Now this is where it's a little different from other plugins, as you only need to use the normal Size [S] column from x2, though initially you'll want to activate the Status.ES [X] column as well which will essentially tell you whether or not it's all working correctly. Once you're sure it's working properly, you can just turn off Status.ES, it's only there for troubleshooting.
Everything Search must be running in the background, recommended version 1.5.0.1399a or later. While the plugin will actually work with the older v1.4 version, there's no reason not to use 1.5 - don't let the "alpha" tag scare you away, it's quite stable and has been in development for a few years with frequent updates. The plugin will work whether Everything is "installed" or portable - it doesn't matter.
If running as a standard user account, the Everything Service must be installed, though this is not necessary if you are an Admin user.
In the Everything menu-bar go to:
Tools -> Options -> Indexes -> Check both "Index Files" and "Index Folders", then click "Force Rebuild"
That's it. Just give x2 a quick refresh.
In terms of speed, if using the newer version (v3+) with Pipes enabled, the arithmetic mean averages out to around 55μs (micro-seconds) latency per folder reference, and if using the older Msg-based IPC, slightly below 3 milliseconds per folder reference. It's persistent (nothing needs to be recalculated if you restart x2 or your PC), and won't suffer long delays like x2's column does when computing large folders in the background.
Troubleshooting:
As of version 3.0.1.0 the zip file also contains a "debug version" of the plugin which will create a log file in the same folder as the plugin file. The log may help clarify what is or is not occurring, though it is primarily intended to provide latency-timing information during runtime. Only use the debug version for testing or troubleshooting, it's not intended for full-time use.
Possible messages (displayed in the Status.ES plugin column):
Code: Select all
<Ok> - All is good: Sizes are read from the Everything index and used directly in x2's native "Size [S]" column
<No ES IPC> - Everything Search isn't running
<No Index> - The drive/folder is not an indexed location (or, "Index Folders" may be unchecked in Everything Search's options)
<No Plugin IPC> - The SizeES IPC receiving window failed to register
<Query Timeout> - A query sent to ES was not processed
<Reply Timeout> - A reply from ES was expected but not received
<Unresolved> - An initially unindexed result tried to "fix" a failed path (links, long paths, etc) and try again, but failed
<No Memory> - If this were to ever actually occur, I would spontaneously combust into toasted marshmallows
<Invalid Plugin Name> - Plugins may ordinarily be renamed (to change their column-suffix), but in this case "ES" is mandatory
<x2 v6 Required> - Plugin will only function with x2 version 6.0.0.1 or above
When you start x2 there will be a temporary message displayed on the statusbar saying which version of Everything you're using, and/or if it's communicating with x2 correctly.
You said there were options? What are they?
Just open the SizeES.ini file (automatically created wherever you placed the plugin itself). You can also right-click the plugin in the plugin manager and select "Edit Plugin Settings". A restart of x2 will be required for changes to take effect. In the download there is a ReadMe.txt file which includes more detailed instructions relating to setup and the following options.
Code: Select all
OnErrorDeferRootFolders = 1
OnErrorDeferNonRootFolders = 1
OnErrorDoNotResolveUncPaths = 1
StartupNotification = 1
SustainPipeConnection = 1
ReparseTreeCacheSize = 500
EnableFallbackIPC = 1
Timeout = 250
; To view properties from Everything Search in x2, they may be declared below
; Use 'property:' followed by the property/column name as displayed within ES
; Qualify with 'folders:' and/or 'files:' to target object types
; To match extensions, declare 'files:png;jpg', or 'ext:png;jpg', or '!ext:png' (all-except)
; By default the column name will be that of the property, use 'alias:' to substitute a custom label
; Declarations are not case-sensitive and their elements may appear in any order
[ColumnsES]
EnableCustomColumns = 1
AllowNonIndexed = 1
IndicateNonIndexed = 1
DisableAliasing = 0
CreateAllSlots = 0
IndicateNullText = "\x00B7" // Raised point
Custom_0 = folders: raw: property:"Descendant Count" alias:"All Children"
Custom_1 = folders: raw: property:"Descendant Folder Count" alias:"Folder Children"
Custom_2 = folders: raw: property:"Descendant File Count" alias:"File Children"
Custom_3 =
Custom_4 =
Custom_5 =
Custom_6 =
Custom_7 =
Custom_8 =
Custom_9 =
Random thoughts about integrating Everything Search and x2:
As I said originally, I don't see complete integration as being useful - for one thing, sticking a toolbar button on x2 to open Everything Search works just fine, and Nikos could (and should) never re-create the custom-built interface that already exists in ES. It would be nice to "automatically" transfer search results into a scrap-container for use in x2, but frankly, copy and paste (into a scrap) does the job - if not being very elegant.
Curiously, ES and file-managers share a common observation amongst their users: Most people only use about 1% of what the programme can actually do. Certainly, when it comes to Everything, people are always impressed with the speed - and it's true - at least when it comes to searching by name. But if you look behind that simple superficial stuff, ES is an amazingly well thought-out and powerful tool which is essentially hidden behind a command-line interface.
Much like Regular-Expressions... if you know how to use them, they are tremendously powerful - but if you don't, it's all just noise to the common user. Everything Search is a command-line paradigm's wet-dream of concision, accuracy, and elegance. But only to the user who takes the time to read the voluminous documentation of how to use it. For everyone else it's just a flashy and superficial "Wow, look at that" sort of experience, absent a true understanding of what's really possible.
Except for this folder size thingy. This bit was an obvious candidate to integration.
Technical Stuff:
As this now uses the native x2 size column, I obviously don't have any control over that display, so incorporating status messages (in case of misconfiguration or error) has to be done separately using the Status.ES [X] column. You only need to use this column if you are having troubles or suspect SizeES isn't working correctly - when finished, just turn it off.
To make things easier for the end-user, if the plugin cannot access the ES index for whatever reason, the column will just fall back to using its old slower routines for computation. This is both a good and a bad thing (you get the size either way), but it may not be immediately clear if it's working or not, unless you notice the column becoming very slow (which won't happen when SizeES is configured correctly).
To help with this, the INI includes the option 'OnErrorDeferNonRootFolders' which by default is set to '1' (enabled) which means x2 will kick-in and do it's thing. A good reason to disable this setting is that if anything were to go wrong with how ES is configured, it will become immediately obvious to the user as all foldersizes will be instantly shown as zero bytes, and you can then turn on Status.ES to see what may be wrong.
Note that x2 only calls the plugin for folder sizes, not file sizes - those are actually faster when handled natively, so the Status.ES column is blank for files.
Ordinarily, with a project like this, any integration with Everything would go through the SDK and distributed DLL the author provides as part of the kit, as the IPC (Inter-Process Communication) code depends on it. However, it can all be handled manually, so you don't really need any of that. This plugin is stand-alone (as far as not needing the DLL or even including the SDK itself). It does, however, require ES to be always resident in the background, but that's a normal part of ES's installation process to just sit in the tray anyway and is completely harmless. You don't even have to use it if you prefer other search programmes for whatever reason. Even though this plugin is constantly spamming ES as you browse, it won't interfere with anything you're concurrently doing in ES itself.
Please note that the plugin adds three columns into x2, Status.ES [X] (explained above), Reserved.ES [X] (for now it's the same as Status.ES), and the more curiously named uint64_t.ES [X]. This last one is just the bridge used so x2 can pick up the information from the plugin. You do NOT need to activate, select, use or view this column in any way, x2 does that automatically in the background.
The source-code is included, and it's easy enough to compile yourself if desired. Coincidentally, ES is written in C too, so take that you Bjarne Stroustrup fan-boy commies!
Full changelog:
Code: Select all
3.0.1.1
Fixed: Allocated parsing buffer (INI) was insufficiently sized for very long delim & param declarations (extended to 2K)
Fixed: Format logged hashes as %016llX
Fixed: Missing plugin version in status bar notification
3.0.1.0
Added: Any property from Everything Search may now be imported (previously it was limited to only a practical subset due to data-type limitations)
- Any number of columns may be declared in the INI (not just the 10 slots provided by default), keynames must be numerically contiguous
- Properties render using ES formatting with the caveat of being textual columns, so suffer the usual restrictions (no "true" searching/sorting/justifying)
- Defining "raw:" removes formatting and returns the data-type verbatim, adjusting the column-type as needed (not all properties unavailable)
Added: Improved column/property array scaling (if you were to lose control over your mental faculties you could have all 400 columns co-extant without memory penalty)
Added: Reparse-Tree folder-caching removes the latency incurred from re-checking previously browsed branches known to have no links
- INI option: ReparseTreeCacheSize[=500], the number of paths to track in a circular buffer (set to 0 to disable entirely)
- The above setting replaces the previous SizesIncludeReparseTree option
- If a new reparse-point is created or found in a previously cached location, any associated paths are removed from the cache allowing the size to report correctly
- When both 'Attributes' and 'Reparse Target' properties are indexed within Everything Search, the Reparse-Tree is automatically used (disabled otherwise)
Added: INI option: IndicateNullText[="\x00B7"] to distinguish a blank/empty string value from an unused property - by default: bullet point '·'; use "" to disable
Added: Escape-prefix ('\') for quotation marks, forward/back-slashes, spaces, and inserting 4-digit hex character sequences (\xhhhh)
Fixed: Latency on custom property queries reduced from ~10ms to ~1-3ms
Fixed: Filtering file-extensions for custom-columns was "off with the faeries" (indeterminate matching, limited extension-length buffer, etc)
Fixed: Predefined custom columns rendered textually by default, are now numerical ('raw:') for improved integration
Fixed: Blocking of unindexed properties (AllowNonIndexed = 0) did not report a warning as intended (preserves existing column order)
Fixed: Long-paths in reparse tree recursion could unexpectedly unwind (regression)
Fixed: If attributes are indexed, all direct filesystem references (except INI reading on startup and link resolution) are optimised away
Updated to ES SDK v3.0.0.8 (elided for size & minor LLVM/GCC adaptation)
Updated INI file conformance version
Updated ReadMe.txt file
3.0.0.6
Fixed: Legacy IPC failure regression (again!)
Fixed: UNC prefix path modification
3.0.0.5
Added: Custom declaration support through INI to add up to 10 properties from Everything Search for use in x2 (see ReadMe.txt for full details)
Added: Option to expose the accumulated sizes of all reparse-targets further down in the tree, including sublinks
Fixed: Resolving deep paths in secondary-column queries caused ES to block indefinitely
Fixed: User-filters within ES could interfere with prefix modifiers the plugin reserves for IPC calls
Fixed: legacy IPC failure regression
Fixed: Removed necessity for the end-user to concern themselves with setting/unsetting the ES alpha_instance for compatibility
Fixed: Initial defaults for a customised INI could require an additional x2 restart to register
Fixed: Resolving path references could repeat unnecessarily if the attributes didn't match
Fixed: Columns other than Status.ES remained available despite being non-functional (failed validations, disabled options, etc.)
Updated to ES SDK v3.0.0.4 (Elided for size & minor LLVM/GCC adaptation)
Updated INI file conformance version and layout
Updated ReadMe.txt file
3.0.0.3
Added: Pipe connections remain open between queries, reducing average query/response latency to ~55μs (will auto-reconnect if lost)
Added: Path resolving/re-querying won't be attempted by default for UNC paths, speeding up browsing of unindexed shares
Added: The older message-based IPC may optionally be disabled completely, removing all background windows, events and callbacks
Added: INI options: SustainPipeConnection[=1], OnErrorDoNotResolveUncPaths[=1], EnableFallbackIPC[=1]
Added: An obelus symbol '†' is appended to the Status.ES column result where paths were resolved and re-queried
Added: A plus symbol '+' is appended to the Status.ES column IPC-type when SustainPipeConnection is active ('Pipe+')
Fixed: Browsing through reparse-points would fail to resolve sizes even if the original link-target was itself indexed
Fixed: Deep paths could fail to resolve sizes in some instances (transition & prefixed folders)
Fixed: The world wasn't fully prepared for the WinXP renaissance when (not if) Win11 inevitably dies its deservedly ignominious demise
Fixed: Regression where sneaky underhanded unscrupulous decorations sullied the 32-bit plugin making it rather unhappy
Updated INI file conformance version
Updated ReadMe.txt file
3.0.0.0
Added: Everything SDK3 (aka 1.5 SDK) integration: IPC uses Pipes (if available) instead of Messages
- Reduces average folder query/response latency from under 3ms to ~150μs
- To reduce bloat, the SDK source included is a highly edited subset of the original (only code necessary for this plugin)
- Suggested use of Everything Search version 1.5.0.1387a or later: https://www.voidtools.com/forum/viewtopic.php?t=9787
- If the previous install set /alpha_instance=0 (for this plugin), this must now be reversed [=1] to take advantage of pipes
- A Pipe-connection is attempted first, but if unavailable (ES 1.4 and older 1.5 builds) IPC will fall back to using messages
- The Status.ES [X] column shows current IPC-type: e.g. '<Ok> (Pipe)' or '<Ok> (Msg)'
Fixed: Zero-size folders could not be distinguished between "not indexed" and "legitimately empty"
Fixed: By default, errors on Root-folders were not deferred while non-Root were
2.5.0.5
Added: INI option 'Timeout[=250]' to set max wait-state in milliseconds - applies to all IPC, outgoing and incoming
Added: Queries & Replies share a changing ID to guarantee 1:1 syncing (errant or mis-queued messages are discarded)
Added: HWND_MESSAGE receiver integrated with main thread for stability, removed superfluous thread and message pump
Added: WM_COPYDATA messages explicitly allowed through the message filter for IPC between processes of different elevations
Added: IPC windows for both ES 1.4 and 1.5 automatically detected (removes need for /alpha_instance=0 step in ReadMe.txt)
Added: Status bar messages for failed IPC receiver, incorrect x2 version, invalid plugin name, and disabled integration
Added: Further refined the range of Status.ES error types, priorities, and states
Added: Refactored for -std=c23
Fixed: A default search filter-alias was used which could be removed/modified in the ES settings (thus invalidating results)
Fixed: Ambiguously failed IPC messages didn't always resolve to timeouts
Fixed: Status bar notification wasn't multi-instance friendly
Fixed: Default settings would fail to be applied if the derived INI filename was incorrect/corrupted
Fixed: Receiver-window continued to be generated when integration was disabled
Fixed: If the status bar was modified by x2 while the notification is displayed, any new text was mistakenly erased
Fixed: Resource file had malformed encoding (interfered with editing/compiling in some environments)
Fixed: All source files regularised to UTF-8 from ANSI (LLVM is a touchy mistress)
2.5.0.2
Fixed: Regression causing ES version and indexing status polls to return incorrect values
2.5.0.1
Added: All IPC is handled directly by the plugin allowing for the removal of the ES 1.4 SDK
- Cuts package size in half
- Reduces average folder query/response latency from ~5ms to under 3ms
Added: Timeouts as fail-safe against possible ES/SizeES IPC thread hanging (if ES hangs for some reason, x2 will keep working)
Added: Expanded range of Status.ES message types and priorities
Added: Lower memory footprint by removing unnecessary long-path buffer allocation block
2.0.0.2
Added: Improved deference factors for zero-size results & reparse points
Added: Refactoring for clarity and efficiency
Added: ReadMe.txt file outlining installation steps and troubleshooting
2.0.0.1
Added: Automatic integration with the Size [S] column in x2 v6.0.0.1 or above
- No column selection is necessary, just use the native x2 'Size' column
- The legacy Size.ES column has been replaced with Status.ES which reports this plugin's functional integration with x2 on a folder-by-folder basis
- The legacy Weight.ES column has been deprecated (still available in the v0.0.0.3 download), replaced with Reserved.ES (same as Status.ES for now)
- There is now a third column uint64_t.ES which provides the bridge for Size [S] - there is no need to activate this column
- If an INI from a prior installation is found, it will be overwritten to enforce the new changes
Added: x86 build for users of 32-bit xplorer2
Added: INI option 'OnErrorDeferRootFolders[=0]' to protect against unnecessary xplorer2 processing of roots if ES fails or is not configured properly
- For example, instead of accidentally trying to calculate C:\Windows et al, it will simply return 0 bytes instead, indicating a misconfiguration fault
Added: INI option 'OnErrorDeferNonRootFolders[=1]' (as above)
- Disabling this shows all folder sizes as 0 bytes which makes an ES misconfiguration fault more immediately obvious
Fixed: If the plugin was renamed (and would thus fail to work), Status.ES didn't report the correct cause of the dysfunction
Fixed: Status.ES didn't report error when run on x2 versions below 6.0.0.1
---------------------------------------------------------------------------------------------------------------------------------------------------
Built using: gcc.exe -Os -Wextra -Wall -std=c23 -m64 [-m32] -shared -Wl,--dll -s -static-libgcc -Wl,--kill-at -municode -lVersion -lshlwapi
(Pass -F pe-i386 to winres.exe if targeting x86 from x64)
LLVM 21.1.2 MinGW-w64 MSVCRT - https://github.com/mstorsjo/llvm-mingw
---------------------------------------------------------------------------------------------------------------------------------------------------
