lotsa memory usage during file itemization

[johncesta]

I have a folder that contains 123,000 files. I needed to delete the files. I navigated to the folder using xplorer. It took a while to itemize the files and then I noticed that the memory xplorer consumed just displaying the files was about 205 mb.

When I selected the files and tried to delete them the CPU% was overwhelming. I had to kill the xplorer process in order to recup the cpu% and the memory for other services. I tried this a number of times same results.

I had to use a different program, I wrote a winbatch script, to perform this process.

The winbatch process is currently running at 17 mb of ram and about max of 21 to 40% cpu usage.

Is xplorer written in c++ or c etc? I have another program I used for backup that did the same thing. When it came to selecting files for iteminsing and sorting the memory would jump to over 250 mb and the cpu% was at 100% until the process was complete.

John

[nikos]

well, 130000 files in a folder, you don't see that every day
to its defence, x2 isn't doing the deleting but windows, which obviously wasn't designed to deal with this extreme situation

in these cases good-old DOS comes to your rescue
if you type $ del *.* in the addressbar you'll be able to remove these files much quicker -- but note that del won't delete in the recycle bin, the files will be gone for good

[fgagnon]

wow - that's an extreme case!

& a quick check (205MB-8MB)/123Kfiles=1.6KB/file sounds a bit high for what I would expect to be necessary to display the files.
The few folders I have with hundreds of files appear to add an incremantal "tax" of only about 100-200Bytes/file (depends on filename length & # of columns in detail view mode).
Can you describe your pane setup? [tree? QuickViewer? single/dual pane? view mode(list/details/etc)] {& if details, how many & which columns}

oops/missed your post, nikos.

[johncesta]

The setup is "out of the box" no special pane views etc. Just installed and navigate to the folder that contains the files.

The files are all text files very small about 300 characters max. Getting to the folder and loading the files takes time about 30 seconds to a minute.

But, there isn't anything special about the setup like I mentioned a straight setup. I know it's windows doing the work but the memory and cpu when I run the winbatch program isn't bad at all about 10 mb ram and about 20 - 40% cpu varies and winbatch used windows as well to delete it looks like they just call win apis.

John

[nikos]

do you get the windows "file operation in progress" dialog with that program too?

[johncesta]

Yes, I do. First I get the dialog of "files to delete"

then the progress bar showing the files beign deleted and time remaining.


Thanks,

John

[nikos]

so you mean this winbatch program shows the windows delete progress dialog for all items but somehow consumes less resources? I can't see how this is possible unless it is deleting files in batches

[johncesta]

I am sorry that I didn't get a screen capture but that is the fact. It didn't delete the files in batches. I watched as it deleted them one by one. If you'd like I think I can do the same thing on a backup server. I believe the files are still on that server.

I can take a screen capture of xplorer loaded and displaying the 127,000 files and the memory at about 200 mb and then the winbatch program deleting them and the memory at about 17 mb or whatever I stated earlier.

If you want screens of anything else let me know and I'll produce them.


John

[fgagnon]

I thought I would like to see what happens for this kind of an extreme case myself, so I created a folder 'fatstuff' with 60000 files.
Sure enough, x2 [1.0.0.2] memory usage bloated to 120MB (to display contents in details view).
A "copy" to alternate pane (another temp directory: 'fatstuff2') took 18 minutes, beginning with 3 minutes of x2 being nonresponsive & 100% cpu usage,
followed by the windows copy progress box display & cpu usage dropping to 10-15% range for a while; then up to 90% range for last 5 minutes of the copy process.
x2 memory usage up to 240MB after copy complete;

next tried $ del *.* in address pane of 'fatstuff' directory, activating console²
That delete process took 11 minutes, with cpu usage near 100% for full time, & no progress indication (of course).

next I closed x2 & opened system's DOS command prompt console.
del *.* took 30 seconds for same content in 'fatstuff2'.

all of the above on w2k with 1000MB RAM
[edit]& all files & folders on local machine hard drive [/edit]

Isn't windows wonderful?

[johncesta]

So, you can imagine my results of 127,000 files!

John

[johncesta]

The winbatch folks tell me that their script function just uses a simple call to the following Windows Shell API:

http://msdn.microsoft.com/library/defau ... ration.asp

[fgagnon]

Actually I was going for 250K files, but was taking too long to make them, so I aborted the process 1/4 thru (I'm impatient).

But it seems like the message is to use a native DOS command prompt if at all possible for this task.
(Ignoring whether that's any way to organize files to have so many in one directory** -- but I suspect from the small filesizes you report that this is either a 'catch' directory for some automated process(es), or a situation that just got out of hand. )

** I'm sure we'll hear from narayan about this in morning

[johncesta]

It was a program that saved text files for info. It had been saving them for a few years. I agree it's out of the ordinary. But isn't a lot of the stuff we do?

John

[fgagnon]

@ out of the ordinary - sometimes these cases are good ways to find the limits of our tools.

Upon further 'playing around' {this time on XP-pro SP1} I find an evil threshold at selection size = 2^13 = 8192 files for copy/move/delete operations.
For selection size = 8191 files, most times operations were okay; although a couple of times the operations finished correctly but x2 never recovered to display the results.
At 8193 files, work started, slowed, hung.
At 8192 files, I got a 'move' to work one time; but s-l-o-w recovery for x2 to display the results.
& whenever x2 was trying to recover its senses, it was using max cpu (99-100%).
My conclusion is that a max selection size limit warning should be given by x2 to the user for attempts to copy/move/delete anything over 4K items in a single selection: advising of slow operations & likely freeze-up at or above 8K items. {This may only need to be an interim situation until 'batching' into 1K-2K item blocks is implemented to achieve linear (vs. an apparent quadratic) degradation in speed for these large selection operations.}

[nikos]

i don't know how you people conduct your experiments but i've just did a folder with 10000 items and had no problems whatsoever, deletion happened in something like 10 secs; del *.* was even faster

you have to keep in mind several things here:
* just to list 100000 items x2 needs a lot of memory
* when you copy/delete a selection you need another 500-600 bytes per item since everything is bundled in a single data structure
* finally the autorefresh after the operation takes a lot of time; it uses a trick that is efficient for small-ish folders but not too good for large ones

to me it would seem that you've challenged your system memory and as a result windows spent most time swapping memory than doing any real work. THere's no inherent limit of items x2 can handle, the bottleneck is your PC

i make no pretences, x2 was designed with speed, not memory, in mind. If most of your folders were of such mammoth sizes then you'd have a problem, but overall, if you consider the statistical distribution of folders, optimizing for the small guy makes much more sense!