zabkat support forum

here's the comment area for today's blog post found at
http://zabkat.com/blog/optimal-disk-access.htm

The lads from StackOverflow agreed with you 3 years ago (so somebody's getting younger ), with one remarking that it's "Especially bad for writing, which is why the option exists, you don't get the lazy write-back. Which means that your program can only run as fast as the disk can write."

Using a virtual disk is somewhat unfair, considering that the (rather interesting) observations from MSDN's FILE_FLAG_WRITE_THROUGH vs. FILE_FLAG_NO_BUFFERING has a thing or two to note regarding the relevancy of actual data location:

When using FILE_FLAG_NO_BUFFERING, disk reads and writes must be done on sector boundaries, and buffer addresses must be aligned on disk sector boundaries in memory.

So you might still be shooting your deer out of season, regardless of how many revisits you revisit... which makes one wonder existentially, could revisits themselves be buffered?

your polymath highness failed to observe that the "experiment" only considers reading; it makes no statement either way about writing

Yes, but your post-skimming skimmeryness neglected to read the MSDN quote above, where I explicitly (because I knew you'd be... well... "you") emboldened the word "reads" in the phrase "disc reads and writes" just to stop you from being... well... "you".

Which apparently didn't work.

To reiterate (for the post-skimmers in the audience), MSDN included "reads" in that FILE_FLAG_NO_BUFFERING reference, not I.

For a file manager non-buffered I/O can be important when copying large (that is, very large) files around. If you just read and write a 50GB file using the normal cached functions the system cache will very soon consist of nothing but the file you just copied. This can dramatically affect the performance of the system and any other programs running at the same time.

nothing is impossible and I don't really know about this, but I would find it astonishing if MS guys are not aware of it and they don't provide at least some registry tweak to control such self-destructing behavior !?

nikos wrote:nothing is impossible and I don't really know about this, but I would find it astonishing if MS guys are not aware of it and they don't provide at least some registry tweak to control such self-destructing behavior !?

If you don't ask for non-buffered I/O, why shouldn't the cache manager cache? - I already mentioned this in the comment on your previous post.

Also, with regards to testing in a VM, be aware that the VM might be doing caching of its own. For instance, doing a Windows installation in a VM seems to go faster than doing it on physical hardware (I haven't wall-clock timed it, so my gut feeling might be wrong here).

Also, you should revisit Russinovich's blog about Vista file copying improvements - pay notice to the information about how the cache manager works (especially the block sizes being requested - if you request 4k data, the cache manager reads a lot more from the physical disk).

There's also my micro-benchmark which shows correlation between buffer size and read performance and CPU utilization, which shows that you won't be able to hit full disk speed with too small buffer sizes, and that after you've hit highest possible reading speed, there can be good reason to use larger buffers still, because of the CPU overhead involved in each ReadFile call. Also, keep in mind that 200MB/s read is slow compared to the speeds contemporary SSDs can deliver.

And one more thing: from what I've seen, even with FILE_FLAG_NO_BUFFERING, you'll get data from the cache manager if it was previously cached - this makes a lot of sense, but it's worth keeping in mind for benchmarks... especially since there's no API to flush the read cache , best you can do is either read in a dummy file large enough to evict all the other cache entries, or do a windows reboot.

nikos wrote:Of course the big problem is taking the windows cache out of the picture. How can this be done? a simple solution is to reboot the PC and run the timing code on "fresh" RAM — but that would be tedious and take ages.

snemarch wrote:since there's no API to flush the read cache , best you can do is either read in a dummy file large enough to evict all the other cache entries, or do a windows reboot.

Like any character happily oblivious to the world around them, I somehow missed this wee tool when it was first released. Recently doing some tests myself, and finding the Windows Cache being a nuisance Nelly, I ran across RAMMap (SysInternals) which is a GUI-based programme offering more information about your current machine's RAM usage than you ever cared to know about.

What's notable, though, is that it contains a menu-command called "Empty Standby List" - which, despite the name, really means "completely clear all disc-read-cached data". And it works like a charm, no rebooting, or other hocus-pocus necessary.

In fact, people found it so useful that they petitioned Russinovich to make a dedicated tool specifically for this purpose: EmptyStandbyList. No installation required, you can even ignore specifying any command-line arguments and just click it and presto - the disc-cache is wiped. A really, really, useful thing to add as a toolbar button.

In case anyone is interested in doing this manually through code (C/C++), they can check out this github snippet and just roll their own rather painlessly. (The code provided requires a bit of typo-fixing to compile properly, and contains some unnecessary/unused declarations) but - having just built it myself - I can verify it works as expected.)

If you ignore the privileging make-work, it really just boils down to an NtSetSystemInformation call - albeit one that requires a bit of work to get your head around and set up properly.

I love resurrecting old threads... noli turbare circulos meos...

Thanks for the EmptyStandbyList thingy, I use it prior to accessing my NAS with X2, it speeds things up considerably. NICE.

dunno wrote:I use it prior to accessing my NAS with X2, it speeds things up considerably.

As the Laws of Unintended Consequences go, that sounds more like a speeding ticket making you late for your wedding, and thus annoying your flibbertigibbet fiancée so much that she rejects you for a South-American dwarf-tossing circus star employee - ultimately saving you the trouble of a humiliatingly painful marriage based on her otherwise repressed carnal desires. (Hey, it happens!)

Actually using the StandbyList (what Windows does by default) is what speeds things up, whereas clearing it intentionally slows things down, or, at least, that is the purposeful, logical, and (generally) empirical result (also my reason for posting it here, contextually).

That being said, Networks are fraught with their own pitfalls, and perhaps your usage-model somehow exposes too many null-reference entries - and Windows caching them produces a slowdown... that clearing that sort of thing produces the tabula rasa effect is no surprise, but it should not actually speed anything up. That points more to a faulty NAS controller than anything else.

Not sure what possessed you to try it with the view of gaining access speed (especially where a network is concerned), but the placebo effect has been known to improve peoples lives inexplicably for centuries; oddly, not dissimilar to South-American dwarf-tossing circuses.

"Dem placebo's not for sale here mon", but it does make things faster with my NAS, and yes I'm sober

There is anecdotal evidence that supports your experience, so I don't doubt that the Loch Ness Monster could be somebody's secret love-child , but as long as no one goes looking for verifiable reasons why it might be so, it's harmless enough to swim in deep waters, though it will undeniably slow down repeated "non-network" storage access-times.

While there are different 'Priority' levels within the standbylist itself which can be cleared independently, splitting which datum from which set might be painstaking (for those who might want a more targeted approach).