how's that possible? learn all in today's blog found athttp://www.zabkat.com/blog/17Feb08.htm
blog: go faster by slowing down
Moderators: fgagnon, nikos, Site Mods
Thank You
We discovered (and remedied much as you did) the very same issue albeit for a different application.
Where we differ is in the chastisement due the clownshoe at M$ who originally coded the copy routine. I think his/her name and current phone number should display on the screen during the inept process.
Multi file copy delay drove me to distraction and I complained bitterly to my programmers until they made it clear that Windows itself was to blame. I'd always suspected M$ code was not always optimal, but the Redmond routine is really poor.
Where we differ is in the chastisement due the clownshoe at M$ who originally coded the copy routine. I think his/her name and current phone number should display on the screen during the inept process.
Multi file copy delay drove me to distraction and I complained bitterly to my programmers until they made it clear that Windows itself was to blame. I'd always suspected M$ code was not always optimal, but the Redmond routine is really poor.
Interesting. On my older P4 system with all 7200 RPM IDE drives, I have never really noticed a dramatic slowdown even when copying extremely huge files (multiple GB sized files). However, involved a USB 2.0 device and the story changes entirely.
Granted, there *is* enough of a slowdown that I wouldn't want to be caught doing something time critical *while* doing th copying - however, from the sound of the blog its as if you're saying the the entire system grinds to a halt, which has not been my experience.
Then again, my systems have always been over-powered in terms of memory - I built my XP box on 2002 with 1 GB RAM, and my current iteration is running 3 GB for Vista - so that in and of itself may be a deciding factor....
Granted, there *is* enough of a slowdown that I wouldn't want to be caught doing something time critical *while* doing th copying - however, from the sound of the blog its as if you're saying the the entire system grinds to a halt, which has not been my experience.
Then again, my systems have always been over-powered in terms of memory - I built my XP box on 2002 with 1 GB RAM, and my current iteration is running 3 GB for Vista - so that in and of itself may be a deciding factor....
Copying slowdown
We found the problem to be consistently on every tested system... including those with 10,000 RPM drives, 8GB of RAM, Quad processors, threaded code, etc. In the application I was referring to, moving a set of X-rays with file sizes of 50-120 MB, the maximum we could safely handle simultaneously was eight; any more had to be queued.
Once we modified our application's code to accommodate Billy's Bug Farm, the issue was addressed --- to the extent possible. :(
Once we modified our application's code to accommodate Billy's Bug Farm, the issue was addressed --- to the extent possible. :(
Speaking of queues which you advertised in this blog entry. Is there any option which could default Copy To/Move To dialogs to queue files, if there any copy/move operations running? I keep forgetting about the queues when copying lots of files and end up with several copy operations running simultaneously.
I'm using Xplorer2 - the only file manager that does not suck. Actually, it rocks!
i mean to add a "use queue" checkbox instead of a separate button, which will do what you want, next version!
btw regarding speed and slowdown, i'm sure there's some hardware expert reading this that can illuminate us all on hardware architectures that are better than others in this respect; eg. is SCSI better?
btw regarding speed and slowdown, i'm sure there's some hardware expert reading this that can illuminate us all on hardware architectures that are better than others in this respect; eg. is SCSI better?
This is an old post, but...
You can't really blame MS for disk I/O performance dropping dramatically when you're processing multiple streams of data at once. That's just how mechanical disks work. Higher RPM and deeper command queueing on the drives help, but they're not solutions.
There isn't much you can do at OS level to mitigate this. You could try letting one process get bigger "I/O time slices", but then the other processes would be waiting and developers would be yelling at this. Vista tries to remedy the situation by introducing disk I/O prioritization, but that's mainly useful for separating background, normal and high-priority data streams.'
It's up to application programmers to try and do as little head seeking as possible. For file copying, this is easy: queue your copies/move (this is one of the really nice features of x^2 over explorer). For stuff like datbases, the task is much more difficult.
For slowdowns not related to head seeking, the biggest problem I know of would be drives running in PIO mode rather than UDMA, which results in close to 100% CPU usage with heavy disk I/O (less on multicore machines).
This is usually caused by defective hardware. How much CPU is used by external USB devices seems to vary depending on the chipset used on your motherboard. I've had systems with considerable usage, and systems with very little. Never made my system as a whole feel unresponsive unless a drive has been in PIO mode.
Dunno if SCSI by itself is better, if we compare drives with the same RPM, since SATA has gained stuff like Native Command Queueing from the SCSI protocol. What technology is better? Solid-State Drives. But even those aren't perfect, and in fact a lot of the SSDs currently available exhibit worse performance than mechanical harddrives on workloads where the "no moving parts" should be an advantage (who cares about 120MB/s linear write speed when random-small writes drop to 100KB/s? especially considering the size of today's SSDs).
PS: it's fine blaming Microsoft when Microsoft is actually to blame. But inefficiencies in mechanical disks aren't really their fault...
You can't really blame MS for disk I/O performance dropping dramatically when you're processing multiple streams of data at once. That's just how mechanical disks work. Higher RPM and deeper command queueing on the drives help, but they're not solutions.
There isn't much you can do at OS level to mitigate this. You could try letting one process get bigger "I/O time slices", but then the other processes would be waiting and developers would be yelling at this. Vista tries to remedy the situation by introducing disk I/O prioritization, but that's mainly useful for separating background, normal and high-priority data streams.'
It's up to application programmers to try and do as little head seeking as possible. For file copying, this is easy: queue your copies/move (this is one of the really nice features of x^2 over explorer). For stuff like datbases, the task is much more difficult.
For slowdowns not related to head seeking, the biggest problem I know of would be drives running in PIO mode rather than UDMA, which results in close to 100% CPU usage with heavy disk I/O (less on multicore machines).
This is usually caused by defective hardware. How much CPU is used by external USB devices seems to vary depending on the chipset used on your motherboard. I've had systems with considerable usage, and systems with very little. Never made my system as a whole feel unresponsive unless a drive has been in PIO mode.
Dunno if SCSI by itself is better, if we compare drives with the same RPM, since SATA has gained stuff like Native Command Queueing from the SCSI protocol. What technology is better? Solid-State Drives. But even those aren't perfect, and in fact a lot of the SSDs currently available exhibit worse performance than mechanical harddrives on workloads where the "no moving parts" should be an advantage (who cares about 120MB/s linear write speed when random-small writes drop to 100KB/s? especially considering the size of today's SSDs).
PS: it's fine blaming Microsoft when Microsoft is actually to blame. But inefficiencies in mechanical disks aren't really their fault...
Not being one to herald the saving grace of the technocracy, one thing science is very good at is studying/eradicating the minutia of it's own psychoses, such as the imperfect birth of mass-market SSD's.snemarch wrote:What technology is better? Solid-State Drives. But even those aren't perfect, and in fact a lot of the SSDs currently available exhibit worse performance than mechanical harddrives...
Still slightly beyond the cost mortal men are willing to part with, the technology is progressing, such as the OCZ Vertex. According to review (worth reading for historical perspective), and as odd as it sounds, this is one of the first to live up to expectations, overcoming the faults (including random R/W degradations) of it's older brethren. Give it a few more years for sizes to increase, and prices to decrease, and the end is nigh for mechanical.
One does wonder how the companies marketing Defrag utilities will cope. Never mind the multitudes of RAID 0+1 advocates. (Though, to be fair, if you have a decent RAID controller, the increase on mechanical is impressive.)
Kilmatead: yup, Vertex (and no other of OCZ's drives!) and Intel were the only two consumer level drives that were worth purchasing, at least a month or two ago (new drives have entered the market, and I haven't checked up on their specs). It's scary how ignorant the manufacturers of the SSDs have been, focusing only on marketing blitz and the (useless!) linear read/write speeds.
I would recommend the AnandTech articles on SSDs as they are some of the most thorough I've come across - and it's Anand's "fault" that the Vertex drives had firmware updated and perform properly
I would recommend the AnandTech articles on SSDs as they are some of the most thorough I've come across - and it's Anand's "fault" that the Vertex drives had firmware updated and perform properly
(I love resurrecting old threads...)
As an addendum, consider this news story: 1TB/s Transfer speed attained
Not quite a mass market product, to be sure, but the winds are a-blowin', and right-feisty winds they be.
As an addendum, consider this news story: 1TB/s Transfer speed attained
Not quite a mass market product, to be sure, but the winds are a-blowin', and right-feisty winds they be.
I use TeraCopy
I use TeraCopy, a free utility http://www.codesector.com/teracopy.php. There is a commercial version too.
It is a great utility: fast (especially compared to Vista/Windows 7), not a system resource hog, adds the ability to skip or overwrite individual files or all files, reports on the copy session, and integrates into the Windows Explorer and xplorer2 context menus.
Try it out you for yourself.
It is a great utility: fast (especially compared to Vista/Windows 7), not a system resource hog, adds the ability to skip or overwrite individual files or all files, reports on the copy session, and integrates into the Windows Explorer and xplorer2 context menus.
Try it out you for yourself.
A free alternative for TeraCopy is CopyHandler which I use.
Tux. ; tuxproject.de
registered xplorer² pro user since Oct 2009, ultimated in Mar 2012
registered xplorer² pro user since Oct 2009, ultimated in Mar 2012