PROBLEM: NTFS brings disk access to its knees when large numbers of files are created. The problem is not simply the absolute number of files, it is most likely file naming conventions leading to a massively unbalanced B+ tree in the MasterFileTable [MFT] The solution is subfolders.
In my case it was caused by the quadkey devolution of global sat imagery and digital elevation data into files named 0100123.dem, 01001230.dem,010012301.dem, etc. The filename depth corresponds to the quadkey LOD making the names themselves highly recursive. At LOD 12 that’s 256k files for the northwest quadrant of the planet, a number that increases by a factor of 4 for each higher level. At LOD 15 and higher the number of tiny files for even a one degree band of information becomes considerable enough to choke Windows I/O.
Try to write them all in a single folder and Windows comes to a crawl, with the directory I/O processing eating several seconds per file, even on an SSD.
I suspect the hashing of the filenames results in a massively unbalanced B+ tree, which is the central structure of the Master File Table. In NTFS this file contains a reference to every file on the disk. There might be other issues, size of the MFT, fragmentation of the MFT–a defrag will fix that in Vista or beyond–but they pale compared to the B+ tree problem.
SOLUTION: Structured nested subfolders so the files get distributed in a fashion that results in more balanced hashing. In my case it was subfolders treed according to the quadkey. This not only cuts the absolute number of files per folder, it significantly shortens the number of leaves on the last branch of the B+ tree.
I/0 resumes normal pace–which still sucks, after all these years, and gets charged to Windows System Idle Process. But that is another story.
Paul
