Sorry about the delay in response. I had to get some info on how it works since I didn't write the DEM reading.
On startup the program reads the demIndex.dat file which contains all the header info for the dems. That file should be too large. It also checks for new/deleted DEM files. So startup time shouldn't be bad.
The indexing part of TF is pretty fast, so having many DEM files shouldn't be much of a problem. But the deal killer is actually loading the DEM file.
DEMs are only loaded as needed, but if you need any piece of a particular DEM file you must load the entire thing. So in this respect the smaller the DEMS the better.
I've got all of AZ as quad-sized DEMs, but I rarely use it. I usually just use the 3D download. If I'm doing views of large areas it's often faster. But it is nice to have the DEMs for offline use and when the elevation server is down.