Yes, a plugin architecture for new map sources would be great, that's the direction I was thinking of as well. With proper design, data could be either local on disk, or fetched asynchronously over the net from another source. And it would save you (Krein) from having to write code for each data source.
The big catch might be in reconciling datums. Ideally (in the long run), TF would offer the plugins services to convert coordinates between UTM, state plane, etc. to easily match the data source's layout, and map rectification to warp the image to the right projection. In the short term, the plugins would have to do that themselves. Probably not trivial; haven't coded any of it.