Author Topic: Managing Map .dat files  (Read 19688 times)

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« on: March 15, 2005, 12:48:05 AM »
Unfortunately, I'm still rather living in the dark ages and have limited memory on my laptop, making which maps I choose very important. Is there any chance of getting a feature integrated with TopoFusion or a seperate application to manage the map .dat files?

For example, it would be nice to seperate different levels of maps as well as topo and aerial for specific regions, and so on and so forth.

I know this is a little out of the way and if it's not possible, could release some kind of specs for the .dat files so other developers might be able to make a 3rd party manager?

Krein

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 1203
  • TopoFusion Author
    • View Profile
    • http://www.topofusion.com/diary
Managing Map .dat files
« Reply #1 on: March 17, 2005, 09:10:33 AM »
We'd be happy to release the spec for others to write a program.  Sure.

Are you a programmer that might be interested in writing it?

I'd find it useful as well, but have other features that continue to be of higher priority.

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #2 on: March 19, 2005, 02:05:14 PM »
At most (if lucky) I would be able to put together perl script(s) to be run from the command line, although this is probably way over my head.

I'd still like to see the specs though, who knows.

alizhan

  • Full Member
  • ***
  • Posts: 30
    • View Profile
Managing Map .dat files
« Reply #3 on: March 22, 2005, 09:17:15 AM »
Quote (Krein @ Mar. 17 2005,9:10)
We'd be happy to release the spec for others to write a program.  Sure.

Are you a programmer that might be interested in writing it?

I'm a programmer, and I might well be interested--especially if we can work out a way to easily integrate the result back into TF while protecting your proprietary assets. I'd need to know more about your development environment (e.g., VS 6, required libs, etc.).

Krein

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 1203
  • TopoFusion Author
    • View Profile
    • http://www.topofusion.com/diary
Managing Map .dat files
« Reply #4 on: March 25, 2005, 12:23:48 PM »
Guys, here is the struct used in the index structure:

typedef struct indexEntry {
   int x,y;          // x, y coordinates in UTM divided by TILE_UTM_SIZE, as used to query Terraserver
   int file;         // which mapsX.dat file this tile is found in
   int offset;       // offset from the beginning of the mapsX.dat file where the jpeg is stored
   int size;          // size, in bytes, of jpeg from mapsX.dat file
   char typ,zone;    // typ - tileset number, zone - UTM zone for x,y coordinates
} indexEntry;


The mapsX.dat files are simply binary files with jpegs (and gifs) written sequentially in them.  There's no struct that goes with them, although if you take a look at it we store all of the above information as a header (in ascii text) before each image.  This way the index file can be reconstructed from the maps.dat files.

So, if you find a matching x,y,zone,typ for a given tileset, you read it from the corresponding (file) mapsX.dat by seeking mapsX.dat to 'offset' position, then reading 'size' bytes.  It's fairly simple.

The code is in visual studio 6, with most of the code written in C as an active X control - GPSMaster.ocx.  However, I think any map managing tools make sense as standalone applications, unless you have something else in mind.

Anyone who is willing to work on programs to handle or process maps please post here and let me know how I can help.

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #5 on: March 27, 2005, 02:11:47 AM »
Quote (Krein @ Mar. 25 2005,12:23)
The code is in visual studio 6, with most of the code written in C as an active X control - GPSMaster.ocx.  However, I think any map managing tools make sense as standalone applications, unless you have something else in mind.

If the program is standalone it will most likely be limited to text only to manage the data, which will make it more difficult to select certain areas that you want vs. simply highlighting it on the screen. (You'll have to input the formatted region)
Anything more advanced, such as being able to select the region visually would already be basically 1/4 of TopoFusion. (At least in the sense of viewing map data)

I haven't taken a look at the structure yet, but are the data files organized in any manner? If not, wouldn't it increase TopoFusion's efficiency if images were properly arranged so for example it wouldn't have to go to map1.dat to get a block and then the middle of map4.dat to get the image block right next to it and so on. Indexing does help, but only to a certain point since the head on the HD will still have to move a considerable distance jumping between data files since they aren't any to small. Let me know your thoughts on this (flaws in my theory), maybe somehing else to work on.

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #6 on: March 27, 2005, 03:01:01 PM »
I've actually finished the user interface, I surprised myself (never written a non-web geared program).

All that's left theoretically is to implement the actual functions to do what the user selects. I'll probably begin with simple selection of map types (topo, landsat, etc).

How does TopoFusion handle .dat files that are bigger than 625mb?
Just wondering if I need to comply with that, or for example, allow users to customize output files for other media types.
(Cd's can hold more than 625 at least)

Alan

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 165
    • View Profile
Managing Map .dat files
« Reply #7 on: March 28, 2005, 01:23:07 PM »
TopoFusion can handle .dat files of up to 2GB (signed integer indexes).  There are split arbitrarily at 625mb to allow easy archiving to cd.

Alan

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 165
    • View Profile
Managing Map .dat files
« Reply #8 on: March 28, 2005, 01:43:02 PM »
The images are stored in the order they are downloaded.  So, usually, images near each other  are near each other in the map file.

I don't buy your claim that it takes longer to seek between files that are large in size.  The location of fragments of a file on the hard drive is arbitrary.

The location of an image in one .dat file may be phsically right next to one in a different .dat file.  Or two "adjacent" images in one file may be on different platters in the hard drive, or on different drives altogether (RAID).  Attempting to optimize file layout is pretty fruitless when the access order is not known until runtime, as it is here.

I don't think it would be worth anyone's time or effort to attempt to optimize the file layout.  I once attempted to pre-cache nearby areas of the map, but that effort was for the most part a failure, it simply didn't give much performance boost and just took up a lot of memory.

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #9 on: March 28, 2005, 01:44:46 PM »
Ok, I have a simple function that will store a specific Tileset to a new .dat
At this time I will not have it generate a new MapIndex.dat and let TopoFusion do it, even though it's pretty easy to do.

For anyone else who might be creating a script, I've figured out the tileset to number mapping sytem (tileset number), everything else seems pretty self explanatory.

Topo 2M: 9
Topo 4M: 0
Topo 8M: 10
Topo 16M: 1
Topo 32M: 11
Topo 64M: 2
Topo 512M: 4
Aerial 1M: 5
Aerial 4M: 6
Aerial 16M: 7
Aerial 64M: 8
Urban 0.25M: 12
Urban 1M: 13
Urban 4M: 14
Urban 16M: 15
Urban 64M: 16
LandSat 16M: 17
LandSat 64M: 18
LandSat 256M: 19

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #10 on: March 28, 2005, 01:50:36 PM »
Quote (Alan @ Mar. 28 2005,1:43)
I don't buy your claim that it takes longer to seek between files that are large in size.  The location of fragments of a file on the hard drive is arbitrary.

Just a theory. Since I haven't taken any classes that delve into this, or actually programmed anything for benchmark comparisons.

alizhan

  • Full Member
  • ***
  • Posts: 30
    • View Profile
Managing Map .dat files
« Reply #11 on: April 02, 2005, 09:47:59 AM »
Quote (jmstacey @ Mar. 28 2005,1:44)
For anyone else who might be creating a script, I've figured out the tileset to number mapping sytem (tileset number), everything else seems pretty self explanatory.

I don't see tileset 3 listed. Tilesets 0-2 are increasing by a factor of four each time (so, if 2 is 64M, then 3 = 64M * 4 = 256M), and Topo 256M is the only one in the interface not listed here (excepting the Combo items, which probably don't have their own tileset numbers).

So is tileset 3 Topo 256M?

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #12 on: April 02, 2005, 09:19:35 PM »
From what I could tell when collecting the numbers, Topo 256M is not an individual tileset. It appears to be just Tileset 2 zoomed out. At least that is the way it showed up when I did it.

Krein

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 1203
  • TopoFusion Author
    • View Profile
    • http://www.topofusion.com/diary
Managing Map .dat files
« Reply #13 on: April 03, 2005, 08:20:07 AM »
Nice job figuring it out, but I could (and should) have posted this:

tileMapping[TOPO_4M] = 0;
   tileMapping[TOPO_16M] = 1;
   tileMapping[TOPO_64M] = 2;
   tileMapping[TOPO_256M] = 3;
   tileMapping[TOPO_512M] = 4;
   
   tileMapping[AIR_1M] = 5;
   tileMapping[AIR_4M] = 6;
   tileMapping[AIR_16M] = 7;
   tileMapping[AIR_64M] = 8;
   
   tileMapping[TOPO_2M] = 9;
   tileMapping[TOPO_8M] = 10;
   tileMapping[TOPO_32M] = 11;

   tileMapping[URBAN_POINT25M] = 12;
   tileMapping[URBAN_1M] = 13;
   tileMapping[URBAN_4M] = 14;
   tileMapping[URBAN_16M] = 15;
   tileMapping[URBAN_64M] = 16;

   tileMapping[NASA_16M] = 17;
   tileMapping[NASA_64M] = 18;
   tileMapping[NASA_256M] = 19;


Tileset three is indeed Topo_256, and we do store it.  Topo 2,8,32 were added after the fact (early versions didn't support them -- and they are GIF files) so we had to create a tilemapping so that people wouldn't lose their previously downloaded maps.

Let me know how else I can help.

Krein

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 1203
  • TopoFusion Author
    • View Profile
    • http://www.topofusion.com/diary
Managing Map .dat files
« Reply #14 on: April 03, 2005, 08:21:14 AM »
Combos do not have a number (at least that is used in the map files) since they are not stored.  They're put together on the fly.

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #15 on: April 03, 2005, 09:55:39 PM »
Scratch the interface. Unfortunately my knowledge expired after getting the complete interface setup trying to integrate it with the actual commands (not enough debugging time) ':p'

If I get some time tomorrow afternoon, I'll make available what I have to be run from the command line. Simply execute via cmd and include any tileset numbers you want from the list above after the command. Not pretty I know, but it will get the job done until have time to delve into gui programming a little more.

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #16 on: July 24, 2005, 12:43:31 AM »
Got a little busy with school, life, work, you know the drill.
Finally caught up with this and here's what I've got.
All you need is php (cli)
http://mark87.com/personal/projects/tfmm/tfmm.phps
Save this to a file of your choice and execute from the command line. It will NOT work through a web browser without modification.

Unfortunately since it's windows there's no nice text interface like ncurses for linux you'll have to do with entering the desired tileset numbers from the chart above. The script displays the same chart for your convenience.

Quick Overview:
The purpose of this script is to provide some management of TopoFusion Map Data, hence the name TopoFusion Map Manager (TFMM).
Currently it allows you to specify desired tilesets and then saves them to new .dat files for your use.

At the moment this script is horribly inneficient and takes quite some time to complete, depending on the ammount of maps you have.
This will be optimized in the future however here is my current todo list and priorities:

+ Write graphical interface with php-gtk
+ Make an installer, bundling php-gtk
+ Optimize file transfer by fully parsing MapIndex.dat and organizing file transfer queue more efficiently (minimize # of fopen()'s required to read MapsX.dat files
+ Additional functions
   + Verify data (contact Terraserver and verify complete image (safe-gaurd
   against isp compression))
   + Region parameters


Got a suggestion, request, or comment? have a bugfix? Know gtk?
Drop me a line at admin@mark87.com or post here.

Krein

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 1203
  • TopoFusion Author
    • View Profile
    • http://www.topofusion.com/diary
Managing Map .dat files
« Reply #17 on: July 24, 2005, 06:50:14 AM »
Jon,

Thanks for the post.  It looks like you're making good progress on the util.  I'll give it a try and if it seems like it's working we'll make it available from our site.  Unless you'd like to put up a page on your site that we can link to?

Either way, keep up the good work.  Some additional functionality would be great.

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #18 on: July 24, 2005, 12:16:30 PM »
Don't use v1.0.0
As I was falling asleep last night I realised that I forgot to have the script update the offset that is written to the new MapIndex.dat file so TF won't be able to correctly retrieve the maps correctly with the generated index.
I'll get this fixed in the next version.

I figured out how to make an about screen in gtk ':p'

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #19 on: July 24, 2005, 12:19:48 PM »
Quote (Krein @ July 24 2005,7:50)
Some additional functionality would be great.

Any suggestions?

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #20 on: July 24, 2005, 12:33:38 PM »
Here's a quick fix.
Find $buf = fread($map_source, $tile['size']);
and add next line: $tile['offset'] = ftell($map_desination);

I haven't tested it though:;):

jmstacey

  • Full Member
  • ***
  • Posts: 38
    • View Profile
Managing Map .dat files
« Reply #21 on: July 29, 2005, 11:54:47 PM »
php-gtk only supports gtk+ 1 versions so I'll be holding off on any interface until php-gtk2 is released. gtk+1 just isn't very pretty and personally I don't think it's worth the work, and then have to recode the interface when php-gtk2 finally comes out.
I've looked at winbinder, but I'd rather not spend my time learning something that is soley for windows. (sorry, linux user here ':cool:' )

In the mean time let me know what additional functionality I can add since this involves the backend. This code won't change when an interface is added so no biggie.