Losing GPX files, Garmin USB problems?

[brian.r.hamilton]

I'm using a Garmin eTrex Vista C with USB.  After downloading (follow download in map, add tracks to logbook) the track & viewing it on the map (everything OK), I disconnected the GPS & went away from the computer.  When I came back (screensaver, password protected), the previous track was still displayed (highlighted actually).  I now had a new track to download, but Topofusion wouldn't communicate with the GPS, so I had to close & restart TF.  After doing this, I was able to download the new track just fine.  I save my GPX files using the date only (as in xx-xx-xxxx.gpx).  The new track was for a hike & the previous track was for a mountain bike ride.  I use different directories for each file, but they had the same date (same filename).  Next I wanted to look at the climbing data for each file but I found that my previous GPX file was not in the list of open files!  I believe it was missing when I restarted TF as well, but I didn't look for it until after downloading the current track.  I looked in the "biking" directory & the file was not there!  The hiking GPX file was in the hiking directory as it should have been, but the biking GPX file was gone - as though it was never written to disk or was wiped off the face of the planet when TF saved the hiking file.  My default directory in TF is the biking directory.  My TF logbook shows the complete entry for the missing biking file, but can't load the file since it doesn't exist.  I've had this happen at least 2 times now, each time losing valuable track data.

I really don't think I did this by accident but unfortunately I can't reproduce it at will.  I really think it is an obscure bug in TF that is somehow related to the USB connection troubles and/or some sort of logbook & filename issue.  I don't recall ever having this problem with my serial Garmin although similar loss of communication troubles did occur occassionally.  There seems to be some trouble with the Garmin GPS when communicating & a loss of GPS signal occurs with either serial or USB, although less often with USB.  There is definitely an issue with sharing the USB driver in TF because I always have problems if I have MapSource & TF running at the same time.  I can run ExpertGPS & MapSource together without troubles.  My guess is that there are driver resources that aren't being locked/unlocked properly.  I have no idea if this is also related to the file loss issue though.

I'll try to watch for this again but I'm a bit paranoid about losing my data now, so I'll probably be sure my previous file is always safely on disk before downloading another just out of habit from now on.  If there is anything specific I could test to help track this down, I'd be happy to do so.

[Krein]

Hi,

Nothing worse than losing data, so we'll take this one very seriously.

TopoFusion handles files of the same name but different paths without problem.  But there might be some issue there.  I'll try saving files as you did.

I can't think of anything that would cause a file that was previously saved to go away.  There's no code to delete or move files.

It sounds like both files have entries in the logbook, is this true?  The fact that the biking file made it into the logbook is troubling.  Initially I thought you looked at the data in the "merge tracks" dialog but had forgotten to save.  I suspected this because you said it was still highlighted.  TopoFusion is quick to "unhighlight" a track (almost any right click or change in mouse mode will do it), but if you highlighted the downloaded but not saved track in the merge dialog it will stay highlighted until you close the merge dialog -- no mouse clicks unhighlight it.  This brings up the issue that the program should warn the user if the merge dialog is still open and they are closing the program.  Since the data is displayed on the map you can sometimes be fooled into thinking the data is actually "there" (saved).  It's happened to me.

You know about the 'backup' directory, right?  Every GPX file is saved there.  However, since you named the files the exact same thing, it's sounds like the file in the backup dir will be the most recent one -- the hiking one.  When writing to the backup dir, TopoFusion will overwrite an existing file of the same name.  But it shouldn't effect the actual file in any way.

Let me know if you make any progress of this.  Do a search to make sure the file didn't get moved or saved somewhere else on accident.

I don't think anything in the USB realm could be causing it.  It is possible, but if the download completed and gave the merge dialog, USB is then out of the picture.

Thanks for the report.

[brian.r.hamilton]

That was fast!  '>

The biking file (the one I lost) has the entry in the logbook, but the hiking file doesn't because I unchecked that option when downloading that file for some reason.  When the USB communication was lost, I unchecked both the "follow" & "logbook" options just to see if I could re-establish communications without having to restart TF.  Not that I thought it would really help, but it was just a dumb thing to try before restarting TF.  So when I restarted TF, I unchecked them again for no good reason.  Anyway, the biking entry in the logbook shows the complete path to the missing file.

YES!...  I could have forgotten to actually save the file & just alt-tabbed to another application & came back to TF thinking my file was saved if the download/save dialog has it's own "tab" & can be hidden & I could see my track on the map!  This sounds like a very reasonable explanation since I usually have more than a dozen windows up at any given time!  As long as the track was visible in TF, I could have easily came back assuming I had already saved it & simply did another download!  Would the dialog just come to the front again with the previous tracks still showing or would the previous download info be cleared from the previous session?  If the previous download session status was still visible, I would usually notice that, but maybe not & I may still have thought I already saved it anyway since I saw the track on the map.  Maybe this is a good reason for me to never use the follow on map option?  I do think this scenario could be the problem!  Thanks!

Is the logbook entry created after the download or after a save?  I definitely didn't use Merge though, if that matters.  The Vista C gets much better signal reception than my old GPS, so the tracks are rarely broken.  Really like TF track downloading vs. ExpertGPS or any other program by the way.  Now just add those pretty waypoint icons too.  '>

I'm not positive about the state of the highlighting, but I think what I said is accurate.  I have also been messing with the color options in TF lately, so I could easily have mistaken a change there for highlighting since it may have appeared different than I was used to.  It seems to me that it was indeed a thicker yellow line though.  I'll try to pay more attention to this next time I download 2 tracks on the same day.

I checked the "backup" directory right away & it indeed had the hiking file (apparently overwritten the biking file).  I suppose it could be argued that the auto backup in TF would be a bit safer from fools like myself if it never overwrites a file in the backup directory.  Maybe a future rev could rename a previous version of the file before writing one with the same name?  Better yet would be to make use of the Recycle Bin before overwriting any file at any time since you wouldn't have to worry about duplicate names & the timestamp could be preserved.

[Krein]

The fact that it's in the logbook means that it *should have* been saved out to disk.  No logbook entries are made until you save a file.

Any chance your disk was full, or having other problems?

I tried to recreate just what you did and I got both files in the two directories, and as expected, the backup file only had the most recent one.  This was with a Forerunner 305, which is USB.  I don't have a Vista C, but the different is not large in terms of what TopoFusion is doing.

I agree that the backup system should probably not overwrite files.  It shouldn't happen often, but adding a suffix like "-back" would help to make sure more data isn't loss with multiple downloads.  I'll see if I can get that rolling in the next version.

In general, though, I'd keep all the GPX file names unique just to be safer (not just with this issue, but elsewhere and with other programs, too).

As for what happened, I really don't know.  If you notice it again please let us know.  I can't think of anything that would cause such a loss, and I can't recreate it, so I'm don't know what else to do.

[brian.r.hamilton]

I've got no reason to believe there was any trouble with my drives (4 drive RAID array with over 16 GB free, zero issues to date) & no instabilities or anything unusual going on with Windows XP either.

I've tried recreating the problem too & everything works as you've already discussed.

I double-checked the log entry & it definitely refers to the biking (missing) GPX file.  I searched the entire drive, including the Windows Recycling Bin & can't find any trace of the missing file.  If I deleted it manually, it would be in my Recycle Bin (nowhere near full & only emptied manually).  I'll run a search for the file by creation date & maybe even one based on contents to see if the filename could have been altered somehow.

I've literally got hundreds of GPX files that are all referenced in a spreadsheet journal (no links or anything like that though), so I'm not really eager to change the naming convention just yet.

PS.  I'm a programmer, but not for Windows... I'm curious why you wouldn't just send any file to the Recycle Bin before overwriting instead of worrying about a goofball naming convention for backups?  Isn't there some sort of file operation that will ensure the Recycle Bin is used (when setup accordingly in Windows of course) when TF wants to overwrite or delete a file?  I wouldn't use such a thing where TF normally prompts the user to overwrite, but it seems like a better solution for backup files than changing the filename.  I often wish more Windows programs would use the Recycle Bin for this sort of thing.  I know it's possible because I have some programs whose files indeed end up in the Recycle Bin.

[Krein]

Thanks for reporting back and for taking a thorough look at it.  I'll have another pass over the code to see if any light bulbs go off.

As for the recycle bin, it's not a bad idea, but it just isn't as intuitive (to me, anyway) to send files there without telling the user.  Personally, I almost never use the recycle bin, period.  If I was looking for data I thought was missing, overwritten, lost, altered, etc, I'd expect to find it in the backup directory, not in the recycle bin.  The different filenames would just be different versions of that particular file.  That way you always have the data instead of relying on windows to not empty it.

Make any sense to you?

[brian.r.hamilton]

I thought of one unusual thing that happened.  When I returned to the computer to download the track data from my hike, the download just hung.  I mentioned this above & just assumed it was a USB communications issue, perhaps related to my computer going into a power-saving state while I was away or maybe related to the fact that I probably had other GPS software (Garmin MapSource and/or TopoGrafix ExpertGPS) open.  The thing that I did not mention yet was that I had to terminate TF - it did not close normally!  I closed the download dialog/window & then closed TF.  I believe the Windows "Send Error Report" thing came up too.  I don't know where those go, but if that did happen, there could be a clue in that report.  Normally, after something like that happens, the application is terminated quickly but TF held on for a while for some reason.  I wasn't very patient & had already started another instance of TF which I believe started before the other instance had totally terminated.  I believe I tried the download again & it would not communicate.  The GPS was on & connected this entire time by the way.  I closed TF & restarted TF again without incident & then was able to download the new track.  So perhaps TF was hung on the previous disk write for some reason?

My search for the lost file turned up absolutely nothing, but I've only searched on Creation Date so far.  If you are sure the log file entry means the GPX file should have been written to disk, I have no idea what happened to it!  I know this has happened to me at least twice since I've had my new GPS (not even a month).  I wish I could remember more details about the first occurance.

Thinking out loud here, but I don't see how the screen saver/power savings could be related because the disk write should have been very quick - obviously the log file was written OK.  This is puzzling.

[brian.r.hamilton]

Yes, it makes sense & is just a different way of looking at the whole backup issue.  Your view is more like that of other programs that have an AutoSave feature used in case of abnormal program termination or to revert to the original file currently being edited.  I agree with this approach, but couldn't your approach also coexist with the use of the Recycle Bin in general?

I've assumed it was probably MS's intent that all applications use the Recycle Bin when doing any file manipulation that would result in overwriting one file with new contents or when deleting a file whenever the user isn't prompted to specifically choose to do so (as in "File Exists, overwrite YES or NO?").  Also, when most applications open/close/save files, they use the standard Windows File Open dialog.  This allows the user to delete/rename/select more than one file, etc. as well as usually (always?) using the Recycle Bin when files are deleted.  That Recycle Bin has saved me more times than I can count!  Windows Explorer must be involved too because choosing Undo in there often restores an accidentally deleted file.

Anyway, if every saved track gets a new unique backup file, I guess the problem simply goes away.  The Recycle Bin would only come into play if TF used the Windows Explorer (?) call to open/close/save files.  By the way, I do think that method of opening files would be a little cleaner from a UI standpoint.  Most application do this these days.  The current TF method is just fine though.  Whatever you do, please don't ever consider the oddball UI like used in DeLorme TopoUSA software!  Tabs all over the place.  Bleah!

So when/how often does TF currently write the backup file anyway?

[Krein]

As far as I know, TF does use the standard windows dialog for opening and saving.  When you get a "save" dialog you can do things like new directories, rename file, etc.  The little "mini-explorer" thing on the main window isn't really standard, but it's just a short cut to data.  You can always use File->Open.

I'd think if MS wanted to move overwritten files to the recycle bin, they'd have done it themselves in that dialog.  I guess we could add that functionality, because it does make sense.

The backups are saved only at download time.  No other backups are made.  Usually things can be reconstructed as long as you have the original data, so that's why we save the original, from-the-GPS data.

BTW, the order of operations is:

1. add entry to logbook
2. save original file in directory requested in save dialog
3. save backup file

So best guess is that something happened inbetween 1 and 2.  The program did continue to function after the save, right?

I've never heard of this happening before, but I'll keep it in mind for my own use of the program and if anyone else reports a problem.  Please keep me informed if you learn or think of anything else.

It might be prudent to swap #1 and #2.

Thanks.

[brian.r.hamilton]

I've been using the "mini-explorer" thing for so long that I never even bothered trying the menu commands!  Thanks for pointing that out!  Doh!  However, I noticed that TF doesn't allow me to open/add more than 1 file at a time like I can in the "mini-explorer" thing though.  Would it be difficult to allow multiple file selections thru File Open too?

I agree that saving the original track data from the GPS in a unique backup file makes sense.  Those backups have saved me from a few mis-edits more than once!

I think I misunderstood your earlier explanation of the order of operations.  I thought the fact that the logbook entry appeared for the missing file meant that the file should have been written to disk already.  If not, then my forgetting to actually do the Save operation seems the most likely culprit for sure.  I could easily have alt-tabbed away from the TF Save dialog before saving & later alt-tabbed back to the TF main window with my unsaved track visible!  Thinking I already saved it, since it was displayed on the map, I probably downloaded & saved the new track (to a different directory), clobbering the backup file with the same name.  Definitely sounds like something I could easily have done!

So besides unique backup file names, another solution could be a captive Save dialog?  In other words, if I alt-tab away from TF, then alt-tab back, the Save dialog would be forced to the "front".  Sorry, I don't know the correct Windows terminology.  I notice this behavior with the GPS Transfer dialog, Profile Tool, & Export dialog too.  Should all these "windows" be allowed to exist open at the same time?  At least in the case of file operations, this seems like a potential area for trouble.  Just more thinking out loud here...

[Krein]

Yeah I'm not sure about opening multiple.  I'd like to be able to open multiple, too.

The order is as listed above, however, there's no way to "stop" in the middle.  #1 doesn't happen until after you choose the filename.  

The "save as" dialog is modal (that's the term) meaning you can't do anything until you dismiss it.  But that doesn't matter given the above -- once you give it the filename, TF goes and adds to the logbook, saves and then saves the backup.

But you're right that the fact that the logbook entry is there doesn't mean the file ever got saved out.  It just means that you actually did choose a file name for it, and that you weren't fooled by it being displayed on the map.

The fact that your next GPS transfer hung is indicative of something dying on the GPS thread.  It's threaded so that the program can continue to run while the GPS is downloading.  So something crapped up between writing the logbook entry and saving the file.  Too bad it didn't crash out to TopoFusion's crash reported (dunno where the "windows crash" things go, certainly not to us).

I'll see if I can spot anything with this in mind.

[brian.r.hamilton]

Assuming I did initiate the Save operation from the Merge/Download dialog, TF did continue to run to the extent that  I was able to navigate the menu to initiate another GPS Transfer (that failed & resulted in my having to restart TF).

So it sounds like the most likely scenario is that I did initiate the Save, the log entry was written to disk, then the file operation to save the GPX file failed for some reason but reported success since TF continued to run (well enough to initiate another transfer before crashing at least).

A failed disk write is a scary scenario for me!  My disk drives are good ones (four 10k Quantum SCSI), but they are getting old!

By the way, I just tried another GPS transfer & TF disappeared immediately after my GPS beeped that the download was complete!  It left no trace & all TF windows were just gone!  I believe I briefly saw the Merge/Download dialog before everything vanished.  Restarted TF & tried again & all went well.  Perhaps this is another reason to suspect something going on with my system although the other half dozen or so apps seem to be running just fine.  Don't you just hate things like this?  Does TF use the registry for anything at run-time?

PS.  I was thinking the exact reverse order (3, 2, 1) made the most sense.  Adding some sort of "normal termination" (or state) flag to the INI file would also allow TF to potentially better recover from crashes.  Maybe TF could look at the flag on start-up & do some sort of sanity check & potentially avoid lost data?  It's easy to get too carried away with this sort of thing to though.

[brian.r.hamilton]

So MS is collecting TF program state info & they never forward it to you guys?  That's funny.  I bet you can get it for a fee!  I'll understand if you don't reply to this.  '>

Does the GPS thread go away entirely & release all resources (memory & close GPS USB driver) after the transfer is complete?  What happens if I alt-tab to another app during the transfer & that other app requests something of the GPS USB driver?  Perhaps Garmin has some re-entrancy issues with their USB driver?  Not that I understand Windows multi-tasking & drivers or anything, but I have a good handle on that stuff for embedded systems.

I'll do a scan disk & maybe a few other disk tests tonite too.

[brian.r.hamilton]

It happened again!!!  It has nothing to do with multiple files.  I just downloaded a track from my GPS, saved it, looked at it on the map (actually exported the map too!), closed TF & now the file is nowhere to be found!  There is a logbook entry just like before.  There is also a shortcut to the file I saved  in my Recent Documents, but the file does not exist anywhere on my hard drive!  I'm not having any trouble with other documents.  What in the world is going on here?

[Krein]

Uh oh.  

2 questions:

1 - how about the backup file?  Is that there?

2 - did you close the Merge/Delete dialog after you saved the track.  This would verify that the track was actually there in the active list, not just shown because the dialog is still active and showing downloaded tracks.

Thanks.  I'm looking at it right now, actually.  Adding the non-overwriting backup, and also added some warnings about the merge dialog still being open.

[Krein]

If you can duplicate the problem, check if you can download again after completing a single download.  If you can't this probably tells us that the GPS thread is dying after it saves the logbook but before it saves the backup and/or actual file.

[brian.r.hamilton]

No backup file was created.  I'm not sure if I closed the Merge/Delete dialog, but I think I did not.  In fact, I think I may have closed TF with that dialog still open as well.  I also think I turned off the GPS & unplugged it before closing anything too.

Fortunately I didn't clear my GPS so I was able to simply restart TF & transfer the track again.  I'm not sure if/what I did differently, but it saved it just fine the 2nd time.  I may have closed all the dialogs before disconnecting the GPS the second time.

[brian.r.hamilton]

I was just able to recreate the problem!  Now let me see if I can relate exactly what I did.

I don't think I closed TF since the last time I successfully transfered a track.  I plugged the USB cable into the GPS, then the computer, then powered on the GPS.  Next I did the GPS Transfer, merged 2 tracks (out of 5 or 6), saved the selected track (actually not highlighted in the dialog - i.e., I didn't click on the merged track), did not close any dialogs, alt-tabbed to the main TF window, & double-clicked on the track file (the only one currently loaded & displayed in TF) to view/center it on the Combo map.  Then I think I alt-tabbed back to the Merge dialog before turning off the GPS & unplugging it from the computer.  Finally, I simply closed TF by clicking on the X in the title bar of the main window.  There is an entry in the log book again but no GPX file or backup file.  The GPX file does show up in my Recent Documents again too, but that shortcut is broken since the file is not on disk.

[brian.r.hamilton]

Just tried again & couldn't repeat it so I tried 2 downloads in a row, disconnecting the GPS in-between & returning to download a 2nd track.  The Merge/Delete dialog was flaky in that I selected the original merged track & saved but the GPX file that was saved was only the first track of the two I merged (i.e., the first one off the GPS in the 2nd download too).  Anyway, this file was saved just fine.  Now I'm wondering if the key is to download a track, leave all dialogs open, let the screensaver kick in to power savings or whatever (my discs are always running though), then come back & download.  I can't think of what else I'm missing here but I know we're getting closer.  I'm still not convinced it is a problem with TF.  It could very well be a problem with my PC and/or the Garmin USB driver.  Hmmm....

[brian.r.hamilton]

Hmmm... messing with it some more & can't get Windows to even recognize the GPS is connected!  I tried TF, Expert GPS, & MapSource & none see the GPS.  Windows isn't giving me the usual beep when the device is connected, so it makes sense no SW can see it.  I even tried a different USB port (built into my printer).  No luck.  Connected my camera with no issues using the same USB port & cable.  Apparently the Garmin USB driver is crashed somehow.  I think I'm going to have to reboot my PC.  The USB driver or the use of it seems like it has to be relevant.

[Krein]

Puzzling, puzzling!

It sounds like moving to v2.88 may solve your problem -- at least the data loss part.  I changed the order, as you suggested, so that the backup and real file are written first, then the logbook entry it made (which, incidentally, calls a different thread).  My guess based on the info you've provided is that you won't lose any more files, but you may still be in the situation of not being able to communicate with your GPS until reboot or at least closing of programs, etc.

I think the Garmin USB driver is not 100% reliable, in general.

v2.88 also has 5 or so changes/fixes that arose from this discussion or your other posts over the last few days.  It can almost be called "v2.88-BRH."  '>

Thanks.

[tdcarter]

garmin did recently update the driver. It was included in the latest firmware patch. The created date on the the file was March 13th.

This one does seem to be more stable then the previous.... or are you also using the new one and it still seems flaky....

TC

[Krein]

Thanks for the suggestion.  BRH - have you updated the driver?

I didn't know about that update.  I'll have to see if it got installed inadvertantly when I updated a firmware on one of my devices.

[brian.r.hamilton]

I installed the latest version (2.1.0.0, dated 03-13-2006) of the Garmin USB driver (grmnusb.sys) on 06-04-2006.  I only installed it because it was available about the time I first installed & updated the MapSource software.  I have only had my USB GPS since about that same time.  I was using a serial Garmin GPS prior to that.

[brian.r.hamilton]

Thanks for giving this stuff so much thought & attention.  Hopefully new users will notice the great customer support & support TF in return!  I plug TF every chance I get!

Just thought I'd let you know that I haven't had the problem since you made the updates.  I've updated to the latest version pretty much when each became available & haven't had any data loss with any of them.

[brian.r.hamilton]

Oops.  Forgot to log in for that reply.  Thanks again for taking the data loss issue seriously & making the repairs so quickly!