Page 1 of 1

Comparing adif files

Posted: Sun 11. Dec 2011, 21:05
by g4sjp
As a result of moving between various logging software both for Windows and OSX, I find myself in a very trivial but nevertheless rather irritating situation. Over the past few years I have uploaded my QSOs to LOTW, but I notice that I have more contacts listed in my LOTW account than in Rumlog. It makes very little difference to my statistics but is still annoying. Somewhere along the line I have failed to merge contacts into my main log. So, I have downloaded my full logbook from LOTW and can easily import this into a new logbook in Rumlog. The trouble in doing this, however, is that the data from LOTW is very minimal, and I will lose lots of interesting information that I have typed into the comments field for example.

What I would like to be able to do is to merge the LOTW download with my existing log. However, even if I tick the box to ignore duplicate entries, because LOTW has, for example, changed 10.117MHz to 10.100MHz, the contact is not seen as being a duplicate. So, rather than adding the missing 70 or so QSOs to Rumlog I am adding more like 2000!

I have tried converting the adif files to csv and looking at them in Excel, and in so doing I have spotted a small number of differences, but the time it will take me to find all the differences and then load them into Rumlog is not worth the likely benefit. In testing for duplicate contacts when merging adif files perhaps only callsign and date/time/mode/band should be compared? Perhaps I am missing something?

73 Stephen G4SJP

Re: Comparing adif files

Posted: Sun 11. Dec 2011, 23:03
by DL2RUM
I've just checked the source code. To recognize a dupe, following data are used:
  • callsign
  • band
  • mode
  • date
  • time
For the time, the seconds are ignored and the frequency will not be taken into consideration. So your idea should work. It would be interesting to see few duplicated record set to compare it in detail.

Another Idea is to download the complete LoTW QSL set (using the "LoTW QSL in" routine), setting the date to your first QSO. QSOs not found in the log will be listed in the table. Maybe this will help you. It could be that only during a small time period your QSOs are missing. Then you could download only QSOs from this time slot.

I hope this helps.

Re: Comparing adif files

Posted: Mon 12. Dec 2011, 11:18
by g4sjp
Hello Tom,

Thanks for your reply. I have tried the merge again and I see there are various issues. I should be adding about 50 contacts but the merge adds over 1500! I have only used Rumlog as my sole logger since April, and contacts since April merge perfectly. The problem with the others is that many contacts are duplicated, some I have logged as AMTOR, LOTW returns them as 'TOR'. With other dupes I can't see what the difference would be. Before using Rumlog I used HRD and also MacLoggerDX.

Thanks again,

73 Stephen

Re: Comparing adif files

Posted: Mon 12. Dec 2011, 12:02
by DL2RUM
TOR is the correct for AMTOR according the ADIF specifications. Maybe you should convert all modes to be ADIF conform wording prior importing in RUMlog. Use a text editor and do a search and replace all.

<mode:5>AMTOR should be converted to <mode:3>TOR

When you are familiar with the SQL syntax you can manipulate the RUMlog log database itself, using a database tool to change all wrong logged modes.

Re: Comparing adif files

Posted: Mon 12. Dec 2011, 13:17
by g4sjp
Tom

Thanks very much, I am making progress! I have sorted out the mode problem with TOR/AMTOR.
However, I am puzzled by what is happening below. I have exported from the merged log, two qsos, one of which is a duplicate (same station, two different bands). As you can see, the 20m qso was duplicated when I merged the logs while the 30m one was not! I can't see any difference between the ADIF fields that you tell me that Rumlog uses in dupe checking.

I don't want to trouble you with this Tom, but I am very curious.

73 Stephen
---------------------------------------------
ADIF Export from RumLog by DL2RUM
tom@dl2rum.de
For further info visit: http://www.dl2rum.de

<adif_ver:5>2.2.6
<programid:6>RUMlog
<programversion:5>4.8.4
<station_callsign:6>NoCall
<operator:6>NoCall
<my_name:6>NoName

<eoh>


<call:4>T77C <qso_date:8>20110104 <time_on:6>173500 <band:3>30m <freq:7>10.1019 <mode:2>CW
<rst_sent:3>599 <rst_rcvd:3>599 <qsl_rcvd:1>N <qsl_sent:1>N <dxcc:3>278 <cqz:2>15 <ituz:2>28 <comment:29>TONY CECCOLI 47890 MURATA A-7 <gridsquare:6>JN63AA <app_rumlog_power:4>QRPP <tx_pwr:1>0
<app_rumlog_qsl:1>- <lotw_qsl_sent:1>Y <lotw_qsl_rcvd:1>Y <eqsl_qsl_sent:1>Y <eqsl_qsl_rcvd:1>N <eor>

<call:4>T77C <qso_date:8>20110104 <time_on:6>170139 <band:3>20m <freq:6>14.019 <mode:2>CW
<rst_sent:3>559 <rst_rcvd:3>559 <qsl_rcvd:1>N <qsl_sent:1>N <dxcc:3>278 <cqz:2>15 <ituz:2>28 <comment:29>TONY CECCOLI 47890 MURATA A-7 <gridsquare:6>JN63AA <app_rumlog_power:4>QRPP <tx_pwr:1>0
<app_rumlog_qsl:1>- <lotw_qsl_sent:1>Y <lotw_qsl_rcvd:1>Y <eqsl_qsl_sent:1>Y <eqsl_qsl_rcvd:1>N <eor>

<call:4>T77C <qso_date:8>20110104 <time_on:6>170100 <band:3>20m <freq:6>14.000 <mode:2>CW
<rst_sent:3>599 <rst_rcvd:3>599 <qsl_rcvd:1>Y <qsl_sent:1>N <dxcc:3>278 <cqz:2>15 <ituz:2>28
<app_rumlog_qsl:1>R <lotw_qsl_sent:1>N <lotw_qsl_rcvd:1>N <eqsl_qsl_sent:1>N <eqsl_qsl_rcvd:1>N <eor>

Re: Comparing adif files

Posted: Mon 12. Dec 2011, 14:40
by g4sjp
Tom, the duplication only appears to happen when the original (logbook) time does _not_ end with two zeros.

So a <time_on:6>182000 will not get duplicated when I bring in the merge file, but <time_on:6>182022 _will_ get duplicated, since the incoming merge file from LOTW always has 00 at the end. You wrote that your code does not check the seconds, but it is behaving as if it does - I think!

73 Stephen

Re: Comparing adif files

Posted: Tue 13. Dec 2011, 15:18
by DL2RUM
Now I'm back home from the Caribbean.
You wrote that your code does not check the seconds, but it is behaving as if it does - I think!
This was my intention. It seems I was wrong. I've found an error in the code. Please check out the following version, just move the unzipped file into our existing RUMlog folder.

Try this: http://dl2rum.de/rumsoft/RUMlogToTest.zip

Re: Comparing adif files

Posted: Tue 13. Dec 2011, 18:57
by g4sjp
Hi Tom, I called you when you were in FS, but no luck! I bet the weather in DL has come as a shock to you!

Anyway, the modified code works FB. I discovered all sorts of problems with the LOTW download, but mainly the use of AMTOR rather than TOR. I thought I had fixed it in Excel but I forgot to change the field length, so it was still duping! The time dupe problem has now disappeared completely, so many thanks for your effort there.

On a related matter, I downloaded my log from Clublog to check on DXCC for some special calls - for example Rumlog showed KH6TY as Hawaii when he was in mainland USA at the time of the QSO. Not able to find a way of changing the logged dxcc in the logbook (is there a way?), I imported my clublog log into dxcc.app. That worked very well after I realised that I had not chosen to 'Recount DXCCs', because I didn't know that's what the option meant. It would be better to label it 'Recalculate DXCCs' or 'Re-map DXCCs' or something like that. There is a subtle difference between 'recount' and 'recalculate'.

Thanks very much for your help on this, I have learnt a great deal not only about the way that Rumlog works but also the adif format.

Hope you have a Happy Christmas

73, Stephen

Re: Comparing adif files

Posted: Tue 13. Dec 2011, 20:02
by DL2RUM
There is a "Recalculate DXCC" entry in the QSO menu. This will determine the DXCC from one or more log entries.
There is a "Recount DXCC" entry in the File menu. This will recalculate the DXCCs from the whole log and it counts all together for the DXCC table. The statistic database will be written completely new.

To change the DXCC for one (or even multiple) QSO make the changes in the "dxcc.rsd" database using the tool "EditDXCC.app" and use them Menu-->QSO-->Recalculate DXCC

Re: Comparing adif files

Posted: Tue 13. Dec 2011, 22:40
by g4sjp
Tom,

Thanks for clearing that up, I understand now. For some reason I had never seen the recalculate button - I feel rather stupid!

Thanks for your support, much appreciated by all using Rumlog I'm sure.

73, Stephen