Lazarus

Free Pascal => Beginners => Topic started by: JLWest on April 23, 2019, 07:35:43 am

Title: DataProblems Maybe
Post by: JLWest on April 23, 2019, 07:35:43 am
Code: Pascal  [Select]
  1. procedure TForm1.Edit3AndRefDataSet(aIdx : Integer);
  2.   Var i : Integer = -1;
  3.    LB3Rcd  : String = '';
  4.    Bit1    : String = '';
  5.   begin
  6.    i := aIdx;
  7.    ListBox3.ItemIndex := i;
  8.    Ed3String  =  Listbox3.Items[i];
  9.    LB3Rcd := Listbox3.Items[i];
  10.  
  11.   Bit1  := ExtractWord(6,LB3Rcd,['|']);
  12.   RefData.Lat := StrToFloat(Bit1);      <-- Run Time error 'K2' not a valid float
  13.  
  14.   Bit1 := ExtractWord(7,Ed3String,['|']);  <-- Run Time error 'K2' not a valid float
  15.   RefData.Lat := StrToFloat(Bit1);
  16.  
  17.    Ed3String  := '|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|'; <-- Actual Data
  18.    Bit1  := ExtractWord(6,Ed3String,['|']);
  19.    RefData.Lat := StrToFloat(Bit1);
  20. {  RefData.Lat is defined as a Double:
  21.    RefData.Lat was = 35.349333          <-- Correct value}
  22.  
  23.   end;    

 I copied the data as Raw Data under the debugger and set this test up.
I don't understand wha'ts going on.

Is there a function to convert text to raw data? I have been trying to write one but cant get it to work right.
Title: Re: DataProblems Maybe
Post by: six1 on April 23, 2019, 07:43:23 am
'|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|'
  1       2   3        4  5        6   7                8 9  10    11       12   13


maybe it isn't the 6th and 7th more 11th and 13s word?

Also check result!
The N-th word, or empty if N is out of range.
https://www.freepascal.org/docs-html-3.0.0/rtl/strutils/extractword.html
Title: Re: DataProblems Maybe
Post by: JLWest on April 23, 2019, 07:52:53 am
I ran a test program and

 S := ExtractWord(1,Data,['|']); is using the delimiter as right and left as a pair So:

     1           2            3                 4            5         6               7

'|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|

It's giving me the error saying 'K2' is not a valid Float.

So my thinking is the data coming out of the Listbox is different that the Raw Data I copied from the debugger and it makes a difference of one field.

I think I need to take the data out of the listbox. Somehow convert it to Raw Data and then extract what I need.

Maybe as a RawByteString but I don't know how to write the function.
Title: Re: DataProblems Maybe
Post by: bytebites on April 23, 2019, 08:25:53 am
'|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|'
  1       2   3        4  5        6   7                8 9  10    11       12   13


maybe it isn't the 6th and 7th more 11th and 13s word?
...
This is how ExtractDelimited works.
https://www.freepascal.org/docs-html-3.0.0/rtl/strutils/extractdelimited.html
Title: Re: DataProblems Maybe
Post by: sstvmaster on April 23, 2019, 08:28:56 am
@JLWEST

this could help (in FormCreate?):
Code: Pascal  [Select]
  1. DefaultFormatSettings.DecimalSeparator := '.';

If this works, your decimal separator is "," Comma not point.

See attachment, this works.
Title: Re: DataProblems Maybe
Post by: JLWest on April 23, 2019, 08:40:54 am
@bytebites Maybe your right about the field counts?

I have been running this way for some time. I got the following function going and passing the RCDString thru it.

Limited testing but it seems to work.

'|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|' is my created data format and I would rather have it look like this:

'[00CA|[7826300][Barstow][United States][K2][35.349333][-116.893333]'

But then how do you write ExtractWord?

S := ExtractWord(6,RCDString,['['..']'); or mybe

S := ExtractWord(6,RCDString, ['[']  , [']']  );

Code: Pascal  [Select]
  1. function TForm1.ToAlphanumericString(const AString: String): String;
  2.     var
  3.       Character: Char;
  4.     begin
  5.       Result := '';
  6.  
  7.       for Character in AString do
  8.         if Character in ['0'..'9', 'A'..'Z', '|'] then
  9.           Result += Character;
  10.     end;        

Title: Re: DataProblems Maybe
Post by: six1 on April 23, 2019, 08:50:34 am
'|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|'
  1       2   3        4  5        6   7                8 9  10    11       12   13


maybe it isn't the 6th and 7th more 11th and 13s word?
...
This is how ExtractDelimited works.
https://www.freepascal.org/docs-html-3.0.0/rtl/strutils/extractdelimited.html

Help says:
This means that if 2 delimiter characters appear next to each other, there is an empty part between it.

in my interpretation above countwise is right!
2 delimiter characters appear next to each other counts!
or am i totaly wrong?

i'm using TStringlist with delimiter in such cases and this "||" will give two parts...
Title: Re: DataProblems Maybe
Post by: JLWest on April 23, 2019, 08:52:57 am

Thoid Works: And it returns the 6 th field  39.297611111


   s := '[KMCI][5502297][Kansas City][United States][K3|[39.297611111][-94.713888889]';

  A := ExtractWord(6,s,['['..']']);

Just have to change an a lot of code to but maybe it will be better. 
Title: Re: DataProblems Maybe
Post by: JLWest on April 23, 2019, 08:54:32 am
'|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|'
  1       2   3        4  5        6   7                8 9  10    11       12   13


maybe it isn't the 6th and 7th more 11th and 13s word?
...
This is how ExtractDelimited works.
https://www.freepascal.org/docs-html-3.0.0/rtl/strutils/extractdelimited.html

Help says:
This means that if 2 delimiter characters appear next to each other, there is an empty part between it.

in my interpretation above countwise is right!
2 delimiter characters appear next to each other counts!
or am i totaly wrong?

i'm using TStringlist with delimiter in such cases and this "||" will give two parts...

No I think maybe your right.

Title: Re: DataProblems Maybe
Post by: six1 on April 23, 2019, 09:11:10 am
'|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|'
  1       2   3        4  5        6   7                8 9  10    11       12   13


maybe it isn't the 6th and 7th more 11th and 13s word?
...
This is how ExtractDelimited works.
https://www.freepascal.org/docs-html-3.0.0/rtl/strutils/extractdelimited.html

in addition:
'|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|'
0  1       2   3        4  5        6   7                8 9  10    11       12   13         14

this is how it will count by using TStringlist delimit
Title: Re: DataProblems Maybe
Post by: lucamar on April 23, 2019, 03:24:00 pm
Since you're using ExtracWord() and not ExtractDelimited(), all that talk about empty returns is futile. As the documentation says:
Quote from: Ref. for 'strutils' (#rtl) - ExtractWords
Unlike ExtractDelimited, an empty string is not a valid return value, i.e. is not a word. If an empty string is returned, the index N was out of range.

For ExtractWords(N, AString, ['|']) the count should be:
Code: [Select]
|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|'
 1     2        3        4              5   6          7
as demonstrated by the fact that it works when you test directly with that string.

Rather add something to show you what is getting read from the ListBox. If it's like the string you're expecting then something is wrong with ExtractWord().

And to use '[' and ']' as delimiters it should suffice with:
Code: Pascal  [Select]
  1.  A := ExtractWord(6,s,['[',']']);
Title: Re: DataProblems Maybe
Post by: Thausand on April 23, 2019, 04:52:20 pm
Is correct user lucamar. I think user six1 not understand maybe ? user bytebites good explain...

error JLWest problem StrToFloat... he write so is.

User sstvmaster write good test. Many country have other float separate. Is two "." and "," (but can make any use freepascal if want). Some time data storage float not same separate have you system and make error when use StrToFloat.

add:

I make program test and see better make float string. Can make more difficult if want. Is make freepascal and can test data if file data.LoadFromFile('Name_File');

Code: Pascal  [Select]
  1. program floatfloat;
  2.  
  3. {$MODE OBJFPC}{$H+}
  4.  
  5. uses
  6.   Classes, SysUtils, StrUtils;
  7.  
  8. procedure Info(S: String; const F: array of const);
  9. begin
  10.   WriteLn('Info: ', Format(S, F));
  11. end;
  12.  
  13. procedure Error(S: String; const F: array of const);
  14. begin
  15.   WriteLn('Error: ', Format(S, F));
  16. end;
  17.  
  18. procedure Test(Lines: TStrings);
  19. const
  20.   worddelim             = ['|'];
  21.   radixdelim            = ['.', ','];
  22. var
  23.   line_no               : integer = 0;
  24.   line                  : string;
  25.   nr_words_in_line      : integer;
  26.   float_part_count      : integer;
  27.   float_integer_part    : string;
  28.   float_fractional_part : string;
  29.   bit1, newbit1         : string;
  30.   bit2, newbit2         : string;
  31.   lat1, lat2            : double;
  32. begin
  33.   for line in Lines do
  34.   begin
  35.     inc(line_no);
  36.     nr_words_in_line := WordCount(line, worddelim);
  37.  
  38.     if (nr_words_in_line = 7) then
  39.     begin
  40.  
  41.       // have float bit1
  42.  
  43.       bit1  := ExtractWord(6, line, worddelim);
  44.       float_part_count := WordCount(bit1, radixdelim);
  45.       if (float_part_count = 2) then
  46.       begin
  47.         float_integer_part    := ExtractWord(1, bit1, radixdelim);
  48.         float_fractional_part := ExtractWord(2, bit1, radixdelim);
  49.         newbit1               := float_integer_part + DefaultFormatSettings.DecimalSeparator + float_fractional_part;
  50.         if TryStrToFloat(newbit1, lat1)
  51.         then Info('line %d have %s valid float value = %f', [line_no, 'bit1', lat1])
  52.         else Error('have error data %s. Not have validate float "%s" line = %d', ['bit1', bit1, line_no]);
  53.       end
  54.       else Error('have error data %s. Not have validate float "%s" line = %d', ['bit1', bit1, line_no]);
  55.  
  56.       // have float bit2
  57.  
  58.       bit2  := ExtractWord(7, line, worddelim);
  59.       float_part_count := WordCount(bit2, radixdelim);
  60.       if (float_part_count = 2) then
  61.       begin
  62.         float_integer_part    := ExtractWord(1, bit2, radixdelim);
  63.         float_fractional_part := ExtractWord(2, bit2, radixdelim);
  64.         newbit2               := float_integer_part + DefaultFormatSettings.DecimalSeparator + float_fractional_part;
  65.         if TryStrToFloat(newbit2, lat2)
  66.         then Info('line %d have %s valid float value = %f', [line_no, 'bit2', lat2])
  67.         else Error('have error data %s. Not have validate float "%s" line = %d', ['bit2', bit2, line_no]);
  68.       end
  69.       else Error('have error data %s. Not have validate float "%s" line = %d', ['bit2', bit2, line_no]);
  70.     end
  71.     else Error('not know line %d and have %d word', [line_no, nr_words_in_line]);
  72.   end;
  73. end;
  74.  
  75. var
  76.   data: TStringList;
  77. begin
  78.   data := TStringList.Create;
  79.   data.append('|00CA||7826300||Barstow||United States||K2||35,349333||-116,893333|');
  80.   data.append('|7826300||Barstow||United States||K2||35,349333||-116,893333|');
  81.   data.append('|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333|');
  82.   data.append('|1||hello||12345||67,78||1111K2||fun||calculate|');
  83.   data.append('|00CA||7826300||Barstow||United States||K2||35,349333||-116.893333|');
  84.   data.append('|00CA||7826300||Barstow||United States||K2||35.349333||-116,893333|');
  85.  
  86.   // make many append data for want test
  87.  
  88.   Test(data);
  89.   data.free;
  90. end.
  91.  

that make write

Code: [Select]
Info: line 1 have bit1 valid float value = 35.35
Info: line 1 have bit2 valid float value = -116.89
Error: not know line 2 and have 6 word
Info: line 3 have bit1 valid float value = 35.35
Info: line 3 have bit2 valid float value = -116.89
Error: have error data bit1. Not have validate float "fun" line = 4
Error: have error data bit2. Not have validate float "calculate" line = 4
Info: line 5 have bit1 valid float value = 35.35
Info: line 5 have bit2 valid float value = -116.89
Info: line 6 have bit1 valid float value = 35.35
Info: line 6 have bit2 valid float value = -116.89

And have not error runtime  :)
Title: Re: DataProblems Maybe
Post by: JLWest on April 23, 2019, 09:47:55 pm
@lucamar

Changed all my data to '['..']'.
I used: A := ExtractWord('NIL', s, ['['..']']);
Compared to A := ExtractWord(6,s,['[',']']); and it seems to work.

Still testing.
Title: Re: DataProblems Maybe
Post by: lucamar on April 23, 2019, 10:37:20 pm
I had a little after-supper free-time and nothing better to do so I made a small test/demo. Find it attached.

The "meat" of it is the method Decompose(Index: Integer) which does, more or less, the same as your code in the first post. Rather heavily commented, but WTH! :)
Code: Pascal  [Select]
  1. procedure TMainForm.Decompose(Index: Integer);
  2. { Decompose "Index"th string from SrcList and add the words to DestList }
  3. var
  4.   Count: Integer;
  5.   AWord: String;
  6.   Delims: TSysCharSet;
  7.   AFloat: Extended;
  8. begin
  9.   if Index in [0..SrcList.Count-1] then begin { Sanity check }
  10.     { Select the proper delimiters }
  11.     if SrcList.Items[Index].StartsWith('|') then
  12.       Delims := ['|']     { String format: "|something||other||...| }
  13.     else if SrcList.Items[Index].StartsWith('[') then
  14.       Delims := ['[',']'] { String format: "[something][other][...] }
  15.     else                { Free form string? Use standard delimiters }
  16.       Delims := StdWordDelims;
  17.     { Extract all words }
  18.     DestList.Clear;
  19.     Count := 0;
  20.     repeat
  21.       Inc(Count);
  22.       AWord := ExtractWord(Count, SrcList.Items[Index], Delims);
  23.       if not AWord.IsEmpty then
  24.         if TryStrToFloat(AWord, AFloat) then
  25.           { If a float, show also the StrToFloat conversion}
  26.           DestList.Items.Add(sfWithFloat,[Count, AWord, AFloat])
  27.         else
  28.           DestList.Items.Add(sfNoFloat,[Count, AWord]);
  29.     until AWord.IsEmpty;
  30.     ShowMessageFmt('Found %d words', [Count-1])
  31.   end;
  32. end;
Title: Re: DataProblems Maybe
Post by: JLWest on April 23, 2019, 11:53:20 pm
I thought it was working but that's not the case.
 
ED2String = '[00C][28944][Nil][Nil][K2][37.203177778][-107.869194444]'

RCD.ICAO := ExtractWord(3,ED2String ,['['..']']); <-- Different than your's

RCD.ICAO = ''

Changed to:

RCD.ICAO := ExtractWord(1,ED2String ,['[',']']); <-- Changed to your's

RCD.ICAO = ''

Still not working.

Will take a look at your code example.
 

Title: Re: DataProblems Maybe
Post by: lucamar on April 24, 2019, 01:05:31 am
Doing:
Code: [Select]
RCD.ICAO := ExtractWord(3,ED2String ,['['..']']);is the same as doing:
Code: [Select]
RCD.ICAO := ExtractWord(3,ED2String ,['[', '\', ']']);which may or may not matter to you...

Anyway, getting an empty string from ExtractWord means that there is no such word in your string, i.e. the word-index is out of range, so add:
Code: [Select]
ShowMessage('"' + ED2String + '"');just before the previous line to see what, if anything, ED2String contains (the double quotes are there for you to see something even if the string is empty).

Most probably ED2String is wrong, because if it were right ExtractWord(3,ED2String, ['[',']']) would return the word 'Nil', as demonstrated by the attached image.
Title: Re: DataProblems Maybe
Post by: Thausand on April 24, 2019, 01:11:00 am
@JLWest:
you have file with many data for test ? Example my work also when not data good.

i ask: why change delimiter any time ?
Title: Re: DataProblems Maybe
Post by: josh on April 24, 2019, 01:54:07 am
Attached simple application, that extract the encapsulated strings and puts them into an array.
The array data will start in the array at index 1.
Works fine with your data line.
Title: Re: DataProblems Maybe
Post by: lucamar on April 24, 2019, 03:52:25 am
Attached simple application, that extract the encapsulated strings and puts them into an array.
The array data will start in the array at index 1.
Works fine with your data line.

Nice example but having ExtractWord() (which, as demonstrated, works and is more capable), why reinvent the wheel? :)

ETA: Forgot what I came here for! Lack of sleep combined with insomnia...

Anyway, since I couldn't sleep I have added a few niceties to my example. Now you can:

Well, have fun!
Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 09:37:31 am
GDrive Links

https://drive.google.com/open?id=1MVwkFVJUImSIJBgW_Sl2ox2Eb2wwi7H0
https://drive.google.com/open?id=1vr77NTnzTVnbbXRn413I6rmWtk3xaMZd
https://drive.google.com/open?id=177Lgy7GeOzgwxrRckR_Vx0Uvzv3WrMgM

Here is the files and code.

The pathing will have to change. I should have changed it to the install path.

 I cut the data way down.

Just hit the test button.
On the first record I get 8 fields and then it goes to 7 (which is right).

I posted the code here and on the Gdrive.
Thanks:


Title: Re: DataProblems Maybe
Post by: Thausand on April 24, 2019, 10:00:22 am
Thanksy JLWest.

I make fast test and write:
Code: [Select]
--------------------------------
proc file ByAirport.txt
--------------------------------
Error: not know line 1 and have 8 word
Info: line 2 have bit1 valid float value = 35.35
Info: line 2 have bit2 valid float value = -116.89
Info: line 3 have bit1 valid float value = 33.49
Info: line 3 have bit2 valid float value = -111.64
Info: line 4 have bit1 valid float value = 45.47
Info: line 4 have bit2 valid float value = -105.46
Info: line 5 have bit1 valid float value = 28.85
Info: line 5 have bit2 valid float value = -82.35
Info: line 6 have bit1 valid float value = 27.23
Info: line 6 have bit2 valid float value = -80.97
Info: line 7 have bit1 valid float value = 19.83
Info: line 7 have bit2 valid float value = -155.98
Info: line 8 have bit1 valid float value = 40.29
Info: line 8 have bit2 valid float value = -82.74
Info: line 9 have bit1 valid float value = 41.64
Info: line 9 have bit2 valid float value = -87.12
Info: line 10 have bit1 valid float value = 41.98
Info: line 10 have bit2 valid float value = -89.56
Info: line 11 have bit1 valid float value = 40.03
Info: line 11 have bit2 valid float value = -89.13
Info: line 12 have bit1 valid float value = 38.18
Info: line 12 have bit2 valid float value = -89.81
Info: line 13 have bit1 valid float value = 38.73
Info: line 13 have bit2 valid float value = -94.93
Info: line 14 have bit1 valid float value = 31.95
Info: line 14 have bit2 valid float value = -89.24
Info: line 15 have bit1 valid float value = 43.95
Info: line 15 have bit2 valid float value = -86.42
Info: line 16 have bit1 valid float value = 46.30
Info: line 16 have bit2 valid float value = -95.71
--------------------------------
proc file Composite.txt
--------------------------------
Error: not know line 1 and have 8 word
Info: line 2 have bit1 valid float value = 45.47
Info: line 2 have bit2 valid float value = -105.46
Info: line 3 have bit1 valid float value = 28.85
Info: line 3 have bit2 valid float value = -82.35
Info: line 4 have bit1 valid float value = 27.23
Info: line 4 have bit2 valid float value = -80.97
Info: line 5 have bit1 valid float value = 41.64
Info: line 5 have bit2 valid float value = -87.12
Info: line 6 have bit1 valid float value = 41.98
Info: line 6 have bit2 valid float value = -89.56
Info: line 7 have bit1 valid float value = 40.03
Info: line 7 have bit2 valid float value = -89.13

That write have error. I look hexa and file read start:
Code: [Select]
EF BB FF 5B 30 30 .....
That "EF BB FF" is make error and confuse extractword or TStrings ... i not know why there ? is unicode ?
Title: Re: DataProblems Maybe
Post by: BrunoK on April 24, 2019, 12:01:38 pm
"EF BB FF" seems to be UTF-8 byte order mark (BOM) see https://en.wikipedia.org/wiki/Byte_order_mark
Title: Re: DataProblems Maybe
Post by: Thausand on April 24, 2019, 12:49:07 pm
Thanksy BrunoK.

That good read. I not know and think FFFE and FEFF is bom  (iws many more) :-[

I sorry and not know good how solve program for user JLWest...  :'(
Title: Re: DataProblems Maybe
Post by: lucamar on April 24, 2019, 03:21:46 pm
On the first record I get 8 fields and then it goes to 7 (which is right).

As I surmised: not a problem of code but of data.

You're reading an UTF8 BOM along with the first record which gets taken as the first field (after all, it ends in a '['), so all the rest are off-by one and when you were trying to read the 6th word, it was, from its point of view correctly, returning "K2".

Don't save files with UTF8 BOM, it's an absurd convention invented by Microsoft to avoid having to check if a file really contains UTF-8 data.

If you can't avoid having the files with the UTF-8 BOM, you can load them first in a TMemo and assign the Memo.Lines tio the listbox items.

One other, unimportant, thing: why are you loading the files by hand (with assing, readln, etc.) instead of using ListBox.Items.LoadFromFile()?

ETA By the way, the UTF-8 BOM is not:
Code: [Select]
#$EF + #$BB + #$FFbut
Code: [Select]
#$EF + #$BB + #$BF
Let's be precise with these kind of things :)
Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 04:36:54 pm
@JLWest:
you have file with many data for test ? Example my work also when not data good.

i ask: why change delimiter any time ?

Well I don't need to change the delimiter.

I did change from:

|00CA||7826300||Barstow||United States||K2||35.349333||-116.893333| 

to : [00CA][7826300][Barstow][United States][K2][35.349333][-116.893333]

Which was about 3 hours work.

But I don't need to change anymore or during the running of the program.



 
Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 04:42:40 pm
Attached simple application, that extract the encapsulated strings and puts them into an array.
The array data will start in the array at index 1.
Works fine with your data line.

I'll look at the code. I need the data converted  from the following line to a record:

 ICAO    Haash      City         Country         Code     Lat           Lon
[00CA][7826300][Barstow][United States][K2][35.349333][-116.893333]

To: a record:

 TData = record
   ICAO          : String[8];
   Region        : String[3];
   Hash           : Double;
   HashStr       : String[12];
   Lat              : Double;
   LatStr          : String[12];
   Lon              : Double;
   LonStr          : String[12];
   RCDLine       : String[95];
   Distance       : Double;
   DistanceStr   : String[18];

Title: Re: DataProblems Maybe
Post by: lucamar on April 24, 2019, 04:53:20 pm
Don't know why but your posts keep sticking in the front of my head ... so here is a five-minutes, no-frills, bug-attracting function to load your UTF-8 files into a listbox:
Code: Pascal  [Select]
  1. procedure LoadListFromFile(AListBox: TListBox; const AFileName: String);
  2. {NOTE: should be a Boolean function and check whether the file exists,
  3.        the ListBox exists, the files is of the correct type, etc.}
  4. const
  5.   U8BOM: String[3] = #$EF#$BB#$BF;
  6. var
  7.   AFileStream: TFileStream;
  8.   BOMTest: String[3];
  9. begin
  10.   AFileStream := TFileStream.Create(AFilename, fmOpenRead);
  11.   try
  12.     BOMTest[0] := #3;
  13.     AFileStream.Read(BOMTest[1], 3);
  14.     if BOMTest <> U8BOM then
  15.       {Rewind if no BOM}
  16.       AFileStream.Seek(0, soFromBeginning);
  17.     AListBox.Items.LoadFromStream(AFileStream);
  18.   finally
  19.     FileStream.Free;
  20.   end;
  21. end;

I need the data converted  from the following line to a record:
Code: [Select]
ICAO    Haash      City         Country         Code     Lat           Lon
[00CA][7826300][Barstow][United States][K2][35.349333][-116.893333]

To: a record:
Code: [Select]
TData = record
   ICAO          : String[8];
   Region        : String[3];
   Hash           : Double;
   HashStr       : String[12];
   Lat              : Double;
   LatStr          : String[12];
   Lon              : Double;
   LonStr          : String[12];
   RCDLine       : String[95];
   Distance       : Double;
   DistanceStr   : String[18];
 end;

Once the problems with loading the files are solved, that should be easy; just a matter of
Code: [Select]
  Data.WhateverField := ExtractWord(X, TheLine)and then generating the other (calculated?) fields.
Title: Re: DataProblems Maybe
Post by: Thausand on April 24, 2019, 04:56:21 pm
ETA By the way, the UTF-8 BOM is not:
Code: [Select]
#$EF + #$BB + #$FFbut
Code: [Select]
#$EF + #$BB + #$BF
Let's be precise with these kind of things :)
I sorry lucamar. I make copy-paste error  :-[ (is better write read-write error because hexy-edit no have copy-paste).
Title: Re: DataProblems Maybe
Post by: Thausand on April 24, 2019, 04:59:42 pm
Once the problems with loading the files are solved, that should be easy; just a matter of
Code: [Select]
  Data.WhateverField := ExtractWord(X, TheLine)and then generating the other (calculated?) fields.
I have question. If data utf-8 then record string short and extractword not work. so make ansi. Then away  utf-8 codec and not can write fancy letter greek, hyroglyph etc ?

add:

Oh, you have clever bom skip  :)

i write more wrong all ways skip  :D
Code: Pascal  [Select]
  1.   ...
  2.     FileStream:= TFileStream.Create(Filename, fmOpenRead);
  3.     FileStream.Position:= 3;
  4.     Lines.Clear;
  5.     Lines.LoadFromStream(FileStream);
  6.     FileStream.Free;
  7.   ...
Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 05:07:32 pm
On the first record I get 8 fields and then it goes to 7 (which is right).

As I surmised: not a problem of code but of data.

You're reading an UTF8 BOM along with the first record which gets taken as the first field (after all, it ends in a '['), so all the rest are off-by one and when you were trying to read the 6th word, it was, from its point of view correctly, returning "K2".

Don't save files with UTF8 BOM, it's an absurd convention invented by Microsoft to avoid having to check if a file really contains UTF-8 data.

If you can't avoid having the files with the UTF-8 BOM, you can load them first in a TMemo and assign the Memo.Lines tio the listbox items.

One other, unimportant, thing: why are you loading the files by hand (with assing, readln, etc.) instead of using ListBox.Items.LoadFromFile()?

ETA By the way, the UTF-8 BOM is not:
Code: [Select]
#$EF + #$BB + #$FFbut
Code: [Select]
#$EF + #$BB + #$BF
Let's be precise with these kind of things :)

UTF8 BOM   <--- No Idea what that is.

The data is extracted from a file of 7.9 million records. And I guess the 7.9 million records are UTF8 BOM.

"One other, unimportant, thing: why are you loading the files by hand (with assing, readln, etc.) instead of using ListBox.Items.LoadFromFile()?"

Well basically I'm reading a text file into a listbox.

1. I know how to do it this.
2. Habit. Maybe Bad Habit.


"If you can't avoid having the files with the UTF-8 BOM, you can load them first in a TMemo and assign the Memo.Lines tio the listbox items."

Don't understand the  "and assign the Memo.Lines to the listbox items.

Listbox1.Items.Add(Line) :=   Memo.Lines ???

There are two data files for the program. One is 38,000 records and the other is about 16,000. I guess both are  UTF-8 BOM.

If I load the 38,000 records into a Memo1 and then load them into a listbox and the save them to a text file.

Will that get rid of the UTF-8 BOM in the text file?

Well I can't avoid it as the data is extracted from a 7.9 million data set.
 
Not against pre-processing the file into ASCII if there is a way.

 
Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 05:11:16 pm
Once the problems with loading the files are solved, that should be easy; just a matter of
Code: [Select]
  Data.WhateverField := ExtractWord(X, TheLine)and then generating the other (calculated?) fields.
I have question. If data utf-8 then record string short and extractword not work. so make ansi. Then away  utf-8 codec and not can write fancy letter greek, hyroglyph etc ?

"Once the problems with loading the files are solved,"

Yea, I need to convert the UTF-8 BOM file to ASCII text. Then this problem will be solved.

Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 05:29:07 pm
I'm going to write a program to load the the 38,000 records into a memo1 box lucamar's  LoadListFromFile procedure and then into a listbox and finally write to an ASCII text file.

If I can get this to work I will move the LoadListFromFile procedure back in the processing chain and use it to convert the file when I extract the 38,000 records  from the 7.9 million records.

Title: Re: DataProblems Maybe
Post by: lucamar on April 24, 2019, 05:33:18 pm
UTF8 BOM   <--- No Idea what that is.

BOM means Byte-Order Mark and by convention is the Unicode zero-width space. Head over to wikipedia to learn more: Byte order mark (https://en.wikipedia.org/wiki/Byte_order_mark)

Quote
The data is extracted from a file of 7.9 million records. And I guess the 7.9 million records are UTF8 BOM.

No. The mark is for the file and there should be only one, at the beginning.

Quote
"If you can't avoid having the files with the UTF-8 BOM, you can load them first in a TMemo and assign the Memo.Lines tio the listbox items."

Don't understand the  "and assign the Memo.Lines to the listbox items.

Listbox1.Items.Add(Line) :=   Memo.Lines ???

Rather like this:
Code: [Select]
Listbox1.Items.AddStrings(Memo.Lines, True)
Quote
If I load the 38,000 records into a Memo1 and then load them into a listbox and the save them to a text file.

Will that get rid of the UTF-8 BOM in the text file?

Yes, it should. Lazarus never saves an UTF-8 BOM (unless it's already there, of course).

Quote
Not against pre-processing the file into ASCII if there is a way.

Easiest way? Use file streams and copy from the fourth byte of the source. If you wait a litle I will write you function to do it.

I'm going to write a program to load the the 38,000 records into a memo1 box lucamar's  LoadListFromFile procedure and then into a listbox and finally write to an ASCII text file.

Before doing that, wait a little time and I'll write you a more convenient function involving much less memory. Allow me half an hour or so, OK?


I have question. If data utf-8 then record string short and extractword not work. so make ansi. Then away  utf-8 codec and not can write fancy letter greek, hyroglyph etc ?

No, the problem here is that the file has a BOM that is not being taken account of. FPC and Lazarus, in general, can deal perfectly with mixing strings of various types in differents encodings. If anything, the compiler will warn that some automatic conversion or other may result in lost data which means that yes: assigning hieroglyphics to a short string may not work well.

If the need arises one can always convert the UTF8 string to a string declared with an appropiate codepage, from which conversion to a short string is normally direct, char to char. Of course, if the short string has to be stored to a file (for example) one should take care of storing some sort of reminder of the codepage in which it's stored ... or convert it again to UTF-8 and store that.
Title: Re: DataProblems Maybe
Post by: lucamar on April 24, 2019, 06:08:23 pm
Ok, done. A five-minutes, no-frills, etc. BOM cleaning proc.

Code: Pascal  [Select]
  1. procedure CorrectFile(
  2.   const ASrcName: String; const ADestName: String);
  3. var
  4.   SrcFile, DestFile: TFileStream;
  5.   BOM: String[3] = '   ';
  6. begin
  7.   SrcFile := TFileStream.Create(ASrcName, fmOpenRead or fmShareDenyWrite);
  8.   try
  9.     DestFile := TFileStream.Create(ADestName, fmCreate or fmShareExclusive);
  10.     try
  11.       SrcFile.Read(BOM[1], 3); { Read over the BOM }
  12.       DestFile.CopyFrom(SrcFile, SrcFile.Size - 3); {and copy the rest}
  13.     finally
  14.       DestFile.Free;
  15.     end;
  16.   finally
  17.     SrcFile.Free;
  18.   end;
  19. end;

One caveat: Make sure that you don't pass the same name as source and destination. Rather rename the original to, for example, "ByAirport.original" and call the procedure as:
Code: [Select]
CorrectFile('ByAirport.original', 'ByAirport.txt')
Also note that there is no test to see if the origin actually has a BOM: it just skips the first three bytes, whatever they are. You should add a test like the one in my post above (https://forum.lazarus.freepascal.org/index.php/topic,45150.msg318811.html#msg318811)
Title: Re: DataProblems Maybe
Post by: lucamar on April 24, 2019, 06:42:16 pm
Oops! Didn't see this:

i write more wrong all ways skip  :D
Code: Pascal  [Select]
  1.   ...
  2.     FileStream:= TFileStream.Create(Filename, fmOpenRead);
  3.     FileStream.Position:= 3;
  4.     Lines.Clear;
  5.     Lines.LoadFromStream(FileStream);
  6.     FileStream.Free;
  7.   ...

The problem with the above code (besides not checking if there is actually a BOM) is that FileStream.Position should be set to four (4), not three (3). :)
Title: Re: DataProblems Maybe
Post by: Thausand on April 24, 2019, 06:59:24 pm
I have question. If data utf-8 then record string short and extractword not work. so make ansi. Then away  utf-8 codec and not can write fancy letter greek, hyroglyph etc ?

No, the problem here is that the file has a BOM that is not being taken account of. FPC and Lazarus, in general, can deal perfectly with mixing strings of various types in differents encodings. If anything, the compiler will warn that some automatic conversion or other may result in lost data which means that yes: assigning hieroglyphics to a short string may not work well.

If the need arises one can always convert the UTF8 string to a string declared with an appropiate codepage, from which conversion to a short string is normally direct, char to char. Of course, if the short string has to be stored to a file (for example) one should take care of storing some sort of reminder of the codepage in which it's stored ... or convert it again to UTF-8 and store that.
Thank you for answer and explain lucamar. Ok, then i know  :)

Oops! Didn't see this:
Is good  I add many later to post  :)

Quote
The problem with the above code (besides not checking if there is actually a BOM) is that FileStream.Position should be set to four (4), not three (3). :)
??

file JLWest BOM have 3 bytes
byte 1, position 0
byte 2, position 1
byte 3, position 2

then ansi start position 3, then position is  index (start 0) ?


i have make many wrong BOM detect, please not use. is only many smell dirty example
Code: Pascal  [Select]
  1. procedure HaveBOM(aStream: TStream);
  2. type
  3.   TBummer = record Name:String;BOM:RawByteString;end;
  4. const
  5.   Bummer: array[0..14] of TBummer =
  6.   (
  7.     (Name: 'UTF-7'      ; BOM: #$2B#$2F#$76#$38#$2D),
  8.     (Name: 'UTF-32 (BE)'; BOM: #$00#$00#$FE#$FF),
  9.     (Name: 'UTF-32 (LE)'; BOM: #$FF#$FE#$00#$00),
  10.     (Name: 'UTF-7'      ; BOM: #$2B#$2F#$76#$38),
  11.     (Name: 'UTF-7'      ; BOM: #$2B#$2F#$76#$39),
  12.     (Name: 'UTF-7'      ; BOM: #$2B#$2F#$76#$2B),
  13.     (Name: 'UTF-7'      ; BOM: #$2B#$2F#$76#$2F),
  14.     (Name: 'UTF-EBCDIC' ; BOM: #$DD#$73#$66#$73),
  15.     (Name: 'GB-18030'   ; BOM: #$84#$31#$95#$33),
  16.     (Name: 'UTF-8'      ; BOM: #$EF#$BB#$BF),
  17.     (Name: 'UTF-1'      ; BOM: #$F7#$64#$4C),
  18.     (Name: 'SCSU'       ; BOM: #$0E#$FE#$FF),
  19.     (Name: 'BOCU-1'     ; BOM: #$FB#$EE#$28),
  20.     (Name: 'UTF-16 (BE)'; BOM: #$FE#$FF),
  21.     (Name: 'UTF-16 (LE)'; BOM: #$FF#$FE)
  22.   );
  23. var
  24.   Buffer : AnsiString;
  25.   Bum    : TBummer;
  26. begin
  27.   if AStream.Size > 4 then SetLength(Buffer, 5) else
  28.   begin
  29.     Print('BOM detection not good but try');
  30.     SetLength(Buffer, AStream.Size);
  31.   end;
  32.   AStream.Read(Buffer[1], Length(Buffer));
  33.  
  34.   for BUM in Bummer do
  35.   begin
  36.     if Buffer.StartsWith(BUM.BOM) then
  37.     begin
  38.       Print('file have bom ', BUM.Name);
  39.       exit;
  40.     end;
  41.   end;
  42.   Print('file have not BOM');
  43.  
  44.   // no BOM and reset stream
  45.   AStream.Position := AStream.Position - Length(Buffer);
  46. end;
  47.  
  48.  
  49. procedure procfiles(Filenames: array of string);
  50. var
  51.   FileStream : TFileStream;
  52.   Filename   : string;
  53.   Lines      : TStringList;
  54. begin
  55.   Lines:= TStringList.Create;
  56.   for Filename in Filenames do
  57.   begin
  58.     Print('--------------------------------');
  59.     Print('proc file %s', [filename]);
  60.     Print('--------------------------------');
  61.     FileStream:= TFileStream.Create(Filename, fmOpenRead);
  62.     HaveBOM(FileStream);
  63.     Lines.Clear;
  64.     Lines.LoadFromStream(FileStream);
  65.     FileStream.Free;
  66.     proclines(lines);  // this use example process file JLWest. no here
  67.   end;
  68.   Lines.Free;
  69. end;
  70.  
  71. begin
  72.   procfiles(['ByAirport.txt','Composite.txt']);
  73. end.
  74.  
Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 07:28:38 pm
@lucmar

I will implement the

procedure CorrectFile( const ASrcName: String; const ADestName: String);

 Thanks
Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 08:22:28 pm
Worked I think.

Just tested but appears to be working.

Thanks
Title: Re: DataProblems Maybe
Post by: JLWest on April 24, 2019, 09:42:16 pm
Yes the conversion from UTF8 BOM worked and now I make the call to
RCD := Decompose(RCD); and it passes me back a RCD with all of the fields filled in perfectly with correct data.
Title: Re: DataProblems Maybe
Post by: lucamar on April 24, 2019, 10:56:48 pm
Yes the conversion from UTF8 BOM worked and now I make the call to
RCD := Decompose(RCD); and it passes me back a RCD with all of the fields filled in perfectly with correct data.

Glad it all worked. :)


Quote from: lucamar
The problem with the above code (besides not checking if there is actually a BOM) is that FileStream.Position should be set to four (4), not three (3). :)
??

file JLWest BOM have 3 bytes
byte 1, position 0
byte 2, position 1
byte 3, position 2

then ansi start position 3, then position is  index (start 0) ?

Hmm ... you're right, Stream.Position is zero-based, so three is the correct value.

Don't know what I was thinking about  :-[
Title: Re: DataProblems Maybe
Post by: JLWest on April 25, 2019, 03:44:24 am
I would like to thank all for the help I received.

When I get this working (the program finished) I would be willing to place it on my GDrive along with the large data set so you can run it and see what I'm try to accomplish.

Just as an aside I wouldn't mind in the least if someone would look at the code with a critical eye and make suggestions. I don't mind the criticism, I know I'm not very good and at my age not likely to improve much. But I do read the code and try my best to understand it.

It's very easy to see from the code provided during these problems there are elegant ways of doing things and there are brute blunt force trauma ways (that's me).

I have learned that the Hero Members are experts. What takes me days they do in 10 min. The difficult they do in a minute, the very difficult they do with ease and the impossible takes 15 minutes.

Thank you all.



Title: Re: DataProblems Maybe
Post by: lucamar on April 25, 2019, 03:51:59 am
I have learned that the Hero Members are experts. What takes me days they do in 10 min. The difficult they do in a minute, the very difficult they do with ease and the impossible takes 15 minutes.

Wel, I thank you. Even though that "hero member" thing just means I have posted way too much in the forum :D

For some strange reason your problems and doubts almost always resonate with me in such a way that I feel compelled to give my utmost to find a nice solution. And I've so much fun doing it ...
Title: Re: DataProblems Maybe
Post by: Thausand on April 25, 2019, 03:14:32 pm
Hmm ... you're right, Stream.Position is zero-based, so three is the correct value.

Don't know what I was thinking about  :-[
I think i not understand that i write. Many time i not think about and write wrong. Then i can make shake hand  :D


Yes the conversion from UTF8 BOM worked and now I make the call to
RCD := Decompose(RCD); and it passes me back a RCD with all of the fields filled in perfectly with correct data.
that many good !! i make congratulation. Now can make more program  :)
Title: Re: DataProblems Maybe
Post by: Thausand on April 25, 2019, 03:33:02 pm
I would like to thank all for the help I received.
Make help is good me for learn :) Then i thank you for post problem  :)

Quote
I don't mind the criticism, I know I'm not very good and at my age not likely to improve much. But I do read the code and try my best to understand it.
I think is improve if learn that it no matter what you have result and our have result and not have same data. Have good data and know data is many important. I think that if result you and help not same then give data for test  :)

If understand you code write then is ok. If only code for you and not work or any one then it no many important.
Title: Re: DataProblems Maybe
Post by: JLWest on April 25, 2019, 10:10:51 pm
@lucamar

I'm still have a few residual UT-8 BOM  issues. So i' wonder if I can fix this once for good.

Here is an armature diagram of the data set I require:
   
   Apt.Dat file 7.9 million records. ( I suspect UT-8 BOM File and the root of the problem)

       Airports.txt Extracted from the 7.9 million Apt.Dat file  36,000 + records
 
          APList.txt Extracted from the 7.9 million Apt.Dat 36,000 +

          Regional.txt   Extracted from the Airports.txt  9,000  + records

              AirportsRegions.txt extracted from the Regional.txt    9,000 + Same size as
                   Regional.txt record for record just different format

               RegionByCountry extracted from the AirportsRegions.txt 9,000 +

So my questions is:

Can I write a program using your
procedure CorrectFile( const ASrcName: String; const ADestName: String); to convert the 7.9 Million records in the Apt.Dat file to ASCII.

Because all the data I use comes from the 7.9 million file if I fix it I won't be haveing any more  data issues.

 

Title: Re: DataProblems Maybe
Post by: john horst on April 26, 2019, 12:31:40 am
@JLWest If you just want to use the file and have access to sed (diy) or dos2unix (fix it for you) will do what you want. In parallel on the cli you can achieve this in seconds. Just throwing that out there.
Title: Re: DataProblems Maybe
Post by: JLWest on April 26, 2019, 12:52:33 am
@ John Horst

I didn't understand what you said.

Maybe something about online?
Title: Re: DataProblems Maybe
Post by: john horst on April 26, 2019, 01:08:22 am
 :) Basically, dos2unix is a *nix tool to remove the junk some MS tools like to add. https://www.liquidweb.com/kb/dos2unix-removing-hidden-windows-characters-from-files/

Apparently there is a version for Windows that can be run in your powershell or whatever it is called on windows. https://sourceforge.net/projects/dos2unix/
Title: Re: DataProblems Maybe
Post by: JLWest on April 26, 2019, 01:16:23 am
This is pretty funny. not funny, funny. I have a procedure that @lucamar wrote as an example.

It basically loads a file into a TFileStream and dumps it into a listbox. It strips out the UT-8 BOM problem on the load.

So I thought I would see if I could run the 7.9 million records thru the procedure and then write them out to a clean text file.

Well I got it running and and it loaded about 45% of the 7.9 mil records and then quit.

The records are in the list box and the program is still running.

But this will work.

All I need to do is load  2.4 million records, write them out repeat until done. Maybe have 3 or 4 files. Thenp ut the three or 4 files into one file.

I could even rebuild the 7.9 million.

What I extract from the 7.9 million is only 38,000 records. 

 
Title: Re: DataProblems Maybe
Post by: JLWest on April 26, 2019, 01:19:10 am
:) Basically, dos2unix is a *nix tool to remove the junk some MS tools like to add. https://www.liquidweb.com/kb/dos2unix-removing-hidden-windows-characters-from-files/

Apparently there is a version for Windows that can be run in your powershell or whatever it is called on windows. https://sourceforge.net/projects/dos2unix/

I check it out, thanks.
Title: Re: DataProblems Maybe
Post by: lucamar on April 26, 2019, 01:39:30 am
Can I write a program using your
procedure CorrectFile( const ASrcName: String; const ADestName: String); to convert the 7.9 Million records in the Apt.Dat file to ASCII.

Yes, of course. In fact I made one to test the function before posting, if you want it.

But let me polish it a little before uploading :)

Well I got it running and and it loaded about 45% of the 7.9 mil records and then quit.

The records are in the list box and the program is still running.

I've run sometimes into similar issues: seems to be a gotcha of TStrings when the strings (or the count) grow too much--as you know already from your other posts about that giant file :)
Title: Re: DataProblems Maybe
Post by: JLWest on April 26, 2019, 01:45:34 am
@lucumar

If your code gets any smarter it will be able to write by it;s self without our.

I'll wait.
Title: Re: DataProblems Maybe
Post by: lucamar on April 26, 2019, 04:52:27 am
If your code gets any smarter it will be able to write by it;s self without our.

Nah! The thing is that it was just a quick and dirty test of the BOM clean up function. I had to pretty it up a little to be almost production-ready.

Note that "almost": it lacks a couple or three hours more to get it really ready for public consumption. As it is now it just works, without frills. :)

Find it attached.

Oh, and sorry for the delay; had a couple personal matters to attend to.
Title: Re: DataProblems Maybe
Post by: JLWest on April 28, 2019, 10:31:07 am


Code: Pascal  [Select]
  1. function TForm1.Decompose(ARCD : TData ) : TData;
  2.    var RCD : TData;
  3.     Count  : Integer;
  4.     AWord  : String;
  5.     AFloat : Extended;
  6.     Delims : TSysCharSet;
  7.     RCDString : String[95];
  8.     LatLon  : Extended;
  9.    begin
  10.     RCD := aRCD;
  11.     Delims := ['[',']'];
  12.     {Extract all words}
  13.     RCDString := RCD.RCDLine;
  14.     Count := 0;
  15.     repeat
  16.       Inc(Count);
  17.      RCDString := '[00VA][5890633][Alton][United States][K6][36.576][-78.999166667]';
  18.       AWord := ExtractWord(Count, RCDString, Delims);
  19.       if TryStrToFloat(AWord, AFloat) then  begin LatLon := AFloat; end;
  20.        Case Count of
  21.         0  : ShowMessage('Bad Data  in RCD in Decompose Function');
  22.         1  : RCD.ICAO := AWord;
  23.         2  : begin
  24.              RCD.Hash := LatLon;
  25.              RCD.HashStr := FloatToStr(LatLon);
  26.              end;
  27.         3  :  RCD.City :=  AWord;
  28.         4  :  RCD.Country := AWord;
  29.         5  :  RCD.Region := AWord;
  30.         6  :  Begin
  31.                RCD.Lat := LatLon;
  32.                RCD.LatStr := FloatToStr(LatLon);
  33.               end;
  34.         7  :  Begin
  35.                RCD.Lon := LatLon;
  36.                RCD.LonStr := FloatToStr(LatLon);
  37.               end;
  38.        end;
  39.     until AWord.IsEmpty;
  40.     Result := RCD;
  41.   end;  

I still have data problems. I have redone the data 3 times today and can't get past this problem.

The issue
Code: Pascal  [Select]
  1. unit1.pas(724,6) Fatal: illegal character "'ï'" ($EF)
on line 17.

So under the debugger I copied the data raw and pasted it  into RCDString just before I try to parse it. Still get the issue. I think the $EF is the UTF8 BOM.

I'm more than wiling to post this on my GDrive if someone is willing to look at it.

It's easy to duplicate. Start the program and click once on the first record in a listbox2.

Dosn't happen on any other listbox.

I have five listboxes on the screen and it only  happens on the one.

I change the first record and second record gets the issue.

But If I click one the second record first it's fine. Go back to the first record and I get the issue.

Is there a way aroud this?



 

Title: Re: DataProblems Maybe
Post by: lucamar on April 28, 2019, 11:36:25 am
I'm more than wiling to post this on my GDrive if someone is willing to look at it.

Yeah, let me give it a look. But you have to be quick on the upload. The planning says this coming week I'm not going to sleep  :P

Remember to upload everything: program, data files, ... everything.

And be patient ... although I know you're. ;)
Title: Re: DataProblems Maybe
Post by: JLWest on April 28, 2019, 08:33:56 pm
Program Attached. The likes are for the for the three files the program loads

https://drive.google.com/open?id=1ELUs7J7u3JZEokvXRloGDlgBGFAdi5cR
https://drive.google.com/open?id=1JG67hET6dz7j-WGUMPGpcKJtmn55s9XS
https://drive.google.com/open?id=1kWc1McsRqQHt6y8UgVewCVWmkJXfOFt4

Title: Re: DataProblems Maybe
Post by: JLWest on April 28, 2019, 08:40:08 pm
One of the data files and the program have the same name. MSTRegions.

So there should be 1 program fie posted here and on the Gdrive  and three data files.


https://drive.google.com/open?id=1Ui4xrVTBAxtQTuLlnUiHvlrJxpaGTQIV
Title: Re: DataProblems Maybe
Post by: JLWest on April 28, 2019, 08:53:04 pm
To Recreate the issue step by step:

Shift-F9
Start it under the debugger.

When the program comes up there are 5 list boxes.

It's a very busy screen.

The far left listbox is not loaded by a serrate file. It's extracted while loading by airports.

When it loads double click on the lower middle listbox top record.

should be:
[00VA][5890633][Alton][United States][K6][36.576][-78.999166667]

The issue will come up at line 628.



Title: Re: DataProblems Maybe
Post by: lucamar on April 28, 2019, 09:21:27 pm
OK, downloaded all five files.

I'll give it a look after supper or tomorrow--you have left it for a little late in the day. ;) I'll tell you something afterwards, at least after analyzing the data files (should only take some minutes).
Title: Re: DataProblems Maybe
Post by: JLWest on April 28, 2019, 09:40:49 pm
Thank you.

I made the post last night at 2:00 in the morning. Was going to load the data up but wanted to include some documentation on your copyright and a readme but it was too late form me.
Title: Re: DataProblems Maybe
Post by: JLWest on April 28, 2019, 10:19:56 pm
I have made a change to the program: I added  'FileLocSW  : Boolean = TRUE;' as a Global Var;

When set to true the program and data files can be dumped into a temp directory and executed from one directory.

When Your done delete the directory.

No path problems or changes needed.

Program Attached



   
Title: Re: DataProblems Maybe
Post by: lucamar on April 28, 2019, 11:19:12 pm
[...] wanted to include some documentation on your copyright [...]

There is no copyright, don't worry about it. Almost all of the code (my own, of course) I post in the forum is dedicated to the public domain: I'd find ridiculous for it to be otherwise; I mean, full-fledged copyright for less than 20 common LOC??? If anything, a bare citation down in the credits suffices: "lucamar helped me!"  :P


About the CC0 in full program...

The full programs in attachments now have a CC0 dedication. That's is mainly because our lawyer recommended it.

He said that having a full-deed, legal dedication might strengthen it in countries (like Spain and most of continental Europe) where some author rights are non-renounceable (v.g. moral rights). In those cases the legal deed leaves clear that you renounce all other "rights" and that you "promise" to not enforce those you can't renounce.

As that is (almost) exactly what I want, I followed his advice. :)


ETA: By the way, I checked the data files and all of them are pure test except MstRegions.txt, which has a UTF8 BOM, but that one can easily be converted with cleanbom.

But I won't be able to check on the program until late tomorrow, sorry. :(

I've given it a quick visual and you still use too much the old style of file management. When will you learn to use streams or the numerous "LoadFrom/SaveToFile" which most classes dealing with strings have? ;D
Title: Re: DataProblems Maybe
Post by: JLWest on April 29, 2019, 12:11:47 am
"ETA: By the way, I checked the data files and all of them are pure test except MstRegions.txt, which has a UTF8 BOM, but that one can easily be converted with cleanbom."

I can't get Cleanbom to work. I Have never run the DOS box programs. (Years a go)



I think I get in a dos box cd to where the Cleanbom resides and do the following

Cleanbom E:\files\Regions\MstRegions.txt  E:\files\Regions\MstRegions.bak


But I won't be able to check on the program until late tomorrow, sorry. :

"When will you learn to use streams or the numerous "LoadFrom/SaveToFile" which most classes dealing with strings"

I'll work on trying to use the "LoadFrom/SaveToFile" and if I can figure it out I will make the change.

Take a look at my next post it may help some.



Title: Re: DataProblems Maybe
Post by: JLWest on April 29, 2019, 12:26:29 am
Some Things I have tried.

Remember Your:  Test/demo ExtractWord
                ref: DataProblems Maybe
                     https://forum.lazarus.freepascal.org/index.php/topic,45150.0.html 

I Started it and pointed it a the MstRegions file.

On the first record it Decomposed 11 fields. (That''''s really a nice little utility program).
But all the rest of the records were good.

Just a thought: You also wrote a procedure to get rid of the UTF8 BOM file and save it to a ASCII file.

So I'm thinking about taking the ExtractWord Demo and add the procedure to it.

That way I could bring up a file, see if it is extracting ok, if not then save it as an ASCII.

All of this data comes from one source = Apt.Dat, 7.0 Million lines.

 A program reads that and produces the ByAirports.txt.


          Apt.dat  7.9 Million 

     ByAirports.txt          36,000 +- Extrated from Apt.dat file
        MstRegions.txt       Just an extract of ByAirports.txt records without and 'Nil'
        CountryRegions.txt   Extracted from ByAirports.txt

In the program MstRegions I load in the left hand listbox a list of all the airport ICAO's  (KPHX, KLAS ect.) extracted from MstRegions.txt  when I load MstRegions.txt.

In the middle of the screen, Top istbox I load  ByAirports.txt 36,000 +- record. And in the lower middle box I load MstRegions.txt 9,600+-.
In the far right Lstbox I load CountryRegions.txt which was extracted from MstRegions. this was done in a seperat program.

In order to do what I really want to do I need a clean file with no 'Nil''s in any record and all records accounted for.

So I took the ByAirports.txt file and extracted record with no 'Nil's and made MstRegions.txt.(seperate program).

Now I can determine the Country field of each record that is 'Nil' in two way's.

First Way:
 
Region field isn't Nil but has a valid code determined from the right listbox then I can plug in the country. This doesn't always work because some countries (i.e US, China, Russia all have multi Regional ). But countries have 1 region.

Second Method:

Use the Haverstine formula. Take the record with the Nil country Listbox2 Click on it and it becomes the target record (Displayed in the TEdit contro at the top of the screen).

Click on a MstRegions record (Lower Middle Listbox). It's record will be displayed a TEdit located in the middle of the screen.

Now click on the Calc Dist button.

It will give you the distance between the two airports. (Worked properly I think).

I checked against Online site.

Record: [00E][6781690][Nil][Nil][Nil][33.49421222][-111.64124806] which
is a few records down in the top listbox is good one to try.

Click on it.

In the bottom Listbox MstRegions try and find the record:

[AZ66][7175661][Mesa][United States][K2][33.340547222][-109.494466667] and click on it.

It's going to tell you that you are 124 mi  from Mesa Arizona.

The web says: 4227 North 88th Street, Mesa, Arizona 85207, United States.
So I may have a bug in my haverstine routing.

Under the middle listboxs there is a TEdit labeled Country with a small button next to it. Press the small button and it changes the target record Country to United States.

With the listboxes and data I can write code to fill in a lot of the 'Nil'. 

After running the program I can save a new MstRegions file.

It has a few bugs I need to fix and resolve the data issue.

NOTE:----------------------------------------------------
So after writing this up I got to thinking.

All data comes from Apt.Dat. I generate a 36,000 ByAirports.txt  file in a seperate program.

   I change the MstRegions program to load the ByAirports.txt and then populate  the Listbox1, Listbox3, and Listbox5 from Listbox2.

So far I no longer get the error on the first record in listbox2.

 

 
 




                               

Title: Re: DataProblems Maybe
Post by: Thausand on May 02, 2019, 05:03:52 am
....
It has a few bugs I need to fix and resolve the data issue.

NOTE:----------------------------------------------------
So after writing this up I got to thinking.

All data comes from Apt.Dat. I generate a 36,000 ByAirports.txt  file in a seperate program.

   I change the MstRegions program to load the ByAirports.txt and then populate  the Listbox1, Listbox3, and Listbox5 from Listbox2.

So far I no longer get the error on the first record in listbox2.
Is good no have error any more  :)

I forget i make small program make data verificate and show. This is tell that data have problem in other way (not only you error).

Code: Text  [Select]
  1. Processing file "ByAirport.txt"
  2.  
  3. ... show unique count
  4.     Line   Error    ICAO    hash    City Country  Region     Lat     lon    Time
  5.    34680       0   34680   34680   14138     259     248   34457   34537   35 sec (968.17 lines/second))
  6.  
  7. Statistics dtICAO :
  8.      Shortest = "00C"  (3 characters)
  9.      Longest  = "XCYCDL8"  (7 characters)
  10.  Lowest count = "00C"  (1 times)
  11. Highest count = "00CA"  (1 times)
  12.  
  13. Statistics dtHash :
  14.      Shortest = "4"  (1 characters)
  15.      Longest  = "1022949"  (7 characters)
  16.  Lowest count = "1021615"  (1 times)
  17. Highest count = "1022949"  (1 times)
  18.  
  19. Statistics dtCity :
  20.      Shortest = "Ie"  (2 characters)
  21.      Longest  = "Aberdeen Proving Grounds(Abe"  (28 characters)
  22.  Lowest count = "(Cagayan De Oro)"  (1 times)
  23. Highest count = "Nil"  (9335 times)
  24.  
  25. Statistics dtCountry :
  26.      Shortest = "CA"  (2 characters)
  27.      Longest  = "British Indian Ocean Territo"  (28 characters)
  28.  Lowest count = "Anguilla"  (1 times)
  29. Highest count = "United States"  (14645 times)
  30.  
  31. Statistics dtRegion :
  32.      Shortest = "AG"  (2 characters)
  33.      Longest  = "Nil"  (3 characters)
  34.  Lowest count = "AN"  (1 times)
  35. Highest count = "Nil"  (22463 times)
  36.  
  37. Statistics dtLat :
  38.      Shortest = "9"  (1 characters)
  39.      Longest  = "33.1497222222222"  (16 characters)
  40.  Lowest count = "-0.019722222"  (1 times)
  41. Highest count = "-23.59055556"  (6 times)
  42.  
  43. Statistics dtLon :
  44.      Shortest = "23"  (2 characters)
  45.      Longest  = "-106.869166666667"  (17 characters)
  46.  Lowest count = "-0.000944"  (1 times)
  47. Highest count = "-46.65805556"  (4 times)
  48.  
  49. ---------- List dtCountry ----------
  50.    1 : (10) "Afghanistan"
  51.    2 : (5) "Albania"
  52.    3 : (47) "Algeria"
  53.    4 : (3) "American Samoa"
  54.    5 : (27) "Angola"
  55.    6 : (1) "Anguilla"
  56.    7 : (9) "Antarctica"
  57.    8 : (3) "Antigua and Barbuda"
  58.    9 : (164) "Argentina"
  59.   10 : (1) "Arizona"
  60.   11 : (5) "Armenia"
  61.   12 : (1) "Aruba"
  62.   13 : (1486) "Australia"
  63.   14 : (54) "Austria"
  64.   15 : (9) "Azerbaijan"
  65.   16 : (38) "Bahamas"
  66.   17 : (2) "Bahrain"
  67.   18 : (13) "Bangladesh"
  68.   19 : (1) "Barbados"
  69.   20 : (9) "Belarus"
  70.   21 : (43) "Belgium"
  71.   22 : (1) "Belize"
  72.   23 : (2) "Benin"
  73.   24 : (1) "Bermuda"
  74.   25 : (3) "Bhutan"
  75.   26 : (38) "Bolivia"
  76.   27 : (4) "Bosnia and Herzegovina"
  77.   28 : (28) "Botswana"
  78.   29 : (3758) "Brazil"
  79.   30 : (1) "British Indian Ocean Territo"
  80.   31 : (2) "British Virgin Islands"
  81.   32 : (2) "Brunei"
  82.   33 : (22) "Bulgaria"
  83.   34 : (2) "Burkina Faso"
  84.   35 : (1) "Burkina Faso7"
  85.   36 : (36) "Burma"
  86.   37 : (1) "Burundi"
  87.   38 : (14) "CA"
  88.   39 : (6) "Cambodia"
  89.   40 : (10) "Cameroon"
  90.   41 : (675) "Canada"
  91.   42 : (9) "Cape Verde"
  92.   43 : (3) "Caribbean Netherlands"
  93.   44 : (1) "Carroll"
  94.   45 : (3) "Cayman Islands"
  95.   46 : (2) "Central African Republic"
  96.   47 : (5) "Chad"
  97.   48 : (176) "Chile"
  98.   49 : (102) "China"
  99.   50 : (1) "Christmas Island"
  100.   51 : (1) "Cocos (Keeling) Islands"
  101.   52 : (92) "Colombia"
  102.   53 : (3) "Comoros"
  103.   54 : (4) "Congo (Brazzaville)"
  104.   55 : (27) "Congo (Kinshasa)"
  105.   56 : (10) "Cook Islands"
  106.   57 : (26) "Costa Rica"
  107.   58 : (18) "Croatia"
  108.   59 : (31) "Cuba"
  109.   60 : (1) "Curaçao"
  110.   61 : (3) "Cyprus"
  111.   62 : (99) "Czechia"
  112.   63 : (7) "Côte d'Ivoire"
  113.   64 : (87) "Denmark"
  114.   65 : (1) "Djibouti"
  115.   66 : (2) "Dominica"
  116.   67 : (15) "Dominican Republic"
  117.   68 : (50) "Ecuador"
  118.   69 : (56) "Egypt"
  119.   70 : (4) "El Salvador"
  120.   71 : (2) "Equatorial Guinea"
  121.   72 : (3) "Eritrea"
  122.   73 : (13) "Estonia"
  123.   74 : (22) "Ethiopia"
  124.   75 : (2) "Falkland Islands"
  125.   76 : (8) "Faroe Islands"
  126.   77 : (21) "Fiji"
  127.   78 : (92) "Finland"
  128.   79 : (3) "FR"
  129.   80 : (441) "France"
  130.   81 : (5) "French Guiana"
  131.   82 : (48) "French Polynesia"
  132.   83 : (11) "Gabon"
  133.   84 : (1) "Gambia"
  134.   85 : (9) "Georgia"
  135.   86 : (504) "Germany"
  136.   87 : (8) "Ghana"
  137.   88 : (1) "Gibraltar"
  138.   89 : (56) "Greece"
  139.   90 : (17) "Greenland"
  140.   91 : (1) "Greenville"
  141.   92 : (2) "Grenada"
  142.   93 : (5) "Guadeloupe"
  143.   94 : (2) "Guam"
  144.   95 : (10) "Guatemala"
  145.   96 : (2) "Guernsey"
  146.   97 : (4) "Guinea"
  147.   98 : (3) "Guinea-Bissau"
  148.   99 : (16) "Guyana"
  149.  100 : (6) "Haiti"
  150.  101 : (10) "Honduras"
  151.  102 : (1) "Hong Kong"
  152.  103 : (76) "Hungary"
  153.  104 : (81) "Iceland"
  154.  105 : (195) "India"
  155.  106 : (181) "Indonesia"
  156.  107 : (88) "Iran"
  157.  108 : (16) "Iraq"
  158.  109 : (36) "Ireland"
  159.  110 : (1) "Isle of Man"
  160.  111 : (21) "Israel"
  161.  112 : (120) "Italy"
  162.  113 : (6) "Jamaica"
  163.  114 : (130) "Japan"
  164.  115 : (1) "Jersey"
  165.  116 : (8) "Jordan"
  166.  117 : (1) "K1"
  167.  118 : (21) "Kazakhstan"
  168.  119 : (14) "Kenya"
  169.  120 : (20) "Kiribati"
  170.  121 : (1) "Kosovo"
  171.  122 : (3) "Kuwait"
  172.  123 : (4) "Kyrgyzstan"
  173.  124 : (8) "Laos"
  174.  125 : (7) "Latvia"
  175.  126 : (3) "Lebanon"
  176.  127 : (5) "Lesotho"
  177.  128 : (2) "Liberia"
  178.  129 : (41) "Libya"
  179.  130 : (1) "Liechtenstein"
  180.  131 : (7) "Lithuania"
  181.  132 : (3) "Luxembourg"
  182.  133 : (1) "Macau"
  183.  134 : (12) "Macedonia"
  184.  135 : (28) "Madagascar"
  185.  136 : (7) "Malawi"
  186.  137 : (43) "Malaysia"
  187.  138 : (6) "Maldives"
  188.  139 : (8) "Mali"
  189.  140 : (2) "Malta"
  190.  141 : (7) "Marshall Islands"
  191.  142 : (1) "Martinique"
  192.  143 : (10) "Mauritania"
  193.  144 : (2) "Mauritius"
  194.  145 : (1) "Mayotte"
  195.  146 : (132) "Mexico"
  196.  147 : (4) "Micronesia"
  197.  148 : (3) "Moldova"
  198.  149 : (1) "Monaco"
  199.  150 : (22) "Mongolia"
  200.  151 : (3) "Montenegro"
  201.  152 : (1) "Montserrat"
  202.  153 : (26) "Morocco"
  203.  154 : (18) "Mozambique"
  204.  155 : (1) "Myanmar"
  205.  156 : (55) "Namibia"
  206.  157 : (1) "Namibia."
  207.  158 : (1) "Nauru"
  208.  159 : (41) "Nepal"
  209.  160 : (34) "Netherlands"
  210.  161 : (17) "New Caledonia"
  211.  162 : (109) "New Zealand"
  212.  163 : (10) "Nicaragua"
  213.  164 : (7) "Niger"
  214.  165 : (27) "Nigeria"
  215.  166 : (7099) "Nil"
  216.  167 : (1) "Niue"
  217.  168 : (1) "Norfolk Island"
  218.  169 : (1) "North Korea"
  219.  170 : (9) "Northern Mariana Islands"
  220.  171 : (142) "Norway"
  221.  172 : (6) "Oman"
  222.  173 : (68) "Pakistan"
  223.  174 : (3) "Palau"
  224.  175 : (2) "Palestinian Territory"
  225.  176 : (22) "Panama"
  226.  177 : (75) "Papua New Guinea"
  227.  178 : (11) "Paraguay"
  228.  179 : (1) "Peoples Republic of China"
  229.  180 : (1) "Peru"
  230.  181 : (79) "Perú"
  231.  182 : (73) "Philippines"
  232.  183 : (92) "Poland"
  233.  184 : (52) "Portugal"
  234.  185 : (31) "Puerto Rico"
  235.  186 : (3) "Qatar"
  236.  187 : (1) "Republic of Ireland (Eire)"
  237.  188 : (39) "Romania"
  238.  189 : (167) "Russia"
  239.  190 : (7) "Russian Federation"
  240.  191 : (1) "Russin Federation"
  241.  192 : (5) "Rwanda"
  242.  193 : (2) "Réunion"
  243.  194 : (1) "Saint Barthélemy"
  244.  195 : (2) "Saint Helena"
  245.  196 : (2) "Saint Kitts and Nevis"
  246.  197 : (2) "Saint Lucia"
  247.  198 : (1) "Saint Martin"
  248.  199 : (2) "Saint Pierre and Miquelon"
  249.  200 : (5) "Saint Vincent and the Grenad"
  250.  201 : (4) "Samoa"
  251.  202 : (2) "Sao Tome and Principe"
  252.  203 : (62) "Saudi Arabia"
  253.  204 : (10) "Senegal"
  254.  205 : (7) "Serbia"
  255.  206 : (8) "Seychelles"
  256.  207 : (1) "Sierra Leone"
  257.  208 : (6) "Singapore"
  258.  209 : (30) "Slovakia"
  259.  210 : (16) "Slovenia"
  260.  211 : (23) "Solomon Islands"
  261.  212 : (6) "Somalia"
  262.  213 : (109) "South Africa"
  263.  214 : (77) "South Korea"
  264.  215 : (4) "South Sudan"
  265.  216 : (107) "Spain"
  266.  217 : (10) "Sri Lanka"
  267.  218 : (11) "Sudan"
  268.  219 : (8) "Suriname"
  269.  220 : (1) "Swaziland"
  270.  221 : (164) "Sweden"
  271.  222 : (1) "Swiss"
  272.  223 : (70) "Switzerland"
  273.  224 : (23) "Syria"
  274.  225 : (21) "Taiwan"
  275.  226 : (6) "Tajikistan"
  276.  227 : (14) "Tanzania"
  277.  228 : (61) "Thailand"
  278.  229 : (6) "Timor-Leste"
  279.  230 : (2) "Togo"
  280.  231 : (6) "Tonga"
  281.  232 : (2) "Trinidad and Tobago"
  282.  233 : (13) "Tunisia"
  283.  234 : (74) "Turkey"
  284.  235 : (10) "Turkmenistan"
  285.  236 : (7) "Turks and Caicos Islands"
  286.  237 : (1) "Tuvalu"
  287.  238 : (6) "U.S. Virgin Islands"
  288.  239 : (1) "U.S.A."
  289.  240 : (18) "Uganda"
  290.  241 : (26) "Ukraine"
  291.  242 : (19) "United Arab Emirates"
  292.  243 : (251) "United Kingdom"
  293.  244 : (1) "United Sates"
  294.  245 : (14645) "United States"
  295.  246 : (4) "United States Minor Outlying"
  296.  247 : (2) "United States of America"
  297.  248 : (11) "Uruguay"
  298.  249 : (25) "US"
  299.  250 : (26) "USA"
  300.  251 : (15) "Uzbekistan"
  301.  252 : (23) "Vanuatu"
  302.  253 : (100) "Venezuela"
  303.  254 : (24) "Vietnam"
  304.  255 : (2) "Wallis and Futuna"
  305.  256 : (3) "Western Sahara"
  306.  257 : (13) "Yemen"
  307.  258 : (8) "Zambia"
  308.  259 : (17) "Zimbabwe"
  309.  


I not have print list city and is many big and make read.

The country name error is not very good  :(

Have example how many time want write United States ?

I have make highlight lines that is no good and i not even good know all country english write ...

Is city and country cut origin apt.dat ? How big apt.dat unpack ? And can download apt.dat ?
Title: Re: DataProblems Maybe
Post by: JLWest on May 02, 2019, 06:38:35 am
@Thausand

I'll download your demo procedure and run it up.

The apt.dat file is 7.9 million text record.  Can't upload but, I can post it on my Gdrive and post the link here.

It will take me bit.



Title: Re: DataProblems Maybe
Post by: lucamar on May 02, 2019, 07:45:18 am
I can't get Cleanbom to work. I Have never run the DOS box programs. (Years a go)

I think I get in a dos box cd to where the Cleanbom resides and do the following

Cleanbom E:\files\Regions\MstRegions.txt  E:\files\Regions\MstRegions.bak

Basically, yes that's the procedure: put the cleanbom.exe in the same directory where the data files are, open a console window (or DOS prompt, or however it's called in your system), CD to that directory and type the command, Basically:
Code: [Select]
E:
cd E:\files\Regions
cleanbom MstRegions.txt
exit

It it cleaned anything, the clean file will be MstRegions.txt and the original will have been renamed to MstRegions.bak.

I haven't had time to give your program more than a quick look, sorry, but I think today or tomorrow I'll have some free hours to dedicate it. Cross your fingers!

Quote
So I'm thinking about taking the ExtractWord Demo and add the procedure to it.
That way I could bring up a file, see if it is extracting ok, if not then save it as an ASCII.

Yes, you could do that too. Or take all the ideas together and make a "validation" program like Thausand's. It shouldn't be too long. And you get to play with streams ;)

Quote
   I change the MstRegions program to load the ByAirports.txt and then populate  the Listbox1, Listbox3, and Listbox5 from Listbox2.

So far I no longer get the error on the first record in listbox2.

That's because ByAirports.txt is clean already, at least the one I downloaded here. I suspect that at some point or other you created or edited some of those files with an editor that wrote them back as UTF8 with BOM, and there started your problems.

Another good, solid solution may be to create a TFileStream descendant which ignored that BOM and returned only the normal data: you could then use all those nice LoadFromStream methods :) . It shouldn't be too difficult...
Title: Re: DataProblems Maybe
Post by: JLWest on May 02, 2019, 08:17:09 am
@lucumar

I uploaded everything to my  posted links. Program, data files.

Everything can be dumped into a single folder and run from there. Evan a short Readme.


"Yes, you could do that too. Or take all the ideas together and make a "validation" program like Thausand's. It shouldn't be too long. And you get to play with streams"

I have, loading one of the listboxes with streams. CountryRegions.txt. Works like a dream. Can't believe you didn't tell me about this long ago. Lol.

I have to take a look at  @Thausand latest.

I may be offline for a day. Doctors, Hospital. The last time I went in for this I was there for a week.

At the time my kid got some program that would allow me to operate my computer from his laptop using the hospital's network log in as a guest. Pretty neat.

 and 
Title: Re: DataProblems Maybe
Post by: lucamar on May 02, 2019, 11:58:55 am
I may be offline for a day. Doctors, Hospital. The last time I went in for this I was there for a week.

At the time my kid got some program that would allow me to operate my computer from his laptop using the hospital's network log in as a guest. Pretty neat.

Yeah, I know how you feel. I passed a week in the hospital after one the surgeries, got back home ... and had to go back next day with a meningitis. One month in the hospital, all told!

That program ... a (more or less) standard remote desktop, isn't it? I tried some but the hospital's outbound net had so many redirections and indirections that I ended up just reading novels in the tablet :D
Title: Re: DataProblems Maybe
Post by: Thausand on May 02, 2019, 06:37:37 pm
Hi hello JLWest,

I have to take a look at  @Thausand latest.
Oh, i sorry. I forget attach (i make now).

I have make new program (i learn use other class) and is make different idea is verificate. Other program is idea process. Because new program i not have finish (and you was working so i not continue work program). I now have make comment and have refactor. Is no finish but is now better read (i hope)  :)

If want more option then can add you or write what want and maybe i can add. Better is you make add  :P

Please no upload me apt.dat if zip more big 1GB. Is text so make > 10-20GB unpack and no have space for test now.

Quote
I may be offline for a day. Doctors, Hospital. The last time I went in for this I was there for a week.
I make wish well for you.
Title: Re: DataProblems Maybe
Post by: lucamar on May 02, 2019, 08:14:06 pm
Please no upload me apt.dat if zip more big 1GB. Is text so make > 10-20GB unpack and no have space for test now.

It's not so big, just about 300 MiB and as it's text it compresses well: down to around 100 MiB.
Title: Re: DataProblems Maybe
Post by: JLWest on May 02, 2019, 10:45:40 pm
@Tausand

If it's to big then open with Notepad and cut out a hundred records of so.
You can do that with all the text file. 
Title: Re: DataProblems Maybe
Post by: Thausand on May 03, 2019, 07:34:26 pm
It's not so big, just about 300 MiB and as it's text it compresses well: down to around 100 MiB.
Hi hello lucamar,

That then ok. I can make  :)

Thanksy!

@Tausand

If it's to big then open with Notepad and cut out a hundred records of so.
You can do that with all the text file. 
Hi hello JLWest,

Use notepad (i not have, i Linux) then make begin error BOM  ;D

Then make small no good for make good data analyse.

I read specificate "X-Plane APT 1100" and make many question for file "ByAirport.txt". I make new read what want you because think i not understand all. Have example name ariport then specificate write "Text string (up to 40 characters)" and you program MasterAirport make 28   :o

I have question because i look and no see. Were can find apt.dat for download ?
Title: Re: DataProblems Maybe
Post by: JLWest on May 04, 2019, 01:21:14 am
@Thausand Link to my GoogleDrive.


https://drive.google.com/open?id=1J4EanhRBD5gYr4gWADyI4BCB5sN1dTeE

@Lucumar It's called:

https://www.splashtop.com/

Works good. Has a free trial and a subscription of $5US month.
Title: Re: DataProblems Maybe
Post by: JLWest on May 04, 2019, 01:30:01 am
 @Thausand

Notepad or notepad++ can't handle apt.dat very well.

There is a free editor for very large files that wokrs pretty good. It's called Kainet

http://www.kainet.ch/Editor-for-very-large-files.

It is very basic. but fast.

Title: Re: DataProblems Maybe
Post by: Thausand on May 04, 2019, 03:38:38 am
Hi hello JLWest,

@Thausand Link to my GoogleDrive.
...
Thanksy !

I have download and make fast test. result be low

...
It is very basic. but fast.
Thanksy for suggest ! Is more windows and i not windows :P

No need edit and have full analyse data. If want cut then can use linux terminal cut  ;)


I make fast program and test. Not many time when make so maybe error.

result:
Code: [Select]
./aptidat apt.dat
Processing file "apt.dat"

    Line Airport    Land     Sea    Heli  Land_R Water_R Helipad    Time
 7970312   35242   28458     585    6199   34893     784    9719   88 sec (89996.98 lines/second)
Is mean:
Have apt.dat line 7970312 and have airport 35242 (Land = 28458 / Sea = 585 / Heli = 6199). Have airport Land 34893 runway, have airport sea 784 runway, have airport heli 9719 helipad).

I make program continue maybe tomorrow. Is me other weekend busy then maybe no time.
Title: Re: DataProblems Maybe
Post by: JLWest on May 04, 2019, 09:10:51 pm
@Thausand

... show unique count
    Line   Error    ICAO    hash    City Country  Region     Lat     lon    Time
   34680       0   34680   34680   14138     259     248   34457   34537   35 sec (968.17 lines/second))
 

 The Lat and Lon should be the same. Indicates something is wrong with the data.

This is from ByAirports.

Trying to run this down.
May com from the program that generates ByAirports.txt from the Apt.dat file.
Title: Re: DataProblems Maybe
Post by: Thausand on May 04, 2019, 11:20:09 pm
Hi hello JLWest,

The Lat and Lon should be the same. Indicates something is wrong with the data.

Make safe think:

My program no compare lon and lat combine only separate lon and separate lat.

Then is possible many airport have same lon or have same lat. No lon and lat same (that real world no possible)

If want lat and lon compare combine then i can make program have compare and show lat/lon combine unique number. My program no have because "real world not possible have same" and number is same number ICAO. If want verificate numbers then is good for verificate. I no think that before then maybe i add for better verificate.

Thanksy

I think more problem:

ByAirport.dat you:
Code: [Select]
Line    ICAO
34680   34680

my program apt.dat:
Code: [Select]
Airport
35242
If have any airport unique ICAO then 35242 <> 34680 so if make me program "Airports.dat" then not same you "ByAirports.dat". My program make many more ICAO then have "ByAirports.dat" ICAO.

I have question:
1) if apt.dat airport have 1302 lon/lat then is more important runway lon/lat ? My program now take last find. That mean if program find 1302 lon/lat and later find runway lon/lat then my Airport.dat have runway lon/lat.
2) if want search lat/lon for find data "nil" then not important list look have any runway lat/lon and airport lat/lon for search ? (then make list have runway -> ICAO and Airport -> ICAO for look and make more big list)
Title: Re: DataProblems Maybe
Post by: lucamar on May 04, 2019, 11:56:13 pm
The Lat and Lon should be the same. Indicates something is wrong with the data.
Then is possible many airport have same lon or have same lat. No lon and lat same (that real world no possible)

I think he meant that there should be the same total number of Longitude values than of latitude values; otherwise, some data lines have only longitude (or only latitude), which is an error.

But looking closer, you seem to be computing the total of different values for each, isn't it?, so they might in fact be different if some airports have the same longitude and diffferent latitude or viceversa.
Title: Re: DataProblems Maybe
Post by: Thausand on May 05, 2019, 01:47:12 am
Hi hello lucamar,

I think he meant that there should be the same total number of Longitude values than of latitude values; otherwise, some data lines have only longitude (or only latitude), which is an error.
Yes and then you be low write is good write. I make alone unique count (alone for lat and alone for lat, no combine).

If write technical my program then my program have simple (sort and dupignore) list for any column item and then add. TObject then mis use for count how many time is add for write more statistic later. Procedure ProcessLine() can have check for verificate item (column) data. I have make more before snippet have example verificate longitude and latitude and can add there. If no good verificate can write error and error count write if some thing no good.

(i make program simple so can understand and make add own routine/check simple and fast. I not know what is important JLWest for verificate. I know lon and lat have restrict value so that can add and add error when no validate lat/lon or add error when no lon/lat there. Then error-count write if is wrong data and need better read data and/or error-log).

Quote
But looking closer, you seem to be computing the total of different values for each, isn't it?, so they might in fact be different if some airports have the same longitude and diffferent latitude or viceversa.
Yes, that correct. Is count different lat and count different lon separate/different so is count alone unique.

That why program write:
Quote
... show unique count
:)

Thanksy for make better english write.
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 02:12:20 am
@Thausand

ICAO  34680             Unique  Only 1 in  apt.dat @ airport
City    14138
Lat     34457              Should be 34680  same as ICAO
Lon    34537              Should be 34680  same as ICAO
Regions 259              280

The number OF Lat's and Lon's have to be the same.
The Lat number and Lon number have to be different.
 

When program runs it loads multi Regions for some countries.


Most countries only have one region. However some Canada, China, Australia, US has more then one Region. Therefore, there are multi regions records in CountryRegions.Txt.

RegionCountry = "MU Cuba-I" or "K1 United States-O'

The 'I' in the record tell me I can do automatic processing:

if RegionCode = 'MU' and Country <> 'Cuba' then  begin

   // change Country here

or

 if CounryCode = 'Cuba' and RegionCode <> 'MU'
     //change Region code

1. The Longitude and Latitude of the airport is best. 

2. 14 tower viewpoint record: If the 1301 record Region_Code isn't set in the apt.dat then the next best Longitude and Latitude is the 14 record tower viewpoint. On really small airports they use that anyway.

3. I only use the  Longitude and Latitude of the runway if I have to: Many airports have multiple runways. One airport in Germany is five mile wide. KDEN in Denver is 7 miles wide. If you are trying find an airport 7 miles is a long ways.

I'm trying to work with your programs but I'm not very good with DOS programs and can't make it work. Built a GUI test program but can't get it to compile.

 
Title: Re: DataProblems Maybe
Post by: lucamar on May 05, 2019, 02:27:44 am
The number OF Lat's and Lon's have to be the same.
The Lat number and Lon number have to be different.

What he's doing is counting the number of longitudes after removing duplicates, and the same for latitudes, so if there are two airports on the same longitude or the same latitude, they are counted (for this) as one. That's why the totals are different.

That is, unless you know for sure that there no two airport with the same latitude or no two airports with the same longitude. If so, then there is an error somewhere.
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 02:49:00 am
Chances are good that more than one airport is on the same longitude or latitude. However, each of the records in my byAirports.txt have a latitude and longitude. So if you count the number of latitudes in ByAirp[orts.txt, the number of longitudes  and the number of airports they would each  equal 34,680. Otherwise the MSTRegions program would be throwing all kinds of errors.

Right now (Before the next test Cat Scan with definition) I think I'm going back and try to clean Apt.Dat. Generate a new ByAirports.txt.
Title: Re: DataProblems Maybe
Post by: Thausand on May 05, 2019, 02:52:02 am
The number OF Lat's and Lon's have to be the same.
The Lat number and Lon number have to be different.
I make count unique so i have no same count. i write be low also.

Thanksy for information write line 14. I then have change my program and make better (maybe take time)

i look what can do make program GUI. DOS simple: fpc flyby.pas for compile and then terminal open and make run "flyby name_of_byairport.txt" (not quote) then enter.

That is, unless you know for sure that there no two airport with the same latitude or no two airports with the same longitude. If so, then there is an error somewhere.
That correct.

I make quick add export ByAirport.dat my program aptidat and make check use my program flyby and i write result and explain:
Code: [Select]
$ ./aptidat apt.dat Airport.txt
Processing file "apt.dat"

    Line Airport    Land     Sea    Heli  Land_R Water_R Helipad    Time
 7970312   35242   28458     585    6199   34893     784    9719   95 sec (83210.44 lines/second)

$ ./flyby Airport.txt
Processing file "Airport.txt"

... show unique count
    Line   Error    ICAO    hash    City Country  Region     Lat     lon    Time
   35241       0   35241   35241   14272     259     248   35109   35135   35 sec (983.01 lines/second))
No errors encountered during processing data

Statistics dtICAO :
     Shortest = "00C"  (3 characters)
     Longest  = "XCYCDL8"  (7 characters)
 Lowest count = "00C"  (1 times)
Highest count = "00CA"  (1 times)

Statistics dtHash :
     Shortest = "4"  (1 characters)
     Longest  = "1022949"  (7 characters)
 Lowest count = "1021615"  (1 times)
Highest count = "1022949"  (1 times)

Statistics dtCity :
     Shortest = "Ie"  (2 characters)
     Longest  = "Charlotte Amalie - Harry S. Truman Airport"  (42 characters)
 Lowest count = "(Cagayan De Oro)"  (1 times)
Highest count = "Nil"  (9552 times)

Statistics dtCountry :
     Shortest = "CA"  (2 characters)
     Longest  = "United States Minor Outlying Islands"  (36 characters)
 Lowest count = "Anguilla"  (1 times)
Highest count = "United States"  (14971 times)

Statistics dtRegion :
     Shortest = "AG"  (2 characters)
     Longest  = "Nil"  (3 characters)
 Lowest count = "AN"  (1 times)
Highest count = "Nil"  (23023 times)

Statistics dtLat :
     Shortest = "0.00000000"  (10 characters)
     Longest  = "-10.00499742"  (12 characters)
 Lowest count = "-0.02069980"  (1 times)
Highest count = "-23.59055556"  (6 times)

Statistics dtLon :
     Shortest = "0.00000000"  (10 characters)
     Longest  = "-100.00039500"  (13 characters)
 Lowest count = "-0.00094400"  (1 times)
Highest count = "-46.65805556"  (4 times)

---------- List dtCountry ----------
   1 : (10) "Afghanistan"
   2 : (5) "Albania"
   3 : (47) "Algeria"
   4 : (3) "American Samoa"
   5 : (27) "Angola"
   6 : (1) "Anguilla"
   7 : (9) "Antarctica"
   8 : (3) "Antigua and Barbuda"
   9 : (164) "Argentina"
  10 : (1) "Arizona"
  11 : (5) "Armenia"
  12 : (1) "Aruba"
  13 : (1492) "Australia"
  14 : (54) "Austria"
  15 : (9) "Azerbaijan"
  16 : (38) "Bahamas"
  17 : (2) "Bahrain"
  18 : (13) "Bangladesh"
  19 : (1) "Barbados"
  20 : (9) "Belarus"
  21 : (43) "Belgium"
  22 : (1) "Belize"
  23 : (2) "Benin"
  24 : (1) "Bermuda"
  25 : (3) "Bhutan"
  26 : (38) "Bolivia"
  27 : (4) "Bosnia and Herzegovina"
  28 : (28) "Botswana"
  29 : (3758) "Brazil"
  30 : (1) "British Indian Ocean Territory"
  31 : (2) "British Virgin Islands"
  32 : (2) "Brunei"
  33 : (22) "Bulgaria"
  34 : (2) "Burkina Faso"
  35 : (1) "Burkina Faso7"
  36 : (36) "Burma"
  37 : (1) "Burundi"
  38 : (15) "CA"
  39 : (6) "Cambodia"
  40 : (10) "Cameroon"
  41 : (704) "Canada"
  42 : (9) "Cape Verde"
  43 : (3) "Caribbean Netherlands"
  44 : (1) "Carroll"
  45 : (3) "Cayman Islands"
  46 : (2) "Central African Republic"
  47 : (5) "Chad"
  48 : (176) "Chile"
  49 : (102) "China"
  50 : (1) "Christmas Island"
  51 : (1) "Cocos (Keeling) Islands"
  52 : (92) "Colombia"
  53 : (3) "Comoros"
  54 : (4) "Congo (Brazzaville)"
  55 : (27) "Congo (Kinshasa)"
  56 : (10) "Cook Islands"
  57 : (26) "Costa Rica"
  58 : (18) "Croatia"
  59 : (31) "Cuba"
  60 : (1) "Curaçao"
  61 : (3) "Cyprus"
  62 : (99) "Czechia"
  63 : (7) "Côte d'Ivoire"
  64 : (87) "Denmark"
  65 : (1) "Djibouti"
  66 : (2) "Dominica"
  67 : (15) "Dominican Republic"
  68 : (50) "Ecuador"
  69 : (56) "Egypt"
  70 : (4) "El Salvador"
  71 : (2) "Equatorial Guinea"
  72 : (3) "Eritrea"
  73 : (13) "Estonia"
  74 : (22) "Ethiopia"
  75 : (2) "Falkland Islands"
  76 : (8) "Faroe Islands"
  77 : (23) "Fiji"
  78 : (92) "Finland"
  79 : (3) "FR"
  80 : (441) "France"
  81 : (5) "French Guiana"
  82 : (48) "French Polynesia"
  83 : (11) "Gabon"
  84 : (1) "Gambia"
  85 : (9) "Georgia"
  86 : (505) "Germany"
  87 : (8) "Ghana"
  88 : (1) "Gibraltar"
  89 : (56) "Greece"
  90 : (17) "Greenland"
  91 : (1) "Greenville"
  92 : (2) "Grenada"
  93 : (5) "Guadeloupe"
  94 : (2) "Guam"
  95 : (10) "Guatemala"
  96 : (2) "Guernsey"
  97 : (4) "Guinea"
  98 : (3) "Guinea-Bissau"
  99 : (16) "Guyana"
 100 : (6) "Haiti"
 101 : (10) "Honduras"
 102 : (1) "Hong Kong"
 103 : (76) "Hungary"
 104 : (81) "Iceland"
 105 : (195) "India"
 106 : (181) "Indonesia"
 107 : (88) "Iran"
 108 : (16) "Iraq"
 109 : (36) "Ireland"
 110 : (1) "Isle of Man"
 111 : (21) "Israel"
 112 : (120) "Italy"
 113 : (6) "Jamaica"
 114 : (130) "Japan"
 115 : (1) "Jersey"
 116 : (8) "Jordan"
 117 : (1) "K1"
 118 : (21) "Kazakhstan"
 119 : (14) "Kenya"
 120 : (20) "Kiribati"
 121 : (1) "Kosovo"
 122 : (3) "Kuwait"
 123 : (4) "Kyrgyzstan"
 124 : (8) "Laos"
 125 : (7) "Latvia"
 126 : (3) "Lebanon"
 127 : (5) "Lesotho"
 128 : (2) "Liberia"
 129 : (41) "Libya"
 130 : (1) "Liechtenstein"
 131 : (7) "Lithuania"
 132 : (3) "Luxembourg"
 133 : (1) "Macau"
 134 : (12) "Macedonia"
 135 : (28) "Madagascar"
 136 : (7) "Malawi"
 137 : (43) "Malaysia"
 138 : (6) "Maldives"
 139 : (8) "Mali"
 140 : (2) "Malta"
 141 : (7) "Marshall Islands"
 142 : (1) "Martinique"
 143 : (10) "Mauritania"
 144 : (2) "Mauritius"
 145 : (1) "Mayotte"
 146 : (132) "Mexico"
 147 : (4) "Micronesia"
 148 : (3) "Moldova"
 149 : (1) "Monaco"
 150 : (22) "Mongolia"
 151 : (3) "Montenegro"
 152 : (1) "Montserrat"
 153 : (26) "Morocco"
 154 : (18) "Mozambique"
 155 : (1) "Myanmar"
 156 : (55) "Namibia"
 157 : (1) "Namibia."
 158 : (1) "Nauru"
 159 : (41) "Nepal"
 160 : (34) "Netherlands"
 161 : (17) "New Caledonia"
 162 : (109) "New Zealand"
 163 : (10) "Nicaragua"
 164 : (7) "Niger"
 165 : (27) "Nigeria"
 166 : (7294) "Nil"
 167 : (1) "Niue"
 168 : (1) "Norfolk Island"
 169 : (1) "North Korea"
 170 : (9) "Northern Mariana Islands"
 171 : (142) "Norway"
 172 : (6) "Oman"
 173 : (68) "Pakistan"
 174 : (3) "Palau"
 175 : (2) "Palestinian Territory"
 176 : (22) "Panama"
 177 : (75) "Papua New Guinea"
 178 : (11) "Paraguay"
 179 : (1) "Peoples Republic of China"
 180 : (1) "Peru"
 181 : (79) "Perú"
 182 : (73) "Philippines"
 183 : (92) "Poland"
 184 : (52) "Portugal"
 185 : (32) "Puerto Rico"
 186 : (3) "Qatar"
 187 : (1) "Republic of Ireland (Eire)"
 188 : (39) "Romania"
 189 : (167) "Russia"
 190 : (7) "Russian Federation"
 191 : (1) "Russin Federation"
 192 : (5) "Rwanda"
 193 : (2) "Réunion"
 194 : (1) "Saint Barthélemy"
 195 : (2) "Saint Helena"
 196 : (2) "Saint Kitts and Nevis"
 197 : (2) "Saint Lucia"
 198 : (1) "Saint Martin"
 199 : (2) "Saint Pierre and Miquelon"
 200 : (5) "Saint Vincent and the Grenadines"
 201 : (4) "Samoa"
 202 : (2) "Sao Tome and Principe"
 203 : (62) "Saudi Arabia"
 204 : (10) "Senegal"
 205 : (7) "Serbia"
 206 : (8) "Seychelles"
 207 : (1) "Sierra Leone"
 208 : (6) "Singapore"
 209 : (30) "Slovakia"
 210 : (16) "Slovenia"
 211 : (23) "Solomon Islands"
 212 : (6) "Somalia"
 213 : (109) "South Africa"
 214 : (77) "South Korea"
 215 : (4) "South Sudan"
 216 : (107) "Spain"
 217 : (10) "Sri Lanka"
 218 : (11) "Sudan"
 219 : (8) "Suriname"
 220 : (1) "Swaziland"
 221 : (164) "Sweden"
 222 : (1) "Swiss"
 223 : (70) "Switzerland"
 224 : (23) "Syria"
 225 : (21) "Taiwan"
 226 : (6) "Tajikistan"
 227 : (14) "Tanzania"
 228 : (61) "Thailand"
 229 : (6) "Timor-Leste"
 230 : (2) "Togo"
 231 : (6) "Tonga"
 232 : (2) "Trinidad and Tobago"
 233 : (13) "Tunisia"
 234 : (74) "Turkey"
 235 : (10) "Turkmenistan"
 236 : (7) "Turks and Caicos Islands"
 237 : (1) "Tuvalu"
 238 : (6) "U.S. Virgin Islands"
 239 : (1) "U.S.A."
 240 : (18) "Uganda"
 241 : (26) "Ukraine"
 242 : (19) "United Arab Emirates"
 243 : (251) "United Kingdom"
 244 : (1) "United Sates"
 245 : (14971) "United States"
 246 : (4) "United States Minor Outlying Islands"
 247 : (2) "United States of America"
 248 : (11) "Uruguay"
 249 : (25) "US"
 250 : (26) "USA"
 251 : (15) "Uzbekistan"
 252 : (23) "Vanuatu"
 253 : (100) "Venezuela"
 254 : (24) "Vietnam"
 255 : (2) "Wallis and Futuna"
 256 : (3) "Western Sahara"
 257 : (13) "Yemen"
 258 : (8) "Zambia"
 259 : (17) "Zimbabwe"
First i see i have wrong count airport (i not know why. I make quick export so think have small error. I look later).

then have more better look:
Code: [Select]
Statistics dtLat :
     Shortest = "0.00000000"  (10 characters)
     Longest  = "-10.00499742"  (12 characters)
 Lowest count = "-0.02069980"  (1 times)
Highest count = "-23.59055556"  (6 times)
Lowest count = 1 times, Highest count is 6 times.

That mean Airport latitude "-23.59055556" is count 6 times. Then 6 airport in apt.dat have coordinate latitude -23.59055556

then:
Code: [Select]
Statistics dtLon :
     Shortest = "0.00000000"  (10 characters)
     Longest  = "-100.00039500"  (13 characters)
 Lowest count = "-0.00094400"  (1 times)
Highest count = "-46.65805556"  (4 times)
Lowest count = 1 times, Highest count is 4 times.

That mean Airport longitude "-46.65805556" is count 4 times. Then 4 airport in apt.dat have coordinate longitude -46.65805556.

Is many coincidence have 6 airport (from 35241)  same latitude 23.59055556 and have 4 airport (from 35241) same longitutude -46.65805556 but i think that possible when have big planet earth  :)

add:
is same for country and then no problem understand  :)
Code: [Select]
Country
259 
Code: [Select]
245 : (14971) "United States"
Country name ""United States" is count 14971 and Country count is 259. My program no write Country count = 35242 because have ICAO 35242. Have 259 unique name country  :)
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 02:59:08 am
@Thausand

I guess it possibly or my data extraction program is wrong.

The difference in the number of airports is due to the fact that I don't build a record for Heliport but I do Sea Plane bases.

 
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 03:28:48 am
@lucmar

I don't know. Not getting a apt.bak file

Volume in drive C has no label.
 Volume Serial Number is 846B-146A

 Directory of C:\UserFiles

05/04/2019  06:19 PM    <DIR>          .
05/04/2019  06:19 PM    <DIR>          ..
04/27/2019  10:08 PM       292,818,538 apt.dat
04/27/2019  10:18 PM       292,818,538 apt.txt
05/01/2019  10:17 PM        61,635,578 apt.zip
05/04/2019  06:07 PM           288,787 cleanbom.exe
04/27/2019  09:06 PM        16,164,022 UTF8Clean.exe
05/04/2019  06:19 PM                 0 x
               6 File(s)    663,725,463 bytes
               2 Dir(s)  314,017,177,600 bytes free



C:\UserFiles>cleanbom apt.txt
or
C:\UserFiles>cleanbom apt.dat



 







Title: Re: DataProblems Maybe
Post by: Thausand on May 05, 2019, 03:51:49 am
I guess it possibly or my data extraction program is wrong.
Is can my program extract wrong. So maybe not you program error. That why i ask question  :)

Quote
The difference in the number of airports is due to the fact that I don't build a record for Heliport but I do Sea Plane bases.
For basic is simple. My program have count (unique) airport  35242 and (unique) count Heli 6199. If apt.dat have unique ICAO airport (and any Airport apt.dat have (unique) ICAO) then count = 35242 - 6199 = 29043 unique airport (and ICAO).
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 04:02:50 am
"The difference in the number of airports is due to the fact that I don't build a record for Heliport but I do Sea Plane bases."

if the Heliport is on an airport I don't build a record but if it's like at a hospital I do build a record I think, I'll have to look.

if  your program outputs a file, e-mail me the file.

Goto go somewhere.
Title: Re: DataProblems Maybe
Post by: Thausand on May 05, 2019, 04:06:05 am
Hi hello JLWest,

I sorry i no lucamar

I don't know. Not getting a apt.bak file
04/27/2019  10:08 PM       292,818,538 apt.dat

i make linux:
Code: [Select]
$ ls -l apt.dat
-rw-r--r-- 1 pi pi 292818538 Apr 27 22:08 apt.dat
If read good then see is same size.

I know apt.dat my no have BOM.

I not know program lucamar cleanbom but when have not BOM then why write file ? Then maybe program cleanbom read no BOM and then no write file new and no need file "apt.bak" ? (maybe i can look cleanbom and check and write later).
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 04:29:37 am
Same time stamp and size
Title: Re: DataProblems Maybe
Post by: lucamar on May 05, 2019, 04:49:06 am
@lucmar

I don't know. Not getting a apt.bak file


That should mean your apt.dat and apt.txt don't have the UTF8 BOM. My program is as lazy as myself: if there's nothing to do it does nothing :D

It only does something (and so creates the .bak) if the original file had a UTF8 BOM. But  if not it should tell you so: the message, IIRC, is something like: "That file doesn't need me: doesn't have a BOM."

I downloaded the apt.dat, will test it when (if!) I have a little time.

ETA: Oh, Thausand tested already. OK, then.

EATA
I not know program lucamar cleanbom but when have not BOM then why write file ? Then maybe program cleanbom read no BOM and then no write file new and no need file "apt.bak" ?

Yes, that's exactly how it works :)
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 06:56:16 am
Here is where I'm at:

Went back and copied from:
C:\X-Plane 11\Resources\default scenery\default apt dat\Earth nav data\apt.dat
TO:
 c:\userfiles\apt.dat

Ran CleanBom

Ran MstAirports and generated a new byAirports.txt 34,680 records generated.

Ran Extractwordtext (Nice test) and tested the first record.

7 fields -everything OK

ran MstRegions and clicked on the first record which is always
[00C][28944][Nil][Nil][K2][37.203177778][-107.869194444]

Everything is Ok so I don't have a UTF8 BOM problem

Now to run down Thausand findings

Title: Re: DataProblems Maybe
Post by: Thausand on May 05, 2019, 09:23:52 am
Hi hello JLWest,

I make new test (have fix 1 line miss error, was wrong count  :-[ ) and not can make same count like you program.

Alone when count airport ID 1, 16 and 17 then my program have almost same count you "ByAirport.txt". I have you "ByAirport.txt" size 2383401 and have ICAO 34680.

Then my program have many more ICAO  (have 35242) :o

I write list ICAO have my program and no have you "ByAirport.txt".

Then can see is no there normal ID "1" also in "ByAirport.txt" (have example EDUY) and also miss other ID airport "16" and "17". Then i think maybe error is in program you ?

Is 562 count list ICAO:
Code: [Select]
01MN
03M
05B
08MN
09Y
0FD6
0G5
0MI5
0MN0
0MN2
0MN4
0O0
0TN1
0W0
0W7
0Z3
10MA
11MN
12MN
13S
13Z
14MN
16MN
16WI
16Z
17MN
19Y
1AL9
1FD1
1IS9
1MI0
1MN4
1MN9
1MU1
1O0
1WI5
1Z9
21H
21MN
22MN
28MN
2MA6
2MA7
2MN5
2N7
2OG3
2Q4
2R3
2Y3
2Z1
2Z6
30W
34MN
3FD9
3LA2
3MD9
3MI9
3TN2
3Z8
3Z9
42FL
43FD
44NJ
45D
48B
4FD4
4FD5
4LA9
4MN1
4NC9
4NJ2
4NY2
4Z7
50MI
51MI
51Y
52B
52Z
53LA
54NJ
58NJ
5BL
5CA9
5J2
5KE
5L6
5LA6
5MA8
5N6
5N9
5X3
5Z1
5Z9
60B
61FD
62FD
63A
68A
6FL5
6MA0
6MA8
6MA9
6MD6
6MN4
6N6
6N7
6NJ7
6NY6
70B
70MY
71C
75B
75PN
78B
78U
78Z
7K2
7N3
7VA7
7W8
7WI1
7Y6
80B
81W
83B
83FD
83Q
85B
85VA
87B
89FD
8AK4
8IL7
8IL9
8K9
8MN0
8NA1
8X0
91FL
94D
96WI
96Z
98FD
98FL
99Z
9C0
9FD0
9M0
9MN3
9MN8
9N2
9X6
9Y5
9Y6
9Z3
9Z4
AK16
AK19
AK27
AK29
AK34
AK36
AK43
AK56
AK57
AK58
AK62
AK81
AK84
AL46
ALZ
AQS
B18
BNF
BQV
BXL
C39
C54
CA2E
CA2S
CAA9
CAML
CAN6
CAN8
CAU6
CGA
CJB7
CJD9
CJX8
CJY9
CJZ6
CJZ9
CKB4
CKB5
CKC2
CKE4
CKG6
CKP5
CKU3
CKW5
CML1
CN20
CNC8
CND9
CNH7
CNQ7
CNS2
CPB6
CPT7
CSKC
CT16
CT30
CT78
CT82
CT87
CTM2
CWB1
CXBB
CYBH
CYDX
CYHC
CYIG
CZSW
DE13
E20
EDUY
ES05
ES06
ES07
EXI
F57
FA05
FA17
FA23
FD12
FD79
FD80
FD98
FL04
FL26
FL42
H11
H63
HWI
HYL
I04
ID28
IN59
IS79
K40
K45
KAE
KBE
KCC
KCDD
KCN
KIB
KKB
KKI
KKL
KKPB
KMY
KOY
KPH
KPR
KPY
KQA
KTB
KVU
KWF
KWK
KWP
KXA
L85
LA16
LA64
LEGO
M00
M14
M49
M57
M69
MA01
MA06
MA12
MA25
MA36
MA65
MA74
MA78
MA82
MA87
MD80
ME01
ME04
ME09
ME11
ME12
ME27
ME30
ME43
ME50
ME51
ME63
ME67
ME80
ME90
MI03
MN03
MN05
MN09
MN21
MN29
MN30
MN35
MN72
MN73
MN78
MN80
MO07
MY21
MY22
MY23
MY31
MY32
MY33
MY34
MY36
MY38
MY48
MY73
MY74
MY83
MY96
NFCS
NH36
NH37
NH64
NH71
NJ07
NK13
NK30
NK59
NK60
NY47
NY61
O06
O12
OOH
ORV
PA90
PAAP
PAEL
PAFK
PAHY
PAMM
PANR
PAQC
PEC
PN19
PN24
PN65
PR03
PS88
PWR
PYL
Q83
Q86
Q87
QAHO
QEVG
QFFS
QHNA
QILO
QJON
QKKI
QKRE
QLAN
QLEH
QLHA
QLTO
QMDY
QOHE
QONA
QPLM
QPUU
QPYM
QPYN
QULA
QWLK
QXMS
QZFJ
RI06
S60
SCGR
SP00
SP01
SP02
SP03
SP04
SP05
SP06
SP07
SP08
SP09
SP10
SP11
SP12
SP13
SP14
SP15
SP16
SP17
SP18
SP19
SP20
SP21
SP22
SP23
SP24
SP25
SP26
SP27
SPRH
SYBD
SYPW
SYRI
T38
TKL
TLG1
TTL
VA48
VT57
VT58
W26
W36
W37
W49
W55
WA39
WA83
WI27
WI35
WN19
WN66
WS01
WS24
WSB
WSJ
WV35
WV37
WV38
WV39
WV40
WV41
WV42
WV43
WV44
WV46
WWT
X66
X67
X96
XCKC5
XSA
XXEI
YAIN
YAIS
YARF
YBAT
YBDA
YBEX
YBVR
YBWK
YBWS
YCGP
YCON
YCRE
YDDI
YDHM
YDLR
YDOL
YDRY
YEST
YFLF
YFLY
YFRA
YGN
YGNI
YGUU
YHC
YHED
YHOW
YHPB
YHRN
YHSR
YHYN
YJEB
YKRI
YKT
YLMI
YMBS
YMCY
YMID
YMIE
YMKH
YMRR
YMSD
YNBY
YNOI
YNOM
YOPH
YOPY
YOSI
YOYS
YPAU
YPAW
YPBM
YPBY
YPLB
YPSB
YPTD
YRAP
YRAY
YRMA
YROD
YRPT
YSAY
YSEI
YSEZ
YSHS
YSML
YSMP
YSND
YSTI
YSUD
YSYY
YTB
YTBY
YTFR
YTG
YTHO
YTND
YTRI
YUPC
YVAN
YVND
YWHN
YWLD
YWTE
YWTI
YYDY
Z20
Z33
Z43
Z58
Z59
Z71
Z78
Z87
ZNU
ZOF

Maybe list have help find what wrong my or you program ?
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 06:14:53 pm
@Thausand

I'm working on it.

I think my number of 34,680 is pretty close.

X-Plane claims they have over 35,000 Airports.

I need the following info in ByAirports.txt:

All Land base airports.
All Sea plane bases.
Some Heliports (Those that are not located at an airport but like at a Hospital or on top of a building)

The ByAirports.txt file has 34,680 records. It is suppose to have one record for each airport in the apt.dat file with the following information:

[ICAO]    The ICAO airport code or location indicator is a four-letter code designating
              aerodromes around the world.

[Hash]   Hash number of the record location of the ICAO in Apt.dat.

[City]   City where the airport is located.  Might not have a city. Some airports are in the
           brush. Above the arctic circle, in the middle of no where, no city assigned.

[Country]   All airports as far as I can determine have a country.
[Region]    Should have a Region, However those without a Region are grandfathered in as
                valid data by X-Plane. Some are without Regions as far as X-Plane. The
                International Civil Aviation Organization has the rights to assigned all airports a
                region, but some don't have a region.

[Latitude]     All are suppose to have a Latitude of the airport. Some don't and you have to
                   use  either the Tower or runway center-line.

[Longitude] All are suppose to have a Longitude of the airport. Some don't and you have to
                 use  either the Tower or runway center-line.

Currently I'm showing in the ByAirports.txt  the following:

34,680 Airports
8,194 Cities,    I  think this is about 4 or 5 cities to many.
9,721 City,Region,ICAO  AbileneK4Kabi.


Unique record of City Region is not possible  (AbileneK4) as there is a Abilene K4 8T2S an Abilene K4 KABI, and a Abiline K4 KDYS. Many cities have more than 1 airport and they would all be in the same Region. 

I copied your list of 562.

You are missing ZYYY,ZYYJ 
[ZYYY][7105929][Shenyang][China][Nil][41.77619293][123.48947822] and quite a few more.
Title: Re: DataProblems Maybe
Post by: Thausand on May 05, 2019, 07:19:30 pm
Thanksy for more information JLWest.

X-Plane claims they have over 35,000 Airports.
Yes. have 35242 airport  :)

Quote
I need the following info in ByAirports.txt:
All Land base airports.
All Sea plane bases.
My program have make read from apt.dat and have any ID "1" and have any ID "16"

Quote
Some Heliports (Those that are not located at an airport but like at a Hospital or on top of a building)
My Airport.dat have count any heliport apt.dat. Have program check ID "17". My program not have 'remove'. I not know how decide remove. Specificate write Heliport unique and not combine land/sea port. Make sure i write my program not count helipads.

Quote
[City]   City where the airport is located.  Might not have a city. Some airports are in the brush. Above the arctic circle, in the middle of no where, no city assigned.
Yes, my program have same. Many no city name.

Quote
[Country]   All airports as far as I can determine have a country.
My program airport.dat write there have airport no ID 1302 or have ID 1302 and then no have item "country" then no have name country (=nil)

Have example ZMGT and have example ZMHG.

Quote
[Region]    Should have a Region, However those without a Region are grandfathered in as valid data by X-Plane. Some are without Regions as far as X-Plane. The International Civil Aviation Organization has the rights to assigned all airports a                 region, but some don't have a region.
Yes.  I can see my program to.

Quote
[Latitude]     All are suppose to have a Latitude of the airport. Some don't and you have to  use  either the Tower or runway center-line.

[Longitude] All are suppose to have a Longitude of the airport. Some don't and you have to  use  either the Tower or runway center-line.
This my program big error. I have many no lat/lon and if make then many different you ByAirport.txt. I try work my program make better. I have make read how read "runway center-line" because is have Airport many runway.

Quote
Unique record of City Region is not possible  (AbileneK4) as there is a Abilene K4 8T2S an Abilene K4 KABI, and a Abiline K4 KDYS. Many cities have more than 1 airport and they would all be in the same Region. 
Yes i have read. That is why my program flyby have error. If have London city England and have London city America. Now my program write is same "London" (i know is error my) and no unique city. I not have make hurry and make fix. I think is no many important now (maybe later is more important).

Quote
I copied your list of 562.
I make sure and write: List 562 is list ICAO that my program make and you ByAirports.txt file no have find.

Quote
You are missing ZYYY,ZYYJ 
[ZYYY][7105929][Shenyang][China][Nil][41.77619293][123.48947822] and quite a few more.
No, my program have  :D

Code: [Select]
...
[ZYHB][6945797][Harbin][China][ZY][45.625][126.251666667]
[ZYJM][6678872][Jiamusi][China][ZY][46.841666667][130.465]
[ZYMD][7599597][Mudanjiang][China][ZY][44.523333333][129.57]
[ZYQQ][6750484][Qiqihar][China][ZY][47.238333333][123.916666667]
[ZYTL][2667556][Dalian][China][ZY][38.966666667][121.54]
[ZYTX][7274750][Shenyang][China][ZY][41.641666667][123.485]
[ZYYJ][7134994][Yanji][China][ZY][42.881666667][129.45]
[ZYYY][7105929][Shenyang][China][Nil][Nil][Nil]
ZYYY ICAO is most below line my program have write  :)
Title: Re: DataProblems Maybe
Post by: JLWest on May 05, 2019, 08:12:35 pm
Z20
Z33
Z43
Z58
Z59
Z71
Z78
Z87
ZNU
ZOF

I don't see these in the posting above.

Title: Re: DataProblems Maybe
Post by: Thausand on May 06, 2019, 02:35:08 am
I'm trying to work with your programs but I'm not very good with DOS programs and can't make it work. Built a GUI test program but can't get it to compile.
Then have two option
1) make learn use dos  ;D
2) make learn how write freepascal command program then write for Lazarus

Have 2 and can maybe me help you show.

I attach zip two. 1 is program flyby fpc command 2 is program flybylaz and make small terminal emulate. program 2 have many more lazarus part but no need for flyby program and is many same program (i have document and maybe can help you better understand how can make. Is not difficult copy-paste).

And i make attach picture for see.

If start program Lazarus then have make type "flyby name_of_airport.txt_file" (no quote) for edit then press button run for execute and if ok then show flyby write memo. If no ok then i hope print error for me read  :)

I hope maybe can help you little. If no then i sorry.
Title: Re: DataProblems Maybe
Post by: Thausand on May 06, 2019, 02:54:23 am
Z20 Z33 Z43 Z58 Z59 Z71 Z78 Z87 ZNU ZOF

I don't see these in the posting above.
I have 35242 more for look then take time :)

Code: [Select]
...
[Z01][7723708][Nil][Nil][Nil][Nil][Nil]
[Z04][7829328][Nil][Nil][Nil][Nil][Nil]
[Z10][7829293][Nil][Nil][Nil][Nil][Nil]
[Z13][7937010][Akiachak][United States][Nil][Nil][Nil]
[Z14][7700680][Tazlina][United States][Nil][Nil][Nil]
[Z17][7626587][Ophir][United States][Nil][Nil][Nil]
[Z20][6737744][Tuntutuliak][United States][Nil][Nil][Nil]
[Z25][6831289][Aleknagik][United States][Nil][Nil][Nil]
[Z33][6891092][Aleknagik][United States][Nil][Nil][Nil]
[Z35][6702164][Nil][Nil][Nil][Nil][Nil]
[Z38][7830993][Nil][Nil][Nil][Nil][Nil]
[Z40][1808506][Goose Bay][United States][Nil][Nil][Nil]
[Z43][7091267][Annette][United States][Nil][Nil][Nil]
[Z47][7150686][Basin Creek][United States][Nil][Nil][Nil]
[Z48][7317757][Bear Creek][United States][Nil][Nil][Nil]
[Z52][7172078][Bear Lake][United States][Nil][Nil][Nil]
[Z53][7150799][Nil][Nil][Nil][Nil][Nil]
[Z55][6977748][Lake Louise][United States][Nil][Nil][Nil]
[Z58][7577491][Bethel][United States][Nil][Nil][Nil]
[Z59][7559661][Bethel][United States][Nil][Nil][Nil]
[Z61][7343303][Nil][Nil][Nil][Nil][Nil]
[Z70][7576598][Nil][Nil][Nil][Nil][Nil]
[Z71][7560738][Cape Pole][United States][Nil][Nil][Nil]
[Z78][7231204][Chignik][United States][Nil][Nil][Nil]
[Z81][7057352][Salmon Lake][United States][Nil][Nil][Nil]
[Z86][7261409][Clearwater][United States][Nil][Nil][Nil]
[Z87][7228969][Cold Bay][United States][Nil][Nil][Nil]
[Z90][7146505][Kantishna][United States][Nil][Nil][Nil]
[Z91][7174526][Birch Creek][United States][Nil][Nil][Nil]
[Z92][7262712][Harsens Island][United States][Nil][Nil][Nil]
[Z93][7228512][Copper Center][United States][Nil][Nil][Nil]
[Z98][4191570][Zeeland][United States][K5][42.817247222][-85.928091667]
...

then

Code: [Select]
...
[ZMUL][5990567][Nil][Mongolia][ZM][48.994166667][89.923333333]
[ZNC][6975729][Nyac][United States][Nil][Nil][Nil]
[ZNU][7007874][Nil][Nil][Nil][51.84477329][-127.87740632]
[ZOF][7010682][Nil][Nil][Nil][52.34806244][-127.69314139]
[ZPDL][1971430][Xiaguan][China][Nil][Nil][Nil]
..
I have try, i no can make file attach to big. And then file me have many error (see paste this post many no have city, country and lat/lon). If want i can make ICAO list complete but is no magica. They all ICAO that have apt.dat. That why i write that you no have 562 ICAO and can look in program you why no find ?. I think that is logica. Is maybe possible i have make very wrong error. (i no admit and deny  :-X )  :D

If i have fix my program error lat/lon and city/country name then if good fix i post (maybe can split attach). If i diff you ByAirports.txt and me Airports.txt then diff too many and no usage compare  :-\
Title: Re: DataProblems Maybe
Post by: JLWest on May 06, 2019, 07:14:53 am
Copied both zip to subdirectory.
Unziped them both.
Have 6 files.

Flyby_main.lfm    Form?
Flyby_Main.pas   pascal code
Flybylaz.ico        program ICON
flyblaz.lpi
flybylas.lpr
flyblaz.res

When I click on flublaz.lpr I get the following message:

"flybylaz.lpr(10,10) Fatal: Cannot find flyby_main used by flybylaz of the Project Inspector."

It's there but the Project Inspector can't see it and I'm unable to add it.

got something loaded and compiler gives me this:


Compile Project, Target: flybylaz.exe: Exit code 1, Errors: 1, Hints: 5
Hint: Start of reading config file C:\FPC\3.0.4\bin\i386-Win32\fpc.cfg
Hint: End of reading config file C:\FPC\3.0.4\bin\i386-Win32\fpc.cfg
flyby_main.pas(139,3) Note: Local variable "s" not used
flyby_main.pas(288,4) Hint: "array of const" not yet supported inside inline procedure/function
flyby_main.pas(288,4) Hint: Inlining disabled
Error: Can't open resource file "C:\Users\Jerry\AppData\Local\Temp\Temp1_flybylaz_publish.zip\flybylaz_publish\flyby_main.lfm"

I have it running but I don't know how or why.

I get  "E:\Files\Airports\ByAirport.txt : command not found"

Do I need to do this in a DOS Box you know CMD Line
 
Title: Re: DataProblems Maybe
Post by: Thausand on May 06, 2019, 02:53:30 pm
Copied both zip to subdirectory.
Unziped them both.
Have 6 files.
Lazarus project directory have 6 file:
Code: [Select]
ls -l
total 300
-rw-r--r-- 1 pi pi 137040 May  5 19:59 flybylaz.ico
-rw-r--r-- 1 pi pi   2948 May  6 01:38 flybylaz.lpi
-rw-r--r-- 1 pi pi    400 May  5 20:00 flybylaz.lpr
-rw-r--r-- 1 pi pi 139052 May  6 01:38 flybylaz.res
-rw-r--r-- 1 pi pi   1123 May  6 01:37 flyby_main.lfm
-rw-r--r-- 1 pi pi  15260 May  6 01:37 flyby_main.pas

Free pascal 'project' (is alone one file) directory have:
Code: [Select]
ls -l
total 12
-rw-r--r-- 1 pi pi 11528 May  6 01:53 flyby.pas

They separate project (one lazarus one fpc). Only have common that have source-lines that read exact same. That maybe can make show how can make project freepascal then make to project lazarus. Simple copy-paste  :)

Lazarus project make with Lazarus 2.0.0 release (use fpc 3.0.4) official and use publish project lazarus.

If some thing not work if load project lazarus then i not know why no work  :-[.

Quote
When I click on flublaz.lpr I get the following message:
Is small suggest: maybe start lazarus and load project "flybylaz.lpi" ? (that is how i do)

Quote
"flybylaz.lpr(10,10) Fatal: Cannot find flyby_main used by flybylaz of the Project Inspector."
That is problem and big error. flyby_main is project main form (have form, panel, memo, button and edit).

Quote
Compile Project, Target: flybylaz.exe: Exit code 1, Errors: 1, Hints: 5
Hint: Start of reading config file C:\FPC\3.0.4\bin\i386-Win32\fpc.cfg
Hint: End of reading config file C:\FPC\3.0.4\bin\i386-Win32\fpc.cfg
flyby_main.pas(139,3) Note: Local variable "s" not used
flyby_main.pas(288,4) Hint: "array of const" not yet supported inside inline procedure/function
flyby_main.pas(288,4) Hint: Inlining disabled
Error: Can't open resource file "C:\Users\Jerry\AppData\Local\Temp\Temp1_flybylaz_publish.zip\flybylaz_publish\flyby_main.lfm"
As write more before. Not load project main form is error and problem. I not know why. What is version/platform  you Lazarus ? (i know you have tag-line but is not always use same lazarus and tag some time expire)

Quote
I have it running but I don't know how or why.
I not know to. If have error write above then in theory impossible ... maybe you wizard ?  :)

Quote
I get  "E:\Files\Airports\ByAirport.txt : command not found"
If run project then have main form open then make edit in editbox "flyby E:\Files\Airports\ByAirport.txt" (no quote) then push button that next editbox (is to right).

"command not found" mean you not have type flyby before  :)

Quote
Do I need to do this in a DOS Box you know CMD Line
lazarus project is run windows/desktop. That why i write because you write not know terminal. Now can have windows run small simulate terminal  ;)
Title: Re: DataProblems Maybe
Post by: JLWest on May 06, 2019, 06:59:08 pm
If run project then have main form open then make edit in editbox "flyby E:\Files\Airports\ByAirport.txt" (no quote) then push button that next editbox (is to right).

"command not found" mean you not have type flyby before

Yes: I typed : E:\Files\Airports\ByAirport.txt and pushed the execute button.

I copied the ByAirports.txt file to the directory where the flyfy is and get the same results.

Looking at the code your suggesting that I copy part of the code out to a GUI New windows program and it.

That will take me a bit of time.

Title: Re: DataProblems Maybe
Post by: Thausand on May 06, 2019, 07:54:27 pm
Yes: I typed : E:\Files\Airports\ByAirport.txt and pushed the execute button.
And is wrong  ;D

Look picture: i type many wrong purpose for see. In picture final is good
Code: [Select]
flyby E:\Files\Airports\ByAirport.txt
And press button.

i write important: see "ByAirport.txt" is not have "s" letter and you copy have name "s" letter "ByAirports.txt" ?

I write many wrong english and not write all word and can maybe understand my write. Computer not so very smart and no understand if name no have match. If no have letter then not can find.

Quote
I copied the ByAirports.txt file to the directory where the flyfy is and get the same results.
if copy same directory "ByAirports.txt" then can type editbox:

Code: [Select]
flyby ByAirports.txt

Quote
Looking at the code your suggesting that I copy part of the code out to a GUI New windows program and it.
Yes. My program show and comment how can do.

Quote
That will take me a bit of time.
For now not you have do (i have make for you in zip)  :).

Is alone show you how can do.

You no like terminal (is ok) and i no like Lazarus (for me is slow, have many bug and i many fight code-tool). I like more work terminal have vim, can make remote many and fast. Have example browse if use desktop browse then many MB/GB take website memory and in terminal have many website and more small 50-100 MB take memory for any.

Have other example "apt.dat" have 250 MB then doublecommand (or any) open and take more 5 minute for edit and then can look. Use terminal can make fast look (is make < 5 second):
Code: [Select]
sed -n '4797454,+20p' apt.dat
1   5511 1 0 ZMHG Khatgal
1302 city
1302 country
1302 datum_lat 50.441388889
1302 datum_lon 100.134166667
1302 faa_code
1302 iata_code
1302 icao_code ZMHG
1302 region_code ZM
1302 state
1302 transition_alt 10499
1302 transition_level
100 35.00 2 0 0.50 0 1 1 15  50.44968173  100.12729071    0  110 2 0 0 0 33  50.43036657  100.14203570    0   38 2 0 0 0
100 25.00 5 0 0.75 0 0 1 16  50.44324210  100.13571124    0    0 2 0 0 0 34  50.43055660  100.14265402    0    0 2 0 0 0
110 2 0.50 302.8900 New Taxiway 1
111  50.44418478  100.13549750
111  50.44662872  100.12959735
111  50.44615845  100.13004682
111  50.44399104  100.13524242
111  50.44335146  100.13637160
111  50.44396626  100.13726546

For me desktop no efficient  :)
Title: Re: DataProblems Maybe
Post by: JLWest on May 07, 2019, 09:43:44 am
Yea I think there is something wrong with the data.

Not sure how to fix it.
Title: Re: DataProblems Maybe
Post by: JLWest on May 07, 2019, 05:16:27 pm
@Thausand

Attached 3 Screen Shots.

1. Listing of dir where your program along with byAirports.txt
2. What I entered.
3. Error Message.
In procedure RunCommandLine it goes from Line 13 to 23 no matter what I enter.

I can't figure out what the line is doing.

Code: Pascal  [Select]
  1. procedure TTermiForm.RunCommandLine(CommandLine: string);
  2. var
  3.   index : SizeInt;
  4.   s : string;
  5. begin
  6.   // have command print
  7.   PrintLine(' $ ' + CommandLine);
  8.   ParameterList := nil;
  9.   ParameterList := CommandLine.Split([' ',#8,#9,#10,#13], '"');
  10.   if Length(ParameterList) > 0 then
  11.   begin
  12.     // have command find in list
  13.     index := CommandList.IndexOf(ParameterList[0]);
  14.     // is find ?
  15.     if (index >= 0) then
  16.     begin
  17.       // have execute command
  18.       TProcedure(CommandList.Objects[index])();
  19.     end
  20.     else PrintLine(ParameterList[0] + ' : command not found');
  21.   end
  22.   else PrintLine('Empty command');
  23. end;      
Title: Re: DataProblems Maybe
Post by: lucamar on May 07, 2019, 06:22:26 pm
I can't figure out what the line is doing.

This line?
Code: Pascal  [Select]
  1. index := CommandList.IndexOf(ParameterList[0]);

It tries to find in CommandList (some kind of TStrings?) the string in ParameterList[0]. The if following it checks if it has found it and, if so, apparently executes a procedure whose pointer was saved in the "object" part of the string in CommandList.

I would have to see more code to explain better.
Title: Re: DataProblems Maybe
Post by: Thausand on May 07, 2019, 07:39:12 pm
Attached 3 Screen Shots.
I sorry. i no see screenshot.... maybe you forget attach ?

Quote
In procedure RunCommandLine it goes from Line 13 to 23 no matter what I enter.

I can't figure out what the line is doing.
Is what write user lucamar.

Other place in source make commandlist:
Code: Pascal  [Select]
  1. procedure TTermiForm.FormCreate(Sender: TObject);
  2. begin
  3.   // make setup
  4.   .. have remove for write forum look in zip for see.
  5.  
  6.   // have command list
  7.   CommandList := TStringList.Create;
  8.   CommandList.AddObject('flyby', TObject(@flyby));
  9. end;
  10.  
That mean CommandList (TStringList) have add item string read 'flyby' and Object of line have pointer procedure flyby.

If press button:
Code: Pascal  [Select]
  1. procedure TTermiForm.RunButtonClick(Sender: TObject);
  2. begin
  3.   // make action
  4.   (Sender as TButton).Enabled := false;  // Make button disable (is possible run 1 command, not many)
  5.   RunCommandLine(ParameterEdit.Text);  // RunCommandLine and have ParameterEdit.Text for "command-line"
  6.   (Sender as TButton).Enabled := true;  // If command have run then have button enable and can run other command
  7. end;
  8.  
Then is call RunCommandLine(...)


Then i make write more RunCommandLine:
Code: Pascal  [Select]
  1. procedure TTermiForm.RunCommandLine(CommandLine: string);
  2. var
  3.   index : SizeInt;
  4.   s : string;
  5. begin
  6.   // have line command enter edit print Memo
  7.   PrintLine(' $ ' + CommandLine);
  8.  
  9.   // have make ParameterList empty (ParameterList type TStringArray)
  10.   ParameterList := nil;
  11.  
  12.   // Have split line command.
  13.   // Make split use space, #8,#9,#10,#13 and no split quote part double quote.
  14.   // Split is all most work same ExtractWord()
  15.   ParameterList := CommandLine.Split([' ',#8,#9,#10,#13], '"');
  16.   // ParameterList now can have many parts. First part have command. Other part after have (many) parameter.
  17.   // If have ParameterEdit.Text empty then ParameterList empty
  18.  
  19.   // ParameterList empty or no empty ?
  20.   if Length(ParameterList) > 0 then
  21.   // is no empty ...
  22.   begin
  23.     // .. then have "command" find in CommandList
  24.     index := CommandList.IndexOf(ParameterList[0]);  // ParameterList[0] = Split item first and mean "command".
  25.     // is find "command" ? (is from ParameterEdit)
  26.     if (index >= 0) then  
  27.     // Yes, is find command ....
  28.     begin
  29.       // ... then have execute command
  30.       TProcedure(CommandList.Objects[index])();
  31.     end
  32.     // If index < 0 then IndexOf not have find "command" and print error
  33.     else PrintLine(ParameterList[0] + ' : command not found');
  34.   end
  35.   // if Length(ParameterList) <= 0 then have empty ParameterEdit then have press button. Is mean "command" empty
  36.   else PrintLine('Empty command');
  37. end;      

What is more no understand ? I happy write explain  :)

Add: have write more explain.
Title: Re: DataProblems Maybe
Post by: JLWest on May 07, 2019, 08:23:26 pm
Can't attach too big.


Attached 3 Screen Shots.
I sorry. i no see screenshot.... maybe you forget attach ?

Quote
In procedure RunCommandLine it goes from Line 13 to 23 no matter what I enter.

I can't figure out what the line is doing.
Is what write user lucamar.

Other place in source make commandlist:
Code: Pascal  [Select]
  1. procedure TTermiForm.FormCreate(Sender: TObject);
  2. begin
  3.   // make setup
  4.   .. have remove for write forum look in zip for see.
  5.  
  6.   // have command list
  7.   CommandList := TStringList.Create;
  8.   CommandList.AddObject('flyby', TObject(@flyby));
  9. end;
  10.  
That mean CommandList (TStringList) have add item string read 'flyby' and Object of line have pointer procedure flyby.

If press button:
Code: Pascal  [Select]
  1. procedure TTermiForm.RunButtonClick(Sender: TObject);
  2. begin
  3.   // make action
  4.   (Sender as TButton).Enabled := false;  // Make button disable (is possible run 1 command, not many)
  5.   RunCommandLine(ParameterEdit.Text);  // RunCommandLine and have ParameterEdit.Text for "command-line"
  6.   (Sender as TButton).Enabled := true;  // If command have run then have button enable and can run other command
  7. end;
  8.  
Then is call RunCommandLine(...)


Then i make write more RunCommandLine:
Code: Pascal  [Select]
  1. procedure TTermiForm.RunCommandLine(CommandLine: string);
  2. var
  3.   index : SizeInt;
  4.   s : string;
  5. begin
  6.   // have line command enter edit print Memo
  7.   PrintLine(' $ ' + CommandLine);
  8.  
  9.   // have make ParameterList empty (ParameterList type TStringArray)
  10.   ParameterList := nil;
  11.  
  12.   // Have split line command.
  13.   // Make split use space, #8,#9,#10,#13 and no split quote part double quote.
  14.   // Split is all most work same ExtractWord()
  15.   ParameterList := CommandLine.Split([' ',#8,#9,#10,#13], '"');
  16.   // ParameterList now can have many parts. First part have command. Other part after have (many) parameter.
  17.   // If have ParameterEdit.Text empty then ParameterList empty
  18.  
  19.   // ParameterList empty or no empty ?
  20.   if Length(ParameterList) > 0 then
  21.   // is no empty ...
  22.   begin
  23.     // .. then have "command" find in CommandList
  24.     index := CommandList.IndexOf(ParameterList[0]);  // ParameterList[0] = Split item first and mean "command".
  25.     // is find "command" ? (is from ParameterEdit)
  26.     if (index >= 0) then  
  27.     // Yes, is find command ....
  28.     begin
  29.       // ... then have execute command
  30.       TProcedure(CommandList.Objects[index])();
  31.     end
  32.     // If index < 0 then IndexOf not have find "command" and print error
  33.     else PrintLine(ParameterList[0] + ' : command not found');
  34.   end
  35.   // if Length(ParameterList) <= 0 then have empty ParameterEdit then have press button. Is mean "command" empty
  36.   else PrintLine('Empty command');
  37. end;      

What is more no understand ? I happy write explain  :)

Add: have write more explain.
Title: Re: DataProblems Maybe
Post by: Thausand on May 07, 2019, 09:12:15 pm
Can't attach too big.
Have copy-paste Memo ?

I no understand what is no work for you  :(

In theory it work simpe: TEdit read "flyby byAirports.txt"  (no quote) then press TButton.

Then can wrong not find file then make read TEdit command + filename complete , have example "flyby E:\MyFiles\MyStorage\some\were\can\find\byAirports.txt"

If directory file or name file have space then my program no work ok and have need fix. That because if space then need quote TEdit command and/or need quote TEdit filename then type helper strip() no remove quote  :o

Other time my program no work correct and have error if no have "read/access right" for file or no have "read/acces right" for directory. That no problem my program and error windows/user.

I no want my program "hard-code" name file/directory "byAirports.txt"
Title: Re: DataProblems Maybe
Post by: Thausand on May 07, 2019, 10:07:53 pm
Ok, user JLWest me send picture private.

Then i write conclude:
- program flybylaz executable locate: F:\CodeLib\Flyby\flybylaz_publish\flybylaz.exe
- file airport locate: F:\CodeLib\Flyby\ByAirport.txt
- JLWest user start my program and make write edit: F:\CodeLib\Flyby\byairport.txt then button click
- Then FlyByLaz program write: command not found

How make work:
- Make sure file airport locate: F:\CodeLib\Flyby\ByAirport.txt
- start flybylaz program
- write Edit box: flyby F:\CodeLib\Flyby\byairport.txt
- click button

Is that work JLWest ?
Title: Re: DataProblems Maybe
Post by: JLWest on May 07, 2019, 11:08:50 pm
Ok, user JLWest me send picture private.

Then i write conclude:
- program flybylaz executable locate: F:\CodeLib\Flyby\flybylaz_publish\flybylaz.exe
- file airport locate: F:\CodeLib\Flyby\ByAirport.txt
- JLWest user start my program and make write edit: F:\CodeLib\Flyby\byairport.txt then button click
- Then FlyByLaz program write: command not found

How make work:
- Make sure file airport locate: F:\CodeLib\Flyby\ByAirport.txt
- start flybylaz program
- write Edit box: flyby F:\CodeLib\Flyby\byairport.txt
- click button

Is that work JLWest ?

Make sure file airport locate: F:\CodeLib\Flyby\ByAirport.txt

Look at the screen .PNG's from my GDrive.

I sent the link via a private message.
 
Title: Re: DataProblems Maybe
Post by: Thausand on May 08, 2019, 01:37:11 am
Ok, user JLWest me send picture private.

Then i write conclude:
- program flybylaz executable locate: F:\CodeLib\Flyby\flybylaz_publish\flybylaz.exe
- file airport locate: F:\CodeLib\Flyby\ByAirport.txt
- JLWest user start my program and make write edit: F:\CodeLib\Flyby\byairport.txt then button click
- Then FlyByLaz program write: command not found

How make work:
- Make sure file airport locate: F:\CodeLib\Flyby\ByAirport.txt
- start flybylaz program
- write Edit box: flyby F:\CodeLib\Flyby\byairport.txt
- click button

Is that work JLWest ?

Make sure file airport locate: F:\CodeLib\Flyby\ByAirport.txt

Look at the screen .PNG's from my GDrive.

I sent the link via a private message.

I have write answer post reply #108  :)
Title: Re: DataProblems Maybe
Post by: JLWest on May 08, 2019, 05:24:40 am


GDrive link of the following:
https://drive.google.com/open?id=1RgTBOWDfciSqyUeyBc2nTb3-QEmKa9_Q


OK:

Step by step:

Step 1:

   - Make sure file airport locate: F:\CodeLib\Flyby\ByAirport.txt

Go to Cmd (Dos Box)
CD F:
F:\>cd Codelib\Flyby

F:\Codelib\flyby>Dir

Shows:

Volume in drive F is Storage
 Volume Serial Number is 34B9-7BA5

 Directory of F:\CodeLib\Flyby

05/07/2019  05:00 PM    <DIR>          .
05/07/2019  05:00 PM    <DIR>          ..
05/05/2019  12:25 AM         2,383,288 ByAirport.txt   <------- File Same dir as program
05/06/2019  01:53 AM            11,528 flyby.pas
05/07/2019  08:25 AM            15,848 flyby.txt
05/05/2019  09:57 PM             3,433 flybyfpc_publish.zip
05/07/2019  08:43 AM    <DIR>          flybylaz_publish
05/05/2019  09:57 PM           134,992 flybylaz_publish.zip
05/06/2019  02:00 PM           686,339 Screenshot (43).zip
05/07/2019  07:56 AM         1,711,837 Screenshot (45).png
05/07/2019  08:05 AM         1,700,495 Screenshot (45).zip
05/07/2019  08:06 AM         1,640,720 Screenshot (46).png
05/07/2019  08:08 AM         1,625,823 Screenshot (46).zip
05/07/2019  05:00 PM                 0 x
              11 File(s)      9,914,303 bytes
               3 Dir(s)  750,504,476,672 bytes free

Step 2:

       - start flybylaz program

See .png on GDrive.

 convert to program and I will try to run in dos box (Not windows program)

Included list of all airports, all Cities and all Countries.      
      
      
      
      
      
      
      
I Show:

7,970,312 in Apt.txt
1,421 Cities
258 Countries


Title: Re: DataProblems Maybe
Post by: Thausand on May 08, 2019, 09:02:13 am
Hi hello JLWest,

GDrive link of the following:
https://drive.google.com/open?id=1RgTBOWDfciSqyUeyBc2nTb3-QEmKa9_Q
Yes thanksy for picture. I have see many time.

Quote
Go to Cmd (Dos Box)
CD F:
F:\>cd Codelib\Flyby

F:\Codelib\flyby>Dir

Shows:

Volume in drive F is Storage
 Volume Serial Number is 34B9-7BA5

 Directory of F:\CodeLib\Flyby

05/07/2019  05:00 PM    <DIR>          .
05/07/2019  05:00 PM    <DIR>          ..
05/05/2019  12:25 AM         2,383,288 ByAirport.txt   <------- File Same dir as program
No  ;D

Quote
05/06/2019  01:53 AM            11,528 flyby.pas
05/07/2019  08:25 AM            15,848 flyby.txt
05/05/2019  09:57 PM             3,433 flybyfpc_publish.zip
05/07/2019  08:43 AM    <DIR>          flybylaz_publish  <------------ this directory program flybylaz  lazarus

That no important. Important is know were is locate file directory Airport.txt

Quote
Step 2:

       - start flybylaz program

See .png on GDrive.
Yes  :)

You make show many many many time you no type editbox "flyby F:\CodeLib\Flyby\ByAirport.txt".

I have write answer problem reply post #108. I have suggest you read.

Quote
convert to program and I will try to run in dos box (Not windows program)
I sorry and write if can you no make work lazarus flybylaz then can also no make work command-line flyby version. Is make start same.

Quote
I Show:

7,970,312 in Apt.txt
1,421 Cities
258 Countries
Country count is ok if no count nil  :).

City count i have make calculate and write later.
Is later:
City count unique me and no count nil = 4719. That read my program have many more count city program you. How is count you program city ?

add:

I not know. Why you write:
Code: [Select]
7,970,312 in Apt.txt
[u]1,421 Cities[/u]
258 Countries
....then have package.zip show good count ICAO/Airport, Country and City ? (*)

(*) package show screenshot you no type "flyby" and alone show editbox "F:\CodeLib\Flyby\ByAirport.txt". You have break keyboard and no have letter "f", "l", "y", "b", "y" ?  :D