DataProblems Maybe

[Thausand]

Hi hello JLWest,

@Thausand Link to my GoogleDrive.
...

Thanksy !

I have download and make fast test. result be low

...
It is very basic. but fast.

Thanksy for suggest ! Is more windows and i not windows

No need edit and have full analyse data. If want cut then can use linux terminal cut 


I make fast program and test. Not many time when make so maybe error.

result:

Code: [Select]

./aptidat apt.dat
Processing file "apt.dat"

    Line Airport    Land     Sea    Heli  Land_R Water_R Helipad    Time
 7970312   35242   28458     585    6199   34893     784    9719   88 sec (89996.98 lines/second)
Is mean:
Have apt.dat line 7970312 and have airport 35242 (Land = 28458 / Sea = 585 / Heli = 6199). Have airport Land 34893 runway, have airport sea 784 runway, have airport heli 9719 helipad).

I make program continue maybe tomorrow. Is me other weekend busy then maybe no time.

[JLWest]

@Thausand

... show unique count
    Line   Error    ICAO    hash    City Country  Region     Lat     lon    Time
   34680       0   34680   34680   14138     259     248   34457   34537   35 sec (968.17 lines/second))
 

 The Lat and Lon should be the same. Indicates something is wrong with the data.

This is from ByAirports.

Trying to run this down.
May com from the program that generates ByAirports.txt from the Apt.dat file.

[Thausand]

Hi hello JLWest,

The Lat and Lon should be the same. Indicates something is wrong with the data.

Make safe think:

My program no compare lon and lat combine only separate lon and separate lat.

Then is possible many airport have same lon or have same lat. No lon and lat same (that real world no possible)

If want lat and lon compare combine then i can make program have compare and show lat/lon combine unique number. My program no have because "real world not possible have same" and number is same number ICAO. If want verificate numbers then is good for verificate. I no think that before then maybe i add for better verificate.

Thanksy

I think more problem:

ByAirport.dat you:

Code: [Select]

Line    ICAO
34680   34680

my program apt.dat:

Code: [Select]

Airport
35242
If have any airport unique ICAO then 35242 <> 34680 so if make me program "Airports.dat" then not same you "ByAirports.dat". My program make many more ICAO then have "ByAirports.dat" ICAO.

I have question:
1) if apt.dat airport have 1302 lon/lat then is more important runway lon/lat ? My program now take last find. That mean if program find 1302 lon/lat and later find runway lon/lat then my Airport.dat have runway lon/lat.
2) if want search lat/lon for find data "nil" then not important list look have any runway lat/lon and airport lat/lon for search ? (then make list have runway -> ICAO and Airport -> ICAO for look and make more big list)

[lucamar]

The Lat and Lon should be the same. Indicates something is wrong with the data.

Then is possible many airport have same lon or have same lat. No lon and lat same (that real world no possible)

I think he meant that there should be the same total number of Longitude values than of latitude values; otherwise, some data lines have only longitude (or only latitude), which is an error.

But looking closer, you seem to be computing the total of different values for each, isn't it?, so they might in fact be different if some airports have the same longitude and diffferent latitude or viceversa.

[Thausand]

Hi hello lucamar,

I think he meant that there should be the same total number of Longitude values than of latitude values; otherwise, some data lines have only longitude (or only latitude), which is an error.

Yes and then you be low write is good write. I make alone unique count (alone for lat and alone for lat, no combine).

If write technical my program then my program have simple (sort and dupignore) list for any column item and then add. TObject then mis use for count how many time is add for write more statistic later. Procedure ProcessLine() can have check for verificate item (column) data. I have make more before snippet have example verificate longitude and latitude and can add there. If no good verificate can write error and error count write if some thing no good.

(i make program simple so can understand and make add own routine/check simple and fast. I not know what is important JLWest for verificate. I know lon and lat have restrict value so that can add and add error when no validate lat/lon or add error when no lon/lat there. Then error-count write if is wrong data and need better read data and/or error-log).

But looking closer, you seem to be computing the total of different values for each, isn't it?, so they might in fact be different if some airports have the same longitude and diffferent latitude or viceversa.

Yes, that correct. Is count different lat and count different lon separate/different so is count alone unique.

That why program write:

... show unique count

Thanksy for make better english write.

[JLWest]

@Thausand

ICAO  34680             Unique  Only 1 in  apt.dat @ airport
City    14138
Lat     34457              Should be 34680  same as ICAO
Lon    34537              Should be 34680  same as ICAO
Regions 259              280

The number OF Lat's and Lon's have to be the same.
The Lat number and Lon number have to be different.
 

When program runs it loads multi Regions for some countries.


Most countries only have one region. However some Canada, China, Australia, US has more then one Region. Therefore, there are multi regions records in CountryRegions.Txt.

RegionCountry = "MU Cuba-I" or "K1 United States-O'

The 'I' in the record tell me I can do automatic processing:

if RegionCode = 'MU' and Country <> 'Cuba' then  begin

   // change Country here

or

 if CounryCode = 'Cuba' and RegionCode <> 'MU'
     //change Region code

1. The Longitude and Latitude of the airport is best. 

2. 14 tower viewpoint record: If the 1301 record Region_Code isn't set in the apt.dat then the next best Longitude and Latitude is the 14 record tower viewpoint. On really small airports they use that anyway.

3. I only use the  Longitude and Latitude of the runway if I have to: Many airports have multiple runways. One airport in Germany is five mile wide. KDEN in Denver is 7 miles wide. If you are trying find an airport 7 miles is a long ways.

I'm trying to work with your programs but I'm not very good with DOS programs and can't make it work. Built a GUI test program but can't get it to compile.

 

[lucamar]

The number OF Lat's and Lon's have to be the same.
The Lat number and Lon number have to be different.

What he's doing is counting the number of longitudes after removing duplicates, and the same for latitudes, so if there are two airports on the same longitude or the same latitude, they are counted (for this) as one. That's why the totals are different.

That is, unless you know for sure that there no two airport with the same latitude or no two airports with the same longitude. If so, then there is an error somewhere.

[JLWest]

Chances are good that more than one airport is on the same longitude or latitude. However, each of the records in my byAirports.txt have a latitude and longitude. So if you count the number of latitudes in ByAirp[orts.txt, the number of longitudes  and the number of airports they would each  equal 34,680. Otherwise the MSTRegions program would be throwing all kinds of errors.

Right now (Before the next test Cat Scan with definition) I think I'm going back and try to clean Apt.Dat. Generate a new ByAirports.txt.

[Thausand]

The number OF Lat's and Lon's have to be the same.
The Lat number and Lon number have to be different.

I make count unique so i have no same count. i write be low also.

Thanksy for information write line 14. I then have change my program and make better (maybe take time)

i look what can do make program GUI. DOS simple: fpc flyby.pas for compile and then terminal open and make run "flyby name_of_byairport.txt" (not quote) then enter.

That is, unless you know for sure that there no two airport with the same latitude or no two airports with the same longitude. If so, then there is an error somewhere.

That correct.

I make quick add export ByAirport.dat my program aptidat and make check use my program flyby and i write result and explain:

Code: [Select]

$ ./aptidat apt.dat Airport.txt
Processing file "apt.dat"

    Line Airport    Land     Sea    Heli  Land_R Water_R Helipad    Time
 7970312   35242   28458     585    6199   34893     784    9719   95 sec (83210.44 lines/second)

$ ./flyby Airport.txt
Processing file "Airport.txt"

... show unique count
    Line   Error    ICAO    hash    City Country  Region     Lat     lon    Time
   35241       0   35241   35241   14272     259     248   35109   35135   35 sec (983.01 lines/second))
No errors encountered during processing data

Statistics dtICAO :
     Shortest = "00C"  (3 characters)
     Longest  = "XCYCDL8"  (7 characters)
 Lowest count = "00C"  (1 times)
Highest count = "00CA"  (1 times)

Statistics dtHash :
     Shortest = "4"  (1 characters)
     Longest  = "1022949"  (7 characters)
 Lowest count = "1021615"  (1 times)
Highest count = "1022949"  (1 times)

Statistics dtCity :
     Shortest = "Ie"  (2 characters)
     Longest  = "Charlotte Amalie - Harry S. Truman Airport"  (42 characters)
 Lowest count = "(Cagayan De Oro)"  (1 times)
Highest count = "Nil"  (9552 times)

Statistics dtCountry :
     Shortest = "CA"  (2 characters)
     Longest  = "United States Minor Outlying Islands"  (36 characters)
 Lowest count = "Anguilla"  (1 times)
Highest count = "United States"  (14971 times)

Statistics dtRegion :
     Shortest = "AG"  (2 characters)
     Longest  = "Nil"  (3 characters)
 Lowest count = "AN"  (1 times)
Highest count = "Nil"  (23023 times)

Statistics dtLat :
     Shortest = "0.00000000"  (10 characters)
     Longest  = "-10.00499742"  (12 characters)
 Lowest count = "-0.02069980"  (1 times)
Highest count = "-23.59055556"  (6 times)

Statistics dtLon :
     Shortest = "0.00000000"  (10 characters)
     Longest  = "-100.00039500"  (13 characters)
 Lowest count = "-0.00094400"  (1 times)
Highest count = "-46.65805556"  (4 times)

---------- List dtCountry ----------
   1 : (10) "Afghanistan"
   2 : (5) "Albania"
   3 : (47) "Algeria"
   4 : (3) "American Samoa"
   5 : (27) "Angola"
   6 : (1) "Anguilla"
   7 : (9) "Antarctica"
   8 : (3) "Antigua and Barbuda"
   9 : (164) "Argentina"
  10 : (1) "Arizona"
  11 : (5) "Armenia"
  12 : (1) "Aruba"
  13 : (1492) "Australia"
  14 : (54) "Austria"
  15 : (9) "Azerbaijan"
  16 : (38) "Bahamas"
  17 : (2) "Bahrain"
  18 : (13) "Bangladesh"
  19 : (1) "Barbados"
  20 : (9) "Belarus"
  21 : (43) "Belgium"
  22 : (1) "Belize"
  23 : (2) "Benin"
  24 : (1) "Bermuda"
  25 : (3) "Bhutan"
  26 : (38) "Bolivia"
  27 : (4) "Bosnia and Herzegovina"
  28 : (28) "Botswana"
  29 : (3758) "Brazil"
  30 : (1) "British Indian Ocean Territory"
  31 : (2) "British Virgin Islands"
  32 : (2) "Brunei"
  33 : (22) "Bulgaria"
  34 : (2) "Burkina Faso"
  35 : (1) "Burkina Faso7"
  36 : (36) "Burma"
  37 : (1) "Burundi"
  38 : (15) "CA"
  39 : (6) "Cambodia"
  40 : (10) "Cameroon"
  41 : (704) "Canada"
  42 : (9) "Cape Verde"
  43 : (3) "Caribbean Netherlands"
  44 : (1) "Carroll"
  45 : (3) "Cayman Islands"
  46 : (2) "Central African Republic"
  47 : (5) "Chad"
  48 : (176) "Chile"
  49 : (102) "China"
  50 : (1) "Christmas Island"
  51 : (1) "Cocos (Keeling) Islands"
  52 : (92) "Colombia"
  53 : (3) "Comoros"
  54 : (4) "Congo (Brazzaville)"
  55 : (27) "Congo (Kinshasa)"
  56 : (10) "Cook Islands"
  57 : (26) "Costa Rica"
  58 : (18) "Croatia"
  59 : (31) "Cuba"
  60 : (1) "Curaçao"
  61 : (3) "Cyprus"
  62 : (99) "Czechia"
  63 : (7) "Côte d'Ivoire"
  64 : (87) "Denmark"
  65 : (1) "Djibouti"
  66 : (2) "Dominica"
  67 : (15) "Dominican Republic"
  68 : (50) "Ecuador"
  69 : (56) "Egypt"
  70 : (4) "El Salvador"
  71 : (2) "Equatorial Guinea"
  72 : (3) "Eritrea"
  73 : (13) "Estonia"
  74 : (22) "Ethiopia"
  75 : (2) "Falkland Islands"
  76 : (8) "Faroe Islands"
  77 : (23) "Fiji"
  78 : (92) "Finland"
  79 : (3) "FR"
  80 : (441) "France"
  81 : (5) "French Guiana"
  82 : (48) "French Polynesia"
  83 : (11) "Gabon"
  84 : (1) "Gambia"
  85 : (9) "Georgia"
  86 : (505) "Germany"
  87 : (8) "Ghana"
  88 : (1) "Gibraltar"
  89 : (56) "Greece"
  90 : (17) "Greenland"
  91 : (1) "Greenville"
  92 : (2) "Grenada"
  93 : (5) "Guadeloupe"
  94 : (2) "Guam"
  95 : (10) "Guatemala"
  96 : (2) "Guernsey"
  97 : (4) "Guinea"
  98 : (3) "Guinea-Bissau"
  99 : (16) "Guyana"
 100 : (6) "Haiti"
 101 : (10) "Honduras"
 102 : (1) "Hong Kong"
 103 : (76) "Hungary"
 104 : (81) "Iceland"
 105 : (195) "India"
 106 : (181) "Indonesia"
 107 : (88) "Iran"
 108 : (16) "Iraq"
 109 : (36) "Ireland"
 110 : (1) "Isle of Man"
 111 : (21) "Israel"
 112 : (120) "Italy"
 113 : (6) "Jamaica"
 114 : (130) "Japan"
 115 : (1) "Jersey"
 116 : (8) "Jordan"
 117 : (1) "K1"
 118 : (21) "Kazakhstan"
 119 : (14) "Kenya"
 120 : (20) "Kiribati"
 121 : (1) "Kosovo"
 122 : (3) "Kuwait"
 123 : (4) "Kyrgyzstan"
 124 : (8) "Laos"
 125 : (7) "Latvia"
 126 : (3) "Lebanon"
 127 : (5) "Lesotho"
 128 : (2) "Liberia"
 129 : (41) "Libya"
 130 : (1) "Liechtenstein"
 131 : (7) "Lithuania"
 132 : (3) "Luxembourg"
 133 : (1) "Macau"
 134 : (12) "Macedonia"
 135 : (28) "Madagascar"
 136 : (7) "Malawi"
 137 : (43) "Malaysia"
 138 : (6) "Maldives"
 139 : (8) "Mali"
 140 : (2) "Malta"
 141 : (7) "Marshall Islands"
 142 : (1) "Martinique"
 143 : (10) "Mauritania"
 144 : (2) "Mauritius"
 145 : (1) "Mayotte"
 146 : (132) "Mexico"
 147 : (4) "Micronesia"
 148 : (3) "Moldova"
 149 : (1) "Monaco"
 150 : (22) "Mongolia"
 151 : (3) "Montenegro"
 152 : (1) "Montserrat"
 153 : (26) "Morocco"
 154 : (18) "Mozambique"
 155 : (1) "Myanmar"
 156 : (55) "Namibia"
 157 : (1) "Namibia."
 158 : (1) "Nauru"
 159 : (41) "Nepal"
 160 : (34) "Netherlands"
 161 : (17) "New Caledonia"
 162 : (109) "New Zealand"
 163 : (10) "Nicaragua"
 164 : (7) "Niger"
 165 : (27) "Nigeria"
 166 : (7294) "Nil"
 167 : (1) "Niue"
 168 : (1) "Norfolk Island"
 169 : (1) "North Korea"
 170 : (9) "Northern Mariana Islands"
 171 : (142) "Norway"
 172 : (6) "Oman"
 173 : (68) "Pakistan"
 174 : (3) "Palau"
 175 : (2) "Palestinian Territory"
 176 : (22) "Panama"
 177 : (75) "Papua New Guinea"
 178 : (11) "Paraguay"
 179 : (1) "Peoples Republic of China"
 180 : (1) "Peru"
 181 : (79) "Perú"
 182 : (73) "Philippines"
 183 : (92) "Poland"
 184 : (52) "Portugal"
 185 : (32) "Puerto Rico"
 186 : (3) "Qatar"
 187 : (1) "Republic of Ireland (Eire)"
 188 : (39) "Romania"
 189 : (167) "Russia"
 190 : (7) "Russian Federation"
 191 : (1) "Russin Federation"
 192 : (5) "Rwanda"
 193 : (2) "Réunion"
 194 : (1) "Saint Barthélemy"
 195 : (2) "Saint Helena"
 196 : (2) "Saint Kitts and Nevis"
 197 : (2) "Saint Lucia"
 198 : (1) "Saint Martin"
 199 : (2) "Saint Pierre and Miquelon"
 200 : (5) "Saint Vincent and the Grenadines"
 201 : (4) "Samoa"
 202 : (2) "Sao Tome and Principe"
 203 : (62) "Saudi Arabia"
 204 : (10) "Senegal"
 205 : (7) "Serbia"
 206 : (8) "Seychelles"
 207 : (1) "Sierra Leone"
 208 : (6) "Singapore"
 209 : (30) "Slovakia"
 210 : (16) "Slovenia"
 211 : (23) "Solomon Islands"
 212 : (6) "Somalia"
 213 : (109) "South Africa"
 214 : (77) "South Korea"
 215 : (4) "South Sudan"
 216 : (107) "Spain"
 217 : (10) "Sri Lanka"
 218 : (11) "Sudan"
 219 : (8) "Suriname"
 220 : (1) "Swaziland"
 221 : (164) "Sweden"
 222 : (1) "Swiss"
 223 : (70) "Switzerland"
 224 : (23) "Syria"
 225 : (21) "Taiwan"
 226 : (6) "Tajikistan"
 227 : (14) "Tanzania"
 228 : (61) "Thailand"
 229 : (6) "Timor-Leste"
 230 : (2) "Togo"
 231 : (6) "Tonga"
 232 : (2) "Trinidad and Tobago"
 233 : (13) "Tunisia"
 234 : (74) "Turkey"
 235 : (10) "Turkmenistan"
 236 : (7) "Turks and Caicos Islands"
 237 : (1) "Tuvalu"
 238 : (6) "U.S. Virgin Islands"
 239 : (1) "U.S.A."
 240 : (18) "Uganda"
 241 : (26) "Ukraine"
 242 : (19) "United Arab Emirates"
 243 : (251) "United Kingdom"
 244 : (1) "United Sates"
 245 : (14971) "United States"
 246 : (4) "United States Minor Outlying Islands"
 247 : (2) "United States of America"
 248 : (11) "Uruguay"
 249 : (25) "US"
 250 : (26) "USA"
 251 : (15) "Uzbekistan"
 252 : (23) "Vanuatu"
 253 : (100) "Venezuela"
 254 : (24) "Vietnam"
 255 : (2) "Wallis and Futuna"
 256 : (3) "Western Sahara"
 257 : (13) "Yemen"
 258 : (8) "Zambia"
 259 : (17) "Zimbabwe"
First i see i have wrong count airport (i not know why. I make quick export so think have small error. I look later).

then have more better look:

Code: [Select]

Statistics dtLat :
     Shortest = "0.00000000"  (10 characters)
     Longest  = "-10.00499742"  (12 characters)
 Lowest count = "-0.02069980"  (1 times)
Highest count = "-23.59055556"  (6 times)
Lowest count = 1 times, Highest count is 6 times.

That mean Airport latitude "-23.59055556" is count 6 times. Then 6 airport in apt.dat have coordinate latitude -23.59055556

then:

Code: [Select]

Statistics dtLon :
     Shortest = "0.00000000"  (10 characters)
     Longest  = "-100.00039500"  (13 characters)
 Lowest count = "-0.00094400"  (1 times)
Highest count = "-46.65805556"  (4 times)
Lowest count = 1 times, Highest count is 4 times.

That mean Airport longitude "-46.65805556" is count 4 times. Then 4 airport in apt.dat have coordinate longitude -46.65805556.

Is many coincidence have 6 airport (from 35241)  same latitude 23.59055556 and have 4 airport (from 35241) same longitutude -46.65805556 but i think that possible when have big planet earth 

add:
is same for country and then no problem understand 

Code: [Select]

Country
259 


Code: [Select]

245 : (14971) "United States"
Country name ""United States" is count 14971 and Country count is 259. My program no write Country count = 35242 because have ICAO 35242. Have 259 unique name country 

[JLWest]

@Thausand

I guess it possibly or my data extraction program is wrong.

The difference in the number of airports is due to the fact that I don't build a record for Heliport but I do Sea Plane bases.

 

[JLWest]

@lucmar

I don't know. Not getting a apt.bak file

Volume in drive C has no label.
 Volume Serial Number is 846B-146A

 Directory of C:\UserFiles

05/04/2019  06:19 PM    <DIR>          .
05/04/2019  06:19 PM    <DIR>          ..
04/27/2019  10:08 PM       292,818,538 apt.dat
04/27/2019  10:18 PM       292,818,538 apt.txt
05/01/2019  10:17 PM        61,635,578 apt.zip
05/04/2019  06:07 PM           288,787 cleanbom.exe
04/27/2019  09:06 PM        16,164,022 UTF8Clean.exe
05/04/2019  06:19 PM                 0 x
               6 File(s)    663,725,463 bytes
               2 Dir(s)  314,017,177,600 bytes free



C:\UserFiles>cleanbom apt.txt
or
C:\UserFiles>cleanbom apt.dat



 

[Thausand]

I guess it possibly or my data extraction program is wrong.

Is can my program extract wrong. So maybe not you program error. That why i ask question 

The difference in the number of airports is due to the fact that I don't build a record for Heliport but I do Sea Plane bases.

For basic is simple. My program have count (unique) airport  35242 and (unique) count Heli 6199. If apt.dat have unique ICAO airport (and any Airport apt.dat have (unique) ICAO) then count = 35242 - 6199 = 29043 unique airport (and ICAO).

[JLWest]

"The difference in the number of airports is due to the fact that I don't build a record for Heliport but I do Sea Plane bases."

if the Heliport is on an airport I don't build a record but if it's like at a hospital I do build a record I think, I'll have to look.

if  your program outputs a file, e-mail me the file.

Goto go somewhere.

[Thausand]

Hi hello JLWest,

I sorry i no lucamar

I don't know. Not getting a apt.bak file
04/27/2019  10:08 PM       292,818,538 apt.dat

i make linux:

Code: [Select]

$ ls -l apt.dat
-rw-r--r-- 1 pi pi 292818538 Apr 27 22:08 apt.dat
If read good then see is same size.

I know apt.dat my no have BOM.

I not know program lucamar cleanbom but when have not BOM then why write file ? Then maybe program cleanbom read no BOM and then no write file new and no need file "apt.bak" ? (maybe i can look cleanbom and check and write later).

[JLWest]

Same time stamp and size