Recent

Author Topic: Won't load Apt.Dat.  (Read 14912 times)

GetMem

  • Hero Member
  • *****
  • Posts: 3459
Re: Won't load Apt.Dat.
« Reply #45 on: January 11, 2019, 07:21:55 am »
@JLWest
Quote
This is probably going to take me a couple of days.
:o  Don't forget that the dat file is mostly text, archived should be under 100MB. You can even send it to me via mail.   

wp

  • Hero Member
  • *****
  • Posts: 5737
Re: Won't load Apt.Dat.
« Reply #46 on: January 11, 2019, 10:21:06 am »
@wp
You opened Pandora's box. :D Now OP will load the big file in VTV, which of course will work, VTV easily can handle 10M records. Still wrong approach in my opinion, the array must be filtered first. He mentioned somewhere that after filtering, only a few thousand item remains, this can be easily shown in any control.
Loading only the filtered data into memory will make it more difficult to write a valid file back after editing some records. In order to save an edited record you must read the original file line by line and write each line immediately to the new file; when the line which was edited is read the modified content must be written to the new file instead of the original line.

Or, during first reading of the file you collect an index of the stream positions of the start of all lines. After editing a specific line you look up in the index at which stream position this line begins in the original file, read everything en-block before that line and write it to a new file, write the new line, then jump to the next line in the original file and read and write the rest of the file. The index must be recreated after the new file is written.
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

GetMem

  • Hero Member
  • *****
  • Posts: 3459
Re: Won't load Apt.Dat.
« Reply #47 on: January 11, 2019, 10:50:06 am »
What I meant to say by "Still wrong approach in my opinion, the array must be filtered first" is: load the whole file into a dynamic array, filter it, then show the filtered records to the user(Listbox, VTV, whatever). In my #12 reply, I even store the index of the large array for future reference.

wp

  • Hero Member
  • *****
  • Posts: 5737
Re: Won't load Apt.Dat.
« Reply #48 on: January 11, 2019, 11:14:46 am »
OK, I misunderstood this. But I think the idea of filtering the file during reading to load only the records needed is good. It is mentioned somewhere that the file has several hundreds of MB in size. Depending on the system and what else is running this may be too much for 32 bit systems. And maybe there are files which are even larger, who knows? Of course when the OP allows his program to be 64bit then this point can be neglegted.

P.S. I find it interesting that we reached the point to discuss out-of-memory issues again after burying DOS.
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

lucamar

  • Hero Member
  • *****
  • Posts: 1596
Re: Won't load Apt.Dat.
« Reply #49 on: January 11, 2019, 01:38:39 pm »
Yes, Correct to that point in your narrative. 

But the lines that need to be edited come in pairs of text lines. a 1301 and a corresponding 1302 text line. The 1301 is the gate number Gate A-12 and the 1302 is the Gate Specifications. Cargo, Airline, Military, A gate, Tie down and who owns the right to park at the gate. SWA, DAL AAL. Often the gates are co-leased in large airports.

That second point isn't really important to the problem at hand, which is how to avoid having to handle all 280 MiB. If the rest is correct then a pretty easy solution is to divide the huge file into smaller files and an index.

The basic process would be: read all lines pertaining to an airport and save them to a new file with just the data for that airport and save the base airport line (those coded with 1, 16, 17) to an index file. Then on to the next airport, etc.

Once you have all those smaller fles you cn load the index, allow the user to select and load (and parse?), for modification, only the file corresponding to that airport. Finally, add an order in your program to regenerate the APP.DAT, which can be easily done by concatenating all those smaller files in the same order they were added to the index file, if the order is important.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4 & 2.0.2 w/FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

JLWest

  • Hero Member
  • *****
  • Posts: 542
Re: Won't load Apt.Dat.
« Reply #50 on: January 11, 2019, 02:40:28 pm »
@JLWest
Quote
This is probably going to take me a couple of days.
:o  Don't forget that the data file is mostly text, archived should be under 100MB. You can even send it to me via mail.

I'll zip and send it.

Oh, by the way, I looked at the VTV. What a nice object. but, the OP won't load the file into the VTV. I'll load the big file into the dynamic array then what I need (1 airport into the VTV and then the records for editing into the lower box of the VTV.

  Zipped it's 48,755KB. I'll try to send.
JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

JLWest

  • Hero Member
  • *****
  • Posts: 542
Re: Won't load Apt.Dat.
« Reply #51 on: January 11, 2019, 03:06:26 pm »
@ lucmar

Yes, the order is important. Lamair Research or X-Plane reads the file to verify integrity when you start the program. If it's wrong they give  a message of corrupted file and exit to the Desktop. In my case I always have a backup of X-Plane, but I think there is a way to download a new file from the site.

There are 35,000+ airports, Seaplane bases and Helipads. The only way I would feel comfortable with this approach is to beak each out in a separate file: F00001.txt, F00002.txt . . .F35187.txt.

And it's maybe the way to go. 
JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

GetMem

  • Hero Member
  • *****
  • Posts: 3459
Re: Won't load Apt.Dat.
« Reply #52 on: January 11, 2019, 03:17:40 pm »
@JLWest
I received an empty mail, no attachment.

PS: If you have a gmail account you can add the file to the google drive then paste the link here.

lucamar

  • Hero Member
  • *****
  • Posts: 1596
Re: Won't load Apt.Dat.
« Reply #53 on: January 11, 2019, 03:25:09 pm »
There are 35,000+ airports, Seaplane bases and Helipads. The only way I would feel comfortable with this approach is to beak each out in a separate file: F00001.txt, F00002.txt . . .F35187.txt.

And it's maybe the way to go.

Yes, that's what I sugested: dividing the big file into small one airport/heli/seaplane base files, whatever the name. Then all you have to do is select the appropiate file to get all data for airport/heli/seaplane base and regenerating the big file is just a matter of concatenating those files.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4 & 2.0.2 w/FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

440bx

  • Hero Member
  • *****
  • Posts: 836
Re: Won't load Apt.Dat.
« Reply #54 on: January 11, 2019, 04:57:52 pm »
Lamair Research or X-Plane reads the file to verify integrity when you start the program. If it's wrong they give  a message of corrupted file and exit to the Desktop.

<snip>

There are 35,000+ airports, Seaplane bases and Helipads. The only way I would feel comfortable with this approach is to beak each out in a separate file: F00001.txt, F00002.txt . . .F35187.txt.
Breaking the large file into separate files will make update and maintenance of your program rather cumbersome.  If you do that, every time there is a new version of the file, you'll have to break it into separate files for your program to use them.  That's a headache you don't want.

You should do it the way Lamair research is doing it, which most likely is, they map (CreateFile/CreateFileMapping/MapViewOfFile) the file, which is extremely quick, then scan it to ensure it is as required.  On your machine, scanning the file would most likely not even be noticeable.

If I knew of a Lazarus component that does that (map the file for you), I would suggest it but, I know next to nothing about what components are available, much less what they do.  Hopefully someone else can fill that gap.

In your case, your first step would be the same, create a file mapping of the file, the second step would be to scan the file to create an array of pointer indexes based on whatever information you want to show (the "filter" box in wp's example.)  This second step, you do it using the technique wp posted an example for in this thread, that is using a component such as VTV (which wp used) so you don't copy thousands (or more) strings into the component that displays them.

You can keep the code very simple if you scan the file twice.  The first scan to determine how many records/lines you need to keep track of, which would allow you to set the length of the dynamic array upfront before you proceed to load it with the pointers to the strings/lines.  The second scan to load the array.  That keeps memory management as simple as it gets.

The basic concept is very simple, you load the file into memory (mapping it is the fastest way also, memory efficient if you set it to PAGE_READONLY) then, create an index based on what the user (you in this case) wants to look at (as wp's example does.) 

On your machine, the above would happen in less than the blink of an eye.

HTH.

ETA:
correction: the component wp used is a ListView placed in virtual mode, not VTV.   See wp's post below for full detail.


« Last Edit: January 11, 2019, 06:08:22 pm by 440bx »
using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

wp

  • Hero Member
  • *****
  • Posts: 5737
Re: Won't load Apt.Dat.
« Reply #55 on: January 11, 2019, 05:30:56 pm »
using a component such as VTV (which wp used)
I think it's worth to put this right: I did not use VTV - which is hard to learn - but a plain old TListView. Using virtual mode in TListView is just three simple actions:
  • Activate virtual mode (ListView.OwnerData := true)
  • Define how many items are contained in the List (ListView.Items.Count := <number>);
  • Write an event handler OnData to populate the TListItem passed as a parameter with the strings to be displayed
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

440bx

  • Hero Member
  • *****
  • Posts: 836
Re: Won't load Apt.Dat.
« Reply #56 on: January 11, 2019, 06:05:32 pm »
using a component such as VTV (which wp used)
I think it's worth to put this right: I did not use VTV - which is hard to learn - but a plain old TListView. Using virtual mode in TListView is just three simple actions:
  • Activate virtual mode (ListView.OwnerData := true)
  • Define how many items are contained in the List (ListView.Items.Count := <number>);
  • Write an event handler OnData to populate the TListItem passed as a parameter with the strings to be displayed
Thank you for clarifying that.  The comments about VTV after the example you posted left me with that mistaken impression.    Virtual mode is definitely the way to go which, I believe, was the point you were making with the example you posted.
using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

JLWest

  • Hero Member
  • *****
  • Posts: 542
Re: Won't load Apt.Dat.
« Reply #57 on: January 11, 2019, 08:17:23 pm »
@JLWest
I received an empty mail, no attachment.

PS: If you have a gmail account you can add the file to the google drive then paste the link here.
I couldn't figure out how to attach a file. So I tried to abandon the email.
The size of the file 48,755 KB using 7zip.

I sent it to  Handoko as I have his direct E-mail and I think it went thru. He said he could host.

Is there a way I could move the file to one of my empty drives (yes obviously to that), share the drive and publish something so you could download the file.

Right now I setlength to the 7.9 and I'm getting an error. I was loading the small data set 62 records. Need to go back and make sure.
 
JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

JLWest

  • Hero Member
  • *****
  • Posts: 542
Re: Won't load Apt.Dat.
« Reply #58 on: January 11, 2019, 08:36:26 pm »
I'm able to load the small data set 62 records into the array and display in a listbox.

So I'm doing the setlength in the wrong place I guess.
JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

JLWest

  • Hero Member
  • *****
  • Posts: 542
Re: Won't load Apt.Dat.
« Reply #59 on: January 12, 2019, 04:07:29 am »
@JLWest
I received an empty mail, no attachment.

PS: If you have a gmail account you can add the file to the google drive then paste the link here.

I sorry I didn't see the notification. I was running down the Copyright of the file but think I have that to bed. I think I do have a gmail account. I have to look it up or get Google to give it to me. But they never throw anything away.


News: During the copyright flap I had the program running in the background and I think I loaded the 7.9 unwashed and unloved records. I can't setlength on the array in one chunk without getting an error. So I did it the way you wrote it. But I think it all loaded.

On to google
JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)