Recent

Author Topic: Won't load Apt.Dat.  (Read 15182 times)

wp

  • Hero Member
  • *****
  • Posts: 5839
Re: Won't load Apt.Dat.
« Reply #15 on: January 09, 2019, 10:31:07 am »
In order to have something comparable, can you post how you create the 10M lines file?

Your results indicate that file access seems to be the limiting factor. There has been a recent discussion in which Bart showed that using the correct size of text buffers can speed up conventional file I/O considerably (https://forum.lazarus.freepascal.org/index.php/topic,42629.msg297970.html).
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

GetMem

  • Hero Member
  • *****
  • Posts: 3467
Re: Won't load Apt.Dat.
« Reply #16 on: January 09, 2019, 10:41:20 am »
Quote
In order to have something comparable, can you post how you create the 10M lines file?
http://forum.lazarus.freepascal.org/index.php/topic,43806.msg307101.html#msg307101

wp

  • Hero Member
  • *****
  • Posts: 5839
Re: Won't load Apt.Dat.
« Reply #17 on: January 09, 2019, 11:29:18 am »
Standard I/O
Code: [Select]
BLOCK_SIZE      Time to read
-----------     ------------
         1          42.8 s
      1000          42.5 s
    10,000          36.1 s
   100,000          11.7 s
 1,000,000           8.5 s
10,000,000           8.3 s

ReadLn only          7.8 s

Using SetTextBuf with varied BUFFER_SIZE:
Code: [Select]
BUFFER_SIZE    BLOCK_SIZE     Time to read
-----------    ----------     ------------
 1 kB          10,000,000         5.1 s
 1 MB          10,000,000         4.0 s
16 MB          10,000,000         3.9 s

 1 kB          ReadLn only        4.7 s
 1 MB          ReadLn only        3.6 s
16 MB          ReadLn only        3.6 s
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

CCRDude

  • Sr. Member
  • ****
  • Posts: 491
Re: Won't load Apt.Dat.
« Reply #18 on: January 09, 2019, 01:01:55 pm »
Some thoughts...

No screen is capable of displaying 10 million datasets at once.
No user is capable of viewing 10 million datasets at once.
Which means they don't need to be in memory at the same time.

So whatever kind of display there is for these, think about pagination. Display the visible chunks, make navigation between these chunks as easy as possible.

And once you're there - think about using databases, since they totally simplify (and speed up if you use proper indizes) that kind of use in the background.

lucamar

  • Hero Member
  • *****
  • Posts: 1814
Re: Won't load Apt.Dat.
« Reply #19 on: January 09, 2019, 01:21:20 pm »
Pick another airport or close the file and quit. Tricky, may have to close the file and start  all over at the first record again depending on airport picked.

You can avoid that by using streams and doing a Seek to the point you want to start reading. Use a buffered stream, else your program will slow to a crawl very quickly.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4 & 2.0.2 w/FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

GetMem

  • Hero Member
  • *****
  • Posts: 3467
Re: Won't load Apt.Dat.
« Reply #20 on: January 09, 2019, 02:05:42 pm »
@wp
It looks like for BLOCK_SIZE larger then 100,000 the speed increase is significant, however you can overshoot the length of the array with a large amount. I find a solution which can count the records fast(3 seconds on my computer), then I can set the length to the exact value:
Code: Pascal  [Select]
  1. function GetRecordCount(const AFileName: string): Integer;
  2. var
  3.   DataFile: TextFile;  
  4. begin
  5.   Result := 0;
  6.   AssignFile(DataFile, AFileName);
  7.   try
  8.     Reset(DataFile);  
  9.     while not eof(DataFile) do
  10.     begin
  11.       Readln(DataFile);//this line has changed, basically it reads nothing but is needed otherwise we go into an endless loop
  12.       Inc(Result);
  13.     end;
  14.     CloseFile(DataFile);
  15.   except
  16.     //...
  17.   end;
  18. end;
  19.  


PS: SetTextBuff indeed helps, the whole process(counting + loading the data) takes 5-6 seconds on my computer.
PS1: I think OP has enough info now to code his application.
« Last Edit: January 09, 2019, 02:08:12 pm by GetMem »

howardpc

  • Hero Member
  • *****
  • Posts: 3043
Re: Won't load Apt.Dat.
« Reply #21 on: January 09, 2019, 02:14:52 pm »
No screen is capable of displaying 10 million datasets at once.
No user is capable of viewing 10 million datasets at once.
Which means they don't need to be in memory at the same time.
As CCRDude points out, when trying to edit huge amounts of data, it is essential to decouple the data location, data editing and data display functions.
Which of the 10 million lines require editing?Just a few?How can you locate them without having to display them? What identifies them? Or perhaps you should report to the user that the file has no data that needs editing?
If the file does contain relevant line(s) that require editing, your display function should limit itself just to that line or lines.
And your edit function should know how to reinsert edited lines into the original text file at the correct location.

JLWest

  • Hero Member
  • *****
  • Posts: 545
Re: Won't load Apt.Dat.
« Reply #22 on: January 09, 2019, 08:00:23 pm »
Wow. a dynamic array. I'll give it a try. Have a few questions like How to view the data after loaded into the array. I guess you could load the ones you need into a listbox maybe?


@howardpc
Yes I agree, no one can handle 7.9 million records. My original thinking was to load all records into a listbox. Find the ones I needed and place them into a second listbox and then allow editing and put them back into the 1st listbox and write out to file.

@GetMem
Oh I know exactly how many records are in the file. 7,970, 312. So can i do this:

Block_Size = 7970312;
SetLength(Data, Block_Size);

I can locate the lines needed for editing with a recordID of 1301 and 1302 First field in the record. They come as a pair in the record set of say KPDX, Portland Or within the Apt.Dat file.

The 1301 is the gate number 'Gate-A12' just like when  you go to the airport and the second record defines what type of gate, Cargo, Airline, Military and what ICAO Operations can use the gate, DAL SWA AAL. Almost all gate at a large airport are co-leased by several ICAO Operators.
 

I don't know if the question was directed to me but I didn't create the file. It comes with X-Plane 11. You can get it by downloading a Demo copy of X-Plane 11 for free and saving off the file and deleting X-Plane 11. It's a big download. Or try it out. It's interesting, it's not really a game.

Think of the movie "Scully' and they are flying the simulators to duplicate his landing on the Hudson. That's X-Plane 11. Of course I don't have the commercial version but the retail version.

I'll have some questions but I think I can download the code provided and get an array loaded.

Thanks All
 
JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

JLWest

  • Hero Member
  • *****
  • Posts: 545
Re: Won't load Apt.Dat.
« Reply #23 on: January 09, 2019, 11:55:26 pm »
@wp
It looks like for BLOCK_SIZE larger then 100,000 the speed increase is significant, however you can overshoot the length of the array with a large amount. I find a solution which can count the records fast(3 seconds on my computer), then I can set the length to the exact value:
Code: Pascal  [Select]
  1. function GetRecordCount(const AFileName: string): Integer;
  2. var
  3.   DataFile: TextFile;  
  4. begin
  5.   Result := 0;
  6.   AssignFile(DataFile, AFileName);
  7.   try
  8.     Reset(DataFile);  
  9.     while not eof(DataFile) do
  10.     begin
  11.       Readln(DataFile);//this line has changed, basically it reads nothing but is needed otherwise we go into an endless loop
  12.       Inc(Result);
  13.     end;
  14.     CloseFile(DataFile);
  15.   except
  16.     //...
  17.   end;
  18. end;
  19.  


PS: SetTextBuff indeed helps, the whole process(counting + loading the data) takes 5-6 seconds on my computer.
PS1: I think OP has enough info now to code his application.

I put this in FormCreaete and on my machine by the time the form shows it has read the file and posted the number to BlockSize which is a global. I didn't time it but about 2 seconds.

Can't seem to get it to load the array.
JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

josh

  • Hero Member
  • *****
  • Posts: 746
Re: Won't load Apt.Dat.
« Reply #24 on: January 10, 2019, 01:01:37 am »
Hi

Your not reading any data into any array

I have not tested it at all, just typed it; so may not work straight off, just a concept..

Code: [Select]
// set some const and variables up
const  MyMaxArraySize=8000000;
Var  s:string;
     MyFileHandle:textfile;
     MyCounter:longint=0;
     myline:string='';
     MyArray:Array of String;
     MyFileName:AnsiString;


// code to read in a text file into dynamic array, and adjust array after
// some minimal test during routine

// please note I would not do this iin the on create event; if this routine takes some time
// to complete on an old slow machine, with standard hd, users will not see anything until it has finished.
// best to add into form show event; and have a global variable so that it is only run once.
// that way your application will show; whilst its working.


Setlength(MyArray,MyMaxArraySize);
if fileexists(MyFileName) Then
begin
  Assignfile(MyFileHandle,MyFileName);
  Reset(MyFileHandle);
  myGetOut:=False;
  While ((not eof(MyFileHandle)) and (MyGetOut=False)) do
  begin
    readln(MyFileHandle,myline);
    if myline<>'' then
    begin
      myArray[MyCounter]:=s;
      inc(MyCounter);
      If MyCounter>MyMaxArraySize then
      begin
        showmessage('Too Much Data, exceeded :'+inttostr(MyMaxArraySize))
        MyGetOut:=True;
      End;
    End;
  end;
  closefile(MyFileHandle);
  setlength(MyArray,MyCounter);
end;                           
Development Installation Lazarus 1.3, FPC 2.7.1,Windows 7/8 32/64, OSX, *nix

Test Environment Lazarus & FPC Trunk on Windows and OSX (Cocoa Mainly on OSX). Testing also Crosscompile windows to OSX.. 
Any posts made from 2015 will be based on Lazarus Trunk.

wp

  • Hero Member
  • *****
  • Posts: 5839
Re: Won't load Apt.Dat.
« Reply #25 on: January 10, 2019, 01:03:46 am »
Or use the code posted in reply #12 or #13
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

josh

  • Hero Member
  • *****
  • Posts: 746
Re: Won't load Apt.Dat.
« Reply #26 on: January 10, 2019, 01:18:53 am »
Oops

Should have looked back a page...

Too many different topics on same subject; making it awkward to follow...

This is massive dataset; do we know how this is going to made into a user-workable UI yet?
I think the UI should be thought out first; as this will guide the route of how to handle it.

What spec machine is this designed to run on?



Development Installation Lazarus 1.3, FPC 2.7.1,Windows 7/8 32/64, OSX, *nix

Test Environment Lazarus & FPC Trunk on Windows and OSX (Cocoa Mainly on OSX). Testing also Crosscompile windows to OSX.. 
Any posts made from 2015 will be based on Lazarus Trunk.

JLWest

  • Hero Member
  • *****
  • Posts: 545
Re: Won't load Apt.Dat.
« Reply #27 on: January 10, 2019, 02:57:29 am »
Yea, Some of that has been thought out ahead of time. It will run on Windows 7 and above. Designed to run on a Windows maching simular to my spec's at the bottom of the post.

Getting it loaded is the problem right this min. The current proposal is a dynamic array.

Each text line in the file has a record indicator, '1', '16' and '17' in the first column denotes an airport, seaplane base or helipad. Within those records in the fifth field is the name of the airport. Followed by 2 or 3,000 records of lights, runways, taxiways, painted lines, signs - most of which I don't have to deal with.

What I need to edit is the 1301 (RampStarts) and 1302(GateAssigments) records. Maybe 30 for a small airport to 400 records for an International. LAX or KJFK, Chicago O'Hare. 

Load those into a listbox and allow editing. Then dump the file to disk. The idea is the file would remain open but only the 1301 and 1302 would be visible and in memory.


JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

JLWest

  • Hero Member
  • *****
  • Posts: 545
Re: Won't load Apt.Dat.
« Reply #28 on: January 10, 2019, 03:05:31 am »
Or use the code posted in reply #12 or #13

Yes, I'm using #12, #13 and the record count function. The record count function works great and and returns the correct number of records.
JLWEST
Lazuras ver 2.0.2 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

GetMem

  • Hero Member
  • *****
  • Posts: 3467
Re: Won't load Apt.Dat.
« Reply #29 on: January 10, 2019, 05:42:58 am »
@JLWest
Quote
I put this in FormCreaete and on my machine by the time the form shows it has read the file and posted the number to BlockSize which is a global. I didn't time it but about 2 seconds.
Don't do that, on a slower computer it might take much longer to count the records. Please drop a TTimer to your form, set Interval to 100, Enabled to false, create an OnTimer event then:
Code: Pascal  [Select]
  1. procedure TfMain.FormShow(Sender: TObject);
  2. begin
  3.   Timer1.Enabled := True;
  4. end;
  5.  
  6. procedure TfMain.Timer1Timer(Sender: TObject);
  7. begin
  8.   Timer1.Enabled := False;
  9.   Caption := 'Loading data. Please wait...';
  10.   BlockSize  := GetRecordCount(ExtractFilePath(Application.ExeName) + 'klax.dat');
  11.   if BlockSize = 0 then
  12.   begin
  13.     ShowMessage('Error loading file.');
  14.     Exit;
  15.   end;
  16.   //...
  17. end;  

Quote
Getting it loaded is the problem right this min. The current proposal is a dynamic array.
What is the exact problem? Do you get an error? Please be more specific. If there is no error, add the following code(after the array is loaded):
Code: Pascal  [Select]
  1.   for I := High(Data) downto High(Data) - 100 do
  2.      ListBox1.Items.Add(DataCondition[Len].FLine);
  3.  
You should see the last 100 records in the listbox.



« Last Edit: January 10, 2019, 06:11:48 am by GetMem »