Recent

Author Topic: Won't load Apt.Dat.  (Read 10065 times)

JLWest

  • Sr. Member
  • ****
  • Posts: 340
Won't load Apt.Dat.
« on: January 08, 2019, 11:00:54 pm »
Tried loading the whole file into a memo box and it fails. No error message just stops as if everything is OK but when I look at the file it should have a '99' as the last record.

I can verify this by opening a DOS box and entering 'Type Apt.Dat'. Scrolls the file to the screen and the last record is in fact '99'.

The last record in the memo box is a 1302 .............

It should give a failure to load message.

Is this a bug?
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig 1.5 Terabyte SSD
3 Terabyte conventional disk space.

lucamar

  • Hero Member
  • *****
  • Posts: 734
Re: Won't load Apt.Dat.
« Reply #1 on: January 08, 2019, 11:07:26 pm »
Sounds more like dying by exhaustion :)

How big is the file? If it's too big the memo may simply load as much as it can and stop there, although IIRC it also should spout some error.

And ... didn't we already discuss this or something like this? (I seriously don't remember)
« Last Edit: January 08, 2019, 11:09:32 pm by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4/FPC 3.0.4 on:
(K)Ubuntu 12..16, Windows XP SP3 (Home/Prof.) and various DOS incarnations.

JLWest

  • Sr. Member
  • ****
  • Posts: 340
Re: Won't load Apt.Dat.
« Reply #2 on: January 08, 2019, 11:28:51 pm »
The file size is 285,986 KB according to Windows Explorer.
By count 7,970,312 Records.

They are not fixed length but have variable size records with some records quite large others small.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig 1.5 Terabyte SSD
3 Terabyte conventional disk space.

lucamar

  • Hero Member
  • *****
  • Posts: 734
Re: Won't load Apt.Dat.
« Reply #3 on: January 08, 2019, 11:38:49 pm »
If by "records" you mean "lines" then it's too big. Remember that the base storage for memos is TStrings and its Count property is a standard (32 bits) Integer, i.e. it can contain a max of 4 MLines. The ~280MiB of data don't help either, because TStrings isn't memory efficient and it can use (depending on the operations performed) as much as four times the assumed capacity.

Humm ... Are you sure we haven't discussed all this already?
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4/FPC 3.0.4 on:
(K)Ubuntu 12..16, Windows XP SP3 (Home/Prof.) and various DOS incarnations.

JLWest

  • Sr. Member
  • ****
  • Posts: 340
Re: Won't load Apt.Dat.
« Reply #4 on: January 09, 2019, 12:19:32 am »
Yea, We have discussed all this and then Handoko wrote a program that generated a 10,000,000 record file into a Memo box. Worked fine. So I thought I would try and load the 7,970,312 Records, seemed reasonable.

I think something isn't right. He loading more than 2,000,000 records that I am trying to do. Should make up for my variable length records. And I think they should load.

I also think there is bug with this. FP should report a failure.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig 1.5 Terabyte SSD
3 Terabyte conventional disk space.

garlar27

  • Hero Member
  • *****
  • Posts: 559
Re: Won't load Apt.Dat.
« Reply #5 on: January 09, 2019, 12:43:11 am »
Humm ... Are you sure we haven't discussed all this already?

For what I saw there are 2 other threads 7,970,312 As it turns out. (on: 2019-01-04, 15:34:24) and Just advise (on: 2018-12-29, 21:32:43). I don't know if there are more...

JLWest

  • Sr. Member
  • ****
  • Posts: 340
Re: Won't load Apt.Dat.
« Reply #6 on: January 09, 2019, 01:03:11 am »
"Yea, We have discussed all this and then Handoko wrote a program that generated a 10,000,000 record file (Text Lines)  into a Memo box. Worked fine. So I thought I would try and load the 7,970,312 (Text Lines) Records, seemed reasonable.

I think something isn't right. He loading more than 2,000,000 records that I am trying to do. Should make up for my variable length records. And I think they should load.

I also think there is bug with this. FP should report a failure."

As stated above. It has been discussed. What I'm asking now is:

Is this a bug. Should it be reported?


If Handoko can generate 10,000,000 records into a memo box but you can't load 7.9 million something is wrong.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig 1.5 Terabyte SSD
3 Terabyte conventional disk space.

josh

  • Hero Member
  • *****
  • Posts: 700
Re: Won't load Apt.Dat.
« Reply #7 on: January 09, 2019, 01:34:08 am »
Hi

I think you hitting the limitation of a memo, the actual amount of memory a memo utilizes can double/triple/quadruple in size during memory maintenance.

You should also notice the memo filling up slows down after a few thousand additions, as the manager is handling larger and larger chunks of memory...

Attached a quick test, that will attempt to fill a memo with 8000000 largest lines one by one,
It will give you a runnign progress on the length of the memo used.

If you open taskmanager and monitor it when you run it; you should see the memory temporarily jump.

I have also created multiple debug modes, so run it in the debug mode; as runnign it in normal or release various checks will be disabled.

Hope it helps

ps. not the most eligent or efficient code. No checks etc....
« Last Edit: January 09, 2019, 01:56:59 am by josh »
Development Installation Lazarus 1.3, FPC 2.7.1,Windows 7/8 32/64, OSX, *nix

Test Environment Lazarus & FPC Trunk on Windows and OSX (Cocoa Mainly on OSX). Testing also Crosscompile windows to OSX.. 
Any posts made from 2015 will be based on Lazarus Trunk.

JLWest

  • Sr. Member
  • ****
  • Posts: 340
Re: Won't load Apt.Dat.
« Reply #8 on: January 09, 2019, 02:51:15 am »
Thanks I'll test it. Mine didn't really slow down until about 5mil and then just a little.

I did find a free editor that can handle very large text files. And it does it with ease. It's called Kainet Editor. It's very basic but fast and a way to check results with large file. Notepad and Notepad++ can't handle the records but Kainet with ease.

I recommend it if anyone is interested.

Thanks Josh
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig 1.5 Terabyte SSD
3 Terabyte conventional disk space.

Handoko

  • Hero Member
  • *****
  • Posts: 2766
  • My goal: build my own game engine using Lazarus
Re: Won't load Apt.Dat.
« Reply #9 on: January 09, 2019, 07:00:00 am »
It seems you misunderstood and didn't get to points I want to tell you on my previous posts.
  • The example I wrote does not load 10 millions items into a TMemo. It generates a text file that contains 10 millions of lines.

  • It scans that text file and loads only some items that passes a filter test into memory, which is about 100,000 items only.

  • For storing that 100,000 items in a TMemo, it consume less than 64 MB memory.

  • My 4 GB RAM computer is not capable to load 10 millions items to TMemo/TListBox. On my test it started to use swap file to hold the data when it reachs 6 millions item on both a TMemo + a TListbox.

  • Scanning the text file that contains 10 millions items and load some of them into memory took about 20 seconds only in my test. But it took more than 5 minutes if I tried to load 5 millions items into the TMemo/TListBox.

  • Lazarus TMemo/TListBox is not capable to store 10 millions items (at least on my computer). It fails because the way it store data, memory fraqmentation, etc.

  • If you still want to load that huge items, you cannot use the default TMemo/TListBox. You have you write your own component. Not many programmers are capable to do it, including me.

The workarounds I suggest are:

For memory friendly
Load a chunk of data into memory, do some processing, store the result, free the data from memory and continue to do the same on the next chunk.

For user friendly
You mentioned you want to load the items into a ListBox, do you show the ListBox to user for selecting some items?
Selecting one or several items from a list box that contains thousands of items is a bad user-experience design. You should separate them into several categories.

Write your own component
It's a possible solution but I do not recommend. Because it is harder you can imagine.
« Last Edit: January 09, 2019, 07:04:41 am by Handoko »

JLWest

  • Sr. Member
  • ****
  • Posts: 340
Re: Won't load Apt.Dat.
« Reply #10 on: January 09, 2019, 07:15:41 am »
After about an hour got an out of memory message and continue abort box.

I think I now know how to write this program.

1. build a double hash file for the Airports, Sea Planes Bases and Heliports. 35,000 plus records. They would look like:

Airport      SRCD     EREC
KPHX      721478   751523
KPDX      751524   762621

Load this into a listbox and allow double click of a airport.

Open the 7.9 million file read to the first record (SRCD) and load to the last record (ERCD) of the airport you are working with. You don't have that much in memory.

When your done write them out.

Pick another airport or close the file and quit. Tricky, may have to close the file and start  all over at the first record again depending on airport picked.

I would like to eliminate the page up and page down buttons and maybe the up and down arrows.  What would be best is to control how far they could move the focus in the list box.

I would need to stop them at the SRCD vector  and EREC  vector.

JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig 1.5 Terabyte SSD
3 Terabyte conventional disk space.

JLWest

  • Sr. Member
  • ****
  • Posts: 340
Re: Won't load Apt.Dat.
« Reply #11 on: January 09, 2019, 07:38:15 am »
@Handako
For user friendly
You mentioned you want to load the items into a ListBox, do you show the ListBox to user for selecting some items?
Selecting one or several items from a list box that contains thousands of items is a bad user-experience design. You should separate them into several categories.


For user friendly Agree: Have to load all of KPDX and then have a working listbox with just the records to be modified. Good call. Putting it back to gatherer will be a little tricky.

I went back and figured out that you weren't loading that many records. Actually, I modified your code to read and load in a memo and I got close. Because at the time I couldn't load the file in any editor I couldn't tell where in the file it quit but I think about 300,000 from the end from what the counters showed. I think I was out of memory so I ordered more memory today. My kid who is something of a guru with computers says it won't help on a 32 bit program.

Looks like a big turn and burn to get to a 64 bit compiler and it may not help.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig 1.5 Terabyte SSD
3 Terabyte conventional disk space.

GetMem

  • Hero Member
  • *****
  • Posts: 3332
Re: Won't load Apt.Dat.
« Reply #12 on: January 09, 2019, 07:54:08 am »
@JLWest

I do not understand why you and others keep insisting on using Memos, Listboxes for processing data. This is ridiculous. You should switch to dynamic arrays, they are fast and can handle much more then 7M records. Loading 7M lines to a memo make no sense anyways, the user cannot process that much information. Lots of bad advice in my opinion.
Although I'm not familiar with the exact requirements, you should try something like this:
Code: Pascal  [Select]
  1. type
  2.   PData = ^TData;
  3.   TData = record
  4.     FLine: String;
  5.     FIndex: Integer;
  6.     //other data if needed
  7.   end;
  8.  
  9. var
  10.   Data: array of TData;
  11.   DataCondition: array of TData;
  12.  
  13. function LoadDataFromFile(const AFileName: String): Boolean;
  14. var
  15.   DataFile: TextFile;
  16.   Len: Integer;
  17.   Line: String;
  18. begin
  19.   Result := True;
  20.   AssignFile(DataFile, AFileName);
  21.   try
  22.     Reset(DataFile);
  23.     while not eof(DataFile) do
  24.     begin
  25.       Readln(DataFile, Line);
  26.       Len := Length(Data);
  27.       SetLength(Data, Len + 1);
  28.       Data[Len].FLine := Line;
  29.       Data[Len].FIndex := Len;
  30.     end;
  31.     CloseFile(DataFile);
  32.    except
  33.      Result := False;
  34.    end;
  35. end;
  36.  
  37. procedure TForm1.Button1Click(Sender: TObject);
  38. var
  39.   Len, I: Integer;
  40. begin
  41.   Data := nil;
  42.   DataCondition := nil;
  43.   if not LoadDataFromFile(ExtractFilePath(Application.Exename) + 'klax.dat') then
  44.   begin
  45.     MessageDlg('Cannot load file! ', mtError, [mbOk], 0);
  46.     Exit;
  47.   end;
  48.   //for some reason you wish to start from the last record, I saw it in one of your previous thread
  49.   for I := High(Data) downto Low(Data) do
  50.   begin
  51.     //check for some conditions, if the conditions are met copy the record/line to another array DataCondition(Listbox2,3 equivalent)
  52.    //you can define more arrays/records if needed)
  53.     Len := Length(DataCondition);
  54.     SetLength(DataCondition, Len + 1);
  55.     DataCondition[Len].FLine := Data[I].FLine;
  56.     DataCondition[Len].FIndex := Data[I].FIndex; //keep the original index for later reference, maybe you wish to delete from the file
  57.   end;
  58.   //do more stuff here, if you have to show to the user informations from DataCondition,
  59.   //you can add it to a listbox or even better to a virtualtreeview
  60. end;
  61.  

« Last Edit: January 09, 2019, 08:34:04 am by GetMem »

wp

  • Hero Member
  • *****
  • Posts: 5294
Re: Won't load Apt.Dat.
« Reply #13 on: January 09, 2019, 09:11:48 am »
@JLWest

I do not understand why you and others keep insisting on using Memos, Listboxes for processing data. This is ridiculous. You should switch to dynamic arrays, they are fast and can handle much more then 7M records. Loading 7M lines to a memo make no sense anyways, the user cannot process that much information. Lots of bad advice in my opinion.
Although I'm not familiar with the exact requirements, you should try something like this:
Absolutely. However, in this sample code, the array length is incremented by 1. This means that the array must be copied to another place whenever a new line is read which will become very slow after some time in case of large files. I would pre-allocate a larger block of array items and use a line counter to set the final dimension at the end:

Code: Text  [Select]
  1. type
  2.   PData = ^TData;
  3.   TData = record
  4.     FLine: String;
  5.     FIndex: Integer;
  6.     //other data if needed
  7.   end;
  8.  
  9. function LoadDataFromFile(const AFileName: String): Boolean;
  10. const
  11.   BLOCK_SIZE = 1000;  // Experiment with this value
  12. var
  13.   DataFile: TextFile;
  14.   Line: String;
  15.   Counter: Integer = 0;
  16. begin
  17.   Result := True;
  18.   AssignFile(DataFile, AFileName);
  19.   try
  20.     Reset(DataFile);
  21.     while not eof(DataFile) do
  22.     begin
  23.       Readln(DataFile, Line);
  24.       if Counter mod BLOCK_SIZE = 0 then
  25.         SetLength(Data, Length(Data) + BLOCK_SIZE);
  26.       Data[Counter].FLine := Line;
  27.       Data[Counter].FIndex := Counter;
  28.       inc(Counter);
  29.     end;
  30.     CloseFile(DataFile);
  31.     SetLength(Data, Counter);
  32.    except
  33.      Result := False;
  34.    end;
  35. end;
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

GetMem

  • Hero Member
  • *****
  • Posts: 3332
Re: Won't load Apt.Dat.
« Reply #14 on: January 09, 2019, 09:47:20 am »
@wp
Quote
This means that the array must be copied to another place whenever a new line is read which will become very slow after some time in case of large files. I would pre-allocate a larger block of array items and use a line counter to set the final dimension at the end
OK. Good point :), however after running a few tests the results are not so convincing. For 10M line text file(Lazarus 32bit, FPC 3.0.4, win10):
My method: 28125 ms
Your method: 27530 ms
Your version is still better but not by much. Please run a few tests yourself, I'm curious what the results are at your side.


« Last Edit: January 09, 2019, 09:48:58 am by GetMem »