### Bookstore

 Computer Math and Games in Pascal (preview) Lazarus Handbook (preview only)

### Recent

#### JLWest

• Hero Member
• Posts: 545
« on: December 30, 2018, 02:32:43 am »

I'm reading 8,000,000+ records out of a text file into a listbox. But I only need to examine 78,000 plus or minus. The rest I just throw on the floor.
Of the 78,000 record I need 3 pieces of data.

1 ICAO Airport  Example: KLAX, EGGE KJFK
2 Latitude of the airport.
3 Longitude of the airport.

Several gotcha along the way.
The Lat and Lon may be in a "1302 data_lat" or a 1302 data_lon.
They would look like this in the file.
1302 data_lat 32.15477889
1302 data_lon -14.25897456
Problem is there are version of of this file so the 1302 data_lat and 1302 dat_lon  might be there but empty.

So you try and get the location of the Tower. It may have a Lat/Lon and may not.

Finally you go for the center line of one of the runways. That is required. Can be a couple miles off in really big airports. But if you land the plane 2 miles east of Gatwick on a city street who's gonna complain.

Now here is what i'm doing:

Step 1: (Done)
I read the 8 million saving off in listboxs1
RecType   1    Airport  ICAO,
14   Tower
16   Seaplane Base
100  Runway(s)

The last record in the file is 99.

So starting at the last record I read backwards looking for a Record type 1, 16 or 17 loading them into listbox3 as i go. Once loaded I can parse out my data based on the record type format a record to go into Listbox2 that looks like:

KLAX 45.254879 -124.24587

Should be about 35,000 of those but with them I can calculate anywhere to anywhere.

Step 2: (Trouble)

After Processing a screen of data from listbox3 to listbox2 I would like to delete them from listbox1.
I either get out of bounds error or it wont delete them.

Step 3: (Done Untested)

Write Listbox2 to a file.

JLWEST
Lazuras ver 2.0.2
FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

#### Trenatos

• Hero Member
• Posts: 508
• Software developer - Open source contributor
« Reply #1 on: December 30, 2018, 02:43:03 am »
Code: Pascal  [Select]
1. ListBox.Clear;

Do you really need to display most of this on-screen though?

#### JLWest

• Hero Member
• Posts: 545
« Reply #2 on: December 30, 2018, 02:49:19 am »
No I don't need to display it on screen. but, I can see what going on a little easier.

As far as listbox.clear. Which one. I have three. Listbox3 I clear before every load. Process the load into Listbox2 and then clear listbox3.

Listbox2 is where I store my results and listbox1 has 79,000 I have to examine.
JLWEST
Lazuras ver 2.0.2
FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

#### dbannon

• Hero Member
• Posts: 587
« Reply #3 on: December 30, 2018, 03:23:26 am »
Not sure what your question is, is it about getting an error when clearing ListBox1 ?

I guess its possible you have exceeded the capacity of the ListBox in some way ?  I doubt the designers expected anyone to load that many lines into it and perhaps they neglected to prevent that from happening ? 78K records, 100 bytes each record, 8Meg ?  maybe .....

It should work OK and if that really is the problem, I'd see it as a reportable bug but its still a lot of data in a visual component....

I think loading into a TStringList would be a heap faster and we know for sure it can handle that sort of data.  Might be a good place to start.

Davo

EDIT - just re-read the question, you initially put all 8Million records into the ListBox ? Oh. wow ? These LCL components are pretty good arn't they ?
« Last Edit: December 30, 2018, 03:38:54 am by dbannon »
Lazarus 1.8, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng

#### lucamar

• Hero Member
• Posts: 1814
« Reply #4 on: December 30, 2018, 03:25:17 am »
Step 2: (Trouble)

After Processing a screen of data from listbox3 to listbox2 I would like to delete them from listbox1.
I either get out of bounds error or it wont delete them.

If I were you I would delete the item from ListBox1 as soon as you have read it. Something like this (seudo-)code:

Code: Pascal  [Select]
1.   repeat
2.     AStr := Listbox1.Items[Listbox1.Items.Count-1];
3.     ListBox.Items.Delete(Listbox1.Items.Count-1)
4.     if GoesToList3(AStr) then
6.   until ListBox.Items.Count = 0;
7.

You get the idea, I'm sure. It has the advantage that you'll not end up consuming all the memory

ETA: dbannon's idea of using a TStringList is good too, and it may speed up things by avoiding the visual updates of the list box.
« Last Edit: December 30, 2018, 03:31:09 am by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!)
Lazarus 1.8.4 & 2.0.2 w/FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

#### Trenatos

• Hero Member
• Posts: 508
• Software developer - Open source contributor
« Reply #5 on: December 30, 2018, 03:26:10 am »
Yeah, maybe I misunderstood what you're asking..

Do you want to delete a single item from the listbox? Or the entire thing?

What's the end goal of your processing?

#### JLWest

• Hero Member
• Posts: 545
« Reply #6 on: December 30, 2018, 04:59:42 am »
The goal is to process all the record and write out a file with three fields:

KLAX 44.125891 -12.45122

I just got this working I think.

Deleting records in Listboxz1 as there are stored in Listbox3 would have been the way to go.
Easier to write.
JLWEST
Lazuras ver 2.0.2
FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

#### jamie

• Hero Member
• Posts: 1656
« Reply #7 on: December 30, 2018, 04:30:51 pm »
list boxes and the like are limited of what They can hold...

You are better off filtering the unwanted as you load them.

Plus they slow down a lot when handling large data.

#### JLWest

• Hero Member
• Posts: 545
« Reply #8 on: December 30, 2018, 05:37:39 pm »
I do filter from a file of 8 million down to 104,000 on the initial on listbox1.

I think I could load into a list and gain speed. But I like the visual defect, something happening on the screen that tells me and the user that it's not in a locked loop.
JLWEST
Lazuras ver 2.0.2
FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
3952 GB (1.5 SSD)

• Hero Member
• Posts: 1814