Recent

Author Topic: Just advise  (Read 911 times)

JLWest

  • Sr. Member
  • ****
  • Posts: 396
Just advise
« on: December 30, 2018, 02:32:43 am »

I'm reading 8,000,000+ records out of a text file into a listbox. But I only need to examine 78,000 plus or minus. The rest I just throw on the floor.
 Of the 78,000 record I need 3 pieces of data.

1 ICAO Airport  Example: KLAX, EGGE KJFK
2 Latitude of the airport.
3 Longitude of the airport.

Several gotcha along the way.
The Lat and Lon may be in a "1302 data_lat" or a 1302 data_lon.
They would look like this in the file.
1302 data_lat 32.15477889
1302 data_lon -14.25897456
Problem is there are version of of this file so the 1302 data_lat and 1302 dat_lon  might be there but empty.

So you try and get the location of the Tower. It may have a Lat/Lon and may not.

Finally you go for the center line of one of the runways. That is required. Can be a couple miles off in really big airports. But if you land the plane 2 miles east of Gatwick on a city street who's gonna complain.

Now here is what i'm doing:

Step 1: (Done)
I read the 8 million saving off in listboxs1
RecType   1    Airport  ICAO,
              14   Tower
              16   Seaplane Base
              17   Helicapoter pad
             100  Runway(s)

The last record in the file is 99.

So starting at the last record I read backwards looking for a Record type 1, 16 or 17 loading them into listbox3 as i go. Once loaded I can parse out my data based on the record type format a record to go into Listbox2 that looks like:

KLAX 45.254879 -124.24587

Should be about 35,000 of those but with them I can calculate anywhere to anywhere.

Step 2: (Trouble)

After Processing a screen of data from listbox3 to listbox2 I would like to delete them from listbox1.
I either get out of bounds error or it wont delete them.

Step 3: (Done Untested)

Write Listbox2 to a file.
   




 

JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

Trenatos

  • Sr. Member
  • ****
  • Posts: 493
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: Just advise
« Reply #1 on: December 30, 2018, 02:43:03 am »
Code: Pascal  [Select]
  1. ListBox.Clear;

Do you really need to display most of this on-screen though?

JLWest

  • Sr. Member
  • ****
  • Posts: 396
Re: Just advise
« Reply #2 on: December 30, 2018, 02:49:19 am »
No I don't need to display it on screen. but, I can see what going on a little easier.

As far as listbox.clear. Which one. I have three. Listbox3 I clear before every load. Process the load into Listbox2 and then clear listbox3.

Listbox2 is where I store my results and listbox1 has 79,000 I have to examine.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

dbannon

  • Hero Member
  • *****
  • Posts: 520
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Just advise
« Reply #3 on: December 30, 2018, 03:23:26 am »
Not sure what your question is, is it about getting an error when clearing ListBox1 ?

I guess its possible you have exceeded the capacity of the ListBox in some way ?  I doubt the designers expected anyone to load that many lines into it and perhaps they neglected to prevent that from happening ? 78K records, 100 bytes each record, 8Meg ?  maybe .....

It should work OK and if that really is the problem, I'd see it as a reportable bug but its still a lot of data in a visual component....
 
I think loading into a TStringList would be a heap faster and we know for sure it can handle that sort of data.  Might be a good place to start.

Davo

EDIT - just re-read the question, you initially put all 8Million records into the ListBox ? Oh. wow ? These LCL components are pretty good arn't they ?
« Last Edit: December 30, 2018, 03:38:54 am by dbannon »
Lazarus 1.8, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng

lucamar

  • Hero Member
  • *****
  • Posts: 1072
Re: Just advise
« Reply #4 on: December 30, 2018, 03:25:17 am »
Step 2: (Trouble)

After Processing a screen of data from listbox3 to listbox2 I would like to delete them from listbox1.
I either get out of bounds error or it wont delete them.

If I were you I would delete the item from ListBox1 as soon as you have read it. Something like this (seudo-)code:

Code: Pascal  [Select]
  1.   repeat
  2.     AStr := Listbox1.Items[Listbox1.Items.Count-1];
  3.     ListBox.Items.Delete(Listbox1.Items.Count-1)
  4.     if GoesToList3(AStr) then
  5.       AddToList3(AStr);
  6.   until ListBox.Items.Count = 0;
  7.  

You get the idea, I'm sure. It has the advantage that you'll not end up consuming all the memory :)

ETA: dbannon's idea of using a TStringList is good too, and it may speed up things by avoiding the visual updates of the list box.
« Last Edit: December 30, 2018, 03:31:09 am by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4/FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3 (Home/Prof.), various DOSes.

Trenatos

  • Sr. Member
  • ****
  • Posts: 493
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: Just advise
« Reply #5 on: December 30, 2018, 03:26:10 am »
Yeah, maybe I misunderstood what you're asking..

Do you want to delete a single item from the listbox? Or the entire thing?

What's the end goal of your processing?

JLWest

  • Sr. Member
  • ****
  • Posts: 396
Re: Just advise
« Reply #6 on: December 30, 2018, 04:59:42 am »
The goal is to process all the record and write out a file with three fields:

KLAX 44.125891 -12.45122

I just got this working I think.

Deleting records in Listboxz1 as there are stored in Listbox3 would have been the way to go.
Easier to write.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

jamie

  • Hero Member
  • *****
  • Posts: 1291
Re: Just advise
« Reply #7 on: December 30, 2018, 04:30:51 pm »
list boxes and the like are limited of what They can hold...

You are better off filtering the unwanted as you load them.

Plus they slow down a lot when handling large data.

JLWest

  • Sr. Member
  • ****
  • Posts: 396
Re: Just advise
« Reply #8 on: December 30, 2018, 05:37:39 pm »
I do filter from a file of 8 million down to 104,000 on the initial on listbox1.

I think I could load into a list and gain speed. But I like the visual defect, something happening on the screen that tells me and the user that it's not in a locked loop.
JLWEST 
 FPC 3.0.4, Lazarus IDE v1.8.2 Windows 10 Pro
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
500 GB SSD
3 Terabyte conventional disk space.

lucamar

  • Hero Member
  • *****
  • Posts: 1072
Re: Just advise
« Reply #9 on: December 30, 2018, 05:45:22 pm »
I think I could load into a list and gain speed. But I like the visual defect, something happening on the screen that tells me and the user that it's not in a locked loop.

If that's all, use a TProgressBar! :D
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4/FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3 (Home/Prof.), various DOSes.