extreamly slow text file loading (55,000 lines text) into listbox

[Vincent Snijders]

Ah, that buffer is on the stack, 1 MB on the stack may be too large.

Make the buffer a global variable.

[luthepa1]

Ah, yes!  Good point.  Ok on my windows XP side I ran the app again but commented out the try.. finally block and commented out the BeginUpdate and EndUpdate.  It open the text file in 25 seconds.  Only 5 sec slower.   So I am convinced its something else.  Maybe I need to change compiler options on Ubuntu?  Or maybe windows has better caching or something?

[luthepa1]

Ah, ok.  Global.

Also out of interest I tried a LoadFromFile funciton and it can load the text file in 21sec.  Slight improvement.  Considering the finished app will be loading many 4MB text files (up to 200MB) a 20-25sec time to load 4MB is still too slow for me but all the other code I had been trying to speed things up should probably work in that regards.

So I guess the remaining question for this topic is, why is it so slow on Ubuntu 8.1 x64?  If anyone can answer this you will be the king!

Thanks Guys.

[luthepa1]

Just tested on a Ubuntu 8.1 x32 system I have.  Same issue.

[Vincent Snijders]

Listboxes are not designed to handle 1000's of items and MB's of data.

[luthepa1]

HOLY CRAP!!! I FIXED IT!

It was Lazarus 0.9.26 and possible a gtk combination.

What I did was to perform complete removal of the Lazarus that I installed from Synaptic in Ubuntu 8.1 x64.  I had Lazarus 0.9.26 installed with fpc 2.2.2.  Removed Lazarus and fpc from synaptic (purge).  Then in terminal type "sudo apt-get autoremove" to remove all the fpc dependencies.

Then I followed this post from this forum. http://forum.lazarus.freepascal.org/index.php/topic,5647.0.html

But instead of downloading the version in post I downloaded the latest obviously which was ;
fpc-2.2.5-20090320.x86_64.rpm
fpc-src-2.2.5-20090320.x86_64.rpm
lazarus-0.9.27.19037-20090320.x86_64.rpm

Also these rpms are the GTK2 builds, so after creating the deb detailed in the post (link above) I did not have to worry about rebuilding lazarus to GTK2.  So yeah, once you got the rpm packages, convert to deb with alien then install.  Then when loading Lazarus I got message that the fpcscr/2.2.5 didn't look correct.  Clicked OK and all is working.

Now the good news!

Using the same demo code for the purpose of troubleshooting i got a incredible nice surprise.  My app loaded all the 55,000+ lines into the listbox in 3 seconds!!!!

On windows I am getting 30seconds and previously on Ubuntu with the older Lazarus 0.9.26 /w fpc2.2.2 I still had nothing loaded after 11mins.  So I am not sure whether its a bug fix in 0.9.27 /w fpc 2.2.5 or GTK2 support now, or both.  But I am happy now that I got it loading in 3 seconds.  Now to go back to my original log reviewer app I am building to see if that works.

Thanks for your help guys.