Still hitting head against wall

[eny]

Nah dont work like that.

wish it did. here's pseudocode of what needs to be done.

What step of the process takes so long?
And what is done in this 3msec?

You could work with in memory hash tables.

it's not the size of the database it's thenumber of transactions. try hitting sqlite with 3 GB of transactions and see what happens 

There are few relational databases that run 3Gig transactions in less than 3 msecs so in your case indeed that might not be the solution.

[update on hashlist]
Quick test on XP: 3M hash entries.
2M lookups take a little less than 1 second.

[liewald]

the full process of read lookup and write used to take 3msec. it's the lookups that are killing it now because indexes cannot be established all calls to lookup are linear searches. kills the performance.

[ludob]

Have you tried using hash lists or any other specialised list?
I have written a double sorted string list which is a good compromise between insert speed and lookup speed. A small test program shows that such a list populated with 500000 different random string values (2-17 characters) will do 100000 lookups per second on a not so fast laptop. Loading the list with 500000 unique values from an array of 1 million random values takes 9 seconds. This includes the test to see if the value was already in the list before storing it.
Let me know if you are interested in this unit.

[liewald]

worth trying.

[eny]

the full process of read lookup and write used to take 3msec. it's the lookups that are killing it now because indexes cannot be established all calls to lookup are linear searches. kills the performance.

OK, clear.
Setting up an in mem db with the approx. 1M records takes a couple of seconds obviously.
But lookups are in the microsecond range.
I can do approx. 40K lookups in a second with SQLite i.e. 0,025 msec per lookup!!

[ludob]

Attached is the test program I used.

TDSStringList (unit doublesortedlist) is used much like a TStringList with the difference that all strings are sorted when added. With every string you can associate a pointer that can point to anything: a record, an object, .... TDSStringList destroys all strings when destroyed but you need to provide a RemoveCallback procedure or  RemoveCallbackMethod method if you want TDSStringList to destroy the attached object for you.

TDSStringList.FindEntry (const s: string; var pentry: PDSStringListEntry ): boolean; is used to locate a string. It returns true if the string is found. pentry.value is the string found and pentry. p is the pointer associated with the string. If the string is not found, pentry will point to the string that is just before the string searched. Useful for fuzzy searches.

The string sort order is the normal ansi sort order and can be overridden by setting the CompareCallback property with your own string compare function.

[eny]

Or use the out of the box TFPHashList or TFPHashObjectList (in contnrs).

[ludob]

Or use the out of the box TFPHashList or TFPHashObjectList (in contnrs).

Modified my little test program to use TFPHashList and it is roughly 10-13 times faster than TDSStringList. So if you just need the lookup feature then TFPHashList is the better choice with more than 1M lookups per second.

[avra]

Looks like they found then bug and fixed it in 2.7.1 but just cant get 2.7.1 to work and no 2.6 patch was released.

CodeTyphon v2.70 works out of the box with Lazarus 1.1 and FPC 2.7.1 from 19.6.2012 and SVN 37429 CT.

[liewald]

Ok I used a TFPStringHashTable to hold the lookup values and populated it using virtually the same routines as the bufdataset.

Process the 21 files and created output file in about  45seconds... WOW

love saving millisecs

thanks Ludob for the heads up.

would still like to get the bufdatset working for some other stuff though!

[BigChimp]

would still like to get the bufdatset working for some other stuff though!

Hello? Are you receiving me?

BTW, if it works with FPC 2.7, you could ask for a backport to 2.6.1 (if it isn't in the pipeline already), e.g. via raising a bugtracker issue. Marcov would probably do the backporting...

Will insytall a local copy on mysql and tect but not hopefull

I'd also try getting an FPC 2.7.1 version in (much easier if you don't need Lazarus - FPC revision 21757 compiles fine... it's just Lazarus that won't work well... at least in my setup).

Either use fpcup or get it using subversion yourself... fpcup is the easiest way though.

You could use e.g. ludob's and my fpcup tool (see 3rd party announcements for details) to do a parallel installation of FPC trunk+Lazarus in c:\development\ (on Windows) or ~/lazarus and ~/fpc.
Note: you'll have to specify the --fpcurl parameter as fpcup defaults to fixes 2.6/2.6.1; see fpcup --help for details.
Works on Linux and Windows.

[liewald]

Receiving loud and clear.....

Busy running about 40 analyses I have cued to be followed by 150 3d brain reconstructions from MRI Scans and 80 other MRI analyses. Also building my new 24 processor thread 96 GB linux processing server.

Lifes a bitch aint it

Dave

[BigChimp]

As long as you're having fun

[liewald]

Fun is relative

much rather be playing about doing 3D art..... saving humanity takes so much bloody time!

[eny]

Ok I used a TFPStringHashTable to hold the lookup values and populated it using virtually the same routines as the bufdataset.

Process the 21 files and created output file in about  45seconds... WOW

love saving millisecs

thanks Ludob for the heads up.

Using hash classes, wow, I wish I would have thought of that...