Recent

Author Topic: [SOLVED] SQLite - I've created a really slow implementation. How to speed up?  (Read 6179 times)

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: SQLite - I've created a really slow implementation. How to speed up?
« Reply #15 on: September 20, 2017, 09:21:33 am »
It's better than that. I am its developer! :-) (see profile URL).
Nice, looks good. (although I had some trouble with a second run getting an empty stringgrid)

In that case things become more easy.
You know the MD5File is fast (the way you do it).

But ASerge also already showed code where a lot of records could be written to SQLite in 300 ms.

The only thing I can imagine is that the problem is combining them. When you do a FindFirst/FindNext and MD5File and combine that with writing to the database, it really puts a strain on the harddrive (heads). You can test this by writing to the database AFTER collecting all the file-information. Begin with just adding the loop ASerge wrote at the end of your run. If that is also fast you could remove the StringGrid and replace it with a simple array or collection in which you first gather all the information and then write it to the database (don't forget to put the Commit at the end and not per record, btw. did you try that yet in your current tries?).

Gizmo

  • Hero Member
  • *****
  • Posts: 831
Re: SQLite - I've created a really slow implementation. How to speed up?
« Reply #16 on: September 20, 2017, 02:38:09 pm »
RVK

Once again, you are the man!!

Yes, I moved the commit out to the end (basically the last thing to happen before the DBGrid is displayed), so it is only executed once per 1K files. And sure enough, it was faster than I could blink. In less than 1 second, the files were traversed, hashed and listed.

If I then revert it back to a commit per file, and it takes 38 seconds with hashing, or 36 without hashing (so if I just comment out the hash line and get it to write a random string).

So it seems there is the asnwer - SQLIte is fast, as long as you're careful with your commit frequency. As you say RVK, one per 1K queries is perhaps wise.

 

TinyPortal © 2005-2018