Recent

Author Topic: Heap overflow (203) errors with SQLite starting with Laz 2.0.10  (Read 584 times)

bobonwhidbey

  • Sr. Member
  • ****
  • Posts: 430
    • Double Dummy Solver - free download
These overflow errors only cropped up after switching to 2.0.10.  I was able to eliminate some of my problems using
Code: Pascal  [Select][+][-]
  1.     UniDirectional:=True;  
at least when using the TSQLQuery Next function.

However, when using the Last function or the Locate function I'm still getting the heap errors. Is this a bug in 2.0.10 or is there some solution. Formerly it wasn't even unnecessary to use UniDirectional, although it seems like a good approach when processing a table from top to bottom.

« Last Edit: August 09, 2020, 08:02:55 pm by bobonwhidbey »
Win10 64-bit / Lazarus 32-bit 2.0.10 / FPC 3.2.0

jamie

  • Hero Member
  • *****
  • Posts: 3518
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #1 on: August 09, 2020, 09:12:58 pm »
You need to study your code better..

Things like that usually happen when you are in a event and while you are in that event you do something that triggers the same event and thus while doing this it eats up the stack until  you run out..
 :o
The only true wisdom is knowing you know nothing

howardpc

  • Hero Member
  • *****
  • Posts: 3517
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #2 on: August 09, 2020, 09:40:59 pm »
@jamie
This is a heap overflow, not a stack overflow.

ccrause

  • Sr. Member
  • ****
  • Posts: 303
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #3 on: August 10, 2020, 11:35:39 am »
Could be a memory leak.  Try compiling your project with the heaptrc unit and line info enabled (-ghl compiler command line option, or go to Project Options | Compiler options | Debugging and select Display line numbers... and Use Heaptrc unit). This wiki article and FPC reference contains more information.

Obviously when you run your project until it crashes there will be lots of memory leaks, so try and test small portions of your program, then close it and check for memory leaks.  Of course there may not be a formal memory leak, it could just be a case of buffering so much data that the OS runs out of memory.

Can you correlate the problem with the size of the table returned by the query?

Thaddy

  • Hero Member
  • *****
  • Posts: 10446
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #4 on: August 10, 2020, 11:41:11 am »
Does a console application accessing the same database work? Because that is easier to analyse.
The core code is pure FPC, not Lazarus.
« Last Edit: August 10, 2020, 11:50:59 am by Thaddy »
When you ask a question that is actually answered in the documentation, you are either lazy or a moron.

jamie

  • Hero Member
  • *****
  • Posts: 3518
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #5 on: August 10, 2020, 01:32:40 pm »
@jamie
This is a heap overflow, not a stack overflow.

Sorry about that, but it most likely a uncontrolled memory allocation in a loop.

I think using heaptrc may help this greatly.
The only true wisdom is knowing you know nothing

bobonwhidbey

  • Sr. Member
  • ****
  • Posts: 430
    • Double Dummy Solver - free download
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #6 on: August 10, 2020, 05:46:26 pm »
I switched to 2.0.10 when I got a new PC. The new PC has only 8 GM memory while the former PC had 16. The problem isn't a memory leak. In fact I'm using a heck of a lot of memory in the query - very likely all the data in the query could be 16GB.

It's easy to get into lazy/wasteful techniques when you have "unlimited" resources. It always amazed my how SQLite could handle an unlimited amount of data without batting an eye! Now that my RAM is more limited I'm discovering that there is a limit.

It seems that all of the data held in a query is kept in RAM; or more accurately - all records already accessed in the Query are held in RAM. I get the impression that SQLite pulls into RAM 20 records at a time as it processes a queue, so RAM gradually gets filled, unless UniDirectional := true. Perhaps the Locate function actually brings into RAM 20 record blocks until the desired record is located, thus filling up RAM.
Win10 64-bit / Lazarus 32-bit 2.0.10 / FPC 3.2.0

jamie

  • Hero Member
  • *****
  • Posts: 3518
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #7 on: August 10, 2020, 10:51:16 pm »
Must be the reason whey they "Lite" is attached to the name?

In any case,  maybe there is a switch to use files for cache instead..

Or you could purge it now and then ..
The only true wisdom is knowing you know nothing

bobonwhidbey

  • Sr. Member
  • ****
  • Posts: 430
    • Double Dummy Solver - free download
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #8 on: August 10, 2020, 11:31:12 pm »
Perhaps LITE is the problem. My query (inner joining some table records) returns a dataset of roughly 500 byes x 100,000 records, or 50 MB.  I have some work-around options to use a much smaller record for that query. I'll try that approach before thinking about another DB engine.
Win10 64-bit / Lazarus 32-bit 2.0.10 / FPC 3.2.0

jamie

  • Hero Member
  • *****
  • Posts: 3518
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #9 on: August 10, 2020, 11:46:50 pm »
The small records in that count is most likely fragmented the heap manager..
Maybe you should write a class with a single chunk of memory that can grow and then partition that memory with your records using a managed table of indexes with in the chunk of memory.

 Create large chunks of allocated memory as contiguous memory..
The only true wisdom is knowing you know nothing

Warfley

  • Sr. Member
  • ****
  • Posts: 302
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #10 on: August 11, 2020, 12:12:16 am »
Maybe this is interesting: https://sqlite.org/malloc.html

SQLite does pretty fancy memory allocation to be fast and it is highly configurable, maybe there is something misconfigured?

bobonwhidbey

  • Sr. Member
  • ****
  • Posts: 430
    • Double Dummy Solver - free download
Re: Heap overflow (203) errors with SQLite starting with Laz 2.0.10
« Reply #11 on: August 11, 2020, 01:42:09 am »
Thanks everyone. I can see there's a lot more I can learn about SQLite.
Win10 64-bit / Lazarus 32-bit 2.0.10 / FPC 3.2.0

 

TinyPortal © 2005-2018