Lazarus

Programming => Databases => Topic started by: Zvoni on March 26, 2019, 06:11:58 am

Title: [SOLVED] This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 26, 2019, 06:11:58 am
Hi folks,

close to deleting FPC/Lazarus. >:D >:D >:D

Ubuntu18.10-64bit/FPC304/Laz200

Library with exported functions.
First call to the library creates a Thread, which takes over as MainThread (because of CheckSynchronize).

This FakeMainthread creates a/many childthreads. Everything woking so far.
The moment i add DB-Components to the Childthreads, everything goes south.

If i use MySQL in the TSQLConnector, it works (with one thread! Many threads it still goes haywire)
If i use SQLite3 i get the Error-MEssage
"Inconsistency detected by ld.so: dl-minimal.c: 126: realloc: Assertion `ptr == alloc_last_block' failed!"
I found exactly one thread in the forum having the error-message. And the discussion there didn't help.

cthreads is the first Unit in the libraray.lpr
and i don't use cmem, because that one doesn't work for me (heaptrc always saying no Memory leaks), and i get a lot of corrupted memory from cmem.

Help?

EDIT: I forgot: The error occurs after connection has been established (connection.connected returns true), on the first try to grab any information from the database (Query.Open, Connection.GetTableNames etc.)

EDIT2: Before anyone asks: YEs, i have successfully worked with SQLite under FPC304, so i don't understand what's happening here, since i don't do anything different than before

EDIT3: And yes: Each thread has its own DB-Conn, Trans and Query! I'm not mixing them, and the threads/db-components don't have to talk with each other.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Thaddy on March 26, 2019, 07:38:21 am
and i don't use cmem, because that one doesn't work for me (heaptrc always saying no Memory leaks), and i get a lot of corrupted memory from cmem.
That is because you overlooked  that heaptrc IS a memory manager by itself, so if you hook cmem, heaptrc won't work anymore.
You can safely use heaptrc and if no leaks are detected change it to cmem. That's what heaptrc is designed for.
Just to be sure: you should let the compiler insert heaptrc and not put it in any uses clause.

The above problems can be wholly caused by mixing two different memory managers, so try first heaptrc without cmem and then with just cmem and without the heaptrc option set.
If there is still memory corruption we need to look further.

The reason I suspect this is the case is the following:
When the compiler inserts heaptrc it does so *before* anything else is done and heaptrc already allocates memory by itself to be able to register all memory allocations/de-allocations. (that's why it is done by the compiler! and not through a uses clause)
That memory is lost when you subsequently set a different memory manager: acute memory corruption.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 26, 2019, 09:07:04 am
and i don't use cmem, because that one doesn't work for me (heaptrc always saying no Memory leaks), and i get a lot of corrupted memory from cmem.
That is because you overlooked  that heaptrc IS a memory manager by itself, so if you hook cmem, heaptrc won't work anymore.
You can safely use heaptrc and if no leaks are detected change it to cmem. That's what heaptrc is designed for.
Just to be sure: you should let the compiler insert heaptrc and not put it in any uses clause.
No heaptrc in Uses-clause. I added it in Project-Options-Debugger (Dwarf2)
And i've removed cmem from uses-clause a "long time ago", but i admit: didn't know that you can't/shouldn't use those two together.
Maybe put that on the HeapTrc-Wiki: DON'T USE IT WITH ANY OTHER MEMORY-MANAGER.
Quote
The above problems can be wholly caused by mixing two different memory managers, so try first heaptrc without cmem and then with just cmem and without the heaptrc option set.
If there is still memory corruption we need to look further.

The reason I suspect this is the case is the following:
When the compiler inserts heaptrc it does so *before* anything else is done and heaptrc already allocates memory by itself to be able to register all memory allocations/de-allocations. (that's why it is done by the compiler! and not through a uses clause)
That memory is lost when you subsequently set a different memory manager: acute memory corruption.

EDIT: Another thing i don't understand: The Error-Message i'm getting implies a problem with the linker.
But both, the library and the Test-Program compile without hitches, and the linker throws the error, when it reaches the point in the code as described above.
Funny thing: only with SQLite. A single thread accessing MySQl works (2/more threads accessing MySQL is another can of worms).

The library is accessed during runtime via the exported functions (lib is not statically linked into the main program)
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Cyrax on March 26, 2019, 11:39:21 am
Open a terminal; issue these commands and check how high is the stack size set in Ubuntu system.

Quote
#soft limits for a process/thread
ulimit -a
#hard limits for a process/thread
ulimit -Ha

Then set the soft limit of a stack size for reasonable size eg. 512 (ulimit -s 512) and try to re-execute your program.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Thaddy on March 26, 2019, 11:48:36 am
I have tried to replicate your problem on win64 and Raspbian, but I used 3.2.0 and 3.3.1 (from today).
I have no problem using sqlite3 in any way, whatever memory manager I use.
Note that the Sqlite3 bindings in 3.2.0 and trunk have changed a bit to support threading in a better way.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 26, 2019, 12:12:29 pm
I have tried to replicate your problem on win64 and Raspbian, but I used 3.2.0 and 3.3.1 (from today).
I have no problem using sqlite3 in any way, whatever memory manager I use.
Note that the Sqlite3 bindings in 3.2.0 and trunk have changed a bit to support threading in a better way.
Thaddy,
thx for that information. I'll postpone on SQLite-use until second half of the year then, when FPC320 becomes officially available
(not familiar with building from svn/trunk/wherever)
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 26, 2019, 12:14:28 pm
Open a terminal; issue these commands and check how high is the stack size set in Ubuntu system.

Quote
#soft limits for a process/thread
ulimit -a
#hard limits for a process/thread
ulimit -Ha

Then set the soft limit of a stack size for reasonable size eg. 512 (ulimit -s 512) and try to re-execute your program.
thx, will try that and report back
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Cyrax on March 26, 2019, 01:43:20 pm
I have tried to replicate your problem on win64 and Raspbian, but I used 3.2.0 and 3.3.1 (from today).
I have no problem using sqlite3 in any way, whatever memory manager I use.
Note that the Sqlite3 bindings in 3.2.0 and trunk have changed a bit to support threading in a better way.
Thaddy,
thx for that information. I'll postpone on SQLite-use until second half of the year then, when FPC320 becomes officially available
(not familiar with building from svn/trunk/wherever)

You can use fpcupdeluxe for that : http://wiki.lazarus.freepascal.org/fpcupdeluxe
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 26, 2019, 06:29:57 pm
Open a terminal; issue these commands and check how high is the stack size set in Ubuntu system.

Quote
#soft limits for a process/thread
ulimit -a
#hard limits for a process/thread
ulimit -Ha

Then set the soft limit of a stack size for reasonable size eg. 512 (ulimit -s 512) and try to re-execute your program.
thx, will try that and report back

My result on my machine
Code: [Select]
zvoni@ZvoniLinux:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31133
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
zvoni@ZvoniLinux:~$ ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31133
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Thaddy on March 27, 2019, 10:20:37 am
Maybe put that on the HeapTrc-Wiki: DON'T USE IT WITH ANY OTHER MEMORY-MANAGER.
FYI: I just adapted the wiki and added a warning, but also a note how to handle alternative memory managers and heaptrc in the same codebase:
http://wiki.freepascal.org/heaptrc
http://wiki.freepascal.org/heaptrc#Why_heaptrc_should_not_be_added_to_the_uses_clause_manually
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Cyrax on March 27, 2019, 10:39:15 am
Open a terminal; issue these commands and check how high is the stack size set in Ubuntu system.

Quote
#soft limits for a process/thread
ulimit -a
#hard limits for a process/thread
ulimit -Ha

Then set the soft limit of a stack size for reasonable size eg. 512 (ulimit -s 512) and try to re-execute your program.
thx, will try that and report back

My result on my machine
Code: [Select]
zvoni@ZvoniLinux:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31133
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
zvoni@ZvoniLinux:~$ ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31133
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

What does ulimit -Ha say?
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 27, 2019, 11:14:04 am

Code: [Select]
zvoni@ZvoniLinux:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31133
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
zvoni@ZvoniLinux:~$ ulimit -Ha
Code: [Select]
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31133
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

What does ulimit -Ha say?
huh?
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 27, 2019, 11:14:21 am
Maybe put that on the HeapTrc-Wiki: DON'T USE IT WITH ANY OTHER MEMORY-MANAGER.
FYI: I just adapted the wiki and added a warning, but also a note how to handle alternative memory managers and heaptrc in the same codebase:
http://wiki.freepascal.org/heaptrc
http://wiki.freepascal.org/heaptrc#Why_heaptrc_should_not_be_added_to_the_uses_clause_manually

Thaddy, thx
+1
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Cyrax on March 27, 2019, 12:10:23 pm

Code: [Select]
zvoni@ZvoniLinux:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31133
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
zvoni@ZvoniLinux:~$ ulimit -Ha
Code: [Select]
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31133
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31133
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

What does ulimit -Ha say?
huh?

Oh sorry, I didn't see the ulimit -Ha part from the quote text field.  :-[

It seems there is no limits. Please open a terminal in your program directory, execute ulimit -s 512 and then execute your program and see if there is errors or program misbehaves .

EDIT : Oh, there is soft limit:

Quote
stack size              (kbytes, -s) 8192
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 27, 2019, 12:26:15 pm

Oh sorry, I didn't see the ulimit -Ha part from the quote text field.  :-[

It seems there is no limits. Please open a terminal in your program directory, execute ulimit -s 512 and then execute your program and see if there is errors or program misbehaves .
Definitely no expert here, but why should i lower my Stack-size from 8196 KB to 512? (ulimit -s is for the "soft"-limit?!)
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Cyrax on March 27, 2019, 12:41:22 pm

Oh sorry, I didn't see the ulimit -Ha part from the quote text field.  :-[

It seems there is no limits. Please open a terminal in your program directory, execute ulimit -s 512 and then execute your program and see if there is errors or program misbehaves .
Definitely no expert here, but why should i lower my Stack-size from 8196 KB to 512? (ulimit -s is for the "soft"-limit?!)

Just for testing purposes. Every created thread (yes, every thread too and there can be several hundred of them running) and process allocates the default 8196 KB of memory.  And if there is about 1024 threads running then kernel will allocate about 8 192 MB ( that is 8 GB!)  for them. It is bit too much. So lowering it to between 512KB - 1024KB should suffice.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Cyrax on March 27, 2019, 12:45:50 pm
Info about on this : https://unix.stackexchange.com/questions/127602/default-stack-size-for-pthreads
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: engkin on March 27, 2019, 02:33:14 pm
Don't let us guess, provide a sample project that shows the problem. ld.so is the dynamic linker. Works when you run your app.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 27, 2019, 04:54:45 pm
Info about on this : https://unix.stackexchange.com/questions/127602/default-stack-size-for-pthreads
Thx for the link. Learned something new
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 27, 2019, 04:57:36 pm
Don't let us guess, provide a sample project that shows the problem. ld.so is the dynamic linker. Works when you run your app.
The problem seems to be the sqlite-connection-object in context of running the connection in its own thread.
I don‘t have the error with MySQL, and as Thaddy mentioned, they changed some sqlite-bindings in trunk, where it doesn‘t report the error, while i‘m on fpc304
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 30, 2019, 10:36:10 am
@Thaddy
By any chance: In trunk, are you using sqlite_open_v2 for InternalConnect?
I‘ve looked through the sourcecode, and found sqlite_open being used, and the docs for sqlite say, in that case sqlite is run with compiletime-options regarding multiple threads.
The default compile-options for distro-binaries is mode 1 (Serialized?), so right now i don‘t have a clue if it‘s my sqlite-lib being in singlethread-mode, or what you mentioned.
I‘m going to setup a small test, if the error occurs in a naked, stripped down, code
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Thaddy on March 30, 2019, 10:52:49 am
@Thaddy
By any chance: In trunk, are you using sqlite_open_v2 for InternalConnect?
I‘ve looked through the sourcecode, and found sqlite_open being used, and the docs for sqlite say, in that case sqlite is run with compiletime-options regarding multiple threads.
The default compile-options for distro-binaries is mode 1 (Serialized?), so right now i don‘t have a clue if it‘s my sqlite-lib being in singlethread-mode, or what you mentioned.
I‘m going to setup a small test, if the error occurs in a naked, stripped down, code
Yes. The Sqlite3 component for SqlDb is adapted to use sqlite_open_v2 by default on my request. Target release is 3.2.0 .
See https://bugs.freepascal.org/view.php?id=34278 which also documents briefly the new options available.
(That change definitely made Sqlite3 more robust., on my side all kind of quirks disappeared after the change!)

In your case I would suggest a defensive approach and use:
Code: Pascal  [Select][+][-]
  1. OpenFlags := [sofReadWrite,sofCreate,sofFullMutex]; // and maybe one of the caching strategies
Unless you run into performance issues.  See the Sqlite3 documentation, which you can now use a bit better with FPC.
You also still have to keep in mind Cyrax remark about lowering the stack size on some platforms.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 30, 2019, 11:58:30 am
You also still have to keep in mind Cyrax remark about lowering the stack size on some platforms.
About that: Any pointers how to do that? Still fairly new to everything.
At a guess: there is somewhere in Project-Options a Flag i can set the stacksize....

EDIT. Found it.
The "-Cs"-Option?
Value is what? in Kb?
Like "-Cs 512" is setting Stacksize to 512 Kb?
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Thaddy on March 30, 2019, 12:05:19 pm
You also still have to keep in mind Cyrax remark about lowering the stack size on some platforms.
About that: Any pointers how to do that? Still fairly new to everything.
At a guess: there is somewhere in Project-Options a Flag i can set the stacksize....

EDIT. Found it.
The "-Cs"-Option?
Value is what? in Kb?
Like "-Cs 512" is setting Stacksize to 512 Kb?
You can also set {$MINSTACKSIZE number} and  {$MAXSTACKSIZE number} in your code. The minimum value for {$maxstacksize} is 2048, though, you can't set it to 512, but 2048 should suffice in most cases.
Note that this goes for threading with TThread. If you want your threads to have a different or smaller stacksize, you should not use TThread but any of the beginthread methods: https://www.freepascal.org/docs-html/rtl/system/beginthread.html

I rarely use TThread in scenario's such as yours, because beginthread is more comfortable and has more control.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Cyrax on March 30, 2019, 12:19:13 pm
You also still have to keep in mind Cyrax remark about lowering the stack size on some platforms.
About that: Any pointers how to do that? Still fairly new to everything.
At a guess: there is somewhere in Project-Options a Flag i can set the stacksize....

EDIT. Found it.
The "-Cs"-Option?
Value is what? in Kb?
Like "-Cs 512" is setting Stacksize to 512 Kb?

Open terminal window and execute command ulimit -s 512 and after executing that command, execute your program.

How to open terminal window : https://askubuntu.com/questions/183775/how-do-i-open-a-terminal
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 30, 2019, 01:06:42 pm
I‘m going to setup a small test, if the error occurs in a naked, stripped down, code

Oh, come on. I just set up a barebone test in the scenario i have (albeit not in a library, but a console-program), and sqlite doesn't throw any errors in the threads.
Create FakeMainThread, take over as MainThread,
FakeMainThread creates Childthreads, which itself create the TSQL-Trinity with Type 'SQLite3',
connect to test-database, and no error is thrown by any of the TSQL-Objects accessing data of the DB

*sigh*
Back to square one, and off i go bug-hunting.......

[CLOSED]
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 30, 2019, 01:46:22 pm
You also still have to keep in mind Cyrax remark about lowering the stack size on some platforms.
About that: Any pointers how to do that? Still fairly new to everything.
At a guess: there is somewhere in Project-Options a Flag i can set the stacksize....

EDIT. Found it.
The "-Cs"-Option?
Value is what? in Kb?
Like "-Cs 512" is setting Stacksize to 512 Kb?

How curious.
Setting Stacksize in Project-Options to anything other than 0 results:
512 --> won't complile. No messages, no nothing
1024 (and higher) --> Stackoverflow Error

Switching off "-Ct"
512 --> won't complile. No messages, no nothing
1024 (and higher) --> seems (!) to work
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: engkin on March 30, 2019, 03:15:44 pm
It is a parameter you pass to the thread constructor.
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 30, 2019, 06:56:26 pm
It is a parameter you pass to the thread constructor.

Ahhh, OK. Have seen it, looked it up, but since no example ever shows its use, i ignored it.

EDIT: Nope, whatever i pass to the inherited constructor (512,1024,2048,4096) results in an Exception "Could not Create Thread"
The Stacksize in Project-options was set to 0 for this test.

EDIT2:
Writeln('DefaultStackSize = '+DefaultStackSize.ToString);    --> 4194304
What's that? Apples? Bytes? Beers?

EDIT3:
*GNARF*
passing 524288 works (a.k.a 512kb)
Title: Re: This is driving me mad: "Inconsistency detected by ld.so"
Post by: Zvoni on March 31, 2019, 02:21:17 pm
OK, we can close this Thread.
I've rebuild my code-base in a new project, and nowhere the afore mentioned error occurs. Seems to have been a bug in my code (no idea where)
TinyPortal © 2005-2018