Weird thing happens. Str:='Const'+Str; spoils other data

[mm7]

Linux Ubuntu 3.13.0-48-lowlatency #80-Ubuntu SMP PREEMPT Thu Mar 12 11:33:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
FPC 2.6.4 [2015/02/21] for x86_64
No optimization, all checks and assertions on, debugging info for GDB Dwarf2 line numbers,

Code: [Select]

function UserString(Index:Integer):String;
var Str     : string;
    Section : String;
    val     : string;
    I,N     : Integer;
begin
   Result:='';
   if CurrentLanguage<>nil then
   begin
      Section:='User';
       Val:=IntToStr(Index);
       while length(val)<4 do val:='0'+val;
       Val:='User'+Val;
...
And here, on last simple statement, it spoils completely different object in program!
Most interesting that address of that object is lower, so buffer overrun is not possible.

here when it enters into fpc_AnsiStr_Concat
and then into fpc_AnsiStr_SetLength
and then into reallocmem, MemoryManager.ReAllocMem
and then MemoryManager.GetMem returns new pointer.
This new pointer is larger than the pointer that will be spoiled by following Move (heap.inc, 1442).

p is $00007FFFEDD95C40
p2 is $00007FFFEDD80C80
minsize is 24

Move(p^,p2^,minsize);
causes another alive object with pointer $00007FFFEDCE7600 become pointer $00007F0030393230 

Of course it destroys its data and causes access violation SIGSEGV later when program tries to access it.

I can work it around replacing
       while length(val)<4 do val:='0'+val;
       Val:='User'+Val;
with something different, like
      Key := Copy('User0000',1,8-length(Val)) + Val;

But, why on the Earth Move() changes other objects?!

[Basile B.]

instead of

Code: [Select]

Val:=IntToStr(Index);
while length(val)<4 do val:='0'+val;
Val:='User'+Val;

could you try this:

Code: [Select]

val := format('User%.4d', [index]);

?

[FPK]

But, why on the Earth Move() changes other objects?!

Without a complete example, this cannot be answered.

[marcov]

Usually this means you corrupt memory somewhere in your program, so that two allocations point to the same block and overwrite eachother.

[Martin_fr]

Independent of the actual problem (which could well be mem corruption anywhere else)

causes another alive object with pointer $00007FFFEDCE7600 become pointer

If you refer to an "Object": var foobar: TFooObject
(also applies to strings, or dyn array)

Then it is not clear which pointer value you refer to in the above statement.

Objects/strings/dyn-array are internally pointers.

So "foobar" the variabel itself may be at address 00000012345678, and (on 64 bit) occupies 8 bytes, holding a pointer Lets say 00002222222222,
Then at address 00002222222222 there are sizeof(TFooObject) bytes holding the fields of that object.

So which Pointer to you refer too?
@foobar or pointer(foobar)

----------
Though that would only matter if this was something writing cross allocated boundaries right then. Which is not necessarily the case.

A good idea is to run your code in valgrind.

[mm7]

Usually this means you corrupt memory somewhere in your program, so that two allocations point to the same block and overwrite eachother.

I understand this. But I've checked those "other" object. It was positive on Assigned(TheAnotherObject) call.
How come MemoryManager gave out its address ($00007FFFEDD80C80) to new string (the result of 'User'+Val expression)?
So TheAnotherObject is at address $00007FFFEDD80C80, and it is not Nil.
Is there a way to check if object is actually assigned in MemoryManager?

[taazz]

Usually this means you corrupt memory somewhere in your program, so that two allocations point to the same block and overwrite eachother.

I understand this. But I've checked those "other" object. It was positive on Assigned(TheAnotherObject) call.

[snip..]

step 1 create a small (preferably a console) application that demonstrates the problem.
Step 2 post it here so we can take a closer look.

If it is impossible to recreate the problem outside your application then as marcov said the position that the error is raised is random and has nothing to do with the error. The memory has been corrupted long before the execution reached that point, you need to trace everything back and find where the problem is.

[mm7]

Folks, please,
1. It was not "my" application and code. I am just migrating it to Lazarus from Delphi. Other people who are very strong in hydrodynamics and ship modeling created that program. May be in some places it is not very elegant and efficient programming wise, but as soon as it produces correct result, I do not touch it.
(please take a look into "elegantly looking" format(). You will see whole bunch of heavy things that are not necessary for the case, like Regexp. No need to fire a cannon at sparrows here.)
2. I asked about a reliable way to get information from Memory Manager regarding a pointer of object if its memory is allocated. Is Assigned() a good one?

[mm7]

step 1 create a small (preferably a console) application that demonstrates the problem.
Step 2 post it here so we can take a closer look.

If it is impossible to recreate the problem outside your application then as marcov said the position that the error is raised is random and has nothing to do with the error. The memory has been corrupted long before the execution reached that point, you need to trace everything back and find where the problem is.

1. I tried to create a small console application that demonstrates the problem. Does not work.
2. The error is not random. It is reproducible. Always same place and same pointers.
I agree it is most likely that "The memory has been corrupted long before the execution reached that point". What is a best way to find where? I've checked the objects right before that string concatenation called, all looked good. May be I missed something?

[Martin_fr]

Assigned does not tell you if mem is allocated.

Code: [Select]

  A:= TObject.create;
  b := a;
  a.free;

a is nil (because free does this, if you used destroy, then a would be a pointer to memory that is not allocated, but later can be allocated again).

b is a pointer to memory that is not allocated, but later can be allocated again

Until such memory is allocated again, it still has the objects data, so it will seem to work.
Except if you use heaptrc -gh. This will fill the memory with trash.

There is -gc http://www.freepascal.org/docs-html/user/usersu15.html#x38-450005.1.4

-gc
    Generate checks for pointers. This must be used with the -gh command line option. When this options is enabled, it will verify that all pointer accesses are within the heap.

This is going to make the code real slow.

Also, I do not know if that is pointers you declared, or also pointers that are implicit (objects). And what happens if an OS call allocated memory (outside the fpc heap).

And more, it can only check, if the pointer is in allocated memory. It can not check if that memory was allocated for that pointer. Memory may be allocated again later, and a dangling pointer could point into this mem.


If you can check on linux, then valgrind may be a real good idea.


EDIT: see also http://forum.lazarus.freepascal.org/index.php/topic,27880.msg172936.html#msg172936

[Martin_fr]

One more, just a random idea. Search the entire code for

Code: [Select]

procedure(CONST abc: string)
Remove "const" from any string (ansistring) param. And also any dyn-array.

This is one of the most often misunderstood features, and can easily corrupt memory if incorrectly used (yet will compile without even a hint or warning).

[Martin_fr]

Another way of debugging this.

But only if you can get the code to run in a way, that the error happens at the exact same address each time.
In this case, you can try to find out, who accesses this address before, and maybe find something.

In order to find all write access to an address (e.g. 00007FFFEDD80C80), create a watchpoint (under menu "Run" > "Add breakpoints"):
options: global / on write
value: ^longint(0x00007FFFEDD80C80)^

I haven't tested it, but the value should work like that.

Untick the checkbox "break", and instead tick "Take Snapshot".

Then when the error happens, open the "view" > "debug windows" > "debug history", and you have a list of where the address was accessed.

Only you need to know the address upfront.
And the execution might be slowed down a lot.

In general, this kind of error is one of the hardest to trace....

[Nitorami]

This is one of the most often misunderstood features, and can easily corrupt memory if incorrectly used (yet will compile without even a hint or warning)

Would you mind explaining this ? I understood that "const" is a hint to the compiler that the variable will not be changed. In your procedure if I assign abc[1] := 'A' the compiler stops with an error.

[engkin]

2. The error is not random. It is reproducible. Always same place and same pointers.
I agree it is most likely that "The memory has been corrupted long before the execution reached that point". What is a best way to find where?

I would use a *hardware* breakpoint at one of the addresses that get corrupted.

[Martin_fr]

Would you mind explaining this ? I understood that "const" is a hint to the compiler that the variable will not be changed. In your procedure if I assign abc[1] := 'A' the compiler stops with an error.

It is a promise from you to the compiler that you will not change it (neither directly, nor through any reference to it).

The compiler will stop obvious cases, but only obvious ones.

Code: [Select]

var someglobal: ansistring;

procedure Foo(const s: ansistring);
begin
    someglobal := foo;
end;

Foo(someglobal)

You are not changing "s", but someglobal is the same variable, and you change someglobal.
That breaks the promise.

Depending on the complexity of code:
- you may get away, with no mem corruption and no error.
- you may get a mem leak
- you may corrupt the mem managment.