Frustrating Error When using getmem to allocate memory and dare to ask

[jamie]

I thought this was your project.

Of course I have the skill to do that, how did you think I found it? 

I didn't dig in deep but using the tools at hand, I was able to deduce this using HeapTrc where it becomes slower than snail snots at the start of that code and this is in 64 bit mode and fails with a memory dump in the console.

 And yes, there are much better ways to remove the Iine-Endings and compact the results.

Jamie

[jamie]

I looked some more at that code, and I must say there was a lot of work put into it, I take my hat off to you for that!

BUT!
  I can't find any obvious problem after straining my eyes on it, I am sure there is a memory intrusion that is hammering the RTL with all the twisted dynamic array reallocations that are taking place and somewhere in the soup is an error.

 It's possible you have found a memory management error but at this point, I believe what is happening is you have fragmented the memory pool so bad it's hard to decide if it's your code overwriting somewhere or you have just created an unmanageable pile of fragmentation?

My suggestions are this:
 
  Create a static CONST of your op-codes using the compiler at compile time instead of doing this dynamically.

Code: Pascal  [Select][+][-]

1.   Specs: array[0..MaxModes] of TmodeSpecs = (
2.     (Name: 'SC1'; Trim: 1; TotalLineCount: 3965; ModeVisCode: (1, 60);
3.     GrayScaleLines: 16;
4.     ModeLineCount: 256; ModeGammaCount: $82; FrontPorch: 14; EndPorch: 0;
5.     ColorSync: 0; VideoRepeats: 4; ModeMaxSync: 83; ModeSpeed: 0.0001079999831; ),
6.     (Name: 'SC2'; Trim: 1; TotalLineCount: 2018; ModeVisCode: (1, 56);
7.     GrayScaleLines: 16;
8.     ModeLineCount: 256; ModeGammaCount: $82; FrontPorch: 11; EndPorch: 0;
9.     ColorSync: 0; VideoRepeats: 2; ModeMaxSync: 65; ModeSpeed: 0.0001376070802; ),
10.     (Name: 'SC3'; Trim: 1; TotalLineCount: 3965; ModeVisCode: (1, 52); GrayScaleLines: 8;        
11.  
12.  

An example from some code to build a Op-Code table via the compiler.
Also, I see  you use Allocmem(4) many times on a pointer for a field that can be represented within the structure without allocation of small chucks like that, that is what causes fragmentation.!

  There is much more in code optimization that can be done. more later.

Jamie

[Bart]

Google City Hash must be used in disabling overflow switch to work.
I don't do anything in 32-bit due to I work on 64-bit machines.
Don't Simplified the process due to the error is on system.pas as GDB backtrace said although I don't know why.

It would have been nice to say so at the beginning (and again: make this happen in the source code. The first thing we do when bughunting is turning all checks on).
Next issue: hardcoded path's. Give nt elayout of your procte, using relative paths would be a bit more logical.
Next issue: range check error:

Code: [Select]

C:\Users\Bart\LazarusProjecten\bugs\forum\unias\unias>unias
Initialize Success!
Read Success!
An unhandled exception occurred at $0000000100090967:
ERangeError: Range check error
  $0000000100090967  unias_assembly_string_to_binary,  line 13722 of uniasinfo.pas
  $000000010002D3A7  unias_generate_assembly,  line 2996 of unianalysis.pas
  $00000001000018AD  $main,  line 13 of unias.lprHere you assing a signed integer to an unsigned variable.
If you know at forehand this is safe, just typecast the signed integer to unsigned.

Again: if some part of the code relies on some checks being off, turn it off in the source code (and document the reason).

Debugging code that requires all sanity checks being off is a nightmare.

Don't Simplified the process due to the error is on system.pas as GDB backtrace said although I don't know why.

You say so, but rather unlike since nobody ever reported this before you.
And trying to reduce the problem to it's bare minimum is essential if this would really turn out to be a problem in fpc.

Bart

[Bart]

Next:

Code: [Select]

An unhandled exception occurred at $000000010000C900:
EAccessViolation: Access violation
  $000000010000C900
  $000000010000CA09
  $000000010000CB33
  $000000010000BBAB
  $0000000100034191  unias_generate_final,  line 3756 of unianalysis.pas
  $000000010000190A  $main,  line 15 of unias.lpr

Code: Pascal  [Select][+][-]

1.     begin
2.      if(OutputFile.SectionType[i-1]<>elf_section_type_nobit) and (OutputFile.SectionSize[i-1]>0) then
3.       begin
4.        Move(OutputFile.SectionContent[i-1]^,(FileContent+OutputFile.SectionOffset[i-1])^,
5.        OutputFile.SectionSize[i-1]);
6.        FreeMem(OutputFile.SectionContent[i-1]); //the offending line
7.       end;

Looks like you have some sort of memory corruption.

[Bart]

Tried to build with fpc main (since I build the compiler/trl with debug info), but then the executable just hangs after "Read Success!".

Bart

[Bart]

Tried using CMem memory manger.
No crash, but the program terminates early:

Code: [Select]

C:\Users\Bart\LazarusProjecten\bugs\forum\unias\unias>unias
Initialize Success!
Read Success!

C:\Users\Bart\LazarusProjecten\bugs\forum\unias\unias>
Having the whole procedure unias_generate_assembly() in a try..except block shows there's some EAccesViolation in that code.
Somewhere between:
 {Then ReHandle the Other Codes}
and
 {Then ReHandle the File}


So, with both memory managers (default and cmem) the program crashes/hangs.
Changes are that your code is at fault, not both memorymanagers.

Bart

[jamie]

Code: Pascal  [Select][+][-]

1.  {Then ReHandle the File}
2.   i := 1;
3.   j := 1;
4.   k := 1;
5.   n := 1;
6.   while (i <= BaseAssembly.LineCount) do
7.   begin
8.     n := 1;
9.     if (Result.Section[j - 1].ContentAssemblyCount = 0) then
10.     begin
11.       Inc(i);
12.       continue;
13.     end;
14.  // but it happens here before then next while loop, .... ParamCount contains a QWORD(-1); FFF........etc
15.     while (n <= Result.Section[j - 1].ContentAssemblyCode[k - 1].ParamCount) do           //<< Range error not in this loop but crashes here.
16.  
17.  

So before it reaches that While Loop, around 388x the value turns bad.

Jamie

[jamie]

Code: Pascal  [Select][+][-]

1.  Assert(k-1  < Length(Result.Section[j-1].ContentAssemblyCode),'Index out of bounds');
2.     while (n <= Result.Section[j - 1].ContentAssemblyCode[k - 1].ParamCount) do        
3.  

There you go, index out of bounds.

Jamie

[jamie]

Ok, I've finally got it to fully exe to the point where there is a user inserted code about some FPC_ something not being valid.

"Error in Coluum 11, Row 1 "
"__FPC_Specfiic_Hnad1 "
"ERROR Detail;__fpc_Specific_hand1 cannot to be recognized"

What I did because I remember an issue like this with Generics, using managed strings that land in the records, they cause some issue, so I just went and made most of them short strings in the record defines.

Jamie

[TYDQ]

Code: Pascal  [Select][+][-]

1.  Assert(k-1  < Length(Result.Section[j-1].ContentAssemblyCode),'Index out of bounds');
2.     while (n <= Result.Section[j - 1].ContentAssemblyCode[k - 1].ParamCount) do        
3.  

There you go, index out of bounds.

Jamie

Ok,Thank you for providing when the error truly occurs.

[jamie]

Read above..

The problem is related to managed strings and maybe even arrays, living inside a record.

But it seems I had an issue with managed strings in generics, and I switched to Short strings which are inline of the record in your code.

your code now executes completely less the error you placed in there for non-support of FPC items.

I changed all of your strings  in the records to string[ 20 ];

Jamie

[TYDQ]

Read above..

The problem is related to managed strings and maybe even arrays, living inside a record.

But it seems I had an issue with managed strings in generics, and I switched to Short strings which are inline of the record in your code.

your code now executes completely less the error you placed in there for non-support of FPC items.

I changed all of your strings  in the records to string[ 20 ];

Jamie

Can I send an issue to the FPC Team for this error where the problem is on string internally?

[TYDQ]

Read above..

The problem is related to managed strings and maybe even arrays, living inside a record.

But it seems I had an issue with managed strings in generics, and I switched to Short strings which are inline of the record in your code.

your code now executes completely less the error you placed in there for non-support of FPC items.

I changed all of your strings  in the records to string[ 20 ];

Jamie

I looked up FPC documentation that Runtime Error 203 is Access Violation,I don't know why the Heap Cannot Grow as heap allocated 19.942MiB.
The Access Violation occurs when allocating more than 19.942MiB memory.Does FPC Have Some restriction?

[marcov]

If I take your sourcecode and put a break point on getmem , I get an exception at around line 2312 near a freemem. So before I get to the getmem.

And it is not the first time it passes by that line. I suspect the heap manager gets confused due to repeated freemems of random pointers.

[Zvoni]

If I take your sourcecode and put a break point on getmem , I get an exception at around line 2312 near a freemem. So before I get to the getmem.

And it is not the first time it passes by that line. I suspect the heap manager gets confused due to repeated freemems of random pointers.

Now that you mention it: Which Memory-Manager IS he using?
I think to remember, that using cmem AND HeapTrc at the same time leads to unexpected hiccups