Recent

Author Topic: Why CMem?  (Read 2330 times)

440bx

  • Hero Member
  • *****
  • Posts: 1086
Re: Why CMem?
« Reply #30 on: August 19, 2019, 10:35:07 am »
what happens when I increment that pointer? Now its no longer at its original starting point..
For a number of reasons, performance among them, most memory managers require the original pointer they returned to free the memory block.

With most memory managers, passing a pointer that points somewhere within an allocated memory block rarely, if ever, yields a desirable result.
using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

Tz

  • New Member
  • *
  • Posts: 40
  • Tz with FPC Pen Cil
Re: Why CMem?
« Reply #31 on: August 19, 2019, 04:58:36 pm »
In my newbie point of view, if I want platform specific memory manager
for example tcmalloc or jemalloc, 
I just use CMem and change

{$else}
  LibName = 'c';  // change to tcmalloc or jemalloc and install the library
{$endif}

just simple as that. Its like plug the MM and play. 
Its freedom to play, other programming? maybe just a few?
Pascal Programming is for teaching, indeed.

I tried that before, but stick to c, cause its already there.
No need tcmalloc or jemalloc deployment check. (This is admin view)

or if I compiled with -glh it's cost me zero :)  (this newbie view, need explanation from hero)

program mem;
{$mode objfpc}
{$h+}
uses CMem, CThreads;
var b :array of Byte;
begin
  SetLength(b, 1000);
end.

Once I had memory leak, if I declare CThreads first, as like most of examples.
Since then I put CMem first.


For speed, I think all Hero here, has got a point.


Excuse me julkas, its nice topic, you gather all Hero, here.


For all Hero, a bit request from batalion of newbie :)

program Project1;
{$mode objfpc}
{$h+}
uses
  {$IFDEF UNIX}{$IFDEF UseCThreads}      // why in each program have to declare like this
  cthreads,
  {$ENDIF}{$ENDIF}

is there a way for platform specific like

program Project1;

uses FpcWindows or DelphiWindows ...

program Project1;

uses FpcLinux or DelphiLinux ...


and all CMem, CThreads, directives, etc are handle seamlessly?

long live newbie,  cheers :)



Leledumbo

  • Hero Member
  • *****
  • Posts: 8103
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Why CMem?
« Reply #32 on: August 19, 2019, 05:29:05 pm »
For all Hero, a bit request from batalion of newbie :)

program Project1;
{$mode objfpc}
{$h+}
uses
  {$IFDEF UNIX}{$IFDEF UseCThreads}      // why in each program have to declare like this
  cthreads,
  {$ENDIF}{$ENDIF}
Single source cross platform compatibility in mind, otherwise you have to change the uses clause everytime you change target platform. Some of us, like me, uses the build mode to build binaries for multiple platform in a single click, it won't work if I have to do what I said earlier, nor it will be a pleasing experience having blocks of ifdef for every platform.
is there a way for platform specific like

program Project1;

uses FpcWindows or DelphiWindows ...

program Project1;

uses FpcLinux or DelphiLinux ...


and all CMem, CThreads, directives, etc are handle seamlessly?
First, it's completely the opposite of the above principle. Second, it cannot be done so. Not all (Unix targeting) programs require threads, so it can't be forced so (since not just unnecessary, but including cthreads pull a significant dependency to the binary size). Directives work at compile time, hence it cannot be included in a compiled unit. Or do you prefer FpcWindows, FpcUnixNoThreads, FpcUnixWithThreads, etc.? It's a big NAH from me, both from users POV and maintainers POV.
« Last Edit: August 19, 2019, 05:30:38 pm by Leledumbo »

jamie

  • Hero Member
  • *****
  • Posts: 1901
Re: Why CMem?
« Reply #33 on: August 20, 2019, 12:06:19 am »
what happens when I increment that pointer? Now its no longer at its original starting point..
For a number of reasons, performance among them, most memory managers require the original pointer they returned to free the memory block.

With most memory managers, passing a pointer that points somewhere within an allocated memory block rarely, if ever, yields a desirable result.
I was thinking on the lines of it keeping an ADDRESS reference to the pointer variable itself.
by giving the pointer via reference back to a memory handler to release it, it can then scan a table of allocated chunks to locate a matching reference to a pointer.

 This way the pointer itself can change value but the entity will always be the same.

 Which means of course if you make one pointer := another, the pointer can not be used to release memory from the "another" pointer.


440bx

  • Hero Member
  • *****
  • Posts: 1086
Re: Why CMem?
« Reply #34 on: August 20, 2019, 12:40:50 am »
I was thinking on the lines of it keeping an ADDRESS reference to the pointer variable itself.
by giving the pointer via reference back to a memory handler to release it, it can then scan a table of allocated chunks to locate a matching reference to a pointer.

 This way the pointer itself can change value but the entity will always be the same.

 Which means of course if you make one pointer := another, the pointer can not be used to release memory from the "another" pointer.
What you described sounds like how Windows 16bit managed memory.  GlobalAlloc and LocalAlloc returned a pointer to a pointer, this second pointer being the actual pointer to the memory block.  This is what allowed Win16 to move memory blocks.  Whenever it moved a block, usually to lower heap fragmentation, it would update that second pointer. 

Even using that method, Win16 depended on the second pointer referencing the beginning of the memory block to either, free it or move it around.

A generic memory manager has to balance the performance of allocating and freeing blocks while doing its best to keep fragmentation at a minimum. 

Under Win32 and Win64 and, equivalent virtual memory based O/Ses, moving memory is not a concern since a virtual address is not a reference to a real physical memory block and, the real memory block can be changed anytime by updating the page directory or page table entries but, memory fragmentation is still a concern both, for virtual address space and any heap manager.

When memory allocation/deallocation is a concern, the best and often simplest solution is to allocate a large enough block for what the program needs and manage the allocation/deallocation of blocks within it yourself.


using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

PascalDragon

  • Hero Member
  • *****
  • Posts: 573
  • Compiler Developer
Re: Why CMem?
« Reply #35 on: August 20, 2019, 09:12:15 am »
what happens when I increment that pointer? Now its no longer at its original starting point..
For a number of reasons, performance among them, most memory managers require the original pointer they returned to free the memory block.

With most memory managers, passing a pointer that points somewhere within an allocated memory block rarely, if ever, yields a desirable result.
I was thinking on the lines of it keeping an ADDRESS reference to the pointer variable itself.
by giving the pointer via reference back to a memory handler to release it, it can then scan a table of allocated chunks to locate a matching reference to a pointer.

 This way the pointer itself can change value but the entity will always be the same.

 Which means of course if you make one pointer := another, the pointer can not be used to release memory from the "another" pointer.
I personally think that performance is more important than the "inconvenience" to keep the original pointer, which the memory function returned to me, around.

BrunoK

  • Full Member
  • ***
  • Posts: 174
  • Retired programmer
Re: Why CMem?
« Reply #36 on: August 20, 2019, 03:09:15 pm »
Where to start.

1° cmem on windows. As Grumpy writes _msize can be used to retrieve the size of the allocated memory instead of having a header sizeuint containing the size.
Note that heap.inc has a similar approach in its own structures.
cmem using _msize. Tested with rebuilding FPC 3.0.4 latest (3.0.6 in my private branch) and rebuilding a relatively recent lazarus.
Code: Pascal  [Select]
  1. {
  2.     This file is part of the Free Pascal run time library.
  3.     Copyright (c) 1999 by Michael Van Canneyt, member of the
  4.     Free Pascal development team
  5.  
  6.     Implements a memory manager that uses the C memory management.
  7.  
  8.     See the file COPYING.FPC, included in this distribution,
  9.     for details about the copyright.
  10.  
  11.     This program is distributed in the hope that it will be useful,
  12.     but WITHOUT ANY WARRANTY; without even the implied warranty of
  13.     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  14.  
  15.  **********************************************************************}
  16. unit cmem;
  17.  
  18. {$MODE OBJFPC}
  19.  
  20. {$DEFINE BK_FEATURES} { ~bk Code regarding agnostic heaptrc and initializing
  21.                             memorymanager and heaptrc BEFORE system.pp }
  22.  
  23. interface
  24.  
  25. const
  26.  
  27. {$if defined(go32v2) or defined(wii)}
  28.   {$define USE_STATIC_LIBC}
  29. {$endif}
  30.  
  31. {$if defined(win32)}
  32.   LibName = 'msvcrt';
  33. {$elseif defined(win64)}
  34.   LibName = 'msvcrt';
  35. {$elseif defined(wince)}
  36.   LibName = 'coredll';
  37. {$elseif defined(netware)}
  38.   LibName = 'clib';
  39. {$elseif defined(netwlibc)}
  40.   LibName = 'libc';
  41. {$elseif defined(macos)}
  42.   LibName = 'StdCLib';
  43. {$elseif defined(beos)}
  44.   LibName = 'root';
  45. {$else}
  46.   LibName = 'c';
  47. {$endif}
  48.  
  49. {$ifdef USE_STATIC_LIBC}
  50.   {$linklib c}
  51. function malloc(Size: ptruint): Pointer; cdecl; external;
  52. procedure Free(P: pointer); cdecl; external;
  53. function realloc(P: Pointer; Size: ptruint): pointer; cdecl; external;
  54. function calloc(unitSize, UnitCount: ptruint): pointer; cdecl; external;
  55. {$else not USE_STATIC_LIBC}
  56. Function Malloc (Size : ptruint) : Pointer; cdecl; external LibName name 'malloc';
  57. Procedure Free (P : pointer); cdecl; external LibName name 'free';
  58. function ReAlloc (P : Pointer; Size : ptruint) : pointer; cdecl; external LibName name 'realloc';
  59. Function CAlloc (unitSize,UnitCount : ptruint) : pointer; cdecl; external LibName name 'calloc';
  60. {$if defined(Windows)}
  61. Function _MemSize(P : Pointer) : ptruint; cdecl; external LibName name '_msize';
  62. {$endif}
  63. {$endif not USE_STATIC_LIBC}
  64.  
  65. implementation
  66.  
  67. uses
  68.   SysUtils;
  69. function CGetMem(Size: ptruint): Pointer;
  70. {var
  71.   lErrno: integer; }
  72. begin
  73.   {$if defined(Windows)}
  74.   CGetMem := Malloc(Size);
  75.   {$else}
  76.   CGetMem := Malloc(Size + sizeof(ptruint));
  77.   if (CGetMem <> nil) then begin
  78.     Pptruint(CGetMem)^ := size;
  79.     Inc(CGetMem, sizeof(ptruint));
  80.   end;
  81.   {$endif}
  82. end;
  83.  
  84. function CFreeMem(P: pointer): ptruint;
  85. begin
  86.   {$if not defined(Windows)}
  87.   if (p <> nil) then
  88.     Dec(p, sizeof(ptruint));
  89.   {$endif}
  90.   Free(P);
  91.   CFreeMem := 0;
  92. end;
  93.  
  94. function CFreeMemSize(p: pointer; Size: ptruint): ptruint;
  95.  
  96. begin
  97.   if size <= 0 then
  98.     exit;
  99.   {$if not defined(Windows)}
  100.   if (p <> nil) then begin
  101.     if (size <> Pptruint(p - sizeof(ptruint))^) then
  102.       runerror(204);
  103.   end;
  104.   {$endif}
  105.   CFreeMemSize := CFreeMem(p);
  106. end;
  107.  
  108. function CAllocMem(Size: ptruint): Pointer;
  109. begin
  110.   {$if defined(Windows)}
  111.   CAllocMem := calloc(Size, 1);
  112.   {$else}
  113.   CAllocMem := calloc(Size + sizeof(ptruint), 1);
  114.   if (CAllocMem <> nil) then begin
  115.     Pptruint(CAllocMem)^ := size;
  116.     Inc(CAllocMem, sizeof(ptruint));
  117.   end;
  118.   {$endif}
  119. end;
  120.  
  121. function CReAllocMem(var p: pointer; Size: ptruint): Pointer;
  122. begin
  123.   {$if defined(WINDOWS)}
  124.   p := Realloc(p, size);
  125.   CReAllocMem := p;
  126.   {$else}
  127.   if size = 0 then begin
  128.     if p <> nil then begin
  129.       Dec(p, sizeof(ptruint));
  130.       Free(p);
  131.       p := nil;
  132.     end;
  133.   end
  134.   else begin
  135.     Inc(size, sizeof(ptruint));
  136.     if p = nil then
  137.       p := malloc(Size)
  138.     else begin
  139.       Dec(p, sizeof(ptruint));
  140.       p := realloc(p, size);
  141.     end;
  142.     if (p <> nil) then begin
  143.       Pptruint(p)^ := size - sizeof(ptruint);
  144.       Inc(p, sizeof(ptruint));
  145.     end;
  146.   end;
  147.   {$endif}
  148.   CReAllocMem := p;
  149. end;
  150.  
  151. function CMemSize(p: pointer): ptruint;
  152.  
  153. begin
  154.   {$if defined(WINDOWS)}
  155.   CMemSize := _MemSize(p);
  156.   {$else}
  157.   CMemSize := Pptruint(p - sizeof(ptruint))^;
  158.   {$endif}
  159. end;
  160.  
  161. function CGetHeapStatus: THeapStatus;
  162.  
  163. var res: THeapStatus;
  164.  
  165. begin
  166.   fillchar(res, sizeof(res), 0);
  167.   CGetHeapStatus := res;
  168. end;
  169.  
  170. function CGetFPCHeapStatus: TFPCHeapStatus;
  171.  
  172. begin
  173.   fillchar(CGetFPCHeapStatus, sizeof(CGetFPCHeapStatus), 0);
  174. end;
  175.  
  176. const
  177.   CMemoryManager: TMemoryManager =
  178.     (
  179.     NeedLock: False;
  180.     GetMem: @CGetmem;
  181.     FreeMem: @CFreeMem;
  182.     FreememSize: @CFreememSize;
  183.     AllocMem: @CAllocMem;
  184.     ReallocMem: @CReAllocMem;
  185.     MemSize: @CMemSize;
  186.     InitThread: nil;
  187.     DoneThread: nil;
  188.     RelocateHeap: nil;
  189.     GetHeapStatus: @CGetHeapStatus;
  190.     GetFPCHeapStatus: @CGetFPCHeapStatus;
  191.     );
  192.  
  193. var
  194.   OldMemoryManager: TMemoryManager;
  195.  
  196. initialization
  197.   GetMemoryManager(OldMemoryManager);
  198.   SetMemoryManager(CMemoryManager, mmtCmem, nil, @FinalizeCMem);
  199.  
  200. Finalization
  201.   SetMemoryManager (OldMemoryManager);
  202.  
  203. end.
  204.  

2° Usage of MemoryManager.memsize  :

@PascalDragon
In general, but focussing on cmem:
If you leave out the size, Lazarus doesn't work - but Fpc does! -. Try it yourself using cmem as a template, simply rip out the silly size management.
So Lazarus relies on implementation detail.

Could any of you 2 indicate where in lazarus memsize is wrongly used, please, and how those limited number of places do affect Lazarus. Also how would ansistring work in FPC without MemSize ?

MemSize MUST IMPERATIVELY BE IMPLEMENTED CORRECTLY in any memory manager otherwise the
following FPC routines would never work. They are
- fpc_AnsiStr_SetLength
- fpc_UnicodeStr_SetLength
- fpc_WideStr_SetLength

The least one can say is these are pretty commonly used ...

I mention them because I did a few changes (3.0.6) in these places to put all memory manager (that is heap.inc and cmem) on equal footing regarding string pre-overallocation.
Lazarus trunk r. 59978/03.01.2019 (+/- patches regarding enabled, TScrollBar, TCursorImage). FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903)

PascalDragon

  • Hero Member
  • *****
  • Posts: 573
  • Compiler Developer
Re: Why CMem?
« Reply #37 on: August 21, 2019, 09:43:51 am »
2° Usage of MemoryManager.memsize  :

@PascalDragon
In general, but focussing on cmem:
If you leave out the size, Lazarus doesn't work - but Fpc does! -. Try it yourself using cmem as a template, simply rip out the silly size management.
So Lazarus relies on implementation detail.

Could any of you 2 indicate where in lazarus memsize is wrongly used, please, and how those limited number of places do affect Lazarus. Also how would ansistring work in FPC without MemSize ?
I don't know what is failing, I have not tried. It could even be that it's not MemSize that is the problem, but that some code tries to simply access the hidden size field. Either Thaddy has more info or you could play around with it yourself.

Thaddy

  • Hero Member
  • *****
  • Posts: 8680
Re: Why CMem?
« Reply #38 on: August 21, 2019, 10:01:22 am »
[I don't know what is failing, I have not tried. It could even be that it's not MemSize that is the problem, but that some code tries to simply access the hidden size field. Either Thaddy has more info or you could play around with it yourself.
The info that I have is that a memory manager implementation for Delphi doesn't need to store length info.
When I subsequently rip out the length information from cmem in Freepascal that manager seems to work in Freepascal, but not in Lazarus.

Further: many platforms have other means to obtain the size of an allocated block, such OS's store the size themselves.
Examples are
- Windows  _msize
- OSX, malloc_size
- Linux and BSD's malloc_usable_size (This is a GNU extension)

So at least for those platforms there is technically no need to store the size, although they store the size of the allocated block and not the actual used length.
Freepascal stores length at negative offset from the variable if applicable, so that should be enough. Should not be done in the memory manager.

Once I can drop the size, the memory manager can also have better alignment.

An experimental example (linux):
Code: Pascal  [Select]
  1. unit fastcmem;  
  2. {$mode delphi}
  3. interface  
  4. // slightly faster than cmem, all size management code removed.
  5. Const  
  6.   LibName = 'libc';  
  7.  
  8. Function Malloc (Size : ptrint) : Pointer;  
  9.   cdecl; external LibName name 'malloc';  
  10. Procedure Free (P : pointer);  
  11.   cdecl; external LibName name 'free';  
  12. function ReAlloc (P : Pointer; Size : ptrint) : pointer;  
  13.   cdecl; external LibName name 'realloc';  
  14. Function CAlloc (unitSize,UnitCount : ptrint) : pointer;  
  15.   cdecl; external LibName name 'calloc';  
  16. Function MemSizeUsed(p:pointer):ptrint;
  17.   cdecl; external libname name 'malloc_usable_size';
  18.  
  19. implementation  
  20. Function CGetMem  (Size : ptruint) : Pointer;  
  21. begin  
  22.   CGetMem:=Malloc(Size);  
  23. end;  
  24.  
  25. Function CFreeMem (P : pointer) : ptruint;  
  26. begin  
  27.   Free(P);  
  28.   CFreeMem:=memsizeused(p);  
  29. end;  
  30.  
  31. Function CFreeMemSize (P : pointer;size:ptruint) : ptruint;  
  32. begin  
  33.   Free(p);
  34.   CFreeMemSize:=MemSizeUsed(p);  
  35. end;  
  36.  
  37.  
  38. Function CAllocMem(Size : ptruint) : Pointer;  
  39. begin  
  40.   CAllocMem:=calloc(Size,1);  
  41. end;  
  42.  
  43. Function CReAllocMem (const p:pointer;Size:ptruint):Pointer;  
  44. begin  
  45.   CReAllocMem:=realloc(p,size);    
  46. end;  
  47.  
  48. Function CMemSize (p:pointer): ptruint;  
  49. begin  
  50.   CMemSize:=memsizeused(p);
  51. end;  
  52.  
  53. function CGetHeapStatus:THeapStatus;  
  54. begin  
  55.   CGetHeapStatus :=Default(THeapStatus);  
  56. end;  
  57.  
  58. function CGetFPCHeapStatus:TFPCHeapStatus;  
  59. begin  
  60.   CGetFPCHeapStatus:=Default(TFPCHeapStatus);  
  61. end;  
  62.  
  63.  
  64. Const  
  65.  CMemoryManager : TMemoryManager =  
  66.     (  
  67.       NeedLock : false;  
  68.       GetMem : @CGetmem;  
  69.       FreeMem : @CFreeMem;  
  70.       FreememSize : @CfreeMemSize;  
  71.       AllocMem : @CAllocMem;  
  72.       ReallocMem : @CReAllocMem;  
  73.       MemSize : @CMemSize;  
  74.       InitThread : Nil;  
  75.       DoneThread : Nil;  
  76.       RelocateHeap : Nil;  
  77.       GetHeapStatus :@CGetHeapStatus;
  78.       GetFPCHeapStatus:@CGetFPCHeapStatus;
  79.     );  
  80.  
  81. Var  
  82.   OldMemoryManager : TMemoryManager;  
  83.  
  84. Initialization  
  85.   OldMemoryManager:=Default(TMemoryManager);
  86.   GetMemoryManager (OldMemoryManager);  
  87.   SetMemoryManager (CmemoryManager);  
  88.  
  89. Finalization  
  90.   SetMemoryManager (OldMemoryManager);  
  91. end.

Fails with Lazarus....works with Freepascal.
Code: Pascal  [Select]
  1. program testcmem ;
  2. {$if not declared(useheaptrc)}uses fastcmem;{$endif}{$H+}{$I-}
  3. type
  4.   TMyrec=record
  5.   s:string;
  6.   end;
  7.   PMyRec=^TMyrec;
  8. var
  9.   P:array[0..99] of PMyRec;
  10.   i:integer;
  11. begin
  12.   //allocate some memory
  13.   for i := 0 to 99 do begin New(P[i]); writestr(P[i]^.s,'test me ',i);end;
  14.   // de-allocate
  15.   for i := 99 downto 0 do begin writeln(P[i]^.s);Dispose(P[i]);end;
  16. end.
   

Aside:
It is really true that modern C allocators are faster than the default Freepascal manager. and threadsafe without locking, even scale better: they use a.o. lockfree patterns.
In the past this was not the case. Now it is the case for almost any application, not only threaded ones.

 
« Last Edit: August 21, 2019, 10:57:53 am by Thaddy »
Most people that want to use threading should learn to patch their jeans first: use a needle.

BrunoK

  • Full Member
  • ***
  • Posts: 174
  • Retired programmer
Re: Why CMem?
« Reply #39 on: August 21, 2019, 01:02:19 pm »
The info that I have is that a memory manager implementation for Delphi doesn't need to store length info.
When I subsequently rip out the length information from cmem in Freepascal that manager seems to work in Freepascal, but not in Lazarus.
Without knowing the Memsize of an assigned pointer, the ansistring, unicodestring and widestring would just NOT work in FPC, try recompiling FPC, any recent version without use of MemSize, you wont go very far.
Quote
So at least for those platforms there is technically no need to store the size, although they store the size of the allocated block and not the actual used length.
That is exactly what happens (and is exploited by FPC) in heap.inc. FPC system unit is not clean because it assumes that memory blocks are allocated rounded up to 16 bytes (see note in astrings.inc.NewAnsiString  that goes like   { request a multiple of 16 because the heap manager alloctes anyways chunks of 16 bytes } ). The comment itself is a lie because newansistring does not round up its request to upper 16 bytes bound, this is something I changed in my 3.0.6.

Heap.inc makes quite a few abusive assumptions about growing blocks. As an example when TFPList and siblings expand it may very well be that allocating additional memory isn't requested because Heap.inc has already decided to give much more memory, and it gets worse and worse as list grows.
Quote
Once I can drop the size, the memory manager can also have better alignment.
Except for special case, aligning on SizeOf(PtrUInt)  is quite sufficient.

Fails with Lazarus....works with Freepascal.
Code: Pascal  [Select]
  1. program testcmem ;
  2. {$if not declared(useheaptrc)}uses fastcmem;{$endif}{$H+}{$I-}
  3. type
  4.   TMyrec=record
  5.   s:string;
  6.   end;
  7.   PMyRec=^TMyrec;
  8. var
  9.   P:array[0..99] of PMyRec;
  10.   i:integer;
  11. begin
  12.   //allocate some memory
  13.   for i := 0 to 99 do begin New(P[i]); writestr(P[i]^.s,'test me ',i);end;
  14.   // de-allocate
  15.   for i := 99 downto 0 do begin writeln(P[i]^.s);Dispose(P[i]);end;
  16. end.
   
Works without a glitch on my W10 system (heap.inc or cmem and with/without my heaptrc  new stuff).
I don't know what is failing, I have not tried. It could even be that it's not MemSize that is the problem, but that some code tries to simply access the hidden size field. Either Thaddy has more info or you could play around with it yourself.
I unsderstand well what you suggest, (And imagine very well doing a hack myself :-) ) but that would be like searching for a needle in a haystack.
If Grumpy has an  example of failed program in lazarus, specially what units are used, it would help to spot a potential hack.
I've rebuild fpc and lazarus with cmem and the -msize change and my lazarus works OK.

heap.inc is stable, so it is good for general use. Also, since it is written in free pascal, one can examine the code and rebuild it.
Lazarus trunk r. 59978/03.01.2019 (+/- patches regarding enabled, TScrollBar, TCursorImage). FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903)

k1ng

  • New Member
  • *
  • Posts: 36
Re: Why CMem?
« Reply #40 on: August 21, 2019, 01:11:31 pm »
[I don't know what is failing, I have not tried. It could even be that it's not MemSize that is the problem, but that some code tries to simply access the hidden size field. Either Thaddy has more info or you could play around with it yourself.
The info that I have is that a memory manager implementation for Delphi doesn't need to store length info.
When I subsequently rip out the length information from cmem in Freepascal that manager seems to work in Freepascal, but not in Lazarus.

Report it to Lazarus bugtracker? >:D

Once I can drop the size, the memory manager can also have better alignment.
It is really true that modern C allocators are faster than the default Freepascal manager. and threadsafe without locking, even scale better: they use a.o. lockfree patterns.
In the past this was not the case. Now it is the case for almost any application, not only threaded ones.

So how about submitting your changes for FPC trunk? So everyone can profit from it in the future  ;D

Thaddy

  • Hero Member
  • *****
  • Posts: 8680
Re: Why CMem?
« Reply #41 on: August 21, 2019, 01:16:02 pm »
[Without knowing the Memsize of an assigned pointer, the ansistring, unicodestring and widestring would just NOT work in FPC, try recompiling FPC, any recent version without use of MemSize, you wont go very far.
As you can see from my code, I implemented it.
Quote
That is exactly what happens (and is exploited by FPC) in heap.inc. FPC system unit is not clean because it assumes that memory blocks are allocated rounded up to 16 bytes (see note in astrings.inc.NewAnsiString  that goes like   { request a multiple of 16 because the heap manager alloctes anyways chunks of 16 bytes } ). The comment itself is a lie because newansistring does not round up its request to upper 16 bytes bound, this is something I changed in my 3.0.6.
Yes. Indeed if I follow you correctly.
Quote
Heap.inc makes quite a few abusive assumptions about growing blocks. As an example when TFPList and siblings expand it may very well be that allocating additional memory isn't requested because Heap.inc has already decided to give much more memory, and it gets worse and worse as list grows.
Quote
Quote
Once I can drop the size, the memory manager can also have better alignment.
Except for special case, aligning on SizeOf(PtrUInt)  is quite sufficient.
Not the case on modern processors, depending on optimizations (sse, mmx,avx etc on x86 and vppX on arm.

About examples: any Lazarus project will fail with my unit on Linux but seems to work (with _msize) on windows.
My freepascal example simply works, even with types that store length.
« Last Edit: August 21, 2019, 01:18:47 pm by Thaddy »
Most people that want to use threading should learn to patch their jeans first: use a needle.

BrunoK

  • Full Member
  • ***
  • Posts: 174
  • Retired programmer
Re: Why CMem?
« Reply #42 on: August 21, 2019, 01:18:43 pm »
Actually I found at least 2 lines that could cause troubles they are
Code: Pascal  [Select]
  1. D:\fpc-laz-asus\Lazarus\laz-svn-trunk\components\sparta\generics\source\inc\generics.dictionaries.inc
  2. components\sparta\generics\source\inc\generics.dictionaries.inc (278,40) Result := PSizeInt(PByte(@((@Self)^))-SizeOf(SizeInt))^;
  3. components\sparta\generics\source\inc\generics.dictionaries.inc (1318,42) Result := SizeInt((@PByte(@((@Self)^))[-SizeOf(SizeInt)])^);
  4.  
Lazarus trunk r. 59978/03.01.2019 (+/- patches regarding enabled, TScrollBar, TCursorImage). FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903)

Thaddy

  • Hero Member
  • *****
  • Posts: 8680
Re: Why CMem?
« Reply #43 on: August 21, 2019, 01:19:35 pm »
 :) Yes, I found these too.... these are quite easy to patch, though. And indeed that makes my fastcmem fail too.(except on systems that also use negative offset, which happens to be the case)

@hnb
Maciej,  if you read this, you need to fix that anyway, because it relies on implementation detail of the memory manager.
« Last Edit: August 21, 2019, 01:31:55 pm by Thaddy »
Most people that want to use threading should learn to patch their jeans first: use a needle.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7359
Re: Why CMem?
« Reply #44 on: August 21, 2019, 03:30:02 pm »
Where to start.

1° cmem on windows. As Grumpy writes _msize can be used to retrieve the size of the allocated memory instead of having a header sizeuint containing the size.
Note that heap.inc has a similar approach in its own structures.
cmem using _msize. Tested with rebuilding FPC 3.0.4 latest (3.0.6 in my private branch) and rebuilding a relatively recent lazarus.

This would make the cmem interface variable. If for some reason you need a specific windows version of CMEM, just use make a windows specific unit of cmem. (WCmem)

 
Quote
MemSize MUST IMPERATIVELY BE IMPLEMENTED CORRECTLY in any memory manager otherwise the
following FPC routines would never work. They are
- fpc_AnsiStr_SetLength
- fpc_UnicodeStr_SetLength
- fpc_WideStr_SetLength

The least one can say is these are pretty commonly used ...

I mention them because I did a few changes (3.0.6) in these places to put all memory manager (that is heap.inc and cmem) on equal footing regarding string pre-overallocation.

And there you hit a problem that these are often used functions which a procvar would slow down. Which is probably why it is as it is now. (just doesn't show up in the simplistic benchmarks)