Recent

Author Topic: SetLength() is not reliable?  (Read 1127 times)

senglit

  • New Member
  • *
  • Posts: 12
SetLength() is not reliable?
« on: August 21, 2019, 03:35:47 pm »
I have several dynamic arraies of record in my program. All of the records have some members of string. I use SetLength to resize these arries in my program. And I found those arries who were been resized for around 10 times (the size grow 1 each time) are ok. but others who are resized for more than 40 times (size grow from 0 to 40+), the program throw a memory violation and crushed.

Is SetLength() reliable? or there are somewhere I can set the stack size (or something other) to avoid this kind of crush?

julkas

  • Sr. Member
  • ****
  • Posts: 348
  • KISS principle / Lazarus 2.0.0 / FPC 3.0.4
Re: SetLength() is not reliable?
« Reply #1 on: August 21, 2019, 03:43:12 pm »
Is SetLength() reliable?
Yes it is.
Please give more details about , provide code example (source).
procedure mulu64(a, b: QWORD; out clo, chi: QWORD); assembler;
asm
  mov rax, a
  mov rdx, b
  mul rdx
  mov [clo], rax
  mov [chi], rdx
end;
(* Pointer game *) Inc(ptr, 1); (* vs *) ptr := ptr + 1;

Handoko

  • Hero Member
  • *****
  • Posts: 3121
  • My goal: build my own game engine using Lazarus
Re: SetLength() is not reliable?
« Reply #2 on: August 21, 2019, 03:45:05 pm »
My programs use SetLength, so far it woks perfectly.

Thaddy

  • Hero Member
  • *****
  • Posts: 8664
Re: SetLength() is not reliable?
« Reply #3 on: August 21, 2019, 03:53:07 pm »
Woks?
Yes it cooks fine and taste good too  O:-)
serious: I am not aware setlength() has problems. That would have showed up may moons before.

Note that increasing length continuously by one is a problem (of the programmer): every time a relocation is not sane programming.
« Last Edit: August 21, 2019, 03:57:12 pm by Thaddy »
Most people that want to use threading should learn to patch their jeans first: use a needle.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5566
    • wiki
Re: SetLength() is not reliable?
« Reply #4 on: August 21, 2019, 04:13:20 pm »
And I found those arries who were been resized for around 10 times (the size grow 1 each time) are ok. but others who are resized for more than 40 times (size grow from 0 to 40+), the program throw a memory violation and crushed.

SetLength is fine. You seem to simply run out of memory.

The short answer: Use TStringList (for strings, otherwise TList or similar), instead of array.

The long answer:
If you do "SetLength(MyArray, length(MyArray)+1);" (or any other value instead of +1)
Then the old memory is "freed" That is, the old memory is added to a pool of "memory that can be re-used". So in your OS you will not see memory shrinking (doing so would be slower).

If you do a lot of such "SetLength", then they will all add there old memory to that pool. They will try to re-use memory from that pool too. But that pool can become fragmented. Then it has lots of small bits of memory (summing up to a huge amount), each to small to be re-used.... => And your app keeps using new mem, until there is none left.

So instead of doing 10 times "+1", you should do 1 time "+10". Pre-allocate what you will need later.
TList/TStringList have that build in (it's called capacity).

senglit

  • New Member
  • *
  • Posts: 12
Re: SetLength() is not reliable?
« Reply #5 on: August 21, 2019, 04:22:19 pm »
there is the main part of the code:

Code: Pascal  [Select]
  1. type
  2.   PRecDevInfo=^RecDevInfo;
  3.   RecDevInfo = record  
  4.     DevId:integer;
  5.     Name:string;
  6.     Model:string;
  7.     ConnTo:string;
  8.     ComPort: integer;
  9.     ComListId:integer;
  10.     Page:integer;      
  11.     dType:integer;    
  12.     Addr: integer;
  13.     IsOn:boolean;
  14.     PlcData:array[1..8] of char;
  15.     PlcAddr:array[1..8] of integer;
  16.     Value1:real;
  17.     Value2:real;
  18.     Range1:real;
  19.     Range2:real;
  20.   end;          
  21.  
  22. var
  23.   DevCount:integer=0;
  24.   MyDev:array of RecDevInfo;    
  25.  
  26. function TFrmMain.LoadCmdFile():boolean;
  27. var
  28.   MyIniFile:TextFile;  
  29.   LineNow:integer=0;
  30.   s,tmp:string;
  31.   strs:TStrings;                                                    
  32. begin
  33.   SetLength(MyDev,1);    
  34.   MyDev[0].Name:='head';
  35.   MyDev[0].IsOn:=false;
  36.   MyDev[0].DevId:=0;
  37.   MyDev[0].Model:='empty';
  38.   AssignFile(MyIniFile,'sys.pzwj');
  39.   Reset(MyIniFile);
  40.   strs:=TStringList.Create;  
  41.   try
  42.     while not eof(MyIniFile) do
  43.       begin
  44.         inc(LineNow);
  45.         Readln(MyIniFile,s);    
  46.       s:=UpperCase(Trim(s));
  47.       if Length(s)=0 then continue;
  48.       if leftStr(s,1)='#' then continue;
  49.       strs.DelimitedText:=s;
  50.       strs.Delimiter:=#9;
  51. //check here, if the format of MyIniFile is wrong ,return false
  52.           inc(DevCount);
  53.           SetLength(MyDev,DevCount+1);          
  54.           MyDev[DevCount].DevId:=DevCount;
  55.           MyDev[DevCount].Name:=strs[0];
  56.           MyDev[DevCount].ComPort:=StrToInt(strs[1]);
  57.           MyDev[DevCount].Addr:=StrToInt(strs[2]);
  58.           MyDev[DevCount].Model:=strs[3];
  59.           MyDev[DevCount].Page:=strtoint(strs[4]);
  60.           MyDev[DevCount].dType:=strtoint(strs[5]);
  61.           MyDev[DevCount].ConnTo:=strs[6];
  62.           MyDev[DevCount].IsOn:=false;
  63.           ...... the strs.count may go to 20
  64.     end;
  65.   finally
  66.     CloseFile(MyIniFile);
  67.   end;
  68.   result := true;
  69. end;              
  70.  

After I declear the array MyDev as "MyDev:array[0..127] of RecDevInfo;" and not use setlength, it goes ok. What's the wrong thing did I do?

Thaddy

  • Hero Member
  • *****
  • Posts: 8664
Re: SetLength() is not reliable?
« Reply #6 on: August 21, 2019, 04:51:06 pm »
After I declear the array MyDev as "MyDev:array[0..127] of RecDevInfo;" and not use setlength, it goes ok. What's the wrong thing did I do?
Ah, much simpler than you explained before:

You can not call setlength() on a static array, only on dynamic arrays. MyDev:array[0..127] is a static array, not a dynamic array......
Code: Pascal  [Select]
  1. var
  2.   MyDev:array of RecDevInfo; //dynamic array
  3. begin
  4.   Setlength(MyDev:array, 128); // set the size to 128
Should solve it.
« Last Edit: August 21, 2019, 04:54:22 pm by Thaddy »
Most people that want to use threading should learn to patch their jeans first: use a needle.

rvk

  • Hero Member
  • *****
  • Posts: 3798
Re: SetLength() is not reliable?
« Reply #7 on: August 21, 2019, 04:53:44 pm »
After I declear the array MyDev as "MyDev:array[0..127] of RecDevInfo;" and not use setlength, it goes ok. What's the wrong thing did I do?
Ah, much simpler than you explained before:

You can not call setlength() on a static array, only on dynamic arrays. MyDev:array[0..127] is a static array, not a dynamic array......
That's NOT what (s)he said.

If you read carefully it says running it with a static array, it all works ("when NOT using setlength").
I highlighted the part for you in the quoted post.

The question would by, "Why does SetLength() crap out with just 128 iterations of +1".

But that could depend on the size of the record.

(And you're correct in saying +1 in a loop isn't good programming practice, as explained earlier)
« Last Edit: August 21, 2019, 04:56:30 pm by rvk »

Thaddy

  • Hero Member
  • *****
  • Posts: 8664
Re: SetLength() is not reliable?
« Reply #8 on: August 21, 2019, 04:55:29 pm »
After I declear the array MyDev as "MyDev:array[0..127] of RecDevInfo;" and not use setlength, it goes ok. What's the wrong thing did I do?
Ah, much simpler than you explained before:

You can not call setlength() on a static array, only on dynamic arrays. MyDev:array[0..127] is a static array, not a dynamic array......
That's NOT what (s)he said.

If you read carefully it says running it with a static array, it all works ("when NOT using setlength").
I highlighted the part for you in the quoted post.

I read it and interpreted it and tested it, Rik.  This is what one means.
Furthermore his/hers inc() is premature... should after . Likely also  the case.
« Last Edit: August 21, 2019, 04:58:45 pm by Thaddy »
Most people that want to use threading should learn to patch their jeans first: use a needle.

rvk

  • Hero Member
  • *****
  • Posts: 3798
Re: SetLength() is not reliable?
« Reply #9 on: August 21, 2019, 04:57:58 pm »
I read it and interpreted it and tested it, Rik.  This is what one means.
One never said SetLength is called on a static array.

One said, when using a static array "and NOT using SetLength", all works correctly.

Thaddy

  • Hero Member
  • *****
  • Posts: 8664
Re: SetLength() is not reliable?
« Reply #10 on: August 21, 2019, 05:00:10 pm »
Look at the place of the inc() in that case.... initialized to 0, should be -1 or performed later in the code. It is really bad code anyway, but therefor we put the time in... O:-)
Most people that want to use threading should learn to patch their jeans first: use a needle.

FTurtle

  • Sr. Member
  • ****
  • Posts: 259
Re: SetLength() is not reliable?
« Reply #11 on: August 21, 2019, 05:51:19 pm »
DevCount is global initialized variable that permanently grows.
What is it?!

FTurtle

  • Sr. Member
  • ****
  • Posts: 259
Re: SetLength() is not reliable?
« Reply #12 on: August 21, 2019, 06:41:30 pm »
Code: Pascal  [Select]
  1.       strs.DelimitedText:=s;
  2.       strs.Delimiter:=#9;
  3.  

Does it really work?

1. First set Delimiter and only after - DelimitedText (the text which will processed with respect of setted earlier Delimiter)
2. Check strs.Count before things like MyDev[DevCount].ConnTo:=strs[6];



Munair

  • Sr. Member
  • ****
  • Posts: 468
  • Try to KISS (keep it simple, smart)
    • Ditrianum
Re: SetLength() is not reliable?
« Reply #13 on: August 21, 2019, 07:13:13 pm »
Note that increasing length continuously by one is a problem (of the programmer): every time a relocation is not sane programming.

There are situations where increasing by one is the only solution. You only need to take a look at UTF8LowerCase() where it is used extensively. Here is a snippet from that function:
Code: Pascal  [Select]
  1. begin
  2.           p:=OutStr - PChar(Result);
  3.           SetLength(Result,Length(Result)+1);// Increase the buffer
  4.           OutStr := PChar(Result)+p;
  5.           OutStr^ := #$C4;
  6.           inc(OutStr);
  7.           OutStr^ := #$B1;
  8.           dec(CounterDiff);
  9.         end

AFAIK even with large texts, it doesn't give any problems (as long as there is enough memory).
« Last Edit: August 21, 2019, 07:35:04 pm by Munair »
Lazarus 2.0.4; Manjaro, Debian, Windows

Handoko

  • Hero Member
  • *****
  • Posts: 3121
  • My goal: build my own game engine using Lazarus
Re: SetLength() is not reliable?
« Reply #14 on: August 21, 2019, 07:23:36 pm »
@senglit

Can you please provide the source code that we can compile and test? Usually with compile-able code to test and inspect, I can debug and find the trouble line in hours. So do the many other seniors here.

Create a new folder, copy and paste all the necessary files except: the binary (exe file), *.bak, lib and backup folders. Compress the folder and send the zip file here. If you're not willing to publicize your project, you can write a demo that shows the issue.

Woks?
Yes it cooks fine and taste good too  O:-)

I was in the middle of having my meal when posting the post, so I unconsciously typed the word related with it.  :D
« Last Edit: August 21, 2019, 07:29:26 pm by Handoko »