Recent

Author Topic: I can't see the error  (Read 6569 times)

TRon

  • Hero Member
  • *****
  • Posts: 2435
Re: I can't see the error
« Reply #15 on: August 02, 2020, 08:26:51 pm »
There can basically only be two instances where a dynamic array isn't freed:
I do not wish to confuse TS or other interested parties but currently it is actually 3.

I agree that you probably covered that by using the word "basically"  ;)

See Mantis #35536

kupferstecher

  • Hero Member
  • *****
  • Posts: 583
Re: I can't see the error
« Reply #16 on: August 02, 2020, 08:28:50 pm »
"Btw, the SetLength(UpperBoxAP,(i+1)) at the end is useless and should be removed because the array is already the correct size."

I'll remove it and see if works
I think it's required, otherwise the array might be larger than the actual content because of the blocksize.

@TE: Is this your actual code or did you strip it to the essential lines? If it's the actual code, then I guess you copied the code from somewhere and modified to your needs. You should try to understand what is happening in each line. The concept with blockwise expansion is used when the number of lines is not known at the beginning of the loop.

howardpc

  • Hero Member
  • *****
  • Posts: 4144
Re: I can't see the error
« Reply #17 on: August 02, 2020, 08:30:47 pm »
@JLWest
Can you share the code and data? It is not usually possible to help with debugging based only on a code snippet.

process_1

  • Guest
Re: I can't see the error
« Reply #18 on: August 02, 2020, 08:41:34 pm »
...

BTW, with such large amount of data, searching and filtering, why not consider using database, as SQlite, for instance? Then simply create SQL queries, table subests etc. Much, much easier approach....
« Last Edit: August 02, 2020, 08:46:50 pm by process_1 »

JLWest

  • Hero Member
  • *****
  • Posts: 1293
Re: I can't see the error
« Reply #19 on: August 03, 2020, 01:47:02 am »
@ process_1



 
SQL - Great idea and it should be in SQL. I don't know SQL, I have barley started to learn SQl and need the program finished in two weeks. Whats my chances with SQL?
 
FPC 3.2.0, Lazarus IDE v2.0.4
 Windows 10 Pro 32-GB
 Intel i7 770K CPU 4.2GHz 32702MB Ram
GeForce GTX 1080 Graphics - 8 Gig
4.1 TB

process_1

  • Guest
Re: I can't see the error
« Reply #20 on: August 03, 2020, 07:32:36 am »
Whats my chances with SQL?

They are great. Just ask here for any help and many people will help you.

But, regarding deadline, better to stick with your original plan if you are not familiar with SQL and SQlite.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: I can't see the error
« Reply #21 on: August 03, 2020, 09:20:51 am »
There can basically only be two instances where a dynamic array isn't freed:
I do not wish to confuse TS or other interested parties but currently it is actually 3.

I agree that you probably covered that by using the word "basically"  ;)

See Mantis #35536

Well, bugs not withstanding obviously. ;)

If shrink or delete list, it will remove array length, but not affected strings. You have to manually set empty string on all affected items before action, otherwise you will end up with massive memory leaks.

Ehm... no. Shrinking a dynamic array will also free managed elements. If it doesn't then this is a bug and should be reported.

Take this for example:

Code: Pascal  [Select][+][-]
  1. program tarrtest;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. procedure Test;
  6. var
  7.   arr: array of String;
  8.   s: String;
  9. begin
  10.   arr := ['Alpha', 'Beta', 'Gamma'];
  11.   for s in arr do
  12.     Writeln(s);
  13.   SetLength(arr, Length(arr) - 2);
  14.   for s in arr do
  15.     Writeln(s);
  16. end;
  17.  
  18. begin
  19.   Test;
  20. end.

Compiled with -gh to trace memory leaks the output will be:

Code: [Select]
Alpha
Beta
Gamma
Alpha
Heap dump by heaptrc unit of D:\fpc\git\testoutput\tarrtest.exe
10 memory blocks allocated : 197/232
10 memory blocks freed     : 197/232
0 unfreed memory blocks : 0
True heap size : 65536 (96 used in System startup)
True free heap : 65440

process_1

  • Guest
Re: I can't see the error
« Reply #22 on: August 03, 2020, 09:28:30 am »
Ehm... no. Shrinking a dynamic array will also free managed elements. If it doesn't then this is a bug and should be reported.

Nah, I'm not talking about freeing occupied memory using memory internal manager tables when program is finished. I'm talking about memory leak is cumulated while program is still running. Repeat that N times and look in system activity monitor how memory consumption rising constantly...

That is unavoidable, unless FPC use some automatic garbage collector...
« Last Edit: August 03, 2020, 09:33:06 am by process_1 »

rvk

  • Hero Member
  • *****
  • Posts: 6111
Re: I can't see the error
« Reply #23 on: August 03, 2020, 09:33:25 am »
If shrink or delete list, it will remove array length, but not affected strings. You have to manually set empty string on all affected items before action, otherwise you will end up with massive memory leaks.
Ehm... no. Shrinking a dynamic array will also free managed elements. If it doesn't then this is a bug and should be reported.
Not according to the documentation.

Quote
In case the length is set to a smaller length than the current one, the existing elements (characters in case of a string) are kept.
https://www.freepascal.org/docs-html/rtl/system/setlength.html

But strings are cleaned up at the end of the program (managed variable) so that's probably why you don't have a leak. Try to get the available memory during your program.

If your program runs a long time and you do this often you can run out of memory.


PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: I can't see the error
« Reply #24 on: August 03, 2020, 09:42:24 am »
Ehm... no. Shrinking a dynamic array will also free managed elements. If it doesn't then this is a bug and should be reported.

Nah, I'm not talking about freeing occupied memory using memory internal manager tables when program is finished. I'm talking about memory leak is cumulated while program is still running. Repeat that N times and look in system activity monitor how memory consumption rising constantly...

That is simply a side effect of the heap manager. In the most common case you have pages of size 4096 Byte that the OS is handing to the application. Now the heap manager might allocate multiple elements in there (e.g. there could be 4 different strings placed in there). Now if all allocations inside that page are freed the heap manager will release that page back to the OS. However if only one allocation is still in there (e.g. a string that you still hold onto) then the page will be kept. The heap manager will reuse the free space in such pages, but if it can't find suitable space then it will allocate a new page, leading to the behaviour that you observe.

If shrink or delete list, it will remove array length, but not affected strings. You have to manually set empty string on all affected items before action, otherwise you will end up with massive memory leaks.
Ehm... no. Shrinking a dynamic array will also free managed elements. If it doesn't then this is a bug and should be reported.
Not according to the documentation.

Quote
In case the length is set to a smaller length than the current one, the existing elements (characters in case of a string) are kept.
https://www.freepascal.org/docs-html/rtl/system/setlength.html

But strings are cleaned up at the end of the program (managed variable) so that's probably why you don't have a leak. Try to get the available memory during your program.

*rolls eyes* What the documentation tries to tell (and apparently fails to do) is that if you reduce the size of an array of length 10 to length 5 that the elements 0 to 4 will be kept. Any increase you might see in memory usage is otherwise an effect of what I mentioned above.

process_1

  • Guest
Re: I can't see the error
« Reply #25 on: August 03, 2020, 10:04:08 am »

That is simply a side effect of the heap manager. In the most common case you have pages of size 4096 Byte that the OS is handing to the application. Now the heap manager might allocate multiple elements in there (e.g. there could be 4 different strings placed in there). Now if all allocations inside that page are freed the heap manager will release that page back to the OS. However if only one allocation is still in there (e.g. a string that you still hold onto) then the page will be kept. The heap manager will reuse the free space in such pages, but if it can't find suitable space then it will allocate a new page, leading to the behaviour that you observe.

In good old days, when every byte of the RAM and every CPU clock was precious, you had to keep eye on everyting. That is what is sometimes long, long time ago called:  The art of programming.

Nowadays, with such large amount of GBs, so many GHz and CPU cores, good old manners are just forgotten, or even worst - ignored!

But I'm still thinking on every byte and optimize my code to the point it is the most efficient and keep resource as low as possible. That is also a reason why I'm working nowadays with MCUs without an issue even complex projects many people would say it is impossible to create on specific MCU...

The secret is to stick with good programming habbits all the time.

TRon

  • Hero Member
  • *****
  • Posts: 2435
Re: I can't see the error
« Reply #26 on: August 03, 2020, 10:33:22 am »
Well, bugs not withstanding obviously. ;)
Fair enough  :)

Thank you for the acknowledgement.

kupferstecher

  • Hero Member
  • *****
  • Posts: 583
Re: I can't see the error
« Reply #27 on: August 03, 2020, 01:54:55 pm »
if you reduce the size of an array of length 10 to length 5 that the elements 0 to 4 will be kept.
That leads to heap fragmentation, doesn't? On second thought its the same problem when the whole array is deleted, so don't mind. :-)

unit1.pas(1101,27) Fatal: Syntax error, "CREATE" expected but "ITEMS" found
The error message is quite confusing.
Use TStrings declaration and TStringList clas instance if you use Items. That is why compiler assumed you should create it first.
That explanation seems far fetched. Can someone explain (or confirm), why the compiler proposes a "Create"?

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: I can't see the error
« Reply #28 on: August 03, 2020, 02:54:17 pm »
if you reduce the size of an array of length 10 to length 5 that the elements 0 to 4 will be kept.
That leads to heap fragmentation, doesn't? On second thought its the same problem when the whole array is deleted, so don't mind. :-)

In any case the heap manager will try to reuse the fragment when a suitable new allocation comes in.

unit1.pas(1101,27) Fatal: Syntax error, "CREATE" expected but "ITEMS" found
The error message is quite confusing.
Use TStrings declaration and TStringList clas instance if you use Items. That is why compiler assumed you should create it first.
That explanation seems far fetched. Can someone explain (or confirm), why the compiler proposes a "Create"?

Seems like the compiler suggests the array constructor (TSomeNamedArray.Create(…)) even though it shouldn't in this case, cause it's a variable and not a type name.

Handoko

  • Hero Member
  • *****
  • Posts: 5131
  • My goal: build my own game engine using Lazarus
Re: I can't see the error
« Reply #29 on: August 03, 2020, 07:10:40 pm »
This code below does not cause any compile time error, tested on Lazarus 2.0.10. Is it a bug?

Code: Pascal  [Select][+][-]
  1. type
  2.   TNewType = array of string;
  3.  
  4. procedure TForm1.Button1Click(Sender: TObject);
  5. var
  6.   Z: TNewType;
  7. begin
  8.   Z := TNewType.Create;
  9.   Z := Z.Create;
  10. end;
« Last Edit: August 03, 2020, 07:13:11 pm by Handoko »

 

TinyPortal © 2005-2018