if you suspect heap and stack collision, you can find some information here [AVR] Investigating stack and heap management (https://github.com/ccrause/freepascal/wiki/%5BAVR%5D-Investigating-stack-and-heap-management). Also in AVR Freaks you can find information ( c only ) for stack painting (https://www.avrfreaks.net/forum/soft-c-avrgcc-monitoring-stack-usage) to determine if you have stack problem ( I think I have something like that written in Free Pascal. I'll take a look tomorrow when I return home .... )
Do you use the heap manager and/or recursive function/procedures ?
No recursive functions, only consts, vars, arrays, records.
< snip >
So finally my question: how can I see last location used with variables?
I can see current SP value in debugger, so last variable location would help me estimate how much space I have to 'free'.
No recursive functions, only consts, vars, arrays, records.
No recursive functions, only consts, vars, arrays, records.
Any strings (as distinct from shortstrings) or dynamic arrays?
MarkMLl
All arrays are with defined boundaries, but I have types declared like this:
can you share your AVR development setup ?
OS, FPC version ?
How do you debug your AVR project ?
AVRSim, hardware debugger ( snap, medbg, atmel ICE ), Microchip( Atmel ) Studio's ATBackend, gdb version ?
Also, if you use objects, IMHO, I recommend to switch to advancedrecords. I wasn't able to successfully debug objects when they're in different units ( I suspect bugs in dwarf information ).
So your TInt16_array is dynamic and goes on the heap, and is possibly reallocated (possibly resulting in fragmentation) during operation.
All arrays are with defined boundaries, but I have types declared like this:
i don't thinkthatyou can use dynamic array's without using the heapmgr unit ...
I use objects. Is there any other advantages/disadvantages of switching to advanced records (in regarding to AVR development)?
I use objects. Is there any other advantages/disadvantages of switching to advanced records (in regarding to AVR development)?
Objects or instances of classes? :-)
Old style object, like this:
< snip >
Also, if you use objects, IMHO, I recommend to switch to advancedrecords. I wasn't able to successfully debug objects when they're in different units ( I suspect bugs in dwarf information ).
I use objects. Is there any other advantages/disadvantages of switching to advanced records (in regarding to AVR development)?
Lazarus 2.3.0 (rev main-2_3-2515-g008af994c1) FPC 3.3.1 i386-win32-win32/win64 under Windows 11. Debug using atbackend.exe simulator described in your post here https://forum.lazarus.freepascal.org/index.php/topic,53342.0.html (https://forum.lazarus.freepascal.org/index.php/topic,53342.0.html) (btw. thanks for that :)).
I noticed one thing with constants: although I have {$WRITEABLECONST OFF} if I use typed constant like below they go into SRAM.
if you declare the constant as {$WRITEABLECONST OFF} ... the storage location will be the binary's rdata (aka read only data!) section
So your TInt16_array is dynamic and goes on the heap, and is possibly reallocated (possibly resulting in fragmentation) during operation.All arrays are with defined boundaries, but I have types declared like this:
i don't thinkthatyou can use dynamic array's without using the heapmgr unit ...
I must check in assembler, but I don't think it is used as dynamic array. Array is not declared as dynamic array, I just use this type so in my other unit I don't have to know size of the array.
Do values match?
73744 73744 73744
22 22 8
73746 73746 2
If I change lines 8 and 11 to the code below I got the same address.
I noticed one thing with constants: although I have {$WRITEABLECONST OFF} if I use typed constant like below they go into SRAM.
That doesn't sound right, since a few days ago Sven saidQuoteif you declare the constant as {$WRITEABLECONST OFF} ... the storage location will be the binary's rdata (aka read only data!) section
https://forum.lazarus.freepascal.org/index.php/topic,60466.msg452560.html#msg452560
MarkMLl
objdump -j .data -s project1 | grep ccbb
42a120 00000000 00000000 ddccbbaa 00000000 ................You did not show your actual code, so i can only be guessing how you do accessing with type override...
Again, you did not show the exact error.
I suggest you to google for ESR's "how to ask questions smart way" - it is a harsh reading, from a man who wrote "a right to be rude", but it is helpful.
Back to the AVR situation: currently FPC can either treat a const as a literal, or store the value in flash and copy to SRAM during startup. In principle read-only data could be accessed in flash, but there is no compiler support for this (at the moment).
accessor should be a typed pointer (with pointer math one) or a static array of maximum attainable size
Regarding the original question: __bss_end is address location which I was looking for - stack pointer must stay above it not to corrupt variable data.
From ccrause's github account:
No recursive functions, only consts, vars, arrays, records.
Any strings (as distinct from shortstrings) or dynamic arrays?
MarkMLl
In all units I have {$H-} on the top. Additionally, all strings are declared as String[10], String[16] or String[20].
All arrays are with defined boundaries, but I have types declared like this:
I use to access this array:
like this (Fvars_Int16 is pointer to vars_Int16):
Old style object, like this:
Just checking, because of the naming ambiguity :-)
I noticed one thing with constants: although I have {$WRITEABLECONST OFF} if I use typed constant like below they go into SRAM.
That doesn't sound right, since a few days ago Sven saidQuoteif you declare the constant as {$WRITEABLECONST OFF} ... the storage location will be the binary's rdata (aka read only data!) section
https://forum.lazarus.freepascal.org/index.php/topic,60466.msg452560.html#msg452560
I should have clarified this a bit more: the location is read only data if the OS supports it and the compiler utilizes it correctly. On AVR this currently isn't supported and on Linux there are problems with relocations if variables are put into the .rdata section (at least that's what I remember).
No recursive functions, only consts, vars, arrays, records.
Any strings (as distinct from shortstrings) or dynamic arrays?
MarkMLl
In all units I have {$H-} on the top. Additionally, all strings are declared as String[10], String[16] or String[20].
All arrays are with defined boundaries, but I have types declared like this:
I use to access this array:
like this (Fvars_Int16 is pointer to vars_Int16):
These are not compatible. Don't use this. If it works for you it's by accident and not guaranteed to continue working.
Regarding the original question: __bss_end is address location which I was looking for - stack pointer must stay above it not to corrupt variable data.
From ccrause's github account:
This may not always be accurate, at the time I didn't realize there is one more section that follows __bss_end, the .noinit (https://gitlab.com/freepascal.org/fpc/source/-/blob/main/compiler/systems/t_embed.pas#L1139) section. If variables are located in .noinit, then __bss_end does not point to the end of all allocated data. For reference, a more robust end of allocated data reference would be:
It is better to stick to shortstring instead of ansistring or others. On most if not all platforms this reserves just 256 byte or even less if declared properly.
E.g. if you know before hand what the maximum string length is, you can declare a shortstring type with the appropriate size:will occupy just 17 bytes on embedded targets. (one for size)
It is better to stick to shortstring instead of ansistring or others. On most if not all platforms this reserves just 256 byte or even less if declared properly.
E.g. if you know before hand what the maximum string length is, you can declare a shortstring type with the appropriate size:will occupy just 17 bytes on embedded targets. (one for size)
The compiler uses the fpc_shortstr_concat (https://gitlab.com/freepascal.org/fpc/source/-/blob/main/rtl/inc/generic.inc#L934) helper function to copy the source strings into the destination string [in this context string means shortstring for brevity]. The destination string is passed to this function via a var parameter with a shortstring type, i.e. the compiler has to create a temporary variable of type shortstring (as Thaddy mentioned) which will take up 256 bytes of stack space.
To avoid this large temporary variable, consider using move (https://www.freepascal.org/docs-html/rtl/system/move.html) to copy the string fragments directly into the destination. This way there is no temporary variable at play, but the programmer has to handle a couple more responsibilities to ensure you don't unintentionally overwrite data, find the correct offset into the destination, ensure the destination is correctly sized, and update the string length after adding data. Basically re-implement the functionality of fpc_shortstr_concat, but without the use of a temporary variable.
This is a good question. Since fpc_shortstr_concat can handle the situation where either of the two sources can also be the destination, the compiler can omit the temp in this case. However, in the disassembly listing @dseligo showed, the compiler is using a temporary shortstring to pass to fpc_shortstr_concat as the dests parameter, followed by a fpc_shortstr_to_shortstr call to copy the temporary string into the final destination. Seems like there is an opportunity for optimization here?The compiler uses the fpc_shortstr_concat (https://gitlab.com/freepascal.org/fpc/source/-/blob/main/rtl/inc/generic.inc#L934) helper function to copy the source strings into the destination string [in this context string means shortstring for brevity]. The destination string is passed to this function via a var parameter with a shortstring type, i.e. the compiler has to create a temporary variable of type shortstring (as Thaddy mentioned) which will take up 256 bytes of stack space.
If that is the case, why is that code comparing the address of the destination with those of the source strings?
MarkMLl
I did that, although I didn't use built-in move (I am careful about string sizes, but since I use strings only for LCD display I just cut if it doesn't fit):
This is a good question. Since fpc_shortstr_concat can handle the situation where either of the two sources can also be the destination, the compiler can omit the temp in this case. However, in the disassembly listing @dseligo showed, the compiler is using a temporary shortstring to pass to fpc_shortstr_concat as the dests parameter, followed by a fpc_shortstr_to_shortstr call to copy the temporary string into the final destination. Seems like there is an opportunity for optimization here?
Eventually I discovered what consumed my SRAM: it was string concatenation.I strongly recommend using PChar instead of ShortString in such a restricted environments.
After confirming that stack overwrites allocated data, with the compiler switch -at (together with -al) I easily found out how much SRAM was consumed for temporary variables in every unit. For each concatenation of strings, 256 bytes were consumed and in some units more than 1KB was reserved.
I solved it by making my own string concatenation procedures and now even AVRs with 2KB SRAM are enough for this project of mine.
Bottom line: be careful with strings in the embedded world.
I strongly recommend using PChar instead of ShortString in such a restricted environments.
Then use strlen(), strcopy(), strcat() from the strings unit. It will be a bit tedious at first, but you won't hit the curbs as you did now.
Yes.I strongly recommend using PChar instead of ShortString in such a restricted environments.
Then use strlen(), strcopy(), strcat() from the strings unit. It will be a bit tedious at first, but you won't hit the curbs as you did now.
What would you use for the base variables- array[something] of char?
It certainly sounds like a useful workaround, but the bottom line is that this shouldn't be happening: if OP really did define his strings as string[something] then they all have a fixed length and there should be no need of any temporaries... I can't see where shortstring has got into this.It happens because you can use assignments ( := ) and expressions with ShortString. Actually only the ( + ) operator is defined and you would say it can be easily translated, but (I suspect) the compiler don't bother for the associativity, commutativity, etc. of each type + (e.g. string concatenation and numeric addition have different properties) and makes it the easy way - first computes the expression in temporary, then makes the assignment. In the common case, the infix notation for expressions always may need some temporaries for the evaluation.
PASCALMAIN:
.Lc2:
.Ll1:
# [project1.lpr]
# [7] begin
call FPC_INIT_FUNC_TABLE
.Ll2:
# [8] s1 := s1 + s2;
ldi r18,lo8(TC_sPsPROJECT1_ss_S2)
ldi r19,hi8(TC_sPsPROJECT1_ss_S2)
ldi r20,lo8(TC_sPsPROJECT1_ss_S1)
ldi r21,hi8(TC_sPsPROJECT1_ss_S1)
ldi r24,lo8(TC_sPsPROJECT1_ss_S1)
ldi r25,hi8(TC_sPsPROJECT1_ss_S1)
ldi r22,5
mov r23,r1
call fpc_shortstr_concat
.Ll3:
# [9] writeln(s1);
call fpc_get_output
movw r2,r24
ldi r18,lo8(TC_sPsPROJECT1_ss_S1)
ldi r19,hi8(TC_sPsPROJECT1_ss_S1)
movw r20,r2
mov r22,r1
mov r23,r1
mov r24,r1
mov r25,r1
call fpc_write_text_shortstr
call fpc_iocheck
movw r24,r2
call fpc_writeln_end
call fpc_iocheck
.Ll4:
# [10] end.
call fpc_do_exitThis is a good question. Since fpc_shortstr_concat can handle the situation where either of the two sources can also be the destination, the compiler can omit the temp in this case. However, in the disassembly listing @dseligo showed, the compiler is using a temporary shortstring to pass to fpc_shortstr_concat as the dests parameter, followed by a fpc_shortstr_to_shortstr call to copy the temporary string into the final destination. Seems like there is an opportunity for optimization here?
Removal of pessimisation more like :-) My suspicion would be that the compiler is doing something odd to that it can get High() of the destination parameter... particularly since it's a var I certainly hope it's not doing spurious copies.
For the adventurous programmer, here is a proposed compiler enhancement that skips creating a temporary variable when assigning a shortstring concatenation expression to a shortstring variable: https://gitlab.com/ccrause/fpc-source/-/commits/shortstrconcat
It might be better to extend fpc_shortstr_concat (and maybe also fpc_shortstr_concat_multi) by a parameter that contains the length of the target variable and use a PShortString instead of a reference to a ShortString. This way the assignment can be done better in the general way.
The ( + ) operator isn't any good for defensive programming, something that should be practiced from all embedded programmers. I avoid using it even on a PC (+ have also other side effects that I'm not happy with).
Without underestimating your effort (that is an improvement on itself) for the embedded it shouldn't change the picture. One can get a false confidence and pretty common statements as:Indeed, this patch seems like a quick fix for a specific situation, not a general fix for other similar situations.will have the same (bad) effects. Again, with the left/right associativity of concat, it can be optimized, but it will require much more effort.
Simply calling a procedures for assignment and concatenation, even with a shortstrings, removes all that burden of the expression evaluation.This is good advice. I am used to using + for string concatenation, so would like to explore how much one can improve the situation. I still need to digest PascalDragon's comment about using PShortString, perhaps a general fix is not too difficult :o...
The ( + ) operator isn't any good for defensive programming, something that should be practiced from all embedded programmers. I avoid using it even on a PC (+ have also other side effects that I'm not happy with).
It might be better to extend fpc_shortstr_concat (and maybe also fpc_shortstr_concat_multi) by a parameter that contains the length of the target variable and use a PShortString instead of a reference to a ShortString. This way the assignment can be done better in the general way.
It's still unclear where shortstring- as distinct from string[something] is coming from.
QuoteSimply calling a procedures for assignment and concatenation, even with a shortstrings, removes all that burden of the expression evaluation.This is good advice. I am used to using + for string concatenation, so would like to explore how much one can improve the situation. I still need to digest PascalDragon's comment about using PShortString, perhaps a general fix is not too difficult :o...
The ( + ) operator isn't any good for defensive programming, something that should be practiced from all embedded programmers. I avoid using it even on a PC (+ have also other side effects that I'm not happy with).
I don't get why that's unclear. The function needs to work with any kind of ShortString and the only one that covers that is ShortString aka String[255] itself.
Just checked again and remembered that the compiler allows to pass a String[X] (with X < 255) to a var parameter of type ShortString so you wouldn't need to change that. But you as I had written you'd need to add the maximum length of the destination string as that gets lost then:That doesn't sound right. "var" parameters have to match exactly for good reason and one of them is not allowing passing a string[10] (or whatever other size) for a shortstring type which is supposed to be string[255].
Strict parameter enforcement would prevent the problem (bug?), but would also prevent the current ease of mixing different sized shortstrings. The ease of use is specifically mentioned in the documentation (https://www.freepascal.org/docs-html/ref/refsu9.html): Whatever the actual type, single byte strings can be used interchangeably. The compiler always takes care of the necessary type conversions.Just checked again and remembered that the compiler allows to pass a String[X] (with X < 255) to a var parameter of type ShortString so you wouldn't need to change that. But you as I had written you'd need to add the maximum length of the destination string as that gets lost then:That doesn't sound right. "var" parameters have to match exactly for good reason and one of them is not allowing passing a string[10] (or whatever other size) for a shortstring type which is supposed to be string[255].
... but if basic text handling can't be done safely this negates the value of using Pascal- particularly now that C/C++ have, generally speaking, taken on board Wirth's ideas about type checking etc. (as far as allowed by their modus operandi and the amount of legacy code).
MarkMLl
I don't get why that's unclear. The function needs to work with any kind of ShortString and the only one that covers that is ShortString aka String[255] itself.
But the destination /isn't/ a string[255], it's got a known maximum length which has to be respected.
I just tried it with Delphi 2 and Delphi 2 accepts passing a string[10] for a shortstring BUT, when the program is run, the Test function knows that only 10 characters fit in the string.
IOW, when high(s) is taken, the result is _not_ 255, it is 10, consequently the string constant moved into the parameter is truncated thereby leaving s2 and s3 unaffected (which is how it should be.)
I think FreePascal for embedded must be rewritten. And striped. That means a new project? I don't think it will happen...
BTW, how good is FreePascal at dead code removal? Personally, I think that the Unit system is already a burden for embedded...
That's why the compiler passes a temporary ShortString variable and then assigns that to the shorter string. Yes, that might mean that data gets lost then, but that's how short strings work!
That's why the compiler passes a temporary ShortString variable and then assigns that to the shorter string. Yes, that might mean that data gets lost then, but that's how short strings work!
I'm sorry, it's working wrong in that case: the size of the destination is known, and even if data loss is expected there's absolutely no need to move more than that number of characters.
But the fact that the result is truncated is not relevant here. What /is/ relevant is that the stack has to have the potential of being 256 bytes larger than expected, and that this requirement is multiplied if the overall program uses multiple threads or coroutines. And speaking as a sometime embedded system programmer, that is a grievous fault.
This is fundamental, 1980s stuff. And if it breaks something more recent like RTTI, then it's the more recent stuff that has to be adjusted.
If you have a function or method with a var parameter of type ShortString then it also passes the maximum size of the provided string variable as a hidden parameter. This is only done if the parameter is declared as ShortString, but not for any other String[<N>] even String[255]! Also this is not done for out parameters. Something like this also isn't done if the result is a ShortString.As @ccrause already mentioned above, it really seems that, in some cases, the extra hidden "max size" parameter to manage the string size correctly is a necessity.
If you have a function or method with a var parameter of type ShortString then it also passes the maximum size of the provided string variable as a hidden parameter. This is only done if the parameter is declared as ShortString, but not for any other String[<N>] even String[255]! Also this is not done for out parameters. Something like this also isn't done if the result is a ShortString.As @ccrause already mentioned above, it really seems that, in some cases, the extra hidden "max size" parameter to manage the string size correctly is a necessity.
It is better to stick to shortstring instead of ansistring or others. On most if not all platforms this reserves just 256 byte or even less if declared properly.
E.g. if you know before hand what the maximum string length is, you can declare a shortstring type with the appropriate size:will occupy just 17 bytes on embedded targets. (one for size)
Yes, I use shortstring with appropriate size and it occupies size + 1. Problem is when strings are concatenated.
Consider this program:
High(s) is correct if parameter is "const {/var/out} s: OpenString;",
regardless of {$P} directive.
... Note the highlighted stack allocations and also the missing “Temps allocated” line in the second example.
Regarding my feedback to ccrause: it turns out that the System unit contains an explicit $P+ in rtl/inc/systemh.inc which means that any of the helper functions take a var or out parameter of type ShortString already contains the correct High value of the parameter. It was simply a matter of implementing ccrause's change correctly and now (https://gitlab.com/freepascal.org/fpc/source/-/commit/02aac653d2599332b70af2cb3433e8f5d76e6400) FPC will handle the example by dseligo better:Multi string concatenation still has an issue, as illustrated by the following example:
# [20] s1 := s1 + s2 + s1;
ldi r18,lo8(TC_sPsPROJECT1_ss_S1)
ldi r21,hi8(TC_sPsPROJECT1_ss_S1)
ldi r24,lo8(8)
ldi r25,hi8(8)
add r24,r28
adc r25,r29
ldi r22,-1
mov r23,r1
mov r20,r18
call fpc_shortstr_to_shortstr
ldi r18,lo8(8)
ldi r19,hi8(8)
add r18,r28
adc r19,r29
std Y+2,r18
std Y+3,r19
ldi r20,lo8(TC_sPsPROJECT1_ss_S2)
ldi r21,hi8(TC_sPsPROJECT1_ss_S2)
ldi r24,lo8(264)
ldi r25,hi8(264)
add r24,r28
adc r25,r29
ldi r22,-1
mov r23,r1
call fpc_shortstr_to_shortstr
ldi r18,lo8(264)
ldi r19,hi8(264)
add r18,r28
adc r19,r29
std Y+4,r18
std Y+5,r19
ldi r20,lo8(TC_sPsPROJECT1_ss_S1)
ldi r21,hi8(TC_sPsPROJECT1_ss_S1)
ldi r24,lo8(520)
ldi r25,hi8(520)
add r24,r28
adc r25,r29
ldi r22,-1
mov r23,r1
call fpc_shortstr_to_shortstr
ldi r18,lo8(520)
ldi r19,hi8(520)
add r18,r28
adc r19,r29
std Y+6,r18
std Y+7,r19
ldi r20,lo8(2)
ldi r21,hi8(2)
add r20,r28
adc r21,r29
ldi r24,lo8(TC_sPsPROJECT1_ss_S1)
ldi r25,hi8(TC_sPsPROJECT1_ss_S1)
ldi r18,2
mov r19,r1
ldi r22,4
mov r23,r1
call fpc_shortstr_concat_multi
The strings on the RHS are copied to temporaries before being passed to fpc_shortstr_concat_multi. A bit more digging required...
... Note the highlighted stack allocations and also the missing “Temps allocated” line in the second example.
Nicely done. Leaving aside all consideration of compiler version and mode for a moment, is there any quick way, e.g. by using a SizeOf(), that something can be put into application code to report that an oversize temporary has been allocated, either at compilation or runtime?
The strings on the RHS are copied to temporaries before being passed to fpc_shortstr_concat_multi. A bit more digging required...
No, you need to look at the generated assembly code.
The strings on the RHS are copied to temporaries before being passed to fpc_shortstr_concat_multi. A bit more digging required...
Great, thank you! The patch works for the couple of cases I tested against.The strings on the RHS are copied to temporaries before being passed to fpc_shortstr_concat_multi. A bit more digging required...
Should be fixed (https://gitlab.com/freepascal.org/fpc/source/-/commit/bb51ac77dd8ba9ad3512f3552515f5ddcd051d11).