* * *

Author Topic: Is there a way to detect "Trash local variables" at compile time?  (Read 3116 times)

ASerge

  • Hero Member
  • *****
  • Posts: 965
Simple program:
Code: Pascal  [Select]
  1. {$MODE OBJFPC}
  2. {$APPTYPE CONSOLE}
  3.  
  4. type
  5.   TR = record
  6.     F: Integer;
  7.   end;
  8.  
  9. procedure Test(const R: TR; out O: TR);
  10. begin
  11.   Writeln('In Test ', R.F);
  12.   O := R;
  13. end;
  14.  
  15. var
  16.   R: TR;
  17. begin
  18.   R.F := 123;
  19.   Writeln('Before ', R.F);
  20.   Test(R, R);
  21.   Writeln('After ', R.F);
  22.   Readln;
  23. end;
Normal execution is:
Code: [Select]
Before 123
In Test 123
After 123
But if use -gt options (or "Trash local variables" in Lazarus/Options/Debugging), than behavior is changed. On my machine:
Code: [Select]
Before 123
In Test 1431655765
After 1431655765
I want to determine if this option is "on" at compile time to issue a warning that this can cause errors. Ideally, disable this option locally.

440bx

  • Sr. Member
  • ****
  • Posts: 332
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #1 on: July 12, 2018, 09:07:57 am »
Hi Serge,

After spending some time looking at the compiler source code, I don't believe there is a compiler directive that corresponds to that compiler command line switch.    If that is correct then the answer to your question is no.    Take that answer as "likely correct", I looked carefully but, I could have missed it.

On a related note, I think it is likely that you've found a bug because if you compile to produce a 32bit executable, you'll notice that it runs fine (it should since there are no locals to trash in your example program).  On the other hand, a 64bit executable gets the wrong results.  I was looking at the code the compiler generated and, in 64bit, it trashes the arguments (which are in registers).  I don't know if it would do that if your program had locals but, it definitely should never trash the arguments/parameters.

That's all I've got at this time.  Hopefully, it you'll find that information useful.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 4891
    • wiki
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #2 on: July 12, 2018, 09:27:13 am »
You are using "const" incorrectly.

https://www.freepascal.org/docs-html/ref/refsu67.html
Quote
Remark: Note that specifying const is a contract between the programmer and the compiler. It is the programmer who tells the compiler that the contents of the const parameter will not be changed when the routine is executed, it is not the compiler who tells the programmer that the parameter will not be changed.

You must make sure,  that while you are in "Test" nothing will change "R". The compiler may give you an error, if it detects that you try to break this promise. But you must obey it even if the compiler does not detect it.
E.g., if you would pass a global var, then you are not allowed to change that global var.

"Out O" also is passed by ref. That means if you pass the same variable to R and O, then you are not allowed to change O (because it would change R.
With -gt you change O.

The error you get is a warning that it is wrong to pass the same variable as R and O.

And again: With any future version of fpc it may be possible that your code breaks, even if you do not use -gt.

-----
In your simple (simplified?) example changing R (indirectly by changing O) at the end of the end of Test does no damage (with the current version of fpc / it may well crash even without -gt in future versions of fpc).

For reference, with most versions of fpc the below will print 12.
But with little changes, it can get worse, maybe even crash.
Code: Pascal  [Select]
  1. program Project1;
  2. {$mode objfpc}{$H+}
  3. var
  4.   Bar: String;
  5.  
  6. procedure Foo(const s: string);
  7. begin
  8.   Bar:= copy('123', 1,2); // changes s, not allowed
  9.   writeln(s)
  10. end;
  11. begin
  12.   Bar:=copy('abc', 1, 2);
  13.   Foo(Bar); // pass "ab"
  14.   readln;
  15. end.
  16.  

440bx

  • Sr. Member
  • ****
  • Posts: 332
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #3 on: July 12, 2018, 10:33:01 am »
"Out O" also is passed by ref. That means if you pass the same variable to R and O, then you are not allowed to change O (because it would change R.
Nice observation.  I totally missed that he was passing the same variable for both parameters.

With -gt you change O.
That I don't understand.  -gt is supposed to trash locals not arguments.  I don't see how or why the -gt switch would affect O (or any of the arguments for that matter, since they are not local variables, unless FPC considers arguments local variables, which would not make any sense.)

The error you get is a warning that it is wrong to pass the same variable as R and O.
But, when his example is compiled in 32bit, everything works fine (as I believe it should, in spite of his breaking the implied contract with the const parameter, since there are no locals in his function.)

Am I missing something ?


marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6579
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #4 on: July 12, 2018, 10:38:26 am »
That I don't understand.  -gt is supposed to trash locals not arguments. 

It trashes the out parameter before it reads the const value. Then the const value is read corrupted because they are the same.

Thaddy

  • Hero Member
  • *****
  • Posts: 7130
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #5 on: July 12, 2018, 10:53:13 am »
But, when his example is compiled in 32bit, everything works fine (as I believe it should, in spite of his breaking the implied contract with the const parameter, since there are no locals in his function.)
No it does not! If it worked for you that is by accident.
32 bit output here:
Code: Bash  [Select]
  1. pi@raspberrypi:~ $ fpc -gt trash.pas
  2. Free Pascal Compiler version 3.1.1-r39423 [2018/07/10] for arm
  3. Copyright (c) 1993-2018 by Florian Klaempfl and others
  4. Target OS: Linux for ARMHF
  5. Compiling trash.pas
  6. Linking trash
  7. 23 lines compiled, 0.8 sec
  8. pi@raspberrypi:~ $ ./trash
  9. Before 123
  10. In Test 1431655765
  11. After 1431655765
  12.  

Also note that out implies local.
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

440bx

  • Sr. Member
  • ****
  • Posts: 332
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #6 on: July 12, 2018, 01:22:45 pm »
No it does not! If it worked for you that is by accident.
Your "example" only shows that the compiler generates different code for a different CPU.  That's what you're calling an accident. 

Also note that out implies local.
That statement implies something very different.   Locals are variables for which space is reserved when setting up the stack frame.  All parameters, whether var, out, or value, unless they are being passed in registers, are already on the stack  _before_ the stack frame is set up, which means they are most definitely not locals (they could be locals in the caller's frame but certainly not in the callee's.)

Parameters/arguments are local in the sense of their visibility NOT their allocation.  Here is a stack frame for you:
Code: [Select]
trashlocalsdelphi.dpr.29: begin
00405A7C 55               push ebp
00405A7D 8BEC             mov ebp,esp
00405A7F 51               push ecx

trashlocalsdelphi.dpr.30: AVar := $ABCDEF12;
00405A80 C745FC12EFCDAB   mov [ebp-$04],$abcdef12  <- THAT is a local
                                                      negative offset from ebp. 

trashlocalsdelphi.dpr.34: Writeln('In Test ', R.F);
00405A87 A118784000       mov eax,[$00407818]
00405A8C BAC45A4000       mov edx,$00405ac4
00405A91 E802E5FFFF       call @Write0UString
00405A96 8B5508           mov edx,[ebp+$08]
00405A99 E81AE3FFFF       call @Write0Long
00405A9E E8F5E5FFFF       call @WriteLn
00405AA3 E860D3FFFF       call @_IOTest
trashlocalsdelphi.dpr.37: O := R;
00405AA8 8B450C           mov eax,[ebp+$0c]        <- parameters, positive
00405AAB 8B5508           mov edx,[ebp+$08]        <- offset from ebp (unless in a register)
00405AAE 8910             mov [eax],edx
trashlocalsdelphi.dpr.40: end;
00405AB0 59               pop ecx
00405AB1 5D               pop ebp                  <- tear down the frame
00405AB2 C20800           ret $0008                <- get rid of arguments

From the Free Pascal Reference Guide, page 191:
The difference of out parameters and parameters by reference is very small (however, see below
for managed types): the former gives the compiler more information about what happens to the
arguments when passed to the procedure: it knows that the variable does not have to be initialized
prior to the call.


Your statement implies that var parameters are also locals, which is downright absurd since it would imply that space for them is allocated on the callee's stack before the procedure has even been called.  That would be quite an accident.   Locally visible and locally allocated are very different things.  Arguments/parameters are locally visible but _not_ locally allocated, as such they can _never_ be the target of a routine designed to trash locals.

Here is something that would be very good reading for you:

https://software.intel.com/en-us/articles/intel-sdm

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6579
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #7 on: July 12, 2018, 01:50:11 pm »
Your statement implies that var parameters are also locals,

Var parameters have a defined value on entry. OUT parameters not, hence that the fuzzer also randomizes them.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 4891
    • wiki
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #8 on: July 12, 2018, 02:09:36 pm »
Also note that out implies local.
No they are not. But as Marcov wrote, they (like locals) have no defined values when the procedure is entered.

-gt is to help detecting when an app uses such undefined/uninitialized values.
So it is only right that out param are affected.

If anything, the doc may need an update to mention this.

440bx

  • Sr. Member
  • ****
  • Posts: 332
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #9 on: July 12, 2018, 02:52:30 pm »
Var parameters have a defined value on entry. OUT parameters not, hence that the fuzzer also randomizes them.

That's reasonable and it is a nice feature to ensure that the caller gets a "return" value but, an out parameter isn't a variable allocated in the callee's stack.   The -gt switch, as documented, is used to ensure that local variables (by definition, allocated on the callee's stack) are initialized before they are used by the callee, or to make it obvious that one or more of them weren't before they were used.  It's nice that it does that for out parameters as well but, it is beyond its document purpose.

It's also a bit problematic to implement the feature that way because its proper (say, consistent) operation depends on how the compiler has chosen to pass the parameters and the sequence in which the "fuzzer" is going to scramble the values.  The 32bit version of the program is a good example, the const is passed by value, the out by reference, the out gets scrambled but it doesn't scramble the parameter passed by value, that's why the 32bit version works, because a unique location has effectively been split into two separate locations.  The 64bit version passes the parameters in registers which ends "working" in the sense that the unique value does get scrambled.

Anyway, as a result of looking into this problem, I noticed that the fuzzer uses 1 of 3 values (if I recall correctly) none of which is particularly interesting.  A bunch of 5s sometimes, a bunch of As in other occasions.  Microsoft chose to use $CC, which is the opcode for int 3.  If the stack gets corrupted, when the ret is executed, it's likely, though, not guaranteed of course, that the code will end up executing a breakpoint, which is good because, it's consistent.  A bunch of 5_s or a bunch of A_s could end up doing god knows what.  For that reason, I think the FPC team should consider using $CC to scramble the values.  Ends up being simpler too.  Always use one value instead of picking one out of three.

Peace.

ETA:

If anything, the doc may need an update to mention this.
I was typing my reply while you posted yours.  I wholeheartedly agree with your suggestion.
« Last Edit: July 12, 2018, 03:13:21 pm by 440bx »

ASerge

  • Hero Member
  • *****
  • Posts: 965
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #10 on: July 12, 2018, 02:58:29 pm »
You are using "const" incorrectly.
...
You must make sure,  that while you are in "Test" nothing will change "R".
The const parameter in the procedure is not changed.
Quote
"Out O" also is passed by ref. That means if you pass the same variable to R and O, then you are not allowed to change O (because it would change R.
With -gt you change O.
I don't agree. It's just a side effect. If the procedure is written correctly, there is no problem. I have never seen such a restriction.
But the fact that the procedure erases the contents of the memory by the out parameter pointer looks bad. But even so, the impossibility of excluding it is even worse.
Moreover, such "incorrect" use is of course the problem of the caller, but it occurs inside the correct code.
And besides, it's inconsistent. This is the code that runs the same regardless of option -gt
Code: Pascal  [Select]
  1. {$APPTYPE CONSOLE}
  2.  
  3. var
  4.   Vglb: Integer = 1;
  5.  
  6. procedure Test(const Vin: Integer; out Vout: Integer);
  7. begin
  8.   Vout := 2;
  9.   Writeln('In test: ', Vin);
  10. end;
  11.  
  12. begin
  13.   Test(Vglb, Vglb);
  14.   Writeln('After: ', Vglb);
  15.   Readln;
  16. end.

440bx

  • Sr. Member
  • ****
  • Posts: 332
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #11 on: July 12, 2018, 03:10:28 pm »
And besides, it's inconsistent.
Consistency, as you pointed out, that's the real problem with having -gt fuzz out parameters. 


This is the code that runs the same regardless of option -gt

Code: [Select]
{$APPTYPE CONSOLE}

var
  Vglb: Integer = 1;

procedure Test(const Vin: Integer; out Vout: Integer);
begin
  Vout := 2;
  Writeln('In test: ', Vin);
end;

begin
  Test(Vglb, Vglb);
  Writeln('After: ', Vglb);
  Readln;
end.
That always works because you're not assigning one parameter to the other.  You are returning a value, which is not dependent on the other parameter, that should make -gt happy and, as expected, it does.

Thaddy

  • Hero Member
  • *****
  • Posts: 7130
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #12 on: July 12, 2018, 03:28:18 pm »
Trashing out is the only thing that is consistent. That's GOOD not bad.
How else can you determine if an out parameter's behavior is correct?

If you happen to feed out from a const parameter I would expect a warning, though.
Hence I use the term "implied" local, not "inferred" local.
The former means "treat it as locally initialized".
The latter means "needs to be locally initialized".

Those are two different things, but in both cases -gt should trash out.
« Last Edit: July 12, 2018, 03:31:03 pm by Thaddy »
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6579
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #13 on: July 12, 2018, 03:45:57 pm »
Microsoft chose to use $CC, which is the opcode for int 3.  If the stack gets corrupted, when the ret is executed, it's likely, though, not guaranteed of course, that the code will end up executing a breakpoint, which is good because, it's consistent. 

Stack corruption with FPC is rarely because of buffer overflows, since dynamic arrays, strings and classes are all on the heap. 5 and A are both 101 bit patterns, which are easy to spot.

Thaddy

  • Hero Member
  • *****
  • Posts: 7130
Re: Is there a way to detect "Trash local variables" at compile time?
« Reply #14 on: July 12, 2018, 04:12:21 pm »
int3 is an intel implementation, not microsoft's suggestion. It is cross-platform for Intel x86 families.
I used it -from Delphi 2!!! - to use in a memory-manager to detect heap corruption, so not stack as is the case in point.
https://en.wikipedia.org/wiki/INT_%28x86_instruction%29

It also works with GDB and linux or FreeBSD, likely all others for x86 families.
« Last Edit: July 12, 2018, 04:19:45 pm by Thaddy »
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus