Recent

Author Topic: "Ghost Lining"  (Read 10906 times)

RedOctober

  • Sr. Member
  • ****
  • Posts: 450
"Ghost Lining"
« on: December 03, 2021, 05:43:24 pm »
This is not a bug report.  It is merely an observation, and a question to see if anyone else has run into this.  My platform is Lazarus 2.0.10, FPC 3.2.0, Windows Server 2016.  This exact same problem occurred in Delphi (years ago), and now Lazarus (yesterday).

Description of problem:

During run time, in the production environment, a bug appears, with error dialog displayed. Bug can be recreated consistently.  Should be easy to find, just retrace steps.  Open project in dev environment, Release mode, run, retrace steps, bug appears as expected.  Close, switch compiler to Default mode, retrace steps, app works perfectly, no bug.  Repeat this several times, ensuring there are no compiler directives that would remove or add lines of code between the Release and Default modes. 

After several minutes, several iterations, it is determined that the only difference is the compiler mode.  Source code is correct.

Cure:

Simply add at least ONE line of code, even if it does nothing.  Recompile, in Release mode, bug goes away.

Conclusion:

Something in the compiler (Delphi or Lazarus) is skipping a line of source code, or, compiling it wrong.

In my Delphi project I had the following code:
Code: Pascal  [Select][+][-]
  1.   if Length(s) > 5 then  // <-- THIS LINE
  2.     begin
  3.       ShowMessage('I am greater than five');
  4.     end
  5.  

The line marked THIS LINE would ALWAYS resolve to FALSE even if the length of s was more than five.  The cure was to add a comment line above this section, like this.

Code: Pascal  [Select][+][-]
  1.   // By adding this comment, above the IF, the line below started working
  2.   if Length(s) > 5 then  // <-- THIS LINE
  3.     begin
  4.       ShowMessage('I am greater than five');
  5.     end
  6.  

In the Lazarus example:

  (in the database the AVL field is defined as NOT NULL with constraint IN('N','Y')

Code: Pascal  [Select][+][-]
  1.   qryStock.ParamByName('STOCK_NUMBER').AsString := my_stock_nmbr;
  2.   qryStock.ParamByName('DSC').AsString := my_item_description;
  3.   qryStock.ParamByName('AVL').AsString := my_item_available;  // <-- THIS LINE
  4.  
  5.   ... commit to database
  6.  
  7.  

In dev environment, compiling to Default mode, run, everything works.  Compiling to Release mode, run, the line marked THIS LINE is skipped, the AVL parameter is left NULL and an error is displayed upon attempt to commit to database.

Cure:

Code: Pascal  [Select][+][-]
  1.   qryStock.ParamByName('STOCK_NUMBER').AsString := my_stock_nmbr;
  2.   qryStock.ParamByName('DSC').AsString := my_item_description;
  3.   qryStock.ParamByName('AVL').AsString := my_item_available;  
  4.  
  5.   // I add this section AFTER the lines above
  6.   Screen.Cursor := crHourGlass;
  7.   Application.ProcessMessages;
  8.   try
  9.     ... commit to database
  10.  
  11.   finally
  12.     Screen.Cursor := crDefault;
  13.     Application.ProcessMessages;
  14.   end;
  15.  
  16.  

Now, this app works when compiled to Default mode OR Release mode.  All I did was add a few lines of code, or a comment, which SHIFTED the lines of code, and changed the total line count, then everything worked reliably.

Now that I am aware of this, I am not stumped for hours trying to resolve it.  If none of my code has changed, and there are no compiler directives that are used or changed between Default and Release modes... I just add a comment or a few lines of code, and the problem goes away.

This has happened to me once in Delphi and now twice in Lazarus, about a year or more apart.

Just wondering if anyone else has run into this, and what, if anything, I am doing wrong, or not looking for that I should be looking for.








SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: "Ghost Lining"
« Reply #1 on: December 03, 2021, 06:23:39 pm »
Yes, after some serious debugging, sometimes things go wrong. Resetting the debugger after each use can help, otherwise restarting Lazarus. Sometimes rebooting, but that is rare. Most of the time, it works very well.

My main complaint with debugging is the limited view of the insides of objects.

And a new, native FPC debugger is in the works, I understand. So feel free to jump in :)

RedOctober

  • Sr. Member
  • ****
  • Posts: 450
Re: "Ghost Lining"
« Reply #2 on: December 03, 2021, 07:14:29 pm »
Interesting Jamie, thanks. That kinda sounds close to what is going on.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9755
  • Debugger - SynEdit - and more
    • wiki
Re: "Ghost Lining"
« Reply #3 on: December 03, 2021, 07:48:37 pm »
Well first of all the info about the issue is rather vague.

The usual suspects....

- Release / Debug mode have diff optimizer settings.
It is possible that faulty user code works with one optimizer setting but not another. Memory layout changes can lead to the exposure of bugs with dangling data or invalid/incorrect pointers or range violation... E.g. the data *after* an array may be allocated, unused and hold a value that matches the expected value. Or not.
Inserting dummy code, can fix such bugs (because the also change memory layout). Comments would have no effect on this.


In the same way, bug in the compilers optimizer can be triggered. And valid code is suddenly incorrectly compiled. (There is one of those in 3.2.0 and 3.2.2, fixed in 3.2.3 - but that is really hard to trigger - it's in the peephole opt, so -OoNoPeephole or whatever the opt is.)
Here too, dummy code can make a diff, because it changes what can be optimize.

- Cross unit compiler issues.
If for some obscure reason some unit wasn't recompiled, but another unit used by the first was changed (and compiled), then that not recompiled unit is in trouble. In that case, even inserting comments could trigger the recompilation.
Monitor the timestamps of your ppu files.

Note: There is a real danger of that going wrong, if your project, or any package has custom path for: searching units, libs, inc, or outputing ppu (other than including target/buildmode in the ppu output).
I have seen that happen to many people. As soon as any directory  is accessed by more that a single package (the project counting as single package too).

----
You can always save the bad exe. Add the comments, and compare the files. Disassemble with objdump or the like.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: "Ghost Lining"
« Reply #4 on: December 03, 2021, 08:00:40 pm »
Interesting Jamie, thanks. That kinda sounds close to what is going on.

A Heisenbug. One thing you can do- rather more easily with FPC/Lazarus than with Delphi- is set up a new development system and copy your source files over by hand. If you're up to it compile FPC and Lazarus from source, and definitely don't trust any VCS in case it's accidentally got a binary library/object file copied into it and it's that that's screwing things up.

And it goes without saying that anything you can do to serialise interaction between threads is worth doing, even if that impacts performance beyond what would be acceptable for production code.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11353
  • FPC developer.
Re: "Ghost Lining"
« Reply #5 on: December 03, 2021, 09:55:13 pm »

Run a riot!  Compile with all checks enabled "-CRriot -gt"


RedOctober

  • Sr. Member
  • ****
  • Posts: 450
Re: "Ghost Lining"
« Reply #6 on: December 04, 2021, 12:22:29 am »
Ya, it's totally weird.  Thanks for the tip about the external debug file. I'm going to use that. 

Mr.Madguy

  • Hero Member
  • *****
  • Posts: 844
Re: "Ghost Lining"
« Reply #7 on: December 04, 2021, 07:27:02 am »
This is floating bug. It's most likely caused by heap corruption. It depends on what is corrupted. And as memory layout can be different for 32bit and 64bit, debug and release, different OS versions - this bug can appear and disappear in different environments, depending on how critical corrupted data is. It's one of hard to debug bugs. In most cases it should be debugged via so called exclusion method, where some parts of code are excluded in order to find exact place, where bug happens.

But you can always try to use pure ASM debugger and try to debug release version. Linker should usually be able to build memory map file. Run release version of program in debugger. Notice address, where bug happens. Use memory map file to find exact routine, where it happens.

And yeah. Use external debug file, if possible.
« Last Edit: December 04, 2021, 07:28:42 am by Mr.Madguy »
Is it healthy for project not to have regular stable releases?
Just for fun: Code::Blocks, GCC 13 and DOS - is it possible?

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: "Ghost Lining"
« Reply #8 on: December 04, 2021, 08:50:26 am »
What he describes almost sounds like fail of a final buffer purge at the end of linking period ?

 I think this would only cause a Resource error though so it can't be that.

 oh well.

Don't "oh well" yourself, it's a valid suggestion. OTOH I find it difficult to tie in with the problem affecting both Delphi and Lazarus.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: "Ghost Lining"
« Reply #9 on: December 04, 2021, 08:53:57 am »
This is floating bug. It's most likely caused by heap corruption. It depends on what is corrupted. And as memory layout can be different for 32bit and 64bit, debug and release, different OS versions - this bug can appear and disappear in different environments, depending on how critical corrupted data is.

Also the fact that it apparently affects both Delphi and Lazarus, which can be reasonably expected to have different heap layouts.

BUT... that in itself is a valuable clue: it implies that it's something being done by the program itself, rather than by the linker or underlying libraries since if they were at fault the problem would be unlikely to manifest itself in the same way.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Mr.Madguy

  • Hero Member
  • *****
  • Posts: 844
Re: "Ghost Lining"
« Reply #10 on: December 04, 2021, 04:34:37 pm »
Also the fact that it apparently affects both Delphi and Lazarus, which can be reasonably expected to have different heap layouts.

BUT... that in itself is a valuable clue: it implies that it's something being done by the program itself, rather than by the linker or underlying libraries since if they were at fault the problem would be unlikely to manifest itself in the same way.

MarkMLl
The most recent bug in my project, that was fixed - was exactly such bug, that was happening from time to time in release version, but almost never in debug one, so it was really hard to pinpoint it. After researching this problem, I determined, that problem was happening during CreateWindow. First I thought, that it was some sort of race condition in video driver due to video mode switch. It would have been weird, because old version of my project didn't have such problem. But then I realized, that problem was actually happening inside window proc, not CreateWindow itself. At the end I found, that it was caused by wrong window class, that was used to create temporary window. Each dll should have had it's own window class for this purpose. But due to error in window class name wrong one was used, that was causing creating window from other dll, that could have been unloaded. But it was some sort of race condition or this bug relied on dll load addresses, so that had never happened in debugger.

Another example of such floating bug - is forgetting to use ShareMem in Delphi, when it's required, that causes random crashes.
« Last Edit: December 04, 2021, 04:46:07 pm by Mr.Madguy »
Is it healthy for project not to have regular stable releases?
Just for fun: Code::Blocks, GCC 13 and DOS - is it possible?

PascalDragon

  • Hero Member
  • *****
  • Posts: 5444
  • Compiler Developer
Re: "Ghost Lining"
« Reply #11 on: December 06, 2021, 01:48:21 pm »
Just wondering if anyone else has run into this, and what, if anything, I am doing wrong, or not looking for that I should be looking for.

If you're sure that your own code is correct (see remarks of the others) you might want to compare the assembly output t you should ensure that the same optimization settings are used. You could also experiment with the various optimization settings.

speaking of MAP files, how is that done in fpc ? Because I did that many times in Delphi so I could archive that file with the backups of releases. This was handy because I could reference an address in the file reported by a user to determine a general location of the error ..

The -Xm parameter asks the linker to generate a map file.

 

TinyPortal © 2005-2018