* * *

Author Topic: Recompile the RTL to debug  (Read 12015 times)

Thaddy

  • Hero Member
  • *****
  • Posts: 7137
Re: Recompile the RTL to debug
« Reply #30 on: December 15, 2017, 01:39:17 pm »
Nope !  Removed the -Xs from two places in Makefile but build still includes the infamous -Xs

I note -dRELEASE too, got to be related.

Sigh.

Too late, going to bed.

Davo

Note I verified the patch (3) thoroughly and you WILL get the correct line number information now if you do a make clean all install 'OPT=-dDEBUG -glh'
I tested on Windows64 10 and on Raspbian Debian Stretch arm eabihf.
But indeed, like molly says, if you have environment variables - exports on linux - set they need to be cleared.
« Last Edit: December 15, 2017, 01:58:01 pm by Thaddy »
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

dbannon

  • Sr. Member
  • ****
  • Posts: 328
Re: Recompile the RTL to debug
« Reply #31 on: December 16, 2017, 12:01:16 am »
Molly, its building fine, there are no errors.  Its just its building with things like -dRELEASE and -Xs

Thats building with the 3.0.2 compiler.  I cannot get a ppc386 from the interweb thing. I have a files of that name in both my functional compile trees. I'm using the 3.0.2 one, if I use the 3.0.4 one, The Makefile spits it out saying I must use 3.0.2    So, I don't quite get what you mean by a bootstrap compiler ??

Thaddy, I have not looked at your patch, I take it it needs to be applied "upstream" ?  And it fixes the stray -Xs that appear in fpc.cfg ?

Well, I have commented out all those -Xs from fpc.cfg as a test, a test that did not solve problem.

I think the compiler is finding another fpc.cfg somewhere, going to have a good search.  But have to go out somewhere right now.

Davo 
Lazarus 1.8, Linux (and reluctantly Win10, OSX)

dbannon

  • Sr. Member
  • ****
  • Posts: 328
Re: Recompile the RTL to debug
« Reply #32 on: December 16, 2017, 11:28:51 am »
OK, to be sure just which fpc.cfg I'm using, I put one in root's home and by putting a printf in there, I can be sure it is, in fact, being used.

That fpc.cfg has both mentions of -Xs commented out and it does not define RELEASE anywhere. My build script does not mention -Xs or RELEASE.

But .....

Looking through the log file generated during Make, I see that the offending options appear quite early, about line #91 of a 3K line log. They are not being used there, they are just mentioned. But they are there ! Thats way, way before anything appears to look at fpc.cfg which does not happen until about #900 - after all the clean up.

And this is a clean source tree.

So, where on earth are these options coming from ?

Lazarus 1.8, Linux (and reluctantly Win10, OSX)

Thaddy

  • Hero Member
  • *****
  • Posts: 7137
Re: Recompile the RTL to debug
« Reply #33 on: December 16, 2017, 12:18:27 pm »
You are looking too deep: apply my patch, make clean all install OPT='-dDEBUG -glh'. Done.
mollie's analyses and my patch are correct.
« Last Edit: December 16, 2017, 12:20:22 pm by Thaddy »
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

molly

  • Hero Member
  • *****
  • Posts: 2345
Re: Recompile the RTL to debug
« Reply #34 on: December 16, 2017, 02:10:38 pm »
Ok, seems i need to be more clear on some topics, so i write it out  :)

Quote
I cannot get a ppc386 from the interweb thing.
As i wrote before. In case you have 3.0.4 compiler installed then the fpc/.../bin directory contains that file for your platform.

Quote
So, I don't quite get what you mean by a bootstrap compiler ??
It is a single executable file capable of compiling Pascal sources (provided that you have a linker and assembler installed correctly on your system).

Bootstrap compiler is just _a_ name to specify a specific executable within the Free Pascal eco system, because otherwise we could end up with quite a mess when two persons are talking about "the compiler".

Think about that for a moment and ask yourself what do _you_ mean when you refer to "the compiler".

More on the term bootstrapping can be found at wikipedia.

Quote
Molly, its building fine, there are no errors.  Its just its building with things like -dRELEASE and -Xs

Thats building with the 3.0.2 compiler.
add option -va to your make instruction. Don't wait it out but let it run for a minute, then break the build and see in the log what options the build is using.

Quote
I have a files of that name in both my functional compile trees. I'm using the 3.0.2 one, if I use the 3.0.4 one, The Makefile spits it out saying I must use 3.0.2   

Currently in this discussion there are two methods being advertised to be able to add debug information to your units. Please don't mix them (unless you're absolutely sure what you're doing).

Method 1:
This method is my own, not officially supported, but should work for building rtl and packages units in order to add debug information to them.

You have 3.0.4 compiler installed and do not want to install 3.0.2 compiler, nor do you want to use 3.0.2 compiler to build a _complete_ 3.0.4 compiler.

That is the method i described in my post including batch file which should work for windows.

This is a method seldom used, as most have _a_ previous fpc compiler installed and are therefor able to build their way up along the different fpc releases.

In case with lazarus (and a fresh OS) there is only one compiler installed, so this method might work as a quick fix to add debug information.

Because this method does not build from the root you can get away with using a starting compiler _not_ required to be of version 3.0.2


Method 2:
The official advertised method.

You _must_ use the 3.0.2 compiler to build the (complete) 3.0.4 compiler. If not, then you receive the error that you mentioned.

In order to add debug information to your compiled units you would need to override default build options because by default no debug information is added.



Both described methods have a couple of things in common, as they both use the fpc build eco-system. So you can see many similarities with regards to providing parameters. They are simply the same.

The difference is that for method 1 you instruct the build system to only build RTL and packages. Because in the end that is all that is required in order to add debug/linenumbers attached to your units (and therefor produced executable, when instructed)

Additionally, in method 1 i'm being more explicit because a) you can (hopefully) learn from that b) my batch is a quick mock-up from a larger script that also builds cross-compilers (there you have to be very explicit with providing parameters).


Whatever method you choose, you still have the second part to handle. For method 1 you _must_ integrated those build units into your existing 3.0.4 installation by modifying your fpc.cfg.

For method 2 you have the choice between chosing to integrate the debug compiled units into your exiting installation _or_ decide to have a complete debug installation of 3.0.4 compiler living next to your original 3.0.4 compiler (that has no debug information attached to compiled units).

That choice is yours to make.

Have you noticed how i _don't_ refer to Lazarus whatsoever ? Since your build instructions fail there is no point in getting it to work for lazarus. First let it work for Free Pascal then we can think about getting it fixed for Lazarus. Just wanting to make sure.

Quote
I think the compiler is finding another fpc.cfg somewhere, going to have a good search.
fpc -va or add -va to your build options and it is able to tell you where (if any) .cfg is coming from.

I repeat: for a build process, there are also environment variables that are able to override your build parameters.


Quote
ne #91 of a 3K line log. They are not being used there, they are just mentioned. But they are there ! Thats way, way before anything appears to look at fpc.cfg which does not happen until about #900 - after all the clean up.

Quote
I see that the offending options appear quite early, about line #91 of a 3K line log. They are not being used there, they are just mentioned. But they are there ! Thats way, way before anything appears to look at fpc.cfg which does not happen until about #900 - after all the clean up.

So, where on earth are these options coming from ?
See my earlier answers.

Note however that when you build the complete compiler that it might be perfectly ok for _some_ projects to compile with these options being active. There is nothing wrong with that (and sometimes even necessary).


Having said all that, this does not seem to be leading somewhere.

Hard facts are required. So a dump of environment variables, build-logs, and -va logs from a test-code build.

I would start with the latter. then a dump from your environment and after that build-logs.

PS: keep note though that i do not have any personal experience with compiling/building on a mac so i might be overlooking something (obvious). I'm relying on others for corrections in case they are necessary.
« Last Edit: December 16, 2017, 02:35:59 pm by molly »

molly

  • Hero Member
  • *****
  • Posts: 2345
Re: Recompile the RTL to debug
« Reply #35 on: December 16, 2017, 02:57:07 pm »
fwiw as additional information to the methods i described in my previous post.

user Thaddy is (more or less) expecting you to patch your sources. You can either do that to existing 3.0.4 sources or use the sources from trunk (in which case you would need 3.0.4 compiler to build fpc 3.1.1 from trunk and eventually use that compiler for your debug builds).

Either that (i won't recommend that you do though because that will add complexity to your topic) or you'll have to wait for his patch to be applied to trunk and/or being back-ported. No idea where a back-port would end up though (e.g. i would suspect in 3.0.6 release if that ever happens ?)

Removing the offending -Xs from fpc.cfg can be done manually (as was explained and you've already done so yourself) but Thaddy's patch include some additional fixing.

dbannon

  • Sr. Member
  • ****
  • Posts: 328
Re: Recompile the RTL to debug
« Reply #36 on: December 17, 2017, 01:07:04 am »
Ah, I did not realise that Thaddy's patch could be applied to my src, I thought it was aimed at trunk and I really did not want to start playing over there !

Hmm, its a seriously large patch. I'll try to apply it to my 3.0.4 src tree.

Quote
Ok, seems i need to be more clear on some topics, so i write it out

... and for that, I am seriously grateful ! I really appriciate the time and effort you and Thaddy have put into this.
Quote
Removing the offending -Xs from fpc.cfg can be done manually ...
Yes, it can, but it does not make any difference, with every -Xs manually removed from every fpc.cfg, -Xs still appears in the Make command lines.

Quote
you have 3.0.4 (bootstrap) compiler installed then the fpc/..
OK, perhaps I need to check that out. I was pretty sure thats the compiler I first used when I get the messages about needing 3.0.2. I think you are saying that it will work as a bootstrap compiler IFF I use your specific command line ?  OK ...

Quote
there are two methods being advertised to be able to add debug information

Ah, that clarify things a bit. Two different methods that should not be mixed ! Got it !

Quote
i _don't_ refer to Lazarus whatsoever

Yep, fully understand that. No point in thinking about Lazarus until I see a compile that does not say things like -Xs -dRELEASE

-va produces a lot of output  :o I'll need to post that (eg) to google drive but it is a big read, and a big ask to ask you to read it !  Env posted below, I did not see anything unexpected there so did not post before.

Quote
i do not have any personal experience with compiling/building on a mac
Trouble is I don't either !  I'm using an old mac I brought on eBay for this project, never used one before. Its "almost" Unix - enough to be frustrating when its not exactly what I expect. Very frustrating !   But my project needs to be "cross platform" and to me, that has to include Mac.

My problem may also be that I am just not the sort of person who can follow a recipe without understanding what and why it does. And if I think I understand, I just must fiddle a bit....

Plan A right now is apply Thaddy's patch locally and see what that does. I don't understand how putting it on 3.0.4 helps but he knows his stuff ! And I have discovered that the build process is a lot more complicated that I'd have thought ...

My env looks like this (Its root's env, I have to build as root ) -
Code: [Select]
sh-3.2# env
TERM_PROGRAM=Apple_Terminal
TERM=xterm-256color
SHELL=/bin/sh
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.29nVnnwS6D/Render
TERM_PROGRAM_VERSION=388.1.1
OLDPWD=/usr/local/share/fpcsrc
TERM_SESSION_ID=E09E06E9-D8DC-433B-9F59-64BA47ACBBDA
USER=dbannon
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.CaWAKR1eqi/Listeners
__CF_USER_TEXT_ENCODING=0x1F6:0x0:0xF
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
PWD=/Users/dbannon/Desktop/Projects
LANG=en_AU.UTF-8
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
HOME=/var/root
SHLVL=2
LOGNAME=dbannon
SECURITYSESSIONID=18713
_=/usr/bin/env



Lazarus 1.8, Linux (and reluctantly Win10, OSX)

molly

  • Hero Member
  • *****
  • Posts: 2345
Re: Recompile the RTL to debug
« Reply #37 on: December 17, 2017, 02:08:59 am »
Ah, I did not realise that Thaddy's patch could be applied to my src, I thought it was aimed at trunk and I really did not want to start playing over there !
The original patch is made against trunk. it should be back-ported. If you understand patches you can do that manually.

Quote
Hmm, its a seriously large patch. I'll try to apply it to my 3.0.4 src tree.
I haven't tried myself yet so be careful, the diff might get confused. I'm not sure if there are other changes that interfere.

Quote
Yes, it can, but it does not make any difference, with every -Xs manually removed from every fpc.cfg, -Xs still appears in the Make command lines.
Yes, it is still coming from somewhere. that is what needs to be figured out.

Quote
OK, perhaps I need to check that out. I was pretty sure thats the compiler I first used when I get the messages about needing 3.0.2.
Yes, but the difference is in a) compiling the _complete_ compiler or b) compiling only rtl and packages

Quote
I think you are saying that it will work as a bootstrap compiler IFF I use your specific command line ?  OK ...
That is correct. my instruction uses bootstrap compiler and can only be used under these circumstances.

Quote
-va produces a lot of output  :o
Yes it does  :D.

 I understand that it probably too much for most people but be happy that fpc is such a compiler that it is able to tell you every little tidbit into details. There are far less informative compilers that leaves you with riddles as error message(s). The trick is to understand/learn what is written in that output. With a bit experience everything in there makes sense  :)

Quote
I'll need to post that (eg) to google drive but it is a big read, and a big ask to ask you to read it !  Env posted below, I did not see anything unexpected there so did not post before.
Be sure to do it first for you testing project, compiled with what is believed to be compiled with debug information using heaptrace. That is where it should end, but allows for us to see if at least the correct debug units and/or fpc.cfg is being used (amongst your other fpc settings).

How (in the end) do you compile your test project ? do you integrate your debug units into your existing 3.0.4 compiler or do you use a 'standalone' 3.04 compiler that has debug information compiled into it's units ?

Quote
My problem may also be that I am just not the sort of person who can follow a recipe without understanding what and why it does. And if I think I understand, I just must fiddle a bit....
In this case, please don't fiddle too much. You seem to fiddle but in doing so seem to be touching some keys that shouldn't be pressed. The make process is clear, precise, is tested and should work (also on a mac). If not the case then you're allowed to file a bug-report. But before filing bug-reports you/we need to make sure that there isn't something on your setup interfering (as seems to be the case right now).

Quote
Plan A right now is apply Thaddy's patch locally and see what that does. I don't understand how putting it on 3.0.4 helps but he knows his stuff !
Take note of the notes i wrote above.

Quote
And I have discovered that the build process is a lot more complicated that I'd have thought ...
It is pretty complicated, especially when you are not familiar with makefiles. And even if you are familiar with them then they can still throw you off balance. fwiw: i'm still learning new things/trick from there, even though it does not have my particular interest.

Quote
My env looks like this (Its root's env, I have to build as root ) -
Ok, at first sight there seems nothing strange to see inside your exports. thanks for posting.

Oh, build as root ? hmz. that is usually strange for a linux. you should be able to build as user. But, i seem to remember something like that when i helped someone with cross-compiling from his mac. In that regards it would be more helpful if another mac user who knows his/her stuff would be able to help you out there a little.

dbannon

  • Sr. Member
  • ****
  • Posts: 328
Re: Recompile the RTL to debug
« Reply #38 on: December 17, 2017, 01:08:12 pm »
OK, new src tree, Thaddy's patch applied (cleanly) and then a build based on Molly's recipe, that is, building only the RTL and Packages using the 3.0.4 compiler.

And it seems to have build without any (f$5^#) -Xs or -dRELEASE. Worth noting, there appears to be a number of memory leaks revealed during the build process.

Its built  usr/lib/fpc/3.0.4/units/i386-darwin/* and even three files in usr/bin

I guess thats going to go into my existing 3.0.4 tree and then I'll generate a new fpc.cfg

Quote
Oh, build as root ? hmz.
25 years as a Unix and Linux systems admin .....

Quote
Yes, it (-dRELEASE and/or -Xs) is still coming from somewhere. that is what needs to be figured out.
Yes, I have not solved that problem. I'm afraid still all there. I'd still like to get it identfied if I can but right now need to concentrate elsewhere. Not good.

Anyway, if I can get the generated stuff working as expected, I'll post the script I used. Its nothing more than a unix'ed version of Molly's.

Big thanks Folks for your help so far, been fantastic.

David



Lazarus 1.8, Linux (and reluctantly Win10, OSX)

dbannon

  • Sr. Member
  • ****
  • Posts: 328
Re: Recompile the RTL to debug
« Reply #39 on: December 18, 2017, 01:56:38 am »
Nope, complete flop again !

Put the newly generated /lib/fpc/3.0.4/units/i386-darwin/* content into my 3.0.4 debug tree. This content was built using Molly's method where I built only the RTL and Utils with the 4.0.3 compiler and options like OPT="-dDEBUG -glh -O-"

It built with no sign of the dreaded -Xs and -dRELEASE so I am fairly confident it does contain appropriate debug info. It was built using this -

Code: [Select]
.....
make -C rtl install INSTALL_PREFIX=$INSTALL_DIR $TARGET PP=$START_COMPILER OPT="-dDEBUG -glh -O-" >> $LOG-rtl
make -C packages install INSTALL_PREFIX=$INSTALL_DIR $TARGET PP=$START_COMPILER OPT="-dDEBUG -glh -O-" >> $LOG-package
.....

But here is the result, the compile of a leaky app (note the heaptrc dump here comes from the compiler, not my app, disregard) -

Code: [Select]
admins-MBP:LeakTest2 dbannon$ /fpc-usr/3.0.4debug/lib/fpc/3.0.4/ppc386 -glh -dDEBUG -va project1.lpr > proj1.txt
Heap dump by heaptrc unit
287651 memory blocks allocated : 23573825/24357464
287651 memory blocks freed     : 23573825/24357464
0 unfreed memory blocks : 0
True heap size : 1736704 (80 used in System startup)
True free heap : 1736624


Run that App -

Code: [Select]
admins-MBP:LeakTest2 dbannon$ ./project1
Heap dump by heaptrc unit
25 memory blocks allocated : 1271/1312
21 memory blocks freed     : 1255/1280
4 unfreed memory blocks : 16
True heap size : 655360 (16 used in System startup)
True free heap : 655024
Should be : 655056
Call trace for block $002ED150 size 4
Call trace for block $002ED100 size 4
Call trace for block $002ED0B0 size 4
Call trace for block $002ED060 size 4

Several blocks of unfreed memory found but still no mention of where they were found.

What happened in the compile -
Code: [Select]
admins-MBP:LeakTest2 dbannon$ grep Xs proj1.txt
[0.005] interpreting file option "#  -Xs   DRB"
[0.018] interpreting file option "# -Xs  DRB"
admins-MBP:LeakTest2 dbannon$ grep RELEASE proj1.txt
[0.005] interpreting file option "# Try compiling with the -dRELEASE or -dDEBUG on the commandline"
[0.005] interpreting file option "#IFDEF RELEASE"
[0.023] Macro FPC_RELEASE set to 0
admins-MBP:LeakTest2 dbannon$ grep DEBUG proj1.txt
[0.005] interpreting file option "# Try compiling with the -dRELEASE or -dDEBUG on the commandline"
[0.005] interpreting file option "#IFDEF DEBUG"
[0.019] Handling option "-dDEBUG"
[0.019] interpreting option "-dDEBUG"
[0.019] Macro defined: DEBUG

No reason to believe that compile stripped out any debug info at all.

And, finally, the App itself, it leaks  -
Code: Pascal  [Select]
  1. program Project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.                 {$IFDEF UNIX}{$IFDEF UseCThreads}
  7.                 cthreads, Lineinfo,
  8.                 {$ENDIF}{$ENDIF}
  9.                 Classes
  10.                 { you can add units after this }, SysUtils;
  11.  
  12.  
  13. procedure BadProcedure();
  14. var
  15.     Ptr : ^longint;
  16. begin
  17.     new(Ptr);
  18.     new(Ptr);
  19. end;
  20.  
  21. begin
  22.     BadProcedure();
  23.     BadProcedure();
  24. end.
I also use another test app that leaks in a more complicated manner, same result.

Conclusion
I have to assume at this stage that adding debug info to RTL and UTILS does not solve the problem of not seeing the line numbers and source code file names of memory leaks.

I think the problem exists in the Mac OSX versions of either HeapTRC or LineInfo units.

I am pretty sure there is no reason to believe there is anything unique in my install or what I am doing. On the Mac, it does not work. And thats probably why there are memory leaks in the Mac code, other people have found them too hard to find.
 
Further research is indicated.
Lazarus 1.8, Linux (and reluctantly Win10, OSX)

molly

  • Hero Member
  • *****
  • Posts: 2345
Re: Recompile the RTL to debug
« Reply #40 on: December 18, 2017, 02:11:02 am »
Thank you for reporting back dbannon.

Do you have time and patience to attempt something else ?

I've located this page (might be outdated) which sets some explicit debug parameters for cross-compilation. When you compile natively these options should be default (in case they are really necessary, as mentioned in that wiki paragraph), but it can't hurt to add them manually for your native (debug) build of rtl and packages.

edit: to make sure there is no misunderstanding this time  :)
Quote
make -C rtl install INSTALL_PREFIX=$INSTALL_DIR $TARGET PP=$START_COMPILER OPT="-dDEBUG -glh -gw -godwarfsets -O-" >> $LOG-rtl
make -C packages install INSTALL_PREFIX=$INSTALL_DIR $TARGET PP=$START_COMPILER OPT="-dDEBUG -glh -gw -godwarfsets -O-" >> $LOG-package

« Last Edit: December 18, 2017, 02:21:31 am by molly »

dbannon

  • Sr. Member
  • ****
  • Posts: 328
Re: Recompile the RTL to debug
« Reply #41 on: December 18, 2017, 03:26:02 am »
Very easy Molly. All in the script, it deletes all source, copy clean version, apply Thaddy's patch, build. Then I copy the result into the 3.0.4debug tree.

However, like most easy fixes, it did not help. Same result.

Now, I did not generate a new fpc.cfg for the debug tree. Because I'm not building a new compiler and specifically a new fpcmkcfg, Thaddy's patch wont change the fpc.cfg - IMHO !  I have edited it to comment out both references to -Xs.

David
Lazarus 1.8, Linux (and reluctantly Win10, OSX)

molly

  • Hero Member
  • *****
  • Posts: 2345
Re: Recompile the RTL to debug
« Reply #42 on: December 18, 2017, 03:30:48 am »
Did you also remember to compile your test-project (that leaks) with those options ?

dbannon

  • Sr. Member
  • ****
  • Posts: 328
Re: Recompile the RTL to debug
« Reply #43 on: December 18, 2017, 04:04:19 am »
Yep, looks like this -

Code: [Select]
/fpc-usr/3.0.4debug/lib/fpc/3.0.4/ppc386 -glh -gw -godwarfsets -O- -dDEBUG project1.lpr
Davo
Lazarus 1.8, Linux (and reluctantly Win10, OSX)

molly

  • Hero Member
  • *****
  • Posts: 2345
Re: Recompile the RTL to debug
« Reply #44 on: December 18, 2017, 04:28:31 am »
Ok, thank you.

Next  assignment :)

What does the following code show you (it doesn't matter which fpc compiler (or even lazarus) you use as long as it does not strip debug information).
Code: [Select]
program showlineinfo;

{$MODE OBJFPC}{$H+}

// compile with fpc -B -dDEBUG showlineinfo.pas (with fixed fpc.cfg)
// -dDebug should supply the -gl option

var
  li_address : PtrUInt;
  li_func    : Shortstring;
  li_source  : Shortstring;
  li_Line    : LongInt;
begin
  li_Address := PtrUInt(Get_pc_addr);
  if GetLineInfo(li_address, li_func, li_source, li_line) then
  begin
    WriteLn('Debug Information:');
    WriteLn('Function = ', li_func);
    WriteLn('Source   = ', li_source);
    WriteLn('Line     = ', li_Line);
  end
  else WriteLn('Retrieving debug information failed');
end.

?

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus