Lazarus

Programming => General => Topic started by: Tony Stone on September 12, 2021, 10:12:30 pm

Title: Lazarus Profiler
Post by: Tony Stone on September 12, 2021, 10:12:30 pm

I am trying to setup a profiler for my Linux dev machine.  I have tried KCacheGrind with no success.  Ran into issues which probably need to be addressed in another forum.  Anyway, I would really like to get the LazProfiler addon available at https://github.com/PascalRiekenberg/LazProfiler working.  I have tried on my Windows machine as well and run into issues installing it.  I did eventually get it to install on my Linux machine with a trunk version of Lazarus.  Problem is i get continuous Access Violation errors with no other information available.  Even if I could get it to work the trunk version of FPC hangs when trying to compile a crypto library I am using, so it wouldn't be helpful anyway.

Has anyone had success using a profiler on Linux with your Lazarus programs?  Do you have any suggestions or other options I can look into?

Thanks

PS
I would also like to add that I am willing to pay a bounty for a functional profiler add-on to Lazarus.  Maybe there are other people interested in contributing to the development of a good profiler?  Let me know.  I would probably start by offering a few hundred dollars to try and fix the existing LazProfiler... if that is not a reasonable amount/approach I would discuss further payment options for other solutions...

Title: Re: Lazarus Profiler
Post by: Tony Stone on September 13, 2021, 01:10:53 am

I found https://orbitprofiler.com/ which says it is for compiled c and c++ binaries... doesnt require source modification or recompiling.  I wonder if it would be able to profile a FPC application?  Just putting it here if anyone is familiar and thinks i am wasting my time... but i am gonna try it out anyway....

Title: Re: Lazarus Profiler
Post by: 440bx on September 13, 2021, 02:51:51 am

I cannot help you with anything that involves Linux but, from what you stated in your first post, it sounds like your app runs on Windows too.

If you can compile your app as a 64bit app for Windows then it may be possible to use the Visual Studio profiler (I haven't tried but, I've been told it works well.)

Check the comments in this thread https://forum.lazarus.freepascal.org/index.php/topic,42000.msg295128.html#msg295128
in particular, don't miss the last two comments.

HTH.

Title: Re: Lazarus Profiler
Post by: trev on September 13, 2021, 05:07:21 am

See also the Wiki article on Profiling (https://wiki.lazarus.freepascal.org/Profiling).

Title: Re: Lazarus Profiler
Post by: JuhaManninen on September 13, 2021, 05:17:17 pm

Quote from: Tony Stone on September 12, 2021, 10:12:30 pm

I am trying to setup a profiler for my Linux dev machine.  I have tried KCacheGrind with no success.  Ran into issues which probably need to be addressed in another forum.

You can tell the details here as well.
Did you run "valgrind --tool=callgrind ..."?
Did it produce an output file?
For me it has worked extremely well. The visualization provided by KCacheGrind is almost ecstatic. It does complex analysis and complex graphics output very fast. I can see they have profiled and optimized their own application well.
Valgrind slows down the profiled program a lot, but that is true with any high quality profiler.
Read the wiki page provided by trev.

Title: Re: Lazarus Profiler
Post by: Tony Stone on September 14, 2021, 12:46:24 am

Quote from: JuhaManninen on September 13, 2021, 05:17:17 pm

Quote from: Tony Stone on September 12, 2021, 10:12:30 pm

I am trying to setup a profiler for my Linux dev machine.  I have tried KCacheGrind with no success.  Ran into issues which probably need to be addressed in another forum.

You can tell the details here as well.
Did you run "valgrind --tool=callgrind ..."?
Did it produce an output file?
For me it has worked extremely well. The visualization provided by KCacheGrind is almost ecstatic. It does complex analysis and complex graphics output very fast. I can see they have profiled and optimized their own application well.
Valgrind slows down the profiled program a lot, but that is true with any high quality profiler.
Read the wiki page provided by trev.

So I am back to KCacheGrind... i thought i was finally having success... now I am at this point here which I am trying to work my way through:

Code: Pascal  [Select][+][-]

1. tony@IrontTowel-Tower11 /media/tony/storpart/GIT/bitbroom $ valgrind --tool=callgrind ./bindbgbitbroom
2. ==484764== Callgrind, a call-graph generating cache profiler
3. ==484764== Copyright (C) 2002-2017, and GNU GPL'd, by Josef Weidendorfer et al.
4. ==484764== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
5. ==484764== Command: ./bindbgbitbroom
6. ==484764==
7. ==484764== For interactive control, run 'callgrind_control -h'.
8. --484764-- WARNING: unhandled amd64-linux syscall: 315
9. --484764-- You may be able to write your own handler.
10. --484764-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
11. --484764-- Nevertheless we consider this a bug.  Please report
12. --484764-- it at http://valgrind.org/support/bug_reports.html.

Title: Re: Lazarus Profiler
Post by: Tony Stone on September 14, 2021, 01:53:13 am

Wow... ok, it was actually in fact working.  It is just painfully slow doing anything with my program using valgrind.  But it did actually output the files in the directory and i am able to open them with KCacheGrind now.  Thanks for pushing me back to KCacheGrind!  I think this may offer me what I was looking for.

Update:
As I go through the callgrind output file I am realizing it doesnt exactly show my function names.  It shows thousands of functions, mostly GTK stuff...  trying to make sense of this all is beyond my ability at the moment.  Would love to find a simpler solution... something less advanced.  In the meantime I am gonna try to learn more about reading and understanding all of this info it is throwing at me.

So wether this is a success for me or not... it is very cool and if anyone wants to try, read the WIKI posted above, install KCacheGrind... and have patience! 

Title: Re: Lazarus Profiler
Post by: JuhaManninen on September 14, 2021, 10:30:42 am

KCacheGrind has many different views and ways to filter the data. Play with them and you may find some of them useful. Frankly, I also didn't understand the idea of all of them.
I attach a Callee Map screenshot from my Lazarus startup optimization endeavour.
The GTK stuff must be the LCL interface and FPC lib calls. If they are called too many times, probably there is a problem. Playing with the data helps you get the idea.

Quote

It is just painfully slow doing anything with my program using valgrind.

Yes. In practice you need a fast machine. For example my old ASRock Beebox TV-box gadget with Manjaro Linux is fast enough for anything I do, including Lazarus development, except for valgrind / callgrind. I use a faster machine for that.

Quote

--484764-- WARNING: unhandled amd64-linux syscall: 315

I don't get that. Sounds like a kernel problem. What is your Linux distro and how old is the kernel?

Title: Re: Lazarus Profiler
Post by: winni on September 14, 2021, 10:55:31 am

Quote from: Tony Stone on September 14, 2021, 12:46:24 am

--484764-- WARNING: unhandled amd64-linux syscall: 315

Have a look at this:

https://bugs.launchpad.net/ubuntu/+source/valgrind/+bug/1909683 (https://bugs.launchpad.net/ubuntu/+source/valgrind/+bug/1909683)

Winni

Title: Re: Lazarus Profiler
Post by: JuhaManninen on September 14, 2021, 11:43:20 am

Quote from: winni on September 14, 2021, 10:55:31 am

https://bugs.launchpad.net/ubuntu/+source/valgrind/+bug/1909683 (https://bugs.launchpad.net/ubuntu/+source/valgrind/+bug/1909683)

Ok, yes. My Manjaro has a newer version of valgrind.

Code: [Select]

$ valgrind --version
valgrind-3.17.0

Title: Re: Lazarus Profiler
Post by: prof7bit on September 14, 2021, 02:50:14 pm

On Linux I have had great success with OProfile in the past. It is a sampling profiler and does not need any code instrumentation, so it can profile basically anything.

Because there is no instrumentation your program will not be slowed down or impacted in any way during the profiling run!

A sampling profiler will make a snapshot of the callstack every x milliseconds and over time it will aggregate the information how many percent of the total time you spent inside function x (including all its callees) for every function in your program and if you run it long enough and have enough samples for a hotspot in your program it will be accurate down to individual machine instructions!

It is the consequent automation of the "poor man's profiler": You press pause in the debugger and write down inside which function you currently are. If you do this often enough you see where it spends most of the time or how many percent of the time it spends in this function. Oprofile can do this tens of thousands of times within a few seconds and give you very accurate reults.

The output of OProfile can also be converted into a callgrind file you can analyze with KCacheGrind.

Title: Re: Lazarus Profiler
Post by: Martin_fr on September 14, 2021, 03:00:14 pm

Quote from: Tony Stone on September 14, 2021, 01:53:13 am

As I go through the callgrind output file I am realizing it doesnt exactly show my function names.  It shows thousands of functions, mostly GTK stuff...  trying to make sense of this all is beyond my ability at the moment.

Did you enable debug info for your project? and added the checkbox for valgrind (project opitons) -gv

If you want functions in packages too, then you need to enable debug info for those too.
For RTL (if needed), you have to recompile fpc.

You can then filter by source file in KCachegrind.

Title: Re: Lazarus Profiler
Post by: JuhaManninen on September 14, 2021, 03:30:30 pm

Quote from: prof7bit on September 14, 2021, 02:50:14 pm

It is the consequent automation of the "poor man's profiler": You press pause in the debugger and write down inside which function you currently are. If you do this often enough you see where it spends most of the time or how many percent of the time it spends in this function. Oprofile can do this tens of thousands of times within a few seconds and give you very accurate reults.

Do you have any data about how accurate it is compared to other profilers?
My understanding is that a sampling profiler is not the most accurate but I may be wrong.

Title: Re: Lazarus Profiler
Post by: prof7bit on September 14, 2021, 05:45:39 pm

Quote from: JuhaManninen on September 14, 2021, 03:30:30 pm

Quote from: prof7bit on September 14, 2021, 02:50:14 pm

It is the consequent automation of the "poor man's profiler": You press pause in the debugger and write down inside which function you currently are. If you do this often enough you see where it spends most of the time or how many percent of the time it spends in this function. Oprofile can do this tens of thousands of times within a few seconds and give you very accurate reults.

Do you have any data about how accurate it is compared to other profilers?
My understanding is that a sampling profiler is not the most accurate but I may be wrong.

The more samples you acquire the more accurate it should become, but I have no hard data. I can remember having done some experiments with it to see how far I can go and after a few minutes in a tight loop I could see execution times for individual machine instructions that seemed quite reasonable.

I used OProfile to fix a huge CPU hog in the TurboPower Ipro HTML component 9 years ago that made lhelp slow down to the point of unsability. I remember it was actually really easy to find it, it turned out they used a linear search in a stringlist (it was completely obvious to see in the visualization of the callgraph from the output of OProfile) and I replaced it with a hash map.

The advantage of sampling profilers is that they have almost no effect on the execution of the program, you can attach to a running process without slowing it down, you can even profile a running kernel without disturbing anything. And for quickly identifying CPU hogs the accuracy is usually plenty enough.

I just tried to get OProfile working again an hour ago (after 5 years not using it) and it seems they changed some things (at least in the Ubuntu distribution) and I can't get it to work like I used to (or I am still missing something). But there is also perf, it needs a kernel module and can be installed with apt-get, it comes with it's own GUI (hotspot) to visualize the results, and from a quick look at it it seems it should be able to replace OProfile for most needs.

Title: Re: Lazarus Profiler
Post by: JuhaManninen on September 14, 2021, 07:55:23 pm

Quote from: prof7bit on September 14, 2021, 05:45:39 pm

I just tried to get OProfile working again an hour ago (after 5 years not using it) and it seems they changed some things (at least in the Ubuntu distribution) and I can't get it to work like I used to (or I am still missing something). But there is also perf, it needs a kernel module and can be installed with apt-get, it comes with it's own GUI (hotspot) to visualize the results, and from a quick look at it it seems it should be able to replace OProfile for most needs.

Ok, I found that recent versions of OProfile use the Linux Kernel Performance Events Subsystem.
 https://oprofile.sourceforge.io/about/
The perf thing could be useful. I must have a close look some day.

Title: Re: Lazarus Profiler
Post by: prof7bit on September 14, 2021, 09:37:05 pm

and then there are google gperftools, this is also a sampling profiler.

just for fun I ran it on fpdoc (see attachment)

Title: Re: Lazarus Profiler
Post by: Tony Stone on September 14, 2021, 09:41:48 pm

Quote from: prof7bit on September 14, 2021, 09:37:05 pm

and then there are google gperftools, this is also a sampling profiler.

just for fun I ran it on fpdoc (see attachment)

When I get home tonight I will check oProfile out and this.  Thank you.  I was able to get function names from KCacheGrind as well... was burried in the settings and features. lol... i don't understand what a lot of the information is it gives you....  This sampling method is very interesting to me though.