Recent

Author Topic: FPCUnit / FPTest & code coverage  (Read 9892 times)

CCRDude

  • Hero Member
  • *****
  • Posts: 596
FPCUnit / FPTest & code coverage
« on: June 12, 2017, 02:20:00 pm »
I had to write some PHP code the past days, and there are two things I do on all platforms: documenting comments (phpDocumentor in this case) and code tests (phpunit).

One intriguing feature of phpunit is code coverage. If xdebug is installed, it can create (in various output formats) information which parts of the tested code are actually tested; by method and function names as well as by line numbers.

This is a simple text output:

Code: [Select]
Code Coverage Report:       
  2017-06-12 11:53:34       
                           
 Summary:                   
  Classes: 100.00% (1/1)   
  Methods: 100.00% (20/20) 
  Lines:   100.00% (213/213)

MyClassName
  Methods: 100.00% (20/20)   Lines: 100.00% (213/213)

I noticed that this helps enormously to cover more test cases.

Has FPCUnit some similar hidden under a different name maybe?

I tried to see what would be required:
  • New or updated test runner
  • Run parser on test code to find @covers comment for test cases.
  • Run parser on tested code to find what needs to be tested.
  • Get information (from debugger?) which code lines were executed during the test.
  • Wrap it all up in a compatible output format
1-3 and 5 seem to be easy, I have absolutely no idea about 4 though, sounds like the debugger needs special interfaces for that. Maybe the included profiler support might produce output that could help here?
« Last Edit: June 20, 2017, 08:39:05 am by CCRDude »

CCRDude

  • Hero Member
  • *****
  • Posts: 596
Re: FPCUnit & code coverage
« Reply #1 on: June 13, 2017, 02:06:24 pm »
I tried a few things and used PParser to cover parsing tested and test units. Code is already creating XML documents mostly compatible to phpunit. Two things remain:

  • Test integration. Would love RunTest(...) of the TGUITestRunner from to be virtual, since that's what I did locally to display information right now. Would supply report on Mantis if I finish this.
  • Executed line tracking. Right now I can track which methods/functions are covered (using @covers() comments), but not which lines are executed. Does anyone know enough about the debugger maybe to help out here? Has "the debugger" some hookup mechanism that could inform me about every processed line, for example?

CCRDude

  • Hero Member
  • *****
  • Posts: 596
Re: FPCUnit & code coverage
« Reply #2 on: June 16, 2017, 03:56:44 pm »
There's a basic implementation now: only GUITestRunner, no console yet, and coverage determined by called functions, not yet by executed lines.

Readme with screenshot here

The huge remaining task would be to find out, during runtime of the test program, from the debugger, which lines were executed.
If that's not possible, I might need to check whether I can combine this with a IDE extension (which I want to create anyway to be able to show as a dedicated gutter for example, which lines were tested).

CCRDude

  • Hero Member
  • *****
  • Posts: 596
Re: FPCUnit & code coverage
« Reply #3 on: June 20, 2017, 08:38:11 am »
Some more updates, including basic FPTest support.

Screenshot shows the new Gutter within the FPCUnit information window, next step will be to add it to the IDE (as long as the original file is untouched).

Biggest remaining task is the line-based coverage information.  I've been looking into the debugger system, but it's not helping a lot yet. I haven't found out how to access the main debugger yet (it seems to be in main code, not in a package), the DebuggerIntf package does not seem to return access to the current debugger. Enhancing one of the included experimental debuggers would be an approach, but lazdebuggergdbmi for example isn't even true to the actual line. That might be the only approach possible though... enhancing an existing experimental debugger, replacing the Run method with a loop of Step Intos, for each step, check the callstack for one of the coverage methods, and if so, log the line access (the callstack is necessary to have the information which test has covered a line).
« Last Edit: June 20, 2017, 08:41:30 am by CCRDude »

CCRDude

  • Hero Member
  • *****
  • Posts: 596
Re: FPCUnit / FPTest & code coverage
« Reply #4 on: June 21, 2017, 08:15:01 pm »
I've filed a bug tracker ticket asking for the required changes to FPCUnits GuiTestRunner :) Requesting both the virtual RunTest(), and a more complex but also more flexible handler registration system.

Leledumbo

  • Hero Member
  • *****
  • Posts: 8744
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: FPCUnit / FPTest & code coverage
« Reply #5 on: June 21, 2017, 10:51:24 pm »
That's interesting. It's been hard to do in a purely compiled language, that a C/C++ solution I used when I was working with VC++ a couple of years ago must do source code weaving that dramatically increases compilation time. But the coverage can go down to every conditional expression, which is damn great. The code coverage plugin was a commercial product, though, hence the quality.

Reference: http://www.testwell.fi/ctcdesc.html
Maybe you can get the idea on how the tool does weaving for code coverage and get inspiration from. FPProfiler also does weaving through its tools by using fcl-passrc parser, so you can get FPC POV for doing that.
« Last Edit: June 21, 2017, 10:56:18 pm by Leledumbo »

CCRDude

  • Hero Member
  • *****
  • Posts: 596
Re: FPCUnit / FPTest & code coverage
« Reply #6 on: June 22, 2017, 04:17:43 pm »
FPProfiler only works blockwise, and with temporary modification, that's less than line based information.

Once I get breakpoints in the Lazarus debugger classes working, I'll have line based coverage information ready.

I agree that down to every conditional expression the result would be even finer, but that's even more difficult to reach... I've looked through the link though, it's interesting to see what is possible, and maybe indeed an inspiration :)

As for the commercial = quality thing, I have to disagree. Lazarus is non-commercial and (imho) way better than Delphi, for which I would have to spend 1k+€ a year, with most bugs I reported to Borland/Codegear/Emba still open a dozen IDE versions later, while Lazarus bugs are treated within days.

Leledumbo

  • Hero Member
  • *****
  • Posts: 8744
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: FPCUnit / FPTest & code coverage
« Reply #7 on: June 22, 2017, 09:03:24 pm »
As for the commercial = quality thing, I have to disagree. Lazarus is non-commercial and (imho) way better than Delphi, for which I would have to spend 1k+€ a year, with most bugs I reported to Borland/Codegear/Emba still open a dozen IDE versions later, while Lazarus bugs are treated within days.
Delphi is an exception to this :P

CCRDude

  • Hero Member
  • *****
  • Posts: 596
Re: FPCUnit / FPTest & code coverage
« Reply #8 on: June 23, 2017, 03:56:05 pm »
I've got working test code using the TGDBMIDebugger now, but its downside is that it is extremly slow (my test code loops through approx. 8000 lines of text, debugged with StepInto it's two hours, run directly it's less than a second).

As an alternative, I tried TFpDebugDebugger, but that needs debugging information in dwarf format, while I need the stabs format to be able to extract file paths from the gdb file.

One workaround would be to build with stabs, rename the .gdb, then rebuild with dwarf, and test whether TfpDebugDebugger is faster than TGDBMIDebugger.

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: FPCUnit / FPTest & code coverage
« Reply #9 on: July 14, 2017, 01:00:13 am »
I've got working test code using the TGDBMIDebugger now, but its downside is that it is extremly slow (my test code loops through approx. 8000 lines of text, debugged with StepInto it's two hours, run directly it's less than a second).

StepInto?

Have you tried setting a breakpoint on every line, and removing it once a line is reached?

CCRDude

  • Hero Member
  • *****
  • Posts: 596
Re: FPCUnit / FPTest & code coverage
« Reply #10 on: July 14, 2017, 08:06:39 am »
That might be an idea. It would reduce the value of the output (currently also including how often a line was executed), but would still allow line based coverage in faster time. Thanks for the idea!

dimag0g

  • Newbie
  • Posts: 1
Re: FPCUnit / FPTest & code coverage
« Reply #11 on: June 20, 2023, 03:33:12 am »
Is  Lazarus CodeCoverage Helper maintained? I can't install the package on latest Lazarus (2.2.6), it complains about "FoldView" being an unknown identifier.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
Re: FPCUnit / FPTest & code coverage
« Reply #12 on: June 20, 2023, 09:37:14 am »
As an alternative, I tried TFpDebugDebugger, but that needs debugging information in dwarf format, while I need the stabs format to be able to extract file paths from the gdb file.

One workaround would be to build with stabs, rename the .gdb, then rebuild with dwarf, and test whether TfpDebugDebugger is faster than TGDBMIDebugger.

You should be able to read dwarf, and extract filenames => see lazarus\components\fpdebug\test\dwarfviewer

Or,  once an app is loaded into a debugger, see how the IDE gets line info.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
Re: FPCUnit / FPTest & code coverage
« Reply #13 on: June 20, 2023, 10:17:18 am »
Also InfoDwrf (or similar) from fpc can read this. Look at the ifdefed code in LeaksAndTraces

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1088
  • Professional amateur ;-P
Re: FPCUnit / FPTest & code coverage
« Reply #14 on: June 21, 2023, 01:38:54 pm »
Hey CCRDude,

This is awesome project!!! Many thanks for your effort.

Have you been maintaining it? Just curious!!

Cheers,
Gus
Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

 

TinyPortal © 2005-2018