* * *

Author Topic: FPCUnit / FPTest & code coverage  (Read 1130 times)

CCRDude

  • Sr. Member
  • ****
  • Posts: 331
FPCUnit / FPTest & code coverage
« on: June 12, 2017, 02:20:00 pm »
I had to write some PHP code the past days, and there are two things I do on all platforms: documenting comments (phpDocumentor in this case) and code tests (phpunit).

One intriguing feature of phpunit is code coverage. If xdebug is installed, it can create (in various output formats) information which parts of the tested code are actually tested; by method and function names as well as by line numbers.

This is a simple text output:

Code: [Select]
Code Coverage Report:       
  2017-06-12 11:53:34       
                           
 Summary:                   
  Classes: 100.00% (1/1)   
  Methods: 100.00% (20/20) 
  Lines:   100.00% (213/213)

MyClassName
  Methods: 100.00% (20/20)   Lines: 100.00% (213/213)

I noticed that this helps enormously to cover more test cases.

Has FPCUnit some similar hidden under a different name maybe?

I tried to see what would be required:
  • New or updated test runner
  • Run parser on test code to find @covers comment for test cases.
  • Run parser on tested code to find what needs to be tested.
  • Get information (from debugger?) which code lines were executed during the test.
  • Wrap it all up in a compatible output format
1-3 and 5 seem to be easy, I have absolutely no idea about 4 though, sounds like the debugger needs special interfaces for that. Maybe the included profiler support might produce output that could help here?
« Last Edit: June 20, 2017, 08:39:05 am by CCRDude »

CCRDude

  • Sr. Member
  • ****
  • Posts: 331
Re: FPCUnit & code coverage
« Reply #1 on: June 13, 2017, 02:06:24 pm »
I tried a few things and used PParser to cover parsing tested and test units. Code is already creating XML documents mostly compatible to phpunit. Two things remain:

  • Test integration. Would love RunTest(...) of the TGUITestRunner from to be virtual, since that's what I did locally to display information right now. Would supply report on Mantis if I finish this.
  • Executed line tracking. Right now I can track which methods/functions are covered (using @covers() comments), but not which lines are executed. Does anyone know enough about the debugger maybe to help out here? Has "the debugger" some hookup mechanism that could inform me about every processed line, for example?

CCRDude

  • Sr. Member
  • ****
  • Posts: 331
Re: FPCUnit & code coverage
« Reply #2 on: June 16, 2017, 03:56:44 pm »
There's a basic implementation now: only GUITestRunner, no console yet, and coverage determined by called functions, not yet by executed lines.

Readme with screenshot here

The huge remaining task would be to find out, during runtime of the test program, from the debugger, which lines were executed.
If that's not possible, I might need to check whether I can combine this with a IDE extension (which I want to create anyway to be able to show as a dedicated gutter for example, which lines were tested).

CCRDude

  • Sr. Member
  • ****
  • Posts: 331
Re: FPCUnit & code coverage
« Reply #3 on: June 20, 2017, 08:38:11 am »
Some more updates, including basic FPTest support.

Screenshot shows the new Gutter within the FPCUnit information window, next step will be to add it to the IDE (as long as the original file is untouched).

Biggest remaining task is the line-based coverage information.  I've been looking into the debugger system, but it's not helping a lot yet. I haven't found out how to access the main debugger yet (it seems to be in main code, not in a package), the DebuggerIntf package does not seem to return access to the current debugger. Enhancing one of the included experimental debuggers would be an approach, but lazdebuggergdbmi for example isn't even true to the actual line. That might be the only approach possible though... enhancing an existing experimental debugger, replacing the Run method with a loop of Step Intos, for each step, check the callstack for one of the coverage methods, and if so, log the line access (the callstack is necessary to have the information which test has covered a line).
« Last Edit: June 20, 2017, 08:41:30 am by CCRDude »

CCRDude

  • Sr. Member
  • ****
  • Posts: 331
Re: FPCUnit / FPTest & code coverage
« Reply #4 on: June 21, 2017, 08:15:01 pm »
I've filed a bug tracker ticket asking for the required changes to FPCUnits GuiTestRunner :) Requesting both the virtual RunTest(), and a more complex but also more flexible handler registration system.

Leledumbo

  • Hero Member
  • *****
  • Posts: 7620
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: FPCUnit / FPTest & code coverage
« Reply #5 on: June 21, 2017, 10:51:24 pm »
That's interesting. It's been hard to do in a purely compiled language, that a C/C++ solution I used when I was working with VC++ a couple of years ago must do source code weaving that dramatically increases compilation time. But the coverage can go down to every conditional expression, which is damn great. The code coverage plugin was a commercial product, though, hence the quality.

Reference: http://www.testwell.fi/ctcdesc.html
Maybe you can get the idea on how the tool does weaving for code coverage and get inspiration from. FPProfiler also does weaving through its tools by using fcl-passrc parser, so you can get FPC POV for doing that.
« Last Edit: June 21, 2017, 10:56:18 pm by Leledumbo »

CCRDude

  • Sr. Member
  • ****
  • Posts: 331
Re: FPCUnit / FPTest & code coverage
« Reply #6 on: June 22, 2017, 04:17:43 pm »
FPProfiler only works blockwise, and with temporary modification, that's less than line based information.

Once I get breakpoints in the Lazarus debugger classes working, I'll have line based coverage information ready.

I agree that down to every conditional expression the result would be even finer, but that's even more difficult to reach... I've looked through the link though, it's interesting to see what is possible, and maybe indeed an inspiration :)

As for the commercial = quality thing, I have to disagree. Lazarus is non-commercial and (imho) way better than Delphi, for which I would have to spend 1k+€ a year, with most bugs I reported to Borland/Codegear/Emba still open a dozen IDE versions later, while Lazarus bugs are treated within days.

Leledumbo

  • Hero Member
  • *****
  • Posts: 7620
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: FPCUnit / FPTest & code coverage
« Reply #7 on: June 22, 2017, 09:03:24 pm »
As for the commercial = quality thing, I have to disagree. Lazarus is non-commercial and (imho) way better than Delphi, for which I would have to spend 1k+€ a year, with most bugs I reported to Borland/Codegear/Emba still open a dozen IDE versions later, while Lazarus bugs are treated within days.
Delphi is an exception to this :P

CCRDude

  • Sr. Member
  • ****
  • Posts: 331
Re: FPCUnit / FPTest & code coverage
« Reply #8 on: June 23, 2017, 03:56:05 pm »
I've got working test code using the TGDBMIDebugger now, but its downside is that it is extremly slow (my test code loops through approx. 8000 lines of text, debugged with StepInto it's two hours, run directly it's less than a second).

As an alternative, I tried TFpDebugDebugger, but that needs debugging information in dwarf format, while I need the stabs format to be able to extract file paths from the gdb file.

One workaround would be to build with stabs, rename the .gdb, then rebuild with dwarf, and test whether TfpDebugDebugger is faster than TGDBMIDebugger.

BeniBela

  • Hero Member
  • *****
  • Posts: 557
    • homepage
Re: FPCUnit / FPTest & code coverage
« Reply #9 on: July 14, 2017, 01:00:13 am »
I've got working test code using the TGDBMIDebugger now, but its downside is that it is extremly slow (my test code loops through approx. 8000 lines of text, debugged with StepInto it's two hours, run directly it's less than a second).

StepInto?

Have you tried setting a breakpoint on every line, and removing it once a line is reached?

CCRDude

  • Sr. Member
  • ****
  • Posts: 331
Re: FPCUnit / FPTest & code coverage
« Reply #10 on: July 14, 2017, 08:06:39 am »
That might be an idea. It would reduce the value of the output (currently also including how often a line was executed), but would still allow line based coverage in faster time. Thanks for the idea!

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus