Recent

Author Topic: Include Informative Line Count Statistics  (Read 2148 times)

munair

  • Hero Member
  • *****
  • Posts: 884
  • compiler developer @SharpBASIC
    • SharpBASIC
Include Informative Line Count Statistics
« on: May 09, 2025, 09:46:10 am »
For larger projects in particular, it would be helpful if Lazarus had an option to provide line count statistics. Working on a project for 10 years, I started wondering about the line count stats, so I made a simple command line tool that can serve as an example. My project appears to have close to 30k lines of code and I had no idea.

The stat results are also exported to a CSV file.

Code: Pascal  [Select][+][-]
  1. {
  2.   countlines: Counts lines in *.pas, *.pp, *.dpr, and *.lfm files
  3.   - PAS/PP/DPR: Non-empty (TotalLines), Code (incl. code with // comments), Comment (TotalLines - CodeLines), Empty
  4.   - LFM: Total lines, Empty lines
  5.   - Supports comment blocks
  6.   - Ignores nested/malformed comments (bad practice)
  7.   - Warns on unclosed comment blocks
  8.   - Single-pass file reading, CSV output
  9.   - Safeguard for empty directories
  10.   copyright (C) Frank Hoogerbeets (https://github.com/ssgeos/)
  11.   License: FPC modified LGPL Version 2 (https://wiki.lazarus.freepascal.org/FPC_modified_LGPL)
  12. }
  13. program countlines;
  14.  
  15. uses
  16.   SysUtils;
  17.  
  18. type TSource = record
  19.   CodeLines: integer;
  20.   CommentLines: integer;
  21.   EmptyLines: integer;
  22.   TotalLines: integer;
  23. end;
  24.  
  25. var
  26.   Directory: string;
  27.   PasFile: TSource;
  28.   LfmFile: TSource;
  29.   PasFilesTotal: TSource;
  30.   LfmFilesTotal: TSource;
  31.  
  32. procedure CountLinesInFile(const FileName: string; var Source: TSource);
  33. // count code lines
  34. var
  35.   F: TextFile;
  36.   Line: string;
  37.   InCommentBlock: Boolean;
  38.   TrimmedLine: string;
  39.   i: Integer;
  40. begin
  41.   InCommentBlock := False;
  42.  
  43.   AssignFile(F, FileName);
  44.   try
  45.     Reset(F);
  46.     while not EOF(F) do
  47.       begin
  48.         ReadLn(F, Line);
  49.         TrimmedLine := Trim(Line);
  50.  
  51.         if Trim(Line) = '' then
  52.           begin
  53.             Inc(Source.EmptyLines);
  54.             Continue;
  55.           end;
  56.  
  57.         Inc(Source.TotalLines);
  58.  
  59.         // Check for single-line // comment
  60.         if not InCommentBlock then
  61.           begin
  62.             i := Pos('//', TrimmedLine);
  63.             if i > 0 then
  64.               begin
  65.                 TrimmedLine := Trim(Copy(TrimmedLine, 1, i - 1));
  66.                 if TrimmedLine = '' then
  67.                   Continue;
  68.               end;
  69.           end;
  70.  
  71.         // Handle multi-line comment blocks
  72.         if InCommentBlock then
  73.           begin
  74.             i := Pos('}', TrimmedLine);
  75.             if i = 0 then
  76.               i := Pos('*)', TrimmedLine);
  77.             if i > 0 then
  78.               begin
  79.                 InCommentBlock := False;
  80.                 TrimmedLine := Trim(Copy(TrimmedLine, i + 1, Length(TrimmedLine)));
  81.                 if TrimmedLine = '' then
  82.                   Continue;
  83.               end
  84.             else
  85.               Continue;
  86.           end;
  87.  
  88.         // Check for start of multi-line comment block
  89.         i := Pos('{', TrimmedLine);
  90.         if i = 0 then
  91.           i := Pos('(*', TrimmedLine);
  92.         if i > 0 then
  93.           begin
  94.             if Pos('}', TrimmedLine) > i then
  95.               begin
  96.                 TrimmedLine := Trim(Copy(TrimmedLine, 1, i - 1) +
  97.                                     Copy(TrimmedLine, Pos('}', TrimmedLine) + 1, Length(TrimmedLine)));
  98.                 if TrimmedLine = '' then
  99.                   Continue;
  100.               end
  101.             else if Pos('*)', TrimmedLine) > i then
  102.               begin
  103.                 TrimmedLine := Trim(Copy(TrimmedLine, 1, i - 1) +
  104.                                     Copy(TrimmedLine, Pos('*)', TrimmedLine) + 1, Length(TrimmedLine)));
  105.                 if TrimmedLine = '' then
  106.                   Continue;
  107.               end
  108.             else
  109.               begin
  110.                 InCommentBlock := True;
  111.                 TrimmedLine := Trim(Copy(TrimmedLine, 1, i - 1));
  112.                 if TrimmedLine = '' then
  113.                   Continue;
  114.               end;
  115.           end;
  116.  
  117.         // Count non-empty, non-comment lines
  118.         if TrimmedLine <> '' then
  119.           Inc(Source.CodeLines);
  120.       end;
  121.     if InCommentBlock then
  122.       Writeln('WARNING: Unclosed comment block in ', FileName);
  123.   except
  124.     on E: Exception do
  125.       Writeln('Error reading file ', FileName, ': ', E.Message);
  126.   end;
  127.   CloseFile(F);
  128. end;
  129.  
  130. procedure CountFilesInDirectory(const Directory: string);
  131. var
  132.   SearchRec: TSearchRec;
  133.   FilePath, Ext: string;
  134. begin
  135.   // reset totals
  136.   PasFilesTotal := default(TSource);
  137.   LfmFilesTotal := default(TSource);
  138.  
  139.   if FindFirst(IncludeTrailingPathDelimiter(Directory) + '*.*', faAnyFile, SearchRec) = 0 then
  140.     try
  141.       repeat
  142.         if (SearchRec.Attr and faDirectory) = 0 then
  143.           begin
  144.             FilePath := IncludeTrailingPathDelimiter(Directory) + SearchRec.Name;
  145.             Ext := LowerCase(ExtractFileExt(SearchRec.Name));
  146.             // source code units
  147.             if (Ext = '.pas') or (Ext = '.pp') or (Ext = '.dpr') then
  148.               begin
  149.                 PasFile := default(TSource);
  150.                 CountLinesInFile(FilePath, PasFile);
  151.                 Inc(PasFilesTotal.TotalLines, PasFile.TotalLines);
  152.                 Inc(PasFilesTotal.EmptyLines, PasFile.EmptyLines);
  153.                 Inc(PasFilesTotal.CodeLines, PasFile.CodeLines);
  154.                 PasFile.CommentLines := PasFile.TotalLines - PasFile.CodeLines;
  155.                 Inc(PasFilesTotal.CommentLines, PasFile.CommentLines);
  156.                 Writeln(
  157.                   'File: ', SearchRec.Name,
  158.                   ', Total Lines (non-empty): ', PasFile.TotalLines,
  159.                   ', Code Lines: ', PasFile.CodeLines,
  160.                   ', Comment Lines: ', PasFile.CommentLines,
  161.                   ', Empty Lines: ', PasFile.EmptyLines
  162.                 );
  163.               end
  164.             // auto-generated form files
  165.             else if Ext = '.lfm' then
  166.               begin
  167.                 LfmFile := default(TSource);
  168.                 CountLinesInFile(FilePath, LfmFile);
  169.                 Inc(LfmFilesTotal.TotalLines, LfmFile.TotalLines);
  170.                 Inc(LfmFilesTotal.EmptyLines, LfmFile.EmptyLines);
  171.                 Writeln(
  172.                   'File: ', SearchRec.Name,
  173.                   ', Total Lines: ', LfmFile.TotalLines,
  174.                   ', Empty Lines: ', LfmFile.EmptyLines
  175.                 );
  176.               end;
  177.           end;
  178.       until FindNext(SearchRec) <> 0;
  179.     finally
  180.       FindClose(SearchRec);
  181.     end;
  182. end;
  183.  
  184. procedure SaveCountsToCSV(const Directory: string);
  185. var
  186.   F: TextFile;
  187.   SearchRec: TSearchRec;
  188.   FilePath, Ext: string;
  189. begin
  190.   try
  191.     AssignFile(F, Directory + '/line_counts.csv');
  192.     Rewrite(F);
  193.     Writeln(F, 'File,NonEmpty,Code,Comment,Empty');
  194.     if FindFirst(IncludeTrailingPathDelimiter(Directory) + '*.*', faAnyFile, SearchRec) = 0 then
  195.                         try
  196.                                 repeat
  197.                                   if (SearchRec.Attr and faDirectory) = 0 then
  198.                                     begin
  199.                                       FilePath := IncludeTrailingPathDelimiter(Directory) + SearchRec.Name;
  200.                                       Ext := LowerCase(ExtractFileExt(SearchRec.Name));
  201.                                       if Ext = '.pas' then
  202.                                         begin
  203.                                           PasFile := default(TSource);
  204.                                           CountLinesInFile(FilePath, PasFile);
  205.                                           Writeln(F, SearchRec.Name + ',' +
  206.                                                   IntToStr(PasFile.TotalLines) + ',' +
  207.                                                   IntToStr(PasFile.CodeLines) + ',' +
  208.                                                   IntToStr(PasFile.TotalLines - PasFile.CodeLines) + ',' +
  209.                                                   IntToStr(PasFile.EmptyLines));
  210.                                         end
  211.                                       else if Ext = '.lfm' then
  212.                                         begin
  213.                                           LfmFile := default(TSource);
  214.                                           CountLinesInFile(FilePath, LfmFile);
  215.                                           Writeln(F, SearchRec.Name + ',' +
  216.                                                   IntToStr(LfmFile.TotalLines) + ',,,' +
  217.                                                   IntToStr(LfmFile.EmptyLines));
  218.                                         end;
  219.                                     end;
  220.                                 until FindNext(SearchRec) <> 0;
  221.                         finally
  222.                                 FindClose(SearchRec);
  223.                         end;
  224.                         Writeln(F, 'Total PAS,' + IntToStr(PasFilesTotal.TotalLines) + ',' +
  225.                                         IntToStr(PasFilesTotal.CodeLines) + ',' +
  226.                                         IntToStr(PasFilesTotal.CommentLines) + ',' +
  227.                                         IntToStr(PasFilesTotal.EmptyLines));
  228.                         Writeln(F, 'Total LFM,' + IntToStr(LfmFilesTotal.TotalLines) + ',,,' +
  229.                                         IntToStr(LfmFilesTotal.EmptyLines));
  230.       if PasFilesTotal.TotalLines > 0 then
  231.         begin
  232.           Writeln(F);
  233.           Writeln(F, '# Comment lines percentage (of non-empty): ', (PasFilesTotal.CommentLines / PasFilesTotal.TotalLines * 100):0:1, '%');
  234.           Writeln(F, '# Empty lines percentage (of total): ', (PasFilesTotal.EmptyLines / (PasFilesTotal.TotalLines + PasFilesTotal.EmptyLines) * 100):0:1, '%');
  235.         end;
  236.       CloseFile(F);
  237.   except
  238.     on E: Exception do
  239.       Writeln('Error writing CSV: ', E.Message);
  240.   end;
  241. end;
  242.  
  243. begin
  244.   if ParamCount > 0 then
  245.     Directory := ParamStr(1)
  246.   else
  247.     Directory := GetCurrentDir;
  248.  
  249.   if not DirectoryExists(Directory) then
  250.     begin
  251.       Writeln('Error: Directory ', Directory, ' does not exist.');
  252.       Halt(1);
  253.     end;
  254.  
  255.   Writeln('Counting lines in directory: ', Directory);
  256.  
  257.   CountFilesInDirectory(Directory);
  258.   SaveCountsToCSV(Directory);
  259.  
  260.   WriteLn;
  261.   Writeln('Total lines in all *.pas files (non-empty): ', PasFilesTotal.TotalLines);
  262.   Writeln('Total code lines (non-comment, non-empty): ', PasFilesTotal.CodeLines);
  263.   Writeln('Total comment lines: ', PasFilesTotal.CommentLines);
  264.   Writeln('Total empty lines: ', PasFilesTotal.EmptyLines);
  265.   if PasFilesTotal.TotalLines > 0 then
  266.     begin
  267.       Writeln('Comment lines percentage (of non-empty): ', (PasFilesTotal.CommentLines / PasFilesTotal.TotalLines * 100):0:1, '%');
  268.       Writeln('Empty lines percentage (of total): ', (PasFilesTotal.EmptyLines / (PasFilesTotal.TotalLines + PasFilesTotal.EmptyLines) * 100):0:1, '%');
  269.     end;
  270.   Writeln('Total lines in all *.lfm files: ', LfmFilesTotal.TotalLines);
  271.   Writeln('Total empty lines: ', LfmFilesTotal.EmptyLines);
  272.   WriteLn;
  273.   WriteLn('Results saved to -> line_counts.csv');
  274. end.

There is a package on Github that will install a menu entry in the Lazarus IDE under Project to provide the same metrics.
https://github.com/ssgeos/ProjectMetrics
« Last Edit: May 14, 2025, 10:57:28 am by munair »
It's only logical.

Thaddy

  • Hero Member
  • *****
  • Posts: 17176
  • Ceterum censeo Trump esse delendam
Re: Include Informative Line Count Statistics
« Reply #1 on: May 09, 2025, 10:55:55 am »
Any TStrings descedant, including TMemo.Items, has a count property that equals the number of lines. I would use that.
Due to censorship, I changed this to "Nelly the Elephant". Keeps the message clear.

n7800

  • Sr. Member
  • ****
  • Posts: 358
Re: Include Informative Line Count Statistics
« Reply #2 on: May 09, 2025, 07:21:51 pm »
For larger projects in particular, it would be helpful if Lazarus had an option to provide line count statistics.

I've seen a similar topic before: https://forum.lazarus.freepascal.org/index.php?topic=71084

munair

  • Hero Member
  • *****
  • Posts: 884
  • compiler developer @SharpBASIC
    • SharpBASIC
Re: Include Informative Line Count Statistics
« Reply #3 on: May 09, 2025, 10:46:55 pm »
That's just yesterday. What a coincidence.
It's only logical.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4572
  • I like bugs.
Re: Include Informative Line Count Statistics
« Reply #4 on: May 10, 2025, 10:47:34 pm »
A "Project Statistics" entry is needed in the Project menu.
If you want to implement it, a patch would be accepted.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

munair

  • Hero Member
  • *****
  • Posts: 884
  • compiler developer @SharpBASIC
    • SharpBASIC
Re: Include Informative Line Count Statistics
« Reply #5 on: May 11, 2025, 12:13:59 am »
A "Project Statistics" entry is needed in the Project menu.
If you want to implement it, a patch would be accepted.
If you can point me to the latest official source, I'm happy to provide the patch.
It's only logical.

TRon

  • Hero Member
  • *****
  • Posts: 4377
Re: Include Informative Line Count Statistics
« Reply #6 on: May 11, 2025, 12:54:12 am »
If you can point me to the latest official source, I'm happy to provide the patch.
Lazarus sources (trunk) are located here. It is possible to either a merge request or a patch. In case the latter make sure to note the revision (hash) to patch against.
Today is tomorrow's yesterday.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4572
  • I like bugs.
Re: Include Informative Line Count Statistics
« Reply #7 on: May 11, 2025, 08:33:34 am »
Lazarus sources (trunk) are located here. It is possible to either a merge request or a patch. In case the latter make sure to note the revision (hash) to patch against.
Yes. All development happens in the "main" branch, historically also called as trunk.
No need to mention any revisions when providing a patch. We assume it is against the latest trunk. Even if it is made from an older revision, it usually applies when the relevant code is not touched for a while. Only when there are conflicting changes, patch tools report an error. Thus the patch format is very safe, eg. no earlier changes will be reverted by accident.

You get the sources easily by:
Code: [Select]
$ git clone https://gitlab.com/freepascal.org/lazarus/lazarus.git
$ cd lazarus
$ make
Later just "git pull" for latest changes.
The "main" branch is active by default.
BTW, is "main" branch very usable most of the time. For maintaining a critical application it is not recommended, but for anything else (hobby projects, learning) I can recommend it.

About the actual Statistics feature: It should work on files of a project, not files in some directory.
Your parser for code / empty / comment lines is good at least initially. Later maybe other parsers could be involved, like Codetools for finding .inc files.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

dbannon

  • Hero Member
  • *****
  • Posts: 3379
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Include Informative Line Count Statistics
« Reply #8 on: May 12, 2025, 02:25:08 am »

I just use -
Code: Pascal  [Select][+][-]
  1. $> wc -l *.pas
  2.     134 autostart.pas
  3.    4828 editbox.pas
  4.     ...
  5.     167 uqt_colors.pas
  6.   29553 total
  7. $>

But, of course, I don't differentiate between code, comments and blank lines. All are equally valuable IMHO.

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

munair

  • Hero Member
  • *****
  • Posts: 884
  • compiler developer @SharpBASIC
    • SharpBASIC
Re: Include Informative Line Count Statistics
« Reply #9 on: May 12, 2025, 03:08:08 pm »
About the actual Statistics feature: It should work on files of a project, not files in some directory.
Your parser for code / empty / comment lines is good at least initially. Later maybe other parsers could be involved, like Codetools for finding .inc files.
Got it.
It's only logical.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 11349
  • Debugger - SynEdit - and more
    • wiki
Re: Include Informative Line Count Statistics
« Reply #10 on: May 12, 2025, 03:47:14 pm »
Your parser for code / empty / comment lines is good at least initially. Later maybe other parsers could be involved, like Codetools for finding .inc files.

There are already several other features that work on "files of project".

One way to get them is "ProjectIntf" which can be found in the package "IdeIntf". It allows to iterate the files of the project.
However that is files that have been added to the project and are listed in the Project-Inspector. It won't scan for other files in directories used by the project.

Ideally (and for best experience with other IDE features) all files are added to the project, then the above is fully sufficient. If not then you can get directories from the ProjectIntf. But that may then contain files (like "Copy of MyUnit.pas) that aren't used. For finding only files used by the Project you would indeed need codetools or similar. But that may stop working if parts of the code contain errors and can't be parsed.

munair

  • Hero Member
  • *****
  • Posts: 884
  • compiler developer @SharpBASIC
    • SharpBASIC
Re: Include Informative Line Count Statistics
« Reply #11 on: May 13, 2025, 08:25:07 pm »
After several hours finding my way around the package manager and how to create packages, I actually wrote my first package "ProjectMetrics". I created it first on Lazarus 3.2 on Debian 12, and then tested it on Lazarus 4 on Windows 11. It seems to work fine.

Now how and where to upload/offer it?
It's only logical.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 11349
  • Debugger - SynEdit - and more
    • wiki
Re: Include Informative Line Count Statistics
« Reply #12 on: May 13, 2025, 08:31:09 pm »
Gitlab/Github/... would be a good place.

And then OPM?
(Though someone else needs to explain the steps)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 11349
  • Debugger - SynEdit - and more
    • wiki
Re: Include Informative Line Count Statistics
« Reply #13 on: May 13, 2025, 08:50:09 pm »
Oh, and congrats on your work.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12264
  • FPC developer.
Re: Include Informative Line Count Statistics
« Reply #14 on: May 13, 2025, 08:59:06 pm »
FPC reports compiled line stats, don't know if that filters conditional code though. Check your -v settings, and pass -B to compile as complete as possible.

 

TinyPortal © 2005-2018