Lazarus

Miscellaneous => Documentation (Maintaining -) => Topic started by: avra on July 22, 2021, 02:32:44 pm

Title: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: avra on July 22, 2021, 02:32:44 pm
This is my first FPC documentation update effort. I have expanded some type helpers in latest trunk code, so I would like to update sysutils documentation. When I click in Code Editor on method name (for example on HighestSetBitPos or ToHexString in TByteHelper) that I would like to see or update, I see their description in Lazarus FPDoc Editor and I can change them.

However as soon as I do this, original sysutils.xml encoding is changed from "ISO-8859-1" to "UTF-8", and many additional line endings get inserted (but only on specific tags - not all) as could be seen in attached screenshot. This means that after I make the needed changes, generated patch will be huge and probably rejected.

Can this be solved somehow? How to prevent FPDoc Editor in Lazarus making these unnecessary changes to XML documentation source?

I am trying to do this with 32bit fpcupdeluxe based trunks of both fpc and laz, on Win10 x64.
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: wp on July 22, 2021, 02:56:41 pm
Yes this is an annoying "feature" of the FPDoc editor. I already had reported it but Michael van Canneyt either did not understand me or was not interested. Please post a bug report (or wait a few days until the transition to git is completed).
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: avra on July 22, 2021, 03:05:35 pm
Does this happen on Linux, too? If it doesn't then as a temporary workaround I might try to update docs on Linux.
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: wp on July 22, 2021, 03:09:37 pm
Never tried, but since this is "high-level" code I'd assume that this issue is independent of OS.
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: Martin_fr on July 22, 2021, 03:10:40 pm
Yes this is an annoying "feature" of the FPDoc editor. I already had reported it but Michael van Canneyt either did not understand me or was not interested. Please post a bug report (or wait a few days until the transition to git is completed).

Does Michael maintain the IDE build in editor?
Afaik that is a separate package independent of the stand-alone fpdoc editor.

For a start point: ide/fpdoceditwindow.pas

Or did I misread the OT?
It sounded to me like the build in editor
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: avra on July 22, 2021, 11:42:04 pm
Or did I misread the OT?
It sounded to me like the build in editor
You're right. Bug is in the built in Lazarus FPDoc Editor.

For a start point: ide/fpdoceditwindow.pas
Tried a quick look but got lost pretty soon, so beyond my reach. Reported as https://bugs.freepascal.org/view.php?id=39282.

Can anyone confirm this bug on Linux Lazarus trunk?
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: avra on July 23, 2021, 01:55:34 am
I have just opened original "ISO-8859-1" encoding sysutils.xml with external LAZDE (Lazarus Documentation Editor) built from trunk, added short description to TByteHelper.ToBinString and saved XML. It was the same bug. Newly saved sysutils.xml had "UTF-8" encoding and many line endings added. The same as with simpler Lazarus built in FPDoc Editor described initially.
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: avra on July 23, 2021, 02:54:21 am
Just checked the Debian x64 with trunk and the bug is the same there, too.
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: avra on July 26, 2021, 03:53:59 pm
It looks like LAZDE is using WriteXMLFile() from xmlwrite to write content of TXMLDocument to file. According to https://wiki.lazarus.freepascal.org/xmlwrite, the only supported encoding is UTF8, '>' is converted to '>' because of forced character escaping, indent is forced and LineEndings are OS specific (TDOMWriter can help with indent and lineendings).

Before, and even after calling WriteXMLFile(FDocument, FN), encoding is still ISO-8859-1, so it is clear that WriteXMLFile() is to blame for creating a XML diff nightmare. Is there some alternative to WriteXMLFile() without using 3rd party XML lib?
Title: Re: FPDoc Editor in Lazarus changes XML files as soon as it opens it
Post by: avra on July 31, 2021, 12:38:47 am
Ok, here is a quick hack to fix FPC XML files after edited with FPDocEditor or LazDE, using simple replacement transformation pairs. XML will not be formatted like the original, but close enough for diff patch trial if patch applyer skips obvious. Tested only on sysutils.xml. Feel free to improve.

Code: Pascal  [Select][+][-]
  1. ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
  2. //                                                                                                               //
  3. //  fixfpcxml utility to fix FPC XML files after edited with FPDocEditor or LazDE, by Zeljko Avramovic (c) 2021  //
  4. //                                                                                                               //
  5. //  When FPDocEditor or LazDE are used for editing FPC XML documentation files (like sysutils.xml from RTL),     //
  6. //  encoding and formating of saved XML files are changed so much that creating a diff patch file is pointless.  //
  7. //  This utility tries to fix that. Although not perfect, result XML is much closer to the original format.      //
  8. //  Tested only on sysutils.xml. Feel free to improve it further.                                                //
  9. //                                                                                                               //
  10. //  Dual license:                                                                                                //
  11. //  1. FPC modified LGPL (chosen for compatibility with FreePascal and Lazarus)                                  //
  12. //  2. BSD3              (chosen for compatibility with everything else)                                         //
  13. //                                                                                                               //
  14. //  Usage:                                                                                                       //
  15. //  fixfpcxml <xmlfilename>                                                                                      //
  16. //                                                                                                               //
  17. //  Examples:                                                                                                    //
  18. //  fixfpcxml sysutils.xml                                                                                       //
  19. //  fixfpcxml c:\Prg\Lazarus\fpdoctrunk\sysutils.xml                                                             //
  20. //                                                                                                               //
  21. ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
  22.  
  23. program fixfpcxml;
  24.  
  25. {$mode objfpc} {$H+}
  26.  
  27. uses
  28.   classes, sysutils, strutils, dom, xmlwrite, xmlread;
  29.  
  30. procedure ShowHelpAndExit;
  31. begin
  32.   WriteLn('');
  33.   WriteLn('Usage:');
  34.   WriteLn('   fixfpcxml <xmlfilename>');
  35.   WriteLn('');
  36.   WriteLn('Examples:');
  37.   WriteLn('   fixfpcxml sysutils.xml');
  38.   WriteLn('   fixfpcxml c:\Prg\Lazarus\fpdoctrunk\sysutils.xml');
  39.   WriteLn('');
  40.   Halt;
  41. end;
  42.  
  43. const
  44.   CRLF: string = #13 + #10;
  45.   Blanks64 = '                                                                ';
  46.   Blanks14 = '              ';
  47.   Blanks10 = '          ';
  48.   Blanks4  = '    ';
  49. var
  50.   i: integer;
  51.   XmlFileName: string;
  52.   S: TFileStream;
  53.   W: TDOMWriter;
  54.   XmlDoc: TXMLDocument;
  55.   StrList: TStringList;
  56. begin
  57.   try
  58.     try
  59.       WriteLn('fixfpcxml utility to fix FPC XML files after edited with FPDocEditor or LazDE, by Zeljko Avramovic (c) 2021');
  60.       if ParamCount() <> 1 then
  61.         ShowHelpAndExit;
  62.  
  63.       XmlFileName := ParamStr(1);
  64.  
  65.       if not FileExists(XmlFileName) then
  66.       begin
  67.         WriteLn('ERROR: File ', XmlFileName, ' does not exist');
  68.         Exit;
  69.       end;
  70.  
  71.       if UpperCase(ExtractFileExt(Trim(XmlFileName))) <> '.XML' then
  72.       begin
  73.         WriteLn('ERROR: File ', XmlFileName, ' must have XML extension');
  74.         Exit;
  75.       end;
  76.  
  77.       ReadXMLFile(XmlDoc, XmlFileName);
  78.       S := TFileStream.Create(XmlFileName, fmCreate);
  79.       W := TDOMWriter.Create(S, XmlDoc);
  80.       W.UseTab := false;
  81.       W.IndentSize := 0;
  82.       W.WriteNode(XmlDoc);
  83.       W.Free;
  84.       S.Free;
  85.  
  86.       StrList := TStringList.Create;
  87.       StrList.LoadFromFile(XmlFileName);
  88.       StrList.Text := StringReplace(StrList.Text, 'encoding="utf-8"', 'encoding="ISO-8859-1"', [rfReplaceAll, rfIgnoreCase]);
  89.       StrList.Text := StringReplace(StrList.Text, '<descr>This documentation', '<descr>' + CRLF + 'This documentation', [rfReplaceAll, rfIgnoreCase]);
  90.       StrList.Text := StringReplace(StrList.Text, '&gt;', '>', [rfReplaceAll, rfIgnoreCase]);
  91.       StrList.Text := StringReplace(StrList.Text, '<p>', '<p>' + CRLF, [rfReplaceAll, rfIgnoreCase]);
  92.       StrList.Text := StringReplace(StrList.Text, CRLF + '<td>', '<td>', [rfReplaceAll, rfIgnoreCase]);
  93.       StrList.Text := StringReplace(StrList.Text, CRLF + '</th>', '</th>', [rfReplaceAll, rfIgnoreCase]);
  94.       StrList.Text := StringReplace(StrList.Text, CRLF + '</tr>', '</tr>', [rfReplaceAll, rfIgnoreCase]);
  95.       StrList.Text := StringReplace(StrList.Text, CRLF + '</td>', '</td>', [rfReplaceAll, rfIgnoreCase]);
  96.       StrList.Text := StringReplace(StrList.Text, CRLF + '</dd>', '</dd>', [rfReplaceAll, rfIgnoreCase]);
  97.       StrList.Text := StringReplace(StrList.Text, CRLF + '<dd>', '<dd>', [rfReplaceAll, rfIgnoreCase]);
  98.       StrList.Text := StringReplace(StrList.Text, '<dd>' + CRLF + '<var>', '<dd><var>', [rfReplaceAll, rfIgnoreCase]);
  99.       StrList.Text := StringReplace(StrList.Text, '<tr><td>' + CRLF, '<tr><td>', [rfReplaceAll, rfIgnoreCase]);
  100.       StrList.Text := StringReplace(StrList.Text, Blanks14 + CRLF, '', [rfReplaceAll, rfIgnoreCase]);
  101.       StrList.Text := StringReplace(StrList.Text, Blanks10 + CRLF, '', [rfReplaceAll, rfIgnoreCase]);
  102.       StrList.Text := StringReplace(StrList.Text, Blanks64, '', [rfReplaceAll, rfIgnoreCase]);
  103.       StrList.Text := StringReplace(StrList.Text, Blanks4, '', [rfReplaceAll, rfIgnoreCase]);
  104.       StrList.Text := StringReplace(StrList.Text, '<li>' + CRLF + '<link', '<li><link', [rfReplaceAll, rfIgnoreCase]);
  105.       StrList.Text := StringReplace(StrList.Text, CRLF + '<var>PChar', '<var>PChar', [rfReplaceAll, rfIgnoreCase]);
  106.       StrList.Text := StringReplace(StrList.Text, CRLF + '<var>TDateTime', '<var>TDateTime', [rfReplaceAll, rfIgnoreCase]);
  107.       StrList.Text := StringReplace(StrList.Text, CRLF + '<var>DateTime', '<var>DateTime', [rfReplaceAll, rfIgnoreCase]);
  108.       StrList.Text := StringReplace(StrList.Text, CRLF + '<var>ShortDate', '<var>ShortDate', [rfReplaceAll, rfIgnoreCase]);
  109.       StrList.Text := StringReplace(StrList.Text, CRLF + '<var>LongTime', '<var>LongTime', [rfReplaceAll, rfIgnoreCase]);
  110.       StrList.Text := StringReplace(StrList.Text, CRLF + '<printshort', '<printshort', [rfReplaceAll, rfIgnoreCase]);
  111.       StrList.Text := StringReplace(StrList.Text, '<errors>', '<errors>' + CRLF, [rfReplaceAll, rfIgnoreCase]);
  112.       StrList.Text := StringReplace(StrList.Text, CRLF + '  <var', '  <var', [rfReplaceAll, rfIgnoreCase]);
  113.       StrList.Text := StringReplace(StrList.Text, CRLF + '  ' + CRLF + '  ', '', [rfReplaceAll, rfIgnoreCase]);
  114.       StrList.Text := StringReplace(StrList.Text, CRLF + '  <var>', CRLF + '<var>', [rfReplaceAll, rfIgnoreCase]);
  115.       StrList.Text := StringReplace(StrList.Text, ' ' + CRLF + '<var>', ' <var>', [rfReplaceAll, rfIgnoreCase]);
  116.       StrList.Text := StringReplace(StrList.Text, ' ' + CRLF + '  <link', ' <link', [rfReplaceAll, rfIgnoreCase]);
  117.       StrList.Text := StringReplace(StrList.Text, ' ' + CRLF + '<link', ' <link', [rfReplaceAll, rfIgnoreCase]);
  118.       StrList.Text := StringReplace(StrList.Text, '/' + CRLF + '<link', '/<link', [rfReplaceAll, rfIgnoreCase]);
  119.       StrList.Text := StringReplace(StrList.Text, '<seealso/>', '<seealso>' + CRLF + '</seealso>', [rfReplaceAll, rfIgnoreCase]);
  120.       StrList.Text := StringReplace(StrList.Text, CRLF + '<file>C:', ' <file>C:', [rfReplaceAll, rfIgnoreCase]);
  121.       StrList.Text := StringReplace(StrList.Text, '<remark>', '<remark>' + CRLF, [rfReplaceAll, rfIgnoreCase]);
  122.       StrList.SaveToFile(XmlFileName);
  123.       StrList.Free;
  124.       WriteLn('Finished fixing file ', XmlFileName);
  125.     except
  126.       on e: Exception do
  127.         WriteLn(e.Message);
  128.     end;
  129.   finally
  130.     ;
  131.   end;
  132. end.
  133.  

If you do not change any old info, and just add new stuff which did not exist in original XML, then after saving XML from FPDocEditor all new stuff is at the end of the document, as shown in the screenshot. It may be optimal to just copy/paste and report these changes to FPC XML doc maintanier, instead of complicating with fixfpcxml.
TinyPortal © 2005-2018