Recent

Author Topic: Making the semicolon useless  (Read 24812 times)

Otto

  • Full Member
  • ***
  • Posts: 226
Re: Making the semicolon useless
« Reply #75 on: April 06, 2020, 11:32:43 am »
Hello PascalDragon.

It's not only about this specific topic (the semicolons), but your generalisation I don't agree with:

I believe that any proposal that aims to discuss a possible greater spread of Lazarus/FPC is to be considered.
I respect your opinion.

However, I believe that in order to increase the use of FPC/Lazarus in the professional field, it is necessary to assess the criticism of the users of these products. Especially when such users have made and continue to make an important contribution to the community (as e.g. circular did).

In the generalization I have made, I wanted to point out that a proposal can be taken into account without the need to accept it. The most important thing is, in my opinion, to discuss the cause that led a programmer to ask that specific question. Discussing the cause might be helpful for other programmers to discuss their opinions. Not all of us have had the same kind of training, so discussing basic topics could help someone clarify some aspect of programming in FPC that doesn't match the one in the language he had previously learned.

Of course what I said is just my opinion, I do not pretend to be right.

Otto.
Kind regards.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Making the semicolon useless
« Reply #76 on: April 06, 2020, 12:03:33 pm »
I wanted to point out that a proposal can be taken into account without the need to accept it. The most important thing is, in my opinion, to discuss the cause that led a programmer to ask that specific question. Discussing the cause might be helpful for other programmers to discuss their opinions. Not all of us have had the same kind of training, so discussing basic topics could help someone clarify some aspect of programming in FPC that doesn't match the one in the language he had previously learned.

I've always seen reasons given in rejections. So they were evaluated/considered.

Just as users request get the response, the users must also trust the developers in their judgement. :)

circular

  • Hero Member
  • *****
  • Posts: 4195
    • Personal webpage
Re: Making the semicolon useless
« Reply #77 on: April 06, 2020, 03:37:27 pm »
Since a compiler error needs that info. But the scanner (which includes the FPC internal preprocessor) and the parser are intertwined. It would be a colossal job to detangle them, create a new level inbetween to splice the external preprocessor into etc.
I see so I would not try that then.

Quote
Keep in mind the external tool would be extremely complicated since any manipulations must keep the token stream equivalent or the errors would be on the a different line and number. Study how the C preprocessor works etc etc.
That's ok because it is just appending a token at the end of the line.

Quote
And before to even attempt such thing (and then again, already purely hypothetical), you would have to demonstrate the concept using an own fork of FPC first, before one starts rearranging the whole parser.
I am not requesting that any action be taken as of now.

Quote
Anyway, I think this is a dead end. If you really want something like that, you're better off crafting some language based on syntax concepts of a language that has no need for statement terminators like python or basic.
I disagree, I think there is potential for an alternate source editor and for tools that convert text code from and to semicolon-less.

I will see if I have the time to try it out.
Conscience is the debugger of the mind

Otto

  • Full Member
  • ***
  • Posts: 226
Re: Making the semicolon useless
« Reply #78 on: April 06, 2020, 08:54:45 pm »
@GypsyPrince
Hello.
I have read the "De Principatibus" in the original language and I can assure you that it is a literary work of which it is very easy to misunderstand the meaning.
Surely your interpretation does not match mine.
In short, my interpretation (limited to the argument you mentioned) is that in order to maintain order in a state, the governor must be forced, even in spite of himself, to make unpopular decisions to protect the state and therefore its people. To counteract the bad faith of some individuals will sometimes have to use cunning other times force. In these cases the Prince's behavior is not despicable but necessary.
So if Moderators/Administrators used this kind of behavior within the Forums, it would be, in my opinion, only a good thing.

Short quote of the original text:

Quote
Sendo, dunque, uno principe necessitato sapere bene usare la bestia, debbe di quelle pigliare la golpe e il lione; perché il lione non si defende da’ lacci, la golpe non si defende da’ lupi. Bisogna, adunque, essere golpe a conoscere e’ lacci, e lione a sbigottire e’ lupi. Coloro che stanno semplicemente in sul lione, non se ne intendano. Non può, pertanto, uno signore prudente, né debbe, osservare la fede, quando tale osservanzia li torni contro e che sono spente le cagioni che la feciono promettere. E se gli uomini fussino tutti buoni, questo precetto non sarebbe buono; ma perché sono tristi, e non la osservarebbono a te, tu etiam non l’hai ad osservare a loro.
Kind regards.

Otto

  • Full Member
  • ***
  • Posts: 226
Re: Making the semicolon useless
« Reply #79 on: April 06, 2020, 08:58:55 pm »
@marcov

Hello.
I've always seen reasons given in rejections. So they were evaluated/considered.

Just as users request get the response, the users must also trust the developers in their judgement. :)
Marcov you are certainly right and I agree with you.
However, I do not understand why you made this statement to me. I have never said otherwise and I have never criticized the responses and judgments of the developers; if you had time I would ask you to check my previous posts.
The speech I had made earlier (which you only partially mentioned) had been made without polemical purpose; but just to better explain my previous post.

(-: I hope, this time, I'm better explained. :-)

Otto.
Kind regards.

440bx

  • Hero Member
  • *****
  • Posts: 3944
Re: Making the semicolon useless
« Reply #80 on: April 06, 2020, 09:27:12 pm »
I disagree, I think there is potential for an alternate source editor and for tools that convert text code from and to semicolon-less.

I will see if I have the time to try it out.
Putting aside my doubts about the feasibility, it seems really extreme to invest that much time and work to accomplish such a trivial and, not particularly useful, goal of getting rid of semicolons in Pascal.

Did a semicolon do something bad to you ? ;)  Here is a more pragmatic solution: go to church, admit your hatred of semicolons, repent, be forgiven after reciting 20 "our semicolon" and move on.



« Last Edit: April 06, 2020, 09:30:30 pm by 440bx »
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

circular

  • Hero Member
  • *****
  • Posts: 4195
    • Personal webpage
Re: Making the semicolon useless
« Reply #81 on: April 06, 2020, 11:32:24 pm »
Yeah, that wasn't even close to what I was talking about. I give up on this forum.
:'(
We can stick together and be strong despite negative feedbacks.

Here is a sample project to remove/add/fix semicolons:
Code: Delphi  [Select][+][-]
  1. program semicolonless;
  2.  
  3. {$mode objfpc}{$H+}
  4. {$MODESWITCH AdvancedRecords}
  5.  
  6. uses
  7.   {$IFDEF UNIX}{$IFDEF UseCThreads}
  8.   cthreads,
  9.   {$ENDIF}{$ENDIF}
  10.   Classes
  11.   { you can add units after this },
  12.   PScanner,
  13.   SysUtils;
  14.  
  15. type
  16.   TPascalScannerIgnoreDirectives = class; // this ending semicolon is necessary
  17.  
  18. var
  19.   outTokens: array of record
  20.     TokenKind: TToken;
  21.     TokenString: string;
  22.   end;
  23.   outTokenCount: integer;
  24.  
  25. procedure AddToOutput(ATokenType: TToken; ATokenString: string); forward;
  26. procedure InsertIntoOutput(AIndex: integer; ATokenType: TToken; ATokenString: string); forward;
  27.  
  28. const
  29.   TokensWhereSemicolonIndicatesEmptyStatement = [tkColon, tkdo, tkelse, tkthen{, tkOtherwise}];
  30.   TokensNeutrallyFollowedBySemicolon =
  31.    [tkIdentifier, tkString, tkNumber, tkChar, tkBraceClose, tkSquaredBraceClose, tkCaret,
  32.     tkdispinterface, tkend, tkfalse, tkfile, tkinherited, tkinline, tkinterface,
  33.     tknil, tkobject, tkself, tkthreadvar, tktrue]; { - private protected public published otherwise }
  34.   TokensNotIndicatingStatementContinuation =
  35.    [tkWhitespace, tkComment, tkIdentifier, tkString, tkNumber, tkChar,
  36.     tkBraceOpen, tkAt, tkasm, tkbegin, tkcase, tkconst, tkend, tkexports,
  37.     tkfinalization, tkfor, tkfunction, tkgoto, tkif, tkimplementation, tkinherited,
  38.     tkinitialization, tklabel, tklibrary, tkoperator, tkprocedure, tkprogram, tkproperty,
  39.     tkraise, tkrepeat, tkResourceString, tkthreadvar, tktry, tktype, tkunit, tkuntil,
  40.     tkuses, tkvar, tkwhile, tkwith];
  41.  
  42. type
  43.   { TPascalScannerIgnoreDirectives }
  44.  
  45.   TPascalScannerIgnoreDirectives = class(TPascalScanner)
  46.   protected
  47.     function HandleDirective(const {%H-}ADirectiveText: String): TToken; override;
  48.   end;
  49.  
  50. function TPascalScannerIgnoreDirectives.HandleDirective(const ADirectiveText: String): TToken;
  51. begin
  52.   Result:= tkComment; // treat directives as comments
  53. end;
  54.  
  55. {********************** Output *********************}
  56.  
  57. procedure AddToOutput(ATokenType: TToken; ATokenString: string);
  58. begin
  59.   if outTokenCount >= length(outTokens) then
  60.     SetLength(outTokens, outTokenCount * 2 + 4);
  61.   outTokens[outTokenCount].TokenKind:= ATokenType;
  62.   outTokens[outTokenCount].TokenString:= ATokenString;
  63.   inc(outTokenCount);
  64. end;
  65.  
  66. procedure InsertIntoOutput(AIndex: integer; ATokenType: TToken; ATokenString: string);
  67. var
  68.   i: Integer;
  69. begin
  70.   if (AIndex < 0) or (AIndex > outTokenCount) then raise exception.Create('Index out of bounds');
  71.   if outTokenCount >= length(outTokens) then
  72.     SetLength(outTokens, outTokenCount * 2 + 4);
  73.   for i := outTokenCount-1 downto AIndex do
  74.     outTokens[i+1] := outTokens[i];
  75.   outTokens[AIndex].TokenKind:= ATokenType;
  76.   outTokens[AIndex].TokenString:= ATokenString;
  77.   inc(outTokenCount);
  78. end;
  79.  
  80. procedure ReadInput(AInputName: string);
  81. var
  82.   resolver: TFileResolver;
  83.   scanner: TPascalScanner;
  84.   curToken: TToken;
  85.   curColStart, curColEnd: integer;
  86.   curInputLine: string;
  87.  
  88.   procedure FetchNextToken;
  89.   var
  90.     inputRow: Integer;
  91.   begin
  92.     inputRow := scanner.CurRow;
  93.     curColStart := scanner.CurColumn;
  94.     curInputLine := scanner.CurLine;
  95.     curToken := scanner.FetchToken;
  96.     if scanner.CurRow > inputRow then
  97.       curColEnd := length(curInputLine)
  98.       else curColEnd:= scanner.CurColumn;
  99.   end;
  100.  
  101. begin
  102.   outTokens := nil;
  103.   outTokenCount := 0;
  104.   resolver := TFileResolver.Create;
  105.   scanner := TPascalScannerIgnoreDirectives.Create(resolver);
  106.   try
  107.     scanner.OpenFile(AInputName);
  108.     FetchNextToken;
  109.     while curToken <> tkEOF do
  110.     begin
  111.       case curToken of
  112.         tkWhitespace, tkComment: AddToOutput(curToken,
  113.             copy(curInputLine, curColStart + 1, curColEnd - curColStart));
  114.         tkIdentifier..tkChar: AddToOutput(curToken, scanner.CurTokenString);
  115.         tkTab: AddToOutput(tkTab, #9); (* keep tabs *)
  116.         tkLineEnding: AddToOutput(tkLineEnding, LineEnding);
  117.         otherwise
  118.           AddToOutput(curToken, TokenInfos[curToken]);
  119.       end;
  120.       FetchNextToken;
  121.     end;
  122.   finally
  123.     scanner.Free;
  124.     resolver.Free;
  125.   end;
  126. end;
  127.  
  128. procedure WriteOutput(AOutputName: string);
  129. var
  130.   i: Integer;
  131.   fout: TextFile;
  132. begin
  133.   assignfile(fout, AOutputName);
  134.   Rewrite(fout);
  135.   for i := 0 to outTokenCount-1 do
  136.     write(fout, outTokens[i].TokenString);
  137.   closefile(fout);
  138. end;
  139.  
  140. {******************** Processing ********************}
  141.  
  142. procedure RemoveSemicolonsInOutput;
  143. var
  144.   i, j: Integer;
  145.   prevToken, nextToken, nextLineToken: TToken;
  146.   nextTokenPos: integer;
  147.   prevTokenStr: String;
  148. begin
  149.   for i := 0 to outTokenCount-1 do
  150.     if (outTokens[i].TokenKind = tkSemicolon) then
  151.     begin
  152.       nextToken := tkLineEnding;
  153.       nextTokenPos:= i;
  154.       for j := i+1 to outTokenCount-1 do
  155.         if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace] then
  156.           continue else
  157.         begin
  158.           nextToken := outTokens[j].TokenKind;
  159.           nextTokenPos := j;
  160.           break;
  161.         end;
  162.       // found a semicolon that might be removed
  163.       if nextToken = tkLineEnding then
  164.       begin
  165.         nextLineToken := tkEOF;
  166.         for j := nextTokenPos+1 to outTokenCount-1 do
  167.           if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding] then
  168.             continue else
  169.           begin
  170.             nextLineToken:= outTokens[j].TokenKind;
  171.             break;
  172.           end;
  173.         if nextLineToken in TokensNotIndicatingStatementContinuation then
  174.         begin
  175.           prevToken := tkLineEnding;
  176.           prevTokenStr := '';
  177.           for j := i-1 downto 0 do
  178.             if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding]
  179.                 then continue else
  180.               begin
  181.                 prevToken := outTokens[j].TokenKind;
  182.                 prevTokenStr := outTokens[j].TokenString;
  183.                 break;
  184.               end;
  185.           if (prevToken in TokensWhereSemicolonIndicatesEmptyStatement) or
  186.                   (compareText(prevTokenStr, 'otherwise') = 0) then
  187.           begin
  188.             outTokens[i].TokenKind:= tknil;
  189.             outTokens[i].TokenString:= 'nil';
  190.           end
  191.           else if (prevToken in TokensNeutrallyFollowedBySemicolon) and
  192.                   (compareText(prevTokenStr, 'otherwise') <> 0) and
  193.                   (compareText(prevTokenStr, 'private') <> 0) and
  194.                   (compareText(prevTokenStr, 'protected') <> 0) and
  195.                   (compareText(prevTokenStr, 'public') <> 0) and
  196.                   (compareText(prevTokenStr, 'published') <> 0) then
  197.           begin
  198.             outTokens[i].TokenKind:= tkWhitespace;
  199.             outTokens[i].TokenString:= '';
  200.           end;
  201.         end;
  202.       end;
  203.     end;
  204. end;
  205.  
  206. procedure RemoveSemicolons(AInputName, AOutputName : string);
  207. begin
  208.   ReadInput(AInputName);
  209.   RemoveSemicolonsInOutput;
  210.   WriteOutput(AOutputName);
  211. end;
  212.  
  213. procedure AddSemicolonsInOutput;
  214. var
  215.   i, j, prevTokenPos: Integer;
  216.   prevToken, prevToken2, nextToken, classToken: TToken;
  217.   prevTokenStr, prevTokenStr2: String;
  218.   forFound: Boolean;
  219. begin
  220.   for i := outTokenCount downto 0 do
  221.     if (i = outTokenCount) or (outTokens[i].TokenKind = tkLineEnding) then
  222.     begin
  223.       prevToken := tkLineEnding;
  224.       prevTokenPos := i;
  225.       for j := i-1 downto 0 do
  226.         if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace] then
  227.           continue else
  228.         begin
  229.           prevToken := outTokens[j].TokenKind;
  230.           prevTokenPos := j;
  231.           prevTokenStr := outTokens[j].TokenString;
  232.           break;
  233.         end;
  234.       if prevToken = tknil then
  235.       begin
  236.         prevToken2 := tkLineEnding;
  237.         prevTokenStr2 := '';
  238.         for j := prevTokenPos-1 downto 0 do
  239.           if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding] then
  240.             continue else
  241.           begin
  242.             prevToken2 := outTokens[j].TokenKind;
  243.             prevTokenStr2 := outTokens[j].TokenString;
  244.             break;
  245.           end;
  246.         if (prevToken2 in TokensWhereSemicolonIndicatesEmptyStatement) or
  247.            (CompareText(prevTokenStr2, 'otherwise')=0) then
  248.         begin
  249.           outTokens[prevTokenPos].TokenKind := tkSemicolon;
  250.           outTokens[prevTokenPos].TokenString:= ';';
  251.           continue;
  252.         end;
  253.       end;
  254.       if (prevToken in TokensNeutrallyFollowedBySemicolon) and
  255.         (compareText(prevTokenStr, 'otherwise') <> 0) and
  256.         (compareText(prevTokenStr, 'private') <> 0) and
  257.         (compareText(prevTokenStr, 'protected') <> 0) and
  258.         (compareText(prevTokenStr, 'public') <> 0) and
  259.         (compareText(prevTokenStr, 'published') <> 0) then
  260.       begin
  261.         // class|record|type helper for T
  262.         if prevToken = tkIdentifier then
  263.         begin
  264.           forFound := false;
  265.           for j := prevTokenPos-1 downto 0 do
  266.             if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding,
  267.                                           tkDot, tkIdentifier] then
  268.               continue else
  269.             if outTokens[j].TokenKind = tkfor then
  270.             begin
  271.               forFound := true;
  272.               break;
  273.             end else break;
  274.           if forFound then continue;
  275.         end;
  276.         // type class|object|interface|dispinterface(parent1...)
  277.         if prevToken = tkBraceClose then
  278.         begin
  279.           classToken:= tkLineEnding;
  280.           for j := prevTokenPos-1 downto 0 do
  281.             if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding,
  282.                                           tkDot, tkIdentifier, tkComma,
  283.                                           tkGreaterThan, tkLessThan, tkspecialize] then
  284.               continue else
  285.             if outTokens[j].TokenKind = tkBraceOpen then
  286.             begin
  287.               if j > 0 then classToken := outTokens[j-1].TokenKind;
  288.               break;
  289.             end else break;
  290.           if classToken in[tkclass, tkobject, tkinterface, tkdispinterface] then continue;
  291.         end;
  292.  
  293.         nextToken := tkLineEnding;
  294.         for j := i+1 to outTokenCount-1 do
  295.           if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding] then
  296.             continue else
  297.           begin
  298.             nextToken := outTokens[j].TokenKind;
  299.             break;
  300.           end;
  301.         if nextToken in TokensNotIndicatingStatementContinuation then
  302.           InsertIntoOutput(prevTokenPos+1, tkSemicolon, ';');
  303.       end;
  304.     end;
  305. end;
  306.  
  307. procedure AddSemicolons(AInputName, AOutputName : string);
  308. begin
  309.   ReadInput(AInputName);
  310.   AddSemicolonsInOutput;
  311.   WriteOutput(AOutputName);
  312. end;
  313.  
  314. procedure FixSemicolons(AInputName, AOutputName : string);
  315. begin
  316.   ReadInput(AInputName);
  317.   RemoveSemicolonsInOutput;
  318.   AddSemicolonsInOutput;
  319.   WriteOutput(AOutputName);
  320. end;
  321.  
  322. {********************** Testing *********************}
  323.  
  324. type
  325.   { TPointHelper }
  326.  
  327.   TPointHelper = record helper for TPoint
  328.     function Sum: integer;
  329.   end;
  330.  
  331.   function TPointHelper.Sum: integer;
  332.   begin
  333.     result := x + y;
  334.   end;
  335.  
  336. procedure TestStatements;
  337. var c: char;
  338. begin
  339.   write('hihi');;
  340.  
  341.   //test empty statement
  342.   if true then ;
  343.  
  344.   //test case else
  345.   case random(4) of
  346.     1: ;
  347.     7: if true then write('haha');
  348.     else write('hoho');
  349.   end;
  350.  
  351.   c := 'A';
  352.   case c of
  353.     'B': ;
  354.     'C': if true then write('haha');
  355.     otherwise
  356.       ;
  357.   end;
  358. end;
  359.  
  360. {******************* Main program *******************}
  361.  
  362. var
  363.   sourceFile, targetFile, mode: string;
  364.  
  365. begin
  366.   if ParamCount < 2 then
  367.   begin
  368.     writeln('Usage: semicolonless -L|-M|-F <input file> [<output file>]');
  369.     writeln;
  370.     writeln('-L : less semicolons (default extension .less)');
  371.     writeln('-M : more semicolons (default extension .pas)');
  372.     writeln('-F : fix semicolons');
  373.     halt;
  374.   end;
  375.  
  376.   mode := paramstr(1);
  377.   if copy(mode,1,1) <> '-' then
  378.   begin
  379.     writeln('Expecting mode');
  380.     halt;
  381.   end;
  382.  
  383.   sourceFile := paramstr(2);
  384.   if not FileExists(sourceFile) then
  385.   begin
  386.     writeln('Input file not found');
  387.     halt;
  388.   end;
  389.  
  390.   if ParamCount >= 3 then
  391.     targetFile := paramStr(3)
  392.     else targetFile := '';
  393.  
  394.   if mode = '-L' then
  395.   begin
  396.     if targetFile = '' then targetFile := ChangeFileExt(sourceFile,'.less') else
  397.     if ExtractFileExt(targetFile) = '' then targetFile += '.less';
  398.     RemoveSemicolons(sourceFile, targetFile);
  399.   end else
  400.   if mode = '-M' then
  401.   begin
  402.     if targetFile = '' then targetFile := ChangeFileExt(sourceFile, '.pas') else
  403.     if ExtractFileExt(targetFile) = '' then targetFile += '.pas';
  404.     AddSemicolons(sourceFile, targetFile);
  405.   end else
  406.   if mode = '-F' then
  407.   begin
  408.     if targetFile = '' then targetFile := sourceFile;
  409.     FixSemicolons(sourceFile, targetFile);
  410.   end else
  411.     writeln('Unknown mode');
  412. end.
Conscience is the debugger of the mind

circular

  • Hero Member
  • *****
  • Posts: 4195
    • Personal webpage
Re: Making the semicolon useless
« Reply #82 on: April 06, 2020, 11:33:51 pm »
Same code without useless semicolons:
Code: Delphi  [Select][+][-]
  1. program semicolonless
  2.  
  3. {$mode objfpc}{$H+}
  4. {$MODESWITCH AdvancedRecords}
  5.  
  6. uses
  7.   {$IFDEF UNIX}{$IFDEF UseCThreads}
  8.   cthreads,
  9.   {$ENDIF}{$ENDIF}
  10.   Classes
  11.   { you can add units after this },
  12.   PScanner,
  13.   SysUtils
  14.  
  15. type
  16.   TPascalScannerIgnoreDirectives = class; // this ending semicolon is necessary
  17.  
  18. var
  19.   outTokens: array of record
  20.     TokenKind: TToken
  21.     TokenString: string
  22.   end
  23.   outTokenCount: integer
  24.  
  25. procedure AddToOutput(ATokenType: TToken; ATokenString: string); forward
  26. procedure InsertIntoOutput(AIndex: integer; ATokenType: TToken; ATokenString: string); forward
  27.  
  28. const
  29.   TokensWhereSemicolonIndicatesEmptyStatement = [tkColon, tkdo, tkelse, tkthen{, tkOtherwise}]
  30.   TokensNeutrallyFollowedBySemicolon =
  31.    [tkIdentifier, tkString, tkNumber, tkChar, tkBraceClose, tkSquaredBraceClose, tkCaret,
  32.     tkdispinterface, tkend, tkfalse, tkfile, tkinherited, tkinline, tkinterface,
  33.     tknil, tkobject, tkself, tkthreadvar, tktrue] { - private protected public published otherwise }
  34.   TokensNotIndicatingStatementContinuation =
  35.    [tkWhitespace, tkComment, tkIdentifier, tkString, tkNumber, tkChar,
  36.     tkBraceOpen, tkAt, tkasm, tkbegin, tkcase, tkconst, tkend, tkexports,
  37.     tkfinalization, tkfor, tkfunction, tkgoto, tkif, tkimplementation, tkinherited,
  38.     tkinitialization, tklabel, tklibrary, tkoperator, tkprocedure, tkprogram, tkproperty,
  39.     tkraise, tkrepeat, tkResourceString, tkthreadvar, tktry, tktype, tkunit, tkuntil,
  40.     tkuses, tkvar, tkwhile, tkwith]
  41.  
  42. type
  43.   { TPascalScannerIgnoreDirectives }
  44.  
  45.   TPascalScannerIgnoreDirectives = class(TPascalScanner)
  46.   protected
  47.     function HandleDirective(const {%H-}ADirectiveText: String): TToken; override
  48.   end
  49.  
  50. function TPascalScannerIgnoreDirectives.HandleDirective(const ADirectiveText: String): TToken
  51. begin
  52.   Result:= tkComment // treat directives as comments
  53. end
  54.  
  55. {********************** Output *********************}
  56.  
  57. procedure AddToOutput(ATokenType: TToken; ATokenString: string)
  58. begin
  59.   if outTokenCount >= length(outTokens) then
  60.     SetLength(outTokens, outTokenCount * 2 + 4)
  61.   outTokens[outTokenCount].TokenKind:= ATokenType
  62.   outTokens[outTokenCount].TokenString:= ATokenString
  63.   inc(outTokenCount)
  64. end
  65.  
  66. procedure InsertIntoOutput(AIndex: integer; ATokenType: TToken; ATokenString: string)
  67. var
  68.   i: Integer
  69. begin
  70.   if (AIndex < 0) or (AIndex > outTokenCount) then raise exception.Create('Index out of bounds')
  71.   if outTokenCount >= length(outTokens) then
  72.     SetLength(outTokens, outTokenCount * 2 + 4)
  73.   for i := outTokenCount-1 downto AIndex do
  74.     outTokens[i+1] := outTokens[i]
  75.   outTokens[AIndex].TokenKind:= ATokenType
  76.   outTokens[AIndex].TokenString:= ATokenString
  77.   inc(outTokenCount)
  78. end
  79.  
  80. procedure ReadInput(AInputName: string)
  81. var
  82.   resolver: TFileResolver
  83.   scanner: TPascalScanner
  84.   curToken: TToken
  85.   curColStart, curColEnd: integer
  86.   curInputLine: string
  87.  
  88.   procedure FetchNextToken
  89.   var
  90.     inputRow: Integer
  91.   begin
  92.     inputRow := scanner.CurRow
  93.     curColStart := scanner.CurColumn
  94.     curInputLine := scanner.CurLine
  95.     curToken := scanner.FetchToken
  96.     if scanner.CurRow > inputRow then
  97.       curColEnd := length(curInputLine)
  98.       else curColEnd:= scanner.CurColumn
  99.   end
  100.  
  101. begin
  102.   outTokens := nil
  103.   outTokenCount := 0
  104.   resolver := TFileResolver.Create
  105.   scanner := TPascalScannerIgnoreDirectives.Create(resolver)
  106.   try
  107.     scanner.OpenFile(AInputName)
  108.     FetchNextToken
  109.     while curToken <> tkEOF do
  110.     begin
  111.       case curToken of
  112.         tkWhitespace, tkComment: AddToOutput(curToken,
  113.             copy(curInputLine, curColStart + 1, curColEnd - curColStart))
  114.         tkIdentifier..tkChar: AddToOutput(curToken, scanner.CurTokenString)
  115.         tkTab: AddToOutput(tkTab, #9) (* keep tabs *)
  116.         tkLineEnding: AddToOutput(tkLineEnding, LineEnding)
  117.         otherwise
  118.           AddToOutput(curToken, TokenInfos[curToken])
  119.       end
  120.       FetchNextToken
  121.     end;
  122.   finally
  123.     scanner.Free
  124.     resolver.Free
  125.   end
  126. end
  127.  
  128. procedure WriteOutput(AOutputName: string)
  129. var
  130.   i: Integer
  131.   fout: TextFile
  132. begin
  133.   assignfile(fout, AOutputName)
  134.   Rewrite(fout)
  135.   for i := 0 to outTokenCount-1 do
  136.     write(fout, outTokens[i].TokenString)
  137.   closefile(fout)
  138. end
  139.  
  140. {******************** Processing ********************}
  141.  
  142. procedure RemoveSemicolonsInOutput
  143. var
  144.   i, j: Integer
  145.   prevToken, nextToken, nextLineToken: TToken
  146.   nextTokenPos: integer
  147.   prevTokenStr: String
  148. begin
  149.   for i := 0 to outTokenCount-1 do
  150.     if (outTokens[i].TokenKind = tkSemicolon) then
  151.     begin
  152.       nextToken := tkLineEnding
  153.       nextTokenPos:= i
  154.       for j := i+1 to outTokenCount-1 do
  155.         if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace] then
  156.           continue else
  157.         begin
  158.           nextToken := outTokens[j].TokenKind
  159.           nextTokenPos := j
  160.           break
  161.         end
  162.       // found a semicolon that might be removed
  163.       if nextToken = tkLineEnding then
  164.       begin
  165.         nextLineToken := tkEOF
  166.         for j := nextTokenPos+1 to outTokenCount-1 do
  167.           if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding] then
  168.             continue else
  169.           begin
  170.             nextLineToken:= outTokens[j].TokenKind
  171.             break
  172.           end
  173.         if nextLineToken in TokensNotIndicatingStatementContinuation then
  174.         begin
  175.           prevToken := tkLineEnding
  176.           prevTokenStr := ''
  177.           for j := i-1 downto 0 do
  178.             if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding]
  179.                 then continue else
  180.               begin
  181.                 prevToken := outTokens[j].TokenKind
  182.                 prevTokenStr := outTokens[j].TokenString
  183.                 break
  184.               end
  185.           if (prevToken in TokensWhereSemicolonIndicatesEmptyStatement) or
  186.                   (compareText(prevTokenStr, 'otherwise') = 0) then
  187.           begin
  188.             outTokens[i].TokenKind:= tknil
  189.             outTokens[i].TokenString:= 'nil'
  190.           end
  191.           else if (prevToken in TokensNeutrallyFollowedBySemicolon) and
  192.                   (compareText(prevTokenStr, 'otherwise') <> 0) and
  193.                   (compareText(prevTokenStr, 'private') <> 0) and
  194.                   (compareText(prevTokenStr, 'protected') <> 0) and
  195.                   (compareText(prevTokenStr, 'public') <> 0) and
  196.                   (compareText(prevTokenStr, 'published') <> 0) then
  197.           begin
  198.             outTokens[i].TokenKind:= tkWhitespace
  199.             outTokens[i].TokenString:= ''
  200.           end
  201.         end
  202.       end
  203.     end
  204. end
  205.  
  206. procedure RemoveSemicolons(AInputName, AOutputName : string)
  207. begin
  208.   ReadInput(AInputName)
  209.   RemoveSemicolonsInOutput
  210.   WriteOutput(AOutputName)
  211. end
  212.  
  213. procedure AddSemicolonsInOutput
  214. var
  215.   i, j, prevTokenPos: Integer
  216.   prevToken, prevToken2, nextToken, classToken: TToken
  217.   prevTokenStr, prevTokenStr2: String
  218.   forFound: Boolean
  219. begin
  220.   for i := outTokenCount downto 0 do
  221.     if (i = outTokenCount) or (outTokens[i].TokenKind = tkLineEnding) then
  222.     begin
  223.       prevToken := tkLineEnding
  224.       prevTokenPos := i
  225.       for j := i-1 downto 0 do
  226.         if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace] then
  227.           continue else
  228.         begin
  229.           prevToken := outTokens[j].TokenKind
  230.           prevTokenPos := j
  231.           prevTokenStr := outTokens[j].TokenString
  232.           break
  233.         end
  234.       if prevToken = tknil then
  235.       begin
  236.         prevToken2 := tkLineEnding
  237.         prevTokenStr2 := ''
  238.         for j := prevTokenPos-1 downto 0 do
  239.           if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding] then
  240.             continue else
  241.           begin
  242.             prevToken2 := outTokens[j].TokenKind
  243.             prevTokenStr2 := outTokens[j].TokenString
  244.             break
  245.           end
  246.         if (prevToken2 in TokensWhereSemicolonIndicatesEmptyStatement) or
  247.            (CompareText(prevTokenStr2, 'otherwise')=0) then
  248.         begin
  249.           outTokens[prevTokenPos].TokenKind := tkSemicolon
  250.           outTokens[prevTokenPos].TokenString:= ';'
  251.           continue
  252.         end
  253.       end
  254.       if (prevToken in TokensNeutrallyFollowedBySemicolon) and
  255.         (compareText(prevTokenStr, 'otherwise') <> 0) and
  256.         (compareText(prevTokenStr, 'private') <> 0) and
  257.         (compareText(prevTokenStr, 'protected') <> 0) and
  258.         (compareText(prevTokenStr, 'public') <> 0) and
  259.         (compareText(prevTokenStr, 'published') <> 0) then
  260.       begin
  261.         // class|record|type helper for T
  262.         if prevToken = tkIdentifier then
  263.         begin
  264.           forFound := false
  265.           for j := prevTokenPos-1 downto 0 do
  266.             if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding,
  267.                                           tkDot, tkIdentifier] then
  268.               continue else
  269.             if outTokens[j].TokenKind = tkfor then
  270.             begin
  271.               forFound := true
  272.               break
  273.             end else break
  274.           if forFound then continue
  275.         end
  276.         // type class|object|interface|dispinterface(parent1...)
  277.         if prevToken = tkBraceClose then
  278.         begin
  279.           classToken:= tkLineEnding
  280.           for j := prevTokenPos-1 downto 0 do
  281.             if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding,
  282.                                           tkDot, tkIdentifier, tkComma,
  283.                                           tkGreaterThan, tkLessThan, tkspecialize] then
  284.               continue else
  285.             if outTokens[j].TokenKind = tkBraceOpen then
  286.             begin
  287.               if j > 0 then classToken := outTokens[j-1].TokenKind
  288.               break
  289.             end else break
  290.           if classToken in[tkclass, tkobject, tkinterface, tkdispinterface] then continue
  291.         end
  292.  
  293.         nextToken := tkLineEnding
  294.         for j := i+1 to outTokenCount-1 do
  295.           if outTokens[j].TokenKind in [tkComment, tkTab, tkWhiteSpace, tkLineEnding] then
  296.             continue else
  297.           begin
  298.             nextToken := outTokens[j].TokenKind
  299.             break
  300.           end
  301.         if nextToken in TokensNotIndicatingStatementContinuation then
  302.           InsertIntoOutput(prevTokenPos+1, tkSemicolon, ';')
  303.       end
  304.     end
  305. end
  306.  
  307. procedure AddSemicolons(AInputName, AOutputName : string)
  308. begin
  309.   ReadInput(AInputName)
  310.   AddSemicolonsInOutput
  311.   WriteOutput(AOutputName)
  312. end
  313.  
  314. procedure FixSemicolons(AInputName, AOutputName : string)
  315. begin
  316.   ReadInput(AInputName)
  317.   RemoveSemicolonsInOutput
  318.   AddSemicolonsInOutput
  319.   WriteOutput(AOutputName)
  320. end
  321.  
  322. {********************** Testing *********************}
  323.  
  324. type
  325.   { TPointHelper }
  326.  
  327.   TPointHelper = record helper for TPoint
  328.     function Sum: integer
  329.   end
  330.  
  331.   function TPointHelper.Sum: integer
  332.   begin
  333.     result := x + y
  334.   end
  335.  
  336. procedure TestStatements
  337. var c: char
  338. begin
  339.   write('hihi');;
  340.  
  341.   //test empty statement
  342.   if true then nil
  343.  
  344.   //test case else
  345.   case random(4) of
  346.     1: nil
  347.     7: if true then write('haha');
  348.     else write('hoho')
  349.   end
  350.  
  351.   c := 'A'
  352.   case c of
  353.     'B': nil
  354.     'C': if true then write('haha')
  355.     otherwise
  356.       nil
  357.   end
  358. end
  359.  
  360. {******************* Main program *******************}
  361.  
  362. var
  363.   sourceFile, targetFile, mode: string
  364.  
  365. begin
  366.   if ParamCount < 2 then
  367.   begin
  368.     writeln('Usage: semicolonless -L|-M|-F <input file> [<output file>]')
  369.     writeln
  370.     writeln('-L : less semicolons (default extension .less)')
  371.     writeln('-M : more semicolons (default extension .pas)')
  372.     writeln('-F : fix semicolons')
  373.     halt
  374.   end
  375.  
  376.   mode := paramstr(1)
  377.   if copy(mode,1,1) <> '-' then
  378.   begin
  379.     writeln('Expecting mode')
  380.     halt
  381.   end
  382.  
  383.   sourceFile := paramstr(2)
  384.   if not FileExists(sourceFile) then
  385.   begin
  386.     writeln('Input file not found')
  387.     halt
  388.   end
  389.  
  390.   if ParamCount >= 3 then
  391.     targetFile := paramStr(3)
  392.     else targetFile := ''
  393.  
  394.   if mode = '-L' then
  395.   begin
  396.     if targetFile = '' then targetFile := ChangeFileExt(sourceFile,'.less') else
  397.     if ExtractFileExt(targetFile) = '' then targetFile += '.less'
  398.     RemoveSemicolons(sourceFile, targetFile)
  399.   end else
  400.   if mode = '-M' then
  401.   begin
  402.     if targetFile = '' then targetFile := ChangeFileExt(sourceFile, '.pas') else
  403.     if ExtractFileExt(targetFile) = '' then targetFile += '.pas'
  404.     AddSemicolons(sourceFile, targetFile)
  405.   end else
  406.   if mode = '-F' then
  407.   begin
  408.     if targetFile = '' then targetFile := sourceFile
  409.     FixSemicolons(sourceFile, targetFile)
  410.   end else
  411.     writeln('Unknown mode')
  412. end.
Conscience is the debugger of the mind

ASBzone

  • Hero Member
  • *****
  • Posts: 678
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
Re: Making the semicolon useless
« Reply #83 on: April 07, 2020, 12:08:12 am »
Same code without useless semicolons:
Code: Delphi  [Select][+][-]
  1. program semicolonless
  2. ...
  3. end.

I'm sorry...  I usually take great interest in the feature request posts (regardless of my own personal feelings about the feature in question, the discussions and reasoning are generally helpful, intriguing and enlightening to me.

For this specific request, however, my mind is boggled.

Not only can I not see what the point is, or what would be gained, but in order to help various parsers figure out what the programmer deemed a single statement to be, all line endings would become important, and an additional character would need to be provided to support multi-line statements.

 :o :o :o

Yes, yes, the request will be "optional", yet that code will have to exist and be a branch for the code that is used by people who have no desire for this feature.

I really can't see the benefit in the least.

And I looked at the semicolonless code and generated a seg fault.  The simultaneous recognition of pascal like syntax, but without semicolons, was quite odd.

Okay... Back to my corner...
-ASB: https://www.BrainWaveCC.com/

Lazarus v2.2.7-ada7a90186 / FPC v3.2.3-706-gaadb53e72c
(Windows 64-bit install w/Win32 and Linux/Arm cross-compiles via FpcUpDeluxe on both instances)

My Systems: Windows 10/11 Pro x64 (Current)

circular

  • Hero Member
  • *****
  • Posts: 4195
    • Personal webpage
Re: Making the semicolon useless
« Reply #84 on: April 07, 2020, 09:47:53 am »
To put things in perspective...

I made a comment on a topic that proposes a "Go-flavoured" Pascal compiler. And as you may know, Go does not have semicolons at the end of lines. So I suggested to remove the semicolons. This sparked a discussion and marcov reproached me of invading the thread with another topic. So I created this stand-alone topic.

I personally don't think semicolons are the most important thing. Though I find it beautiful without semicolons to be honest.

Quote
an additional character would need to be provided to support multi-line statements.
I am not sure, but I can see one case where this may be useful. For example if you make a call on the next line:
Code: Pascal  [Select][+][-]
  1. MyProc
  2. (MyParameter)

This would be interpreted as two lines. Though one can avoid that by writing:
Code: Pascal  [Select][+][-]
  1. MyProc(
  2. MyParameter)

In a way that makes the code less ambiguous anyway at first glance.
Conscience is the debugger of the mind

munair

  • Hero Member
  • *****
  • Posts: 798
  • compiler developer @SharpBASIC
    • SharpBASIC
Re: Making the semicolon useless
« Reply #85 on: April 07, 2020, 10:16:57 am »
The semicolon is useless most of the time. Though sometimes, it makes a difference. So I am thinking that we could simply avoid these cases and remove the semicolon completely. It seems that many people agree, though we can make it a compiler option so that anyone is free to do as they want.
I have not read all the comments to your idea so my comment might be superfluous, but I have two remarks.

1. Have you ever looked at how a compiler is designed and what kind of impact it has to change basic syntax rules? It might be analogous to designing a brand new language.

2. The semicolon as a statement terminator is very useful, both from a readability and from a practical point of view. Consider the following BASIC example without semicolon support:
Code: FreeBasic  [Select][+][-]
  1. buf = "command " _
  2.   + spec _
  3.   + ext _
  4.   + libs _
  5.   + switch _
  6.   + outp
... and the same with semicolon support:
Code: FreeBasic  [Select][+][-]
  1. buf = "command "
  2.   + spec
  3.   + ext
  4.   + libs
  5.   + switch
  6.   + outp;
As you can see, without a statement terminator, every statement must be written on a single line. IMO a statement terminator is preferable over a statement continuation symbol.
keep it simple

munair

  • Hero Member
  • *****
  • Posts: 798
  • compiler developer @SharpBASIC
    • SharpBASIC
Re: Making the semicolon useless
« Reply #86 on: April 07, 2020, 11:41:12 am »
For completeness, there is a difference between statement separators and statement terminators. In Pascal the semicolon is a statement separator, whereas in C it is a statement terminator. I prefer the latter.

Because Go was mentioned in this discussion, it also uses the semicolon as a statement separator. While optional, the Go compiler inserts it where omitted, which IMO is not a good approach.

I have been programming with BASIC type languages that use the line ending as statement terminator. While simple and seemingly convenient, many times I wished for an explicit terminator if it were only for code readability, which is always an issue with high level programming languages.

The only language that really qualifies for not needing a statement separator or terminator is assembly.
(EDIT: an explicit separator symbol of course)
« Last Edit: April 07, 2020, 12:23:24 pm by Munair »
keep it simple

lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: Making the semicolon useless
« Reply #87 on: April 07, 2020, 12:11:03 pm »
The only language that really qualifies for not needing a statement separator or terminator is assembly.

Note that (almost?) all assemblers do require that statements end (or is separated from the next) with a line end, and assembly sintax is (usually) even stricter than Pascal, which is why it's so (relatively) easy to write one. Not really a good example, is it?  ;D

Indeed, explicit statement separators/terminator where introduced to be able to disregard extra spaces, including line ends, and thus allow for "free-format" gramars (as compared to the strict column/line oriented ones of the far past).
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

munair

  • Hero Member
  • *****
  • Posts: 798
  • compiler developer @SharpBASIC
    • SharpBASIC
Re: Making the semicolon useless
« Reply #88 on: April 07, 2020, 12:19:57 pm »
The only language that really qualifies for not needing a statement separator or terminator is assembly.

Note that (almost?) all assemblers do require that statements end (or is separated from the next) with a line end, and assembly sintax is (usually) even stricter than Pascal, which is why it's so (relatively) easy to write one. Not really a good example, is it?  ;D
I was speaking in terms of explicit separators. I consider the line ending an implicit separator if it is treated as such by the compiler; it is impossible to forget a line ending by the programmer unless you put two statements on one line which really doesn't make sense: mov ax, 1 mov dx, 2  :(
keep it simple

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Making the semicolon useless
« Reply #89 on: April 07, 2020, 01:14:36 pm »
Because Go was mentioned in this discussion, it also uses the semicolon as a statement separator. While optional, the Go compiler inserts it where omitted, which IMO is not a good approach.

For Goers (Goists?), it would be interesting to see how GO solves the quoted error generation case.

 

TinyPortal © 2005-2018