If you've ever wanted to be able to use multi-line strings in FPC...

[jamie]

Why can't we have alternate C escape strings on the fly and simply allow the compiler to assemble the
string line by line until it encounters the closing Quote in use ?

 There is many locations in code where this would benefit everyone..

 I've seen versions of Pascal compilers, not FPC that can do this. it makes it clean looking to insert
line feeds  and also easy to port code from other languages.

[Akira1364]

Ok, engkin's third bug-find is fixed! Same as before: changes have been pushed to my github fork-branch, and there's also a new pair of patches on the bugtracker issue.

Additionally, I've now implemented both "ALL" and "AUTO" for MultiLineStringTrimLeft. To quote myself from a comment I just left on the tracker issue:

"ALL is basically just a more user-friendly way of passing the highest possible value, meaning it just guarantees that literally all leading whitespace is trimmed from each line (essentially by ensuring that the scanner runs out of whitespace to actually trim long before it decrements the temporary local trim count to zero, at which point it of course always stops immediately anyways).

AUTO, however, is based on the column position of the opening backtick of the current multi-line string in the source file, as some people had suggested would be what they'd want."

So, for example, BrunoK's SQL example can now be written like this:

Code: Pascal  [Select][+][-]

1. program tmultilinestring25;
2.  
3. {$modeswitch MultiLineStrings}
4. {$MultiLineStringTrimLeft Auto}
5.  
6. // ↓↓↓ interpreted the same as if {$MultiLineStringTrimLeft 10} was set
7.  
8. const
9.   Str1 = `SELECT o.*, C.Company
10.           from Orders O
11.           join Customer C
12.             on o.CustNo=C.ID
13.           where
14.             O.saledate=DATE '2001.03.20'`;
15.  
16. // ↓↓↓ interpreted the same as if {$MultiLineStringTrimLeft 5} was set
17.  
18. const
19.   Str2 =
20.     `SELECT o.*, C.Company
21.      from Orders O
22.      join Customer C
23.        on o.CustNo=C.ID
24.      where
25.        O.saledate=DATE '2001.03.20'`;
26.  
27. begin
28.   WriteLn(Str1);
29.   WriteLn(Str2);
30. end.
31.  

Which will give identical output for both strings:

Code: Text  [Select][+][-]

1. SELECT o.*, C.Company
2. from Orders O
3. join Customer C
4.   on o.CustNo=C.ID
5. where
6.   O.saledate=DATE '2001.03.20'
7. SELECT o.*, C.Company
8. from Orders O
9. join Customer C
10.   on o.CustNo=C.ID
11. where
12.   O.saledate=DATE '2001.03.20'

Hopefully this will make the feature a bit more convenient to use overall.

Lastly, again, if you're someone who is willing / able to do so, please feel free to test the feature and report any bugs you might find!

[VTwin]

AUTO, however, is based on the column position of the opening backtick of the current multi-line string in the source file, as some people had suggested would be what they'd want."

The AUTO feature does make it more intuitive, and probably less susceptible to typing errors.

[VTwin]

Having PLATFORM as the default would be much less expected IMO, and likely frustrating for anyone who did not want the compiler to impose a certain line ending that may have been not what was actually written.

Considering that the idea is to replace StrConst = 'Foo' + sLineBreak + 'Bar' making PLATFORM the default might indeed be the most sensible approach.

Using PLATFORM as default seems logical to me.

[VTwin]

Not a replacement for Akira's very nice work, but perhaps an option for some:

Code: Pascal  [Select][+][-]

1. operator/ (const a, b: string): string;
2. begin
3.   result := a + LineEnding + b;
4. end;
5.  
6. procedure TForm1.Button1Click(Sender: TObject);
7. var
8.   s: string;
9. begin
10.   s :=
11.     'Hello' /
12.     'World';
13.   Memo1.Append(s);
14. end;  

[Akira1364]

Using PLATFORM as default seems logical to me.

This seems to be the consensus, or something close to one at least. Will probably change it over to PLATFORM sometime tonight.

I also plan to re-use the variable that tracks the column position of the opening backticks for AUTO to add a multi-line-string specific error, shown in the event that one is "unbalanced" or "unterminated", that tells you where exactly it started (as opposed to the more general syntax error you'd get in that case currently.) Probably the last real addition to the feature I'll do for the time being.

Not a replacement for Akira's very nice work, but perhaps an option for some:

Code: Pascal  [Select][+][-]

1. operator/ (const a, b: string): string;
2. begin
3.   result := a + LineEnding + b;
4. end;
5.  
6. procedure TForm1.Button1Click(Sender: TObject);
7. var
8.   s: string;
9. begin
10.   s :=
11.     'Hello' /
12.     'World';
13.   Memo1.Append(s);
14. end;  

That works, but I would just... not recommend that you do it, ever. It's adding a non-trivial amount of runtime overhead to your application for no reason (as opposed to single-line or multi-line string concatenation with either the '+' operator or the actual Concat intrinsic function, which is always resolved entirely at compile time and results in a single constant string.)

Take a look at this Compiler Explorer link to get an idea of what I mean.

[VTwin]

That works, but I would just... not recommend that you do it, ever. It's adding a non-trivial amount of runtime overhead to your application for no reason (as opposed to single-line or multi-line string concatenation with either the '+' operator or the actual Concat intrinsic function, which is always resolved entirely at compile time and results in a single constant string.)

 "ever" Good point, and fair enough.

[JernejL]

Also, " is already known in the (Pascal) programming universe as a string delimiter of strings containing escaped characters. (E. g. the GNU Pascal compiler allowed that.)

Interesting, i cannot find any mention of that, i assume that FPC doesn't support that at all?
 

[BrunoK]

Using PLATFORM as default seems logical to me.

This seems to be the consensus, or something close to one at least. Will probably change it over to PLATFORM sometime tonight.

I also plan to re-use the variable that tracks the column position of the opening backticks for AUTO to add a multi-line-string specific error, shown in the event that one is "unbalanced" or "unterminated", that tells you where exactly it started (as opposed to the more general syntax error you'd get in that case currently.) Probably the last real addition to the feature I'll do for the time being.

Not a replacement for Akira's very nice work, but perhaps an option for some:

Code: Pascal  [Select][+][-]

1. operator/ (const a, b: string): string;
2. begin
3.   result := a + LineEnding + b;
4. end;
5.  
6. procedure TForm1.Button1Click(Sender: TObject);
7. var
8.   s: string;
9. begin
10.   s :=
11.     'Hello' /
12.     'World';
13.   Memo1.Append(s);
14. end;  

Akira, as I wrote somewhere above, the idea of having an easier looking way of expressing long string literals is very engaging.

But : we have only JEDI code formatter to salvage strangely indented code and this I need more than this new feature.

So you could slightly reconsider the syntax using kupferstecher / VTwin formulation.
It, of course, is not as easy to use as just having a backtick at the beginning and end, but pascal in general does not use position in the source code layout and a formulation that would remain 'conventional' seems to be a better idea.
The inconvenience is that it still would require quoting strings (and '' single quotes); in this regard this is not as good as the back tick.

Since you are in it, you could parse the literal applying this rule :
Any non terminated (terminator being ;) literal string that meets operator / outside a the single quoted literal is interpreted by the scanner as LineEnding +.
This would have the additional quality of letting // and {} comments appear in the source stream. This should also be simpler for adapting the JEDI code formatter taking in account the new syntax.

Now just a slightly more complicated SQL script straight out of a text file with the back tick idea, would it pass ?

Code: Pascal  [Select][+][-]

1.   DBIScript =
2.    `/* ImportEcr2012.sql : Reprendre toutes écritures 2012,
3.                            injecter dans écritures
4.                            mettre à jour références comptes DB/CR
5.                            Mettre à jour code catégorie selon Categorie de compte */
6.     /* SQL-92 Table Creation Script with DBISAM Extensions */
7.     DROP TABLE IF EXISTS "\Memory\EcrituresNew";
8.     CREATE TABLE IF NOT EXISTS "\Memory\EcrituresNew"
9.     (
10.        "UId" AUTOINC,
11.        "DateEcr" DATE,
12.        "DB" VARCHAR(10),
13.        "CR" VARCHAR(10),
14.        "Ref" VARCHAR(10),
15.        "Libelle" VARCHAR(100),
16.        "MntEcr" MONEY,
17.     PRIMARY KEY ("RecordID") COMPRESS NONE
18.     TEXT INDEX ("Libelle")
19.     STOP WORDS ':TEXT_INDEXPARAMS=_DBI_TIPS_FR :TEXT_INDEXRULES=CH_FR'
20.     SPACE CHARS #1+#2+#3+#4+#5+#6+#7+#8+#9+#10+#11+#12+#13+
21.     #14+#15+#16+#17+#18+#19+#20+#21+#22+#23+
22.     #24+#25+#26+#27+#28+#29+#30+#31+#32+'*+'+
23.     ',-./:;<=>\`'
24.     INCLUDE CHARS '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_ab' +
25.     'cdefghijklmnopqrstuvwxyz€‚ƒ„…†‡ˆ‰Š‹ŒŽ'+
26.     '‘’“”•–—˜™š›œžŸ ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·'+
27.     '¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß'+
28.     'àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþ'
29.     LOCALE CODE 4108
30.     USER MAJOR VERSION 1
31.     );
32.  
33.     import table "\Memory\EcrituresNew"
34.       from
35.         "C:\DBI_DB\Cresus08\Ecritures12.txt"
36.       DELIMITER #9
37.       Columns (DateEcr,DB,CR,Ref,Libelle,MntEcr)
38.       DATE 'dd.mm.yy'
39.       DECIMAL '.';
40.  
41.     DROP INDEX IF EXISTS Ecritures.ixDB;
42.     DROP INDEX IF EXISTS Ecritures.ixCR;
43.     DROP INDEX IF EXISTS Ecritures.ixRef;
44.  
45.     delete  FROM Ecritures
46.       where DateEcr>='2012-01-01';
47.  
48.     INSERT INTO
49.       "Ecritures" (DateEcr,DB,CR,Ref,Libelle,MntEcr)
50.     SELECT DateEcr,DB,CR,Ref,Libelle,MntEcr
51.     FROM "\Memory\EcrituresNew";
52.  
53.     CREATE NOCASE INDEX IF NOT EXISTS "ixDB" ON "Ecritures" ("DB");
54.     CREATE NOCASE INDEX IF NOT EXISTS "ixCR" ON "Ecritures" ("CR");
55.  
56.     update Ecritures E
57.       set E.DBUId=P1.UId,
58.           E.CRUId=P2.UId
59.     from
60.       Ecritures
61.     left join Plan P1 on (E.DB=P1.CompteUnique)
62.     left join Plan P2 on (E.CR=P2.CompteUnique);
63.  
64.     update Ecritures E
65.        set CatCodes=P1.CatCode+P2.CatCode
66.     from
67.       Ecritures
68.     left join Plan P1 on (E.DBUId=P1.UId)
69.     left join Plan P2 on (E.CRUId=P2.UId);
70.  
71.     update Ecritures E
72.        set CatCodes=P1.CatCode+P2.CatCode
73.     from
74.       Ecritures
75.     left join Plan P1 on (E.DBUId=P1.UId)
76.     left join Plan P2 on (E.CRUId=P2.UId);
77.  
78.     update Ecritures
79.        set Ref='0'+Ref
80.     where length(Ref)<6 and SUBSTRING(Ref,1,1) between '1' and '9';
81.  
82.     CREATE INDEX IF NOT EXISTS "ixRef" ON "Ecritures" ("Ref");
83.     /* Clean up */
84.     DROP TABLE IF EXISTS "\Memory\EcrituresNew";`;  
85.  

[440bx]

Code: Pascal  [Select][+][-]

1. operator/ (const a, b: string): string;
2. begin
3.   result := a + LineEnding + b;
4. end;
5.  
6. procedure TForm1.Button1Click(Sender: TObject);
7. var
8.   s: string;
9. begin
10.   s :=
11.     'Hello' /
12.     'World';
13.   Memo1.Append(s);
14. end;  

<snipped>

So you could slightly reconsider the syntax using kupferstecher / VTwin formulation.

If I understood what you're proposing correctly, that is, use "/" (without quotes of course) to indicate a line continuation, that seems to have the same problem as using "+" for the same purpose.  The possibility of overloading the operator would force the entire construction to be parsed instead of just scanned.

Refer to post #34 by PascalDragon, in this thread, for a more complete explanation of the problems it would create.

if a line-break continuation character is to be used, it cannot be a character that is already in use as an operator (or one which may be used in the future as an operator.)

[kupferstecher]

If I understood what you're proposing correctly, that is, use "/" (without quotes of course) to indicate a line continuation, that seems to have the same problem as using "+" for the same purpose.  The possibility of overloading the operator would force the entire construction to be parsed instead of just scanned.

Refer to post #34 by PascalDragon, in this thread, for a more complete explanation of the problems it would create.

This would mean that the parser would need to handle new lines which it currently is totally ignorant of.

I understand that sentence that the problem is the Parser can't distinguish, if the tokenized "+" is at the end of a line which would result in a inserted linebreak or it is between two strings in the same line, which wouldn't result in a inserted linebreak. Making fpc able to distinguish both cases of course would be intrusive.
But with a new operator "/" this is not necessary. The token "/" is just passed to the parser, the lexer perhaps may be even untouched (But I don't know). It's actually desirable that the parser deals with it, because then it also can treat string variables, not only source strings.

The result is a clean syntax without mode switches, different types of strings, trimming mechanisms etc.

Code: Pascal  [Select][+][-]

1. var
2.   s,s2: string;
3. begin
4.   s2:= 'sweet'
5.  
6.   s := 'Hello,' / s2 + 'world'
7.   Memo1.Append(s);
8. end;

[440bx]

But with a new operator "/" this is not necessary.

The problem is "/" is not a new operator, it is currently used to indicate division and it is legal, though it would look a bit strange, to define division between two strings. There is nothing stopping/preventing a programmer from doing that which would cause an ambiguity, i.e, is it a division or a line continuation ?.

[Akira1364]

But : we have only JEDI code formatter to salvage strangely indented code and this I need more than this new feature.

I'm afraid I'm not quite sure what you're getting at with this.

Now just a slightly more complicated SQL script straight out of a text file with the back tick idea, would it pass ?

You'd need to escape the backtick literal (by adding a second one) on the line that looks like this, if you wanted to have it in the output:

Code: Text  [Select][+][-]

1. ',-./:;<=>\`'

But apart from that, yes.

I understand that sentence that the problem is the Parser can't distinguish, if the tokenized "+" is at the end of a line which would result in a inserted linebreak or it is between two strings in the same line, which wouldn't result in a inserted linebreak. Making fpc able to distinguish both cases of course would be intrusive.
But with a new operator "/" this is not necessary. The token "/" is just passed to the parser, the lexer perhaps may be even untouched (But I don't know). It's actually desirable that the parser deals with it, because then it also can treat string variables, not only source strings.

It doesn't really work like you think it does.

FPC uses exactly one completely context-free procedure (tscannerfile.readtoken) to do basically all reading of strings (whether they be initialized variables, or "true constants", or resourcestrings, or a standalone literal in a function call, and so on.)

This is why multiline strings "just work" everywhere single-line strings do: it boils down to doing exactly the same thing, except between two backticks instead of between two single-quotes, and without intentionally stopping when it hits a newline character.

The result is a clean syntax without mode switches, different types of strings, trimming mechanisms etc.

A) The mode switch was laid out as an explicit requirement for this feature to ever be considered at all, by the core team.

B) This feature doesn't introduce a new type of string. It just enables the compiler to read strings the way it always has, but across multiple lines.

C) IMO the "/" operator thing is just not really any better than the "+" we've always had, and provides none of the advantages that real unseparated multi-line string constants do.

D) At this point I simply have no intention of changing the core way the feature works, regardless.

[kupferstecher]

The problem is "/" is not a new operator, it is currently used to indicate division and it is legal, though it would look a bit strange, to define division between two strings. There is nothing stopping/preventing a programmer from doing that which would cause an ambiguity, i.e, is it a division or a line continuation ?.

The same holds for "+".
Because the "+" operator is already used, thus it's not far-fetched to use "/" as well.

What has "+" to do with strings? Well, it "adds" them together.
What has "/" to do with strings? It "devides" them to two lines.
Quite intuitive, I'd say.

Anyways, that was only a suggestion.

It doesn't really work like you think it does.

FPC uses exactly one completely context-free procedure (tscannerfile.readtoken) to do basically all reading of strings (whether they be initialized variables, or "true constants", or resourcestrings, or a standalone literal in a function call, and so on.)

This is why multiline strings "just work" everywhere single-line strings do: it boils down to doing exactly the same thing, except between two backticks instead of between two single-quotes, and without intentionally stopping when it hits a newline character.

I didn't assume any different. My comment was about the claim, that a "/" would be intrusive.
I can see the charm of your solution that everything is covered in the Lexer, but it also means there is still no better way to insert linebreaks concatenated with variable strings.

A) The mode switch was laid out as an explicit requirement for this feature to ever be considered at all, by the core team.

Well a mode switch for enabling a feature is one thing. I was talking about configuration mode switches: trim 5, trim auto...

B) This feature doesn't introduce a new type of string. It just enables the compiler to read strings the way it always has, but across multiple lines.

New delimiters, new syntax. The syntax issue I guess is what BrunoK meant with "JEDI code formatter". Tools and syntax highlighters may stumble over the new syntax. (But thats evolution...)
A freepascal programmer then has to learn both the standard strings and multiline strings.

C) IMO the "/" operator thing is just not really any better than the "+" we've always had, and provides none of the advantages that real unseparated multi-line string constants do.

As said before, this doesn't contradict to your proposal, they could coexist.

D) At this point I simply have no intention of changing the core way the feature works, regardless.

Thats clear. And I can understand that its not very motivating to get contra. But such implementations have an effect on all freepascal users, so it should be discussed.

[Akira1364]

Well a mode switch for enabling a feature is one thing. I was talking about configuration mode switches: trim 5, trim auto...

I mean, it's just one directive. IMO the "Auto" option I added should pretty much cover the common use case for anyone who doesn't want to worry about indentation at all, also.

New delimiters, new syntax. The syntax issue I guess is what BrunoK meant with "JEDI code formatter". Tools and syntax highlighters may stumble over the new syntax. (But thats evolution...)
A freepascal programmer then has to learn both the standard strings and multiline strings.

Adapting formatters and syntax highlighters to the backticks is certainly far more straightforward than it would be with "/". In the case of a formatter, all it has to do is treat the column position of the opening backtick as the "left side" (much like {$MultiLineStringLeft Auto} does) and reposition the entire string block based on that (while just not altering anything that comes between the two backticks at all.)

I also don't think there's anything to "learn", really. Anywhere you can use a single-line string denoted with single quotes, you can use a multi-line string denoted with backticks. They are exactly the same thing apart from the fact that one of them can span multiple lines while the other cannot.

As said before, this doesn't contradict to your proposal, they could coexist.

Well, of course. I guess my real point is that I just don't really understand why or where the "/" thing would ever be preferable.

Thats clear. And I can understand that its not very motivating to get contra. But such implementations have an effect on all freepascal users, so it should be discussed.

I don't mind the "contra" at all, really. We did discuss it at length on the mailing list though, and I specifically made this thread afterwards to give the whole thing additional visibility.