Using UTF8

[J-G]

I'm told that modern programs ought to be using UTF8  - rather than cp1252 - which I know is the 'default' in Lazarus.  The fact that as a hobbyist who now only writes programs for personal or relative/friends use and only using English I had thought would make me immune from such edicts but today I've been investigating how to add a 'Thin Space' to assist with layout issues.

As a physical, hands-on, printer it would be a simple matter of selecting an 'n space' lead from the fount rather than an 'm space' lead. Sadly my days of composing in lead type are well behind me 

I've 'Googled' many times and am always pointed to the fact that ASCII does not have a 'Thin Space' but UTF8 does in the form of U+2009.  Now I'm very well aware of how to assign an ASCII character to a Char() -  and I'm aware that characters beyond 255 need more than one byte so cannot be assigned in this way, but I have yet to find any text that makes it clear how a UTF8 charater can be assigned to a var (or const even).

In my blind fumbling I did attempt to create a Var  ThinSpace : String;  by concatinating Char($E2)+Char($80)+Char($89) which one of my searches tells me is how Character U+2009 can be made.  - - - -   Whilst this was acceptable to the compiler, the resultant text doesn't show a thin space - it shows an empty square taking up the width of a normal space.

Can someone point me to my error and explain how I can use UTF8 characters? - and how to assign them to vars knowing the U+number.

[ALLIGATOR]

Does that make it clearer? If not, please explain in more detail.

Code: Pascal  [Select][+][-]

1. program app;
2. {$codepage utf8}
3. var
4.   s: String = 'балалайка';
5. begin
6.   WriteLn(s);
7. end.

Code: Bash  [Select][+][-]

1. > chcp 65001
2. > app.exe
3. балалайка

[dsiders]

I'm told that modern programs ought to be using UTF8  - rather than cp1252 - which I know is the 'default' in Lazarus.  The fact that as a hobbyist who now only writes programs for personal or relative/friends use and only using English I had thought would make me immune from such edicts but today I've been investigating how to add a 'Thin Space' to assist with layout issues.

As a physical, hands-on, printer it would be a simple matter of selecting an 'n space' lead from the fount rather than an 'm space' lead. Sadly my days of composing in lead type are well behind me 

I've 'Googled' many times and am always pointed to the fact that ASCII does not have a 'Thin Space' but UTF8 does in the form of U+2009.  Now I'm very well aware of how to assign an ASCII character to a Char() -  and I'm aware that characters beyond 255 need more than one byte so cannot be assigned in this way, but I have yet to find any text that makes it clear how a UTF8 charater can be assigned to a var (or const even).

In my blind fumbling I did attempt to create a Var  ThinSpace : String;  by concatinating Char($E2)+Char($80)+Char($89) which one of my searches tells me is how Character U+2009 can be made.  - - - -   Whilst this was acceptable to the compiler, the resultant text doesn't show a thin space - it shows an empty square taking up the width of a normal space.

Can someone point me to my error and explain how I can use UTF8 characters? - and how to assign them to vars knowing the U+number.

I would declare the variable using:

Code: Pascal  [Select][+][-]

1. var ThinSpace: String = #$E2+#$80 + #$89;

If you see the missing the missing character glyph, it is telling you that the specific font does not have that glyph. Try using one like Noto Sans... it seems to have the best Unicode glyph coverage.

HTH

[ALLIGATOR]

...it is telling you that the specific font does not have that glyph...

And most likely, the reason is that the string's encoding isn't specified either. By default, it will be 1252 - the system encoding. To change it to UTF-8, you need to add {$codepage utf8} or call SetCodePage

[J-G]

D'Oh!  -   how did I miss the 'missing Glyph'  ???

Thanks both.

Of course the font in use must have the Glyph - as it happens, in the particular use case,  I have no control over the font in use - - - It's the caption in the Window Title bar - - -  but under other conditions I've now sorted it.

Of course I'd also missed the {$CodePage utf8}  but in fact I've just tested with that disabled and the output is good  -  ie. the thinSpace is still used (as long as the font has the Glyph)  which seems to indicate that CodePage UTF8 is already in use . . . .

Assigning the variable using Char(1)+Char(2)  etc.  works, and I can see that #$ prefix would also be sensible, but is there not a way that the U+number can be used?  I only came across the E2 80 89 by chance but the list of U+number codes is generally available.

[J-G]

  s: String = 'балалайка';

The fact that I could recognise the name of the three string musical instrument was very rewarding   🤣🤣

[ALLIGATOR]

This is a relatively simple solution. But, of course, it’s far from optimal. It’s wastefully suboptimal.
On the other hand, there’s not much code )

Code: Pascal  [Select][+][-]

1. program UnicodeEscapeSequenceDecoder;
2. {$mode objfpc}{$H+}
3.  
4. uses
5.   SysUtils, fpjson, jsonparser;
6.  
7. function DecodeUnicodeEscapeSequenceToString(const S: string): string;
8. var
9.   JsonData: TJSONData;
10. begin
11.   Result := '';
12.   JsonData := GetJSON('"' + S + '"');
13.   try
14.     Result := JsonData.AsString;
15.   finally
16.     JsonData.Free;
17.   end;
18. end;
19.  
20. begin
21.   Writeln(DecodeUnicodeEscapeSequenceToString('\u041C\u0435\u0434\u0432\u0435\u0434\u044C\u0020\u0438\u0020\u0412\u043E\u0434\u043A\u0430'));
22.  
23.   ReadLn;
24. end.

Co-authored-by: Google Gemini <gemini@google.com>

[Nimbus]

LazUtils has UnicodeToUTF8
https://lazarus-ccr.sourceforge.io/docs/lazutils/lazutf8/unicodetoutf8.html

This should do it

Code: Pascal  [Select][+][-]

1. uses
2.   LazUTF8;
3.  
4. var
5.   ThinSpace: String;
6.  
7. begin
8.   ThinSpace := UnicodeToUTF8($2009);
9.  
10.   Writeln('Free', ThinSpace, 'Pascal');
11. end.

[J-G]

That example was very useful indeed @Nimbus  -  Not that yours wasn't @Alligator -  it was just that from Nimbus' code I wrote a general converter in a few moments with only six lines of code - including the Proc declaration !  viz.

Code: Pascal  [Select][+][-]

1. procedure Get_UChar(var S:String;N:Cardinal);
2. begin
3.   S := UnicodeToUTF8(N);
4. end;
5.  
6. ===============
7.  
8. Var n_Spc : String;
9. […]
10. Get_UChar(n_Spc,$2009);
11.  

Oh yes, I did have to add LazUTF8 to the Uses Clause 

The first 4 lines are in a Utilities Unit and the last two in the main Program.

Works like a charm  and has given me access to a whole new world of extra characters, how I'll use them is yet to be discovered 

You'll notice that I've used 'n_Spc' as a name, which is shorthand for 'En Space' - the print compositors name for a narrow space - ie. one taking up the space of a lower case 'n'  - a Standard width space is known as an 'm' Space (that normally being the widest character in the fount's lower case).

I haven't used these terms 'in anger' for more than 50 years!

[EDIT]  Just translated the Medveded l Vodka  (I think).

[ADMGNS]

hello

pls look at these links first:

1. https://www.w3.org/International/i18n-tests/

2. https://www.moserware.com/2008/02/does-your-code-pass-turkey-test.html

3. https://blog.codinghorror.com/whats-wrong-with-turkey/

Or, take it easy

[Xenno]

I use Format function all the time. For example:

Code: Pascal  [Select][+][-]

1.   Button1.Caption := Format('%s Undo', [#$E10E]);
2.   Button2.Caption := Format('%s Cut', [#$E16B]);
3.   Button3.Caption := Format('%s Copy', [#$E16F]);
4.   Button4.Caption := Format('%s Paste', [#$E16D]);
5.   Button5.Caption := Format('%s Delete', [#$E106]);

to produce attached screenshot.

[J-G]

Another useful 'tip' @Xenno - thanks.

Not that I write programs that need those specific buttons but I do see and understand the concept which I will be able to adapt.

[Xenno]

Yes. In your case, you may simply code:

Code: Pascal  [Select][+][-]

1. var
2.   ThinSpace: String;
3.   ...
4.   ThinSpace := Format('%s', [#$2009]);

Then ThinSpace is ready to use.

[ADMGNS]

I use Format function all the time. For example:

Code: Pascal  [Select][+][-]

1.   Button1.Caption := Format('%s Undo', [#$E10E]);
2.   Button2.Caption := Format('%s Cut', [#$E16B]);
3.   Button3.Caption := Format('%s Copy', [#$E16F]);
4.   Button4.Caption := Format('%s Paste', [#$E16D]);
5.   Button5.Caption := Format('%s Delete', [#$E106]);

to produce attached screenshot.

hello @xenno

on linux mint, it doesnt work. perhaps, the default font doesnt support utf8 ??!!..

do you have any experience on linux issues that?

tenx

[Xenno]

on linux mint, it doesnt work. perhaps, the default font doesnt support utf8 ??!!..

Apparently it does not present (unicode) chars as in Windows. I am sorry, I only have Windows 10. Lazarus picks Segoe UI as default.

What happens if you change to other font instead default or change those character codes?