How can a TSynMemo handle an UTF-8 string ?

[Basile B.]

I'm developing an IDE based on the SynEdit suite. Until a few days ago I was very happy. But I've realized that it doesn't handle well UTF-8 chars.

for example take the following string:

> "ຂອ້ຍກິນແກ້ວໄດ້ໂດຍທີ່ມັນບໍໄດ້ເຮັດໃຫ້ຂອ້ຍເຈັບ"

it's displayed, in an editor (default charset, monospaced font):

if I put the cursor before the first double quote:
> "ຂອ້ຍກິນແກ້ວໄດ້ໂດຍທີ່ມັນບໍໄດ້ເຮັດໃຫ້ຂອ້ຍເຈັບ                            ",

if I put the cursor at the end of line, after the comma:
> "ຂອ້ຍກິນແກ້ວໄດ້ໂດຍທີ່ມັນບໍໄດ້ເຮັດໃຫ້ຂອ້ຍເຈັບ"                            ,

If I choose the OEM charset, then the display is better, except that I only get squared characters.

> "□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□□",

Actually I noticed that even in Lazarus there's the same display issue, for example, play with:

> 'ຂອ້ຍກິນແກ້ວໄດ້ໂດຍທີ່ມັນບໍໄດ້ເຮັດໃຫ້ຂອ້ຍເຈັບ',

if you put the cursor before and after the first char you'll see that the final quote moves.

Actually that's not a big big problem, but I'd like to see if it can be fixed.

[typo]

Yes, I notice also that the selection behavior is very strange when I try to select a part of the string.

[Martin_fr]

This is an issue with the font. Not an utf8 issue

SynEdit can only deal with mono spaced fonts.

However, I have noticed, that some systems use proportional fonts for certain languages, even if the font should be monospaced.

Since SynEdit believes that the system would have drawn the text monospaced, SynEdit calculates position of the caret, the selection highlight, or the next char after a highlight change or RTL/LTR break based on the monospaced width of a char.

Setting "Extra Char Spacing" <> 0 will force  strictly monospaced output (but not look good. (And script languages, will look broken on gtk/carbon)



In result you need to find a font that works for those letters. And if you distribute your app, so does every user.

To fix this in SynEdit, it needs support for proportional fonts. But that is still far amay.

[Martin_fr]

EDIT: this will not work. See below

You can edit

components\synedit\lazsyntextarea.pp

line 1535 (trunk)

Code: [Select]

      // Prepare FETOBuf
      if FTextDrawer.NeedsEto or ATokenInfo.HasDoubleWidth
         {$IFDEF Windows} or ATokenInfo.RtlInfo.IsRtl {$ENDIF}  // RTL may have script with ligature
      then begin

change to "If true"

Or any condition based on the text in the token.

On GTK and Carbon, this will break script based languages. Instead of drawing the letters as a continuous word, you will see each letter individually (as if separated by (invisible) spaces)

[Martin_fr]

Actually, I might be wvrong.

It may not be the font.

some chars ar made from several code points.

ອ້

I made from 2 codepoints. SyntEdit counts them wrong.

This clearly is a bug.

You can see this, when selecting text from left to right, one by one. The char becomes separated.


Yes SynEdit does not handle those combined codepoint correctly yet.

What language is this?

It needs to be checked, maybe they need to be added to
components\synedit\synedittextbuffer.pp  line 891

Code: [Select]

function TSynEditStringList.LogicPosIsCombining(const AChar: PChar): Boolean;


[Bart]

This is an issue with the font. Not an utf8 issue

SynEdit can only deal with mono spaced fonts.

However, I have noticed, that some systems use proportional fonts for certain languages, even if the font should be monospaced.

Hmm, I've seen this happening with Courier New also.
For some of my test for handling files with UTF8 characters, I have resorted to using #-notation for the string constants, otherwise I experience the same problem as above.

Bart

[Martin_fr]

Some of the chars are not supported by SynEdit. SynEdit does not yet use the OS to handle unicode correctly, and it only implements a small subset.

The text contains
 'ZERO WIDTH NO-BREAK SPACE'
which A am not sure SynEdit will deal with.

'LAO TONE MAI THO'  which is a Non-Spacing Mark, but SynEdit will allocate space for it.

I dont know the full unicode spec on this, so I do not know, when it will be rendered which way.

You can try to add it to combining marks. It is not the correct fix, but if you are lucky it will work.

[Basile B.]

Ok, so this not an error from my side. Good to know.
I let someone else filling a bug report if needed.