Lazarus in non-UTF8 mode (-dDisableUTF8RTL): does anyone still use this?

[Bart]

Hi,

Long time ago, before codepage-aware ANSI strings existed in fpc, all strings in Lazarus were treated as being UTF8 encoded.
With the introduction of codepage aware strings (fpc 3.0 and later) the LazUTF8 unit sets DefaultSystemCodePage to CP_UTF8. This ensures "String" in an LCL application will have CP_UTF8 as it's codepage.
At the same time we introduced a possibility to build Lazarus (and the LCL etc.) with a define (DisableUTF8RTL) that disabled this "initialization" to CP_UTF8.
When using this define, "String" in an application will have CP_ACP as it's codepage.
IIRC this was done for Lazarus users who used databases that were encode in a Windows codepage.

The Lazarus team is thinking about completely removing this workaround.
Of course it may very well be the case that users still depend on building LCL applications with the DisableUTF8RTL define.
If that is the case, we would ask you to please report this in this thread.

Bart

[Thaddy]

I used to hate utf8 on Windows since on that platform it was not native to the OS which is since NT/XP days either ucs2 or utf16, but since windows 10 that has become transparent so I have no objections, since nowadays I explicitly specify my string type at declaration, so shortstring, ansistring,utf8string, unicodestring etc.
The ambiguity of just "string" does not affect me. (and that behavior stems indeed from my former dislike of utf8 on Windows)

Of course, when dropped, I expect Lazarus to honor the specified string types when used. The latter makes me slightly worried.... but as long as this keeps working on unit level:

Code: Pascal  [Select][+][-]

1. type
2.     string = type shortstring;// or any specified string type

then it's fine with me.

[teco]

Removing this "workaround" would make Lazarus/Free Pascal mostly unusable for us.

- The Database engine we are using works with UTF16 or Codepage. Only with this it can be used without issues.
- We import Records from different systems. Import would cause issues without this because of codetables, etc.

[abouchez]

In most projects, "string" is used as an alias to "TFileName", and when the RTL has to call the OS.

At RTL level, there are not the same: you have SetMultiByteConversionCodePage() and SetMultiByteRTLFileSystemCodePage().

In all my projects, and even in mormot.core.os.pas initialization, I force:

Code: Pascal  [Select][+][-]

1.   {$ifdef ISFPC27}
2.   // we force UTF-8 everywhere on FPC for consistency with Lazarus
3.   SetMultiByteConversionCodePage(CP_UTF8);
4.   SetMultiByteRTLFileSystemCodePage(CP_UTF8);
5.   {$endif ISFPC27}

This is done even  for console/server applications, for simplicity and consistency.

IMHO CP_UTF8 is much better than UTF-16, even on Windows.
https://utf8everywhere.org/
The time to make a UTF8/UTF16 translation with the API is negligeable.
And I would not trust Microsoft to have properly implemented the CP_UTF8 at system level. Just use the existing code with UTF-16 parameters.

I would not change any default behavior. Removing -dDisableUTF8RTL would make no difference to me.
Keep it as such, with the SetMultiByte*() functions, unless you want string = UnicodeString like in Delphi and in that case, just use the new mode available in trunk.
But I suspect no-one is using this -dDisableUTF8RTL anywhere but in some very old and unmaintained project, which could stay with a FPC 3.2 anyway.

[abouchez]

- The Database engine we are using works with UTF16 or Codepage. Only with this it can be used without issues.
- We import Records from different systems. Import would cause issues without this because of codetables, etc.

I don't see how those two points would be affected.

Even removing this -dDisableUTF8RTL support won't change anything in code using code pages.
If I understand correctly, it is not about removing code pages, it is about removing a potential RTL without CP_UTF8 for strings, when used with Lazarus.
I already thought it was the case: I expected Lazarus to require CP_UTF8 (or CP_UTF16 in the trunk in Unicode mode).

[Thaddy]

Hence I mentioned that on unit level the string type should be definable and on project level the string type should work with explicit string types.
As long as that remains the case there really are no(t many) problems.

[teco]

ElevateDB is original developed for Delphi.

With Free Pascal it has issues with german umlauts (äöüß)

Solution from the manufacturer is to use -dDisableUTF8RTL or switching to Delphi.

[abouchez]

Solution from the manufacturer is to use -dDisableUTF8RTL or switching to Delphi.

Maybe switch to the new FPC Unicode mode may be a good option.
That is, remove "-dDisableUTF8RTL" when FPC Unicode mode is finalized.

[paweld]

Removing this "workaround" would make Lazarus/Free Pascal mostly unusable for us.

- The Database engine we are using works with UTF16 or Codepage. Only with this it can be used without issues.
- We import Records from different systems. Import would cause issues without this because of codetables, etc.

In LCL you have the LConvEncoding unit, which allows conversion between multiple code pages.
I use Lazarus for applications using the MSSQL database server, and most of the databases have the collation set to Polish_CI_AS and it works very well with Lazarus using the CP1250ToUTF8 and UTF8ToCP1250 functions. To handle UTF-16 there is UTF8ToUTF16 from the LazUTF8 unit.
I also use these functions when importing data from various files.

[Thaddy]

Simply try {$mode delphiunicode} on the units where code is affected.
As I understand utf8 code should be assignment compatible to utf16.
But the problem you ran into is exactly why I initially hated Lazarus default utf8.

[JuhaManninen]

I would not change any default behavior. Removing -dDisableUTF8RTL would make no difference to me.
Keep it as such, with the SetMultiByte*() functions, unless you want string = UnicodeString like in Delphi and in that case, just use the new mode available in trunk.

Nobody is planning to change the default behavior now. The question is about removing the DisableUTF8RTL define and old codepage support.

Default UTF-16 in LCL using UnicodeString will require big changes. I don't even know how it can be done.
At that point however the old codepage support will complicate things further. I hope it gets removed before that.

ElevateDB is original developed for Delphi.
With Free Pascal it has issues with german umlauts (äöüß)
Solution from the manufacturer is to use -dDisableUTF8RTL or switching to Delphi.

OK, that is a valid reason to use dDisableUTF8RTL, meaning it cannot be removed now.
You should however push ElevateDB developers to improve their Unicode support. Having issues with german umlauts (äöüß) probably means they treat strings as having fixed-width chars. That means they treat also UTF-16 as fixed-width which it is not. With UTF-8 you must do it correctly because codepoint lengths vary so often.

Another thought: Maybe you could easily solve the problem in your own code using explicit conversion functions as paweld explained.
Convert the database values to Unicode as soon as possible after reading and convert them to the database encoding as late as possible before writing. That is what everybody else does (AFAIK).
If the database already has UTF-16, then just convert it (automatically) to UTF-8. And back to UTF-16.
If it does not work, please explain me why.

[Thaddy]

Yes it looks that they still treat utf16 as ucs2, which was correct a long time ago, but not for the past 20 years. They seem to reserve two bytes per char instead of four, which should be the case for utf8, utf16 and utf32 at storage point and there is no escape from that in a database setting.

[JuhaManninen]

IMO all textual data in databases and all code manipulating it should be converted to Unicode, regardless of what encoding.
A locale dependent codepage is a truly inferior system. Somebody else's data will show as "????????".
I thought we got rid of that in 1990's. But no!

[teco]

Another thought: Maybe you could easily solve the problem in your own code using explicit conversion functions as paweld explained.
Convert the database values to Unicode as soon as possible after reading and convert them to the database encoding as late as possible before writing. That is what everybody else does (AFAIK).
If the database already has UTF-16, then just convert it (automatically) to UTF-8. And back to UTF-16.
If it does not work, please explain me why.

The catch is that there are not only our software accessing the ElevateDB. We write data into Elevate and another software do further processing. This software is Delphi based.  We must make it work that the Delphi Software see the right information, so we can not see it independently with a solution. The manufacturer says that Delphi is his main sales for the Desktop and Server Edition and Free Pascal is only a small part. To small for additional work to increase compatibility.

[Thaddy]

To small for additional work to increase compatibility.

Yes, that makes sense: that is your job. (btw it is too, double o)
FPC supports many conversions.