String functions. conversion from Delphi

[mm7]

I am converting project from Delphi7 into Lazarus 1.4R2.
Delphy converter has converted file functions to UTF8.
But in code lots of calls of string functions Copy, Pos, Trunc etc ANSI ones.
Not converted for some reason (why?).
I understand that because LCL is mainly UTF8, the according functions, like UTF8Copy, should be used instead of ANSI ones.

What would be a best approach:
A. replace literally all functions calls with their UTF8 brothers;
B. create unit MyUTF8StringUtils where UTF8 calls are wrapped into Copy, Pos, etc ANSI-like functions, and use it instead of standard one;
C. Other?

I read docs about conversion and UTF8 support, and searched here. Have not found a clear solution for this issue. Sorry if I have not found something and the issue is well known. Please advise.

[skalogryz]

B. seems to be the best and the most flexible.

[JuhaManninen]

What would be a best approach:
A. replace literally all functions calls with their UTF8 brothers;
B. create unit MyUTF8StringUtils where UTF8 calls are wrapped into Copy, Pos, etc ANSI-like functions, and use it instead of standard one;
C. Other?

The right answer is C.
It is comforting to see that an experienced programmer, skalogryz, is as clueless with Unicode as I was just a short time ago.

Now, the best solution in your case is this :
  http://wiki.lazarus.freepascal.org/Better_Unicode_Support_in_Lazarus
The downside is that you need development versions of FPC and Lazarus. FPC 3.0 is released at some point which improves the situation.
I don't see it as a big problem because you just started converting your code. By the time it is ready, both FPC and Lazarus have evolved more.

If you want to continue with the release versions, the instructions are here :
  http://wiki.lazarus.freepascal.org/LCL_Unicode_Support

I read docs about conversion and UTF8 support, and searched here. Have not found a clear solution for this issue. Sorry if I have not found something and the issue is well known. Please advise.

Here is an answer to your original question about why Copy and Pos are not converted :
Becase they already work well with UTF-8 in most cases. See :
  http://wiki.lazarus.freepascal.org/UTF8_strings_and_characters
This holds true in any case, whether you use the new "better" UTF-8 support or the current official one.

(skalogryz, hint: you should also read the wiki links, you are a FPC/Lazarus developer for God's sake

[mm7]

Now, the best solution in your case is this :
  http://wiki.lazarus.freepascal.org/Better_Unicode_Support_in_Lazarus
The downside is that you need development versions of FPC and Lazarus. FPC 3.0 is released at some point which improves the situation.
I don't see it as a big problem because you just started converting your code. By the time it is ready, both FPC and Lazarus have evolved more.

Juha, many thanks for your detailed reply.
Yes, I agree Better_Unicode_Support_in_Lazarus would be the best. But FPC3 is not released an I do not know when it will be. Currently 2.7 is in trunk.
I already converted the project. I relied on "Delphy Converter" and the project in general works. Except bugs here and there. Current bug is caused by Copy. I'll illustrate later.

If you want to continue with the release versions, the instructions are here :
  http://wiki.lazarus.freepascal.org/LCL_Unicode_Support

Yes, of course, I read it.

Here is an answer to your original question about why Copy and Pos are not converted :
Becase they already work well with UTF-8 in most cases. See :
  http://wiki.lazarus.freepascal.org/UTF8_strings_and_characters
This holds true in any case, whether you use the new "better" UTF-8 support or the current official one.

Keyword is "in most cases". But not in every one. Unfortunately.
Not sure about Pos, but Str:=Copy('Трапециедальная',1,23); causes funny results if you add that Str to TMemo.Lines or to another GTK2 component, even if it is writeln to Gnome xterm. It erases everything printed above!
The Copy cuts 23 bytes(!) leaving part of UTF8 char at the end and this makes GTK go crazy.

PS/ Трапециедальная in Russian means Trapezoidal.

I think using my unit (B) is also good because it can be turned off by setting some {$IfNdef FPC3} (or something that tells that FPC3 compiler/RTL uses UTF8) around it, for future compatibility with FPC3.

[Cyrax]

...
Currently 2.7 is in trunk.
...

Err, FPC trunk have 3.1.1 version numer now.

[mm7]

...
Currently 2.7 is in trunk.
...

Err, FPC trunk have 3.1.1 version numer now.

Well, it was from http://www.freepascal.org/develop.var

...You can download today's development (trunk - currently v2.7.x) sources...

If you know about FPC so well, could you tell when FPC3 will be released?

[JuhaManninen]

Keyword is "in most cases". But not in every one. Unfortunately.
Not sure about Pos, but Str:=Copy('Трапециедальная',1,23); causes funny results ...

Yes, then you must use UTF8Copy but fortunately you don't need to copy a fixed number of characters very often.
If you deal with valid UTF-8 strings, you can safely use Pos, Copy and Length. They work with byte positions but due to inherent properties of UTF-8 they work correctly ALWAYS.
Say you want to cut your text to a user specified UTF-8 character. The user has typed the char in Edit1. Here you go :

Code: [Select]

  Txt := 'Трапециедальная';
  i := Pos(Edit1.Text, Txt);
  if i > 0 then
    Txt := Copy(Txt, 1, i-1);

See other examples from the wiki page I linked and from internet.

I think using my unit (B) is also good because it can be turned off by setting some {$IfNdef FPC3} (or something that tells that FPC3 compiler/RTL uses UTF8) around it, for future compatibility with FPC3.

No, you clearly have not understood the issue.
The "future compatibility with FPC3" does not change anything when dealing with individual UTF-8 characters. You need to use either Pos/Copy/Length or UTF8Pos/UTF8Copy/UTF8Length depending on your particular code. This one detail stays the same when we move to the "better" UTF-8 support.

Pos/Copy/Length work with byte positions.
UTF8Pos/UTF8Copy/UTF8Length work with UTF-8 character positions.
Most often you can continue using the byte positions but sometimes you need UTF-8 character positions.
You should not blindly convert everything to use UTF-8 character positions. It slows down your code in any case and leads to wrong results in some cases.
You must go through your code case by case.

Unicode is unbelievable complex. Now we only touched the surface. It happened to me at least 10 time that I though I understood it, but then something new came up again.

[Cyrax]

...
Currently 2.7 is in trunk.
...

Err, FPC trunk have 3.1.1 version numer now.

Well, it was from http://www.freepascal.org/develop.var

...You can download today's development (trunk - currently v2.7.x) sources...

That page needs some updating.

If you know about FPC so well, could you tell when FPC3 will be released?

It's in stabilization phase : http://svn.freepascal.org/cgi-bin/viewvc.cgi/branches/fixes_3_0/?sortby=date&view=log
So probably in near future.

[mm7]

...
You should not blindly convert everything to use UTF-8 character positions. It slows down your code in any case and leads to wrong results in some cases.
You must go through your code case by case.
...

I agree, this is wise.
Especially those parsed strings read from text files, that are usually just ASCII.

Is it harmful if UTF8 functions applied to plain ASCII 7 data like CSV (digits with simple separators...)?

[JuhaManninen]

Is it harmful if UTF8 functions applied to plain ASCII 7 data like CSV (digits with simple separators...)?

No harm. UTF-8 is backwards compatible with ascii.

[mm7]

Is it harmful if UTF8 functions applied to plain ASCII 7 data like CSV (digits with simple separators...)?

No harm. UTF-8 is backwards compatible with ascii.

Thank you Yuha!

Does it mean I can safely replace all ASCII functions with UTF8 ones?

[JuhaManninen]

Does it mean I can safely replace all ASCII functions with UTF8 ones?

Safely, I am not sure. You should always understand what happens in your code.
Anyway, you can try that. It will work most of times although it is not a very clever solution.

[mm7]

Problem with overload of function length.

there are two of them one for string another for dynamic arrays.

If I overload one for string I get compilation error
my.pas(173,28) Error: Incompatible type for arg no. 1: Got "Array[0..8] Of TUserstring", expected "AnsiString"

Code: [Select]

unit MyStringUtils;

{$mode delphi}

interface

uses
  Classes, SysUtils, LazUTF8;

function Length(const s: string): PtrInt; overload;

implementation

function Length(const s: string): PtrInt;
begin
     result := UTF8Length(s);
end;

end.

Why FPC does not understand different signatures in this case?

[JuhaManninen]

Problem with overload of function length.

Length is a built-in function, not a library function.
Just do as I told, use UTF-8 specific functions only when they are needed. You will understand Unicode and UTF-8 better during the process.

[Leledumbo]

Length is a built-in function, not a library function.

Which is user-overridable, it's a matter of usage, looking at the error message.