How do I parse UTF8 strings?

[typo]

For example, one can parse an ANSI string like that:

Code: [Select]

var
 i :integer;
 s :string;
begin
 for i := 1 to length(s) do
 begin
    if (s[i] = 'a') and (s[i+1] = 'b') then go;
 end;
end;
But UTF8 strings can not be analyzed this way. If you use pointers, how to have access to the previous and next character?

[dfeher]

I think you can parse an UTF8 string the same way, except that you must typecast 'a' and 'b' to WideChar or decoding the UTF8 string with UTF8Decode function and then compare.

[Zoran]

You can write this:

Code: [Select]

uses
  ..., LCLProc;

...

for I := 1 to UTF8Length(S) do begin
  if (UTF8Copy(S, I, 1) = 'a') and (UTF8Copy(S, I, 1) = 'b') then go;
end;


For manipulating strings, you should use procedures from LCLProc.
First notice that Length(S) is replaced with UTF8Length(S).
Now, for this purpose, the char S[ I] cannot be used, because it's just one byte, but UTF8Copy(S, I, 1) returns UTF8String whose UTF8Lenth is 1, althought it might contain more than one byte (which means that its Length might be more than 1).

See also other analog procedures and functions in LCLProc unit, like UTF8Insert, UTF8Delete...

[theo]

You could use the utf8scanner:
http://wiki.lazarus.freepascal.org/Theodp

[typo]

Code: [Select]

if (UTF8Copy(S, I, 1) = 'a') and (UTF8Copy(S, I, 1) = 'b') then go;

But if I don't know which is the length of 'a', I could not know where to search 'b'.

[theo]

Code: [Select]

if (UTF8Copy(S, I, 1) = 'a') and (UTF8Copy(S, I, 1) = 'b') then go;

But if I don't know which is the length of 'a', I could not know where to search 'b'.

It should work. "I" is the char index (code point) not the byte index.
But utf8scanner is easier to work with and faster.

[typo]

I installed it, but Lazarus does not show me the corresponding set of components.

[Zoran]

Code: [Select]

if (UTF8Copy(S, I, 1) = 'a') and (UTF8Copy(S, I, 1) = 'b') then go;

But if I don't know which is the length of 'a', I could not know where to search 'b'.

Oh, I see that I put "I" twice, instead of "I + 1" in second term, sorry.
Of course, it should be UTF8Copy(S, I + 1, 1) = b.

It is a solution which will work for you.
I beleive that using the package that Theo recomends is faster, I haven't tried it, but I will.

[theo]

I installed it, but Lazarus does not show me the corresponding set of components.

It's not a visual component. Add the dependency "utf8tools" to the project (in project inspector) and add "uses utf8scanner" to your unit.

[typo]

"I" is the char index (code point) not the byte index.

If so, "I" should be incremented by UTF8Length('a'), which does not occur. I am not understanding. To me, it should be:

Code: [Select]

if (UTF8Copy(s, i, 1) = 'a')
              and (UTF8Copy(s, i + Length('a'), 1) = 'b') then go;   
See the example in Wiki, using PChar:

Code: [Select]

uses LCLProc;
...
procedure IterateUTF8Characters(const AnUTF8String: string);
var
  p: PChar;
  unicode: Cardinal;
  CharLen: integer;
begin
  p:=PChar(AnUTF8String);
  repeat
    unicode:=UTF8CharacterToUnicode(p,CharLen);
    writeln('Unicode=',unicode);
    inc(p,CharLen);
  until (CharLen=0) or (unicode=0);
end;


I think also loop for is inappropriate in this case.

[typo]

Well, I tested this code and it works:

         

Code: [Select]

repeat
  unicode := UTF8CharacterToUnicode(p, charlen);

  if  (utf8copy(s, i, 1) = 'c')
  and (utf8copy(s, i+length('c'), 1) = 'd')              // caractere à direita
  and (utf8copy(s, i+length('c')+length('d'), 1) = 'e')  // segundo caractere à direita
  and (utf8copy(s, i-length('b'), 1) = 'b')              // caractere à esquerda
  and (utf8copy(s, i-length('b')-length('a'), 1) = 'a')  // segundo caractere à esquerda
  then
    showmessage('abcde');

  inc(p, charlen);
  inc(i, charlen);
until (charlen = 0) or (unicode = 0);