BUG? Can someone explain

[derek.john.evans]

AnsiCompareStr returns a different order compared to AnsiCompareText, for strings with underscores.

This happen with FPC 3.x. I've known about it for a while, but, I thought it would have been fixed before releasing 3.x with Lazarus.

I found it because TSynAnySyn uses a binary search for keywords using AnsiCompareStr(), and keywords with underscores are not found correctly.

Code: [Select]

  ShowMessage(IntToStr(AnsiCompareStr('_APPLE', 'APPLE')));
  ShowMessage(IntToStr(AnsiCompareText('_APPLE', 'APPLE')));   


[taazz]

AnsiCompareStr returns a different order compared to AnsiCompareText, for strings with underscores.

This happen with FPC 3.x. I've known about it for a while, but, I thought it would have been fixed before releasing 3.x with Lazarus.

I found it because TSynAnySyn uses a binary search for keywords using AnsiCompareStr(), and keywords with underscores are not found correctly.

Code: [Select]

  ShowMessage(IntToStr(AnsiCompareStr('_APPLE', 'APPLE')));
  ShowMessage(IntToStr(AnsiCompareText('_APPLE', 'APPLE')));   


try it with out lcl or lazarus on a console application and see if that helps.

[engkin]

LazUTF8 changes the widestringmanager functions.

Its UTF8CompareStr has a "bug":

Code: Pascal  [Select][+][-]

1. function UTF8CompareStr(S1: PChar; Count1: SizeInt; S2: PChar; Count2: SizeInt
2.   ): PtrInt;
3. var
4.   Count: SizeInt;
5. begin
6.   Result := 0;
7.   if Count1>Count2 then
8.     Count:=Count2
9.   else
10.     Count:=Count1;
11.   Result := CompareMemRange(Pointer(S1),Pointer(S2), Count); // Note: CompareMemRange can handle nil if Count=0
12.   if Result<>0 then exit;
13.   if Count1>Count2 then
14.     Result:=1
15.   else if Count1<Count2 then
16.     Result:=-1
17.   else
18.     Result:=0;
19. end;
20.  

See?:

Code: Pascal  [Select][+][-]

1.   Result := CompareMemRange(Pointer(S1),Pointer(S2), Count); // Note: CompareMemRange can handle nil if Count=0
2.   if Result<>0 then exit;  <----- This
3.  

[JuhaManninen]

The underscore in '_APPLE' is a normal low ASCII underscore, isn't it?
I believe CompareMemRange should work ok with all ASCII single-byte characters.
I am not sure about multi-byte UTF-8 characters, I don't know the sorting rules well.

[derek.john.evans]

try it with out lcl or lazarus on a console application and see if that helps.

Yep, as a console app, AnsiCompareStr() := AnsiCompareText() when the cases are the same.

What ever is going on, this breaks the methods: IsKeyword(), IsConstant() & IsObject() in the unit SynHighlighterAny.

Which causes invalid syntax highlighting. Personally, I'd say there is a bug in AnsiCompareStr(). But, I dont know the full story behind the unicode update.

[Bart]

AnsiCompareStrdepends on the WideStrigManager that is being used.
Unfortunately this is platform dependant.

Bart

[derek.john.evans]

What! Platform = Windows
So now we are redefining the ASCII set.

No offense, but, thats just bullshit.

Ive outlined one effect of this bug. I can only imagine the other effects.

[Bart]

No offense, but, thats just bullshit.

See the source if you do not believe me.

Code: [Select]

function AnsiCompareStr(const S1, S2: string): integer;{$ifdef SYSUTILSINLINE}inline;{$endif}
  begin
    result:=widestringmanager.CompareStrAnsiStringProc(s1,s2);
  end;

Bart

[derek.john.evans]

I agree. Ive seen this. And I traced it to functions, and some dll stuff.

Really!? I'm actually a little pissed off. As a coder of some 25+ years, Ive base a lot of my trust on simple ideas, like the ASCII code.

I understand I don't know "the world", but, this just is weird.

[JuhaManninen]

See procedure InitLazUtf8 in winlazutf8.inc. It sets the UTF-8 functions for widestringmanager correctly:

Code: Pascal  [Select][+][-]

1. {$IFDEF UTF8_RTL}
2. ...
3. widestringmanager.UpperAnsiStringProc:=@UTF8UpperString;
4. widestringmanager.LowerAnsiStringProc:=@UTF8LowerString;
5. widestringmanager.CompareStrAnsiStringProc:=@UTF8CompareStr;
6. widestringmanager.CompareTextAnsiStringProc:=@UTF8CompareText;
7. widestringmanager.StrCompAnsiStringProc:=@UTF8StrCompAnsiString;
8. widestringmanager.StrICompAnsiStringProc:=@UTF8StrICompAnsiString;
9. widestringmanager.StrLCompAnsiStringProc:=@UTF8StrLCompAnsiString;
10. widestringmanager.StrLICompAnsiStringProc:=@UTF8StrLICompAnsiString;
11. ...
12. {$IFEND}

UTF8CompareStr has worked for years. It uses CompareMemRange which should be OK at least for ASCII characters.
If there really is such a serious error it must be fixed. What can cause it? Somebody else must study the issue, I don't currently use Windows and somehow screwed my Lazarus installation under Wine. Will fix it later...

[Bart]

I agree. Ive seen this. And I traced it to functions, and some dll stuff.

Really!? I'm actually a little pissed off. As a coder of some 25+ years, Ive base a lot of my trust on simple ideas, like the ASCII code.

I understand I don't know "the world", but, this just is weird.

Notice that CompareStr gives the same results as Utf8CompareStr:

Code: [Select]

Utf8CompareStr(APPLE,_APPLE)  = -30
 Utf8CompareStr(apple,_apple)  = 2
 CompareStr(APPLE,_APPLE)      = -30
 CompareStr(apple,_apple)      = 2

This is because  Ord('A') < Ord('_') < Ord('a');
(CompareMemRange returns this)
CompareText compares uppercase values, while Utf8CompareText compares lowercase values.

Default AnsicompareText/AnsiCompareStr (as in a pure console app under windows, no LazUtf8 involvement) ask Windows for the result:

Code: [Select]

function Win32AnsiCompareStr(const S1, S2: string): PtrInt;
  begin
    result:=CompareStringA(LOCALE_USER_DEFAULT,0,pchar(s1),length(s1),
      pchar(s2),length(s2))-2;
  end;

If your binary search relies on the fact that comparing 'apple' vs '_apple' must give a positive value always (case independant) then using AnsiCompareText (which maps to Utf8CompareText in Lazarus) should do just that.

If your compariosn must always be a positive value, but the the keywords itself have different meaning depending on case (as in C), then you basically are screwed, since AnsiCompareStr's result will always depend on the widestringmanager and therefore the results can differ across platforms and locales.

Bart

[Bart]

Maybe you can use WideCompareStr() instead?
At least on Windows, it gives this results:

Code: [Select]

WideCompareStr(APPLE,_APPLE)  = 1
WideCompareStr(apple,_apple)  = 1
WideCompareText(APPLE,_APPLE) = 1
WideCompareText(apple,_apple) = 1

Bart

[derek.john.evans]

It is not an issue about what can be used. It is an issue of what was used.

A function result changed with FPC3.X. Which means, every app which thought it should do one thing, will now do another.

FreePascal has now defined itself as being unreliable.

[JuhaManninen]

CompareText compares uppercase values, while Utf8CompareText compares lowercase values.

Maybe Utf8CompareText should be changed to be consistent with CompareText. With our new UTF-8 system this function is more critical than it used to be.
Is there any good reason for using lowercase?

[JuhaManninen]

A function result changed with FPC3.X.

No, this time you can blame Lazarus and its UTF-8 functions mapped to widestringmanager.
Now I understand the reason. It can be fixed by using UTF8UpperCase instead of UTF8LowerCase in UTF8CompareText. Isn't this correct? Does it cause any potential problems?