Recent

Author Topic: BufDataset foCaseInsensitive not working with "ñ" "Ñ"  (Read 1483 times)

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
BufDataset foCaseInsensitive not working with "ñ" "Ñ"
« on: January 03, 2020, 04:16:46 pm »
"Tempo" is the TEdit text.
The filter works fine for any text, except these special characters.

Code: Pascal  [Select][+][-]
  1. Productos.Filtered := False;
  2.       Productos.FilterOptions := [foCaseInsensitive];
  3.       //Productos.Filter   := ‘(sNombre = ’ + QuotedStr(‘*’ + Tempo + ‘*’)
  4.       //                    + ' OR sCodProducto = ' + QuotedStr(Tempo)
  5.       //                    + ' OR sEan = ' + QuotedStr(Tempo)
  6.       //                    + ' OR sEan2 = ' + QuotedStr(Tempo)+‘) ’
  7.       //                    + ' AND bActivo <> ' + QuotedStr(‘F’);
  8.  
  9.       Productos.Filter := ArmarFiltroProductos(Tempo); // the same as the commented lines above
  10.       Productos.Filter := Productos.Filter + ' AND sTipoProducto <> ' + QuotedStr(‘I’);
  11.       Productos.Filtered := True;

The lowercase of Ñ is ñ, but setting foCaseInsensitive has no effect.

If I type Ñ only uppercase elements of the DB are returned in the DBGrid. Like "Ñandú" but not "Cañón".
If I type ñ only lowercase elements of the DB are returned in the DBGrid. Like "Cañón" but not "Ñandú".

The DB has mixed content, some are upper case and some are lower case.

In BufDataset.pas I found this

Code: Pascal  [Select][+][-]
  1. function DBCompareText(subValue, aValue: pointer; size: integer; options: TLocateOptions): LargeInt;
  2.  
  3. begin
  4.   if [loCaseInsensitive,loPartialKey]=options then
  5.     Result := AnsiStrLIComp(pchar(subValue),pchar(aValue),length(pchar(subValue)))
  6.   else if [loPartialKey] = options then
  7.     Result := AnsiStrLComp(pchar(subValue),pchar(aValue),length(pchar(subValue)))
  8.   else if [loCaseInsensitive] = options then
  9.     Result := AnsiCompareText(pchar(subValue),pchar(aValue))
  10.   else
  11.     Result := AnsiCompareStr(pchar(subValue),pchar(aValue));
  12. end;

Anyways, the ñ and Ñ are in the ASCII table

http://ascii-table.com/ansi-codes.php

209   D1   U+00D1   Ñ   Latin Capital Letter N With Tilde
241   F1   U+00F1   ñ   Latin Small Letter N With Tilde

Any ideas on how to fix? I'm on Windows but the project works also on Linux and macOS.

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: BufDataset foCaseInsensitive not working with "ñ" "Ñ"
« Reply #1 on: January 03, 2020, 11:55:23 pm »
Attached a simple test project that replicates the Issue, I'm reporting it at the bugtracker too.

How to test:
- Click on the first button, and see the filtered records (it displays only 1 record, must display 2)
- Click on the second button, and see the filtered records (it displays only 1 record, must display 2)

Edit: Reported here
https://bugs.freepascal.org/view.php?id=36512
« Last Edit: January 03, 2020, 11:58:56 pm by lainz »

amartitegui

  • Jr. Member
  • **
  • Posts: 84
Re: BufDataset foCaseInsensitive not working with "ñ" "Ñ"
« Reply #2 on: February 01, 2023, 04:58:13 pm »
Hello
No news on this right?

LacaK

  • Hero Member
  • *****
  • Posts: 691
Re: BufDataset foCaseInsensitive not working with "ñ" "Ñ"
« Reply #3 on: February 22, 2023, 09:44:20 am »
IMO problem is in AnsiCompareText function. On Windows this functions calls Win API function CompareStringA(), which "Compares two character strings, for a locale specified by identifier" - so SompareStringA(LOCALE_USER_DEFAULT, NORM_IGNORECASE, ...) is called.

And there is remark "If your application is calling the ANSI version of CompareString, the function converts parameters via the default code page of the supplied locale. Thus, an application can never use CompareString to handle UTF-8 text."

But in Lazarus string are by default UTF-8 encoded so probably you supply UTF-8 strings to AnsiCompareText function, which will not work ...

In my experiments on Windows (ansi code page 1250):
- CompareStringA(LOCALE_USER_DEFAULT, NORM_IGNORECASE, 'ábc',3,'ÁBC',3)-2 == 0 ... OK
- CompareStringA(LOCALE_USER_DEFAULT, NORM_IGNORECASE, 'ábc',3,'ABC',3)-2 <> 0 ... OK
- CompareStringA(LOCALE_USER_DEFAULT, NORM_IGNORECASE+LINGUISTIC_IGNOREDIACRITIC, 'ábc',3,'ABC',3)-2 == 0 ... OK
« Last Edit: February 22, 2023, 10:22:11 am by LacaK »

Thaddy

  • Hero Member
  • *****
  • Posts: 14201
  • Probably until I exterminate Putin.
Re: BufDataset foCaseInsensitive not working with "ñ" "Ñ"
« Reply #4 on: February 22, 2023, 12:54:21 pm »
As per msdn, CompareStrEx should be called for UTF8/16. It looks like it has no impact except for windows versions before Vista.
See the advice here:
https://learn.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-comparestringa

So this indeed is a bug and needs to be corrected.

Also note that even CompareStrEx has some security considerations as announced as per:
https://learn.microsoft.com/en-us/windows/win32/intl/security-considerations--international-features
« Last Edit: February 22, 2023, 01:01:41 pm by Thaddy »
Specialize a type, not a var.

 

TinyPortal © 2005-2018