Recent

Author Topic: Sharing a function (Also: Is there a BUG in lazUTF8.UTF8LengthFast)  (Read 17255 times)

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Sharing a function (Also: Is there a BUG in lazUTF8.UTF8LengthFast)
« Reply #30 on: December 04, 2017, 01:07:41 pm »
I could not replicate the problem on my side. The compiler neglected my simple request to produce PIC code. :-/
I get the same error with the new code, too. I am using FPC trunk now, maybe it matters. I will switch to 3.0.4 in near future.

Forget the previous ones. It turns out it is better (shorter?) generating the values by code:
Code: Pascal  [Select][+][-]
  1. function UTF8Len_aligned16(p16: pchar; BlockCount: PtrInt):PtrInt;assembler;
  2. Label
  3.  loop;
  4. asm
  5.   PUSH ecx
  6.   PUSH edi
  7.   PUSH ebx
  8.  
  9.   { tmp counter }
  10.   MOV ecx, 0
  11.  
  12.   { masks }
  13.   //xmm0 := ($00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00,$00);
  14.   PXOR xmm0,  xmm0
  15.  
  16.   //xmm1 := ($01,$01,$01,$01,$01,$01,$01,$01,$01,$01,$01,$01,$01,$01,$01,$01);
  17.   PCMPEQW xmm1, xmm1
  18.   PSRLW xmm1, 15
  19.   PACKUSWB xmm1, xmm1
  20.  
  21.   //xmm2 := ($80,$80,$80,$80,$80,$80,$80,$80,$80,$80,$80,$80,$80,$80,$80,$80);
  22.   MOVAPD xmm2, xmm1
  23.   PSLLQ xmm2, 7
  24.  
  25.   //xmm3 := ($FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF);
  26.   PCMPEQW xmm3, xmm3
  27.  
  28. Loop:
  29.   { On big data it increased the speed from 13.x to 15.x times }
  30.   { On small data it decreased the speed }
  31.   //PREFETCHNTA [p16+256]
  32.  
  33.   { get 16 bytes }
  34.   MOVDQA xmm4, [p16]
  35.  
  36.   { Invert 16 bytes }
  37.   MOVDQA xmm5, xmm4
  38.   ANDNPD xmm5, xmm3 { xmm5 = not xmm4}
  39.  
  40.   { Shift the inverted bytes 6 bits to the right }
  41.   PSRLQ  xmm5, 6
  42.  
  43.   { Keep msb of each non-inverted byte }
  44.   PAND xmm4, xmm2//ONEMASKx80
  45.  
  46.   { Shift them to right 7 bits }
  47.   PSRLQ  xmm4, 7 { Shift Right Logical QWord }
  48.  
  49.   { A one in the 1st bit means: NOT the first byte of a codepoint }
  50.   PAND xmm5, xmm4
  51.  
  52.   { Count them ;-) }
  53.   PSADBW xmm5, xmm0
  54.  
  55.   MOVD edi, xmm5
  56.   PEXTRW ebx, xmm5, 4
  57.  
  58.   ADD ecx, ebx
  59.   ADD ecx, edi
  60.  
  61.   { Next 16 bytes }
  62.   ADD p16, 16
  63.   DEC edx
  64.   JNZ Loop
  65.  
  66.   { Result }
  67.   MOV eax, ecx
  68.  
  69.   POP ebx
  70.   POP edi
  71.   POP ecx
  72. end;

tomitomy

  • Sr. Member
  • ****
  • Posts: 251
Re: Sharing a function (Also: Is there a BUG in lazUTF8.UTF8LengthFast)
« Reply #31 on: December 11, 2017, 05:01:03 am »
Hi, @engkin.

Can you write an inverse function with this CPU feature? similar to the following one:

Code: Pascal  [Select][+][-]
  1. // convert codepoint index to byte index
  2. // PStr  :The source string pointer
  3. // ASize :The source string size
  4. // UPos  :The codepoint index (based of 1)
  5. // Return value: The byte index (based of 1, if return 0 means that UPos is unlawful, large than ASize means that UPos is beyond the scope
  6. function UTF8PosToBytePos(const PStr: PChar; const ASize: SizeInt; UPos: SizeInt): SizeInt;
  7. begin
  8.   if UPos < 1 then Result := -1 else Result := 0;
  9.  
  10.   while (UPos > 1) and (Result < ASize) do begin
  11.     case PStr[Result] of
  12.       // #0  ..#127: Inc(Pos);
  13.       #192..#223: Inc(Result, 2);
  14.       #224..#239: Inc(Result, 3);
  15.       #240..#247: Inc(Result, 4);
  16.       else Inc(Result);
  17.     end;
  18.     Dec(UPos);
  19.   end;
  20.   Inc(Result);
  21. end;

This should be very useful.
« Last Edit: December 11, 2017, 06:15:23 am by tomitomy »

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Sharing a function (Also: Is there a BUG in lazUTF8.UTF8LengthFast)
« Reply #32 on: December 11, 2017, 08:27:12 pm »
Hi, @engkin.

Can you write an inverse function with this CPU feature? similar to the following one:

Code: Pascal  [Select][+][-]
  1. // convert codepoint index to byte index
  2. // PStr  :The source string pointer
  3. // ASize :The source string size
  4. // UPos  :The codepoint index (based of 1)
  5. // Return value: The byte index (based of 1, if return 0 means that UPos is unlawful, large than ASize means that UPos is beyond the scope
  6. function UTF8PosToBytePos(const PStr: PChar; const ASize: SizeInt; UPos: SizeInt): SizeInt;
  7. begin
  8.   if UPos < 1 then Result := -1 else Result := 0;
  9.  
  10.   while (UPos > 1) and (Result < ASize) do begin
  11.     case PStr[Result] of
  12.       // #0  ..#127: Inc(Pos);
  13.       #192..#223: Inc(Result, 2);
  14.       #224..#239: Inc(Result, 3);
  15.       #240..#247: Inc(Result, 4);
  16.       else Inc(Result);
  17.     end;
  18.     Dec(UPos);
  19.   end;
  20.   Inc(Result);
  21. end;

This should be very useful.
I can try.  :)

tomitomy

  • Sr. Member
  • ****
  • Posts: 251
Re: Sharing a function (Also: Is there a BUG in lazUTF8.UTF8LengthFast)
« Reply #33 on: December 12, 2017, 06:09:28 am »
I can try.  :)

Thank you very much, engkin !  :)

BrunoK

  • Sr. Member
  • ****
  • Posts: 452
  • Retired programmer
Re: Sharing a function (Also: Is there a BUG in lazUTF8.UTF8LengthFast)
« Reply #34 on: May 27, 2018, 07:42:15 pm »
My entry in pure pascal for little endian processors :
Code: Pascal  [Select][+][-]
  1. unit uUTF8CharSize;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7.  
  8. const
  9.   cNAK_Hex15 = DWord($15);     // Internal functions return value for a bad CPx1 character
  10.                                // (or maybe (???) $1A to indicate a non translatable char)
  11.                                // istead of '?' which is a relatively frequent char
  12.   cUTF8Diamond = DWord($FDFF); // and uUTF8Diamond for a bad UTF8 sequence { '�' or diamond }
  13.                                // this is invariant
  14.  
  15. { Returns nb of charater for UTF8 code point without checking restricted 2nd byte
  16.   Returns : [1..4] nb bytes for UTF-8 codepoint
  17.             -2 : Diamond char
  18.             -1 : Invalid sequence                                             }
  19.  
  20. function UTF8Size(apChar : PChar): integer; {  inline; }
  21.  
  22. implementation
  23.  
  24. const
  25.   // 2 bytes UTF-8
  26.   cUtf2Mask = DWORD($0000C0E0); // And mask to extract useful bits
  27.   cUtf2XOR  = DWord($000080C0); // XOR mask to a 0 result
  28.  
  29.   // 3 bytes UTF-8
  30.   cUtf3Mask = DWORD($00C0C0F0);
  31.   cUtf3XOR  = DWord($008080E0);
  32.  
  33.   // 4 bytes UTF-8
  34.   cUtf4Mask = DWORD($C0C0C0F8);
  35.   cUtf4XOR  = DWord($808080F0);
  36.  
  37.   // Diamond UTF-8
  38.   cUtf4DiamondMask  = DWORD($0000FFFF);
  39.   cUtf4DiamondXOR   = cUTF8Diamond;
  40.  
  41. function UTF8Size(apChar : PChar): integer;
  42. var
  43.   lDWordI: DWord;          // aPDWord^ value
  44. begin
  45.   if apChar^<#128 then
  46.     Exit(1);
  47.  
  48.   lDWordI:=PDWord(apChar)^;
  49.   { Test 2 bytes }
  50.   if ((lDWordI and cUtf2Mask) xor cUtf2XOR) = 0 then
  51.     Exit(2);
  52.   { Test 3 bytes }
  53.   if ((lDWordI and cUtf3Mask) xor cUtf3XOR) = 0 then
  54.     Exit(3);
  55.   { Test Diamond }
  56.   if ((lDWordI and cUtf4DiamondMask) xor  cUtf4DiamondXOR)=0 then
  57.     Exit(-2);
  58.   { Test 4 bytes }
  59.   if ((lDWordI and cUtf4Mask) xor cUtf4XOR) = 0 then
  60.     Exit(4);
  61.   Exit(-1);
  62. end;
  63.  
  64. end.
  65.  

 

TinyPortal © 2005-2018