UTF8 & Locale

[JZS]

Hello everyone,
Until recently I was thinking that I understand UTF8 and system locale, but I discovered there is a missing link in my applications.

Some of my clients use Arabic, so I use UTF8 collation in my DB tables, and when viewing the data from my application everything looks OK as long as the system locale is set to Arabic.

But there is a scenario that I need to handle, apparently is annoying to miss.
When a user with System Locale set to Arabic adds some arabic text, all other users, with Arabic Locale, are able to read, but non-arabic locale see ?????, instead of the real text, also the other way around, non-arabic locale users when adding arabic text all non-arabic locale users are able to read, but those who have system locale set to arabic see ?????.

How to make the text appears in both configuration correctly?

I know the difference about UTF8 and ASCII, in bytes and char representation,  but how can I enable both users to see what others enter regardless the system locale:
1. Does it involve collation of my DB Tables (cp1256 <=> UTF8)?
2. Or, I encode (SysToUTF8?) when adding arabic text to my UTF8 Collation Tables?

Appreciate any input on this subject.

I use Laz 1.4 & MariaDB 10.

Regards,

[JuhaManninen]

Until recently I was thinking that I understand UTF8 and system locale ...

I know the feeling

When a user with System Locale set to Arabic adds some arabic text, all other users, with Arabic Locale, are able to read, but non-arabic locale see ?????

It means you are dealing with ASCII + codepages. If everything was in UTF-8, you would not see ?????

How to make the text appears in both configuration correctly?

By using our improved Unicode system :
  http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus
Then if your DB fully supports UTF-8, you don't need to worry about encodings (not much anyway).

[LacaK]

How do you have defined connection charset between your application and MariaDB ?

[JZS]

It means you are dealing with ASCII + codepages. If everything was in UTF-8, you would not see ?????

I knew UTF8 will complicate things

By using our improved Unicode system :
  http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus
Then if your DB fully supports UTF-8, you don't need to worry about encodings (not much anyway).

So if I understand correctly then I must use FPC 2.7.1+ to enable default usage of UTF8?
If yes, can I use FPC 2.7.1 (3.x?) with 1.4?

I will download the latest from trunk just in case.

Thank you Juha.

[JZS]

How do you have defined connection charset between your application and MariaDB ?

Always UTF8

[JuhaManninen]

I knew UTF8 will complicate things

No, quite the opposite. Once everything is UTF-8, it will simplify things.

So if I understand correctly then I must use FPC 2.7.1+ to enable default usage of UTF8?
If yes, can I use FPC 2.7.1 (3.x?) with 1.4?

FPC trunk is now 3.1.1. You can use it or the fixes_3.0 branch which will soon (?) become the FPC 3.0 release.
FPC 2.7.1 is history.

[JZS]

No, quite the opposite. Once everything is UTF-8, it will simplify things.

I hope so.

FPC trunk is now 3.1.1. You can use it or the fixes_3.0 branch which will soon (?) become the FPC 3.0 release.
FPC 2.7.1 is history.

Do you mean this one (for win32):
ftp://ftp.freepascal.org/pub/fpc/snapshot/fixes/i386-win32/

After downloading, shall I just set the compiler under Lazarus 1.4 to this new fixes, or I need to rebuild lazarus from source with this new compiler?

[JuhaManninen]

Do you mean this one (for win32):
ftp://ftp.freepascal.org/pub/fpc/snapshot/fixes/i386-win32/

Yes it looks right. I installed with fpcup myself.

After downloading, shall I just set the compiler under Lazarus 1.4 to this new fixes, or I need to rebuild lazarus from source with this new compiler?

You need Lazarus trunk, not the 1.4 release.
Set the compiler and its sources in Lazarus options, then rebuild Lazarus. Then you are ready to go.

[JZS]

I am going to do that. Will get back if anything goes wrong.

Many Thanks Juha, you are a great help, as always.

Regards,

J

[JZS]

Hi Juha,
I managed to install FPC and Lazarus from trunk.
FPC 3.1.1 revision: 30773
Lazarus 1.5 revision: 48923

Good news:
I got FPC 3.1.1 & Lazarus 1.5 both built successfully using fpc 2.6.4.

Bad news is I can't run any program or rebuild lazarus, I guess there is something I am missing here.

Steps of what I did (as I remember):
1. Built fpc & lazarus by using Martin's directions:

Code: [Select]

PATH=C:\lazarus\fpc\2.6.4\bin\i386-win32;
SET FPC=C:\lazarus\fpc\2.6.4\bin\i386-win32\fpc.exe

make.exe clean  distclean
make.exe all  OPT="-gw -godwarfset -O-1"
make.exe install  INSTALL_PREFIX=c:\FPC\trunk  COPYTREE=echo

Code: [Select]

C:\lazarus\fpc\2.6.4\bin\i386-win32\make clean bigide
2. Started the new Lazarus with> startlazarus.exe --pcp="New Config Path"

3. Ran Lazarus and filled in the Compiler/Make/Source/Debugger fields, as follows:
- Compiler=The trunk fpc that I build -> OK (needed to copy the fpc.cfg over, from 2.6.4)
- Make= from 2.6.4 because there is no make.exe in the new built fpc dir.
- Source= from 2.6.4, again not found in the new fpc.
- Debugger= copied "mingw" directory from Laz 1.4 installation, (I hope this one did not make a big difference).

4. Tried to run the default started project, I got an error message about Unit "Interfaces" then I did step 2 again, then it never showed up again.

5. Tried to rebuild Lazarus (from the menu -> Tools -> Build Lazarus...), got this error:
registerfcl.pas(43,3) Fatal: Cannot find ServiceManager used by RegisterFCL of package FCL.


So now am out of tricks, need some push please.

Regards,

[zeljko]

You're probably missing PPC_CONFIG_PATH if fpc 3.1.1 isn't in the path.
EDIT: PPC_CONFIG_PATH means where fpc.cfg is.

[JZS]

You're probably missing PPC_CONFIG_PATH if fpc 3.1.1 isn't in the path.
EDIT: PPC_CONFIG_PATH means where fpc.cfg is.

i can get Lazarus run & rebuild with fpc 2.6.4, but can not get it working with fpc 3.1.1.

How can I get the missing files (fpc.cfg, make.exe, probably others?) for 3.1.1?

[JZS]

I tried "fpcmkcfg.exe -o fpc.cfg" but that was not accepted (i.e could not run or rebuild Lazarus).

I guess there is one step missing to link 1.5 with 3.1.1.

[totya]

Fcpup sometimes working for me, under windows. You can get fpc and lazarus svn at once.

https://github.com/LongDirtyAnimAlf/Reiniero-fpcup/tree/master/bin/i386-win32

I tired it and working in few times, but isn't perfect yet.

[JZS]

Thanks totya.
fpcup is refusing to work, for sometime now (although it did work for me before), that's why I am in this quest.

I even installed Getlazarus, but same thing, did not rebuild, complaining about Interface unit.

No matter what I did, the different combinations of 1.5 and 3.1.1 did not work out.

The Interface unit error message when run the default project is:
project1.lpr(9,3) Fatal: Cannot find Interfaces used by Project1. Make sure all ppu files of a package are in its output directory. ppu in wrong directory=C:\Development\Lazarus\lcl\units\i386-win32\win32\interfaces.ppu. Clean up package "LCL"..