Recent

Author Topic: String and Database problem  (Read 1043 times)

thehidden

  • Jr. Member
  • **
  • Posts: 56
  • Freelancer
String and Database problem
« on: May 28, 2020, 01:49:19 pm »
Hi,
first Application with german language to enter.
My problem

I have a Memo Field or Edit Field. When - for example - an ä is entered, I see in the database an ä

When I change it in the database back to ä I can get it back to the Memo or Edit Field as ä. But when I store it again, it is shown as ä

Any idea what is going wrong?
I am mostly writing software for Image/Video based Process Documentation and Warehouse/Invoicing.
Other Software only if the project is interesting.

fred

  • Full Member
  • ***
  • Posts: 179
Re: String and Database problem
« Reply #1 on: May 28, 2020, 02:07:52 pm »
I think windows code pages, Memo and Edit uses UTF-8 and your database uses a windows code page.
Look for UTF8ToWinCP, WinCPToUTF8
Started with OmegaSoft Pascal on OS-9/68k , now Lazarus 2.0.8 / FPC 3.0.4 on Windows 7

af0815

  • Hero Member
  • *****
  • Posts: 553
Re: String and Database problem
« Reply #2 on: May 28, 2020, 03:25:05 pm »
What kind of DB ? Which DB-drivers (Connection)?

In Server based drivers you have often options to say, which charset they should use.
 
regards
Andreas

thehidden

  • Jr. Member
  • **
  • Posts: 56
  • Freelancer
Re: String and Database problem
« Reply #3 on: May 28, 2020, 04:17:39 pm »
What kind of DB ? Which DB-drivers (Connection)?

In Server based drivers you have often options to say, which charset they should use.
Desktop Database Elevate DB
All Tables are set to UTF-8.

I have tried to use
{$codepage utf8}
{$modeswith unicodestrings} 

but without success.

I am mostly writing software for Image/Video based Process Documentation and Warehouse/Invoicing.
Other Software only if the project is interesting.

af0815

  • Hero Member
  • *****
  • Posts: 553
Re: String and Database problem
« Reply #4 on: May 28, 2020, 04:56:26 pm »
I have tried to use
{$codepage utf8}
{$modeswith unicodestrings} 

but without success.
This is not for the DB Connection, only for compiler.

Do you use SQLdb or ZEOS ?
regards
Andreas

thehidden

  • Jr. Member
  • **
  • Posts: 56
  • Freelancer
Re: String and Database problem
« Reply #5 on: May 28, 2020, 05:01:32 pm »
I have tried to use
{$codepage utf8}
{$modeswith unicodestrings} 

but without success.
This is not for the DB Connection, only for compiler.

Do you use SQLdb or ZEOS ?

Nothing from both. ElevateDB comes with his own connection API. The only requirement is, you need to make sure that you feed the connector with Unicode.
I am mostly writing software for Image/Video based Process Documentation and Warehouse/Invoicing.
Other Software only if the project is interesting.

af0815

  • Hero Member
  • *****
  • Posts: 553
Re: String and Database problem
« Reply #6 on: May 28, 2020, 10:40:22 pm »
ElevateDB comes with his own connection API. The only requirement is, you need to make sure that you feed the connector with Unicode.
Which Unicode  :o UTF-8 (Lazarus uses) or UTF-16 (like IMHO Delphi uses)
But i think the guys from this DB system can give you more help or samples. I see only a paywall, but no information on their HP.
regards
Andreas

thehidden

  • Jr. Member
  • **
  • Posts: 56
  • Freelancer
Re: String and Database problem
« Reply #7 on: May 28, 2020, 11:34:52 pm »
Which Unicode  :o UTF-8 (Lazarus uses) or UTF-16 (like IMHO Delphi uses)
But i think the guys from this DB system can give you more help or samples. I see only a paywall, but no information on their HP.

I don't know. With Delphi I have no problems.

But when I use the database manager to add a record with german umlauts, I can read them without problem to a edit or memo field. Only when I write back to the database I get an unwanted result.

Any way to find out what string type the database connector is sending?

Something like

Code: Pascal  [Select][+][-]
  1. if whatutf(dataset.field[0].asstring) = 'UTF8' then
  2.   showmessage('UTF-8 and not UTF-16')

I am mostly writing software for Image/Video based Process Documentation and Warehouse/Invoicing.
Other Software only if the project is interesting.

winni

  • Hero Member
  • *****
  • Posts: 1511
Re: String and Database problem
« Reply #8 on: May 28, 2020, 11:53:59 pm »
Hi!

Allways treasures inside of Lazarus:

In the unit LConvEncoding you find
Code: Pascal  [Select][+][-]
  1.  
  2. function GuessEncoding(const s: string): string;

The result returns a string with the name of the codepage.

Winni

thehidden

  • Jr. Member
  • **
  • Posts: 56
  • Freelancer
Re: String and Database problem
« Reply #9 on: May 29, 2020, 09:22:03 am »
Hi!

Allways treasures inside of Lazarus:

In the unit LConvEncoding you find
Code: Pascal  [Select][+][-]
  1.  
  2. function GuessEncoding(const s: string): string;

The result returns a string with the name of the codepage.

Winni
Thank you for the help.

Database returns as UTF-8 and I write as UTF-8.

But I get a compiler message for the line with the SQL Execute and don't see why.

Code: Pascal  [Select][+][-]
  1. var
  2.    temptxt1 : string;
  3. begin
  4. temptxt1 := 'UPDATE "Data" SET content1 = ''' + Memo1.text+''' where ID = ' + IDMarker;
  5.  
  6. EDBDatabase1.StartTransaction(EmptyEDBStringsArray);  //prepare Transaction for all affected tables
  7.  
  8. EDBDatabase1.Execute(temptxt1);   //Compiler warning for this line as below.
  9.  
  10. Databaseform.EDBDatabase1.Commit();
  11.  

Warning message:
Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString"
I am mostly writing software for Image/Video based Process Documentation and Warehouse/Invoicing.
Other Software only if the project is interesting.

Zoran

  • Hero Member
  • *****
  • Posts: 1564
    • http://wiki.lazarus.freepascal.org/User:Zoran
Re: String and Database problem
« Reply #10 on: May 29, 2020, 09:53:32 am »


Warning message:
Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString"

Is this directive:
{$modeswith unicodestrings}
still present in your unit?

Remove it and try again.

thehidden

  • Jr. Member
  • **
  • Posts: 56
  • Freelancer
Re: String and Database problem
« Reply #11 on: May 29, 2020, 10:12:09 am »


Warning message:
Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString"

Is this directive:
{$modeswith unicodestrings}
still present in your unit?

Remove it and try again.

It is now removed. Warning is gone, but problem is not fixed.

Anyway, thank you for clearing the Warning message.
I am mostly writing software for Image/Video based Process Documentation and Warehouse/Invoicing.
Other Software only if the project is interesting.

Zoran

  • Hero Member
  • *****
  • Posts: 1564
    • http://wiki.lazarus.freepascal.org/User:Zoran
Re: String and Database problem
« Reply #12 on: May 29, 2020, 12:21:46 pm »


Warning message:
Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString"

Is this directive:
{$modeswith unicodestrings}
still present in your unit?

Remove it and try again.

It is now removed. Warning is gone, but problem is not fixed.

Anyway, thank you for clearing the Warning message.

That directive tells to compiler to treat "String" type as UnicodeString (not AnsiString). Except in special cases, you should not use it in LCL applications.

Now, make sure you do not use UTF8String anywhere, as LCL handles UTF8 encodings its own way, so, except in special cases, you should not normally let FPC make any conversions in LCL applicatons, it would lead to double conversions, thus corrupting string's content.

Simply, in general, this should be enough:

- make sure that your source files (units) are saved in utf8 encoding (I am not talking about using any compiler directive, just about how the files are saved).
- declare your string variables as just "String", do not use UTF8String, UnicodeString or any specific code page string.
- do not set any code page directive yourself.

thehidden

  • Jr. Member
  • **
  • Posts: 56
  • Freelancer
Re: String and Database problem
« Reply #13 on: May 29, 2020, 01:29:20 pm »


Warning message:
Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString"

Is this directive:
{$modeswith unicodestrings}
still present in your unit?

Remove it and try again.

It is now removed. Warning is gone, but problem is not fixed.

Anyway, thank you for clearing the Warning message.

That directive tells to compiler to treat "String" type as UnicodeString (not AnsiString). Except in special cases, you should not use it in LCL applications.

Now, make sure you do not use UTF8String anywhere, as LCL handles UTF8 encodings its own way, so, except in special cases, you should not normally let FPC make any conversions in LCL applicatons, it would lead to double conversions, thus corrupting string's content.

Simply, in general, this should be enough:

- make sure that your source files (units) are saved in utf8 encoding (I am not talking about using any compiler directive, just about how the files are saved).
- declare your string variables as just "String", do not use UTF8String, UnicodeString or any specific code page string.
- do not set any code page directive yourself.

The directive was only added for testing. I use only string and not switching/selecting codepages.
It should work, but it does not.

If nothing else helps, I will store the information encoded as base64 in the database and transfer it back when I read them.
I am mostly writing software for Image/Video based Process Documentation and Warehouse/Invoicing.
Other Software only if the project is interesting.

Zoran

  • Hero Member
  • *****
  • Posts: 1564
    • http://wiki.lazarus.freepascal.org/User:Zoran
Re: String and Database problem
« Reply #14 on: May 29, 2020, 01:47:48 pm »
Perhaps ElevateDB connection tries to do string conversions based on passed string encoding (which could make problem in LCL).

You might try this:
Code: Pascal  [Select][+][-]
  1. var
  2.    temptxt1 : string;
  3.    temptxt2 : UTF8String;
  4. //...
  5. //...
  6.  
  7.     // Instead of this:
  8.     // EDBDatabase1.Execute(temptxt1);
  9.     // just try:
  10.     temptxt2 := UTF8String(RawByteString(temptxt1));
  11.     EDBDatabase1.Execute(temptxt2);
  12.  

 

TinyPortal © 2005-2018