Recent

Author Topic: "How to program using UNICODE in Pascal"  (Read 5821 times)

Ñuño_Martínez

  • Hero Member
  • *****
  • Posts: 1186
    • Burdjia
"How to program using UNICODE in Pascal"
« on: February 13, 2018, 01:35:23 pm »
I looked for a "step by step" tutorial about how to correctly manage UNICODE texts with Free Pascal but I didn't found any in the whole Internet, I only find partial explanations about data types or how to convert text form this enconding to that other one, but not an actual simple example.  I mean, it shouldn't be hard to "the user inputs text, then I save that text in a UTF-8 text file in a filesystem that uses UTF-8 for filenames, while using LCL internationalization utils for labels and buttons" or something, but reading the wiki explanations I find it extremely hard. May be I'm doing the wrong question?

I understand how UNICODE works, that's not the problem.  I was using UTF-8 for a long time, but I used ANSISTRING to store the texts, and never used UTF-8 to name files (for example).  I know it is wrong but I still used it because it worked for me since I know how the text was encoded, but now and then I'm doing stuff that other people is using and they shouldn't expect to use it wrong, should they?
« Last Edit: February 13, 2018, 01:38:34 pm by Ñuño_Martínez »
Are you interested in game programming? Join the Pascal Game Development community!
Also visit the Game Development Portal

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: "How to program using UNICODE in Pascal"
« Reply #1 on: February 13, 2018, 05:27:35 pm »
I looked for a "step by step" tutorial about how to correctly manage UNICODE texts with Free Pascal but I didn't found any in the whole Internet, I only find partial explanations about data types or how to convert text form this enconding to that other one, but not an actual simple example.  I mean,

... it shouldn't be hard to "the user inputs text, then I save that text in a UTF-8 text file in a filesystem that uses UTF-8 for filenames, while using LCL internationalization utils for labels and buttons" or something, but reading the wiki explanations I find it extremely hard. May be I'm doing the wrong question?
I don't know if you are serious. The very first link from your search is:
 Unicode Support in Lazarus - Free Pascal wiki
Shortly put: everything just works with UTF-8, most of the time you don't need to think of it.
You can find examples here:
 http://wiki.freepascal.org/UTF8_strings_and_characters
See also this if you want to make 100% Delphi compatible code also when iterating codepoints and "user perceived characters":
 http://wiki.freepascal.org/Unicode_Support_in_Lazarus#CodePoint_functions_for_encoding_agnostic_code
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Ñuño_Martínez

  • Hero Member
  • *****
  • Posts: 1186
    • Burdjia
Re: "How to program using UNICODE in Pascal"
« Reply #2 on: February 14, 2018, 01:52:59 pm »
The "UTF8 strings and characters" seems to be what I was looking for. Thank-you.

I don't know if you are serious. The very first link from your search is:
 Unicode Support in Lazarus - Free Pascal wiki
I actually find that "Unicode Support in Lazarus" page quite confusing (may be because I'm not an English native-speaker and I'm not used to it yet, or may be because there are some concepts about UNICODE that I don't have).  As far as I can understand, it enumerates how it works but I can't see how or why or when to use that stuff.
Are you interested in game programming? Join the Pascal Game Development community!
Also visit the Game Development Portal

wp

  • Hero Member
  • *****
  • Posts: 11857
Re: "How to program using UNICODE in Pascal"
« Reply #3 on: February 14, 2018, 02:01:44 pm »
I looked for a "step by step" tutorial about how to correctly manage UNICODE texts with Free Pascal [...]
With "manage" do you mean: How to load and save text files with unicode encoding? What do you mean with "unicode", by the way? UTF-8 (like Lazarus) or UTF-16 (like Delphi)?

Ñuño_Martínez

  • Hero Member
  • *****
  • Posts: 1186
    • Burdjia
Re: "How to program using UNICODE in Pascal"
« Reply #4 on: February 14, 2018, 02:10:38 pm »
I looked for a "step by step" tutorial about how to correctly manage UNICODE texts with Free Pascal [...]
With "manage" do you mean: How to load and save text files with unicode encoding? What do you mean with "unicode", by the way? UTF-8 (like Lazarus) or UTF-16 (like Delphi)?
A bit of everything.  I need to control how text is encoded.

The actual problem is that I'm having some troubles with Allegro.pas.  It works correctly with Free Pascal but Delphi seems to have problems with texts (among others).  May be also because Allegro is written in plain C and its UNICODE support is the weirdest thing I ever saw.
Are you interested in game programming? Join the Pascal Game Development community!
Also visit the Game Development Portal

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: "How to program using UNICODE in Pascal"
« Reply #5 on: February 14, 2018, 04:40:24 pm »
With "manage" do you mean: How to load and save text files with unicode encoding? What do you mean with "unicode", by the way? UTF-8 (like Lazarus) or UTF-16 (like Delphi)?
I understood he made clear he works with UTF-8:
 "the user inputs text, then I save that text in a UTF-8 text file in a filesystem that uses UTF-8 for filenames, while using LCL internationalization utils for labels and buttons"
Hence my earlier answer.

The actual problem is that I'm having some troubles with Allegro.pas.  It works correctly with Free Pascal but Delphi seems to have problems with texts (among others).  May be also because Allegro is written in plain C and its UNICODE support is the weirdest thing I ever saw.
I see from other posts that you have improved Allegro.pas. What are the actual problems? The Lazarus UTF-8 solution is amazingly compatible with Delphi at source code level.
That kind of problems exactly with Unicode are you experiencing?
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Ñuño_Martínez

  • Hero Member
  • *****
  • Posts: 1186
    • Burdjia
Re: "How to program using UNICODE in Pascal"
« Reply #6 on: February 14, 2018, 06:02:12 pm »
Some of the examples renders gibberish instead of the actual texts.  Also Delphi shows a hint when converting from STRING to ANSISTRING (or vice-versa) saying that data may be lost in the operation.

I'm aware the origin of this problem may be different, but I'm investigating.
« Last Edit: February 14, 2018, 06:08:49 pm by Ñuño_Martínez »
Are you interested in game programming? Join the Pascal Game Development community!
Also visit the Game Development Portal

rick2691

  • Sr. Member
  • ****
  • Posts: 444
Re: "How to program using UNICODE in Pascal"
« Reply #7 on: February 14, 2018, 08:16:28 pm »
Quote
Some of the examples renders gibberish instead of the actual texts.

That sounds like FONT BINDING, which Microsoft does with their infinite wisdom, to prevent you from publishing a document that they think is defective. They switch a font with yours, and it displays as gibberish in another language. They are so clever!

Rick
Windows 11, LAZ 2.0.10, FPC 3.2.0, SVN 63526, i386-win32-win32/win64, using windows unit

Ñuño_Martínez

  • Hero Member
  • *****
  • Posts: 1186
    • Burdjia
Re: "How to program using UNICODE in Pascal"
« Reply #8 on: April 24, 2018, 02:00:31 pm »
No, it is not FONT BINDING.  It was that I used the incorrect data type.  Delphi STRING is an alias of WIDESTRING wich is UTF-16 (AFAIK).  I've changed my definitions to use ANSISTRING instead and it works.  Now I have other problems.
Are you interested in game programming? Join the Pascal Game Development community!
Also visit the Game Development Portal

 

TinyPortal © 2005-2018