Detect string encoding

[McClane]

Hi I don't know how to detect the encoding of a string.

I'm making an IRC client and if someone writes in spanish and send a string with á or ¿ i need to know the encoding
to convert it to utf8.

Thanks in advance.

[Imants]

I do not think It is possible to detect string encoding only to guess.

If you are using Lazarus for client then anything user types in control should be in UTF8. If I remember correctly all string variables in Lazarus by default should be in UTF8.

But if user input in not in UTF8 but using local code-page then I think your only option is to find local code page in client side and when sending data to server convert it to UTF8 so that server only receives data in UTF8 format.

[wp]

Try the ChsDet (CharacterSet Detector) available through Online-Package-Manager.

See also https://forum.lazarus.freepascal.org/index.php/topic,34695.msg228321.html#msg228321 and http://chsdet.sourceforge.net/

[Thaddy]

Try the ChsDet (CharacterSet Detector) available through Online-Package-Manager.

See also https://forum.lazarus.freepascal.org/index.php/topic,34695.msg228321.html#msg228321 and http://chsdet.sourceforge.net/

That only works with enough entropy.... On four letters, throw a dice..

A BOM encoding was designed just to prevent that... But programmers are stubborn and forget these "little details" all the time. They are born like that....

[lucamar]

A BOM encoding was designed just to prevent that... But programmers are stubborn and forget these "little details" all the time. They are born like that....

Having a BOM wouldn't help the OP at all: the unique purpose of the BOM is to specify whether a Unicode file is LE or BE encoded and, indirectly, allows one to know if a file is Unicode text. Other than that ...

The OP's question is unsolvable: an array of characters may represent a string in almost any coceivable encoding and there is no simple way to "guess" which it might be. The only exceptions are the (relative) easy of guessing whether a file might be UTF8 and the even easier guess of a pure ASCII file.

Beyond that, how do you differentiate between Windoiws-1252, IBM-850 and IBM-437 encoded texts? You can't ... unless you have a nice expert system that can guess which language the text is and, by testing, the most probable encoding.

[ASerge]

In general, the IRC server knows nothing about encodings and works with a stream of bytes. Only clients can negotiate on the encoding used. Many IRC clients offer an explicit choice of the encoding used. Some of them prefer UTF8 by default, and use/recognize BOM for Unicode/UTF8 encoding.
It is clear that the use of Unicode or UTF8 in languages that are very different from english significantly reduces the size of the message. Therefore, users of one country prefer the local one-byte character set (if the alphabet allows).

[Remy Lebeau]

I'm making an IRC client and if someone writes in spanish and send a string with á or ¿ i need to know the encoding
to convert it to utf8.

Check if the string is already encoded in UTF-8 (easy enough to do manually), and if so then send it as-is, otherwise just use the user's current locale to convert the string to UTF-8, then send it.