XML and encoding='us-ascii'

[Dzandaa]

Hi everybody,

 Working with Lazarus Pascal 2.012 FPC 2.3.0 on Windows, Linux and MacOS

I have to read values from 8000 xml files

The encoding is

<?xml version='1.0' encoding='us-ascii'?>

Using the DOM and XMLRead Units,

When calling:

Code: Pascal  [Select][+][-]

1. procedure XMLAnalyze(FileName: String);
2. var
3.   PassNode: TDOMNode;
4.   Doc: TXMLDocument;
5. begin
6.  try
7.    // Read in xml file from disk
8.    ReadXMLFile(Doc, FileName);  
9.  

I have the error: Encoding: 'us-ascii' is not supported

Is there another XML library supporting 'us-ascii' or another way without modifying the XML files?

Thank you.

[PascalDragon]

Not tested, but in essence you need to register a decoder (best at startup) for us-ascii:

Code: Pascal  [Select][+][-]

1. function UsAsciiDecode(aContext: Pointer; aInBuf: PChar; var aInCnt: Cardinal; aOutBuf: PWideChar; var aOutCnt: Cardinal): Integer; stdcall;
2. var
3.   i, len: Cardinal;
4. begin
5.   if aInCnt < aOutCnt then
6.     len := aInCnt
7.   else
8.     len := aOutCnt;
9.   // us-ascii shouldn't contain any character > $7f
10.   for i := 0 to len - 1 do
11.     if aInBuf[i] < $80 then
12.       aOutBuf[i] := WideChar(aInBuf[i])
13.     else
14.       aOutBuf[i] := '?';
15.   Dec(aInCnt, len);
16.   Dec(aOutCnt, len);
17.   Result := len;
18. end;
19.  
20. function GetUsAsciiDecoder(const aEncoding: string; out aDecoder: TDecoder): Boolean; stdcall;
21. begin
22.   Result := False;
23.   if aEncoding = 'us-acsii' then begin
24.     aDecoder.Context := Nil;
25.     // you should probably also be able to use XMLUtils.Decode_8859_1 instead
26.     // which does the same as UsAsciiDecode except for the ? replacement
27.     aDecoder.Decode := @UsAsciiDecode;
28.     aDecoder.Cleanup := Nil;
29.     Result := True;
30.   end;
31. end;
32.  
33. begin
34.   // in unit XmlTextReader
35.   RegisterDecoder(@GetUsAsciiDecoder);
36.   // and now you can load XML files with us-acsii as usual
37. end.

[Dzandaa]

Hello,

Thank you very much.

Just corrected a few typo's and it works!!!

Code: Pascal  [Select][+][-]

1. function UsAsciiDecode(aContext: Pointer; aInBuf: PChar; var aInCnt: Cardinal; aOutBuf: PWideChar; var aOutCnt: Cardinal): Integer; stdcall;
2. var
3.   i, len: Cardinal;
4. begin
5.   if aInCnt < aOutCnt then
6.     len := aInCnt
7.   else
8.     len := aOutCnt;
9.   // us-ascii shouldn't contain any character > $7f
10.   for i := 0 to len - 1 do
11.     if aInBuf[i] < char($80) then
12.       aOutBuf[i] := WideChar(aInBuf[i])
13.     else
14.       aOutBuf[i] := '?';
15.   Dec(aInCnt, len);
16.   Dec(aOutCnt, len);
17.   Result := len;
18. end;
19.  
20. function GetUsAsciiDecoder(const aEncoding: string; out aDecoder: TDecoder): Boolean; stdcall;
21. begin
22.   Result := False;
23.   if aEncoding = 'us-ascii' then begin
24.     aDecoder.Context := Nil;
25.     // you should probably also be able to use XMLUtils.Decode_8859_1 instead
26.     // which does the same as UsAsciiDecode except for the ? replacement
27.     aDecoder.Decode := @UsAsciiDecode;
28.     aDecoder.Cleanup := Nil;
29.     Result := True;
30.   end;
31. end;    
32.  
33. begin
34.   // in unit XmlTextReader
35.   RegisterDecoder(@GetUsAsciiDecoder);
36.   // and now you can load XML files with us-acsii as usual
37. end.                                            
38.  

Lazarus, best in the world (for portability)

[jollytall]

Good news, I was struggling with it for some time. I will try this too.
Can it be added to the standard libraries? If it is so easy, it would make sense to have it added centrally, rather than everybody doing it separately.
Btw. I chose the simple way. When acquire the xml files, I check them (i also read thousands) and replace us-ascii to utf-8 in the encoding. Since a us-ascii character set is a subset of utf-8, it works.

[PascalDragon]

Can it be added to the standard libraries? If it is so easy, it would make sense to have it added centrally, rather than everybody doing it separately.

File a bug report then, please.

[jollytall]

A bit off: I tried, but cannot login, register. I tried to use my Github account, as well as a simple registration. For both I got an error:
Sign-in failed because Email is not allowed for sign-up. Please use your regular email address. Check with your administrator..
Any idea what can be wrong?

[PascalDragon]

You'll have to ask the administrators of GitLab. We have no control of that part.

[BeniBela]

The XML parser in my Internet Tools  parses encoding='us-ascii' in the default settings

Can it be added to the standard libraries? If it is so easy, it would make sense to have it added centrally, rather than everybody doing it separately.

File a bug report then, please.

I did that 6 years ago: https://gitlab.com/freepascal.org/fpc/source/-/issues/29571

[PascalDragon]

Can it be added to the standard libraries? If it is so easy, it would make sense to have it added centrally, rather than everybody doing it separately.

File a bug report then, please.

I did that 6 years ago: https://gitlab.com/freepascal.org/fpc/source/-/issues/29571

Ah, good to know that there already is a bug report about that.

[jollytall]

With all due respect to the developers knowing how hard they work on all sort of things parallel, I would still ask, if there is a chance that this bug will not live up to its 10th birthday? As i understand it would be rather simple to fix.