Recent

Author Topic: No warning from FPC with ambiguous overload  (Read 2289 times)

440bx

  • Hero Member
  • *****
  • Posts: 2464
No warning from FPC with ambiguous overload
« on: April 12, 2021, 08:40:56 am »
Hello,

FPC determines which overloaded function/procedure to call by matching the parameters in the call to the parameters in the definitions, which is fine but, there are times when that isn't enough information to choose the correct overload.

Consider this example:
Code: Pascal  [Select][+][-]
  1. program AmbiguousOverload;
  2.  
  3. uses
  4.   Windows
  5.   ;
  6.  
  7. const
  8.   ntdll    = 'ntdll';
  9.   kernel32 = 'kernel32';
  10.  
  11.  
  12. function swprintf({ _out_ } OutWideBuffer    : pwidechar;
  13.                   { _in_  } InFormatString   : pwidechar;
  14.                   { _in_  } InWidechars      : pwidechar)
  15.          : integer; cdecl; external ntdll;                          { CDECL !!}
  16.  
  17. function swprintf({ _out_ } OutWideBuffer    : pwidechar;
  18.                   { _in_  } InFormatString   : pwidechar;
  19.                   { _in_  } InAnsiChars      : pchar)
  20.          : integer; cdecl external ntdll;                           { CDECL !!}
  21.  
  22.  
  23. var
  24.   WideBuf : packed array[0..511] of widechar;
  25.  
  26. begin
  27.   writeln;
  28.  
  29.   { the compiler should emit a warning stating that there isn't enough        }
  30.   { information to decide which overload to pick.                             }
  31.  
  32.   swprintf(WideBuf,
  33.            '%s',
  34.            'Hello character world!');  { compiler picks pchar overload        }
  35.  
  36.   writeln(unicodestring(WideBuf));
  37.  
  38.  
  39.   { now the same but explicitly resolving the ambiguity                       }
  40.  
  41.   swprintf(WideBuf,
  42.            '%s',
  43.            pwidechar('Hello character world!'));  { picks widechar overload   }
  44.  
  45.   writeln(unicodestring(WideBuf));
  46.  
  47.  
  48.   writeln;
  49.   writeln('press ENTER/RETURN to end this program');
  50.   readln;
  51. end.            
One of the swprintf overloads specifies a pchar parameter and the other a widechar parameter.

When the compiler encounters the call at line 34 (highlighted), there isn't enough information in that statement for the compiler to determine that it should choose the widechar overload (swprintf requires a widechar but that fact isn't part of the overload resolution.) Instead it simply _assumes_ the characters should be single byte characters and picks the pchar overload resulting in output which is not quite as desired.

The compiler should probably emit at least a warning when it encounters a situation like the one above to inform the user that the "chosen overload" is simply a guess.  The other potential problem of making that guess is that the swprintf function and likely all widechar functions expect the string to be terminated with two null widechars and a pchar string is only guaranteed to provide a single null byte opening the door to buffer overruns.

But, the above situation in a GUI program that uses TextOutW instead of writeln shows some really ornate and quite artistic Asian characters. :)  (gotta always look for the silver lining)
FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

PascalDragon

  • Hero Member
  • *****
  • Posts: 3310
  • Compiler Developer
Re: No warning from FPC with ambiguous overload
« Reply #1 on: April 12, 2021, 09:01:25 am »
There is no guess. As long as a string constant does not contain any characters that require a WideChar (and the UnicodeStrings modeswitch is not enabled) then a PAnsiChar is preferred to a PWideChar.

440bx

  • Hero Member
  • *****
  • Posts: 2464
Re: No warning from FPC with ambiguous overload
« Reply #2 on: April 12, 2021, 09:11:09 am »
There is no guess. As long as a string constant does not contain any characters that require a WideChar (and the UnicodeStrings modeswitch is not enabled) then a PAnsiChar is preferred to a PWideChar.
That's fine but, the compiler should let the programmer know by a warning or at least a hint that PAnsiChar is preferred over PWideChar because there are situations like the one above where that "preference" leads to undesirable results.

That preference is just a built-in guess in the compiler.
FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

ccrause

  • Sr. Member
  • ****
  • Posts: 431
Re: No warning from FPC with ambiguous overload
« Reply #3 on: April 12, 2021, 09:29:10 am »
Do you have documentation for the NTdll version of swprintf which mentions that an ansichar pointer parameter is supported?  The Visual Studio 2019 documentation mentions this:
Quote
swprintf is a wide-character version of sprintf; the pointer arguments to swprintf are wide-character strings.
To me it seems as if an overload for an ansichar pointer version is not provided (at least in the standard C runtime).

PascalDragon

  • Hero Member
  • *****
  • Posts: 3310
  • Compiler Developer
Re: No warning from FPC with ambiguous overload
« Reply #4 on: April 12, 2021, 09:33:01 am »
There is no guess. As long as a string constant does not contain any characters that require a WideChar (and the UnicodeStrings modeswitch is not enabled) then a PAnsiChar is preferred to a PWideChar.
That's fine but, the compiler should let the programmer know by a warning or at least a hint that PAnsiChar is preferred over PWideChar because there are situations like the one above where that "preference" leads to undesirable results.

That preference is just a built-in guess in the compiler.

There will not be a warning for this.

440bx

  • Hero Member
  • *****
  • Posts: 2464
Re: No warning from FPC with ambiguous overload
« Reply #5 on: April 12, 2021, 09:39:01 am »
Do you have documentation for the NTdll version of swprintf which mentions that an ansichar pointer parameter is supported?  The Visual Studio 2019 documentation mentions this:
Quote
swprintf is a wide-character version of sprintf; the pointer arguments to swprintf are wide-character strings.
To me it seems as if an overload for an ansichar pointer version is not provided (at least in the standard C runtime).
Yes, I do have documentation for that.  Specifically, sprintf allows a "%S" (capital S) to indicate that the parameter is a null terminated sequence of widechar(s).  Likewise, swprintf allows a "%S" to indicate that the parameter is a null terminated ansi (single byte) sequence of characters.

IOW, both sprintf and swprintf can take either wide character or single character parameters, it only depends on the format specifier.

Also, that capability _is_ present in the standard C runtime library and it is documented (in the supported format specifiers.)

There will not be a warning for this.
That's very unfortunate because there definitely should be one.  The code is absolutely correct yet, it doesn't work because the compiler made a decision behind the programmer's back and, you're saying the compiler shouldn't inform the user of that fact.  When the compiler makes decisions behind the programmer's back, the least it should do is to let the programmer know.
FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

MarkMLl

  • Hero Member
  • *****
  • Posts: 3033
Re: No warning from FPC with ambiguous overload
« Reply #6 on: April 12, 2021, 10:09:06 am »
There will not be a warning for this.
That's very unfortunate because there definitely should be one.  The code is absolutely correct yet, it doesn't work because the compiler made a decision behind the programmer's back and, you're saying the compiler shouldn't inform the user of that fact.  When the compiler makes decisions behind the programmer's back, the least it should do is to let the programmer know.

This is not a democracy and this will not be going to a vote, but I second your expression of discontent.

C++'s entire system of automatic type conversion etc. badly needs tightening up.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

ccrause

  • Sr. Member
  • ****
  • Posts: 431
Re: No warning from FPC with ambiguous overload
« Reply #7 on: April 12, 2021, 10:17:17 am »
Do you have documentation for the NTdll version of swprintf which mentions that an ansichar pointer parameter is supported?  The Visual Studio 2019 documentation mentions this:
Quote
swprintf is a wide-character version of sprintf; the pointer arguments to swprintf are wide-character strings.
To me it seems as if an overload for an ansichar pointer version is not provided (at least in the standard C runtime).
Yes, I do have documentation for that.  Specifically, sprintf allows a "%S" (capital S) to indicate that the parameter is a null terminated sequence of widechar(s).  Likewise, swprintf allows a "%S" to indicate that the parameter is a null terminated ansi (single byte) sequence of characters.

IOW, both sprintf and swprintf can take either wide character or single character parameters, it only depends on the format specifier.

Also, that capability _is_ present in the standard C runtime library and it is documented (in the supported format specifiers.)
Thanks for clarifying what is available.  This means that there isn't an overloaded function in the library/dll, the type of the string pointer is interpreted according to the format string?

To get back the the compiler choosing between ansistring and widestring: the literal string can be converted (in this case) to either ansi or wide string.  With no further information given to the compiler, why do you think it is wrong to pick the most compact and exact representation of the string literal and choose the provided overload that matches this?

Quote
There will not be a warning for this.
That's very unfortunate because there definitely should be one.  The code is absolutely correct yet, it doesn't work because the compiler made a decision behind the programmer's back and, you're saying the compiler shouldn't inform the user of that fact.  When the compiler makes decisions behind the programmer's back, the least it should do is to let the programmer know.
There are potentially numerous implicit type conversions applied by the compiler, if the compiler should mention each one it would be more than useless as it will swamp the compiler messages and hide the really important messages.

Edit: fixed quoting
« Last Edit: April 12, 2021, 10:51:44 am by ccrause »

MarkMLl

  • Hero Member
  • *****
  • Posts: 3033
Re: No warning from FPC with ambiguous overload
« Reply #8 on: April 12, 2021, 10:36:37 am »
There are potentially numerous implicit type conversions applied by the compiler, if the compiler should mention each one it would be more than useless as it will swamp the compiler messages and hide the really important messages.

Except that the one being considered here is the specific case of there being an arbitrary and silent selection of a user-defined function. This is not a simple assignment issue.

OTOH we were told a few days ago that the order user-defined functions were checked was consistent, even if it was implementation-defined. So once one is aware of this being a potential problem one can have assertions testing the compiler's behaviour in a unit's initialisation.

https://forum.lazarus.freepascal.org/index.php/topic,54027.msg400893.html#msg400893

I've been using Quartus-II software for FPGAs a bit over the last few days, and was impressed by the way that its output messages are sorted into separate tabbed pages: an overall one, another for suppressed messages and so on.

MarkMLl
« Last Edit: April 12, 2021, 10:44:22 am by MarkMLl »
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

440bx

  • Hero Member
  • *****
  • Posts: 2464
Re: No warning from FPC with ambiguous overload
« Reply #9 on: April 12, 2021, 10:52:02 am »

MarkMLI posted while I was writing this, he is exactly right, "silent selection" is not a good thing.


Thanks for clarifying what is available.  This means that there isn't an overloaded function in the library/dll, the type of the string pointer is interpreted according to the format string?
You're welcome and, that is correct.  There cannot be an overload because the "..." is generic, it has no associated type.  Their type will be determined/interpreted according to the format specifier(s) (which better be correct or else... )

With no further information given to the compiler, why do you think it is wrong to pick the most compact and exact representation of the string literal and choose the provided overload that matches this?
because the case is ambiguous and whatever decision the compiler makes, may or may not be right.  In those cases, the least the compiler should do is to let the programmer know that it made a decision which is _not_ guaranteed to be correct.  It is the fact that the decision is potentially incorrect that justifies emitting a warning.

There are potentially numerous implicit type conversions applied by the compiler, if the compiler should mention each one it would be more than useless as it will swamp the compiler messages and hide the really important messages.
But normally those conversions are deterministic.  IOW, the compiler can figure out what is unambiguously correct.  Whenever the compiler doesn't know the answer it's supposed to rely on the programmer to determine what is correct.  Compiler "preferences" cannot be used to justify a choice that may or may not be correct.  That's the programmer's job but, the programmer relies on the compiler to tell him/her where it made a decision not based on correctness but on "preference". 

As far as swamps of compiler messages that hide really important messages, anyone who programs directly to the Windows API (I do) gets a swamp of utterly useless messages because FPC decided that "var" parameters should be initialized.  Somewhere in the ballpark of 3/5 of Windows API calls generates one of those useless messages that get in the way of the important ones.  The only workaround, you have to write your own API definitions, otherwise you get swamped.

The bottom line is, loads of completely useless messages but, the one message the programmer _really_ needs to have telling him/her that the compiler made the wrong choice, that one, you don't get.  It's like giving free shampoo to totally bald people and withholding it from long-hair hippies. 
FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

WooBean

  • Full Member
  • ***
  • Posts: 115
Re: No warning from FPC with ambiguous overload
« Reply #10 on: April 12, 2021, 11:14:47 am »
Hi,

I think that situation as is (in the example) doesn't need any action from FPC team.
When editing the source a programer  hit key "(" after "swprintf" and codetools code completion displays available overloaded versions.  This is enough to use the needed one.
Below a small change to code used by TS to illustrate it (forcing pWideChar variant of the overloaded function) .

Code: Pascal  [Select][+][-]
  1. program project3;
  2.  
  3. uses
  4.   Windows
  5.   ;
  6.  
  7. const
  8.   ntdll    = 'ntdll';
  9.   kernel32 = 'kernel32';
  10.  
  11.  
  12. function swprintf({ _out_ } OutWideBuffer    : pwidechar;
  13.                   { _in_  } InFormatString   : pwidechar;
  14.                   { _in_  } InWidechars      : pwidechar)
  15.          : integer; cdecl; external ntdll;                          { CDECL !!}
  16.  
  17. function swprintf({ _out_ } OutWideBuffer    : pwidechar;
  18.                   { _in_  } InFormatString   : pwidechar;
  19.                   { _in_  } InAnsiChars      : pchar)
  20.          : integer; cdecl external ntdll;                           { CDECL !!}
  21.  
  22.  
  23. var
  24.   WideBuf :  packed array[0..511] of widechar;
  25.   WideChars: wideString;
  26. begin
  27.   writeln;
  28.  
  29.   { the compiler should emit a warning stating that there isn't enough        }
  30.   { information to decide which overload to pick.                             }
  31.   WideChars:='Hello unicode world!';
  32.   swprintf(WideBuf,
  33.            '%s',
  34.            pWideChar(@WideChars[1]));   //my 2 cents /WB
  35.            //'Hello character world!');  { compiler picks pchar overload        }
  36.  
  37.   writeln(unicodestring(WideBuf));
  38.  
  39.   { now the same but explicitly resolving the ambiguity                       }
  40.  
  41.   swprintf(WideBuf,
  42.            '%s',
  43.            pwidechar('Hello character world!'));  { picks widechar overload   }
  44.  
  45.   writeln(unicodestring(WideBuf));
  46.  
  47.  
  48.   writeln;
  49.   writeln('press ENTER/RETURN to end this program');
  50.   readln;
  51. end.
  52.  
Win7/64, Lazarus 2.0.8 win64-win64, FPC 3.0.4

MarkMLl

  • Hero Member
  • *****
  • Posts: 3033
Re: No warning from FPC with ambiguous overload
« Reply #11 on: April 12, 2021, 11:20:43 am »
When editing the source a programer  hit key "(" after "swprintf" and codetools code completion displays available overloaded versions.  This is enough to use the needed one.

Codetools is not maintained by the compiler team, so there is absolutely no guarantee that it will make the same choice.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

WooBean

  • Full Member
  • ***
  • Posts: 115
Re: No warning from FPC with ambiguous overload
« Reply #12 on: April 12, 2021, 11:33:29 am »
@MarkMLI

There is no need to establish some kind of cooperation between codetools code completion Authors and FPC core Team - code completion is showing my or anyone programer's code used for function declarations. So, it is responsibility of programmers to prepare proper parameters  to use exactly needed version of the functions.
Win7/64, Lazarus 2.0.8 win64-win64, FPC 3.0.4

440bx

  • Hero Member
  • *****
  • Posts: 2464
Re: No warning from FPC with ambiguous overload
« Reply #13 on: April 12, 2021, 11:35:37 am »
When editing the source a programer  hit key "(" after "swprintf" and codetools code completion displays available overloaded versions.  This is enough to use the needed one.
No, that's incorrect. The selection of the correct overload is determined by the _format specifier_ not by the function parameters.

Specifically, if the format specifier had been "%S" instead of "%s", your change would result in the call not producing the correct result.

The _only_ way the compiler could make the correct determination is by inspecting the format specifier _and_ by knowing that swprintf normally expects a widechar string.  if it did that then, a "%s" would tell the compiler to pass a widestring while a "%S" would tell the compiler to pass a single byte character string and, the reverse if the function had been sprintf instead of swprintf.


it is responsibility of programmers to prepare proper parameters  to use exactly needed version of the functions.
Yes and, it is the compiler's responsibility to inform the programmer of ambiguous choices it made which it has no way of knowing are correct.
FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

MarkMLl

  • Hero Member
  • *****
  • Posts: 3033
Re: No warning from FPC with ambiguous overload
« Reply #14 on: April 12, 2021, 11:40:23 am »
There is no need to establish some kind of cooperation between codetools code completion Authors and FPC core Team - code completion is showing my or anyone programer's code used for function declarations. So, it is responsibility of programmers to prepare proper parameters  to use exactly needed version of the functions.

The functions and parameters are entirely proper, and are accepted by the compiler without error, warning or other comment. That's the whole point of the thread.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

 

TinyPortal © 2005-2018