Issue with DateUtils.ScanDateTime and milliseconds

[alpine]

Hello,

I'm using ScanDateTime to convert ISO-8601 date/time to Pascal TDateTime format. The scan pattern is 'yyyy"-"mm"-"dd"T"hh":"nn":"ss"."zzz"Z"'.

Noticed that if the milliseconds part is lesser than 3 digits, it won't get converted correctly. For example:
'2021-04-07T08:49:22.8Z' converts to a value which is lesser than
'2021-04-07T08:49:22.035Z'
That is incorrect.

Quick look into the dateutil.inc makes me think that the millisecond part is just used as a value and not as a fraction, i.e. as in above example .8 was used as 8 millis (8/1000) instead of 8/10 of a second.

Anybody have a workaround for this?

Regards,


 

[alpine]

I don't know about other OSes but in windows the MS is a value not a fraction and its not 0..59 of course

IMHO it is conversion issue and it is not OS dependent. The point is when you write some time string it usually looks like: '23:59:01.123', i.e when it has a milliseconds they are with precision of 3. For example, in Microsoft SQL Server, the precision of a decimal fraction is 3, i.e., "yyyy-mm-ddThh:mm:ss[.mmm]". But unfortunately, I'm receiving data from some web service and getting in a same body many ISO8601 timestamps with milliseconds of different precision. Some are with 1 digit, others with 2 or 3 digits. Then I have noticed the issue into the ScanDateTime().

Actually, I wonder is that a real issue in ScanDateTime(), because in the ISO8601 (and MSSQL datetime) context, the milliseconds are fraction. There may be other strings specifying time where the millis are just a whole number, not a fraction. It depends on a context and it must be specified somehow. May be the FormatDateTime() should include special formatting chars for the fraction, or ISO8601 format chars? 

simply use some string functions to trim the right side and use that value give you a fraction if that is what you want.

Yes, I ended up with a custom conversion routine, quite exact but also ugly.

[MarkMLl]

Quick look into the dateutil.inc makes me think that the millisecond part is just used as a value and not as a fraction, i.e. as in above example .8 was used as 8 millis (8/1000) instead of 8/10 of a second.

I didn't realise that this was a "feature" of input parsing as well as output generation.

I believe that it's justified as "the way the Delphi does it". Irrespective of that it's just plain /wrong/ by any reasonable interpretation, you could try reporting it as a bug but I don't think it would be fixed.

My suggestion is that you do a bit of preprocessing, and explicitly insert one or two zeros before feeding it to the standard parser... which is sufficiently good to be worth using where possible.

MarkMLl

[marcov]

Afaik ISO uses zero padded fields everywhere, and not just for milliseconds. This uniformity in order and width is what makes it useful for machine parse-able timestamps.

No doubt some broken software generates broken dates, but is that really scandatetime's problem?

Note: There is no Delphi angle, since Delphi doesn't have this routine.

[MarkMLl]

Note: There is no Delphi angle, since Delphi doesn't have this routine.

No, but it does have FormatDateTime() which emits unpadded milliseconds and when I've raised this issue in that context I've been told it's because of Delphi compatibility.

MarkMLl

[alpine]

Afaik ISO uses zero padded fields everywhere, and not just for milliseconds. This uniformity in order and width is what makes it useful for machine parse-able timestamps.

You're right, but to some extent. In '5.3.1.3 Representation of decimal fractions' standard says that: 'The number of digits in the decimal fraction shall be determined by the interchange parties, dependent upon the application.' So, the following time is quite valid:

23:20:50,5

i.e. that is a seconds fraction and not a milliseconds.

No doubt some broken software generates broken dates, but is that really scandatetime's problem?

May be not... As long as there is no 'fraction' formatting char.

[wp]

The problem is in the number of "z"s used in the format string. Taken from https://www.freepascal.org/docs-html/rtl/sysutils/formatchars.html:

z
    milliseconds
zzz
    milliseconds(leading zero)

You can see that everything is clear, when the "zzz" case is used because leading zeros will be added, i.e. a time of 8 ms past  the full seconds will be displayed as ".008".

It becomes very confusing when the normal time symbols are combined with the single "z". Now the 8ms past the full second are displayed as ".8".

This is observed for FormatDateTime, but when the resulting string is put into ScanDateTime then the original input date/time value is returned, i.e. FormatDateTime and ScanDateTime behave exactly the same regarding the milliseconds.

Here is a little test program:

Code: Pascal  [Select][+][-]

1. program Project1;
2.  
3. uses
4.   SysUtils, DateUtils;
5.  
6. const
7.   FMT3 = 'yyyy-mm-dd"T"hh:nn:ss.zzz"Z"';
8.   FMT2 = 'yyyy-mm-dd"T"hh:nn:ss.zz"Z"';
9.   FMT1 = 'yyyy-mm-dd"T"hh:nn:ss.z"Z"';
10. var
11.   t, t1: TDateTime;
12.   s: String;
13. begin
14.   WriteLn(' *** 800 ms ***');
15.   t := EncodeDate(2021, 4, 7) + EncodeTime(8, 49, 22, 800);  // 800 ms past 8:49:22
16.   s := FormatDateTime(FMT3, t);
17.   WriteLn(s);
18.   t1 := ScanDateTime(FMT3, S);
19.   if t1 = t then
20.     WriteLn('ScanDateTime returns same date/time value')
21.   else
22.     WriteLn('ScanDateTime returns DIFFERENT date/time value');
23.  
24.   WriteLn;
25.  
26.   t := EncodeDate(2021, 4, 7) + EncodeTime(8, 49, 22, 800);  // 800 ms past 8:49:22
27.   s := FormatDateTime(FMT2, t);
28.   WriteLn(s);
29.   t1 := ScanDateTime(FMT2, S);
30.   if t1 = t then
31.     WriteLn('ScanDateTime returns same date/time value')
32.   else
33.     WriteLn('ScanDateTime returns DIFFERENT date/time value');
34.  
35.   WriteLn;
36.  
37.   t := EncodeDate(2021, 4, 7) + EncodeTime(8, 49, 22, 800);  // 800 ms past 8:49:22
38.   s := FormatDateTime(FMT1, t);
39.   WriteLn(s);
40.   t1 := ScanDateTime(FMT1, S);
41.   if t1 = t then
42.     WriteLn('ScanDateTime returns same date/time value')
43.   else
44.     WriteLn('ScanDateTime returns DIFFERENT date/time value');
45.  
46.   WriteLn;
47.   WriteLn;
48.   WriteLn(' *** 8 ms ***');
49.  
50.   t := EncodeDate(2021, 4, 7) + EncodeTime(8, 49, 22, 8);  // 8 ms past 8:49:22
51.   s := FormatDateTime(FMT3, t);
52.   WriteLn(s);
53.   t1 := ScanDateTime(FMT3, S);
54.   if t1 = t then
55.     WriteLn('ScanDateTime returns same date/time value')
56.   else
57.     WriteLn('ScanDateTime returns DIFFERENT date/time value');
58.  
59.   t := EncodeDate(2021, 4, 7) + EncodeTime(8, 49, 22, 8);  // 8 ms past 8:49:22
60.   s := FormatDateTime(FMT2, t);
61.   WriteLn(s);
62.   t1 := ScanDateTime(FMT2, S);
63.   if t1 = t then
64.     WriteLn('ScanDateTime returns same date/time value')
65.   else
66.     WriteLn('ScanDateTime returns DIFFERENT date/time value');
67.  
68.   WriteLn;
69.  
70.   t := EncodeDate(2021, 4, 7) + EncodeTime(8, 49, 22, 8);  // 8 ms past 8:49:22
71.   s := FormatDateTime(FMT1, t);
72.   WriteLn(s);
73.   t1 := ScanDateTime(FMT1, S);
74.   if t1 = t then
75.     WriteLn('ScanDateTime returns same date/time value')
76.   else
77.     WriteLn('ScanDateTime returns DIFFERENT date/time value');
78.  
79.  
80.   ReadLn;
81. end.
82.  

Lessons learned:

• When the milliseconds are meant to a fraction of a second the three-z version of millisecond format must be used: 8:49:22.008.
• But when the milliseconds are given as a separate quantity the single-z version is preferred, e.g. "elapsed time: 8 hours, 49 minutes, 22 seconds, 8 milliseconds".
• A "two-z" version is automatically replaced by the "three-z" verison.

[alpine]

This is observed for FormatDateTime, but when the resulting string is put into ScanDateTime then the original input date/time value is returned, i.e. FormatDateTime and ScanDateTime behave exactly the same regarding the milliseconds.

I humbly disagree with that.

Consider that snippet (and also notice the absence of the double-quote at the end of the pattern):

Code: Pascal  [Select][+][-]

1.   WriteLn( // Output: 2021-04-07T08:49:22.008Z
2.     FormatDateTime('yyyy"-"mm"-"dd"T"hh":"nn":"ss"."zzz"Z',
3.     ScanDateTime(  'yyyy"-"mm"-"dd"T"hh":"nn":"ss"."zzz"Z', '2021-04-07T08:49:22.8Z')));

It is not exact (actually, not exactly opposite) behavior.

Another quirk in the ScanDateTime() - closing the double-quote after the last Z gives an exception. The scan of the argument finishes but the pattern has one more character, the closing quote. Did you tried your own example?

Lessons learned:

• When the milliseconds are meant to a fraction of a second the three-z version of millisecond format must be used: 8:49:22.008.

The lesson was learned, but unfortunately it turned out to be incorrect.

[wp]

Consider that snippet (and also notice the absence of the double-quote at the end of the pattern):

Code: Pascal  [Select][+][-]

1.   WriteLn( // Output: 2021-04-07T08:49:22.008Z
2.     FormatDateTime('yyyy"-"mm"-"dd"T"hh":"nn":"ss"."zzz"Z',
3.     ScanDateTime(  'yyyy"-"mm"-"dd"T"hh":"nn":"ss"."zzz"Z', '2021-04-07T08:49:22.8Z')));

It is not exact (actually, not exactly opposite) behavior.

I see. My test did not cover the case in which the input data are wrong from a mathematical point of view. So, there is another lesson learned (by myself):

• Be very careful when strings with 1- or 2-digit milliseconds are to be converted

Another quirk in the ScanDateTime() - closing the double-quote after the last Z gives an exception. The scan of the argument finishes but the pattern has one more character, the closing quote. Did you tried your own example?

Of course, Laz trunk/FPC 3.2.0, and it works. It does not work with FPC 3.0.4, though, which crashes as reported by you.

[MarkMLl]

• Be very careful when strings with 1- or 2-digit milliseconds are to be converted

Very /very/ careful. I think the problem is the casual use of . as a separator while common sense would imply that should be interpreted as a decimal.

Of course, there are other examples of this questionable practice, including dotted-quad IP addresses, common major.minor.revision numbering, and just about anything influenced by Ted Nelson's Xanadu/Udanax etc.

MarkMLl

[alpine]

I see. My test did not cover the case in which the input data are wrong from a mathematical point of view. So, there is another lesson learned (by myself):

• Be very careful when strings with 1- or 2-digit milliseconds are to be converted

Sorry being so insistent, but there's no mathematical error, nor non-compliance with ISO8601 in the '2021-04-07T08:49:22.8Z' - the digits after the decimal point (it can be also comma in the standard) are not a milliseconds, it is a fraction of a second. In a special case of 3-digits it is a milliseconds count, resp. if there are 2 digits that is a hundreds of a second or tenths of a second if a single digit was given. The 'zzz' pattern treats the digits as a whole number of millis and not like a fraction of second, x*(1/10^n).

My wrong at first place, thinking that 'zzz' in ScanDateTime() can do the work. I was mostly mislead by MS-SQL datetime formats (those are very similar with unqualified ISOs but with 3-digits sharp==milliseconds). Recently, I've dug into the ISO texts and found that 'fraction' thing. Lesson: RFM! 

Of course, Laz trunk/FPC 3.2.0, and it works. It does not work with FPC 3.0.4, though, which crashes as reported by you.

Sorry again, my FPC is 3.1.1, I'm not a very frequent updater. Searched through the bugtracker but found nothing about ScanDateTime() recently.

[wp]

Sorry being so insistent, but there's no mathematical error, nor non-compliance with ISO8601 in the '2021-04-07T08:49:22.8Z' - the digits after the decimal point (it can be also comma in the standard) are not a milliseconds, it is a fraction of a second. In a special case of 3-digits it is a milliseconds count, resp. if there are 2 digits that is a hundreds of a second or tenths of a second if a single digit was given. The 'zzz' pattern treats the digits as a whole number of millis and not like a fraction of second, x*(1/10^n).

No, when the 'zzz' pattern is the whole number of milliseconds - which I agree on - then it IS a fraction of a second, namely the number of thousandths. So, all this is consistent.

The problem is when an input string says "...22.8" and silently assumes that there are 22 full seconds plus 8 milliseconds. Since the input string seems to use a decimal separator here this number is nonsense, it is mathematically incorrect, the correct value would be 22.008. Of course there could be the possibility that the dot means something else, as MarkLML suggests. But still it is misleading to anybody not knowing that the usual decimal notation does not apply here.  Why can't these guys format their string as "22s 8ms"? Unfortunately FPC supports this nonsense because it allows the single-z format string with the decimal separator hh:nn:ss.z

[alpine]

The problem is when an input string says "...22.8" and silently assumes that there are 22 full seconds plus 8 milliseconds. Since the input string seems to use a decimal separator here this number is nonsense, it is mathematically incorrect, the correct value would be 22.008.

Forgive me, may be I'm not a native English speaker, and thus my writing is not as clear as I wish, but, once again - '22.8' means exactly 22.8 seconds, 22 + 8/10 of a second, which also means 22 seconds + 800 milliseconds. '22.8' represents mathematically correct value and also ISO8601 compliant value for seconds with a fractional part. Same is for '22.08' - it is 22 + 8/100 seconds, '22.008' is 22 + 8/1000 and only in this case (with 3 digits) it is equivalent to milliseconds.

And the z-format is supporting milliseconds (only the last case), which is obviously unsuitable for the general case of fraction in the mathematical sense. I have already agreed on that. The thing I do not agree is that 22.8 is a nonsense. It is clearly stated in the standard and it is mathematically sound.

Regards,

[MarkMLl]

@wp, @alpinistbg: I don't think you're disagreeing with each other.

I think we have a consensus that 22.8 *should* be interpreted as 22.800 not 22.008

MarkMLl

[winni]

@wp, @alpinistbg: I don't think you're disagreeing with each other.

I think we have a consensus that 22.8 *should* be interpreted as 22.800 not 22.008

MarkMLl

And everybody who disagrees should go back to school.

Shaking the head.

Winni