TDateTime to number

[Remy Lebeau]

Are there msecs in ISO 8601 ? (i was unable to locate)

Yes.  Technically, ISO 8601 times only have hours, minutes and seconds components, but the last component used can be expressed as a decimal fraction, thus milliseconds can be expressed as a decimal in the seconds component.

[Edson]

I need to compact a TDateTime to a string, before to save to a file.

Why not just encode the TDateTime in a standard ISO 8601 format?

Too large string. I have converted the YYYY/MM/DD HH:NN:SS to a string, using only 8 chars or less.

[molly]

Too large string. I have converted the YYYY/MM/DD HH:NN:SS to a string, using only 8 chars or less.

Now you confuses me.   

1 You require to convert back n forth from tdatetime to whatever format you're using.
2 You 'complained' about Unixdate time functions not being able to do that (bug as we've experienced but, also related to unixdate/time resolution).
3 from the above quote i'm able to extract/conclude that you do not store millisecs.

So, please enlighten. How do you use TDateTime with f.e. function now (using millisecond precision), and be able to convert that to your storage format and back again so they match up ?

[Remy Lebeau]

Too large string. I have converted the YYYY/MM/DD HH:NN:SS to a string, using only 8 chars or less.

ISO 8601 can be condensed in its representation, but it can't be condensed that small unless you omit components.

So, let's go back to the discussion of converting the TDateTime to an integer offset from the Unix epoch (January 1 1970 00:00:00 GMT).

The current date/time (October 5 2016 23:03:26 GMT) as a seconds-precision offset uses 10 decimal digits (1475708606), and a milliseconds-precision offset uses 13 decimal digits (1475708606043).  Neither will fit your 8-char restriction if you encode the offset as a decimal string.

However, if you stored the offset as binary data, a 32bit seconds-precision offset (which will overflow in the year 2038 if you use a signed integer, and in the year 2106 if you use an unsigned integer) can be stored in 4 bytes, and a 64bit milliseconds-precision offset can be stored in 8 bytes.

Alternatively, a 32bit seconds-precision offset can be encoded as an 8-char hex string (57F586BE).  But then you lose milliseconds precision, which would require an 11-char hex string (1579716565B).

If you were to encode the offset value as a packed Binary-Coded Decimal, a 13-digit milliseconds-precision offset would fit in 7 bytes (0x01 0x47 0x57 0x08 0x60 0x60 0x43) (a 16-digit offset would fit in 8 bytes).

Or, you could do what I did in one app, and bit-stuff the components of a complete date/time (with milliseconds and even UTC offset!) into 8 bytes (with bits to spare!). Here is the bit layout I used.

Either way, if you have the date/time in binary format and need to store it as human-readable text, you could just map the binary bytes to an ASCII alphabet, and map them back during decoding.

[Edson]

1 You require to convert back n forth from tdatetime to whatever format you're using.
2 You 'complained' about Unixdate time functions not being able to do that (bug as we've experienced but, also related to unixdate/time resolution).
3 from the above quote i'm able to extract/conclude that you do not store millisecs.

So, please enlighten. How do you use TDateTime with f.e. function now (using millisecond precision), and be able to convert that to your storage format and back again so they match up ?

I just asked for a function (and the reverse function) to convert TDateTime to/from number, like DateTimeToUnix()/UnixToDateTime() do (with no miliseconds). One inconvenient was DateTimeToUnix()/UnixToDateTime() don't work properly.
   
I needed to obtain a number from TDateTime (with no miliseconds) to save this to a file in a compact string, because I need to save many dates.
My idea was simple. Convert a TDateTime (with no miliseconds) to a number, and save this number as a string using hexadecimal format. That's how I can convert a TDateTime (with no miliseconds) to a 8 char string.

[Edson]

So, let's go back to the discussion of converting the TDateTime to an integer offset from the Unix epoch (January 1 1970 00:00:00 GMT).

As I have said (may times  ), I don't need to store miliseconds.

In this case, according to the definition of TDateTime, it start counting from 1900 (exactly 30 Dec 1899). So to the current day, it's 42648 days. Then to represent this date like a number, we can use:

days*86400 + hours*3600 + minutes*60 + seconds

that is some like 3684787200, and DBA16400 in hexa (8 chars). Of course, it can be compressed more using more alphabetic codes. And of course, it will be needed 9 hexa digits in 2036 year.

[J-G]

I've been following this thread with interest since in days gone by I needed to be very frugal with data storage and dates were an area of concern. in the '80s I held dates in 3 bytes (I didn't need to store time) and never had any issue with the 'Millenium Bug' 

Out of interest I've written code to hold date and time - including milliseconds to 2 decimal places - from Year 0000 to 9999 - in 8 bytes. It's not 'pretty' and would need a little extra to add leading zeros to numbers below 10 but it works forward and reverse.

Without milliseconds it only needs 7 bytes.

Code: Pascal  [Select][+][-]

1. Type
2.    date = record
3.             d,m : byte;
4.             y   : integer;
5.            End;
6.     time = record
7.              h,m : byte;
8.              s   : real;
9.            end;
10. Var
11.   DT_String  : string[8];
12.   Today        : date;
13.   Time_Now  : time;
14.  
15. Const
16.   offset = 32;  // this could be anything between 32 and 156  -  
17.                      // below 32 is an issue with control characters
18.                      // above 156 would cause 99 in the year or m-sec to go above 255
19.  
20. Procedure Format_DT(D : Date;T : Time);         // This is just for my testing
21. Var DT,S : String;
22. begin
23.   str(Today.y:4,DT);       DT := DT + '-';
24.   str(Today.m:2,S);        DT := DT + S + '-';
25.   str(Today.d:2,S);        DT := DT + S + ' ';
26.   str(Time_Now.h:2,S);     DT := DT + S + ':';
27.   str(Time_Now.m:2,S);     DT := DT + S + ':';
28.   str(Time_Now.s:5:2,S);   DT := DT + S;
29.   J_Day.VE_Date.caption := DT;
30. end;
31.  
32. Procedure Decode_DT(S : string);
33. begin
34.   With Today do
35.     begin
36.       y := (trunc(ord(S[1]) - offset) * 100) + (Ord(S[2]) - offset);
37.       m := ord(s[3]) - offset;
38.       d := ord(s[4]) - offset;
39.     end;
40.   Time_Now.h := ord(S[5]) - offset;
41.   Time_Now.m := ord(S[6]) - offset;
42.   Time_Now.s := ord(S[7]) - offset + ((ord(S[8]) - offset) / 100);
43. end;
44.  
45. procedure AddToString(ch : char);
46. begin
47.   DT_String := DT_String+ch;
48. end;
49.  
50. Procedure Encode_DT(D : Date;T : Time);
51. Var ch : char;
52. begin
53.   DT_String := '';
54.   ch := char(trunc(D.y / 100) + offset);         // Century 0 - 99     (32 -131)
55.   AddToString(ch);
56.   ch := char(D.y MOD 100 + offset);              // Year in Cent 0 - 99   (32 -131)
57.   AddToString(ch);
58.   ch := char(D.m + offset);                      // Month in year 1 - 12  (33 - 44)
59.   AddToString(ch);
60.   ch := char(D.d + offset);                      // Day in Month 1 - 31   (33 - 63)
61.   AddToString(ch);
62.   ch := char(T.h + offset);                      // Hour in Day 0 - 23    (32 - 55)
63.   AddToString(ch);
64.   ch := char(T.m + offset);                      // minute in Hour 0 - 59 (32 - 91)
65.   AddToString(ch);
66.   ch := char(Trunc(T.s) + offset);               // seconds in minute 0 - 59  (32 - 91)
67.   AddToString(ch);
68.   ch := char(trunc((T.s - Trunc(T.s)) * 100) + offset);   // millisec 0 - 99  (32 -131)
69.   AddToString(ch);
70. end;
71.  
72. procedure TJ_Day.Encode_ButClick(Sender: TObject);
73. begin
74.   Encode_DT(Today,Time_Now);
75.   WS_Date.caption := DT_String;                              // I just happen to have a TLabel on the form I'm testing
76. end;
77. procedure TJ_Day.Decode_ButClick(Sender: TObject);
78. begin
79.   Decode_DT(DT_String);
80.   Format_DT(Today,Time_Now);
81. end;
82.  

[Today] and [Time_Now] naturally need to be pre-qualified of course. I've used separate entities because that's more convenient in the program I'm working on but a TDateTime could be used equally easily.

[molly]

@Edon:
Thank you for the explanation.

I just asked for a function (and the reverse function) to convert TDateTime to/from number, like DateTimeToUnix()/UnixToDateTime() do (with no miliseconds). One inconvenient was DateTimeToUnix()/UnixToDateTime() don't work properly.

The 'bug' was indeed very inconvenient and caused some confusion along the way.

Note that i indicated 'bug', as it isn't actually a bug but simply a wrong assumption on your end (even though these FPC conversion routines do not act similar to Delphi).

The original example (in reply #2) that you started out with could never have worked as expected because of the now function usage.

If you 'store' a date/time-value inside a TDateTime typed variable, a function like Now() will store the number of milliseconds in there. Converting that TDateTime value into something that cuts-off/truncates the number of milliseconds, it is not possible to reverse and match that with the original TDateTime value again.

Even though in your reply #4 you stated to have used another solution, it was never clear that your code in reply #2 started working.

And with good reason because it never could have !

And in case you are telling yourself that it did, then please verify because it only does so on first sight and as such fools you on the output. Use CompareDateTime function to check.

Hence the confusion about the requirements. Sorry for that.

edit:
And to show how you get fooled, a bit more obvious example:

Code: Pascal  [Select][+][-]

1. program foolme;
2.  
3. {$MODE OBJFPC}{$H+}
4.  
5. Uses
6.  SysUtils, DateUtils;
7.  
8. function DT2Number(const dt: TDateTime): Int64;
9. var
10.   hh, nn, ss, MilliSecond: word;
11. begin
12.   DecodeTime(dt, hh, nn, ss, MilliSecond);
13.   Result := trunc(dt)*86400 + hh * 3600 + nn * 60 + ss;
14. end;
15.  
16. function Number2DT(n: Int64): TDateTime;
17. var
18.   day, hh, nn, ss: Int64;
19. begin
20.   day := n div 86400;
21.   n := n mod 86400;
22.   hh := n div 3600;
23.   n := n mod 3600;
24.   nn := n div 60;
25.   ss := n mod 60;
26.   Result := EncodeTime(hh,nn,ss,0) + day;
27. end;
28.  
29. procedure TestDateFooled;
30. var
31.    fec1: TDateTime;
32.    fec2: TDateTime;
33.    n  : int64;
34.    s  : String;
35. begin
36.    fec1 := now;
37.  
38.    s := FormatDateTime('dd/mm/yyyy', fec1);
39.    n := DT2Number(fec1);
40.    writeln(s, ' = ', n);
41.    fec2 := incHour(Number2DT(n), 1);  // Note i get it one hour off
42.    s := FormatDateTime('dd/mm/yyyy', fec2);
43.    writeln(s, ' = ', n);  // yet both dates look exactly similar
44.    WriteLn('relationship = ', CompareDateTime(fec1, fec2));  // Actually they are not
45.    writeln('');
46. end;
47.  
48. var
49.   i: Integer;
50.  
51. begin
52.   for i := 1 to 10 do
53.   TestDateFooled;
54. end.
55.  

[Fungus]

I've read most of this thread and I'm wondering; Is the TDateTime required to be represented as a numeric string value? Why not convert it to hexadecimal or base64 string and preserve 100% accuracy?

[Thaddy]

Once the unix versions get fixed, I think Edson will be happy enough.
He spotted that bug and he is right that atm it is unusable for his purpose.

But that *should* have been the right solution. And I will trust it will be in the very near future.
(There's also the matter of standards to consider as Remy pointed out. The unix  routines are designed to conform to these standards)

[Edson]

I've written code to hold date and time - including milliseconds to 2 decimal places - from Year 0000 to 9999 - in 8 bytes. It's not 'pretty' and would need a little extra to add leading zeros to numbers below 10 but it works forward and reverse.

Great  . That's exactly what I wanted to do. Although I would be worried about coding using some special chars like chr(32), chr(34) and chr(44), because they are used as delimiters in some cases.

[Edson]

Why not convert it to hexadecimal or base64 string and preserve 100% accuracy?

They will work, but are long strings.

[Edson]

Note that i indicated 'bug', as it isn't actually a bug but simply a wrong assumption on your end (even though these FPC conversion routines do not act similar to Delphi).

I have never said it's a "bug".  (maybe I really think it's, but never said  ).
It's because I don't know how these functions are supposed to work (round or not to round).

If you 'store' a date/time-value inside a TDateTime typed variable, a function like Now() will store the number of milliseconds in there. Converting that TDateTime value into something that cuts-off/truncates the number of milliseconds, it is not possible to reverse and match that with the original TDateTime value again.

I know this. I know how the date-times are stored at low level. I know what DateTimeToStr shows, is not the real bits of the TDateTime.

This really make think of a philosophical question: What is the real date-time in your application? All the 64bits? And my philosophical answer is: No. It's what the user see in the screen or in the paper printed.

An for my App, this only go until seconds  .

Coming to the point, like I stated "it's deserable DateTimeToUnix() must works the same way DateTimeToStr() works, and in general, every Date-Time function in Lazarus."

But I repeat "I don't know how these functions are supposed to work."

And by the way, my functions (DT2Number and Number2DT) works well, until seconds  . I suspect they will fail at some date like "1/1/2016 12:00:05.99999", but not tested. If someone can show me one case, I will be thankful.

[molly]

I know this. I know how the date-times are stored at low level. I know what DateTimeToStr shows, is not the real bits of the TDateTime.

Yes, but that is what worries me as your original code did not take this into account (and i doubt your current solution does).

Luckily for us because otherwise the 'bug' with unix DT conversion function would never have surfaced 

No. It's what the user see in the screen or in the paper printed.

And that is where imho you are a bit wrong.

Yes for the end-user this might be true but, for computers and their infinite wisdom to express these kind of things in an exact matter you would have to compensate for that. Well, at least for your initial example code that showed the error.

DateTimeToStr is not aware of your used precision (no matter what functions used), and dumbly truncates the part you do not wish to show. You would have to do the same trunctation for your used DT values in case you wish to compare them.

Coming to the point, like I stated "it's deserable DateTimeToUnix() must works the same way DateTimeToStr() works, and in general, every Date-Time function in Lazarus."

This remark is also an extra indication to me that you still do not seem to understand what is the culprit with your original code, even though you stated that you know that DateTimeToStr is not telling you the truth (and only shows what you want it to show).

The output of your original example using unix conversion functions (still) does not make any sense if i would have configured in my locale options that DateTimeToStr() by default needs to print milliseconds. See where i'm getting at ?

Let me illustrate how to fix your initial example code (We do not need a bug-fix as we made the wrong assumption to begin with).

Code: Pascal  [Select][+][-]

1. program corrected;
2.  
3. {$MODE OBJFPC}{$H+}
4.  
5. uses
6.   SysUtils, Dateutils;
7.  
8. function CorrectDT(DT: TDateTime): TDateTime;
9. var
10.   Year, Month, Day, Hour, Minute, Second, MilliSecond: Word;
11. begin
12.   DecodeDateTime(DT, Year, Month, Day, Hour, Minute, Second, MilliSecond);
13.   MilliSecond := 0;
14.   Result := EncodeDateTime(Year, Month, Day, Hour, Minute, Second, MilliSecond);
15. end;
16.  
17. procedure TestDate;
18. var
19.   fec1: TDateTime;
20.   fec2: TDateTime;
21.   feca1: int64 absolute fec1;
22.   feca2: int64 absolute fec2;
23.  
24.   n: LongInt;
25. begin
26.   fec1 := now;
27.   fec1 := CorrectDT(fec1);
28.  
29.   n := DateTimeToUnix(fec1);
30.   writeln(DateTimeToStr(fec1), ' = ', n, ' (',feca1,')');
31.   fec2 := UnixToDateTime(n);
32.   writeln(DateTimeToStr(fec2), ' = ', n, ' (',feca2,')');
33.   WriteLn('relationship = ', CompareDateTime(fec1, fec2));
34.   writeln;
35. end;
36.  
37. var
38.   i : Integer;
39. begin
40.   for i := 1 to 10 do
41.   begin
42.     TestDate;
43.     sleep(1000);
44.   end;
45. end.
46.  

Quite simple, isn't it ?

You simply have to make sure the resolution matches. Unix epoch format does not know anything about milliseconds so, we make sure we are not bothered by them to begin with ;-)

Sorry for being persistent but, if you do not fully realize what the culprit is/was to begin with then you will end up with this kind of confusion the rest of your programming life (and we can't have that  )

[Edson]

Let me illustrate how to fix your initial example code (We do not need a bug-fix as we made the wrong assumption to begin with).

Code: Pascal  [Select][+][-]

1. function CorrectDT(DT: TDateTime): TDateTime;
2. var
3.   Year, Month, Day, Hour, Minute, Second, MilliSecond: Word;
4. begin
5.   DecodeDateTime(DT, Year, Month, Day, Hour, Minute, Second, MilliSecond);
6.   MilliSecond := 0;
7.   Result := EncodeDateTime(Year, Month, Day, Hour, Minute, Second, MilliSecond);
8. end;
9.  

No need to ilustrate me. That's exactly what my function do:

Code: Pascal  [Select][+][-]

1. function DT2Number(const dt: TDateTime): Int64;
2.   var
3.     hh, nn, ss, MilliSecond: word;
4.   begin
5.     DecodeTime(dt, hh, nn, ss, MilliSecond);
6.     Result := trunc(dt)*86400 + hh * 3600 + nn * 60 + ss;
7.   end;
8.  

Truncate the Miliseconds.

I have the impression you have the impression I ignore this simple solution. 

Thanks anyway for being collaborative.