Precision and jitter on Sysutils.Now().

[MarkMLl]

I have a very simple console program which is monitoring short messages generated every second by an Arduino, arriving over a USB serial port. FPC 3.0.4, 64-bit Linux on x86_64 with KDE desktop.

In the program I have this function:

Code: [Select]

(* This is an experimental replacement for the RTL's Now() based on the POSIX
  clock if available. I've not gone to the trouble of correcting the date,
  assume that as it stands it is only good for relative times.
*)
function posixSecs(): TDateTime;

var     ts: timespec;
        days: extended;

begin
 if clock_gettime(CLOCK_REALTIME, @ts) = 0 then begin
    days := ts.tv_sec / SecsPerDay;
    days += ts.tv_nsec / SecsPerDay / 10E9;
    result := days * SecsPerDay
  end else
    result := (Sysutils.Now() - UnixEpoch) * SecsPerDay
end { posixSecs } ;

If I comment out the part that uses clock_gettime() so that Now() is being used, I see this sequence of minute lengths:

1582918852.627 01:30:00     ............................................................ 59.980       5458.230
1582918912.607 01:31:00     ............................................................ 59.979       5518.209
1582918972.586 01:32:00     ............................................................ 59.979       5578.188
1582919032.565 01:33:00     ............................................................ 59.983       5638.171
1582919092.548 01:34:00     ............................................................ 59.979       5698.150

If I revert to using clock_gettime(), with no other changes to the program or on the system, I see this sequence:

1582919226.035 00:00:00     ............................................................ 59.998         59.998
1582919286.033 00:01:00     ............................................................ 59.998        119.996
1582919346.031 00:02:00     ............................................................ 59.998        179.994
1582919406.029 00:03:00     ............................................................ 59.998        239.992
1582919466.027 00:04:00     ............................................................ 59.998        299.990

This is reproducible. Would anybody like to speculate as to what's going on?

MarkMLl

[lucamar]

It might sound far fetched (and I'm not all sure about it) but, may it be an artifact of floating point operations?

[MarkMLl]

Quite possibly. I was looking at the overall implementation last weekend (because another of my timing functions wasn't working properly and I wanted to know why) and it did seem rather baroque.

I'm trying to run comparisons of various timing sources, including GPS pulse-per-second and the high-precision pulses from 60kHz MSF. Having this amount of disparity on the PC doesn't help, although I would say that on occasion my own code sees

1582920246.003 00:17:00     ............................................................ 59.998       1079.966
1582920306.001 00:18:00     ............................................................ 59.098       1139.064
1582920365.099 00:19:00     ............................................................ 59.998       1199.062

which I haven't investigated in detail yet. If I add a few more digits to the output I see

1582922199.019 00:00:00     ............................................................ 59.998297         59.998
1582922259.018 00:01:00     ............................................................ 59.997993        119.996
1582922319.016 00:02:00     ............................................................ 59.998305        179.995
1582922379.014 00:03:00     ............................................................ 59.998392        239.993
1582922439.012 00:04:00     ............................................................ 59.998304        299.991

which I'm happy to attribute to the attached Arduino, since I know that its own program will have jtter of the order of 88 uSec.

MarkMLl

[MarkMLl]

My original function had a spurious multiply/divide which I've factored out, removing that added a couple of bits of jitter to the mSec digit. If I change the return type to extended (noting the portability aspect), I get this for a direct call to clock_gettime():

1582965605.030945 00:06:00     ............................................................ 59.997588        419.984731
1582965665.028534 00:07:00     ............................................................ 59.997863        479.982594
1582965725.026396 00:08:00     ............................................................ 59.997546        539.980140
1582965785.023943 00:09:00     ............................................................ 59.998314        599.978454
1582965845.022257 00:10:00     ............................................................ 59.998353        659.976807

and this using Now():

1582965918.531000 00:00:00     ............................................................ 59.986000         59.983000
1582965978.517000 00:01:00     ............................................................ 59.980000        119.963000
1582966038.497000 00:02:00     ............................................................ 59.980000        179.943000
1582966098.477000 00:03:00     ............................................................ 59.978000        239.921000
1582966158.455000 00:04:00     ............................................................ 59.982000        299.903000

I ran last night with NTP clock updates disabled, and was still seeing the occasional glitch. When I re-enabled it, I saw the time adjust over the course of roughly a minute as expected.

MarkMLl

[MarkMLl]

There was an out-by-one error in the /exponent/ in my original code, which was both messing up my precision and causing odd behaviour when there was a carry. I've also looked carefully at some of the behaviour during caculations, and am now using the extended type for all intermediate values.

Getting the times from the OS now looks like

Code: [Select]

1582999549.586098 01:14:00     ............................................................ 60.398046       4529.810655
1582999609.984144 01:15:00     ............................................................ 60.398121       4590.208777
1582999670.382265 01:16:00     ............................................................ 60.398178       4650.606954
1582999730.780443 01:17:00     ............................................................ 60.398104       4711.005058
1582999791.178547 01:18:00     ............................................................ 60.397130       4771.402188

Using Sysutils.Now():

Code: [Select]

1582999866.222000 00:00:00     ............................................................ 60.397000         60.397000
1582999926.619000 00:01:00     ............................................................ 60.397000        120.794000
1582999987.016000 00:02:00     ............................................................ 60.398000        181.192000
1583000047.414000 00:03:00     ............................................................ 60.398000        241.590000
1583000107.812000 00:04:00     ............................................................ 60.397000        301.987000

Again, with no other changes except for using the different timing source. So Now() still has more jitter and less precision, but they're broadly compatible.

MarkMLl

[lucamar]

I think most of it is caused because the floating point calcs and the relatively long series of calls through which Now goes to get the date/time.

Just curious: What happens if you use the general function in Unix instead of the Linux "special"?

[MarkMLl]

I think most of it is caused because the floating point calcs and the relatively long series of calls through which Now goes to get the date/time.

Just curious: What happens if you use the general function in Unix instead of the Linux "special"?

There were- as usual- two different things. The first was that in my function (head of this thread) I was dividing nSec by 10E9 rather than 1.0E9, that was causing both a disparity between the two ways of getting the time and occasional carry errors... I blush to admit how much difficulty I had finding that.

The second is a loss of precision as a timestamp progresses through the Now() function, I'm fairly confident that the low-level unix/Linux API is working properly.

I had a lot of hassle, even with 80-bit extended reals, with the order of some of the calculations.

One of the things I've revisited today is the resolution and what I call the "granularity" of the low-level RTC API. The particular Linux 4 kernel I'm running (which might have non-default configuration from Debian) claims a resolution of 1 nSec, but actually testing it shows a granularity of about 420 nSec.

Anyway, for the moment I'm back on comparing the timekeeping of this particular Arduino with what I'm getting from 60kHz MSF... the match is nowhere near as good as I'd like.

-----

Much later: the main chip, i.e. including timers etc., on an Arduino usually uses a resonator rather than a crystal, so the timing accuracy is in the % rather than the ppm range. I'm trying to characterise that.

A particular Arduino tells me that its idea of an hour is 3623.943 seconds. If I hook it onto an MSF receiver I should in principle have an atomic clock reference, I can see falling edges representing the start of seconds and the Arduino tells me that it thinks they're 993497 uSec apart. 3600/0.993497 = 3623.564 which is close enough to the other figure. I'd expect a PPS signal from a GPS receiver to work the same, although I trust MSF more. I've not started exploring variations against temperature.

I can't do this automatically yet, since I don't at present have a good external antenna mount and the signals are sufficiently bad that I'm having to be a bit selective. But this does demonstrate that it's possible to get at least one tertiary standard without having to go to a calibration lab.

MarkMLl

[munair]

It may help if you round off the DateTime value to the precision you require. For example, my calendar library supports precision up to 10s of a second, but I won't get a stable conversion between Julian dates and Gregorian dates unless the decimal part is rounded up by 5 at the right digit:

Code: Pascal  [Select][+][-]

1. HourSerial := (JD + 0.5 - JD0) + 0.00000005;

In the world of computers decimals are approximations by definition because the base-2 number system cannot represent them accurately.

[MarkMLl]

It may help if you round off the DateTime value to the precision you require. For example, my calendar library supports precision up to 10s of a second, but I won't get a stable conversion between Julian dates and Gregorian dates unless the decimal part is rounded up by 5 at the right digit:

Code: Pascal  [Select][+][-]

1. HourSerial := (JD + 0.5 - JD0) + 0.00000005;

In the world of computers decimals are approximations by definition because the base-2 number system cannot represent them accurately.

I fail to see how rounding would help. I'd like sub-mSec resolution, in general modern computers provide roughly uSec. Now() isn't up to it, and since TDateTime is represented internally as a floating-point number its precision has got worse the further we have progressed from its epoch.

Allowing for the ubiquity of Now() and TDateTime, if I were fixing this at the RTL level I'd add a parameter to Now() to allow the epoch to be changed and TZ to be disabled, which would restore the usefulness of TDateTime comparisons which at present are screwed by the increasing number of bits needed to represent the epoch-relative day.

MarkMLl

[Thaddy]

Anyway, if you want high resolution precision, don 't use a PC for timing purposes (they are useless for that) , use a dedicated clock source in hardware.
What would help a little if you set the thread/process affinity of the timing thread to one single dedicated core, but basically a dedicated clock with uSec/pSec resolution is pennies. You still have to account for your converters, though.

[MarkMLl]

But if you do have something with pSec resolution, you need access to something like MSF (60kHz phase-locked to an atomic clock, with an overlaid timecode) to demonstrate confidence in it.

MarkMLl