I kind of solved my problems by adding an additional storage for the extents.
I use an additional function, which sets the extents, and if necessary swaps them and inverts the scales, but does not touch the additional storage. Still my control happens to crash, but I will dig more trough it.
I will check you demo, but if I understand you right, I'd better use
LogicalExtents[1..4]:=something instead of
FixToo.
But the odd thing is that I use FixTo, my minimum X scale is i.e. 1, but both
LogicalExtemt.coords[1] and
CurrentExtemt.coords[1] return i.e.
3,xyz which in not even close neither to 1, nor to the other border

I have no zoom, my chart margins are zeroes, I have no captions on the plot... I am not sure if I had a plot at all.
The bad things is that code is too big, to reproduce it in a simple way.
So as I mentioned I added an additional buffer for the coordinates and stopped reading logical exents, except when an OnChangeExtents event occurs.