I changed the way that I'm reading and writing to the stream and now, all processes that want to read/write from the stream go through two central methods which makes debugging easier.
What has been revealed is alarming.
The data that was written to the stream does not always make it to the physical disk intact!!!!
How do I know this? Well, I still had problems, so I implemented a re-read in the write procedure and I used a hex editor to to check the physical file content and on occasion, they are different.
This is a total shambles because you can't trust that the data written to the stream actually goes to the disk (in several cases, I've found that a single byte can be wrong in the buffer that was written), and re-reading for verification is very expensive.
If I detect a failure, I implemented a 250ms sleep, and a flush to disk, and then retry until successful.
On several occasions, when the write fails, it can remain in the while loop for up to 15-20 seconds until it finally succeeds.
iostat -c 2 indicates virtually no wait time.
Please let me know what I can try to try and sort this out.
Kind regards
Zakiwi