Recent

Author Topic: How do I flush a TFileStream to disk  (Read 17782 times)

zakiwi

  • New Member
  • *
  • Posts: 28
How do I flush a TFileStream to disk
« on: June 02, 2010, 10:10:48 pm »
Hi there,

Please could anyone help me. 

I have an application that performs very fast read and write operations to a TFileStream object.  I have been able to prove that the data being written is not always on the disk.  ie. It's on it's way to the disk, but the FileStream.WriteBuffer() method returns before the data is actually on the disk.  This causes problems when my program tries to read that segment of data shortly after it was written, but before it is actually on the disk.

Your help would be greatly appreciated.

Cheers

Zakiwi

typo

  • Hero Member
  • *****
  • Posts: 3051
Re: How do I flush a TFileStream to disk
« Reply #1 on: June 02, 2010, 10:31:53 pm »
Well, you could try to read from the stream instead of reading from file.

zakiwi

  • New Member
  • *
  • Posts: 28
Re: How do I flush a TFileStream to disk
« Reply #2 on: June 02, 2010, 10:38:52 pm »
Sorry, I should have made myself clear.

I am only reading and writing to and from the stream.

theo

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1930

bflm

  • Jr. Member
  • **
  • Posts: 54
    • Free Pascal Random Bits
Re: How do I flush a TFileStream to disk
« Reply #4 on: June 03, 2010, 03:54:00 am »
This causes problems when my program tries to read that segment of data shortly after it was written, but before it is actually on the disk.

IMO it should have nothing to do with flushing/actually writing the data to the disk. To me it sounds more like a bug - either in TFileStream implementation or in file I/O (*) of the OS you're using or in your program. It could be helpful to provide a minimal example demonstrating this behavior to decide where the problem roots from.

(*) AFAIK typical OS caches file I/O a lot but the caching is and must be invisible to the semantics of reading any part of the file *after* being written to at any other point. At least from the perspective of a single process/thread and a local disk volume.

vvzh

  • Jr. Member
  • **
  • Posts: 58
Re: How do I flush a TFileStream to disk
« Reply #5 on: June 03, 2010, 07:48:26 am »
The following solution was suggested on the Russian FPC forum:
Code: [Select]
function FileFlush(Handle: THandle): Boolean;
{$IFDEF MSWINDOWS}
begin
  Result:= FlushFileBuffers(Handle);
end;
{$ELSE}
begin
  Result:= (fpfsync(Handle) = 0);
end;
{$ENDIF}
...
var
  fs: TFileStream;
...
  FileFlush(fs.Handle);

zakiwi

  • New Member
  • *
  • Posts: 28
Re: How do I flush a TFileStream to disk
« Reply #6 on: June 03, 2010, 07:14:03 pm »
Thank you vvzh.  I found that solution as well, but despite flushing to disk using this technique, I am still getting erratic behavior.

Perhaps some detail will help:
I have a set of objects which manage a large B-Tree, which takes a TStream descendant as an input parameter for storage.  If I pass it a TMemoryStream, then the process that I am running works perfectly every time, but when I pass it a TFileStream, I get totally random errors despite the inputs to the process being consistent.  This appears to be due to the data not being correctly saved into the file.  I can't use a memory stream in production as the file that is processed by this process is in excess of 64GB.

The process involves a significant amount of read and write operations, and so I am not too keen to flush the buffers to disk too often as it kills the performance.

Any advice would be appreciated.  What do you make of the MemoryStream vs FileStream issue?

I should have mentioned that I programming on Debian Lenny 64Bit.

Cheers

Zakiwi
« Last Edit: June 03, 2010, 08:42:01 pm by zakiwi »

Chronos

  • Full Member
  • ***
  • Posts: 242
    • PascalClassLibrary
Re: How do I flush a TFileStream to disk
« Reply #7 on: June 03, 2010, 11:25:59 pm »
It sounds interesting. Please provide some example simplified code where behavior on TMemoryStream and TFileStream differs so others can test it and suggest solution.

zakiwi

  • New Member
  • *
  • Posts: 28
Re: How do I flush a TFileStream to disk / Integrity Problems
« Reply #8 on: June 07, 2010, 03:38:57 am »
I changed the way that I'm reading and writing to the stream and now, all processes that want to read/write from the stream go through two central methods which makes debugging easier.

What has been revealed is alarming.

The data that was written to the stream does not always make it to the physical disk intact!!!!

How do I know this?  Well, I still had problems, so I implemented a re-read in the write procedure and I used a hex editor to to check the physical file content and on occasion, they are different.

This is a total shambles because you can't trust that the data written to the stream actually goes to the disk (in several cases, I've found that a single byte can be wrong in the buffer that was written), and re-reading for verification is very expensive.

If I detect a failure, I implemented a 250ms sleep, and a flush to disk, and then retry until successful.

On several occasions, when the write fails, it can remain in the while loop for up to 15-20 seconds until it finally succeeds.

iostat -c 2 indicates virtually no wait time.

Please let me know what I can try to try and sort this out.

Kind regards

Zakiwi

theo

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1930
Re: How do I flush a TFileStream to disk / Integrity Problems
« Reply #9 on: June 07, 2010, 04:10:30 am »
Please let me know what I can try to try and sort this out.

fpc-devel is a better place for this discussion:
http://www.freepascal.org/maillist.var

bflm

  • Jr. Member
  • **
  • Posts: 54
    • Free Pascal Random Bits
Re: How do I flush a TFileStream to disk / Integrity Problems
« Reply #10 on: June 07, 2010, 04:11:15 am »
I changed the way that I'm reading and writing to the stream and now, all processes that want to read/write from the stream go through two central methods which makes debugging easier.

What has been revealed is alarming.

The data that was written to the stream does not always make it to the physical disk intact!!!!


*Please* provide a minimal compilable test/example case for others to verify/reproduce. If it's a FPC bug then it should be reported on Mantis. If it's bug in your code then there is no reason for such alarming claims.

zakiwi

  • New Member
  • *
  • Posts: 28
Re: How do I flush a TFileStream to disk
« Reply #11 on: June 08, 2010, 09:41:45 pm »
Sorry for the delayed reply.

I did as you requested, and wrote a very simple application that proved the concept, and during that process, appear to have stumbled on the solution.

The error only happens when we run the program on our pre-production server.  We've tried running the program on a number of other servers, and it appears to work correctly.

Further test cases are currently running, and I will report back in about a week as to how they have gone, but it looks as though the problem was to do with a faulty hardware controller or something like that.

Thanks for your help.

Cheers

Zakiwi

 

TinyPortal © 2005-2018