[SOLVED] How to wait until a WebDAV-Upload w. Cache is really completed? (Linux)

[Lutz Mändle]

Please, what would it help for my WebDAV problem to know, whether fpfsync would wait on a NFS share?

It's about abstraction, fpFsync does the work of flushing the file buffers not itself, this task is done by the underlying filesystem and its drivers. Therefore, if there is a problem, first I should determine whether a more common non local filesystem is working as expected.

I have done this and can say that with a nfs filesystem fpFsync waits until the buffers are flushed. With a 625Mb file it takes around 8 seconds in a LAN (with WiFi) environment. Here is the result:

maendle@lutz-nb:/devel/FPC-Projekte/Demos/various_tests> ./fsynctest
2026-05-24 13:08:10.192 Start
2026-05-24 13:08:10.337 BlockWrite is done
<<3,0>>
2026-05-24 13:08:18.175 fpFsync is done

The same with a local file shows this:

maendle@lutz-nb:/devel/FPC-Projekte/Demos/various_tests> ./fsynctest
2026-05-24 13:09:55.682 Start
2026-05-24 13:09:55.822 BlockWrite is done
<<3,0>>
2026-05-24 13:09:56.145 fpFsync is done

Here is the testprogram, it is adapted from your code fragment.

Code: Pascal  [Select][+][-]

1. program fsynctest;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   Unix, sysutils;
7.  
8. var
9.   filespec2:string = '/daten/Temp/test.bin'; //on a non local filesystem nfs
10. //  filespec2:string = '/tmp/test.bin';        //on a local filesystem ext4
11.   f2:file; // destination file
12.   h:THandle;
13.   e:cint;
14.   i:Integer;
15.   data:array[0..255] of byte;
16.  
17. begin
18.   WriteLn(FormatDateTime('yyyy-mm-dd hh:mm:ss.zzz',Now),#9,'Start');
19.   Assign(f2,filespec2); // open destination file
20.   Rewrite(f2,SizeOf(data));
21.     //...
22.     // do the whole copy via blockread() and blockwrite()
23.     //...
24.   for i:=1 to 80000 do
25.     BlockWrite(f2,data,32);
26.  
27.   WriteLn(FormatDateTime('yyyy-mm-dd hh:mm:ss.zzz',Now),#9,'BlockWrite is done');
28.   h:=system.FileRec(f2).Handle;
29.   Write('<<', h);
30.   e:=Unix.fpFsync(h); // 0=OK / rest=Error
31.   WriteLn(',',e,'>>');
32.   WriteLn(FormatDateTime('yyyy-mm-dd hh:mm:ss.zzz',Now),#9,'fpFsync is done');
33.   Close(f2); // close destination file
34. end.
35.  

If with WebDAV fpFsync doesn't wait this means, that you have no chance (and IMHO the filesystem driver is poor implemented).

[Hartmut]

Thank you very much again Lutz for your help. I ran your demo with a mounted WebDAV-folder as destination and the result was:

Code: Text  [Select][+][-]

1. 2026-05-24 17:56:02.843 Start
2. 2026-05-24 17:56:05.741 BlockWrite is done
3. <<3,0>>
4. 2026-05-24 17:56:06.477 fpFsync is done

so fpFsync() waited only 736 ms, although the transfer of 625 MB needed about 164 seconds. So Unix.fpFsync() is really not usable for my purpose.

Has someone another idea, how I can wait, until an upload via WebDAV (using the Cache) is really completed?

E.g. when I disconnect the WebDAV connection by the 'umount' command, then, if an upload is active, the disconnect automatically waits, until the upload is completed, before the connection is disconnected. That means, "some part inside" must be able to detect the active upload...

Maybe is it possible to query some special "status" of the destination file?

[Lutz Mändle]

Have you read the man page of the davfs2 filsystem driver?

man davfs2.conf

There is the following option:

delay_upload
              When a file that has been changed is closed, mount.davfs will wait that many seconds before it will upload it to the server. This will avoid upload‐
              ing of temporary files that will be removed immediately after closing.  If you need the files to appear on the server immediately after closing, set
              this option to 0.
              Default: 10

That (and some other readings) implies for me that writing to a file on a webdav filesystem doesn't transfer data to the server. Instead the real write is triggered by closing the file. This explains the short delay of the fpFsync call, because it's only the time to write to the local cached file.

[Lutz Mändle]

Maybe is it possible to query some special "status" of the destination file?

From my understanding with a webdav filesystem you have no chance for that because you don't operate on a real remote file.
Maybe you can setup a second channel to the webdav server (a second mount) on which you query the file state.

[Hartmut]

Have you read the man page of the davfs2 filsystem driver?

Yes. I know parameter 'delay_upload' which I have changed from 10 to 0 long ago to start uploads immediately.

That (and some other readings) implies for me that writing to a file on a webdav filesystem doesn't transfer data to the server. Instead the real write is triggered by closing the file. This explains the short delay of the fpFsync call, because it's only the time to write to the local cached file.

You are right. I have checked, that an Upload does not begin, before close() is called. I was not aware of that before. This explains *why* fpFsync() does not work. But *that* fpFsync() does not work for my WebDAV problem we know since a couple of days. So we should not invest more effort in fpFsync() I think.

From my understanding with a webdav filesystem you have no chance for that because you don't operate on a real remote file.
Maybe you can setup a second channel to the webdav server (a second mount) on which you query the file state.

Will think about that. Currently I'm playing with function linux.statx (needs FPC > 3.2.2).

[Lutz Mändle]

From your initial post:

But when I call the same things from inside of the program, which has started an upload, then the same actions do not wait:
 - calling sysutils.FileGetAttr() for the destination file
 - or calling sysutils.FindFirst() for the destination file
 - or calling OS command 'ls <destination file>' via TProcess or RunCommandInDir().

Have you tried this things after closing the file?


 

[Hartmut]

From your initial post:

But when I call the same things from inside of the program, which has started an upload, then the same actions do not wait:
 - calling sysutils.FileGetAttr() for the destination file
 - or calling sysutils.FindFirst() for the destination file
 - or calling OS command 'ls <destination file>' via TProcess or RunCommandInDir().

Have you tried this things after closing the file?

Yes.

Looks that I found a solution by using function 'linux.statx' = https://man7.org/linux/man-pages/man2/statx.2.html which is included only in FPC > 3.2.2 (for <= 3.2.2 you must access this System Call by yourself via function syscall.do_syscall). This is my code:

Code: Pascal  [Select][+][-]

1. uses linux,baseunix;
2.  
3. function statx_wait_WebDAV(filespec: ansistring): cint;
4.    {waits during the Upload of file 'filespec' via WebDAV, until the transfer
5.     (using a Cache) is really completed.
6.     Result: 0=success / -1 = Error, then get Errorcode via 'baseunix.fpGetErrno'}
7.    const Flags = AT_STATX_FORCE_SYNC; {"Force the attributes to be synchronized
8.                     with the server. This may require that a network filesystem
9.                     performs a data writeback". 'AT_STATX_DONT_SYNC' did NOT work}
10.          Mask = STATX_TYPE; {asks only for the file type. 'STATX_MODE' and
11.                              'STATX_ALL' did also work, but 'STATX_TYPE' probably
12.                              will produce the smallest server effort}
13.    var STX: linux.TStatx;
14.    begin
15.    exit(linux.statx(AT_FDCWD,PChar(filespec),Flags,Mask, STX));
16.    end;

When I call this function after the close-command for writing the destination file, then this functions waits, until the Upload is really completed :-))

Again thanks a lot to Lutz for his continuous help.

[Lutz Mändle]

Good to know that statx has the power.