[SOLVED] How to wait until a WebDAV-Upload w. Cache is really completed? (Linux)

[Hartmut]

I have a self written FPC console program to upload files to a Cloud, which is connected via WebDAV. When I upload larger files, then this program seems to be finished, but in the Taskmanager of the OS I can see, that the upload needs a couple of more time to finish, which probably is caused by the WebDAV-Cache.

I want to be able to wait in my program, until the upload is really completed. E.g. for time measurements or for to wait, until I can read the file back, to compare if it's content is correct. How can I do this wait?

When I start during such an upload in a separate terminal the OS command 'ls <destination file>', then this ls-command waits, until the upload is finished, before it displays it's result.
Or when I start during such an upload an external (second) FPC program in a separate terminal, which
 - calls sysutils.FileGetAttr() for the destination file
 - or calls sysutils.FindFirst() for the destination file
then this external FPC program also waits, until the upload is finished, before it returns.

But when I call the same things from inside of the program, which has started an upload, then the same actions do not wait:
 - calling sysutils.FileGetAttr() for the destination file
 - or calling sysutils.FindFirst() for the destination file
 - or calling OS command 'ls <destination file>' via TProcess or RunCommandInDir().

Has someone an idea, why the same actions work, when started from outside of the uploading program, but does not work, when started from inside of the uploading program? How can I fix this?

If it matters: WebDAV is connected e.g. by command "mount /mnt/gmx" and in file /etc/fstab there is a line "https://mediacenter.gmx.net /mnt/gmx davfs noauto,user,rw 0 0" for this.

I'm using FPC 3.2.2 on Linux Ubuntu 24.04 with KDE-Desktop.

[marcov]

No idea, but try a fpsync ? That is usually a wait till writes are really finished on *nix.

[paweld]

When using TProcess or RunCommand, do you enable the poWaitOnExit option?

[Hartmut]

No idea, but try a fpsync ?

Thanks marcov for this suggestion, but I do not know "fpsync" and I did not find it. Not with
search the free pascal documentation or Index of all identifiers in package 'fcl' or Index of all identifiers in package 'rtl' or Index of all identifiers in package 'lcl' .
Can somebody help?

When using TProcess or RunCommand, do you enable the poWaitOnExit option?

Yes, I used [poUsePipes,poWaitOnExit] as TProcessOptions.

[Thausand]

Can somebody help?

I think marco is write for fpsync command linux: https://manpages.debian.org/unstable/fpart/fpsync.1.en.html

BTW: For search freepascal identifier RTL can use: https://www.freepascal.org/docs-html/rtl/index-8.html

[Hartmut]

Thank you Thausand for that info and your link. I misleaded myself, thinking "fpsync", because starting with "fp", would be a Free Pascal thing. Did not know this Linux tool.

I had a look in your documentation link for fpsync. I'm not familiar with those network / synchronizing theme. Looks difficult for me.

What I first would like to understand: what was the intention of @marcov, how I should use fpsync:
 - should I replace my current file copying procedure completely by fpsync?
 - or can fpsync be used, after my current file copying procedure did it's job, so that fpsync only waits, until the upload (with Cache) is completely finished?
If possible, I would prefer the second.

[marcov]

Sorry, it is fpFsync:  https://www.freepascal.org/docs-html/rtl/unix/fpfsync.html  But that is if you write the file, and want to block till it is "written".  I used that on NFS shares for similar problems.

[Hartmut]

I tried fpFsync, but I don't get it to work. It returns '0', but it does not wait until the upload is completed.

Your documentation link says, that I have to pass a "file descriptor" to fpFsync. I assumed that this is the same as the file handle. Is this correct?

My code, which uploads a file to the cloud via WebDAV is the same as a "normal file copy" and the relevant part for the destination file looks as this:

Code: Pascal  [Select][+][-]

1. var f2: file; // destination file
2.     h: THandle;
3.     e: cint;
4. ...
5. assign(f2,filespec2); // open destination file
6. rewrite(f2,1);
7. ...
8. // do the whole copy via blockread() and blockwrite()
9. ...
10. h:=system.FileRec(f2).handle;
11. write('<<', h);     // shows always '4'
12. e:=unix.fpFsync(h); // 0=OK / rest=Error
13. write(',',e,'>>')
14. close(f2); // close destination file

I tested uploading with files up to 60 MB, which requires about 18 seconds, but the call to fpFsync never waits.
Do you see something what I made wrong?

If it matters: WebDAV is connected e.g. by command "mount /mnt/gmx" and in file /etc/fstab there is a line "https://mediacenter.gmx.net /mnt/gmx davfs noauto,user,rw 0 0" for this.
I'm using FPC 3.2.2 on Linux Ubuntu 24.04 with KDE-Desktop.

[Hartmut]

Hmm, is here nobody who can tell,
 - whether the "file descriptor" needed to pass to 'fpFsync' is the same as the file handle or not?
 - and whether my code to "use" 'fpFsync' is correct or wrong?

As said, the returned file-handle is always '4'. I'm surprised, that this number ist so small and that it's always the same number - for a file handle I would expect a much larger number and not always to be the same - can this be correct?

[Lutz Mändle]

I guess the 4 is because the file is the second explicit opened file in the process. The numbers 0 to 2 are for standard input, standard output and standard error.
I have digged in the rtl sources to see how assign, reset and rewrite are implemented and found that filerec.handle is indeed the filedescriptor resulting from a call to fpopen. Therefore fpfsync should work.

[Hartmut]

Thanks a lot Lutz for these infos. Seems that my "implementation" of 'fpFsync' is correct. Than this means, that unfortunately 'fpFsync' is not usable for this purpose :-((

Has someone another idea, how I can wait, until an upload via WebDAV (using the Cache) is really completed?

E.g. when I disconnect the WebDAV connection by the 'umount' command, then, if an upload is active, the disconnect automatically waits, until the upload is completed, before the connection is disconnected. That means, "some part inside" must be able to detect the active upload...

How can I query / wait for an active upload (without disconnecting and reconnecting again, which would cost too much time)?

[Lutz Mändle]

Can you test whether fpfsync waits on a NFS share. Maybe you have to use a bigger file (10 Gb for example) if the share is on the local network. 

[Hartmut]

What do you mean with 'NFS share'? I never heard this... Or do you mean filesystem 'NTFS'? What do I have to pay attention for, that the test does, what you want?

What I not understood: what will it help us to know, if fpfsync waits on a NFS share?

I will have a visitor soon for 1 or 2 days, so I can do this test only after he's gone.

[Lutz Mändle]

NFS is the standard networking filesystem on Unix/Linux for decades. I borrowed the term share from samba, in that context its the same what a share (Freigabe) is on Windows. A nfs server calls the shares exports.

The idea was to test this file copy procedure on another non local filesystem which is maybe better supported by the kernel.

[Hartmut]

Sorry Lutz, I still have not understood how this test should look like and what the benefit would be.

Do I need a second computer to do a copy over the network (in a special way, that a 'NFS share' is used)? Or could I do this test with only 1 computer by creating a 'NFS share' and do the copy in a special way, that this share is "used"?

Because my network knowledge is verry small and about 'NFS shares' it is zero, I fear a lot of time-consuming effort without that this would solve my WebDAV problem - that's why I hesitate.

Please, what would it help for my WebDAV problem to know, whether fpfsync would wait on a NFS share?