Create a file when it does not exist or fail

[jollytall]

I am trying to make some synchronization between programs (some FPC, some C++) under Linux. A simple solution would be to make some "lock" files. For it I would need to create the lockfile if it does not exist or return fail if it does. If it creates I can do whatever I want to be "semaphored" and then delete the file. If it fails I simply wait a bit.
The problem is that all functions (simplest being rewrite) I found create the file even if it exists. Simple sharing options do not work under Linux either. To make it worse even the low level FileCreate creates a handler not only if the file exists, but even if it exists AND open.

Code: Pascal  [Select][+][-]

1. var
2.   t:system.text;
3.   f:longint;
4.   i:integer;
5. begin
6. assignfile(t,'test.txt');
7. rewrite(t); // file created and opened for writing
8. for i:=1001 to 1010 do // writing out 10 numbers in readable form
9.   writeln(t,i);
10. f := filecreate('test.txt'); // trying to create it once more
11. if f=-1 then
12.   writeln('File already exists and open') // This does not happen
13. else
14.   for i:=1 to 100 do // instead this code runs (f is happily open)
15.     filewrite(f,i,sizeof(i));
16. fileclose(f); // this is when the content of f is actually flushed to disk if not earlier
17. for i:=2001 to 2010 do // and the same file is written again
18.   writeln(t,i);
19. closefile(t); // this is when the content of t is flushed to disk (including 1001..1010) and thus overwriting the first part that was written in f
20.  

Checking in two steps that the file exists and if not then create it, is not an option, because another program can also create the file between the two steps.

In C fopen had worked the same way, but since 2011 there is an extra option "wx" instead of "w" that fails the open if it exists. I did not find it in FPC.

How could I solve it?

[rsz]

Hi,

This does what you want:

Code: Pascal  [Select][+][-]

1. program FileLock;
2.  
3. uses sysutils;
4.  
5. var
6.   LockFilePath: String;
7.   Lock: THandle;
8. begin
9.   LockFilePath := GetTempDir + 'MyLockFile';
10.   Lock := FileCreate(LockFilePath, fmShareExclusive, &644);
11.   if Lock <> -1 then
12.   begin
13.     WriteLn('Acquired lock.');
14.     WriteLn('Press ENTER to release lock and exit.');
15.     ReadLn;
16.     FileClose(Lock);
17.     DeleteFile(LockFilePath);
18.   end
19.   else
20.   begin
21.     WriteLn('Failed to acquire lock.');
22.   end;
23. end.

Tested it both on Linux and in WINE, run the program twice to see it in action.

Some quick remarks about this:
1. If your program acquired the lock and crashes, there's no nice and simple recovery (see 3.).
2. Creating the lock file in /tmp/ (GetTempDir) on Linux is a good idea, as files in there are generally deleted on bootup so if the entire system crashes your program will be able to acquire the lock on the next boot. This is not the case on Windows as far as I'm aware.
3. You could delete the lock file manually if an error occurs but you could also write the PID of the process which acquired the lock into another file. This would allow other processes to check if the lock is still valid by checking if the process is still alive. If it isn't then it means something went wrong and the lock can be deleted and acquired again.

EDIT: Wrong overloaded FileCreate was being called. It also seems that the way this is implemented with FileCreate, with flock under the hood, it handles crashes gracefully, so my remarks in 1. and 3. are void, at least on Linux.

[jollytall]

Thank you, it works, but I do not understand.
Earlier I also tried including fmShareExclusive in FileCreate as you did. It did not work.
So now I played around a bit with your code and mine and found that the difference that matters is the /tmp directory. Actually your code also fails if the lock file is not placed in the temp directory. It seems that in your point 2 using the /tmp is not only "a good idea", but a must to work.
If you just delete GetTempDir + from line 9, it does not work (always can acquire the lock file).
What can be the reason?

[rsz]

In the above code I was calling the wrong overloaded FileCreate with parameters FileName, Rights instead of the correct one FileName, ShareMode, Rights  . That it worked at all is a coincidence it seems.
I've updated my remarks and the code in my post above.

Instead of the 644 permission you can set it to what suits your use case best, but it shouldn't affect the locking mechanism.

[jollytall]

Thanks, we made a good progress, but still I think somehow we are on the wrong track, mixing up three different things: file locking, file permissions and existence of a file.

My original goal was to hand over "tasks" from program A to program B in a file. For that file locking (flock) would be the ideal solution (no need even for a separate lock file, the task file is locked in itself). While A is updating the tasklist it locks the file and similarly while B cleans the tasks it can also lock. The problem is that the flock mechanism under Linux is unreliable. It is called "advisory", i.e. processes either take it into account or not: https://www.baeldung.com/linux/file-locking. Often it can be made mandatory when the file system is mounted, but in certain cases it does not work at all (NFS). So this is a no solution.
You can test it. If in your program while waiting for ENTER you open the lock file in e.g. Gedit (for non-Linux users it is the "notepad" of gnome under Linux), it does not respect the advisory rights and simply opens it and you can write in it, save it, etc. Once you press ENTER the file is deleted normally, because Linux does not care that someone has also touched it in the meantime.
You are right though that flock is released once the process terminates, so even a program respecting flock can acquire it again (no permanent locking out).

The second thing is the file permissions. Yesterday with the wrong FileCreate this is why tmp work differently than the home directory. Apparently Linux set different permissions in the two cases. Now, as it is explicitly specified there is no difference.
If the lock file is created without read and write permission even for the owner (&xxx AND &600 <> &600) it is created as a non accessible file. In Nautilus (File explorer of gnome/linux) the file is shown with a little lock icon. Now, if the process crashes without deleting the file, it remains there with restricted access. No other process can "acquire" it. Gedit cannot open it either.
Still any process under the same user can delete the file and thus sort-of release the lock.
It can be used as a "locking" mechanism even if access rights are not specified.

Code: Pascal  [Select][+][-]

1. Lock := FileCreate(LockFilePath, &577);

works just as with access right (I chose &577 just to show that it is enough if 2 or 4 is missing from the owner's permissions. Here the problem might again be (I did not test), how it behaves on different file systems or over network access.

This is why I came up with the original problem, create a file if it does not exist and fail if the file exists regardless with what permission, what locking. For this problem I still do not have a solution.

Is my assessment correct, or do I miss something?

[rsz]

Yes, your assessment is correct but given these requirements I don't believe file locking (neither flock or with an atomic create) is the right choice here.
It sounds like you want unix domain sockets or tcp sockets for communication between your programs instead, maybe even a database.

[marcov]

Or ipc.

[jollytall]

Yes, I consider many options. Sockets, pipes, etc. are not really good, because they require that both processes run at the same time or need to build in a mechanism to track what was delivered and what was not. SQL is an option, but I feel it an overkill. A simple file with a list of task is all I need, but with a control that A and B do not access it at the same time.
For this locking the tasklist file would be an easy option if worked (probably I will use it and respect the lock status both in A and B, since I write all the programs and for the time being they are all on the same machine, file system). To be a bit more flexible in terms of network devices, file systems, etc., it seems/seemed the easiest to use a lock file as the semaphore.

So, in this thread I would still like to focus on the original problem. How could I check if a file exists and in the same step(!!!) create it if it does not exist.
It should be possible if C can solve it. Somehow there is an option (x) in C fopen() not to create the file in FileCreate if it exists, unless C also does it internally in two steps first checking if it exists and to create if not. I tested it, but since it is so fast, I never came across this problem, but maybe I was only lucky. I did not go deeper to check the source code for the C solution. Maybe that will be the next step.

[MarkMLl]

Checking in two steps that the file exists and if not then create it, is not an option, because another program can also create the file between the two steps.

Careful with your assumptions there: some of the Linux syscalls are in fact guaranteed to be atomic.

Apart from that I echo what a couple of others have said: Unix-domain sockets or a database.

A Unix-domain socket does still have to be created, and (the file representing it) persists even if the "owner" has terminated. Also you need to be careful with ownership: if you create it in ~ then other programs might not be able to write to it, while if you put it in /var/run then you'll need to give its creator the appropriate POSIX capabilities (or run setuid root, which is increasingly problematic).

Apart from that, noting that use of e.g. FPC's Ipc unit will be problematic since you're talking about multiple languages, I suggest looking at e.g. SQLite since it's reasonable to assume that the author has done everything possible to enforce locking and atomicity across various platforms... and that he gets enough feedback to be better informed in that area than most of us.

MarkMLl

[Kays]

[…] The problem is that all functions (simplest being rewrite) I found create the file even if it exists. […]

Well, then, as a workaround,

• go for it and create a temporary file (e.g. like in a mktemp(3) fashion).
• Attempt hardlinking it to the name of the lock file.
  • If it worked out, you can do your protected stuff. Ensure to addExitProc deleting your lockfile.
  • If it failed, there’s also another process running, or the lock file wasn’t deleted.
• Delete the temporary file, obviously.

Caveat: The used FS needs to support hardlinking (the tmpfs at /var/run→/run does that) and must be mounted readable and writable.