Recent

Author Topic: fpflock vs tcritical  (Read 2259 times)

lucamar

  • Hero Member
  • *****
  • Posts: 2124
Re: fpflock vs tcritical
« Reply #15 on: July 14, 2019, 03:45:33 am »
Is there an Assign(fw, fn); before that? Because otherwise, fw would have an spurious handle, provoking that error.

Either that or the error comes from another call you're missing.

If neither helps, try doing something like:
Code: Pascal  [Select]
  1. {Declare vars lockRes and tempErr before!}
  2. lockRes := fpflock(fw, LOCK_EX);
  3. tempErr := fpgeterrno;
  4. writeln('fpflock EX ', lockRes, ' fpgeterrno : ', tempErr, ' ', syserrormessage(tempErr));
to check that it's not an artifact of WriteLn().

Check also that there are no calls to fpseterrno anywhere!

It shouldn't matter (much): fpflock is returning 0 which means it succeded, but it's somewhat annoying :)
« Last Edit: July 14, 2019, 04:01:13 am by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.2/2.0.4  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

Fungus

  • Sr. Member
  • ****
  • Posts: 352
Re: fpflock vs tcritical
« Reply #16 on: July 15, 2019, 01:28:08 pm »
EDIT: I misinterpreted some of the answers, fpFlock is in fact useable as a inter-process mutex on Linux. Sorry, my bad!

This might be off-topic, but I think your approach is "bad" since neither a critical section nor a file lock will ever work for you. Critical sections are only usefull if all the threads using it is in the same process (group) and a file lock will eventually cause "race conditions" where the result may be worse than without the lock.

Since web-requests (through CGI or what not) may overlap you are in fact dealing with a multi-threaded, inter-process environment. In order to deal with this, you either need to create a "log server" whereto each CGI process can connect (through an IPC, pipe or whatever) or you need to let each request log into its own file and then have a service process amalgamize the logs into one file. The last (and maybe the best) option is to use a mutex which is basically a critical section that works for inter-process multi-threading, but I'm not sure if FPC has an easy, portable fix for this.
« Last Edit: July 15, 2019, 07:27:13 pm by Fungus »

Thaddy

  • Hero Member
  • *****
  • Posts: 9278
Re: fpflock vs tcritical
« Reply #17 on: July 15, 2019, 04:27:14 pm »
This might be off-topic, but I think your approach is "bad" since neither a critical section nor a file lock will ever work for you. Critical sections are only usefull if all the threads using it is in the same process (group) and a file lock will eventually cause "race conditions" where the result may be worse than without the lock.
The lock family functions (even if prefixed by fp) are Linux (Unix) system calls. They are thread safe and will not cause race conditions at all. Either document your reply or withdraw it.
I am not very happy that marcov has prefixed all these calls, because it leads to just the kind of confusion that is obviously in your head.... They are direct mappings to Linux kernel calls.
Yes, they are kernel calls...No overhead is involved.
I am not aware that the current Linux kernel (or BSD) has any race conditions.... :D :D :D
A programmer, though, is usually perfectly capable to screw up multi-threading, including me... So don't be offended.
« Last Edit: July 15, 2019, 04:37:45 pm by Thaddy »
also related to equus asinus.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 635
Re: fpflock vs tcritical
« Reply #18 on: July 15, 2019, 04:38:39 pm »
Ah futexes  :D

Thaddy

  • Hero Member
  • *****
  • Posts: 9278
Re: fpflock vs tcritical
« Reply #19 on: July 15, 2019, 05:16:40 pm »
Ah futexes  :D
That's only part of it.... :D
also related to equus asinus.

toby

  • Jr. Member
  • **
  • Posts: 66
Re: fpflock vs tcritical
« Reply #20 on: July 15, 2019, 10:28:12 pm »

lucamar

i don't think fpgeterrno gives any information unless there is an error

and i get success no matter how many LOCK_EX or LOCK_UN i put in a row (never tried this before)

what happens if you put a number of LOCK_EX in a row? shouldn't if fail if already enabled?

--
       
       to check that it's not an artifact of WriteLn().

       Check also that there are no calls to fpseterrno anywhere!

and the fpgeterrno error message i get is definitely from the if fileexists line

writeln('1 fpgeterrno : ', fpgeterrno, ' ', syserrormessage(fpgeterrno));
assign(fw, fn);
writeln('2 fpgeterrno : ', fpgeterrno, ' ', syserrormessage(fpgeterrno));
if fileexists(fn) then append(fw) else rewrite(fw);
writeln('3 fpgeterrno : ', fpgeterrno, ' ', syserrormessage(fpgeterrno));

output
1 fpgeterrno : 0 Success
2 fpgeterrno : 0 Success
3 fpgeterrno : 25 Not a typewriter








toby

  • Jr. Member
  • **
  • Posts: 66
Re: fpflock vs tcritical
« Reply #21 on: July 15, 2019, 10:48:28 pm »

Thaddy

     The lock family functions (even if prefixed by fp) are Linux (Unix) system calls. They are thread safe and will not cause race conditions at all. Either document your reply or withdraw it.
     I am not very happy that marcov has prefixed all these calls, because it leads to just the kind of confusion that is obviously in your head.... They are direct mappings to Linux kernel calls.
     Yes, they are kernel calls...No overhead is involved.

good to know but isn't the stream coming into a ethernet port always queued so a race condition couldn't really happen anyway?

-

toby

  • Jr. Member
  • **
  • Posts: 66
Re: fpflock vs tcritical
« Reply #22 on: July 15, 2019, 11:12:59 pm »
       to check that it's not an artifact of WriteLn().

       Check also that there are no calls to fpseterrno anywhere!

it isn't an artifact of writeln and no calls to fpseterrno anywhere!!! :)

no matter how many fpflock(fw, LOC_EX) i put in a row its result is always 0 (success) but the file locking doesn't exisit and fpgeterrno is not affected

only with one LOCK_EX is the file actually locked

so the success result from the fpflock only means that the statement executed not that the file is actually locked


toby

  • Jr. Member
  • **
  • Posts: 66
Re: fpflock vs tcritical
« Reply #23 on: July 16, 2019, 12:07:35 am »

sorry for extra posts  but i've been messing with the program so much that i had a fpflock(fw, LOCK_UN); that i didn't see.

so this is what it really is :

you can have a lot of fpflock(fw, LOCK_EX); commands in a row and a lot of fpflock(fw, LOCK_UN); as well all will return 0 (success) using lucamar's code - as long as a fpflock(fw, LOCK_EX); is the last fpflock run before the writeln(fw, xxxx)- the file will be locked

so the 0 (success) returned doesn't mean the file locking/unlocking is actually checked just that the statment is executed with the file opened in my case with append(fw) or rewrite(fw) - you get -1 if the file isn't append(fw) or rewrite(fw)


lucamar

  • Hero Member
  • *****
  • Posts: 2124
Re: fpflock vs tcritical
« Reply #24 on: July 16, 2019, 12:43:24 am »
and the fpgeterrno error message i get is definitely from the if fileexists line

That a makes certain sense, if the file doesn't exists. FileExists() uses a call to fpAccess(), which will fail (and set the error) if the file doesn't exists.

Regarding multiple locks/unlocks, man 2 flock has this to say:
Quote from: Linux Programmer's Manual - FLOCK(2)
A process may only hold one type of lock (shared or exclusive) on a file. Subsequent flock() calls on an already locked file will convert an existing lock to the new lock mode.

About this:

so the 0 (success) returned doesn't mean the file locking/unlocking is actually checked just that the statment is executed
That's not exactly true: there are quite some conditions (one of which is if the file is already locked by other process) in which flock() will return -1, which indicates an error, or simply not return (if you didn't use LOCK_NB) until the file is unlocked.

It's important to note that flock() is an advisory lock, not a forced one; any other process may decide to blithely ignore it and read/write from/to the file without the system trying to impede it. It's alright for what you're using it (preventing simultaneous usage by multiple instances) but it's not a "real" system-level lock.

This is a skeleton (and not very robust) function to do what you want:
Code: Pascal  [Select]
  1. function TryOpenAndLockFile(const Filename: String: var AFile: TextFile): Boolean;
  2. begin
  3.   Assign(AFile, Filename);
  4.   if FileExists(Filename) then Append(AFile) else Rewrite(AFile);
  5.   Result := fpflock(AFile, LOCK_EX or LOCK_NB) = 0;
  6.   if not Result then Close(AFile);
  7. end;
« Last Edit: July 16, 2019, 01:04:31 am by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.2/2.0.4  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

toby

  • Jr. Member
  • **
  • Posts: 66
Re: fpflock vs tcritical
« Reply #25 on: July 16, 2019, 02:33:28 am »

lucamar

     That a makes certain sense, if the file doesn't exists. FileExists() uses a call to fpAccess(), which will fail (and set the error) if the file doesn't exists.

i see that baseunix unit is loaded with -vu  so fpaccess must give faster code then using fileexists
(didn't know about fpaccess)

--

     Regarding multiple locks/unlocks, man 2 flock has this to say:
     Quote from: Linux Programmer's Manual - FLOCK(2)

     A process may only hold one type of lock (shared or exclusive) on a file. Subsequent flock() calls on an already locked file will convert an existing lock to the new lock mode.

thanks didn't know about flock so found these

https://www.gnu.org/software/libc/manual/html_node/File-Locks.html
https://gavv.github.io/articles/file-locks/

man 2 fcntl

so fpfcntl would be better then using fpflock cause it affects the file descriptors themselves (in /proc?)

--

            That's not exactly true: there are quite some conditions (one of which is if the file is already locked by other process) in which flock() will return -1, which indicates an error, or simply not return (if you didn't use LOCK_NB) until the file is unlocked.

            It's important to note that flock() is an advisory lock, not a forced one; any other process may decide to blithely ignore it and read/write from/to the file without the system trying to impede it. It's alright for what you're using it (preventing simultaneous usage by multiple instances) but it's not a "real" system-level lock.

yes i can see that   echo "distrupt file" >> test.txt   shows it isn't locked    just locked against other fpc file access and looks like c also doesn't abide by fpc's fplock!! :)

--

i am confused by LOCK_NB   if it isn't blocked then don't i have to put the check 'result is not -1' in a loop?

that is what paritally made using fileopen(   fmShareExclusive)  so slow

--


lucamar

  • Hero Member
  • *****
  • Posts: 2124
Re: fpflock vs tcritical
« Reply #26 on: July 16, 2019, 05:46:13 am »
Don't take this amiss but, please, use [quote] tags when citing other people; otherwise it is a little difficult to follow your posts and the sense of the thread. Just click on the "quote" link which is on every post (even in the reply screen).

i am confused by LOCK_NB   if it isn't blocked then don't i have to put the check 'result is not -1' in a loop?

that is what paritally made using fileopen(   fmShareExclusive)  so slow

In theory, if you don't use LOCK_NB and the file is locked, the flock() call should not return until the file is unblocked, so it makes no sense using a loop. If, however, you use LOCK_NB and the file is locked the call will return -1 and errno will contain EWOULDBLOCK.

The attached example shows (and allows you to explore) the difference.

With respect to fcntl(), note that it also uses advisory locks by default and the pre-conditions to be able to use mandatory locks are somewhat astringent. Read carefully the "mandatory locks" section of the man page.

I must confess that for serious work I tend to use streams almost exclusively and prefer to let the RTL libs do the "low-level" work by themselves :)
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.2/2.0.4  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

PascalDragon

  • Hero Member
  • *****
  • Posts: 705
  • Compiler Developer
Re: fpflock vs tcritical
« Reply #27 on: July 16, 2019, 09:08:25 am »
lucamar

     That a makes certain sense, if the file doesn't exists. FileExists() uses a call to fpAccess(), which will fail (and set the error) if the file doesn't exists.

i see that baseunix unit is loaded with -vu  so fpaccess must give faster code then using fileexists
(didn't know about fpaccess)
However unlike fpAccess FileExists is cross platform and also deals with conversions of the string argument to the correct code page (not to mention that in the future it will also correctly deal with symlinks depending on the FollowLink parameter).

Fungus

  • Sr. Member
  • ****
  • Posts: 352
Re: fpflock vs tcritical
« Reply #28 on: July 16, 2019, 01:44:47 pm »
This might be off topic again, but web servers usually handle each client in a separate thread in order to serve multiple clients as quick as possible. If each request has to be logged into the same file which is effectively protected by a mutex-type mechanism then the benefit of multi-threading is kinda lost. If there are 50 requests to be served simultaneously then the last request may have to wait for 49 other clients to finish writing to the log. Is that really what you want? :o

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 635
Re: fpflock vs tcritical
« Reply #29 on: July 16, 2019, 01:49:52 pm »
Anyway, the problem described is what a logger does as well. You can use an existing one or write your own. It is basically a thread with a concurrent queue that holds the lines to be written. So you can simply open the file and write to it without locking it, as the locking is in the queue. That will also prevent partial lines.