Recent

Author Topic: fpflock vs tcritical  (Read 2262 times)

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 635
Re: fpflock vs tcritical
« Reply #30 on: July 16, 2019, 08:53:06 pm »
SymbolicFrank

i have 'logger' on my system that ties into the syslog setup (looks like would just take more resources - but syslog is already running on the system) but instead of logger i would just parse the httpd logs but i want to save more then just in the logs

You can use the same mechanism for the normal files. It isn't about writing to syslog, but about how you do it. You just append text to some file in a thread-safe way.

I tend to write my own custom loggers, but I do know there are a few for fpc you could use. Just turn off things like timestamps and such.

lucamar

  • Hero Member
  • *****
  • Posts: 2124
Re: fpflock vs tcritical
« Reply #31 on: July 17, 2019, 02:18:58 am »
my correction to above -> c does abide to the fpc fpflock when flock is actually used in my c code :) (i had said it didn't because the code i used didn't have flock in the version i mistakenly used) (i wonder if fortran wil abide as well) :)
It's not really a question of language but of whether the other program also uses flock (or fcntl) to try to lock the file. In that case the call will return the corresponding error (or lock while waiting). If the other program doesn't care and just goes on with the writing (or reading) the system will hapily go and obey its orders. That's why they are called advisory locks.

Locking and respecting locks is seen (quite rightly) as the programmers' responsabilty. The problem is that while almost each file write (and not a few of the reads!) should be locked (remember, this is a multitasking and multiuser system), lots of programmers don't care about it or don't even know what "this thing" is for.

It's a lot like what happens with "coperative" multitasking (hey, Windows 3, hi!): if programs don't "cooperate" there is no multitasking ;)

Quote
thank you for the code - i'll change the form to a nice silver color though :)
This system does that automatically :D

Quote
https://www.kernel.org/doc/Documentation/filesystems/mandatory-locking.txt    <-- perfect for explaining all thanks for the information
Lots of little gems like that "hidden" there :)

Quote
i will program fcntl/fpfcntl vs flock/fpflock code (just to see if i can do it and to see if it any faster? will it? once compiled?
Probably minimally or not all: Take into account that fcntl() can do a lot more things than flock() so just deciding what it has to do may even make it slower.

Quote
thanks for the heads up about inserting quote
My pleasure.

but is fpaccess faster then fileexists ?   i would guess so but once compiled would it matter?

Again, probably not or minimally. All you really avoid (in LInux) is a call to FileExists(), which turns around and calls fpaccess to do the job. A matter of microseconds, at most, and you'd loose the multiplatform capacity.

Your program now is not multi-platform, if you're using flock(), but if some day you need to convert it to, say, Windows <shudder> that's one less place that would give you problems.

The rule of thumb for this kind of things is: if there's a platform-independent way to do something, use it; if not, $ifdef it :D
« Last Edit: July 17, 2019, 02:23:07 am by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.2/2.0.4  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

toby

  • Jr. Member
  • **
  • Posts: 66
Re: fpflock vs tcritical
« Reply #32 on: July 17, 2019, 04:02:18 am »
SynbolFrank


I tend to write my own custom loggers, but I do know there are a few for fpc you could use. Just turn off things like timestamps and such.

what are you using for file locking? or threading?

---

lucamar

i'm testing the cgi on the server from a program that does 750 threads with curl posts (the maxmimum threads that can be created is 767 though i have 3GB ram - i read that a delphi thread costs 1MB so ...)

i have no trouble running multiple instances of it in separate xterms and the file is good with the fpflocks - without fpflock it is a 'mess' it isn't consistant but running 4 or more instances i sometimes get a segfault but the file is still looks good up to that point (i guess the file is closed when the program ends??)

with decodetime(time, hour, minute, second, sec100); as my time i'm not getting any identical times - though i have to use a sleep(1) or i get the curl post threads really arriving in the file out of order

i've got to pass on trying to code using  fpfcntl (it is to complicated at this time -->- meaning i couldn't find a fpfcntl example using fpflock)
i did find an interesting fpfcntl example for sock/synapse code - but don't need another sidetracked day to try it out
https://forum.lazarus.freepascal.org/index.php/topic,28104.0.html

-

do you know if there is another programming way to 'hook' into a already running program on a server besides shared memory/shm or by writing and then reading a 'shared file'?

---

lucamar

  • Hero Member
  • *****
  • Posts: 2124
Re: fpflock vs tcritical
« Reply #33 on: July 17, 2019, 08:57:01 am »
(i guess the file is closed when the program ends??)
Yes, but it's better if you close it yourself. Do it right after unlocking it, when you have ended writing to it. Unless you need it right after that for other purposes, of course.

Quote
with decodetime(time, hour, minute, second, sec100); as my time i'm not getting any identical times - though i have to use a sleep(1) or i get the curl post threads really arriving in the file out of order
I don't know what you mean by this but if what you want is to save all lines from a single call with the same timestamp, you can simply use a TDateTime var, equate it to Now at the start and use it everywhere.

Quote
i've got to pass on trying to code using  fpfcntl (it is to complicated at this time -->- meaning i couldn't find a fpfcntl example using fpflock)
It's not so difficult, but since all you want to do with it can already be done with flock I don't think it's worth the effort. Take a note to test it when you have some free time and you're done :)

Quote
do you know if there is another programming way to 'hook' into a already running program on a server besides shared memory/shm or by writing and then reading a 'shared file'?
Depends on what you mean by "hook" but if you mean communicating, any IPC mechanism should suffice. Since you're in Linux (aren't you?) you could try using DBus or similar but IMHO it's a little overkill, though that depends (as always) on exactly what you want to achieve.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.2/2.0.4  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

PascalDragon

  • Hero Member
  • *****
  • Posts: 705
  • Compiler Developer
Re: fpflock vs tcritical
« Reply #34 on: July 17, 2019, 09:07:10 am »
This might be off topic again, but web servers usually handle each client in a separate thread in order to serve multiple clients as quick as possible. If each request has to be logged into the same file which is effectively protected by a mutex-type mechanism then the benefit of multi-threading is kinda lost. If there are 50 requests to be served simultaneously then the last request may have to wait for 49 other clients to finish writing to the log. Is that really what you want? :o
Some kind of synchronization is necessary as otherwise you'll have garbled log output. However file locking is indeed not the best approach compared to a queue based mechanism with a separate logging thread as mentioned by SymbolicFrank.

lucamar

  • Hero Member
  • *****
  • Posts: 2124
Re: fpflock vs tcritical
« Reply #35 on: July 17, 2019, 09:24:25 am »
Some kind of synchronization is necessary as otherwise you'll have garbled log output. However file locking is indeed not the best approach compared to a queue based mechanism with a separate logging thread as mentioned by SymbolicFrank.

The problem is that this is a CGI. An instance is run by the server for each request, that is, they are all basically independent programs that need to synchronize with one another. Threads are not the solution here: IIPC is. And the simplest form of IPC is using file locks, which is what the OP is doing.

It would be different if the logger where a separate process to which the CGIs could pass the strings and forget about it. But then you run into other problems, not the less of which is that now is the logger who needs to hog time to serve all those requests as quick as posible, so the "time" problem doesn't go away; it just get shifted.

All in all, and for what the OP is doing, I think he has taken a good approach. Not the best, maybe, but a good one.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.2/2.0.4  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 635
Re: fpflock vs tcritical
« Reply #36 on: July 17, 2019, 10:11:39 am »
At this moment, there are 3460 separate threads running on my computer. Almost all of them are threads waiting on a device or other input. It's what threads do best.

The 1MB of memory mentioned is the size of the stack, you can configure that.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7590
Re: fpflock vs tcritical
« Reply #37 on: July 17, 2019, 12:15:50 pm »
This might be off topic again, but web servers usually handle each client in a separate thread in order to serve multiple clients as quick as possible. If each request has to be logged into the same file which is effectively protected by a mutex-type mechanism then the benefit of multi-threading is kinda lost. If there are 50 requests to be served simultaneously then the last request may have to wait for 49 other clients to finish writing to the log. Is that really what you want? :o

No, but that is easily solved, by having a small log cache per thread (or couple of them). Only when a request aborts or finished or at regular sync intervals (once per few seconds) such requests are pushed to the central system.

Fungus

  • Sr. Member
  • ****
  • Posts: 352
Re: fpflock vs tcritical
« Reply #38 on: July 17, 2019, 12:31:25 pm »
Some kind of synchronization is necessary as otherwise you'll have garbled log output. However file locking is indeed not the best approach compared to a queue based mechanism with a separate logging thread as mentioned by SymbolicFrank.

Yes, if you absolutely have to write all logging into one file. If I should make some code to log the client requests, I would log each request into it's own file which gets a timestamp and/or GUID as file name. Then I would make a cronjob (or a service / daemon) which periodically scans the log directory for new entries and move those entries into an amalgamized log file. This would completely remove the nescessity to synchronize access to one file and allow each thread to do it's task without having to wait for anything. But hey! Just because I don't use cowtit-juice in my coffee doesn't mean it's wrong to do so :P

toby

  • Jr. Member
  • **
  • Posts: 66
Re: fpflock vs tcritical
« Reply #39 on: July 17, 2019, 04:41:10 pm »
lucamar

Quote
Yes, but it's better if you close it yourself. Do it right after unlocking it, when you have ended writing to it. Unless you need it right after that for other purposes, of course.

i do close it but this was mentioned because i got runtime exceptions when running the threaded curl testing code af full speed but this is fixed by putting a sleep(1) after the curl line (using ths sleep(1) also helps get the curl post data arriving in the file in the thread creation order)  i use sysutils.executeprocess(curl, l);  with the l being the curl line - i have always had problems with curl and use to recompile fpc with the exception code commented out (i never got exception problems with anyother code) - but that turned out to be to much work doing it each time upgrading fpc from 2.4.0 to 3,1.1 in steps - i also use aprocessexecute for curl run in a bash shell when i need stdout from it
the libcurl for pascal is not as developed as say for php and posting with it isn't so straight forward - i also use synapse (low level and library level for sending posts - great fun but synapse code isn't my goal here - just quikly testing the fpflock coding on the server


Quote
with decodetime(time, hour, minute, second, sec100); as my time i'm not getting any identical times - though i have to use a sleep(1) or i get the curl post threads really arriving in the file out of order

I don't know what you mean by this but if what you want is to save all lines from a single call with the same timestamp, you can simply use a TDateTime var, equate it to Now at the start and use it everywhere.

i was showing that i get and use the sec100 for timing and timestamping i think that tdatetime now only has seconds?


Quote
i've got to pass on trying to code using  fpfcntl (it is to complicated at this time -->- meaning i couldn't find a fpfcntl example using fpflock)

It's not so difficult, but since all you want to do with it can already be done with flock I don't think it's worth the effort. Take a note to test it when you have some free time and you're done :)

i think you are very correct - and there isn't much fpfcntl example code around anyway to study :)



Quote
do you know if there is another programming way to 'hook' into a already running program on a server besides shared memory/shm or by writing and then reading a 'shared file'?

Depends on what you mean by "hook" but if you mean communicating, any IPC mechanism should suffice. Since you're in Linux (aren't you?) you could try using DBus or similar but IMHO it's a little overkill, though that depends (as always) on exactly what you want to achieve.

yes i'm using linux - started with red hat forked off to mandriva and then replaced all the rpms with source and that is what i have now
i'm currently doing the same for debian - this festival started with desire to just rip all the selinux out of it (systemd was very easy to get out - but boy was selinux infesting the whole thing)

the passing of data to another running program is interesting programming wise as is queuing the data and passing it to another running program as Fungus and PascalDragon discuss so at what point is it 'not overkill' to use it?

my coding somewhat related : I have code that passes data in through procedure variables and passing functions too etc but have never found use for them - always seemed easier to just use global variables??

---


Fungus

  • Sr. Member
  • ****
  • Posts: 352
Re: fpflock vs tcritical
« Reply #40 on: July 17, 2019, 07:38:33 pm »
the passing of data to another running program is interesting programming wise as is queuing the data and passing it to another running program as Fungus and PascalDragon discuss so at what point is it 'not overkill' to use it?

Overkill depends on your use case. DBus is IMHO cumbersome to implement and I would not use it unless I was writing code that either had to communicate with an existig service via DBus or if I was programming a service which had to expose functionality to other programs via DBus. You could use a simple named pipe for IPC, but for your use case even this would be overkill since all you want is to log the requests. If you do not experience any problems with the flock approach, no need to change it. But you should probably try to examine how many instances of your CGI is actually running when stress-testing and then compare it to a non-logging scenario where flock is not used. You may have lots of instances taking up memory whilst waiting for the log to become unlocked.

As mentioned, letting each request log to it's own file would be ideal. You could even do it into shared memory (usually located in /dev/shm/ - create your own subfolder), this would be lightning fast since there is no disk I/O. A service or something could then pick the log fragments, append them to the log file on disk and delete the fragments afterwards. This would be a simple and robust sollution for logging the requests without causing any hanging / waiting at all.

Thaddy

  • Hero Member
  • *****
  • Posts: 9278
Re: fpflock vs tcritical
« Reply #41 on: July 17, 2019, 07:55:52 pm »
Cumbersome? I disagree.
See the dbus articles by Michael van Canneyt.
https://www.freepascal.org/~michael/articles/

Comes with sources and pdf's, hence I could not link directly.

Those two articles also have some basic idea's to abstract complexity away.
also related to equus asinus.

lucamar

  • Hero Member
  • *****
  • Posts: 2124
Re: fpflock vs tcritical
« Reply #42 on: July 18, 2019, 01:09:15 am »
i was showing that i get and use the sec100 for timing and timestamping i think that tdatetime now only has seconds?

No, AFAIK TDateTime has the full-resolution time. Note that the first parameter of DecodeTime() is a TDateTime.

Quote
the passing of data to another running program is interesting programming wise as is queuing the data and passing it to another running program as Fungus and PascalDragon discuss so at what point is it 'not overkill' to use it?

When you need the capabiities it offers :)

Quote
my coding somewhat related : I have code that passes data in through procedure variables and passing functions too etc but have never found use for them - always seemed easier to just use global variables??

You should use global vars only when they really need to be global. In small programs it may not matter much, except from an "ellegance" point of view, but for medium/large programs indiscriminate use of global variables tends to produce subtle and hard-to-find bugs because literally every procedure/function may be changing the variables in ways and in a sequence that is hard to control and predict.

It's usually best if each function/procedure has its own "view" of any passed parameter, without having the possibility of affecting the rest of the program unless specifically needed (which is what var and out parameters are for).

I don't explain this well (I'm told) but almost any basic programming book/tutorial should do it better.
« Last Edit: July 18, 2019, 01:12:36 am by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.2/2.0.4  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.

toby

  • Jr. Member
  • **
  • Posts: 66
Re: fpflock vs tcritical
« Reply #43 on: July 18, 2019, 02:04:23 am »
lucamar

nice to know that 'time' wasn't some magic variable - i never had to define it on the var line and decocdetime always worked so i never looked at the man page!!

did you see my other new thread your input would be greatly appreciaed there also

https://forum.lazarus.freepascal.org/index.php/topic,46127.0.html

lucamar

  • Hero Member
  • *****
  • Posts: 2124
Re: fpflock vs tcritical
« Reply #44 on: July 18, 2019, 02:14:16 am »
nice to know that 'time' wasn't some magic variable - i never had to define it on the var line and decocdetime always worked so i never looked at the man page!!

If you always use literally "DecodeTime(time, ...)" then what you're doing is always calling SysUtils' Time() function, which returns the current time. No wonder, then, that it was giving you a different "time" each time :)

Quote
did you see my other new thread your input would be greatly appreciaed there also

Yeah, I already left an answer there. But I don't know much about programming servers (just the basics and, maybe, a little beyond) so my answers may not be meaningful. In which case I prefer not to make them :)
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 2.0.2/2.0.4  - FPC 3.0.4 on:
(K|L)Ubuntu 12..16, Windows XP SP3, various DOSes.