[SOLVED] Game logging template — StdOut, StdErr

[furious programming]

For the needs of my game, I need a very simple system for logging various data in the form of text, which is ultimately to be done using Write and WriteLn intrinsics (they are powerful, have built-in data type detection and various formatting abilities). Logging should be based on built-in mechanisms and system ones, i.e. System.StdOut and/or System.StdErr handles.

This should work so that if the system console is available, all data should be displayed there — if the application is compiled with {$APPTYPE CONSOLE} switch. However, if the console is not available, all the diagnostic data should be written to a text file (as a log file).

I have a problem understanding how such logging should work, how to properly redirect the output to the custom log file when the game is compiled with {$APPTYPE GUI} mode and where the log file should be created, according to operating system guidelines and security services (like UAC).

For now, I checked and confirmed that if the console is not available, the System.StdOut and System.StdErr handles are initialized and I can write data with WriteLn without any problems, but I don't know where they end up. Are they stored somewhere or are they lost?

Has anyone done something like this before? I would like to mention that I don't want to use any dependencies and logging frameworks — I want to create as simple a login system as possible, which must be based on Write and WriteLn. I will be grateful for any help.

[Bogen85]

The thing is you need a logging framework, even if it is just a simple light one that you create custom for yourself.

The problem is when you create a simple custom one yourself, as time you goes on the more you use it, the more you will update, and it won't be so simple any more. (Even if it is simple to use.)

Been there, done that, many times...

[furious programming]

As I mentioned, I don't want to use any framework for such a simple thing like creating log files. What I need is to understand how these handles works, for both to extend my knowledge and to implement what I want to implement in my project. So please, let's focus on my questions.

[Bogen85]

Yes... understood on the not wanting a framework... But personally find logging that does not universally include file, line, function by default as "difficult". (and the ability to include stack traces...) Yes, better than nothing, but still "difficult"...

But I digress.

Problem on  write and writeln unless you explicitly indicate stderr or stdout, a redirect won't take place. (Unless I'm missing something).

Code: Pascal  [Select][+][-]

1. {$mode objfpc}
2. {$h+}
3. program output_redirect;
4.  
5. uses sysutils;
6.  
7. procedure println(const text: string);
8.   begin
9.     writeln(stdout, text);
10.   end;
11.  
12. procedure eprintln(const text: string);
13.   begin
14.         writeln(stderr, text);
15.   end;
16.  
17. procedure println(const fmt: string; const args: array of const);
18.   begin
19.         println(format(fmt, args));
20.   end;
21.  
22. procedure eprintln(const fmt: string; const args: array of const);
23.   begin
24.         eprintln(format(fmt, args));
25.   end;
26.  
27. procedure hello1;
28.   begin
29.         writeln('hello 1!')
30.   end;
31.  
32. procedure hello2;
33.   begin
34.         writeln(stderr, 'hello 2!')
35.   end;
36.  
37. procedure hello3;
38.   begin
39.         writeln(stdout, 'hello 3!')
40.   end;
41.  
42. procedure hello4;
43.   begin
44.     println('hello 4!');
45.   end;
46.  
47. procedure hello5;
48.   begin
49.     eprintln('hello 5!');
50.   end;
51.  
52. procedure hello6;
53.   begin
54.     println('hello %d!', [6]);
55.   end;
56.  
57. procedure hello7;
58.   begin
59.     eprintln('hello %d!', [7]);
60.   end;
61.  
62. procedure hello_all;
63.   begin
64.         hello1;
65.         hello2;
66.         hello3;
67.         hello4;
68.         hello5;
69.         hello6;
70.         hello7;
71.   end;
72.  
73. begin
74.   hello_all;
75.   close(stdout);
76.   close(stderr);
77.   assign(stdout, 'stdout.txt');
78.   assign(stderr, 'stderr.txt');
79.   rewrite(stdout);
80.   rewrite(stderr);
81.   hello_all;
82. end.

Compiles:

Code: Text  [Select][+][-]

1. $ fpc output_redirect.pas
2. Free Pascal Compiler version 3.3.1 [2022/12/31] for x86_64
3. Copyright (c) 1993-2022 by Florian Klaempfl and others
4. Target OS: Linux for x86-64
5. Compiling output_redirect.pas
6. Linking output_redirect
7. 82 lines compiled, 0.2 sec, 375600 bytes code, 166616 bytes data

Runs:

Code: Text  [Select][+][-]

1. $ ./output_redirect
2. hello 1!
3. hello 2!
4. hello 3!
5. hello 4!
6. hello 5!
7. hello 6!
8. hello 7!
9. hello 1!
10.  
11. $ cat stdout.txt
12. hello 3!
13. hello 4!
14. hello 6!
15.  
16. $ cat stderr.txt
17. hello 2!
18. hello 5!
19. hello 7!

Problem is the second hello 1!, as writeln is not picking out the stdout redirect.

I included println and eprintln as println does work correctly before and after the redirect.
However, they are not intrinsics like write and writeln so they have no automatic type detection.... (and additional arguments to be printed are in an array, and needed to specified in the format string).

Something like println and eprintln (or what ever you would call them) could automatically file/line/function of the caller without specifying those in the call to them, but as you already pointed out, you don't seem to interested in any of that (even though you did say "game logging template"...) but once again, I digress.

If someone else knows how to make write and writeln both respect the stdout redirect implicity (with explicity including it in the call) I'd like to see the mechanism needed.


 

[furious programming]

The first idea that came to my mind is the simplest solution, which is to check the IsConsole variable and if it is False, release the StdOut file and assign to it my own — with a relative path, so that the log file is created always next to the executable.

Code: Pascal  [Select][+][-]

1. if not IsConsole then
2. begin
3.   Close(StdOut);
4.  
5.   Assign(StdOut, 'log.txt');
6.   ReWrite(StdOut);
7. end;

This is simple — if the game does not use the console (that is, build uses the {$APPTYPE GUI} switch), the current StdOut handle is freed, and a file is created instead and all data logged with WriteLn goes to it. Of course, for this to actually happen, all calls to Write and WriteLn must accept an output handle in the first parameter:

Code: Pascal  [Select][+][-]

1. WriteLn(StdOut, 'actual data in next parameters');

This is not a problem for me, so I can not only log data to the console or log file in this way, but also use all WriteLn capabilities (any number of arguments, formatting, support for any data types, etc.). And so that I don't have to explicitly use WriteLn, but my own identifiers, I can always declare a set of macros with names that match the convention used in the project (e.g. names Game_LogLn and similar).

I just don't know if creating a log file using a relative name — Assign(StdOut, 'log.txt') — won't be a bad choice. I don't know if operating systems will always allow application to create a log file next to the executable. However, I'd be in favor of actually creating a log file next to the executable so that the player doesn't have to look for it.

[PascalDragon]

I have a problem understanding how such logging should work, how to properly redirect the output to the custom log file when the game is compiled with {$APPTYPE GUI} mode and where the log file should be created, according to operating system guidelines and security services (like UAC).

Please note that {$APPTYPE GUI/CONSOLE} is only applicable to Windows.

For now, I checked and confirmed that if the console is not available, the System.StdOut and System.StdErr handles are initialized and I can write data with WriteLn without any problems, but I don't know where they end up. Are they stored somewhere or are they lost?

StdOut and StdErr are setup to put the output into message boxes except if NoErrMsg is set to true, then the output is simply discarded.

You simply need to Assign StdOut and StdErr to log files in case no console is available (just like you did in your recent post).

Alternatively you can create custom file types and then use Write(Ln) with those (see for example unit StreamIO which allows to assign a TStream to a file variable and then use Write(Ln) on it).

This is simple — if the game does not use the console (that is, build uses the {$APPTYPE GUI} switch), the current StdOut handle is freed, and a file is created instead and all data logged with WriteLn goes to it. Of course, for this to actually happen, all calls to Write and WriteLn must accept an output handle in the first parameter:

Code: Pascal  [Select][+][-]

1. WriteLn(StdOut, 'actual data in next parameters');

You don't need to explicitely specify StdOut if you reopen the Output instead of the StdOut file variable (as that is what is used when no file parameter is specified).

I just don't know if creating a log file using a relative name — Assign(StdOut, 'log.txt') — won't be a bad choice. I don't know if operating systems will always allow application to create a log file next to the executable. However, I'd be in favor of actually creating a log file next to the executable so that the player doesn't have to look for it.

In case of an installed application (and not something that runs in a portable way e.g. from a USB stick) the directory containing the executable is indeed protected. You need to study the OSes in question to discover where to best place the log files. E.g. on Linux systems it would probably be /var/log/<your app name> and on Windows %APPDATA%/<your app name>.

[furious programming]

Please note that {$APPTYPE GUI/CONSOLE} is only applicable to Windows.

Then what is the correct way to determine whether the system console should be used or not? Way other than checkbox in project settings window?

You don't need to explicitely specify StdOut if you reopen the Output instead of the StdOut file variable (as that is what is used when no file parameter is specified).

Ok, I checked it and it actually works. However, I should check whether the opening of the log file was successful, and if it failed, I have to assign something to the System.Output variable, otherwise the first call to WriteLn without specifying the file variable will generate runtime error.

So far I've done this:

Code: Pascal  [Select][+][-]

1. procedure Game_LogOpen();
2. begin
3.   if not System.IsConsole then
4.   begin
5.     {$PUSH}
6.     {$IOCHECKS OFF}
7.     AssignFile(System.Output, 'log.txt');
8.     ReWrite(System.Output);
9.     {$POP}
10.  
11.     if IOResult() <> 0 then
12.       System.Output := System.StdOut;
13.   end;
14. end;
15.  
16. procedure Game_LogQuit();
17. begin
18.   CloseFile(System.Output);
19. end;

This is just a dirty test, but it works fine both when the log file is created and when I give an invalid file name, preventing the file from creating and opening (in this case the output goes to limbo). Now all that's left is to modify the code so that System.Output doesn't close if the file hasn't been created.

In case of an installed application (and not something that runs in a portable way e.g. from a USB stick) the directory containing the executable is indeed protected. You need to study the OSes in question to discover where to best place the log files. E.g. on Linux systems it would probably be /var/log/<your app name> and on Windows %APPDATA%/<your app name>.

More precisely, the application will have a standard installer and will appear in the system as installed. Under Windows I know where to keep the application files (exactly where you specified, i.e. in %LOCALAPPDATA%), but I'm not sure about other platforms. Well, I still have time for that, so I'll be thinking about this in the far future, if I decide to create versions for other platforms.

[lainz]

You don't will publish on Steam? Then the folder is other, but the settings remains on app data.

[furious programming]

You don't will publish on Steam?

No, I will not be using Steam for sure.

For now, all I know is that read-only files should be in the installation directory, while all the files that will be modified must be in the %APPDATA% and %LOCALAPPDATA% directories. Unfortunately, I'm having some weird problem with creating files in these special directories. For example:

Code: Pascal  [Select][+][-]

1. AssignFile(System.Output, '%LOCALAPPDATA%\producer\product\log.txt');
2. Rewrite(System.Output);

The above code does not create the file even though the path physically exists. The IOResult function returns 3, which is an error about path not existing. Doesn't AssignFile support paths with system phrases?

[Kays]

[…] I just don't know if creating a log file using a relative name — Assign(StdOut, 'log.txt') — won't be a bad choice. I don't know if operating systems will always allow application to create a log file next to the executable. […]

There’s the notion of current working directory. On Linux, path_resolution(7) of non-absolute paths starts from the current working directory:

If the pathname does not start with the '/' character, the starting lookup directory of the resolution process is the current working directory of the process.

Therefore your log.txt will be created wherever the user is invoking your application from.

[TRon]

The above code does not create the file even though the path physically exists. The IOResult function returns 3, which is an error about path not existing. Doesn't AssignFile support paths with system phrases?

No it doesn't.
1. it is windows centric (not applicable for other platforms)
2. you need to 'translate' those environment variables into their real value manually.

Every platform has its own preferred paths for application and userdata. There are freepascal functions to retrieve those.paths but i do not have experience for every platform if they are accurate (for that platform).

edit:

Code: Pascal  [Select][+][-]

1. program test;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.  sysutils;
7.  
8. begin
9.   writeln('GetUserDir                             : ', GetUserDir);
10.   writeln('GetAppConfigDir(global)                : ', GetAppConfigDir(true));
11.   writeln('GetAppConfigDir(non-global)            : ', GetAppConfigDir(false));
12.   writeln('GetAppConfigFile(non-global)           : ', GetAppConfigFile(false));
13.   writeln('GetAppConfigFile(global)               : ', GetAppConfigFile(true));
14.   writeln('GetAppConfigFile(global,subdir)        : ', GetAppConfigFile(true, true));
15.   writeln('GetAppConfigFile(global,non-subdir)    : ', GetAppConfigFile(true, false));
16.   writeln('GetAppConfigFile(non-global,subdir)    : ', GetAppConfigFile(false, true));
17.   writeln('GetAppConfigFile(non-global,non-subdir): ', GetAppConfigFile(false, false));
18.   writeln('ApplicationName                        : ', ApplicationName);
19.   writeln('SysConfigDir                           : ', SysConfigDir);
20.   writeln('VendorName                             : ', VendorName);
21.   writeln('ConfigExtension                        : ', ConfigExtension);
22. end.
23.  

output on linux:

Code: [Select]

etUserDir                             : /home/TRon/
GetAppConfigDir(global)                : /etc/test/
GetAppConfigDir(non-global)            : /home/TRon/.config/test/
GetAppConfigFile(non-global)           : /home/TRon/.config/test.cfg
GetAppConfigFile(global)               : /etc/test.cfg
GetAppConfigFile(global,subdir)        : /etc/test/test.cfg
GetAppConfigFile(global,non-subdir)    : /etc/test.cfg
GetAppConfigFile(non-global,subdir)    : /home/TRon/.config/test/test.cfg
GetAppConfigFile(non-global,non-subdir): /home/TRon/.config/test.cfg
ApplicationName                        : test
SysConfigDir                           : /etc
VendorName                             :.
ConfigExtension                        : .cfg


[furious programming]

@Kays and @TRon: thanks for the explanation.

I think the best solution would be to just use the SDL_GetPrefPath function and save the data wherever SDL comes up with. Although this function returns the path of a given subdirectory located in %APPDATA% (under Windows), but since that's what the developers of SDL have decided, I think they can be trusted.

So to sum up, now the log file initialization and finalization code looks like this:

Code: Pascal  [Select][+][-]

1. var
2.   LogCreated: TGame_Bool = False;
3.  
4. procedure Game_LogOpen();
5. begin
6.   {$PUSH}
7.   {$IOCHECKS OFF}
8.   AssignFile(System.Output, SDL_GetPrefPath('organization name', 'application name') + 'log.txt');
9.   Rewrite(System.Output);
10.   {$POP}
11.  
12.   LogCreated := IOResult() = 0;
13.  
14.   if not LogCreated then
15.     System.Output := System.StdOut;
16. end;
17.  
18. procedure Game_LogQuit();
19. begin
20.   if LogCreated then
21.     CloseFile(System.Output);
22. end;

After initialization, I can easily write diagnostic data with WriteLn, without having to specify a file variable in the first parameter (which is convenient). Of course, in my project I have done so that if the game itself opens the system console, diagnostic data is displayed in the console, and the log file is not created.


One question still remains — since {$APPTYPE} is intended only for Windows, how to force the automatic creation of the system console from the code level? Do I have to allocate the console manually or are there any switches for that?

[TRon]

One question still remains — since {$APPTYPE} is intended only for Windows, how to force the automatic creation of the system console from the code level? Do I have to allocate the console manually or are there any switches for that?

The easiest answer is yes. You need to create the console yourself and route the input/output/error.

However I would like to mention a few considerations that you could perhaps think about.

On Linux the console/terminal is not automatically created for you (by FPC/Lazarus). That requires you to run the users default terminal and reroute the input/output/error handles manually.

You can however 'run' a command from a desktop environment and configure it in such a way that a terminal is automatically opened when a executable is 'launched'.

So, in Linux it is more the (end) user who decides whether or not (s)he wants to start your 'game' from a terminal or directly from a desktop link. In case the latter the user can also decides if a console is wanted/required or not.

Another thing you might wish to consider is whenever the 'game' is launched in full-screen mode. In that situation a terminal window is not visible to the user. That is why some (SDL) games integrate their own custom console window or use something like a virtual terminal (f.e. see https://sourceforge.net/projects/sdl-terminal/ , https://github.com/rlt3/SDL-Console or https://github.com/Crypto2/SDL_console).

Do note however that I do not have any experience with either of the mentioned SDL consoles/terminals so your mileage may vary.

I take it that you are aware of the logging functionality of SDL itself, if not then have a look at that as well. You can reroute the default logging function so that you could for example route it to your own routine that uses write(ln). Whether or not you would then direct that write to a file, socket or terminal is up to you.

[furious programming]

The easiest answer is yes. You need to create the console yourself and route the input/output/error.

In that case, the current solution for Windows can stay because it meets my requirements, while for other platforms a different solution will be selected. Especially considering what you wrote about Linux.

So, in Linux it is more the (end) user who decides whether or not (s)he wants to start your 'game' from a terminal or directly from a desktop link. In case the latter the user can also decides if a console is wanted/required or not.

So, just like under Windows — you can also run the game directly from the disc or from the console, using the appropriate command. The only problem is that currently the build determines whether the Windows console is to be created, and on this basis the game either uses the console to log in, or creates a log file (both whether the program was launched from the disk/IDE or with a command from the console).

However, this means that if I run a build that does not create a console from an external console, the data will not be logged in the console, but in a file. A solution would be useful to check how the game was launched (from the disc or from the console) and if the console is available, the data should be logged to it.

I need the automatically created console only for debugging, but it would be good if the player could specify where the data will be logged. This will not only come in handy on Windows, but also on other platforms.

Another thing you might wish to consider is whenever the 'game' is launched in full-screen mode. In that situation a terminal window is not visible to the user.

The data that is to be ultimately logged in the console or in the file will be only diagnostic data that will be useful to me and the players only when the game works incorrectly — it will be helpful in looking for information that allows me to track bugs and fix them.

Do note however that I do not have any experience with either of the mentioned SDL consoles/terminals so your mileage may vary.

Ultimately, as far as the final product is concerned, the game should not use the console at all, unless the player deliberately launches it from the console so that they can see what is being logged. That would be an extra feature.

And that's why it would be nice if the game could detect from the code level whether the console is available, and on this basis determine whether to create a log file or not. If anyone knows how to check this (for Windows and other platforms), I'd be grateful.

Edit: I found this — http://kriscode.blogspot.com/2018/02/console-vs-gui-application-in-op.html — I will check it later.

I take it that you are aware of the logging functionality of SDL itself, if not then have a look at that as well. You can reroute the default logging function so that you could for example route it to your own routine that uses write(ln). Whether or not you would then direct that write to a file, socket or terminal is up to you.

Yes, but I don't like the SDL API and the log format, so I prefer to use what Free Pascal offers me. I prefer to choose WriteLn, because it allows me to log any data and format it freely, in a simple and convenient way. Such a custom logging system, as you can see, is only a dozen or so lines of code, and the rest is WriteLn calls to log data. There is nothing wrong with doing such a simple thing yourself, especially since it gives you complete freedom.

[TRon]

i understand where you are coming from furious programming so you do not really have to elaborate on your decisions for every answer, although it provides a little more insight on what it is you are looking for exactly.

However, this means that if I run a build that does not create a console from an external console, the data will not be logged in the console, but in a file. A solution would be useful to check how the game was launched (from the disc or from the console) and if the console is available, the data should be logged to it.

understood.

I need the automatically created console only for debugging, but it would be good if the player could specify where the data will be logged. This will not only come in handy on Windows, but also on other platforms.

The tricky part lies in the automatic creation because what console (read terminal) to launch depends on so many variables.

But let's try to keep it simple and not complicate matters (that was actually a note to myself)

One thing you can be sure about is that on startup of your game you are unable to determine whether a terminal/console is available or not (please do not read "was launched from" but rather /is available anywhere/).

However, you could determine if your game was launched from a terminal or not. isatty comes to mind (should also be available on windows but windows has some other convenient API's such as GetConsoleMode etc that are able to determine the same status').

Initially I made some test that depended on locating the parent process and see what executable was responsible for that process. That brought some tears into my eyes because I used about a dozen different launch methods and got exactly the same amount of different answers in return.

In short it is unreliable unless you wish to account for every program launcher that is available for Linux.


So, instead I had to use come up with another approach (see last part of my message)

Ultimately, as far as the final product is concerned, the game should not use the console at all, unless the player deliberately launches it from the console so that they can see what is being logged. That would be an extra feature.

And that's why it would be nice if the game could detect from the code level whether the console is available, and on this basis determine whether to create a log file or not. If anyone knows how to check this (for Windows and other platforms), I'd be grateful.

See my earlier remark about isatty.

But there is a catch as the user can route/pipe the output in which case the isatty check alone will not be helpful enough. But things could be refined if you so wish to do so.

Edit: I found this — http://kriscode.blogspot.com/2018/02/console-vs-gui-application-in-op.html — I will check it later.

Yeah, I've read that.

I have no idea what the writer means with the buzz around his discovery because all that the writer does is check for the presence of a commandline argument and in case it is present then the writer mistakenly use that as a fact that either the program was launched from either a desktop or from a terminal/console and uses that as a check to either start the GUI part of the program or only the non-gui bits.

Besides the fact that I can start a application from a desktop environment /and/ provide it with parameters without using a console/terminal there also is no secret or unwritten behaviour that can be relied on (other then the presence of a commandline argument)


However, the path I would like to suggest is the following:
- use api calls like isatty to determine if your game was launched from a terminal/console
- in case not launched form a terminal/console /and/ user requests (commandline argument ?) debug output
  then either:
  a) use a pre-game-launcher executable that launches the rest of the game using TProcess and that uses bash/sh/cmd.exe to launch the main executable.
  b) do not use a pre-launcher executable but use the same logic to execute the same executable again but in a detached manner (using the same methodology as described above).
  (you could for example use the presence (or not by removing it from second launch) of the argument to determine in which launch 'phase' your game/executable resides.

That way you have control over whether or not /which/ terminal/console needs to be launched and when exactly.

Could something like that be an option for you ?