FPC Unleashed (async/await, parallel for, match, string interpolation & more)

[creaothceann]

Looks like this calls for a separate directive - {$incfilebytes} - emitting an array of bytes instead of a String.

On a side note: on the devel branch {$incfile} already got an upgrade - besides the two-argument form it now also takes a single argument (just the path) and pastes the string literal right where the directive sits, so you can write things like call({$incfile 'path'}). It'll be a while before that reaches main, though.

Meanwhile I'll think through {$incfilebytes}. If you have any thoughts on the name, behavior, element type and so on - go for it.

One option would be

    const Identifier : {$array of Type from Path}

Type would be any type that byte/word/dword/qword can be cast to, and behind the scenes it'd be transformed to

    Identifier : array[{...}] of Type = (FileContent0, FileContent1, FileContent2, ... , FileContentN)

FileContentX would be for example $33221100 for a dword-compatible type.

- - -

Another option would be extending the existing {$include} directive like this:

    {$include Path as Type}

This has the most flexibility (could be any type, doesn't have to be an array), but would require a bit more typing for arrays, a few checks that the sizes fit, and (unfortunately) more than a relatively simple source code transformation behind the scenes. I assume you'd basically have to implement a new literal.

Speaking of literals, afaik the " character is still unused in Free Pascal so the code (a regular language feature, not a directive) could look like this:

    const Identifier : Type = "c:\path\to\file.ext";

(Might be easy to be mistaken at a glance as a string constant?)

[440bx]

Feature request:

A "requires" clause in a function or procedure declaration to declare that using the function should require using another function. 

For instance, the Windows API function GetEnvironmentStrings requires using FreeEnvironmentStrings to free the block of memory GetEnvironmentStrings allocates.  The requires clause would simply act as a reminder.  if a function calls GetEnvironmentStrings and does not call FreeEnvironmentStrings in the same scope then the compiler would emit a cautionary note about the call being required. 

The full syntax would be something along the lines of:

requires_clause       ::= "requires" subprogram_identifier "reason" string_const_expression ";"

where the "reason" string would become part of the cautionary message if the required function call is absent.

Note, that the mechanism is not Windows specific, it is general and could be used to declare that a program's internal function should be followed by a call to another function or procedure for reasons that have nothing to do with freeing resources.  In other words, it can be used to declare any kind of dependency between two functions/procedures and have the compiler emit a "reminder" if the required function is not called.

[Martin_fr]

A "requires" clause in a function or procedure declaration to declare that using the function should require using another function. 

For instance, the Windows API function GetEnvironmentStrings requires using FreeEnvironmentStrings to free the block of memory GetEnvironmentStrings allocates.  The requires clause would simply act as a reminder.  if a function calls GetEnvironmentStrings and does not call FreeEnvironmentStrings in the same scope then the compiler would emit a cautionary note about the call being required. 

Just from a technical side... That seems "awfully" incomplete...

I don't know about you, but I can attest that a great many people have setups were such a "required" function is called in another place. E.g a "setup" and "teardown" pair of functions.

So, that would mean (much alike declaring what exceptions might be thrown in some other language), that requirement can be forwarded.

E.g.
- "setup" would have to be able to say: I require "Teardown" and it will fulfil FreeEnvironmentStrings.
- Or "setup" just says, I require "Teardown", and Teardwon says, I fulfill FreeEnvironmentStrings for any one requiring me.

Also some functions may have several options of finalization, so they may need to say require Foo or Bar.

------------------
If that sounds over engineered to anyone... No problem. I am just pointing out a few cases that may or may not be of interest to anyone who would want the notifications...

For real simple code, my concerns may not matter, but then the overall need for the feature in first would not be great either (I assume). For any more involved code, the feature would have to cover common situations...

[440bx]

It's very likely that there are many cases where the "required" clause is too little but, there are many cases, usually the simple ones, where all that's needed is a reminder that some other function should be called in the same scope.  That's the simple and common case, that's what "require" is meant for.

For more complex cases, it really is up to the programmer to determine if "requires" is practical or not.

In the case of the Windows GetEnvironmentStrings, the darn thing allocates memory behind the programmer's back, that by itself is far from evident and even if the programmer knows it  does that, it is rather easy to forget.  Basically, if the GetEnvironmentStrings declaration has a "requires" clause associated with it, the compiler will remind the programmer that without a call to FreeEnvironmentStrings, a memory leak will occur.  That's a very nice reminder that the compiler can emit with extremely little effort.   It should also be noted that GetEnvironmentStrings is far from the only Windows API function that requires subsequent actions in order to not leave things hanging.  It would be extremely nice and very useful to the programmer, if those functions were clearly identified.  That would relieve the programmer from checking the documentation for every API just in case, to ensure no required "detail" was missed.

[Fibonacci]

@Thaddy

That's exactly the point - I don't want to run a tool for this. With data2inc/bin2obj I'd end up with two files instead of one (the original + the generated include), and I'd have to remember to regenerate it every time the original changes. Too much hassle. With the directive the original is the single source of truth - no extra step.

And a resource script? Resources are trivial to swap out in the finished binary - anyone can replace them. Baked into the code as a const it's not that easy.


@Khrys

You come up with something and then find out it already exists... happens with everything

The byte include is happening regardless, it's just a matter of settling the directive names. Either:

{$incfile} - as a string (current)
{$incfilebytes} - as bytes (maybe)

...or a full rework that'd even make @creaothceann happy: extend the existing {$include} / {$i} by argument count:

• 1 arg = normal include (as today)
• 2 args = {$include <type> <path>} -> in-place include, as <type>
• 3 args = {$include <type> <name> <path>} -> a const <name> of <type> from the file

If <type> is anything other than string, you get an array of <type> (byte/word/dword, ...); string itself gives a plain string.


@creaothceann

What if the file size isn't a multiple of the element type (7 bytes as u16)?


@440bx

Drop some pseudocode of how you picture it - a modifier on the import declaration? And who annotates all the API units (Windows, etc.) with these? Plus, as @Martin_fr said, the free can be in another scope, so the in-scope check won't always hold up.

[creaothceann]

@creaothceann

What if the file size isn't a multiple of the element type (7 bytes as u16)?

Free Pascal seems to be fine with filling extra bits with zeroes, and showing "Warning: Type size mismatch, possible loss of data / range check error" when there's more bits of data than what would fit into the constant, but for this file inclusion feature I think it should be a compilation error in both cases.

[Khrys]

extend the existing {$include} / {$i} by argument count:

• 1 arg = normal include (as today)
• 2 args = {$include <type> <path>} -> in-place include, as <type>
• 3 args = {$include <type> <name> <path>} -> a const <name> of <type> from the file

If <type> is anything other than string, you get an array of <type> (byte/word/dword, ...); string itself gives a plain string.

I like this solution the most. It's simple and self-explanatory, IMO:

Code: Pascal  [Select][+][-]

1. {$include shader_defs.inc}
2.  
3. {$include byte TEX_VIGNETTE 'textures/vignette.png'}
4. {$include string SHADER_POST_VERT 'shaders/post.vsh'}
5.  
6. const
7.   SHADER_POST_FRAG = {$include string 'shaders/post.fsh'};

I still wonder though how  const TEX_VIGNETTE = {$i byte 'textures/vignette.png'};  could be implemented in the compiler, but admittedly I'm not up to speed on FPC Unleashed's type inference capabilities.

[Thaddy]

@Khrys

That option requires a rewrite of the {$ parser, because currently it only accepts the first parameter after the define/include and the rest is treated as comment. IOW may be not as easy as it looks.

[Thaddy]

@Thaddy

That's exactly the point - I don't want to run a tool for this. With data2inc/bin2obj I'd end up with two files instead of one (the original + the generated include), and I'd have to remember to regenerate it every time the original changes. Too much hassle. With the directive the original is the single source of truth - no extra step.

And a resource script? Resources are trivial to swap out in the finished binary - anyone can replace them. Baked into the code as a const it's not that easy.

Does not matter with a binary patcher. (Delphi Pascal code in the old Dream Components)
But anyway, I am more worried about distinguishing between the string and the binary form: it should not be necessary to do any encode/decode when the format is just in byte form. You suggest the string form needs that, which is not the case. Where is the logic in that?
If you can skip that, it is otherwise fine with me.

[Fibonacci]

That option requires a rewrite of the {$ parser, because currently it only accepts the first parameter after the define/include and the rest is treated as comment. IOW may be not as easy as it looks.

Arguments in directives are simple. The fact that the rest is "treated as comment" is exactly what makes it easy - you grab that comment text and split it yourself. Here's the top of {$incfile} on the devel branch, handling both the 1-arg and 2-arg forms:

Code: Pascal  [Select][+][-]

1.       begin
2.         current_scanner.skipspace;
3.         args := current_scanner.readcomment;
4.         token1 := GetToken(args, ' ');
5.         if token1 = '' then
6.           begin
7.             Comment(V_Error, 'incfile: missing arguments');
8.             exit;
9.           end;
10.         token2 := GetToken(args, ' ');
11.         if token2 = '' then
12.           begin
13.             expr_only := true;
14.             varname := '';
15.             pathspec := TCmdStr(token1);
16.           end
17.         else
18.           begin
19.             expr_only := false;
20.             varname := token1;
21.             pathspec := TCmdStr(token2);
22.           end;
23.  

readcomment grabs the rest, GetToken splits on spaces. No parser rewrite involved.

Does not matter with a binary patcher.

Only if the replacement is the same size or smaller than the original. Resources, on the other hand, you can swap for anything.

But anyway, I am more worried about distinguishing between the string and the binary form: it should not be necessary to do any encode/decode when the format is just in byte form. You suggest the string form needs that, which is not the case. Where is the logic in that?
If you can skip that, it is otherwise fine with me.

For me the string form is actually the more convenient one. Take the Base64 unit - EncodeStringBase64 / DecodeStringBase64 both take a string, one call and done. There's no one-shot EncodeBytesBase64 or similar: with an array of bytes you either convert it to a string first, or go through the stream classes - create an instance, write, read, free. Either way, a lot more typing for the same result.

To be clear, I never said the string form needs encode/decode - it doesn't, and neither does the byte form; the byte form has nothing to do with encoding. I'd happily leave {$incfile} string-only. The only reason I'm adding it is that absolute doesn't work over a string const (it's a managed pointer), which is the overlay @creaothceann wanted.

[Martin_fr]

@440bx

Drop some pseudocode of how you picture it - a modifier on the import declaration? And who annotates all the API units (Windows, etc.) with these? Plus, as @Martin_fr said, the free can be in another scope, so the in-scope check won't always hold up.

IMHO you need something like

Code: Pascal  [Select][+][-]

1. procedure Alloc; demands Free;
2. procedure Alloc; demands Free 'You need to call UnAlloc after Alloc'; // optional message

"Free" is just a token, it could be "FooBar" => the fulfiller needs to give the same token => the fulfiller must be in the same unit, or the token must be fully qualified. The token is only valid within the require/provide modifiers, and does not conflict with any other identifier used in the sources (not even the name of the function on which it is used.

"provides Free" in a different unit, would be a different token, and not match the above. It would need to be "provides AboveUnit.Free"

Code: Pascal  [Select][+][-]

1. procedure UnAlloc; provides Free;
2. procedure SaveUnAlloc; provides Free;

Either of the 2 can be used...
If a message is given, it will be used if the the call is made without matching "Alloc". If no message is given dangling calls will not trigger a warning at all.

Code: Pascal  [Select][+][-]

1. procedure Setup; demands Free forward;
2. procedure Setup; demands Free forward 'call teardown or otherwise Free the alloc'; demands Xyz forward;
3. procedure Setup; demands Free, Xyz forward; // OPTIONAL allow bundling of forwards, same as giving both individually
4. procedure Teardown; provides Free forward;

There can be multiple forwards. Setup could forward Free and maybe also ReleaseHandle and others.

Also, if the above signifies that "Setup" can contain a call to "Alloc" that is not freed inside setup, then the question is how many (or should count be kept)?

Code: Pascal  [Select][+][-]

1. procedure Setup; demands Free forward 3; // has 3 calls that it does not resolve, if there are more, then they must be matched within setup / if there are less that is a warning too
2.  

A forward can be matched by an unforwarded call too. E.g. you could call "Alloc" direct, and then Teardown to release it. Or call Setup and directly Unalloc. (that could be controlled by the "as" below)


Optional, but probably would be more a source of errors than help, rename and bundle

Code: Pascal  [Select][+][-]

1. procedure Setup; demands Free, Release forward as teardown // contains 1 call that requires Free and 1 that requires Release, and wants them fulfilled by a provision of "teardown"
2. // This does not work with a number after the "forward", so either tokens need to be repeated, or the number must be after each token
3.  

So if the caller of that "setup" calls UnAlloc directly, that will not fulfil it. It needs to call

Code: Pascal  [Select][+][-]

1. procedure Teardown; provides Free, Release forward as teardown; // The list of provides must match that of the require alias. Otherwise its not valid
2. // That means either "forward as FOO" can have overloaded FOO for different combinations. Or giving a diff combination should be an error.
3.  

This would differ from the below, which also demands teardown, but the 2 calls could be resolved elsewhere. (teardown might not provide them)

Code: Pascal  [Select][+][-]

1. procedure Setup; demands Free forward; demands Release forward; demands teardown;
2.  

There is on checking on matching arguments of the paired calls. They just need to be present (and present in the correct number)

The compiler would always keep count of the demands for each token, and at the end of each routine it would check that they equal the amount of forwards. (or zero if no forwards are declared).
It doesn't care which call matches with other, if there are multiple calls (as variables can have had their content changed...)

Note that any top level code (main begin /end) needs to check finalize sections. (and initialization sections may have had something too).

[Martin_fr]

Further more, this needs to be correctly handled for virtual methods.

Since the compiler does not know which inherited message is called at some point, then all inherited methods must demand, provide and forward the exact same set of tokens.

That somewhat defies the natural expectation that an overridden method should be able to require or provide more (but never less / yes it could have freed the result of the inherited call, but that would be outside the expectations).

But if the overridden TMyClass.Foo has a different set, then the compiler can't keep track if MyInstance.Foo is called as it does not know which class.

Yet, it can be partly solved by

Code: Pascal  [Select][+][-]

1. type TMyClass = class(...)
2.   procedure Foo; virtual; demands a,b as UnFoo of object;
3.   procedure Bar; virtual; provides a,b as UnFoo of object;
4.  

The "of object" means that both methods must be on the same class, and on that class must match their list.

However subclasses can override them both and change the list "a,b" but keep "UnFoo of object".

The compiler can check that within the virtual or overridden method, there is the correct amount of forward-able unpaired calls.
On the caller side, it only needs to track the correct amount of "UnFoo"

It isn't 100% safe, as "Bar" could be called on a different variabel (which may be a copy of the same object, a diff object of the same class, or a diff object of different class altogether. (but that might still be correct)

In this case Unfoo is not limited to the unit. It is limited to all inherited classes (the class tree).

[Martin_fr]

And optional, there might be some thought on generics.

Code: Pascal  [Select][+][-]

1. generic Xyz<A> = // class, function, ...

could then contain

Code: Pascal  [Select][+][-]

1. A.SomeThing()

If SomeThing would have a require then its ok if that is resolved within. But what if it needs forwarding.

[Fibonacci]

@Martin_fr

You've complicated it so much it almost kills the urge to do it at all That's a ton of machinery for what's basically a "reminder" to call the right Free.

I was under the impression this was only meant for library imports - flagging external APIs like GetEnvironmentStrings - not your own functions/procedures/methods.

[Martin_fr]

@Martin_fr

You've complicated it so much it almost kills the urge to do it at all That's a ton of machinery for what's basically a "reminder" to call the right Free.

I was under the impression this was only meant for library imports - flagging external APIs like GetEnvironmentStrings - not your own functions/procedures/methods.

Well, the origin actually does not matter. A library call to GetEnvironmentStrings can be anywhere within the code, including a virtual method. And the release can be anywhere else. And then you need the forwards.

The only other option is, that a flag is added to suppress the warning. That is, if "Procedure abc" does call GetEnvironmentStrings, but not the release, then its just flagged as  (could also be the code body)

Code: Pascal  [Select][+][-]

1. procedure abc; require GetEnvironmentStrings is ignored;

That somehow feel incomplete to me...

On the other hand, even with the forwards, there may be cases where an ignore is needed. So any subset will do in combination with an ignore option.

Anyway, as I said before: I am just writing that as an observer. I.e. if my thoughts do help, well fine. If they don't, it wont hurt me, I am not going to be affected.