Coroutine manager for Free Pascal

[OpenSIMPLY]

COMTAY is free shared library for creating asymmetric stackful coroutines and co-methods.

It is applicable to both procedural and class-based programming to implement cooperative multitasking.

COMTAY currently supports the calling conventions of Free Pascal, Delphi, MS Visual C++, and GNU C++.

It runs on 32- and 64-bit platforms on Windows and Linux.

Try it.

Homepage: https://opensimply.org/comtay

[MarkMLl]

Without wanting to belittle it, also see https://forum.lazarus.freepascal.org/index.php/topic,56072.msg418466.html#msg418466

This turns out to be an enormously complex endeavour, at least in part because of modern languages' support for exceptions (I don't know whether OP's try/finally aggravates the situation).

MarkMLl

[OpenSIMPLY]

You've exactly pointed out one of the problems with the coroutine libraries I've tested so far. (In fact, they didn't work at all.)

When creating COMTAY , this and much more was taken into account.

The case of exceptions, and the case of calling functions such as TFileStream.Create, can easily be bypassed with the COMTAY stack switching functions.

I've attached an example of using exceptions with COMTAY (don't forget to include the correct path to the comtay.pas file in the project code).

[Warfley]

I just tried this out, and wanted in your example to just switch to another task:

Code: Pascal  [Select][+][-]

1. var
2.   CT1: TCoTask1;
3.  
4.   procedure TCotask2.Body;
5.   begin
6.     WriteLn('Task 2');
7.   end;
8.  
9.   procedure TestException;
10.   var
11.     F: file;
12.   begin
13.     try
14.       CT1.Resume(TCotask2.Create);
15.       AssignFile(F, 'TheFileThatDoesNotExist');
16.       Reset(F);
17.     except
18.       on e: EInOutError do
19.         WriteLn('Error: ' + E.ClassName + EOL + E.Message);
20.     end;
21.   end;
22.  
23.   procedure TCotask1.Body;
24.   begin
25.     pJoinThreadStack();
26.     TestException;
27.     pJoinNativeStack();
28.   end;

But this gives the following error:

Code: Pascal  [Select][+][-]

1. Error #24: Rejected while joined non-native stack

While I am not sure how this stack management actually works, but through a bit trial and error I've found it to be the "pJoinThreadStack()" call that causes it, and when removing, the switch to the second task works, but the exceptions don't work anymore.

Could you maybe elaborate a bit on how the stack handling is done and what these JoinXXXStack functions do?

[OpenSIMPLY]

I just tried this out, and wanted in your example to just switch to another task:

But this gives the following error:

While I am not sure how this stack management actually works, but through a bit trial and error I've found it to be the "pJoinThreadStack()" call that causes it, and when removing, the switch to the second task works, but the exceptions don't work anymore.

Could you maybe elaborate a bit on how the stack handling is done and what these JoinXXXStack functions do?

Thank you for your interest in COMTAY. Could you please provide me with the complete code of your project.

[Warfley]

It's just a slight variation from your example above, just adding another task to be run within the first task

[OpenSIMPLY]

It's just a slight variation from your example above, just adding another task to be run within the first task

Thank you. The problem is that you are using the Resume() function out of native task (coroutine) stack.

Code: Pascal  [Select][+][-]

1.   procedure TestException;
2.   var
3.     F: file;
4.   begin
5.     try
6.       CT1.Resume(TCotask2.Create);                           //   Here is the stack of the thread where the ResumeAsMain was called.
7.       AssignFile(F, 'TheFileThatDoesNotExist');
8.       Reset(F);
9.     except
10.       on e: EInOutError do
11.         WriteLn('Error: ' + E.ClassName + EOL + E.Message);
12.     end;
13.   end;
14.  
15.   procedure TCotask1.Body;
16.   begin
17.     pJoinThreadStack();
18.     TestException;
19.     pJoinNativeStack();
20.     Resume(TCotask2.Create);      // <- Here is the stack of the task (coroutine)
21.   end;
22.  

If that doesn't explain the problem, feel free to ask again. 

[Warfley]

What exactly does native task stack mean? When looking for example at the WinAPI Fibers, as Windows native coroutines, there each Fiber has it's own stack, and when switching to another fiber, execution will continue on the stack of that fiber.

When I understand correctly, for COMTAY this works differently, but I don't fully understand how. Could you maybe explain a bit further, how does each co routine handle it's stack, and how to switch from one co routine stack to another (i.e. in this example, how to switch over to CoTask2 within the execution of CoTask1)

[OpenSIMPLY]

What exactly does native task stack mean? When looking for example at the WinAPI Fibers, as Windows native coroutines, there each Fiber has it's own stack, and when switching to another fiber, execution will continue on the stack of that fiber.

When I understand correctly, for COMTAY this works differently, but I don't fully understand how. Could you maybe explain a bit further, how does each co routine handle it's stack, and how to switch from one co routine stack to another (i.e. in this example, how to switch over to CoTask2 within the execution of CoTask1)

A COMTAY task uses its own separate stack such as Fibers. When some A task resumes some B task, then after a context switching the B task uses its own stack. So, COMTAY task  behaves like a typical stackful coroutine.

The main task is resumed using the ResumeAsMain or ResumeAsMainExtra function. This function must be called from the standard thread stack as a normal function.
If the main task completes or terminates, control returns to point right after the call to the ResumeAsMain or ResumeAsMainExtra function.
From this address point and below, the thread stack is unused until the main task is complete or terminated. This is the idle thread stack area. Despite this area must not be used, some compilers can use it.   

When the pJoinThreadStack function is called within a task, the task stack is switched over to the thread stack area. This allows exceptions to be thrown and constructors such as TFileStream.Create to be called. The code for such things contains some thread stack fixed addresses and cause a critical error when called from the task (coroutine). 

After some actions in the thread stack area, the own task stack (native stack) must be returned back. That is done with the pJoinNativeStack function.

Now the answer to the example.
In order for TCoTask1 to resume TCotask2, you need to call  Resume(TCotask2.Create). The task stack must be its own TCoTask1 stack (native stack). Calling the Resume function while thread stack causes an error.

So, I put Resume(TCotask2.Create) in the TCotask1.Body and comment out in the TestException.

This can be seen in the WithExceptions_2 example that I uploaded.

Code: [Select]

  procedure TestException;
  var
    F: file;
  begin
    try
     // CT1.Resume(TCotask2.Create);
      AssignFile(F, 'TheFileThatDoesNotExist');
      Reset(F);
    except
      on e: EInOutError do
        WriteLn('Error: ' + E.ClassName + EOL + E.Message);
    end;
  end;

  procedure TCotask1.Body;
  begin
    pJoinThreadStack();
    TestException;
    pJoinNativeStack();
    Resume(TCotask2.Create);      // <-
  end;

If it's still not clear, feel free to ask again.

[Warfley]

Ok I think I understand now. Thats an interesting approach, because when I wrote my co-routine library STAX I also had to deal with the problem of how to get exceptions (and the implicit exception handlers from constructors) running, and how I solved it was to simply create a patch for the FPC which allows to manipulate the exception state, allowing each fiber to have it's own exception state associated with their stack.

This of course does not work when creating an external library that can be used by different languages and compilers, so I was curious on how you solved it. Defenetly an interesting project.