Recent

Author Topic: Looking to add a new compiler directive (ARRAYMAJORORDER)  (Read 3886 times)

Murmandamus

  • New member
  • *
  • Posts: 8
Looking to add a new compiler directive (ARRAYMAJORORDER)
« on: October 15, 2014, 08:55:17 am »
Greetings,

I am doing some game engine development, specifically working on a more comprehensive version of the Matrix unit, and have discovered some potential performance issues that would be easily solved if I could control the major order of array access for multidimensional arrays (matrices).

For example, OpenGL/ES expects matrices to be in Column-Major order, and DirectX expects them in Row-Major order. I can do some conditionals which let me select which way some of the matrix operations are done so that it is somewhat seamless at compile-time, but I would like to be able to have Array types which can be compiled either way in the same code (so I can have both a DirectX renderer and an OpenGL renderer built-in to the same engine, for example).

I have done a little bit of research about how I would like to go about it, basically, adding a flag to an array type declaration in the symbol table to specify whether each type is either column-major or row-major, and use a new compiler directive  {$ARRAYMAJORORDER COLUMN|ROW} (and/or perhaps{$AMO COLUMN|ROW } for short), and then modify the code generation so that when array indices are composed into the linear memory offset, it checks this flag in the type, and generates the appropriate code to reference the correct physical storage layout of the multidimensional array. I am not certain exactly how FPC does this right now, as it is what I want to research next, but I am not as familiar with the source as some here may be and would like some feedback and perhaps some pointers towards which parts of the source files may be relevant for this kind of patch. I am happy to contribute it (as well as the new Matrix unit I am working on) when I am done for a future version of FPC.

I am kind of looking to this as a first minor test project to getting my claws dirty working with the guts of the compiler.

Thanks in advance for any info!

engkin

  • Hero Member
  • *****
  • Posts: 2513
Re: Looking to add a new compiler directive (ARRAYMAJORORDER)
« Reply #1 on: October 15, 2014, 05:22:42 pm »
The other day I came across this file. It talks about the internals of an older version of FPC. Hope it helps.

Murmandamus

  • New member
  • *
  • Posts: 8
Re: Looking to add a new compiler directive (ARRAYMAJORORDER)
« Reply #2 on: October 15, 2014, 09:56:41 pm »
Thanks! Looks like it has some good information. Might be a little dated, but anything helps. :)

I have been pouring over he compiler source a bit today, looking for potential areas that may be impacted by this kind of change. I found the definitions for the array type (TArrayDef) in symdef.pas, and note that there is an array options bitset for it (TArrayDefOptions) in symconst.pas which I think I can add a flag (something like ado_IsColumnMajor; RowMajor would be the default case). That bitset is written out to the ppufile as a small set (up to 32 elements; it currently has 8 defined), so it shouldn't impact the ppufile format any.

I wasn't sure exactly how multi-dimensional arrays were defined, but it makes sense that they are simply handled as nested array definitions, which matches the syntactical options:

Type
  A1 = Array[0..3,0..3] of Single;
  A2 = Array[0..3] of Array[0..3] of Single;

Which are both valid and equivalent.

The next step is to look at the array node generation and then the code generation for it to deal with reversing the traversal order of the array indices when computing the linear offset if the flag is set, which I think is the route to go right now. I can already see a few special cases that I need to be mindful of, like the is_vector case, and MMX-enabled optimizations for small static arrays.

User137

  • Hero Member
  • *****
  • Posts: 1790
    • Nxpascal home
Re: Looking to add a new compiler directive (ARRAYMAJORORDER)
« Reply #3 on: October 15, 2014, 10:18:51 pm »
Few points:
- If you have the array type switch at compile time, you can't at runtime switch between DirectX or OpenGL. (Sure you can still use a launcher that starts a specific executable for either case)
- Some define a 4x4 matrix as array[0..15] of single. From that you can't distinct the order, since compiler doesn't know row length.

Murmandamus

  • New member
  • *
  • Posts: 8
Re: Looking to add a new compiler directive (ARRAYMAJORORDER)
« Reply #4 on: October 15, 2014, 11:40:48 pm »
Heyas User137! :)

Pardon in advance for the stream-of-consciousness thinking -- this is developing as I research it.

Few points:
- If you have the array type switch at compile time, you can't at runtime switch between DirectX or OpenGL. (Sure you can still use a launcher that starts a specific executable for either case)

Actually, if the difference is at the type definition level, you can have types of both major orders in the same executable, even in the same unit. It will require two definitions for base matrix objects of each major order, but the matrix manipulation code can be shared. I realize that actual dereferences for the matrix array will have to be one or the other, but I think I can get around that issue. For example, consider this code:

Type
{$ARRAYMAJORORDER COLUMN }
  TMatrix4x4CM = Array[0..3,0..3] of Single;
{$ARRAYMAJORORDER ROW }
  TMatrix4x4RM = Array[0..3,0..3] of Single;

Var
  A1 : TMatrix4x4CM;
  A2 : TMatrix4x4RM;
...

// Set Translation parameters

Procedure SetOpenGLTranslate(X,Y,Z: Single);

  Begin
  A1[0,3] := X;
  A1[1,3] := Y;
  A1[2,3] := Z;
  End;

Procedure SetDirectXTranslate(X,Y,Z: Single);

  Begin
  A2[0,3] := X;
  A2[1,3] := Y;
  A2[2,3] := Z;
  End;

...

Obviously, I can't directly dereference one type and have it act as the other, but typecasting like this should work:

If UseDirectX Then
  Begin
  TMatrix4x4RM(A1)[0,3] := X; // stores using Row-Major, even though A1 is Column-Major in default accesses.
  TMatrix4x4RM(A1)[1,3] := Y;
  TMatrix4x4RM(A1)[2,3] := Z;
  End;

Also, as a fallback, I think I can use object inheritance to unify Matrix types, even adding an element access function or method, which I may be able to inline. The existing Matrix unit uses this approach somewhat.

Even if I have to specify the major order unit-wide, it would still be fine; I expect that the API-specific interface parts of the engine will be in separate units for different APIs. I mainly want to avoid the performance hit in having to constantly transpose the matrix before making the API calls for one API (OpenGL normally, since it is the opposite of the current default).

-----

Quote
- Some define a 4x4 matrix as array[0..15] of single. From that you can't distinct the order, since compiler doesn't know row length.

This proposed change is intended for explicitly-declared multi-dimensional arrays. If someone wants to use a linear array and do their own linear matrix element offset calculations, it should not impact their ability to do so, regardless of what the compile-time option is set to (or that it exists at all).

Many thanks for the valuable input, as usual. It helps me a lot to think through the issues and potential pitfalls. :)