* * *

Author Topic: Dynamic array extensions ?  (Read 4505 times)

Nitorami

  • Sr. Member
  • ****
  • Posts: 330
Dynamic array extensions ?
« on: June 06, 2018, 11:11:25 pm »
I would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short:

- support for array constructors using "[...]" syntax
- support for Insert(), Delete() and Concat()
- support for "+" operator
- support for dynamic array constants (and variable initializations)

Sounds ok, however: The "+" operator is effectively an alias for concat(), and it will be an intrinsic, which means that existing "+" operator overloads for dynamic arrays will no longer compile. Intrinsic operators cannot be overloaded, hence it will no longer be possible to custom define the "+" operator to do a mathematical elementwise addition of numerical vectors/matrices.

With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied.

What is your opinion on that  ?

Phil

  • Hero Member
  • *****
  • Posts: 2734
Re: Dynamic array extensions ?
« Reply #1 on: June 07, 2018, 03:13:58 am »
I would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short:

- support for array constructors using "[...]" syntax
- support for Insert(), Delete() and Concat()
- support for "+" operator
- support for dynamic array constants (and variable initializations)

What is your opinion on that  ?

Sounds like Pascal array syntax is getting dragged into the 21st century so it's more like other languages. Take the Swift MLPClassifier example givenhere (under Interpreter):

https://github.com/tensorflow/swift/blob/master/Usage.md

That gives several examples of constant arrays being declared within the code that uses the arrays. I assume something like that will be available in Pascal. If so, that should make for cleaner code, less code.

I'm kind of indifferent to operator overloading. Since you already have to declare a function that gets called when you use the overloaded operator, maybe just make that function available in the interface section so it can be used in case "+" is lost.

The Swift example also uses operator overloading for tensor arithmetic. So something like this:

tanh(matmul(x, w1) + b1)

Could be written without operator overloading like this:

tanh(add(matmul(x, w1), b1))

That still seems pretty clear. After all, you can't get away completely from calling functions when you're doing this kind of stuff. I suppose you could overload operators for both tanh and matmul if there were obvious math symbols that made sense. Are there limitations on what characters you can overload? In Swift, I believe you can use any Unicode character as an overloaded operator. In fact, they actually overload ⊗ for matrix multiplication, so you could write this:

tanh(x ⊗ w1 + b1)

Interesting that they don't use that overloaded operator in their own example. Perhaps cooler heads prevailed.

In my Pascal interface to TensorFlow I overloaded the arithmetic operators too, per Swift, but I don't think I would miss them if they weren't available.

Perhaps just rejoice in the "lower-ceremony" syntax for arrays and make peace with the loss of an overloaded "+" in this one case.

taazz

  • Hero Member
  • *****
  • Posts: 5150
Re: Dynamic array extensions ?
« Reply #2 on: June 07, 2018, 04:34:03 am »
I would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short:

- support for array constructors using "[...]" syntax
- support for Insert(), Delete() and Concat()
- support for "+" operator
- support for dynamic array constants (and variable initializations)

Sounds ok, however: The "+" operator is effectively an alias for concat(), and it will be an intrinsic, which means that existing "+" operator overloads for dynamic arrays will no longer compile. Intrinsic operators cannot be overloaded, hence it will no longer be possible to custom define the "+" operator to do a mathematical elementwise addition of numerical vectors/matrices.

With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied.

What is your opinion on that  ?
on one hand it nice not to have to write those pesky init routines, on the other I have no idea why it is needed to become an intrinsic, until I get some insight on that I can't really comment. I guess it makes things both better and worst. Its better because now + becomes standard, stable, when I see it on piece of code that I debug, I know exactly what it does, it is worst well I guess you know why. Now you have to type all kinds of staff and your code moves a bit farther away from math symbolism. You could say I'm on the fence on that.
ps
Oh by the way becoming like swift is a bad thing not a good one :P (just kidding)
« Last Edit: June 07, 2018, 04:36:09 am by taazz »
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

Phil

  • Hero Member
  • *****
  • Posts: 2734
Re: Dynamic array extensions ?
« Reply #3 on: June 07, 2018, 04:53:09 am »
Oh by the way becoming like swift is a bad thing not a good one :P (just kidding)

You'll probably find this interesting, on why Google selected Swift over Python (considering that Python rules in ML).

https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md

I'm looking to see whether Pascal makes sense to use with TensorFlow.

Thaddy

  • Hero Member
  • *****
  • Posts: 6362
Re: Dynamic array extensions ?
« Reply #4 on: June 07, 2018, 09:19:09 am »
Well the logical thing for dynamic arrays is really concat after all, because it not necessarily holds data that you can perform vector math on.
For use as vectors you can write a type helper Add(), something like:
Code: Pascal  [Select]
  1. Type
  2.  
  3.   TMyIntvector = array of array of integer;
  4.  
  5.   TMyIntVectorHelper = type helper for TMyIntVector
  6.     class function Add(const a,b:TMyIntVector):TMyIntVector;static;
  7.     function Add(const b:TMyIntVector):TMyIntVector;
  8.   end;
Still, how do you retrieve the dimensions easily? (also goes for the operators on vector cells)
These must be known and furthermore dynamic arrays can be ragged arrays, which complicates it even further.
Ragged:
Given an array of array of integer, you can do this:
  Setlength(a,3,5);
  Setlength(a[0],3);
  Setlength(a[4],2);
I am enjoining wine, not whine....

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6327
Re: Dynamic array extensions ?
« Reply #5 on: June 07, 2018, 09:28:01 am »
Well the logical thing for dynamic arrays is really concat after all, because it not necessarily holds data that you can perform vector math on.

You could invert that logic, and say that contrary to strings because dynamic arrays CAN contain data you can do math on, you reserve that operator for its mathematical meaning.

And for appending you can write a type helper Append()

Then you keep the association between static and dynamic arrays instead of strings and dynamics arrays.  Also, Strings are commonly concatenated, but general arrays much less often, while running an operator on every element of an array is more common.




« Last Edit: June 13, 2018, 03:25:35 pm by marcov »

Thaddy

  • Hero Member
  • *****
  • Posts: 6362
Re: Dynamic array extensions ?
« Reply #6 on: June 07, 2018, 09:56:00 am »
That was my initial opinion. That I reversed because of the complexity. Either way is fine with me but I understand the ratio.
I am enjoining wine, not whine....

circular

  • Hero Member
  • *****
  • Posts: 2726
    • Personal webpage
Re: Dynamic array extensions ?
« Reply #7 on: June 13, 2018, 03:04:19 pm »
Where can we say that we want to keep the + operator as it is?
Conscience is the debugger of the mind

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6327
Re: Dynamic array extensions ?
« Reply #8 on: June 13, 2018, 03:25:46 pm »
I updated my answer

GAN

  • Full Member
  • ***
  • Posts: 183
Re: Dynamic array extensions ?
« Reply #9 on: June 13, 2018, 09:43:50 pm »
Where can we say that we want to keep the + operator as it is?
I agree.
Lazarus 1.6 FPC 3.0.0 Linux Mint Mate 17.2 x86_64 GTK-2
Zeos 7.1.3 - Sqlite 3.8.2

Foro Lazarus en español http://forum.lazarus.freepascal.org/index.php/board,73.0.html

hnb

  • Full Member
  • ***
  • Posts: 242
Re: Dynamic array extensions ?
« Reply #10 on: June 13, 2018, 11:30:18 pm »
Where can we say that we want to keep the + operator as it is?
I agree.
probably the only solution is combo of both :


while I like most of improvements, the support for "+" operator seems strange even for me (I generally like new things)... Also strange is breaking backward compatibility (which AFAIK is one of priority) with questionable gain.
Checkout NewPascal initiative and donate beer - ready to use tuned FPC compiler + Lazarus for mORMot project

best regards,
Maciej Izak

schuler

  • Jr. Member
  • **
  • Posts: 82
Re: Dynamic array extensions ?
« Reply #11 on: June 14, 2018, 09:29:57 am »
Hi Nitorami,
These improvements are really fantastic!

I would like to take the opportunity to share what I miss: faster math vector operations. Some vectors I use have 10000 elements that are summed/multiplied with other vectors. I ended coding it in assembler (the full code has more than 4000 lines):

TNeuralFloatArrPtr is a pointer to the first element of a single precision floating point dynamic array.
 
Code: Pascal  [Select]
  1. procedure AVXMulAdd(PtrA, PtrB: TNeuralFloatArrPtr; MulOp: TNeuralFloat; NumElements: integer);
  2. var
  3.   MulOpPtr: pointer;
  4.   localNumElements, MissedElements: integer;
  5. begin
  6.   localNumElements := (NumElements div 4) * 4;
  7.   MissedElements := NumElements - localNumElements;
  8.   if localNumElements > 0 then
  9.   begin
  10.     MulOpPtr := Addr(MulOp);
  11.   asm
  12.   mov ecx, localNumElements
  13.   mov rax, PtrB
  14.   mov rdx, MulOpPtr
  15.  
  16.   VBROADCASTSS ymm5, [rdx]
  17.   mov rdx, PtrA
  18.  
  19.   push rcx
  20.   shr ecx,5  // number of large iterations = number of elements / 32
  21.   jz @SkipLargeAddLoop
  22.  
  23. @LargeAddLoop:
  24.   {$IFDEF AVX2}
  25.   vmovups ymm0, [rdx]
  26.   vmovups ymm1, [rdx+32]
  27.   vmovups ymm2, [rdx+64]
  28.   vmovups ymm3, [rdx+96]
  29.  
  30.   vfmadd231ps ymm0, ymm5, [rax]
  31.   vfmadd231ps ymm1, ymm5, [rax+32]
  32.   vfmadd231ps ymm2, ymm5, [rax+64]
  33.   vfmadd231ps ymm3, ymm5, [rax+96]
  34.   {$ELSE}
  35.   vmulps  ymm0, ymm5, [rax]
  36.   vmulps  ymm1, ymm5, [rax+32]
  37.   vmulps  ymm2, ymm5, [rax+64]
  38.   vmulps  ymm3, ymm5, [rax+96]
  39.  
  40.   vaddps  ymm0, ymm0, [rdx]
  41.   vaddps  ymm1, ymm1, [rdx+32]
  42.   vaddps  ymm2, ymm2, [rdx+64]
  43.   vaddps  ymm3, ymm3, [rdx+96]
  44.   {$ENDIF}
  45.  
  46.   vmovups [rdx],    ymm0
  47.   vmovups [rdx+32], ymm1
  48.   vmovups [rdx+64], ymm2
  49.   vmovups [rdx+96], ymm3
  50.  
  51.   add rax, 128
  52.   add rdx, 128
  53.   dec ecx
  54.   jnz @LargeAddLoop
  55.  
  56. @SkipLargeAddLoop:
  57.   vzeroupper
  58.  
  59.   pop rcx
  60.   and ecx,$0000001F
  61.   jz @EndAdd
  62.   shr ecx, 2 // number of small iterations = (number of elements modulo 16) / 4
  63.  
  64. @SmallAddLoop:
  65.  
  66.   movups  xmm2, [rax]
  67.   movups  xmm4, [rdx]
  68.  
  69.   mulps   xmm2, xmm5
  70.   addps   xmm4, xmm2
  71.  
  72.   movups  [rdx], xmm4
  73.  
  74.   add rax, 16
  75.   add rdx, 16
  76.  
  77.   dec ecx
  78.   jnz @SmallAddLoop
  79.  
  80. @EndAdd:
  81.   end  [
  82.     'RAX', 'RCX', 'RDX',
  83.     'ymm0', 'ymm1', 'ymm2', 'ymm3', 'ymm4', 'ymm5'
  84.   ];
  85.   end; // of if
  86.  
  87.   if MissedElements>0 then
  88.   begin
  89.     PtrA^[localNumElements] += MulOp*PtrB^[localNumElements];
  90.     if MissedElements>1 then
  91.     begin
  92.       PtrA^[localNumElements+1] += MulOp*PtrB^[localNumElements+1];
  93.       if MissedElements>2 then PtrA^[localNumElements+2] += MulOp*PtrB^[localNumElements+2];
  94.     end;
  95.   end;
  96. end;

In the case that you are interested in the full source code, please let me know.

:) Wish everyone happy coding :)

VTwin

  • Sr. Member
  • ****
  • Posts: 425
  • Former Turbo Pascal 3 user
Re: Dynamic array extensions ?
« Reply #12 on: June 14, 2018, 12:57:47 pm »
With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied.

Oh dear!

I consistently use operator overloading on vectors and matrixes. Seems like going backwards to rewrite everything as functions or container classes. If I add two vectors I don't want them concatenated!! :/
« Last Edit: June 14, 2018, 01:02:36 pm by VTwin »
“Talk is cheap. Show me the code.” Linus Torvalds

Lazarus 1.8.4, FPC 3.0.4
macOS 10.11.5 (32 bit Carbon, working on Cocoa) 
Windows 7 (64 bit)
Ubuntu 16.04.3 (64 bit)

circular

  • Hero Member
  • *****
  • Posts: 2726
    • Personal webpage
Re: Dynamic array extensions ?
« Reply #13 on: June 14, 2018, 06:35:04 pm »
Nitorami, could you point out the FPC e-mail that announces those changes?
http://lists.freepascal.org/pipermail/fpc-devel/
Conscience is the debugger of the mind

Nitorami

  • Sr. Member
  • ****
  • Posts: 330
Re: Dynamic array extensions ?
« Reply #14 on: June 14, 2018, 09:15:27 pm »
@circular: Of course.
http://free-pascal-general.1045716.n5.nabble.com/Feature-announcement-Dynamic-array-extensions-td5731410.html
You may have to register with your email address.

@schuler:
1. I think you did not really read the original post; I did not advertize these "improvements", which in my view are unnecessary as part of the core language, as it is trivial to implement them manually. I wanted to hear other user's opinion on the developer's plan to implement the + operator as intrinsic for concatenation of dynamic strings, making it unavailable for simple vector addition: I use vector addition a lot but never needed concatenation.

2. Thanks for the code. For my current project the normal non-vectorised math is sufficient, although optimisation is always tempting. My own knowledge of assembler is poor, and I appreciated the native support for vectorised math which had once been introduced (I think by switch $Sv). But unfortunately it has not been maintained any further and is now broken, at least under windows :-(

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus