* * *

Author Topic: artificial neural networks | back propagation  (Read 12662 times)

mw108

  • New member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #15 on: July 07, 2018, 12:40:25 pm »
Impressive project and implementation @schuler. Your work is gold. :)

I wonder why you declared TVolume as Generic though? From what I've seen it isn't really necessary, is it? Because in the whole project you implement TVolume only once in uvolume.pas as TNNetVolume = class (specialize TVolume<TNeuralFloat>). And TNeuralFloat is of type Single.

schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #16 on: July 11, 2018, 05:30:45 am »
 :) Hello mw108 :)

Thank you for the "impressive". I've been adding new features for an year now and it seems it's going to continue to be like this.

At this moment, the AVX implementation only supports single type and the currently in development OpenCL dot product engine only supports single type too (just tested on Ubuntu 18.04 with NVIDIA K80 and it was a disaster in performance - I hope due to a bug somewhere). Anyway, for now, there is support for the single type only.

When I started coding it, I was asking myself if I will ever use another floating point type. Therefore, for now, there is a generic implementation and AVX/AVX2 implementations for single type only. BTW, in the case that you would like to give it a go, add these defines to your code: "Release, AVX64 and AVX2". Your CPU will fly like a GPU. I've been testing in cloud servers with up to 64 vCPUs with AVX. I must say that it flies.

I haven't formally published results yet. But I intend to do it in the next 12 months.

Let me know if you need any help.

:) Wish everyone happy pascal coding. :)

mw108

  • New member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #17 on: July 11, 2018, 07:45:00 pm »
Thanks for the reply and explanation.

I see. But if you want to implement another floating point type, like Double, you have to get rid of that:

Code: Pascal  [Select]
  1. TNeuralFloatArr = array[0..300000000] of TNeuralFloat;

What was the reason for implementing something like this? I see that you do some System.Move() operations with the Arrays. But can't you use a List type or something? This array is a real memory hog. :D

I actually started to experiment with a List type, but then got stuck at that point where you do the direct memory operations.

Code: Pascal  [Select]
  1. interface
  2.  
  3.   { TNeuralList }
  4.  
  5.   generic TNeuralList<T> = class(specialize TFPGList<T>)
  6.   private
  7.     procedure EnsureRange(Index: Integer);
  8.  
  9.     function Get(Index: Integer): T; {$ifdef FGLINLINE} inline; {$endif}
  10.     procedure Put(Index: Integer; const Item: T); {$ifdef FGLINLINE} inline; {$endif}
  11.   public
  12.     // Copies <Range> amount of items from <Source> at <StartIndex> to <Self> at <TargetIndex>
  13.     procedure Copy(Source: specialize TNeuralList<T>; SourceIndex: Integer; TargetIndex: Integer; Range: Integer);
  14.  
  15.     property Items[Index: Integer]: T read Get write Put; default;
  16.   end;
  17.  
  18.   TNeuralFloatList = class(specialize TNeuralList<TNeuralFloat>);
  19.  
  20.   { TVolume }
  21.   generic TVolume<T> = class(TObject)
  22.     ...
  23.   public
  24.     // FData was made public to allow other fast operations
  25.     FData: specialize TNeuralList<T>; //array of T;
  26.     ...
  27.   end;
  28.  
  29. implementation
  30.  
  31. { TNeuralList }
  32.  
  33. procedure TNeuralList.EnsureRange(Index: Integer);
  34. var
  35.   i: Integer;
  36. begin
  37.   if (Index > Self.Count - 1) then
  38.     begin
  39.       for i := Self.Count - 1 to Index do
  40.         Add(0.0);
  41.     end;
  42. end;
  43.  
  44. function TNeuralList.Get(Index: Integer): T;
  45. begin
  46.   EnsureRange(Index);
  47.   Result := inherited;
  48. end;
  49.  
  50. procedure TNeuralList.Put(Index: Integer; const Item: T);
  51. begin
  52.   EnsureRange(Index);
  53.   inherited;
  54. end;
  55.  
  56. procedure TNeuralList.Copy(Source: specialize TNeuralList<T>; SourceIndex: Integer; TargetIndex: Integer; Range: Integer);
  57. var
  58.   I: Integer;
  59. begin
  60.   EnsureRange(TargetIndex + Range);
  61.   for I := 0 to Range - 1 do
  62.         Self[TargetIndex + I] := Source[SourceIndex + I];
  63. end;
  64.  

The idea of that List is that it takes care of the boundaries itself, so that you can access it like an array:

Code: Pascal  [Select]
  1. var
  2.   TestList: TNeuralFloatList;
  3.   nf: TNeuralFloat;
  4. begin
  5.   TestList := TNeuralFloatList.Create;
  6.   try
  7.     nf := 1.2345;
  8.     TestList[1000] := nf;
  9.     nf := TestList[1000];
  10.     WriteLn(nf, ' Count: ', TestList.Count); // 1.2345
  11.   finally
  12.     TestList.Free;
  13.   end;
  14. end;
  15.  

My idea was to change the TNNetVolumeList.ConcatInto and TNNetVolumeList.SplitFrom functions into something like this:

Code: Pascal  [Select]
  1. procedure TNNetVolumeList.ConcatInto(V: TNNetVolume);
  2. var
  3.   TotalSize: integer;
  4.   I: integer;
  5.   CurrPos: integer;
  6. begin
  7.   if (Count>0) then
  8.   begin
  9.  
  10.     TotalSize := Self.GetTotalSize();
  11.     if V.Size < TotalSize then
  12.     begin
  13.       V.ReSize(TotalSize,1,1);
  14.     end;
  15.  
  16.     CurrPos := 0;
  17.     for I := 0 to Count - 1 do
  18.     begin
  19.       V.FData.Copy(Self[I].FData, 0, CurrPos, Self[I].Size);
  20.       //system.Move(Self[I].FData[0], V.FData[CurrPos], Self[I].Size * SizeOf(TNeuralFloat));
  21.       CurrPos += Self[I].Size;
  22.     end;
  23.   end;
  24. end;
  25.  
  26. procedure TNNetVolumeList.SplitFrom(V: TNNetVolume);
  27. var
  28.   TotalSize: integer;
  29.   I: integer;
  30.   CurrPos: integer;
  31. begin
  32.   if (Count>0) then
  33.   begin
  34.  
  35.     TotalSize := Self.GetTotalSize();
  36.     if V.Size < TotalSize then
  37.     begin
  38.       V.ReSize(TotalSize,1,1);
  39.     end;
  40.  
  41.     CurrPos := 0;
  42.     for I := 0 to Count - 1 do
  43.     begin
  44.       Self[I].FData.Copy(V.FData, CurrPos, 0, Self[I].Size);
  45.       //system.Move(V.FData[CurrPos], Self[I].FData[0], Self[I].Size * SizeOf(TNeuralFloat));
  46.       CurrPos += Self[I].Size;
  47.     end;
  48.   end;
  49. end;
  50.  

But I'm not sure if Self.Size represents the number of records in the List, like List.Count.

What do you think?
« Last Edit: July 11, 2018, 08:00:31 pm by mw108 »

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #18 on: July 11, 2018, 08:06:40 pm »
Thanks for the reply and explanation.

Anyone interested in TensorFlow instead of rolling your own neural network?

https://macpgmr.github.io/MacXPlatform/PascalForTensorFlow.html


mw108

  • New member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #19 on: July 11, 2018, 08:35:29 pm »
But I'm not sure if Self.Size represents the number of records in the List, like List.Count.

Ok, I just saw that Size resembles the size of all dimensions of the Volume:

Code: Pascal  [Select]
  1. FSize := FSizeX * FSizeY * FDepth;
« Last Edit: July 12, 2018, 12:18:08 am by mw108 »

schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #20 on: July 12, 2018, 12:33:32 am »
Hello Phil.

Thank you for sharing the pascal for tensorflow link.

Quote
Anyone interested in TensorFlow instead of rolling your own neural network?

To give perspective where CAI is, have a look at how the tensorflow CIFAR10 example is implemented:

https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10.py

Have a look at how the tensorflow NN architecture is defined in the method def inference(images).

In tensorflow, you need loads of source code lines to define a relatively small model. In CAI, this is how you define a similar model:

Code: Pascal  [Select]
  1.     NN.AddLayer( TNNetInput.Create(32,32,iInputDepth) );        
  2.         NN.AddLayer( TNNetConvolutionReLU.Create(64, 5, 2, 1) );
  3.         NN.AddLayer( TNNetMaxPool.Create(3) );
  4.         NN.AddLayer( TNNetLocalResponseNormDepth.Create(11) );
  5.         NN.AddLayer( TNNetConvolutionReLU.Create(64, 5, 2, 1) );
  6.         NN.AddLayer( TNNetMaxPool.Create(2) );
  7.         NN.AddLayer( TNNetLocalResponseNormDepth.Create(11) );
  8.         NN.AddLayer( TNNetFullConnectReLU.Create(384) );
  9.         NN.AddLayer( TNNetFullConnectReLU.Create(192) );
  10.     NN.AddLayer( TNNetFullConnectReLU.Create(NumClasses) );
  11.     NN.AddLayer( TNNetSoftMax.Create() );
  12.        

As you can see, using CAI it's ultra easy to define similar NNs. Plus, as most of the code is pure Pascal, if you intend to create a new layer type, you can!!!

Besides creating the NN itself, adding data augmentation is super easy:
Code: Pascal  [Select]
  1. //      Random crop and resize
  2.       CropSizeX := 4 + random(5);
  3.       CropSizeY := 4 + random(5);
  4.  
  5.       ImgInputCp.CopyCropping(ImgVolumes[ImgIdx], random(CropSizeX), random(CropSizeY),ImgVolumes[ImgIdx].SizeX-CropSizeX, ImgVolumes[ImgIdx].SizeY-CropSizeY);
  6.       ImgInput.CopyResizing(ImgInputCp, ImgVolumes[ImgIdx].SizeX, ImgVolumes[ImgIdx].SizeY);
  7.  
  8.     // flip is always used in training
  9.     if Random(1000) > 500 then
  10.     begin
  11.       ImgInput.FlipX();
  12.     end;
  13.  
  14.     // one salt and one pepper for each 200 pixels
  15.     ImgInput.AddSaltAndPepper( (ImgInput.SizeX * ImgInput.SizeY) div 200 );
  16.  
  17.     if (color_encoding = csEncodeRGB) and ( Random(1000) > 750 ) then
  18.     begin
  19.       ImgInput.RgbToGray();
  20.     end;
  21.  
  22.         // Random "channel add"
  23.         ImgInput.AddAtDepth(0, ( (Random(1024)-512)*FNoiseLevel) / 2560 );
  24.         if ImgInput.Depth >= 1 then ImgInput.AddAtDepth(1, ( (Random(1024)-512)*FNoiseLevel) / 2560 );
  25.         if ImgInput.Depth >= 2 then ImgInput.AddAtDepth(2, ( (Random(1024)-512)*FNoiseLevel) / 2560 );
  26.  

Besides CIFAR-10 classification, there is another interesting experiment with CAI:
https://www.youtube.com/watch?v=jdFixaZ2P4w

Some results were posted here:
http://forum.lazarus.freepascal.org/index.php?topic=39305.0

I'm currently typing a paper for peer review with impressive results in regards to CIFAR-10 classification.

:) wish everyone happy pascal coding :)

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #21 on: July 12, 2018, 12:43:10 am »
I'm currently typing a paper for peer review with impressive results in regards to CIFAR-10 classification.

It would be interesting if you would code the MNIST.swift example and show the code here. This is a practical example of the sort of thing that people are now doing routinely with software like Apple's Create ML (https://developer.apple.com/documentation/create_ml).

https://github.com/tensorflow/swift-models/tree/master/MNIST

I've included a Pascal version of MNIST.swift that uses the TensorFlow library.

schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #22 on: July 12, 2018, 05:41:00 am »
Hi Phil,
Thank you for sharing links so it gives me opportunity to see an implementation for MNIST.

The equivalent NN architecture with CAI will be:
Code: Pascal  [Select]
  1.     NN := TNNet.Create();
  2.     NN.AddLayer( TNNetInput.Create(28, 28, 1) ); // 28*28*1 = 784      
  3.     NN.AddLayer( TNNetFullConnect.Create(30) );
  4.     NN.AddLayer( TNNetFullConnect.Create(10) );

The main training block (one epoch) will look like this:
Code: Pascal  [Select]
  1.     for Cnt := Low(inputs) to High(inputs) do
  2.     begin
  3.       NN.Compute(inputs[cnt]);
  4.       NN.GetOutput(pOutPut);
  5.       NN.Backpropagate(vOutput[cnt]);
  6.     end;

You can define learning rate and inertia with:
Code: Pascal  [Select]
  1. NN.SetLearningRate(0.001, 0.9);


schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #23 on: July 12, 2018, 05:54:56 am »
@mw108
Quote
What was the reason for implementing something like this?

I needed this:
Code: Pascal  [Select]
  1.   TNeuralFloatArr = array[0..300000000] of TNeuralFloat;

To be able to declare this:
Code: Pascal  [Select]
  1.   TNeuralFloatArrPtr = ^TNeuralFloatArr;

With this type, I can then use:
Code: Pascal  [Select]
  1.   TNNetVolume = class (specialize TVolume<TNeuralFloat>)
  2.     private
  3.       FDataPtr: TNeuralFloatArrPtr;

I can now call assembler code passing pointers as parameters:
Code: Pascal  [Select]
  1. AVXDotProduct(FDataPtr, Original.FDataPtr, FSize)

It's now simple to interface Volumes with OpenCL code. Have a look at this file:
https://sourceforge.net/p/cai/svncode/HEAD/tree/trunk/lazarus/libs/ueasyopencl.pas

Code: Pascal  [Select]
  1. function TEasyOpenCLV.WriteBuffer(buffer: cl_mem; V: TNNetVolume): integer;
  2. begin
  3.   Result := WriteBuffer(buffer, V.GetMemSize(), V.DataPtr);
  4. end;

schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #24 on: July 12, 2018, 07:29:48 am »
@mw108
Quote
Ok, I just saw that Size resembles the size of all dimensions of the Volume:

YES!!! The volume is both a 3D and 1D data structure physically implemented as a dynamic array. In some APIs, you have to transform from 3D to 1D such as when connecting the last convolutional layer to the first fully connected layer. This transformation ins't required here as both representations live together.

You can access as 3D via:
Code: Pascal  [Select]
  1.     property Data[x, y, d: integer]: T read Get write Store; default;

Or, you can access as 1D via:
Code: Pascal  [Select]
  1.     property Raw[x: integer]: T read GetRaw write SetRaw;

SymbolicFrank

  • Sr. Member
  • ****
  • Posts: 394
Re: artificial neural networks | back propagation
« Reply #25 on: July 12, 2018, 12:08:02 pm »
Hi schuler,

Very interesting! Some questions:

Most processes are multi-step (flowchart): they require multiple actions in sequence (probably all a neural network as well) to get a result. How do you calculate a score for the learning from that? Store all intermediate values? But how do you pinpoint the exact step that was weakest and should be tweaked most? Or would you need a kind of unit test for each step?

Is the learning always a separate pass to create a file with biases and weights, or can the network keep on learning as it goes? It would need some feedback for that, which is probably generated by a different process and so might have a different format, and might have to be processed itself before it becomes useful. How would you do that? Use another neural network to process the feedback? But that should have learning feedback as well. Etc.

In human vision, first we detect edges and then shapes. Those shapes are extrapolated and normalized (rotate, tilt, pan, resize, etc), and should then be handled by their own neural network for processing. Handed over to the right sub-process / step in the flowchart, so to say. How would you go about doing that?


Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #26 on: July 12, 2018, 02:09:40 pm »
Hi Phil,
Thank you for sharing links so it gives me opportunity to see an implementation for MNIST.

I meant a working program functionally equivalent to MNIST.

mw108

  • New member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #27 on: July 12, 2018, 07:44:20 pm »
I needed this:
Code: Pascal  [Select]
  1.   TNeuralFloatArr = array[0..300000000] of TNeuralFloat;

To be able to declare this:
Code: Pascal  [Select]
  1.   TNeuralFloatArrPtr = ^TNeuralFloatArr;

With this type, I can then use:
[...]

Yes, I saw that. But you are aware that this array clogs 1.2GB of the max. 2GB global memory? Why does it need to have 300 Mio elements? Why not 250 Mio or 100 Mio or 10 Mio? Is there any specific reason for that number?

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #28 on: July 12, 2018, 07:51:53 pm »
Yes, I saw that. But you are aware that this array clogs 1.2GB of the max. 2GB global memory? Why does it need to have 300 Mio elements? Why not 250 Mio or 100 Mio or 10 Mio? Is there any specific reason for that number?

It may be he's only using it as a way of referencing as an array a block of memory that's never bigger than that dimension.

Another approach might be to use dynamic arrays. That's what I did in MNIST.pas and in the TTensor class:

https://macpgmr.github.io/MacXPlatform/PascalForTensorFlow.html




SymbolicFrank

  • Sr. Member
  • ****
  • Posts: 394
Re: artificial neural networks | back propagation
« Reply #29 on: July 12, 2018, 08:42:43 pm »
This is how the list of a TStrings is defined:

Code: Pascal  [Select]
  1. TStringItemList = array[0..MaxListSize] of TStringItem;

MaxListSize is defined as:

Code: Pascal  [Select]
  1. MaxListSize = Maxint div 16;

That's how it works.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus