Recent

Author Topic: FPC & Efficient Neural Networks  (Read 149 times)

schuler

  • Full Member
  • ***
  • Posts: 101
FPC & Efficient Neural Networks
« on: June 20, 2019, 05:04:00 pm »
:) Hello :),
I’m frequently asked why I’m using pascal to work with neural networks and my frequent reply is: compiled pascal code is usually super efficient. Keeping efficiency in mind, I would like to comment a neural network architecture that is efficient in both trainable parameters (memory) and computing requirements (CPU/GPU/FPGA/…). It seems perfectly logical to me implementing efficient neural architectures with an efficient language (pascal).

I’ve heard about teams working with 60 video cards in parallel with training time from 1 to 2 months. Adding neurons and neuronal layers is surely a possible way to improve artificial neural networks if you have enough hardware and computing time. In the case that you can’t afford time and hardware, you’ll look for efficiency. There is an inspirational paper here: https://arxiv.org/abs/1704.04861 . They base their work on separable convolutions:
https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728 .

As can be seen on above links, a separable convolution is a composition of 2 building blocks: a depth-wise convolution followed by a point-wise convolution. Although I won’t explain here what these 2 building blocks do, I would like to comment that these 2 building blocks allow artificial neural network architectures to solve classification problems with a fraction of trainable parameters and computing power.

The 2 separable convolution building blocks (depth-wise and point-wise convolutions) are ready to use:
Code: Pascal  [Select]
  1. TNNetDepthwiseConv.Create(pMultiplier, pFeatureSize, pInputPadding, pStride: integer);
  2. TNNetDepthwiseConvLinear.Create(pMultiplier, pFeatureSize, pInputPadding, pStride: integer);
  3. TNNetDepthwiseConvReLU.Create(pMultiplier, pFeatureSize, pInputPadding, pStride: integer);
  4. TNNetPointwiseConvReLU.Create(pNumFeatures: integer; pSuppressBias: integer = 0);
  5. TNNetPointwiseConvLinear.Create(pNumFeatures: integer; pSuppressBias: integer = 0);

Or, if you just want to give a go with an out of the box separable convolution, you can add it with these methods:

Code: Pascal  [Select]
  1. TNNet.AddSeparableConvReLU(pNumFeatures{filters}, pFeatureSize, pInputPadding, pStride: integer; pDepthMultiplier: integer = 1; pSuppressBias: integer = 0; pAfterLayer: TNNetLayer = nil): TNNetLayer;
  2. TNNet.AddSeparableConvLinear(pNumFeatures{filters}, pFeatureSize, pInputPadding, pStride: integer; pDepthMultiplier: integer = 1; pSuppressBias: integer = 0; pAfterLayer: TNNetLayer = nil): TNNetLayer;

The most recent convolutional API can be found here:
https://sourceforge.net/p/cai/svncode/HEAD/tree/trunk/lazarus/libs/uconvolutionneuralnetwork.pas

:) Wish everyone happy pascal coding. :)

Thaddy

  • Hero Member
  • *****
  • Posts: 8039
Re: FPC & Efficient Neural Networks
« Reply #1 on: June 20, 2019, 05:15:14 pm »
The same to you. I really enjoy your coding (and knowledge about the subject) and have already applied it in some applications.
Thank you very much!   8) 8) 8) 8) 8) 8) (Actually beyond cool)
« Last Edit: June 20, 2019, 05:22:32 pm by Thaddy »
Hamlet 1.4 (nothing wrong with the Danish, btw)