Hello

,

I’m frequently asked why I’m using pascal to work with neural networks and my frequent reply is: compiled pascal code is usually super efficient. Keeping efficiency in mind, I would like to comment a neural network architecture that is efficient in both trainable parameters (memory) and computing requirements (CPU/GPU/FPGA/…). It seems perfectly logical to me implementing efficient neural architectures with an efficient language (pascal).

I’ve heard about teams working with 60 video cards in parallel with training time from 1 to 2 months. Adding neurons and neuronal layers is surely a possible way to improve artificial neural networks if you have enough hardware and computing time. In the case that you can’t afford time and hardware, you’ll look for efficiency. There is an inspirational paper here:

https://arxiv.org/abs/1704.04861 . They base their work on separable convolutions:

https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728 .

As can be seen on above links, a separable convolution is a composition of 2 building blocks: a depth-wise convolution followed by a point-wise convolution. Although I won’t explain here what these 2 building blocks do, I would like to comment that these 2 building blocks allow artificial neural network architectures to solve classification problems with a fraction of trainable parameters and computing power.

The 2 separable convolution building blocks (depth-wise and point-wise convolutions) are ready to use:

TNNetDepthwiseConv.Create(pMultiplier, pFeatureSize, pInputPadding, pStride: integer);

TNNetDepthwiseConvLinear.Create(pMultiplier, pFeatureSize, pInputPadding, pStride: integer);

TNNetDepthwiseConvReLU.Create(pMultiplier, pFeatureSize, pInputPadding, pStride: integer);

TNNetPointwiseConvReLU.Create(pNumFeatures: integer; pSuppressBias: integer = 0);

TNNetPointwiseConvLinear.Create(pNumFeatures: integer; pSuppressBias: integer = 0);

Or, if you just want to give a go with an out of the box separable convolution, you can add it with these methods:

TNNet.AddSeparableConvReLU(pNumFeatures{filters}, pFeatureSize, pInputPadding, pStride: integer; pDepthMultiplier: integer = 1; pSuppressBias: integer = 0; pAfterLayer: TNNetLayer = nil): TNNetLayer;

TNNet.AddSeparableConvLinear(pNumFeatures{filters}, pFeatureSize, pInputPadding, pStride: integer; pDepthMultiplier: integer = 1; pSuppressBias: integer = 0; pAfterLayer: TNNetLayer = nil): TNNetLayer;

The most recent convolutional API can be found here:

https://sourceforge.net/p/cai/svncode/HEAD/tree/trunk/lazarus/libs/uconvolutionneuralnetwork.pas Wish everyone happy pascal coding.