### Bookstore

 Computer Math and Games in Pascal (preview) Lazarus Handbook

### Author Topic: Conscious Artificial Intelligence - Project Update  (Read 20587 times)

#### zulof

• Newbie
• Posts: 4
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #30 on: November 12, 2019, 11:05:27 am »
Could/Should this be included in online package manager?

#### m.abudrais

• Jr. Member
• Posts: 51
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #31 on: November 22, 2019, 08:14:54 am »
thank you for making this good library.
I am absolute beginner in NN.
i run the CIFAR-10 example (for  50 epochs only)   and  the SimpleImageClassifier.nn is generated.
then I tried to Classify some image  by this code:
Code: Pascal  [Select][+][-]
1. procedure TForm1.Button1Click(Sender: TObject);
2. var
3.   NN: THistoricalNets;
4.   O1: array of TNeuralFloat;
5.   pOutPut, pInput: TNNetVolume;
6.   k, y, x: integer;
7.   OK: extended;
8.   img: TTinyImage;
9. begin
11.   NN := THistoricalNets.Create();
13.     TNNetConvolutionLinear.Create(64, 5, 2, 1, 1).InitBasicPatterns(),
14.     TNNetMaxPool.Create(4), TNNetConvolutionReLU.Create(64, 3, 1, 1, 1),
15.     TNNetConvolutionReLU.Create(64, 3, 1, 1, 1),
16.     TNNetConvolutionReLU.Create(64, 3, 1, 1, 1),
17.     TNNetConvolutionReLU.Create(64, 3, 1, 1, 1), TNNetDropout.Create(0.5),
18.     TNNetMaxPool.Create(2), TNNetFullConnectLinear.Create(10),
19.     TNNetSoftMax.Create()]);
21.   for y := 0 to 31 do
22.   begin
23.     for x := 0 to 31 do
24.     begin
25.       img.B[y, x] := Red(Image1.Canvas.Pixels[y, x]);
26.       img.g[y, x] := Green(Image1.Canvas.Pixels[y, x]);
27.       img.B[y, x] := Blue(Image1.Canvas.Pixels[y, x]);
28.     end;
29.   end;
30.   pInput := TNNetVolume.Create(32, 32, 3, 1);
31.   pOutPut := TNNetVolume.Create(10, 1, 1, 1);
33.   NN.Compute(pInput);
34.   NN.GetOutput(pOutPut);
35.   OK := 0.0;
36.   for k := 0 to 9 do
37.   begin
38.     OK := OK + pOutPut.Raw[k];
39.     WriteLn(pOutPut.Raw[k]);
40.   end;
41.   WriteLn();
42.   WriteLn(OK);
43. end;
i tried image for a dog and car from the same data set but i always get 1 in pOutPut.Raw[0]!
can you please add  example  to show how to use Classifier after training.
« Last Edit: November 22, 2019, 08:16:33 am by m.abudrais »

#### schuler

• Full Member
• Posts: 164
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #32 on: November 25, 2019, 04:56:02 pm »

First of all, you can completely remove "AddLayer" call as CAI stores both architecture and weights into the same nn file.

Then, you need to add RgbImgToNeuronalInput as follows:

Code: Pascal  [Select][+][-]
2. pInput.RgbImgToNeuronalInput(csEncodeRGB);

If the above doesn't work, I'll try to compile at my end.

As you have SoftMax, you can print output class probabilities with:
Code: Pascal  [Select][+][-]
1. pOutPut.Print();

You can get the inferred output class with:
Code: Pascal  [Select][+][-]
1. WriteLn('Inferred Class: ', csTinyImageLabel[pOutPut.GetClass()]);

#### m.abudrais

• Jr. Member
• Posts: 51
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #33 on: November 25, 2019, 07:40:52 pm »
Quote
First of all, you can completely remove "AddLayer" call as CAI stores both architecture and weights into the same nn file.

now it works .
this is the correct code if any one is interesting
Code: Pascal  [Select][+][-]
1. procedure TForm1.Button1Click(Sender: TObject);
2. var
3.   NN: THistoricalNets;
4.   O1: array of TNeuralFloat;
5.   pOutPut, pInput: TNNetVolume;
6.   k, y, x: integer;
7.   OK: extended;
8.   img: TTinyImage;
9. begin
10.   NN := THistoricalNets.Create();
12.   for y := 0 to 31 do
13.   begin
14.     for x := 0 to 31 do
15.     begin
16.       img.R[y, x] := Red(Image1.Canvas.Pixels[y, x]);
17.       img.g[y, x] := Green(Image1.Canvas.Pixels[y, x]);
18.       img.B[y, x] := Blue(Image1.Canvas.Pixels[y, x]);
19.     end;
20.   end;
21.   pInput := TNNetVolume.Create(32, 32, 3, 1);
22.   pOutPut := TNNetVolume.Create(10, 1, 1, 1);
24.   pInput.RgbImgToNeuronalInput(csEncodeRGB);
25.   NN.Compute(pInput);
26.
27.   WriteLn('Inferred Class: ', csTinyImageLabel[pOutPut.GetClass()]);
28.
thank you again for making this good library.

#### schuler

• Full Member
• Posts: 164
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #34 on: November 25, 2019, 09:55:28 pm »
@m.abudrais
Are you placing your code on a public repo?

Anyway, just realized that you might be able to replace both for loops by using the neuralvolumev https://github.com/joaopauloschuler/neural-api/blob/master/neural/neuralvolumev.pas unit:

Code: Pascal  [Select][+][-]
1. /// Loads a Picture into a Volume
2. procedure LoadPictureIntoVolume(LocalPicture: TPicture; Vol:TNNetVolume); {\$IFDEF Release} inline; {\$ENDIF}
3.
4. /// Loads a Bitmat into a Volume

The code will look like:
Code: Pascal  [Select][+][-]
2. pInput.RgbImgToNeuronalInput(csEncodeRGB);

In the case that the input image isn't 32x32, you can resize it (via copying) with:
Code: Pascal  [Select][+][-]
1. TVolume.CopyResizing(Original: TVolume; NewSizeX, NewSizeY: integer);

Last idea: given that you have a trained NN, you could call this:
Code: Pascal  [Select][+][-]
1. procedure TNeuralImageFit.ClassifyImage(pNN: TNNet; pImgInput, pOutput: TNNetVolume);
« Last Edit: November 25, 2019, 10:16:01 pm by schuler »

#### m.abudrais

• Jr. Member
• Posts: 51
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #35 on: November 26, 2019, 04:58:28 pm »
Quote
Are you placing your code on a public repo?
I have just uploaded the code to:
https://github.com/mabudrais/CAI-NEURAL-API-Test
your library has many useful functions, I think more examples may be added to show that.

#### schuler

• Full Member
• Posts: 164
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #36 on: December 06, 2019, 06:57:25 pm »
Hello

It's usually very hard to understand neuron by neuron how a neural network dedicated to image classification internally works. One technique used to help with the understanding about what individual neurons represent is called Gradient Ascent. You can find more about gradient ascent at http://yosinski.com/deepvis .

In this technique, an arbitrary neuron is required to activate and then the same backpropagation method used for learning is applied to an input image producing an image that this neuron expects to see. I'm happy to inform that we have a working example purely coded in FPC/Lazarus at: https://github.com/joaopauloschuler/neural-api/tree/master/examples/GradientAscent

To be able to run this example, you'll need to load an already trained neural network file and then select the layer you intend to visualise. I've attached an example from the last layer (can you see any pattern?) of a neural network trained to classify CIFAR-10 dataset. Can you find patterns for horse, deer, ship, car or truck?

Long life to Pascal
« Last Edit: December 06, 2019, 07:01:36 pm by schuler »

#### agorka

• New Member
• Posts: 25
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #37 on: December 21, 2019, 01:28:15 pm »
Just found this thread, it seems to be a very interesting topic!
Thanks to all the hard work you did!

While I still prefer python/tensorflow/keras approach to train CNNs, I'd like to use the resulting CNN in my Delphi project to help me sorting the sounds. Do you have any method to export the weights from a Keras model and load it to your library so I can do that NN magic in pure pascal without any additional DLLs?

#### schuler

• Full Member
• Posts: 164
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #38 on: December 23, 2019, 06:29:55 pm »
Hello agorka! Glad that you like it.

Supporting Keras file format isn't in the near horizon. At this moment, I'm polishing a super advanced example about producing artificial art: https://github.com/joaopauloschuler/neural-api/tree/master/examples/VisualGAN

Other feature requests are listed as github issues.

#### agorka

• New Member
• Posts: 25
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #39 on: December 24, 2019, 07:38:50 am »
Wow! I couldn't imagine this could be done in Lazarus without such heavy frameworks like tensorflow or pytorch...
Still, my question is, what file formats could be used to load existing weights to the convolution layers?
I could find a converter or make my own export function from Python...
Is there any json, csv, any other format?

Thanks!
Oleg.

#### avra

• Hero Member
• Posts: 1998
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #40 on: December 24, 2019, 08:17:21 am »
Is there any json, csv, any other format?
It would also help for MatLab.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

#### schuler

• Full Member
• Posts: 164
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #41 on: December 24, 2019, 11:35:49 am »
@agorka,
In the case that you try loading weights by yourself, here we go:

A neural network TNNet has Layers:
Code: Pascal  [Select][+][-]
1. property Layers: TNNetLayerList read FLayers;

Each Layer (TNNetLayer) has Neurons:
Code: Pascal  [Select][+][-]
1. property Neurons: TNNetNeuronList read FNeurons;

Each Neuron (TNNetNeuron) has weights:
Code: Pascal  [Select][+][-]
1. property Weights: TNNetVolume read FWeights

Accessing weights from the second neuron from the second layer will look like this:
Code: Pascal  [Select][+][-]
1. NN.Layers[1].Neurons[1].Weights

As you can see, Weights is a TNNetVolume.

TNNetVolume has 2 methods that might interest you:
Code: Pascal  [Select][+][-]
1.     // load and save functions
2.     function SaveToString(): string;

Having a look at the implementation, you'll find that weights are saved as a ";" separated string. The first 4 values are structural: version, SizeX, SizeY and depth. The following values are the actual weights.

#### schuler

• Full Member
• Posts: 164
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #42 on: December 24, 2019, 11:47:46 am »
Quote
Wow! I couldn't imagine this could be done in Lazarus without such heavy frameworks like tensorflow or pytorch
Lazarus/FPC can do impressive things such as compiling to other targets (android and raspberry).

Artificial Art can be useful for texture generation on an artificial environment (such as computer games) as long as you have a pre-trained network. I'll eventually add an example creating very large images (and not just 32x32 pixels).

#### agorka

• New Member
• Posts: 25
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #43 on: December 24, 2019, 10:48:38 pm »
Thank you, I'm getting it running with 1 convolution layer.
NN.Layers.AfterWeightUpdate;
and unprivate this AfterWeightUpdate procedure.
Probably it should be called from LoadFromString function to avoid this inconsistencies.

Thank you for your hard work! I really hope to port my keras network to Pascal tomorrow! =)

#### agorka

• New Member
• Posts: 25
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #44 on: December 24, 2019, 11:09:42 pm »
Sorry, I've run into a problem again.
My network now is as simple as this:

TNNetInput.Create(512, 512, 1),
TNNetConvolutionReLU.Create(24, 7, 0, 1, 0).InitUniform(0),
TNNetConvolutionReLU.Create(1, 5, 0, 1, 0).InitUniform(0)

Though, I run out of memory in
procedure TNNetConvolutionBase.SetPrevLayer(pPrevLayer: TNNetLayer);
when it tries to allocate 502x502x5x5x24 Volume, which is ~600MB.
As I understand, it's needed to optimize convolution calculation, but in my 32 bit application I just can't allocate that much, and my real network needs many more layers.
Is there any way to workaround this problem?
For me it's better to run it slowly than not to run at all. =)

Thanks!