### Bookstore

 Computer Math and Games in Pascal (preview) Lazarus Handbook

### Author Topic: Conscious Artificial Intelligence - Project Update  (Read 48516 times)

#### iLya2IK

• New Member
• Posts: 29
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #120 on: July 02, 2021, 10:34:07 am »
Thank you for a wonderful project.

I would like to start classifying text phrases according to certain patterns. It is about recognizing and classifying car brands and models. The number of combinations is about 2000. But all my attempts run into a fundamental misunderstanding of which class to choose for modeling - now I use TNNetForByteProcessing. Here is the code I am using to simulate:
Code: Pascal  [Select][+][-]
1. const MAX_IN_SIZE = 25;
2.       HIGH_IN_VAL = MAX_IN_SIZE-1;
3.       MAX_OUT_SIZE = 256;
4.       HIGH_OUT_VAL = MAX_OUT_SIZE-1;
5.
6. type TInpArray = Array [0..HIGH_IN_VAL] of Byte;
7.      TOutArray = Array [0..HIGH_OUT_VAL] of Byte;
8.
9. var
10.   DS : TSqlite3Dataset;
11.   MMList : TStringList;
12.
13. procedure SetCorrectOut(var aOut : TOutArray; value : Word);
14. var Pos : Word;
15.     Shift : Byte;
16. begin
17.   Pos := value div 8;
18.   Shift := value mod 8;
19.   aOut[Pos] := \$1 shl Shift;
20. end;
21.
22. function GetCorrectOut(const aOut : TOutArray) : Word;
23. var Pos : Word;
24.     Shift : Byte;
25. begin
26.   for Pos := 0 to HIGH_OUT_VAL do
27.   begin
28.     if (aOut[Pos] > 0) then
29.     begin
30.       case aOut[Pos] of
31.         2   : Shift := 1;
32.         4   : Shift := 2;
33.         8   : Shift := 3;
34.         16  : Shift := 4;
35.         32  : Shift := 5;
36.         64  : Shift := 6;
37.         128 : Shift := 7;
38.       else
39.         Shift := 0;
40.       end;
41.       Exit(Pos * 8 + Shift);
42.     end;
43.   end;
44.   Result := 0;
45. end;
46.
47. function GetCorrectOutList(const aOut : TOutArray) : String;
48. var Pos : Word;
49.     Cnt : Byte;
50.     S : String;
51.
53. begin
54.   if length(S) > 0 then S := S + ' ';
55.   S := S + MMList.Values[inttostr(W)];
56.   Inc(Cnt);
57. end;
58.
59. begin
60.   S := '';
61.   Cnt := 0;
62.   for Pos := 0 to HIGH_OUT_VAL do
63.   begin
64.     if (aOut[Pos] > 0) then
65.     begin
66.       if aOut[Pos] and \$1 > 0 then AddOption(Pos shl 3);
67.       if aOut[Pos] and \$2 > 0 then AddOption(Pos shl 3 or \$1);
68.       if aOut[Pos] and \$4 > 0 then AddOption(Pos shl 3 or \$2);
69.       if aOut[Pos] and \$8 > 0 then AddOption(Pos shl 3 or \$3);
70.       if aOut[Pos] and \$10 > 0 then AddOption(Pos shl 3 or \$4);
71.       if aOut[Pos] and \$20 > 0 then AddOption(Pos shl 3 or \$5);
72.       if aOut[Pos] and \$40 > 0 then AddOption(Pos shl 3 or \$6);
73.       if aOut[Pos] and \$80 > 0 then AddOption(Pos shl 3 or \$7);
74.     end;
75.     if Cnt > 15 then break;
76.   end;
77.   Result := S;
78. end;
79.
80. procedure RunAlgo();
81. const MAX_EPOCH_CNT = 1000;
82.       MAX_EPOCH_SIZE = 512;
83.       MAX_ERROR_CNT  = 1;
84. var
85.   NN: TNNetForByteProcessing;
86.   BAInput : TInpArray;
87.   BAExpected, BAOutput :TOutArray;
88.   L, epoch : integer;
89.   Txt1, Res, Outv : String;
90.   MM, MM1 : Word;
91.   ErrorCnt, EpochSize : Integer;
92.   NeedNextEpoch : Boolean;
93.   Key : char;
94. begin
95.   NN := TNNetForByteProcessing.Create();
96.
97.   try
99.     NN.SetLearningRate(0.1, 0.0);
100.
101.     WriteLn('Computing...');
102.     epoch := 1;
103.     NeedNextEpoch := true;
104.
105.     ErrorCnt := MAX_ERROR_CNT + 1;
106.     EpochSize := MAX_EPOCH_SIZE;
107.     while NeedNextEpoch and (epoch <= MAX_EPOCH_CNT) and (ErrorCnt > MAX_ERROR_CNT) do
108.     begin
109.       WriteLn('Epoch #', epoch);
110.       inc(epoch);
111.       DS.SQL := 'select mrkorig, uniq_ats_schem.id as mmid, '+
112.       'ats_schem.mrk||" "||ats_schem.model as expc '+
113.       'from ats_schem, uniq_ats_marks, uniq_ats_schem '+
114.       'where length(mrkorig) < 25 and uniq_ats_marks.mrk = ats_schem.mrk and '+
115.       '      uniq_ats_schem.mrk = ats_schem.mrk and '+
116.       '      uniq_ats_schem.model = ats_schem.model order by random() limit ' + inttostr(EpochSize);
117.       DS.Open;
118.       try
119.         ErrorCnt := 0;
120.         // tests the learning
121.         while not DS.EOF do
122.         begin
123.           if Keypressed then // if user provided input
124.           begin
126.             if Key = #27 then begin
127.               NeedNextEpoch := false;
128.               Break;
129.             end;
130.           end;
131.
132.           FillChar(BAInput,    MAX_IN_SIZE, #0);
133.           FillChar(BAExpected, MAX_OUT_SIZE, #0);
134.           FillChar(BAOutput,   MAX_OUT_SIZE, #0);
135.           Txt1 := UTF8ToConsole(UTF8LowerCase(DS.FieldByName('mrkorig').AsString));
136.           Res := UTF8ToConsole(DS.FieldByName('expc').AsString);
137.           MM := DS.FieldByName('mmid').AsInteger;
138.
139.           L := Length(Txt1);
140.           if L > HIGH_IN_VAL then
141.           begin
142.             L := HIGH_IN_VAL;
143.           end;
144.           Move((@(Txt1[1]))^, (@(BAInput[0]))^, L);
145.           SetCorrectOut(BAExpected, MM);
146.           NN.Compute(BAInput);
147.           NN.GetOutput(BAOutPut);
148.           NN.Backpropagate(BAExpected);
149.
150.           MM1 := GetCorrectOut(BAOutput);
151.           if MM <> MM1 then Inc(ErrorCnt);
152.           Outv := GetCorrectOutList(BAOutput);
153.           WriteLn
154.           ( 'In:',
155.             ConsoleToUTF8(PChar(@(BAInput[0]))),
156.             ' Out:',
157.             Outv,
158.             ' Expect:',
159.             Res
160.           );
161.           DS.Next;
162.         end;
163.       finally
164.         DS.Close;
165.       end;
166.     end;
167.   finally
168.     NN.Free;
169.     Write('Press ENTER to exit.');
171.   end;
172. end;

There is a feeling that I have chosen the wrong approach. I ask you to suggest how best to approach the solution of this problem. If it is not difficult, please give an example of solving similar problems of text analysis.

#### schuler

• Full Member
• Posts: 201
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #121 on: July 06, 2021, 11:24:18 am »
Hi iLya2IK!
Thank you for sharing your source code example with bleeding edge new features from the API.

If I understood your code, you are placing characters into bytes. Although doable, this is an unusual approach but I must say that I have curiosity to find what the NN will find...

Have you tried much lower learning rates such as 0.001? I would give a try to:
Code: Pascal  [Select][+][-]
1.     NN.SetLearningRate(0.001, 0.9);
2.     NN.SetLearningRate(0.0001, 0.9);

Usually, when you see no convergence at all, you can try lower learning rates.

Are you able to share your output?

#### iLya2IK

• New Member
• Posts: 29
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #122 on: July 06, 2021, 02:46:47 pm »
Are you able to share your output?

I'm afraid there is nothing to show  . For any parameters, the approximation degrades to the output of an empty set (the neural network produces zero bits as a result after a large number of training rounds).
I think I have found a possible alternative for building a neural network. If something works out, I will certainly post the results and the source code.

#### schuler

• Full Member
• Posts: 201
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #123 on: December 26, 2021, 07:16:09 am »
Hello Pascal Lovers!

Besides wishing a long and prosper life to Lazarus/FPC, after 6 months without showing up here, I would like to comment some enhancements recently done to the API.

As usual, I’ll type around the CIFAR-10 https://www.cs.toronto.edu/~kriz/cifar.html image classification task. First, I would like to comment things that you can now do to increase the learning capability of your neural network without adding more neurons/connections to it.

Just by updating the library to the latest library on github, CreateCifar10Volumes and CreateCifar100Volumes now load 2000 images for validation instead of 10000. This small change will push the classification accuracy a bit up. Follows an example:

Code: Pascal  [Select][+][-]
1. CreateCifar10Volumes(ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes, csEncodeRGB, {ValidationSampleSize=}2000);

We have new activation functions:

Regarding TNNetReLU6, TNNetSwish6 and TNNetReLUL, limiting outputs can both prevent overflows and make the learning faster. Follows an example:

Code: Pascal  [Select][+][-]
1. TNNetReLUL.Create(-3, 3, 0)

You could define a ReLU6 equivalent activation function with:

Code: Pascal  [Select][+][-]
1. TNNetReLUL.Create(0, 6, 0)

I coded an example with Swish at https://github.com/joaopauloschuler/neural-api/blob/master/examples/SimpleImageClassifier/SimpleImageClassifierPaddingCroppingSwish.lpr . From experience, Swish doesn’t work well along the first layer or layers. On deeper layers, it works very well. In my experiments, it gives a classification accuracy improvement from 1% to 2%.

In my opinion, one of the most interesting neural network architectures is called ResNet: https://arxiv.org/abs/1512.03385 . The ResNet-20 neural network model has been ported to FPC: https://github.com/joaopauloschuler/neural-api/blob/master/examples/ResNet/ResNet20.lpr . ResNet-20 model is particularly interesting because it’s very floating-point and parameter efficient image classification architecture. Another version of this model was coded taking advantage of the CAI specific activation function TNNetReLUL: https://github.com/joaopauloschuler/neural-api/blob/master/examples/ResNet/CaiResNet20.lpr .

Long and prosperous life to Lazarus and FPC!
« Last Edit: December 26, 2021, 07:27:47 am by schuler »

#### schuler

• Full Member
• Posts: 201
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #124 on: January 29, 2022, 08:39:33 pm »
Hello
I found today a scientific paper incorrectly referencing this API (it's super cool to find a scientific paper referencing a free pascal based work - this is possible thanks to lots of people that worked on FPC and Lazarus). In the case that anyone needs to reference it, this is how in BibTeX format:
Code: Pascal  [Select][+][-]
1. @software{cai_neural_api_2021_5810077,
2.   author       = {Joao Paulo Schwarz Schuler},
3.   title        = {CAI NEURAL API},
4.   month        = dec,
5.   year         = 2021,
6.   publisher    = {Zenodo},
7.   version      = {v1.0.6},
8.   doi          = {10.5281/zenodo.5810077},
9.   url          = {https://doi.org/10.5281/zenodo.5810077}
10. }

BTW, a recommendation via researchgate will be greatly appreciated: https://www.researchgate.net/publication/357717402_CAI_NEURAL_API .

Happy Pascal Coding

#### kagamma

• New member
• Posts: 9
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #125 on: February 17, 2022, 03:59:31 am »
It's nice to see Pascal has a very good native NN library. Too bad it doesn't support RNN/LSTM/GRU so I can't move some of my NLP stuff from PyTorch to it

#### domasz

• Full Member
• Posts: 108
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #126 on: March 15, 2022, 10:05:49 am »
First of all- amazing work!

I don't understand how to use all those neural nets but I'd love to see some nets as sort of ready-to-use functions for morons like me.

Perhaps this is something you could consider?

What would be great:
1) colorizing black & white photos:
https://github.com/mlhubber/colorize

2) upscale:
https://github.com/Araxeus/PNG-Upscale
https://github.com/IBM/MAX-Image-Resolution-Enhancer

3) removing background:
https://github.com/danielgatis/rembg

#### schuler

• Full Member
• Posts: 201
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #127 on: April 05, 2022, 02:20:47 pm »
@domasz,
regarding upscaling, there is a ready to use example:
https://github.com/joaopauloschuler/neural-api/tree/master/examples/SuperResolution

You can find some of the testing at:
https://github.com/joaopauloschuler/neural-api/issues/26

Thank you for the other ideas.

Wish everyone happy pascal coding
« Last Edit: April 06, 2022, 12:40:22 pm by schuler »

#### Dzandaa

• Jr. Member
• Posts: 73
• From C# to Lazarus
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #128 on: September 18, 2022, 07:42:17 pm »
Hello Joao,

I just wanted to tell you that this is a new approach in neural networks which are limited a little too much to Python and OpenCV (apart from Darknet).

The Pascal Lazarus is a great development tool that has the advantage of being independent of obscure external libraries and working on the 3 (4 with Android) main platforms, Linux Mac and Windows.

I hope you will continue to develop CAI.
Dzandaa

#### schuler

• Full Member
• Posts: 201
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #129 on: November 22, 2022, 06:48:19 pm »
Hello

I have a selfish post to make: yesterday, I finished my PhD course in AI. YAY!

My PhD thesis certainly can give clues to where this API is heading: super efficient neural networks.

I wish everyone happy pascal coding
« Last Edit: November 22, 2022, 07:04:10 pm by schuler »

#### nummer8

• Jr. Member
• Posts: 93
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #130 on: November 22, 2022, 07:42:15 pm »
Congrats,

Finishing a PhD is a hell of a job.
Enjoy!!

• Hero Member
• Posts: 12209
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #131 on: November 22, 2022, 10:07:58 pm »
Hello

I have a selfish post to make: yesterday, I finished my PhD course in AI. YAY!
Many, many congratulations! Dr Schuler.
By coincidence I referred to your work at this forum today...

I will owe you some beers and a good meal when you will visit the Netherlands.
Yay,
« Last Edit: November 22, 2022, 10:18:52 pm by Thaddy »
Manuals, manuals, manuals first.
You have incompetence and sheer incompetence.

#### schuler

• Full Member
• Posts: 201
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #132 on: November 22, 2022, 11:05:59 pm »
You supported my work in its very early days. I'm very thankful for the positive thoughts you've provided me.

@nummer8, it's very a hard work indeed.

Dr. Schuler
« Last Edit: November 22, 2022, 11:10:22 pm by schuler »

#### metis

• Full Member
• Posts: 237
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #133 on: November 24, 2022, 12:02:09 pm »
@Dr. Schuler

Congratulations !
Got for Academics Beer&Food and Sun in Tenerife.
Life could be so easy, if there weren't those f*** Details.