Recent

Author Topic: Conscious Artificial Intelligence - Project Update  (Read 47351 times)

iLya2IK

  • New Member
  • *
  • Posts: 29
Re: Conscious Artificial Intelligence - Project Update
« Reply #120 on: July 02, 2021, 10:34:07 am »
Thank you for a wonderful project.

I would like to start classifying text phrases according to certain patterns. It is about recognizing and classifying car brands and models. The number of combinations is about 2000. But all my attempts run into a fundamental misunderstanding of which class to choose for modeling - now I use TNNetForByteProcessing. Here is the code I am using to simulate:
Code: Pascal  [Select][+][-]
  1. const MAX_IN_SIZE = 25;
  2.       HIGH_IN_VAL = MAX_IN_SIZE-1;
  3.       MAX_OUT_SIZE = 256;
  4.       HIGH_OUT_VAL = MAX_OUT_SIZE-1;
  5.  
  6. type TInpArray = Array [0..HIGH_IN_VAL] of Byte;
  7.      TOutArray = Array [0..HIGH_OUT_VAL] of Byte;
  8.  
  9. var
  10.   DS : TSqlite3Dataset;
  11.   MMList : TStringList;
  12.  
  13. procedure SetCorrectOut(var aOut : TOutArray; value : Word);
  14. var Pos : Word;
  15.     Shift : Byte;
  16. begin
  17.   Pos := value div 8;
  18.   Shift := value mod 8;
  19.   aOut[Pos] := $1 shl Shift;
  20. end;
  21.  
  22. function GetCorrectOut(const aOut : TOutArray) : Word;
  23. var Pos : Word;
  24.     Shift : Byte;
  25. begin
  26.   for Pos := 0 to HIGH_OUT_VAL do
  27.   begin
  28.     if (aOut[Pos] > 0) then
  29.     begin
  30.       case aOut[Pos] of
  31.         2   : Shift := 1;
  32.         4   : Shift := 2;
  33.         8   : Shift := 3;
  34.         16  : Shift := 4;
  35.         32  : Shift := 5;
  36.         64  : Shift := 6;
  37.         128 : Shift := 7;
  38.       else
  39.         Shift := 0;
  40.       end;
  41.       Exit(Pos * 8 + Shift);
  42.     end;
  43.   end;
  44.   Result := 0;
  45. end;
  46.  
  47. function GetCorrectOutList(const aOut : TOutArray) : String;
  48. var Pos : Word;
  49.     Cnt : Byte;
  50.     S : String;
  51.  
  52. procedure AddOption(W : Word);
  53. begin
  54.   if length(S) > 0 then S := S + ' ';
  55.   S := S + MMList.Values[inttostr(W)];
  56.   Inc(Cnt);
  57. end;
  58.  
  59. begin
  60.   S := '';
  61.   Cnt := 0;
  62.   for Pos := 0 to HIGH_OUT_VAL do
  63.   begin
  64.     if (aOut[Pos] > 0) then
  65.     begin
  66.       if aOut[Pos] and $1 > 0 then AddOption(Pos shl 3);
  67.       if aOut[Pos] and $2 > 0 then AddOption(Pos shl 3 or $1);
  68.       if aOut[Pos] and $4 > 0 then AddOption(Pos shl 3 or $2);
  69.       if aOut[Pos] and $8 > 0 then AddOption(Pos shl 3 or $3);
  70.       if aOut[Pos] and $10 > 0 then AddOption(Pos shl 3 or $4);
  71.       if aOut[Pos] and $20 > 0 then AddOption(Pos shl 3 or $5);
  72.       if aOut[Pos] and $40 > 0 then AddOption(Pos shl 3 or $6);
  73.       if aOut[Pos] and $80 > 0 then AddOption(Pos shl 3 or $7);
  74.     end;
  75.     if Cnt > 15 then break;
  76.   end;
  77.   Result := S;
  78. end;
  79.  
  80. procedure RunAlgo();
  81. const MAX_EPOCH_CNT = 1000;
  82.       MAX_EPOCH_SIZE = 512;
  83.       MAX_ERROR_CNT  = 1;
  84. var
  85.   NN: TNNetForByteProcessing;
  86.   BAInput : TInpArray;
  87.   BAExpected, BAOutput :TOutArray;
  88.   L, epoch : integer;
  89.   Txt1, Res, Outv : String;
  90.   MM, MM1 : Word;
  91.   ErrorCnt, EpochSize : Integer;
  92.   NeedNextEpoch : Boolean;
  93.   Key : char;
  94. begin
  95.   NN := TNNetForByteProcessing.Create();
  96.  
  97.   try
  98.     NN.AddBasicByteProcessingLayers(MAX_IN_SIZE, MAX_OUT_SIZE, 8, 32);
  99.     NN.SetLearningRate(0.1, 0.0);
  100.  
  101.     WriteLn('Computing...');
  102.     epoch := 1;
  103.     NeedNextEpoch := true;
  104.  
  105.     ErrorCnt := MAX_ERROR_CNT + 1;
  106.     EpochSize := MAX_EPOCH_SIZE;
  107.     while NeedNextEpoch and (epoch <= MAX_EPOCH_CNT) and (ErrorCnt > MAX_ERROR_CNT) do
  108.     begin
  109.       WriteLn('Epoch #', epoch);
  110.       inc(epoch);
  111.       DS.SQL := 'select mrkorig, uniq_ats_schem.id as mmid, '+
  112.       'ats_schem.mrk||" "||ats_schem.model as expc '+
  113.       'from ats_schem, uniq_ats_marks, uniq_ats_schem '+
  114.       'where length(mrkorig) < 25 and uniq_ats_marks.mrk = ats_schem.mrk and '+
  115.       '      uniq_ats_schem.mrk = ats_schem.mrk and '+
  116.       '      uniq_ats_schem.model = ats_schem.model order by random() limit ' + inttostr(EpochSize);
  117.       DS.Open;
  118.       try
  119.         ErrorCnt := 0;
  120.         // tests the learning
  121.         while not DS.EOF do
  122.         begin
  123.           if Keypressed then // if user provided input
  124.           begin
  125.             Key := readkey;
  126.             if Key = #27 then begin
  127.               NeedNextEpoch := false;
  128.               Break;
  129.             end;
  130.           end;
  131.  
  132.           FillChar(BAInput,    MAX_IN_SIZE, #0);
  133.           FillChar(BAExpected, MAX_OUT_SIZE, #0);
  134.           FillChar(BAOutput,   MAX_OUT_SIZE, #0);
  135.           Txt1 := UTF8ToConsole(UTF8LowerCase(DS.FieldByName('mrkorig').AsString));
  136.           Res := UTF8ToConsole(DS.FieldByName('expc').AsString);
  137.           MM := DS.FieldByName('mmid').AsInteger;
  138.  
  139.           L := Length(Txt1);
  140.           if L > HIGH_IN_VAL then
  141.           begin
  142.             L := HIGH_IN_VAL;
  143.           end;
  144.           Move((@(Txt1[1]))^, (@(BAInput[0]))^, L);
  145.           SetCorrectOut(BAExpected, MM);
  146.           NN.Compute(BAInput);
  147.           NN.GetOutput(BAOutPut);
  148.           NN.Backpropagate(BAExpected);
  149.  
  150.           MM1 := GetCorrectOut(BAOutput);
  151.           if MM <> MM1 then Inc(ErrorCnt);
  152.           Outv := GetCorrectOutList(BAOutput);
  153.           WriteLn
  154.           ( 'In:',
  155.             ConsoleToUTF8(PChar(@(BAInput[0]))),
  156.             ' Out:',
  157.             Outv,
  158.             ' Expect:',
  159.             Res
  160.           );
  161.           DS.Next;
  162.         end;
  163.       finally
  164.         DS.Close;
  165.       end;
  166.     end;
  167.   finally
  168.     NN.Free;
  169.     Write('Press ENTER to exit.');
  170.     ReadLn;
  171.   end;  
  172. end;

There is a feeling that I have chosen the wrong approach. I ask you to suggest how best to approach the solution of this problem. If it is not difficult, please give an example of solving similar problems of text analysis.

schuler

  • Full Member
  • ***
  • Posts: 198
Re: Conscious Artificial Intelligence - Project Update
« Reply #121 on: July 06, 2021, 11:24:18 am »
Hi iLya2IK!
Thank you for sharing your source code example with bleeding edge new features from the API.

If I understood your code, you are placing characters into bytes. Although doable, this is an unusual approach but I must say that I have curiosity to find what the NN will find...

Have you tried much lower learning rates such as 0.001? I would give a try to:
Code: Pascal  [Select][+][-]
  1.     NN.SetLearningRate(0.001, 0.9);
  2.     NN.SetLearningRate(0.0001, 0.9);

Usually, when you see no convergence at all, you can try lower learning rates.

Are you able to share your output?

iLya2IK

  • New Member
  • *
  • Posts: 29
Re: Conscious Artificial Intelligence - Project Update
« Reply #122 on: July 06, 2021, 02:46:47 pm »
Are you able to share your output?
Thanks for the answer!

I'm afraid there is nothing to show  :(. For any parameters, the approximation degrades to the output of an empty set (the neural network produces zero bits as a result after a large number of training rounds).
I think I have found a possible alternative for building a neural network. If something works out, I will certainly post the results and the source code.

schuler

  • Full Member
  • ***
  • Posts: 198
Re: Conscious Artificial Intelligence - Project Update
« Reply #123 on: December 26, 2021, 07:16:09 am »
 :) Hello Pascal Lovers!  :)

Besides wishing a long and prosper life to Lazarus/FPC, after 6 months without showing up here, I would like to comment some enhancements recently done to the API.

As usual, I’ll type around the CIFAR-10 https://www.cs.toronto.edu/~kriz/cifar.html image classification task. First, I would like to comment things that you can now do to increase the learning capability of your neural network without adding more neurons/connections to it.

Just by updating the library to the latest library on github, CreateCifar10Volumes and CreateCifar100Volumes now load 2000 images for validation instead of 10000. This small change will push the classification accuracy a bit up. Follows an example:

Code: Pascal  [Select][+][-]
  1. CreateCifar10Volumes(ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes, csEncodeRGB, {ValidationSampleSize=}2000);

We have new activation functions:

Regarding TNNetReLU6, TNNetSwish6 and TNNetReLUL, limiting outputs can both prevent overflows and make the learning faster. Follows an example:

Code: Pascal  [Select][+][-]
  1. TNNetReLUL.Create(-3, 3, 0)

You could define a ReLU6 equivalent activation function with:

Code: Pascal  [Select][+][-]
  1. TNNetReLUL.Create(0, 6, 0)

I coded an example with Swish at https://github.com/joaopauloschuler/neural-api/blob/master/examples/SimpleImageClassifier/SimpleImageClassifierPaddingCroppingSwish.lpr . From experience, Swish doesn’t work well along the first layer or layers. On deeper layers, it works very well. In my experiments, it gives a classification accuracy improvement from 1% to 2%.

In my opinion, one of the most interesting neural network architectures is called ResNet: https://arxiv.org/abs/1512.03385 . The ResNet-20 neural network model has been ported to FPC: https://github.com/joaopauloschuler/neural-api/blob/master/examples/ResNet/ResNet20.lpr . ResNet-20 model is particularly interesting because it’s very floating-point and parameter efficient image classification architecture. Another version of this model was coded taking advantage of the CAI specific activation function TNNetReLUL: https://github.com/joaopauloschuler/neural-api/blob/master/examples/ResNet/CaiResNet20.lpr .

 :) Long and prosperous life to Lazarus and FPC!  :)
« Last Edit: December 26, 2021, 07:27:47 am by schuler »

schuler

  • Full Member
  • ***
  • Posts: 198
Re: Conscious Artificial Intelligence - Project Update
« Reply #124 on: January 29, 2022, 08:39:33 pm »
 :) Hello :)
I found today a scientific paper incorrectly referencing this API (it's super cool to find a scientific paper referencing a free pascal based work - this is possible thanks to lots of people that worked on FPC and Lazarus). In the case that anyone needs to reference it, this is how in BibTeX format:
Code: Pascal  [Select][+][-]
  1. @software{cai_neural_api_2021_5810077,
  2.   author       = {Joao Paulo Schwarz Schuler},
  3.   title        = {CAI NEURAL API},
  4.   month        = dec,
  5.   year         = 2021,
  6.   publisher    = {Zenodo},
  7.   version      = {v1.0.6},
  8.   doi          = {10.5281/zenodo.5810077},
  9.   url          = {https://doi.org/10.5281/zenodo.5810077}
  10. }

BTW, a recommendation via researchgate will be greatly appreciated: https://www.researchgate.net/publication/357717402_CAI_NEURAL_API .

:) Happy Pascal Coding :)

kagamma

  • New member
  • *
  • Posts: 9
Re: Conscious Artificial Intelligence - Project Update
« Reply #125 on: February 17, 2022, 03:59:31 am »
It's nice to see Pascal has a very good native NN library. Too bad it doesn't support RNN/LSTM/GRU so I can't move some of my NLP stuff from PyTorch to it :(

domasz

  • New Member
  • *
  • Posts: 39
Re: Conscious Artificial Intelligence - Project Update
« Reply #126 on: March 15, 2022, 10:05:49 am »
First of all- amazing work!

I don't understand how to use all those neural nets but I'd love to see some nets as sort of ready-to-use functions for morons like me.

Perhaps this is something you could consider?

What would be great:
1) colorizing black & white photos:
https://github.com/mlhubber/colorize

2) upscale:
https://github.com/Araxeus/PNG-Upscale
https://github.com/IBM/MAX-Image-Resolution-Enhancer

3) removing background:
https://github.com/danielgatis/rembg

schuler

  • Full Member
  • ***
  • Posts: 198
Re: Conscious Artificial Intelligence - Project Update
« Reply #127 on: April 05, 2022, 02:20:47 pm »
@domasz,
regarding upscaling, there is a ready to use example:
https://github.com/joaopauloschuler/neural-api/tree/master/examples/SuperResolution

You can find some of the testing at:
https://github.com/joaopauloschuler/neural-api/issues/26

Thank you for the other ideas.

:) Wish everyone happy pascal coding :)
« Last Edit: April 06, 2022, 12:40:22 pm by schuler »

Dzandaa

  • Jr. Member
  • **
  • Posts: 50
Re: Conscious Artificial Intelligence - Project Update
« Reply #128 on: September 18, 2022, 07:42:17 pm »
Hello Joao,

I just wanted to tell you that this is a new approach in neural networks which are limited a little too much to Python and OpenCV (apart from Darknet).

The Pascal Lazarus is a great development tool that has the advantage of being independent of obscure external libraries and working on the 3 (4 with Android) main platforms, Linux Mac and Windows.

I hope you will continue to develop CAI.

 

TinyPortal © 2005-2018