Conscious Artificial Intelligence - Project Update

[Dzandaa]

Hi,

I have problem with TrainingAccuracy, my program is running well, results are good but TrainingAccuracy is always Zero.

I've send you a personal message with code.

B->

[Dzandaa]

Hi Schuler,

I tested "plant leaf diseases" and "Colorectal" networks and  it's working


B->

[schuler]

@Dzandaa,
What a beautiful screenshot! Wondering if I can find your code in an open repository...

[schuler]

Hello pascal lovers ,
I did an experiment few days ago that I would like to share. The experiment is about predicting the next character in a string.

To make things very simple, the dataset has 3 strings:

Code: Pascal  [Select][+][-]

1.     FDataset[0] := 'happy good morning.'+chr(1);
2.     FDataset[1] := 'fantastic good evening.'+chr(1);
3.     FDataset[2] := 'superb good night.'+chr(1);

The neural network is built with:

Code: Pascal  [Select][+][-]

1.     FNN.AddLayer([
2.       TNNetInput.Create(csContextLen, 1, csVocabSize),
3.       TNNetPointwiseConvReLU.Create(32),
4.       TNNetFullConnectReLU.Create(32),
5.       TNNetFullConnectReLU.Create(32),
6.       TNNetFullConnectLinear.Create(csVocabSize),
7.       TNNetSoftMax.Create()
8.     ]);

The constants above are defined with:

Code: Pascal  [Select][+][-]

1. const
2.   csContextLen = 64;  // The input string can have up to 64 characters.
3.   csVocabSize  = 128; // Character based vocabulary/dictionary.
4.   csMinSampleSize = 3; // Minimum of 3 characters.
5.  

After training the NN, I tested with:

Code: Pascal  [Select][+][-]

1.     WriteLn('Testing.');
2.     WriteLn(GenerateStringFromChars(FNN, 'happy'));
3.     WriteLn(GenerateStringFromChars(FNN, 'fantastic'));
4.     WriteLn(GenerateStringFromChars(FNN, 'superb'));
5.  

Then NN gets it 100% right with the following output:

happy good morning.
fantastic good evening.
superb good night.

How does it work?
Each character is encoded into a number from 0 to 127. This number from 0 to 127 is then transformed into a vector with 128 elements where only one element values 1. This is called “one-hot encoding” (https://en.wikipedia.org/wiki/One-hot). This is why the input is defined with (csContextLen, 1, csVocabSize). Then, to decrease the dimensionality from 128 dimensions in the vector, a pointwise convolution is used (TNNetPointwiseConvReLU). Each input character is converted into an 128 elements vector in reverse order so “good” will become “doog”. The last layer of the NN has 128 elements so the element with highest value (or probability) is the next predicted character.

The one-hot encoding is done with one API call:

Code: Pascal  [Select][+][-]

1. procedure TVolume.OneHotEncodingReversed(aTokens: string); overload;

In the case that your input is not a string, you could call the following instead:

Code: Pascal  [Select][+][-]

1. procedure TVolume.OneHotEncoding(aTokens: array of integer); overload;

So, for each character predicted, this character is added to the input string and the process is repeated until a termination token is found (chr(0) or chr(1)). This is how calling GenerateStringFromChars(FNN, 'superb') outputs ‘superb good night’.

The source code can be found at:
https://github.com/joaopauloschuler/neural-api/blob/master/examples/StringManipulation/StringManipulation.lpr

I wish everyone happy pascal coding 

[Dzandaa]

Hi,

Another update
Thank you.

Do you know this ?

https://analyticsindiamag.com/beginners-guide-to-text-generation-with-rnns/

B->

[schuler]

@Dzandaa,
Your question is intelligent and very proper.

At this moment, I'm not coding RNNs nor I have it clear if I will ever code it. To the other readers of this forum that are interested in RNNs, I would recommend the following:

https://www.youtube.com/watch?v=AsNTP8Kwu80 - RNN
https://www.youtube.com/watch?v=YCzL96nL7j0 - LSTM
https://www.youtube.com/watch?v=L8HKweZIOmg - seq2seq
https://www.youtube.com/watch?v=PSs6nxngL6k - Attention

BTW, it's not crazy to think about NLP with convolutional layers:
https://towardsdatascience.com/nlp-with-cnns-a6aa743bdc1e
https://dennybritz.com/posts/wildml/understanding-convolutional-neural-networks-for-nlp/
https://slds-lmu.github.io/seminar_nlp_ss20/convolutional-neural-networks-and-their-applications-in-nlp.html

If you google for "nlp convolutional neural network", you'll find plenty of info.

The example above is a "hello world". ATM, I have 4 VMs running experiments in this area...

[Dzandaa]

Hi,

@schuler

I reproduce your String Manipulation program using Forms and chart.

Feel free to change, share (check errors!!!)  and comment.

Don't forget to change AVX and OpenCL in Project Options!!!

I just tested it with a NVidia card on Windows and Linux Mint.

You have to download CAI from schuler on github : https://github.com/joaopauloschuler/neural-api

and put the "neural" directory from CAI in ../neural , or you can change the path  in Options->Project Options->Path

Have a good day.

B->


 

[schuler]

I've just pushed a tremendous memory optimization (I still have more optimizations to go). It dropped the single thread memory requirement from 4.5GB to 1.7GB for the model below. ATM, each thread consumes an extra 0.6GB.
The model below has 45.6 million parameters also intended to guess the next character in a string from a context of up to 81 characters:

Code: Pascal  [Select][+][-]

1.     FNN := TNNet.Create();
2.     NFit := TNeuralDataLoadingFit.Create();
3.  
4.     FNN.AddLayer([
5.       TNNetInput.Create(csContextLen, 1, csVocabSize),
6.       TNNetPointwiseConv.Create(32,0),
7.       TNNetPadXY.Create(1,0),
8.       TNNetConvolutionReLU.Create(64,3,0,1,0),
9.       TNNetMaxPool.Create(3),
10.       TNNetPadXY.Create(1,0),
11.       TNNetConvolutionReLU.Create(128,3,0,1,0),
12.       TNNetMaxPool.Create(3),
13.       TNNetFullConnectReLU.Create(1024*6),
14.       TNNetFullConnectReLU.Create(1024*6),
15.       TNNetFullConnectLinear.Create(csVocabSize),
16.       TNNetSoftMax.Create()
17.     ]);
18.     NFit.MaxThreadNum := 1;
19.     NFit.LogEveryBatches := 100;
20.     NFit.InitialLearningRate := 0.001;
21.     NFit.LearningRateDecay := 0;
22.     NFit.L2Decay := 0;
23.     NFit.EnableClassComparison();
24.     NFit.EnableDefaultLoss();
25.     NFit.AvgWeightEpochCount := 1; // This is very important for large models
26.     NFit.OnAfterEpoch := @OnAfterEpoch;
27.     NFit.FitLoading(
28.       FNN,
29.       {TrainingVolumesCount=}FDatasetSize*20,
30.       {ValidationVolumesCount=}FDatasetSize,
31.       {TestVolumesCount=}FDatasetSize,
32.       {batchsize=}32,
33.       {epochs=}50,
34.       @GetTrainingPair, @GetValidationPair, @GetTestPair
35.     );
36.  

I have also pushed code for NLP Samplers:

Code: Pascal  [Select][+][-]

1.   { TNNetSamplerGreedy }
2.   TNNetSamplerGreedy = class (TNNetSamplerBase)
3.     public
4.       function GetToken(Origin: TNNetVolume): integer; override;
5.   end;
6.  
7.   { TNNetSamplerTopK }
8.   TNNetSamplerTopK = class (TNNetSamplerBase)
9.     protected
10.       FTopK: integer;
11.     public
12.       constructor Create(TopK: integer);
13.       function GetToken(Origin: TNNetVolume): integer; override;
14.   end;
15.  
16.   { TNNetSamplerTopP }
17.   TNNetSamplerTopP = class (TNNetSamplerBase)
18.     protected
19.       FTopP: TNeuralFloat;
20.     public
21.       constructor Create(TopP: TNeuralFloat);
22.       function GetToken(Origin: TNNetVolume): integer; override;
23.   end;
24.  

I can now talk to my own NNs with:

Code: Pascal  [Select][+][-]

1. procedure TestFromFile;
2. var
3.   S: string;
4.   oSampler: TNNetSamplerBase;
5.   NN: TNNet;
6. begin
7.   oSampler := TNNetSamplerTopP.Create(0.4);
8.   NN := TNNet.Create();
9.   WriteLn('Loading neural network.');
10.   NN.LoadFromFile('JP45A01d2.nn');
11.   NN.DebugStructure();
12.   WriteLn();
13.   WriteLn('Write something and I will reply.');
14.   repeat
15.     Write('User: ');
16.     ReadLn(S);
17.     WriteLn('Neural network: ',GenerateStringFromChars(NN, LowerCase(S), oSampler),'.');
18.   until S = 'exit';
19.   NN.Free;
20.   oSampler.Free;
21. end;

And then, I get outputs:

Code: Pascal  [Select][+][-]

1. User: this is good for
2. Neural network: this is good for the construction of the great deal with the south core comparti.
3. User: this is not good for
4. Neural network: this is not good for the present " consistent with the extended to be seen as r.

The output text is still not human like. But it's improving fast.

[cpicanco]

Hi schuler, do you consider your system feasible to my speech-to-text needs? Please, let me explain.

I am building a custom database with CVCV (consonant-vowel-consonant-vowel) pseudowords. Each pseudoword may have an audio, text and image association. I call them "psedo" because they have no natural meaning, they were artificially created using a mini alphabet composed by 4 vowels and 4 consonants. As you might guess, my goal is to train a model to recognize those words, my urgent need is a speech-to-text solution. 

[schuler]

@cpicanco,
Thank you for considering CAI. CAI is not ready yet for speech recognition. I'll soon have some source code examples in the NLP area but still far from speech recognition. There are mature solutions around for this so I won't drag you into something experimental.

[schuler]

  Hello

In the hope that Neural Networks can be easily learned, I created few videos:

* Basics of Neural Networks in Pascal - Loading and Saving: https://youtu.be/aIy1S7clhQo
* Neural Networks for Absolute Beginners! Learning a Simple Function: https://youtu.be/q56NcgUiAAk
* Coding a Neural Network in Pascal that Learns to Calculate the Hypotenuse: https://youtu.be/PdNTgI_qSyo
* Pre-trained Neural Networks & Transfer Learning with Pascal's CAI Neural API: https://youtu.be/tODsv6Ks2DM
* Coding a Neural Network in Pascal that Learns the OR Boolean Operation: https://youtu.be/f4T9IB-He_k
* A Dive into Identity Shortcut Connection - The ResNet building block: https://youtu.be/o-8NuoSsdck

I wish everyone happy pascal coding