Recent

Author Topic: Conscious Artificial Intelligence - Project Update  (Read 59692 times)

Dzandaa

  • Full Member
  • ***
  • Posts: 203
  • From C# to Lazarus
Re: Conscious Artificial Intelligence - Project Update
« Reply #150 on: August 01, 2023, 12:13:55 pm »
Hi,

I have problem with TrainingAccuracy, my program is running well, results are good but TrainingAccuracy is always Zero.

I've send you a personal message with code.

B->
Dzandaa

Dzandaa

  • Full Member
  • ***
  • Posts: 203
  • From C# to Lazarus
Re: Conscious Artificial Intelligence - Project Update
« Reply #151 on: August 02, 2023, 06:10:17 pm »
Hi Schuler,

I tested "plant leaf diseases" and "Colorectal" networks and  it's working :)


B->
« Last Edit: August 03, 2023, 12:26:16 pm by Dzandaa »
Dzandaa

schuler

  • Full Member
  • ***
  • Posts: 222
Re: Conscious Artificial Intelligence - Project Update
« Reply #152 on: November 27, 2023, 12:10:56 am »
@Dzandaa,
What a beautiful screenshot! Wondering if I can find your code in an open repository...

schuler

  • Full Member
  • ***
  • Posts: 222
Re: Conscious Artificial Intelligence - Project Update
« Reply #153 on: November 27, 2023, 12:48:23 am »
:) Hello pascal lovers :) ,
I did an experiment few days ago that I would like to share. The experiment is about predicting the next character in a string.

To make things very simple, the dataset has 3 strings:
Code: Pascal  [Select][+][-]
  1.     FDataset[0] := 'happy good morning.'+chr(1);
  2.     FDataset[1] := 'fantastic good evening.'+chr(1);
  3.     FDataset[2] := 'superb good night.'+chr(1);

The neural network is built with:

Code: Pascal  [Select][+][-]
  1.     FNN.AddLayer([
  2.       TNNetInput.Create(csContextLen, 1, csVocabSize),
  3.       TNNetPointwiseConvReLU.Create(32),
  4.       TNNetFullConnectReLU.Create(32),
  5.       TNNetFullConnectReLU.Create(32),
  6.       TNNetFullConnectLinear.Create(csVocabSize),
  7.       TNNetSoftMax.Create()
  8.     ]);

The constants above are defined with:
Code: Pascal  [Select][+][-]
  1. const
  2.   csContextLen = 64;  // The input string can have up to 64 characters.
  3.   csVocabSize  = 128; // Character based vocabulary/dictionary.
  4.   csMinSampleSize = 3; // Minimum of 3 characters.
  5.  
After training the NN, I tested with:

Code: Pascal  [Select][+][-]
  1.     WriteLn('Testing.');
  2.     WriteLn(GenerateStringFromChars(FNN, 'happy'));
  3.     WriteLn(GenerateStringFromChars(FNN, 'fantastic'));
  4.     WriteLn(GenerateStringFromChars(FNN, 'superb'));
  5.  
Then NN gets it 100% right with the following output:
Quote
happy good morning.
fantastic good evening.
superb good night.

How does it work?
Each character is encoded into a number from 0 to 127. This number from 0 to 127 is then transformed into a vector with 128 elements where only one element values 1. This is called “one-hot encoding” (https://en.wikipedia.org/wiki/One-hot). This is why the input is defined with (csContextLen, 1, csVocabSize). Then, to decrease the dimensionality from 128 dimensions in the vector, a pointwise convolution is used (TNNetPointwiseConvReLU). Each input character is converted into an 128 elements vector in reverse order so “good” will become “doog”. The last layer of the NN has 128 elements so the element with highest value (or probability) is the next predicted character.

The one-hot encoding is done with one API call:
Code: Pascal  [Select][+][-]
  1. procedure TVolume.OneHotEncodingReversed(aTokens: string); overload;

In the case that your input is not a string, you could call the following instead:

Code: Pascal  [Select][+][-]
  1. procedure TVolume.OneHotEncoding(aTokens: array of integer); overload;

So, for each character predicted, this character is added to the input string and the process is repeated until a termination token is found (chr(0) or chr(1)). This is how calling GenerateStringFromChars(FNN, 'superb') outputs ‘superb good night’.

The source code can be found at:
https://github.com/joaopauloschuler/neural-api/blob/master/examples/StringManipulation/StringManipulation.lpr

:) I wish everyone happy pascal coding  :)
« Last Edit: November 27, 2023, 12:56:22 am by schuler »

Dzandaa

  • Full Member
  • ***
  • Posts: 203
  • From C# to Lazarus
Re: Conscious Artificial Intelligence - Project Update
« Reply #154 on: November 27, 2023, 03:50:13 pm »
Dzandaa

schuler

  • Full Member
  • ***
  • Posts: 222
Re: Conscious Artificial Intelligence - Project Update
« Reply #155 on: November 28, 2023, 04:37:48 pm »
@Dzandaa,
Your question is intelligent and very proper.

At this moment, I'm not coding RNNs nor I have it clear if I will ever code it. To the other readers of this forum that are interested in RNNs, I would recommend the following:

https://www.youtube.com/watch?v=AsNTP8Kwu80 - RNN
https://www.youtube.com/watch?v=YCzL96nL7j0 - LSTM
https://www.youtube.com/watch?v=L8HKweZIOmg - seq2seq
https://www.youtube.com/watch?v=PSs6nxngL6k - Attention

BTW, it's not crazy to think about NLP with convolutional layers:
https://towardsdatascience.com/nlp-with-cnns-a6aa743bdc1e
https://dennybritz.com/posts/wildml/understanding-convolutional-neural-networks-for-nlp/
https://slds-lmu.github.io/seminar_nlp_ss20/convolutional-neural-networks-and-their-applications-in-nlp.html

If you google for "nlp convolutional neural network", you'll find plenty of info.

The example above is a "hello world". ATM, I have 4 VMs running experiments in this area...

Dzandaa

  • Full Member
  • ***
  • Posts: 203
  • From C# to Lazarus
Re: Conscious Artificial Intelligence - Project Update
« Reply #156 on: November 29, 2023, 01:07:14 pm »
Hi,

@schuler

I reproduce your String Manipulation program using Forms and chart.

Feel free to change, share (check errors!!!)  and comment.

Don't forget to change AVX and OpenCL in Project Options!!!

I just tested it with a NVidia card on Windows and Linux Mint.

You have to download CAI from schuler on github : https://github.com/joaopauloschuler/neural-api

and put the "neural" directory from CAI in ../neural , or you can change the path  in Options->Project Options->Path

Have a good day.

B->


 
« Last Edit: November 30, 2023, 03:04:04 pm by Dzandaa »
Dzandaa

schuler

  • Full Member
  • ***
  • Posts: 222
Re: Conscious Artificial Intelligence - Project Update
« Reply #157 on: December 02, 2023, 07:34:31 am »
I've just pushed a tremendous memory optimization (I still have more optimizations to go). It dropped the single thread memory requirement from 4.5GB to 1.7GB for the model below. ATM, each thread consumes an extra 0.6GB.
The model below has 45.6 million parameters also intended to guess the next character in a string from a context of up to 81 characters:
Code: Pascal  [Select][+][-]
  1.     FNN := TNNet.Create();
  2.     NFit := TNeuralDataLoadingFit.Create();
  3.  
  4.     FNN.AddLayer([
  5.       TNNetInput.Create(csContextLen, 1, csVocabSize),
  6.       TNNetPointwiseConv.Create(32,0),
  7.       TNNetPadXY.Create(1,0),
  8.       TNNetConvolutionReLU.Create(64,3,0,1,0),
  9.       TNNetMaxPool.Create(3),
  10.       TNNetPadXY.Create(1,0),
  11.       TNNetConvolutionReLU.Create(128,3,0,1,0),
  12.       TNNetMaxPool.Create(3),
  13.       TNNetFullConnectReLU.Create(1024*6),
  14.       TNNetFullConnectReLU.Create(1024*6),
  15.       TNNetFullConnectLinear.Create(csVocabSize),
  16.       TNNetSoftMax.Create()
  17.     ]);
  18.     NFit.MaxThreadNum := 1;
  19.     NFit.LogEveryBatches := 100;
  20.     NFit.InitialLearningRate := 0.001;
  21.     NFit.LearningRateDecay := 0;
  22.     NFit.L2Decay := 0;
  23.     NFit.EnableClassComparison();
  24.     NFit.EnableDefaultLoss();
  25.     NFit.AvgWeightEpochCount := 1; // This is very important for large models
  26.     NFit.OnAfterEpoch := @OnAfterEpoch;
  27.     NFit.FitLoading(
  28.       FNN,
  29.       {TrainingVolumesCount=}FDatasetSize*20,
  30.       {ValidationVolumesCount=}FDatasetSize,
  31.       {TestVolumesCount=}FDatasetSize,
  32.       {batchsize=}32,
  33.       {epochs=}50,
  34.       @GetTrainingPair, @GetValidationPair, @GetTestPair
  35.     );
  36.  

I have also pushed code for NLP Samplers:
Code: Pascal  [Select][+][-]
  1.   { TNNetSamplerGreedy }
  2.   TNNetSamplerGreedy = class (TNNetSamplerBase)
  3.     public
  4.       function GetToken(Origin: TNNetVolume): integer; override;
  5.   end;
  6.  
  7.   { TNNetSamplerTopK }
  8.   TNNetSamplerTopK = class (TNNetSamplerBase)
  9.     protected
  10.       FTopK: integer;
  11.     public
  12.       constructor Create(TopK: integer);
  13.       function GetToken(Origin: TNNetVolume): integer; override;
  14.   end;
  15.  
  16.   { TNNetSamplerTopP }
  17.   TNNetSamplerTopP = class (TNNetSamplerBase)
  18.     protected
  19.       FTopP: TNeuralFloat;
  20.     public
  21.       constructor Create(TopP: TNeuralFloat);
  22.       function GetToken(Origin: TNNetVolume): integer; override;
  23.   end;
  24.  

I can now talk to my own NNs with:
Code: Pascal  [Select][+][-]
  1. procedure TestFromFile;
  2. var
  3.   S: string;
  4.   oSampler: TNNetSamplerBase;
  5.   NN: TNNet;
  6. begin
  7.   oSampler := TNNetSamplerTopP.Create(0.4);
  8.   NN := TNNet.Create();
  9.   WriteLn('Loading neural network.');
  10.   NN.LoadFromFile('JP45A01d2.nn');
  11.   NN.DebugStructure();
  12.   WriteLn();
  13.   WriteLn('Write something and I will reply.');
  14.   repeat
  15.     Write('User: ');
  16.     ReadLn(S);
  17.     WriteLn('Neural network: ',GenerateStringFromChars(NN, LowerCase(S), oSampler),'.');
  18.   until S = 'exit';
  19.   NN.Free;
  20.   oSampler.Free;
  21. end;

And then, I get outputs:
Code: Pascal  [Select][+][-]
  1. User: this is good for
  2. Neural network: this is good for the construction of the great deal with the south core comparti.
  3. User: this is not good for
  4. Neural network: this is not good for the present " consistent with the extended to be seen as r.

The output text is still not human like. But it's improving fast.
« Last Edit: December 02, 2023, 09:04:25 pm by schuler »

cpicanco

  • Hero Member
  • *****
  • Posts: 615
  • Behavioral Scientist and Programmer
    • Portfolio
Re: Conscious Artificial Intelligence - Project Update
« Reply #158 on: December 07, 2023, 01:18:43 am »
Hi schuler, do you consider your system feasible to my speech-to-text needs? Please, let me explain.

I am building a custom database with CVCV (consonant-vowel-consonant-vowel) pseudowords. Each pseudoword may have an audio, text and image association. I call them "psedo" because they have no natural meaning, they were artificially created using a mini alphabet composed by 4 vowels and 4 consonants. As you might guess, my goal is to train a model to recognize those words, my urgent need is a speech-to-text solution. 
Be mindful and excellent with each other.
https://github.com/cpicanco/

schuler

  • Full Member
  • ***
  • Posts: 222
Re: Conscious Artificial Intelligence - Project Update
« Reply #159 on: December 09, 2023, 05:08:28 am »
@cpicanco,
Thank you for considering CAI. CAI is not ready yet for speech recognition. I'll soon have some source code examples in the NLP area but still far from speech recognition. There are mature solutions around for this so I won't drag you into something experimental.

 

TinyPortal © 2005-2018