AI, NLP and CAI: Text Generation with Convolutional Neural Networks in Pascal

[indydev]

After looking into it. It looks like OpenAI removed the assistant sharing quite some time ago. Probably because the API is tied to accounts and thus charges would be applied to the Assistant owner.

[schuler]

What I'm looking for is how to divide a layer into two branches and join one of the two branches further into another layer.

@Dzandaa,
For CIFAR-10 32x32x3 images, this is an example showing how to create 3 branches being one branch for each channel: R, G and B:

Code: Pascal  [Select][+][-]

1.         InputLayer := NN.AddLayer( TNNetInput.Create(32,32,3) );        
2.        
3.         // Branch1
4.         NN.AddLayerAfter( TNNetSplitChannels.Create([0]), InputLayer);
5.         NN.AddLayer( TNNetConvolutionReLU.Create(FeaturesNum, FeatureSize, iInputPadding, Stride) );
6.         NN.AddLayer( TNNetMaxPool.Create(4) );
7.         Branch1 := NN.AddLayer(TNNetConvolutionReLU.Create(21, iInnerConvFeatureSize, iPadding, 0));
8.  
9.         // Branch2
10.         NN.AddLayerAfter( TNNetSplitChannels.Create([1]), InputLayer);
11.         NN.AddLayer( TNNetConvolutionReLU.Create(FeaturesNum, FeatureSize, iInputPadding, Stride) );
12.         NN.AddLayer( TNNetMaxPool.Create(4) );
13.         Branch2 := NN.AddLayer(TNNetConvolutionReLU.Create(22, iInnerConvFeatureSize, iPadding, 0));
14.  
15.         // Branch3
16.         NN.AddLayerAfter( TNNetSplitChannels.Create([2]), InputLayer);
17.         NN.AddLayer( TNNetConvolutionReLU.Create(FeaturesNum, FeatureSize, iInputPadding, Stride) );
18.         NN.AddLayer( TNNetMaxPool.Create(4) );
19.         Branch3 := NN.AddLayer( TNNetConvolutionReLU.Create(21, iInnerConvFeatureSize, iPadding, 0) );
20.  
21.         NN.AddLayer( TNNetDeepConcat.Create([Branch1, Branch2, Branch3]) );
22.  
23.         if fDropout > 0 then NN.AddLayer( TNNetDropout.Create(fDropout) );
24.  
25.         NN.AddLayer( TNNetLayerStdNormalization.Create() );
26.         NN.AddLayer( TNNetConvolutionReLU.Create(iConvNeuronCount, iInnerConvFeatureSize, iPadding, 0) );
27.  
28.         NN.AddLayer( TNNetFullConnectReLU.Create(32) );
29.         NN.AddLayer( TNNetFullConnectReLU.Create(32) );
30.         NN.AddLayer( TNNetFullConnectLinear.Create(NumClasses) );
31.         NN.AddLayer( TNNetSoftMax.Create() );

The example above uses TNNetSplitChannels and TNNetDeepConcat. The other way of branching (very popular with Keras/PyTorch) is using pointwise convolutions or just any convolution for splitting:

Code: Pascal  [Select][+][-]

1.        
2.         InputLayer := NN.AddLayer( TNNetInput.Create(32,32,3) );        
3.  
4.         // Branch1 (3x3 features)
5.         Branch1 := NN.AddLayerAfter(TNNetConvolutionReLU.Create(FeaturesNum, 3, 1, Stride),InputLayer);
6.  
7.         // Branch2 (5x5 features)
8.         Branch2 := NN.AddLayerAfter(TNNetConvolutionReLU.Create(FeaturesNum, 5, 2, Stride),InputLayer);
9.  
10.         // Branch3 (7x7 features)
11.         Branch3 := NN.AddLayerAfter(TNNetConvolutionReLU.Create(FeaturesNum, 7, 3, Stride),InputLayer);
12.  
13.         // Concats all 3 branches so the NN has only one end.
14.         NN.AddLayer(TNNetDeepConcat.Create([Branch1, Branch2, Branch3]));        
15.         NN.AddLayer( TNNetFullConnectLinear.Create(NumClasses) );
16.         NN.AddLayer( TNNetSoftMax.Create() );

[schuler]

Regarding the chatGPT interface version. Is that available as an assistant in the playground? I don't have a PRO account, but use the API through my (still limited) client.

I probably should create another slightly downgraded assistant that could be used for free (without paid subscription) at poe.com. What do you think?

[Dzandaa]

Hi
@schuler

Just what I need, thank you very much!!!

B->

[indydev]

Regarding the chatGPT interface version. Is that available as an assistant in the playground? I don't have a PRO account, but use the API through my (still limited) client.

I probably should create another slightly downgraded assistant that could be used for free (without paid subscription) at poe.com. What do you think?

That would work. Thank you.

[schuler]

@indydev,
As requested, the free version is ready to use:
https://poe.com/CAI-NEURAL-API-FREE

[indydev]

@indydev,
As requested, the free version is ready to use:
https://poe.com/CAI-NEURAL-API-FREE

Nice! I have had just a short time with this on poe.com. I can see that there is some length limitation with poe that I get around by asking it to continue. It is very good at providing answers and examples to simple questions. But when I uploaded some code (Javascript and Free Pascal) and asked it to evaluate the translation it responded in Spanish (perhaps seeing "translation" or "rewrite" as a request to change the human interactive language). https://poe.com/s/kR9wcAfZ8ezLcLkeJw1A Requesting it in english fixed that problem. However, I finished up my free usage rather quickly. The response is pretty impressive--especially the speed. Thanks.

[avra]

As requested, the free version is ready to use:
https://poe.com/CAI-NEURAL-API-FREE

That is fun to play with 

[schuler]

@indydev,
Regarding the Spanish, I got the same problem here.

You now should be able to use the openai version for free:

https://chatgpt.com/g/g-bqMxEDpIg-neural-api-free-pascal-developer

You can try general pascal coding tasks if you like (besides AI tasks).

I think that you'll be able to use without paying because I got the following email in my inbox:

Today, GPTs in the Store are now available to all ChatGPT users, allowing more than 100 million weekly active users to experience your GPT and the others in the store.

[indydev]

I was just going to let you know that I found the chatGPT version available a couple of days ago. I get more use from it here than with POE.  I haven't looked at your contribution at Hugging Face, other than to see that it is there, but I want to look at it to see what you have done.  When I use the chatGPT version, it does seem to do better than the normal version. Thank you so much!

I am also very much interested in TLMs since it might be possible to run them locally, reducing the need to reach out to an LLM.

[schuler]

Hello,
About 5 months ago, I wrote the following in this very same thread:

Is this the very first time that a transformer block is coded in Pascal? It's probable. Is it ready to use? Probably not. I'll eventually have it fully tested and examples to show for both NLP and Computer Vision. Anyway, it has been born!

Although I can’t say that it’s ready to use, I got a pure free pascal example ready to run on google colab. It contains a stack of 2 transformer decoder blocks and is trained with the tiny stories dataset:
https://huggingface.co/datasets/schuler/TinyStories4Pascal

You can run the free pascal source code example on google colab:
https://colab.research.google.com/github/joaopauloschuler/neural-api/blob/master/examples/SimpleNLP/transformer_decoder_tiny_stories_dataset_3k_vocab.ipynb

For now, it’s a CPU only implementation. After training, it produces sentences such as:

• one day , a little boy named tim went to the park with his mom . they saw a big tree with many leaves . tim wanted to climb the tree , but his mom said , " no , tim , you can ' t climb the tree . it ' s not safe . " tim was sad and said , " but i want to climb the tree . " his mom said , " don '.
  
• once upon a time , there was a little girl named lily . she loved to play outside in the sunshine . one day , she saw a big , scary dog . the dog looked scary and lily was scared . she ran to her mommy and said , " mommy , there ' s a big dog outside ! " her mommy said , " don ' t worry , lily . the dog is friendly and friendly.
• billy was a little boy who loved to play . he had a lot of toys , but his favorite was a big red ball . he would throw it up and catch it , and he would throw it . one day , billy was playing with his ball in the park . he saw a big , shiny red ball and he wanted to play with it . he ran to the ball and tried to grab it.

The above output was produced with:

Code: Pascal  [Select][+][-]

1.     WriteLn(GenerateStringFromCasualNN(NFit.NN, FDictionary, 'one day', nil),'.');
2.     WriteLn(GenerateStringFromCasualNN(NFit.NN, FDictionary, 'once upon a', nil),'.');
3.     WriteLn(GenerateStringFromCasualNN(NFit.NN, FDictionary, 'billy', FSampler),'.');

The 7.2 million parameters neural network is created with:

Code: Pascal  [Select][+][-]

1.     FNN.AddLayer([
2.       TNNetInput.Create(csContextLen, 1, 1),
3.       TNNetTokenAndPositionalEmbedding.Create(csModelVocabSize, csEmbedDim, 0, 0.02, 0.01),
4.       TNNetPointwiseConvLinear.Create(512)
5.     ]);
6.  
7.     for I := 1 to 2 do FNN.AddTransformerBlockCAI(8, 2048, true, false, false);
8.  
9.     FNN.AddLayer([
10.       TNNetPointwiseConvLinear.Create(csEmbedDim),
11.       TNNetPointwiseConvLinear.Create(csModelVocabSize),
12.       TNNetPointwiseSoftMax.Create(1)
13.     ]);

I wish everyone happy pascal coding!

[indydev]

Thank you so much for this!  I will check this out soon. I am traveling right now, but am looking forward to trying this.

[Dzandaa]

Hi Joao-Paulo,

Great Job!!!

I'm happy that you continue to work on CAI.

Is the final training available?

Thank you.

B->

[schuler]

@Dzandaa,
I haven't published any trained transformer decoder model yet. But given that you are asking, I certainly will. I'll let you know when ready to download.

[schuler]

Hello ,
I did something that has 0% of practical usage and 100% of fun. I coded a model similar to GPT-3 Small from Open AI in pascal: https://github.com/joaopauloschuler/gpt-3-for-pascal/tree/main .

I say 0% of practical usage because I can't think about anyone using this model for any practical usage. I say 100% of fun because it feels so good coding it...

Have Fun