Recent

Author Topic: Conscious Artificial Intelligence - Project Update  (Read 33039 times)

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #105 on: October 15, 2020, 11:08:46 am »
Some of my own models have millions of trainable parameters with 224x224x3 (150k) inputs. So, input size isn't a limit.

daringly

  • Jr. Member
  • **
  • Posts: 62
Re: Conscious Artificial Intelligence - Project Update
« Reply #106 on: October 21, 2020, 07:35:33 pm »
I figured out the error, thanks. I was doing something idiotic.

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #107 on: October 22, 2020, 07:42:42 pm »
Cool.

In the case that you are interested, I coded a new Hypotenuse Example without preallocating all the training data into memory. This is useful if you have a too large dataset to fit into memory (I've seen training datasets approaching 1TB).

https://github.com/joaopauloschuler/neural-api/blob/master/examples/HypotenuseFitLoading/HypotenuseFitLoading.lpr

:) Wish everyone happy pascal coding :)

nouzi

  • Full Member
  • ***
  • Posts: 189
Re: Conscious Artificial Intelligence - Project Update
« Reply #108 on: October 22, 2020, 08:04:14 pm »
Very nice  8-)
my english is  bad
Lazarus 2.0.6 free pascal 3.0.4
Lazarus trunk  free pascal trunk 
System : linux mint 19.3 64bit  windows 7 64bit

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #109 on: December 03, 2020, 01:02:18 am »
@nouzi, thank you for your kind words.

I have 2 news:
  • A new layer type: TNNetConvolutionSharedWeights.
  • Beautiful images created by a neural network.

Beautiful Images
I've attached some images created by a generative adversarial network. The source code can be found here:
https://sourceforge.net/p/cai/svncode/1485/tree/trunk/lazarus/examples/VisualGANTinyImagenet/

TNNetConvolutionSharedWeights
Added today to the API a new convolutional layer type: TNNetConvolutionSharedWeights . Instead of having its own neurons and weights, this convolutional layer uses the same neurons and weights from another existing layer. So, if you need 2 layers with the same neurons, you can add TNNetConvolutionSharedWeights to your network. Why would you need something like this? Maybe, your NN needs to learn the same patterns in different scales (such as big and small dogs are all dogs).

I added an example at:
https://github.com/joaopauloschuler/neural-api/blob/master/examples/SimpleImageClassifier/SimpleImageClassifierSharedWeights.lpr

:) wish everyone happy pascal coding :)

PierceNg

  • Full Member
  • ***
  • Posts: 110
Re: Conscious Artificial Intelligence - Project Update
« Reply #110 on: January 31, 2021, 08:56:51 am »
@schuler This is great stuff! Thanks for making this!

My setup: Windows 10 Home 2H20, Ryzen 2500U, latest AMD drivers, Lazarus 2.0.10 with FPC 3.2.0.  GPU probe shows following:

Platform info: 0 ---------------------
PROFILE: FULL_PROFILE
VERSION: OpenCL 2.1 AMD-APP (3188.4)
NAME: AMD Accelerated Parallel Processing
VENDOR: Advanced Micro Devices, Inc.
EXTENSIONS: cl_khr_icd cl_khr_d3d10_sharing cl_khr_d3d11_sharing cl_khr_dx9_media_sharing cl_amd_event_callback cl_amd_offline_devices
<and so on>


However, when running, say, opencl-dot-product-test.exe, I get many lines of the following:

Error: TDotProductCL.Compute - Failed setting parameters: -48

I am not familiar with OpenCL programming. Is -48 the OpenCL error CL_INVALID_KERNEL? More importantly, does the error mean the GPU wasn't used?

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #111 on: February 06, 2021, 06:49:48 pm »
Hi PierceNg,
Do you have any of the following messages?

Code: Pascal  [Select][+][-]
  1. clCreateContext OK!
  2. clCreateCommandQueue OK!
  3. clCreateProgramWithSource OK!
  4. clBuildProgram OK!
  5. clCreateKernel cai_dot_product OK!
  6. clCreateKernel cai_dot_product OK!

I'm wondering if the file cai_dot_product.cl is correctly been loaded.

I logged a bug report for this:
https://github.com/joaopauloschuler/neural-api/issues/41

Can you please confirm if you are using an APU or a discrete video card? In the case of a discrete video card, what is the model?
« Last Edit: February 07, 2021, 12:04:21 pm by schuler »

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #112 on: February 07, 2021, 06:53:04 pm »
@PierceNg,
I've just tested with a Ryzen APU and was able to reproduce the problem and fix it:
https://github.com/joaopauloschuler/neural-api/issues/41

 :) Wish everyone happy pascal coding  :)

PierceNg

  • Full Member
  • ***
  • Posts: 110
Re: Conscious Artificial Intelligence - Project Update
« Reply #113 on: February 08, 2021, 11:03:33 am »
@PierceNg,
I've just tested with a Ryzen APU and was able to reproduce the problem and fix it:
https://github.com/joaopauloschuler/neural-api/issues/41

 :) Wish everyone happy pascal coding  :)

Thank you!

After pulling from your repos,the dot product and visual CIFAR 10 OpenCL versions run, and I see the "OK!" messages in the command prompt window. My machine is using a Ryzen APU.

mika

  • New Member
  • *
  • Posts: 31
Re: Conscious Artificial Intelligence - Project Update
« Reply #114 on: February 11, 2021, 06:46:54 pm »
HypotenuseFitLoading.lpr
Code: Pascal  [Select][+][-]
  1.     Write('Press ENTER to exit.');
  2.     ReadLn;
  3.   end;
  4.  

should be

Code: Pascal  [Select][+][-]
  1.     Write('Press ENTER to exit.');
  2.     ReadLn;
  3.     Terminate;
  4.   end;
  5.  
Otherwise it keeps repeating DoRun method over and over.

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #115 on: February 14, 2021, 07:49:28 pm »
@mika,
the bug has been fixed as per your suggestion (thank you!):
https://github.com/joaopauloschuler/neural-api/issues/42

PierceNg

  • Full Member
  • ***
  • Posts: 110
Re: Conscious Artificial Intelligence - Project Update
« Reply #116 on: March 09, 2021, 11:58:32 am »
@schuler,

I am happy to report that I have OpenCL Dot Product example running also on my Macbook Pro 2012 model, which has Intel i7 CPU, integrated HD Graphics 4000, and GeForce GT 650M.

Running directly by double-clicking on the app in Finder appears to work - total OpenCL runtime reports 0.02s - but not really. When I run the executable from the shell, the program reports:

File neural.cl could not be found.
Error: Failed to create compute kernel:cai_dot_product
<many many lines of error messages>


I copied neural.cl into the the app bundle directory where the executable opencl-dot-product-test is, then run the executable from the shell again:

clCreateContext OK!
clCreateCommandQueue OK!
clCreateProgramWithSource OK!
clBuildProgram OK!
clCreateKernel cai_dot_product OK!
clCreateKernel cai_dot_product OK!


Screenshot of successful run attached.

But double clicking on the app from Finder still didn't work even with neural.cl already copied in, as in OpenCL runtime is again 0.02s.

Oh, I also had to fix a FPC_FULLVERSION test in the file Lazarus/components/multithreadprocs/mtpcpu.pas in order to build the program.

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #117 on: March 11, 2021, 09:07:01 am »
@PierceNg,
Loved seeing the screenshot from your experiment! Thank you for sharing your test results.

:) May the (fo/sou)rce be with you :)

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #118 on: March 23, 2021, 01:42:50 pm »
 :) Hello :)

I would like to comment a data augmentation approach commonly used with CIFAR-10: the dataset is padded with 4 px increasing the original size from 32x32 to 40x40. Then, at the time of training, a random crop is done bringing the size back to 32x32 px. An example with this approach can be found at https://medium.com/fenwicks/tutorial-2-94-accuracy-on-cifar10-in-2-minutes-7b5aaecd9cdd.

Implementing this in Pascal is very simple:

Code: Pascal  [Select][+][-]
  1. const
  2.   // Padding and cropping constants.
  3.   csPadding = 4;
  4.   csCropSize = csPadding * 2;  
  5. ...
  6.     // Load the CIFAR-10 dataset
  7.     CreateCifar10Volumes(ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes);
  8.  
  9.     // Add padding to dataset resizing from 32x32 to 40x40
  10.     ImgTrainingVolumes.AddPadding(csPadding);
  11.     ImgValidationVolumes.AddPadding(csPadding);
  12.     ImgTestVolumes.AddPadding(csPadding);
  13.  
  14.     NeuralFit := TNeuralImageFit.Create;
  15.  
  16.     // Enable cropping while fitting.
  17.     NeuralFit.HasImgCrop := true;
  18.     NeuralFit.MaxCropSize := csCropSize;
  19.     NeuralFit.HasFlipX := true;

This simple trick actually seems to outperform CAI's default image data augmentation method. A fully functional source code example can be found at: https://github.com/joaopauloschuler/neural-api/blob/master/examples/SimpleImageClassifier/SimpleImageClassifierPaddingCropping.lpr

:) May the source be with you :)
« Last Edit: March 23, 2021, 01:48:54 pm by schuler »

schuler

  • Full Member
  • ***
  • Posts: 191
Re: Conscious Artificial Intelligence - Project Update
« Reply #119 on: April 01, 2021, 03:46:08 pm »
:) Hello :)
I think that I have never commented the Neural API threading in detail here. I got a couple of questions in regards to this recently. I hope that I'm not typing too much.

As an example, assuming that you need to run a procedure 10 times in parallel, you can create 10 thread workers as follows:

Code: Pascal  [Select][+][-]
  1. FProcs := TNeuralThreadList.Create( 10 );

As an example, this is the procedure that we intend to run in parallel:

Code: Pascal  [Select][+][-]
  1. procedure MyClassName.RunNNThread(index, threadnum: integer);
  2. begin
  3.   WriteLn('This is thread ',index,' out of ',threadnum,' threads.');
  4. end;

Then, to run the procedure RunNNThread passed as parameter 10 times in parallel, do this:

Code: Pascal  [Select][+][-]
  1. FProcs.StartProc(@RunNNThread);

You can control the blocking mode (waiting threads to finish before the program continues) as per declaration:

Code: Pascal  [Select][+][-]
  1. procedure StartProc(pProc: TNeuralProc; pBlock: boolean = true);

Or, if you prefer, you can specifically say when to wait for threads to finish as per this example:

Code: Pascal  [Select][+][-]
  1. FProcs.StartProc(@RunNNThread, false);
  2. // insert your code here
  3. FProcs.WaitForProc(); // waits until all threads are finished.

When you are done, you should call:

Code: Pascal  [Select][+][-]
  1. FProcs.Free;

neuralthread is the unit that allows parallel processing when using neural fit: https://github.com/joaopauloschuler/neural-api/blob/master/neural/neuralfit.pas

Wish long and prosper life to FPC and Lazarus!
« Last Edit: April 01, 2021, 03:54:00 pm by schuler »

 

TinyPortal © 2005-2018