### Bookstore

 Computer Math and Games in Pascal (preview) Lazarus Handbook

### Author Topic: Conscious Artificial Intelligence - Project Update  (Read 25898 times)

#### schuler

• Full Member
• Posts: 179
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #90 on: October 04, 2020, 07:29:11 am »
This is in reply to "All my outputs are now 0. What have I done?".

Volume is a lot bigger number than hypothenuse. So, the network will converge towards fixing volume first. To solve the problem, you can rescale volume with:
Code: Pascal  [Select][+][-]
1. Volume := (LocalX * LocalY * LocalZ)/(100*100);

With the above, your network should converge.

#### daringly

• Jr. Member
• Posts: 54
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #91 on: October 04, 2020, 10:09:04 pm »
Thanks. That adjustment did make it converge, and I can now run with 2 outputs. I'm still getting an output of 0 for the second output (volume) while the desired outputs are correct. It's not critical, since I could simply rerun the nn for each output, but slightly frustrating.

The next thing I'm starting is much more interesting. Using about 8 custom features to predict spread and total results in a sport. I've had a lot of success doing this in the past without neural nets, and I'm wondering how much better this might be.

Edit: I figured out what was making the second ouput 0 -- dividing by (100*100) made the scale too small, but 100 worked reasonably well. Is it better to build separate models for unrelated outputs? In the next major project, most projected spreads (the target solution for my features) will typically be -25 to 25, and target for totals is typically 40-80.

« Last Edit: October 04, 2020, 10:14:28 pm by daringly »

#### daringly

• Jr. Member
• Posts: 54
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #92 on: October 04, 2020, 11:57:53 pm »
As long as I'm thinking wishfully, can we have GPU support?

#### schuler

• Full Member
• Posts: 179
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #93 on: October 05, 2020, 08:40:11 am »
In regards to "can we have GPU support?":
Yes we can!

GPUs are run via OpenCL. Have a look at this example:
https://github.com/joaopauloschuler/neural-api/tree/master/examples/SimpleImageClassifierGPU

Fully connected (dense) layers have OpenCL implementation for the forward pass. While other APIs support only NVIDIA, GPUs from major vendors are supported here.

In the case that you have more than one GPU on your environment, you can have a look here (how to select the correct GPU from a GPU list):

In the case that you need to infer 20 partially unrelated outputs in the same network, you can try a multi-path neural network (this is fully supported by this API).

These Project/Custom (compilation) options might be important to you: AVX, AVX2, AVX512 and OpenCL.
« Last Edit: October 05, 2020, 10:22:19 am by schuler »

#### schuler

• Full Member
• Posts: 179
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #94 on: October 05, 2020, 08:48:49 am »
About "will typically be -25 to 25, and target for totals is typically 40-80", I have a preference for rescaling inputs/outputs to a monopolar range [0.1,0.9] or
bipolar [-1,+1] or even [-2, +2]. This is a personal preference. Smaller numbers avoid overflows.
« Last Edit: October 06, 2020, 10:17:05 am by schuler »

#### josiaslg

• Newbie
• Posts: 2
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #95 on: October 07, 2020, 02:00:28 am »
Hi.
First of all, congratulations for this project. Is really amazing such vision about performance and optimization.
I'm newer in this camp and leaning about the project and process.
I started compile the SimpleFashionMNIST and make the changes to be OpenCL compatible. Work very well wirhout any problem.
Now i have the challange to read directorys with pure photos (png, jpeg, bmp and etc).
To read is ok, but, how i can implement this on neuraldatasets unit ?.
The photos is one category, but, different sizes. Other thing is: how i can test the the results to know how well is training ?.
Sorry if this questions is too newbie.

#### schuler

• Full Member
• Posts: 179
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #96 on: October 07, 2020, 08:29:24 pm »
@josiaslg,
Welcome aboard! Glad to see you here!

Your question is a very good question. Sometimes, we have images in folders like this:
Code: Pascal  [Select][+][-]
1. /myfolder/<class_id>/img1.jpg
2. /myfolder/<class_id>/img2.jpg
3. /myfolder/<class_id>/img3.jpg

For folder structures like the above, you can look here: https://github.com/joaopauloschuler/neural-api#one-class-per-folder-with-image-classification

Sometimes, inside the class folder, you have a subfolder such as "images". Follows an example:
Code: Pascal  [Select][+][-]
1. /myfolder/<class_id>/images/img1.jpg
2. /myfolder/<class_id>/images/img2.jpg
3. /myfolder/<class_id>/images/img3.jpg

In this case, you'll pass 'images' to the parameter pImageSubFolder. It should look like this:
Code: Pascal  [Select][+][-]
1. // change ProportionToLoad to a smaller number if you don't have enough RAM.
4. CreateVolumesFromImagesFromFolder
5. (
6.   ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes,
7.   {FolderName=}'/myfolder', {pImageSubFolder=}'images',
8.   {color_encoding=}csRGB{RGB},
12.   {NewSizeX=}64, {NewSizeY=}64
13. );

Above code will resize input images to 64x64.

I'll eventually add an example loading images "on the fly" for very large datasets. If you need large amounts of RAM such as 128GB to run your models, you can consider running in the cloud.

#### daringly

• Jr. Member
• Posts: 54
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #97 on: October 11, 2020, 04:15:47 pm »
Edit: Issue resolved. Switching to Linear worked, since output was typically -1 to 1, and ReLU ranges positive.

I'm having another "0 Output" issue. All my outputs show 0, even though everything seemed to run correctly.

I started with the Hypotenuse example, and changed the first layer to allow for 22 features:

The datapoint creation was also changed:
TNNetVolumePair.Create(
TNNetVolume.Create([Features[6], Features[7], Features[8], Features[9],
Features[10], Features[11], Features[12], Features[13], Features[14],
Features[15], Features[16], Features[17], Features[18], Features[19],
Features[20], Features[21], Features[22], Features[23], Features[24],
Features[25], Features[26], Features[27]]),
TNNetVolume.Create([noutputs[28]/50])

The output is typically a number from -50 to 50, and usually -20 to 20.

I used weird indexing because it made it easier to visualize where pieces of a .csv file are going -- the features are columns 6-27, and the desired output is in 28 (and 29).

Any idea how I blew this up?
« Last Edit: October 12, 2020, 02:43:10 am by daringly »

#### schuler

• Full Member
• Posts: 179
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #98 on: October 11, 2020, 07:09:13 pm »
What would the expected output range be (-50 to 50)?

#### daringly

• Jr. Member
• Posts: 54
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #99 on: October 11, 2020, 07:19:27 pm »
Before scaling, -50 to 50. I divided the expected output by 50, so it is typically -1 to 1.

#### daringly

• Jr. Member
• Posts: 54
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #100 on: October 11, 2020, 07:20:52 pm »
One other weird issue:

When I execute the following statement, I inspected Features to ensure they were non-0. After processing and showing the inputs, Features 24-27 are always 0 on output (although they aren't initially). Something isn't copying over somehow.

TNNetVolumePair.Create(
TNNetVolume.Create([Features[6], Features[7], Features[8], Features[9],
Features[10], Features[11], Features[12], Features[13], Features[14],
Features[15], Features[16], Features[17], Features[18], Features[19],
Features[20], Features[21], Features[22], Features[23], Features[24],
Features[25], Features[26], Features[27]]),
TNNetVolume.Create([noutputs[28]/50])
« Last Edit: October 11, 2020, 07:34:52 pm by daringly »

#### josiaslg

• Newbie
• Posts: 2
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #101 on: October 13, 2020, 03:56:52 am »
Hi.
I try to do but not generate nn file.
When execute, the images is founded on folders (the folders "images" is separated in 3 classes).
But, simple don't generate nn file.
The source file is here: https://github.com/josiaslg/cai-exemple-photo-from-directory/blob/main/neural.lpr (i not paste code here to not mess up the text and forum).
The OpenCL is ok, i test other example files and everything goes fine (the driver is installed and working on CAi OpenCL examples).

@josiaslg,
Welcome aboard! Glad to see you here!

Your question is a very good question. Sometimes, we have images in folders like this:
Code: Pascal  [Select][+][-]
1. /myfolder/<class_id>/img1.jpg
2. /myfolder/<class_id>/img2.jpg
3. /myfolder/<class_id>/img3.jpg

For folder structures like the above, you can look here: https://github.com/joaopauloschuler/neural-api#one-class-per-folder-with-image-classification

Sometimes, inside the class folder, you have a subfolder such as "images". Follows an example:
Code: Pascal  [Select][+][-]
1. /myfolder/<class_id>/images/img1.jpg
2. /myfolder/<class_id>/images/img2.jpg
3. /myfolder/<class_id>/images/img3.jpg

In this case, you'll pass 'images' to the parameter pImageSubFolder. It should look like this:
Code: Pascal  [Select][+][-]
1. // change ProportionToLoad to a smaller number if you don't have enough RAM.
4. CreateVolumesFromImagesFromFolder
5. (
6.   ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes,
7.   {FolderName=}'/myfolder', {pImageSubFolder=}'images',
8.   {color_encoding=}csRGB{RGB},
12.   {NewSizeX=}64, {NewSizeY=}64
13. );

Above code will resize input images to 64x64.

I'll eventually add an example loading images "on the fly" for very large datasets. If you need large amounts of RAM such as 128GB to run your models, you can consider running in the cloud.

#### schuler

• Full Member
• Posts: 179
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #102 on: October 13, 2020, 12:31:06 pm »
Quote
Before scaling, -50 to 50. I divided the expected output by 50, so it is typically -1 to 1...are always 0 on output

If your Expected output can be negative, if your last layer is ReLU, replacing ReLU layer (as ReLU result is never negative) by a Linear layer might fix the problem.

#### schuler

• Full Member
• Posts: 179
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #103 on: October 13, 2020, 12:32:23 pm »
Quote
But, simple don't generate nn file.

You need validation data to generate the nn file. The nn file is created each time the validation does better.

#### daringly

• Jr. Member
• Posts: 54
##### Re: Conscious Artificial Intelligence - Project Update
« Reply #104 on: October 15, 2020, 12:30:45 am »
Is there a limit to how many inputs you can add? It looks like adding features stopped working after 21 inputs in the code I ran.