* * *

Author Topic: artificial neural networks | back propagation  (Read 12661 times)

schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #30 on: July 12, 2018, 09:35:04 pm »
Hi everyone :) ,

Quote
But you are aware that this array clogs 1.2GB of the max. 2GB global memory?

I think that SymolicFrank got the idea. This array is never ever allocated. I just need a pointer type. The big number is there just to avoid range checks when debugging. BTW, I think that I should replace the big number by a constant as shown in the SymbolicFrank's post.

This is the heart of TVolume:
Code: Pascal  [Select]
  1.     FData: array of T;

As you can see, it's not a static array. The pointer is always kept updated:
Code: Pascal  [Select]
  1. procedure TNNetVolume.ReSize(pSizeX, pSizeY, pDepth: integer);
  2. begin
  3.   inherited ReSize(pSizeX, pSizeY, pDepth);
  4.   FDataPtr := addr(FData[0]);
  5. end;
  6.  
  7. function TNNetVolume.GetMemSize(): integer;
  8. begin
  9.   Result := FSize * SizeOf(TNeuralFloat);
  10. end;

schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #31 on: July 12, 2018, 09:52:20 pm »
@Phil
Quote
https://macpgmr.github.io/MacXPlatform/PascalForTensorFlow.html

Very interesting project. I might need your help to properly benchmark CAI against TF in the future.

I have a question to you: have you benchmarked PascalForTensorFlow against Python with TF? I would presume that Pas2TF is a lot faster than Python + TF.

schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #32 on: July 12, 2018, 10:26:12 pm »
@SymbolicFrank

Good questions. I'll reply based on my experience. It means that if you look at a computer science book, you might find other points of view.

Quote
Most processes are multi-step (flowchart): they require multiple actions in sequence (probably all a neural network as well) to get a result. How do you calculate a score for the learning from that?

In my opinion, the 2 main drivers in supervised learning are:

* The sometimes called error or distance or delta from the result to the expected result. In modern times, this is called Loss.
* With the delta/error/distance/loss associated to the slope/derivative/gradient, you know where to apply more correction/learning/descent. For each step of the process, the derivative is calculated for each neuron.

CAI implements the Stochastic Gradient Descent algorithm. CAI calculates an error in the last NN layer and then backpropagates through all layers layer by layer. The backpropagation is done recursively.

Quote
But how do you pinpoint the exact step that was weakest and should be tweaked most? Or would you need a kind of unit test for each step?
This is done via derivatives/slope/gradient and delta/error/distance/loss. If you have a big slope with a big delta on any given neuron/weight, there will be big learning/correction.

Quote
Is the learning always a separate pass to create a file with biases and weights, or can the network keep on learning as it goes? It would need some feedback for that, which is probably generated by a different process and so might have a different format, and might have to be processed itself before it becomes useful. How would you do that? Use another neural network to process the feedback? But that should have learning feedback as well

In the supervised learning, you can keep training for as long as you want. Unfortunately, the learning ability for any computational device (biological or artificial) is always limited in space, energy and information (bits).

As an example, when you are happy with the quality of a "face detection" NN, you can freeze your NN, save your NN to file, compile the code to an Android device (resource limited) and run just the forward pass in your mobile device.

To be continued...
« Last Edit: July 12, 2018, 10:29:52 pm by schuler »

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #33 on: July 12, 2018, 10:32:25 pm »
As an example, when you are happy with the quality of a "face detection" NN, you can freeze your NN, save your NN to file, compile the code to an Android device (resource limited) and run just the forward pass in your mobile device.

Good example. Here's a fairly technical discussion of how Apple did the neural network for face detection:

https://machinelearning.apple.com/2017/11/16/face-detection.html

For Face ID, they use an in-processor neural engine:

https://www.pocket-lint.com/phones/news/apple/142207-what-is-apple-face-id-and-how-does-it-work
« Last Edit: July 12, 2018, 10:44:42 pm by Phil »

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #34 on: July 12, 2018, 10:37:47 pm »
@Phil
Quote
https://macpgmr.github.io/MacXPlatform/PascalForTensorFlow.html

Very interesting project. I might need your help to properly benchmark CAI against TF in the future.

I have a question to you: have you benchmarked PascalForTensorFlow against Python with TF? I would presume that Pas2TF is a lot faster than Python + TF.

Yes, benchmarking sounds interesting.

I don't have a Python version of MNIST, just the Swift version to check my Pascal version's results against. I would guess that almost all processing takes place within the TensorFlow library, not in the calling, so choice of language shouldn't matter much for this example (mostly interative execution of tensor operations by TensorFlow, not much really going on in the Swift or Pascal code).

A non-TensorFlow implementation of MNIST would be interesting to compare memory use as well as performance. TensorFlow can use external GPUs and TPUs, but I have no experience working with them.

Having said that, the discussion of Python's challenges (performance, concurrency, type checking, etc.) and why the Google Brain team branched Swift is quite interesting:

https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md

SymbolicFrank

  • Sr. Member
  • ****
  • Posts: 394
Re: artificial neural networks | back propagation
« Reply #35 on: July 12, 2018, 10:46:36 pm »
Thanks, schuler.

So: everything is a single process and should be taught independently, up front.

Yes, but that limits the usability severely.

schuler

  • Jr. Member
  • **
  • Posts: 90
Re: artificial neural networks | back propagation
« Reply #36 on: July 13, 2018, 03:26:37 am »
Quote
In human vision, first we detect edges and then shapes.
In machine vision, it's exactly the same.

There is an attached image. In this image, we can see the first neuronal layer as 2D RGB images so we can have an idea about what the network is learning. As you can see, some patterns are clearly edge detectors while others are clearly color detectors. It's interesting to note that the pascal code doesn't say "learn edges". This aspect (edges and colors) seems to be a good solution in the gradient descent.

The attached image was generated with a prototype inside CAI source code that anyone can run and reproduce the result. With just 180k weights, the NN has 85% accuracy at the CIFAR-10 test dataset. There is a slight overfitting from 90% to 85%.

SymbolicFrank

  • Sr. Member
  • ****
  • Posts: 394
Re: artificial neural networks | back propagation
« Reply #37 on: July 13, 2018, 09:30:18 am »
What does that picture show?

mw108

  • New member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #38 on: July 13, 2018, 01:36:01 pm »
Quote
But you are aware that this array clogs 1.2GB of the max. 2GB global memory?

I think that SymolicFrank got the idea. This array is never ever allocated. I just need a pointer type. The big number is there just to avoid range checks when debugging. BTW, I think that I should replace the big number by a constant as shown in the SymbolicFrank's post.

Ok, I understand now. Thanks for the explanation. :)

The only reason I'm stressing this is that it seems to kind of limit the available resources when working with your framework. I mean 800 MB left is still quite a lot to work with. But in the end it all depends on the project size and complexity you might want to use CAI in.

Also, as I said earlier, if you want to use a different data type than Single, for instance Double, you already hit the limits, even if you reduce the array size significantly.

As Phil said, a dynamic array implementation might be better here.

SymbolicFrank

  • Sr. Member
  • ****
  • Posts: 394
Re: artificial neural networks | back propagation
« Reply #39 on: July 13, 2018, 03:46:25 pm »
mw108, it is already dynamic. There is no array taking up 1.2 GB.

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #40 on: July 13, 2018, 06:36:00 pm »
I don't have a Python version of MNIST, just the Swift version to check my Pascal version's results against. I would guess that almost all processing takes place within the TensorFlow library, not in the calling, so choice of language shouldn't matter much for this example (mostly interative execution of tensor operations by TensorFlow, not much really going on in the Swift or Pascal code).

Tested MNIST.swift and MNIST.pas on Ubuntu in a 1GB VM with 1 core. Both ran with very similar times, as expected. Interesting that TensorFlow can do this in so little memory, considering a 47MB input data file.

Using watch, memory footprint seems a little higher with the Pascal version - not sure why that would be.

Now we just need a non-TensorFlow version of MNIST to test against.

Note I fixed a memory leak in TF.pas, so if you're working with the Pascal interface, be sure to download the latest code:

https://macpgmr.github.io/MacXPlatform/PascalForTensorFlow.html

Also tested against the new 1.9 version of TensorFlow.

mw108

  • New member
  • *
  • Posts: 21
Re: artificial neural networks | back propagation
« Reply #41 on: July 13, 2018, 08:31:25 pm »
mw108, it is already dynamic. There is no array taking up 1.2 GB.

Hm ok. Then I obviously had a misconception.

If you change TNeuralFloat = Single to TNeuralFloat = Double, you can't compile CAI anymore, because the compiler says that TNeuralFloatArr is too large, because it exeeds the prescribed 2GB memory limit. You have to reduce the size of TNeuralFloatArr = array[0..300000000] of TNeuralFloat so that i compiles again.

If you calculate the size of the array using Single type you get 1.2GB and I assumed that 2GB is the (heap?) memory limit for the whole app, leaving you with only 800MB to work with. But I see that it is obviously only per type / variable. I just tested it and I can create as many similar arrays I like:

Code: Pascal  [Select]
  1. type
  2.   TNeuralFloatArr = array[0..300000000] of TNeuralFloat;
  3.   TNeuralFloatArr2 = array[0..300000000] of TNeuralFloat;
  4.   TNeuralFloatArr3 = array[0..300000000] of TNeuralFloat;
  5.   TNeuralFloatArr4 = array[0..300000000] of TNeuralFloat;
  6.   TNeuralFloatArr5 = array[0..300000000] of TNeuralFloat;
  7.   TNeuralFloatArr6 = array[0..300000000] of TNeuralFloat;
  8.  

Long story short: All good. Thanks for the clarification. Forget what I said.  :D

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #42 on: July 13, 2018, 10:08:32 pm »
Long story short: All good. Thanks for the clarification. Forget what I said.  :D

The compiler doesn't know that the full 300000000 items of Double is going to be allocated or not and so, under 32-bits, it considers that an error. Try it under 64-bit - it should compile okay.

Again, dynamic arrays probably a better approach. With the current approach, if you allocate less than the array type's maximum size, you lose normal range-checking above the allocated upper limit.

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: artificial neural networks | back propagation
« Reply #43 on: July 15, 2018, 07:01:31 pm »
This article should be of interest to anyone reading this thread:

https://www.tmssoftware.com/site/blog.asp?post=466

Interesting approach that TMS uses, putting vector and matrix expressions in a string that they then evaluate. This makes for very concise user code and allows them to support their own vector and matrix operators (eg, x for cross product).

Thaddy

  • Hero Member
  • *****
  • Posts: 7130
Re: artificial neural networks | back propagation
« Reply #44 on: July 15, 2018, 09:50:54 pm »
Although that is a smart approach it is not a fast approach.
Usually speed is here the foremost importance. String manipulation does not help here.
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus