Recent

Author Topic: Pascal for TensorFlow  (Read 12477 times)

Phil

  • Hero Member
  • *****
  • Posts: 2737
Pascal for TensorFlow
« on: June 23, 2018, 01:45:32 am »
New article on some first steps toward a Pascal for TensorFlow:

https://macpgmr.github.io

If you have any interest in machine learning, neural networks, matrix arithmetic, this article and downloadable code is for you.

Source is included for several Pascal example programs, from simple to complex. All have been tested against TensorFlow 1.8 on these platforms:

    macOS 10.11 El Capitan
    macOS 10.13 High Sierra
    Ubuntu 14.04

If you want to learn a bit about the things behind machine learning, you might take a look at Google's "crash course". The videos are quite short and easy digestible. Really. I was able to cruise through a half dozen before my brain fogged over.

https://developers.google.com/machine-learning/crash-course


Phil

  • Hero Member
  • *****
  • Posts: 2737
Re: Pascal for TensorFlow
« Reply #1 on: June 23, 2018, 02:15:31 pm »
Related article, possibly inspirational:

https://appleinsider.com/articles/18/06/06/this-third-year-wwdc-scholarship-winner-built-an-ml-model-to-recognize-beer-yesterday


Machine learning projects often fall into one of several areas: image classification (as in the beer app and the Pascal example MNIST program), natural language text classification, and more general stuff related to numerical categorization and estimation.


Phil

  • Hero Member
  • *****
  • Posts: 2737
Re: Pascal for TensorFlow
« Reply #2 on: July 16, 2018, 12:20:51 am »
I was curious about how well TensorFlow scales, so I timed some runs of the MNIST.pas image classification demo program. This program is not memory limited on systems with at least 4GB RAM and does not do any disk I/O within the training loop, so it should be a good example of a compute-bound program.

Note that MNIST.pas works with a 60000 x 784 input matrix (188 MB of memory).

On my old Late 2008 MacBook, MNIST took exactly 60 seconds to complete.

Next I tested MNIST on a 2017 MacBook. This is still a dual-core CPU, but a Core i7 which is clocked faster. It took 25 seconds, meaning a nice 2x bump as expected for a newer chip.

Finally I tested MNIST on an 8-core AWS instance running Ubuntu. In tests of other compute-bound models, this system has similar performance to the 2017 MacBook when utilizing only a single core. This system ran MNIST in only 12 seconds, suggesting that it was able to utilize more cores to do the arithmetic faster, without requiring any changes to the Pascal code.

Note that the TensorFlow library has a permissive Apache license. The Pascal for TensorFlow units are released under the same modified LGPL license as FPC RTL, meaning there shouldn't be many restrictions on your use of TensorFlow in any app you might write.

lainz

  • Hero Member
  • *****
  • Posts: 4449
    • https://lainz.github.io/
Re: Pascal for TensorFlow
« Reply #3 on: July 16, 2018, 02:23:00 am »
Thanks I'm interested. I need to find the time to try.

I once was on a live stream of Microsoft Latinoamérica but it was only - as I see it - a way to sell his own service of machine learning. It was too short and I get nothing with it.

Hope this is something I can afford and use.

Edit: oh this is free  :)
« Last Edit: July 16, 2018, 02:58:37 am by lainz »

Phil

  • Hero Member
  • *****
  • Posts: 2737
Re: Pascal for TensorFlow
« Reply #4 on: July 16, 2018, 02:39:20 pm »
Edit: oh this is free  :)

Yes, free and open source. I have only tested on macOS and Linux, but TensorFlow library is also available for Windows and Raspian. I've also tested on earlier versions of macOS and Ubuntu than what they give here:

https://www.tensorflow.org/install

The easiest way to obtain the TensorFlow library binaries on macOS and Linux is just to use the simple curl script given here:

https://www.tensorflow.org/install/install_c

Or just download the .gz directly. For example, for Linux or macOS with no GPU support:

https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.9.0.tar.gz

https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-darwin-x86_64-1.9.0.tar.gz

And with all platforms, there are instructions for obtaining TensorFlow via Python's pip command, the assumption probably being that you'll use Python with TensorFlow (although we don't need Python at all with the Pascal interface).

One other way to get a copy of the TensorFlow binaries on macOS and Ubuntu is to install Swift for TensorFlow. I've tested that as well:

https://github.com/tensorflow/swift/blob/master/Installation.md


There's also TensorFlow Lite for mobile and embedded, although again I have not tested it:

https://www.tensorflow.org/mobile/tflite

jollytall

  • Sr. Member
  • ****
  • Posts: 303
Re: Pascal for TensorFlow
« Reply #5 on: January 04, 2020, 12:14:06 pm »
Phil,
I tried your code. It is a very good start for a beginner like me, so thanks for it.

In your MNIST.pas I think I found two errors:

- When calculating dz1, you have
Code: [Select]
dz1 := dz2.ExecOp('MatMul', Parameters.w2.ExecOp('Transpose', TTensor.CreateInt32([1,0]).Temp).Temp).Temp *
                  h1 * (TTensor.CreateSingle(1).Temp - h1).Temp;
but the first part
Code: [Select]
dz2.ExecOp('MatMul', Parameters.w2.ExecOp('Transpose', TTensor.CreateInt32([1,0]).Temp).Temp).Temp * h1
is already a TTensor, what is not freed. If the IterationCount is larger than in your example, it is a severe memory leak.
It has to be fixed as
Code: [Select]
dz1 := (dz2.ExecOp('MatMul', Parameters.w2.ExecOp('Transpose', TTensor.CreateInt32([1,0]).Temp).Temp).Temp *
                  h1).Temp * (TTensor.CreateSingle(1).Temp - h1).Temp;


- A bit smaller but similar memory leak is in the calculation of part 1
Code: [Select]
part1 := LabelsT.ExecOp('Neg').Temp * predictions.ExecOp('Log');
I t should be
Code: [Select]
part1 := LabelsT.ExecOp('Neg').Temp * predictions.ExecOp('Log').Temp;
and then no memory leak at all.


jollytall

  • Sr. Member
  • ****
  • Posts: 303
Re: Pascal for TensorFlow
« Reply #6 on: January 04, 2020, 06:19:15 pm »
Can someone help me with a very simple question?

I see that if the T.ExecOp is used then T is passed on as the first parameter of the operation.

I needed some time, how the w2 := TTensor.CreateInt32([30, 10]).Temp.ExecOp('RandomUniform'); line actually first creates a Shape, and using it as a first parameter, generates a Tensor with random numbers, using tf.random.uniform. Looking at it, first I assumed it creates a 2 element Int32 vector and randomizes that. (Usual OO logic, that the method is working on the object and not using the object as an input to a sort-of class method.) Anyway now, I understand how it works.

I also see that ExecOp can take (max 3) additional TTensors as parameters (e.g. for matrix multiplication, etc.), what is useful when the operation needs extra Tensors.

What I do not see, how I can pass other parameters, not Tensors. E.g in Python tf.random.uniform has a format:

tf.random.uniform(
    shape,
    minval=0,
    maxval=None,
    dtype=tf.dtypes.float32,
    seed=None,
    name=None
)

The C interface has seed and seed2, but no min-max. I see that ExecOp declares seed and seed2 and dtype, so, as expected, builds on the C interface.

Does it mean that min-max is a Python extension and neither C, not Pascal has it?

Regarding the other C parameters (e.g. Seed in this case, but in other operations might be more interesting ones), can I specify them in an easy way with the TF unit, or directly calling TensorFlow unit?
« Last Edit: January 04, 2020, 06:28:38 pm by jollytall »

jollytall

  • Sr. Member
  • ****
  • Posts: 303
Re: Pascal for TensorFlow
« Reply #7 on: February 02, 2020, 06:45:44 pm »
FYI - Based on Phil's TensorFlow unit (the pascal wrapper around the C header files) I also made a new unit that can handle Tensors, Graphs, Nodes and Sessions both in the basic way (Graph-Nodes-Tensors-Sessions) and the Eager way, without using any object, class (and Phil's TF unit).
Hence (at least to me), the code is very easy to understand, use and develop further.
If you are interested, please send me a PM and I can send you a copy.

jollytall

  • Sr. Member
  • ****
  • Posts: 303
Re: Pascal for TensorFlow
« Reply #8 on: February 13, 2020, 05:28:44 pm »
In the meantime I developed it a bit further and uploaded it to GitHub. Please check it at GitHub.com/tensorflowforpascal.
It has specific Exec<oper> functions for over 800 TF functions to run them directly without any Graph creation, or you can also use TGraph and TGraphExt where all the operations are available for building a Graph.

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Pascal for TensorFlow
« Reply #9 on: February 13, 2020, 06:19:13 pm »
Please check it at GitHub.com/tensorflowforpascal.
That link is not good. Is https://github.com/zsoltszakaly/tensorflowforpascal the good one?
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

jollytall

  • Sr. Member
  • ****
  • Posts: 303
Re: Pascal for TensorFlow
« Reply #10 on: February 14, 2020, 12:55:31 pm »
Oh yes,
Sorry for mistyping it... Good that you found it.

jollytall

  • Sr. Member
  • ****
  • Posts: 303
Re: Pascal for TensorFlow
« Reply #11 on: January 19, 2023, 08:06:08 pm »
In the last three years Tesnsorflow went from 1.15 to 2.11 and it changed significantly. Now the Pascal interface at https://github.com/zsoltszakaly/tensorflowforpascal has been updated to work without error on 2.11 and more examples were added including a simple but complete MNIST solution.

Root

  • New member
  • *
  • Posts: 7
Re: Pascal for TensorFlow
« Reply #12 on: January 27, 2023, 01:16:32 pm »
https://github.com/Pigrecos/TensorFlow.Delphi

this is the implementation of tensorflow and keras that i'm working on occasionally (little free time) without using python etc. based on tensorflow 2.11, but it's mainly written for delphi (version from 10.3 up). The freePascal port is planned (but not a priority at the moment). If anyone would like to help me, they are welcome.

Thaddy

  • Hero Member
  • *****
  • Posts: 14169
  • Probably until I exterminate Putin.
Re: Pascal for TensorFlow
« Reply #13 on: January 27, 2023, 01:49:07 pm »
As far as I can see it already works in {$mode delphi} apart from the dreaded inline variable declarations.
If you agree with me such code is a bad idea, I can provide patches/pull requests against your repository.
Note that bringing the vars to the declaration section will still keep it Delphi compatible.
Specialize a type, not a var.

Root

  • New member
  • *
  • Posts: 7
Re: Pascal for TensorFlow
« Reply #14 on: January 27, 2023, 03:06:10 pm »
As far as I can see it already works in {$mode delphi} apart from the dreaded inline variable declarations.
If you agree with me such code is a bad idea, I can provide patches/pull requests against your repository.
Note that bringing the vars to the declaration section will still keep it Delphi compatible.

yes you are absolutely right, I don't like it either, I don't even like how delphi handles them (especially under debug). But the changes to be made would be considerable. I started with inline variable because it saves a little bit of time, but it wasn't a good idea. :'(

 

TinyPortal © 2005-2018