Lazarus

Announcements => Third party => Topic started by: Phil on June 23, 2018, 01:45:32 am

Title: Pascal for TensorFlow
Post by: Phil on June 23, 2018, 01:45:32 am
New article on some first steps toward a Pascal for TensorFlow:

https://macpgmr.github.io

If you have any interest in machine learning, neural networks, matrix arithmetic, this article and downloadable code is for you.

Source is included for several Pascal example programs, from simple to complex. All have been tested against TensorFlow 1.8 on these platforms:

    macOS 10.11 El Capitan
    macOS 10.13 High Sierra
    Ubuntu 14.04

If you want to learn a bit about the things behind machine learning, you might take a look at Google's "crash course". The videos are quite short and easy digestible. Really. I was able to cruise through a half dozen before my brain fogged over.

https://developers.google.com/machine-learning/crash-course

Title: Re: Pascal for TensorFlow
Post by: Phil on June 23, 2018, 02:15:31 pm
Related article, possibly inspirational:

https://appleinsider.com/articles/18/06/06/this-third-year-wwdc-scholarship-winner-built-an-ml-model-to-recognize-beer-yesterday


Machine learning projects often fall into one of several areas: image classification (as in the beer app and the Pascal example MNIST program), natural language text classification, and more general stuff related to numerical categorization and estimation.

Title: Re: Pascal for TensorFlow
Post by: Phil on July 16, 2018, 12:20:51 am
I was curious about how well TensorFlow scales, so I timed some runs of the MNIST.pas image classification demo program. This program is not memory limited on systems with at least 4GB RAM and does not do any disk I/O within the training loop, so it should be a good example of a compute-bound program.

Note that MNIST.pas works with a 60000 x 784 input matrix (188 MB of memory).

On my old Late 2008 MacBook, MNIST took exactly 60 seconds to complete.

Next I tested MNIST on a 2017 MacBook. This is still a dual-core CPU, but a Core i7 which is clocked faster. It took 25 seconds, meaning a nice 2x bump as expected for a newer chip.

Finally I tested MNIST on an 8-core AWS instance running Ubuntu. In tests of other compute-bound models, this system has similar performance to the 2017 MacBook when utilizing only a single core. This system ran MNIST in only 12 seconds, suggesting that it was able to utilize more cores to do the arithmetic faster, without requiring any changes to the Pascal code.

Note that the TensorFlow library has a permissive Apache license. The Pascal for TensorFlow units are released under the same modified LGPL license as FPC RTL, meaning there shouldn't be many restrictions on your use of TensorFlow in any app you might write.
Title: Re: Pascal for TensorFlow
Post by: lainz on July 16, 2018, 02:23:00 am
Thanks I'm interested. I need to find the time to try.

I once was on a live stream of Microsoft Latinoamérica but it was only - as I see it - a way to sell his own service of machine learning. It was too short and I get nothing with it.

Hope this is something I can afford and use.

Edit: oh this is free  :)
Title: Re: Pascal for TensorFlow
Post by: Phil on July 16, 2018, 02:39:20 pm
Edit: oh this is free  :)

Yes, free and open source. I have only tested on macOS and Linux, but TensorFlow library is also available for Windows and Raspian. I've also tested on earlier versions of macOS and Ubuntu than what they give here:

https://www.tensorflow.org/install

The easiest way to obtain the TensorFlow library binaries on macOS and Linux is just to use the simple curl script given here:

https://www.tensorflow.org/install/install_c

Or just download the .gz directly. For example, for Linux or macOS with no GPU support:

https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.9.0.tar.gz

https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-darwin-x86_64-1.9.0.tar.gz

And with all platforms, there are instructions for obtaining TensorFlow via Python's pip command, the assumption probably being that you'll use Python with TensorFlow (although we don't need Python at all with the Pascal interface).

One other way to get a copy of the TensorFlow binaries on macOS and Ubuntu is to install Swift for TensorFlow. I've tested that as well:

https://github.com/tensorflow/swift/blob/master/Installation.md


There's also TensorFlow Lite for mobile and embedded, although again I have not tested it:

https://www.tensorflow.org/mobile/tflite
Title: Re: Pascal for TensorFlow
Post by: jollytall on January 04, 2020, 12:14:06 pm
Phil,
I tried your code. It is a very good start for a beginner like me, so thanks for it.

In your MNIST.pas I think I found two errors:

- When calculating dz1, you have
Code: [Select]
dz1 := dz2.ExecOp('MatMul', Parameters.w2.ExecOp('Transpose', TTensor.CreateInt32([1,0]).Temp).Temp).Temp *
                  h1 * (TTensor.CreateSingle(1).Temp - h1).Temp;
but the first part
Code: [Select]
dz2.ExecOp('MatMul', Parameters.w2.ExecOp('Transpose', TTensor.CreateInt32([1,0]).Temp).Temp).Temp * h1
is already a TTensor, what is not freed. If the IterationCount is larger than in your example, it is a severe memory leak.
It has to be fixed as
Code: [Select]
dz1 := (dz2.ExecOp('MatMul', Parameters.w2.ExecOp('Transpose', TTensor.CreateInt32([1,0]).Temp).Temp).Temp *
                  h1).Temp * (TTensor.CreateSingle(1).Temp - h1).Temp;


- A bit smaller but similar memory leak is in the calculation of part 1
Code: [Select]
part1 := LabelsT.ExecOp('Neg').Temp * predictions.ExecOp('Log');
I t should be
Code: [Select]
part1 := LabelsT.ExecOp('Neg').Temp * predictions.ExecOp('Log').Temp;
and then no memory leak at all.

Title: Re: Pascal for TensorFlow
Post by: jollytall on January 04, 2020, 06:19:15 pm
Can someone help me with a very simple question?

I see that if the T.ExecOp is used then T is passed on as the first parameter of the operation.

I needed some time, how the w2 := TTensor.CreateInt32([30, 10]).Temp.ExecOp('RandomUniform'); line actually first creates a Shape, and using it as a first parameter, generates a Tensor with random numbers, using tf.random.uniform. Looking at it, first I assumed it creates a 2 element Int32 vector and randomizes that. (Usual OO logic, that the method is working on the object and not using the object as an input to a sort-of class method.) Anyway now, I understand how it works.

I also see that ExecOp can take (max 3) additional TTensors as parameters (e.g. for matrix multiplication, etc.), what is useful when the operation needs extra Tensors.

What I do not see, how I can pass other parameters, not Tensors. E.g in Python tf.random.uniform has a format:

tf.random.uniform(
    shape,
    minval=0,
    maxval=None,
    dtype=tf.dtypes.float32,
    seed=None,
    name=None
)

The C interface has seed and seed2, but no min-max. I see that ExecOp declares seed and seed2 and dtype, so, as expected, builds on the C interface.

Does it mean that min-max is a Python extension and neither C, not Pascal has it?

Regarding the other C parameters (e.g. Seed in this case, but in other operations might be more interesting ones), can I specify them in an easy way with the TF unit, or directly calling TensorFlow unit?
Title: Re: Pascal for TensorFlow
Post by: jollytall on February 02, 2020, 06:45:44 pm
FYI - Based on Phil's TensorFlow unit (the pascal wrapper around the C header files) I also made a new unit that can handle Tensors, Graphs, Nodes and Sessions both in the basic way (Graph-Nodes-Tensors-Sessions) and the Eager way, without using any object, class (and Phil's TF unit).
Hence (at least to me), the code is very easy to understand, use and develop further.
If you are interested, please send me a PM and I can send you a copy.
Title: Re: Pascal for TensorFlow
Post by: jollytall on February 13, 2020, 05:28:44 pm
In the meantime I developed it a bit further and uploaded it to GitHub. Please check it at GitHub.com/tensorflowforpascal.
It has specific Exec<oper> functions for over 800 TF functions to run them directly without any Graph creation, or you can also use TGraph and TGraphExt where all the operations are available for building a Graph.
Title: Re: Pascal for TensorFlow
Post by: avra on February 13, 2020, 06:19:13 pm
Please check it at GitHub.com/tensorflowforpascal.
That link is not good. Is https://github.com/zsoltszakaly/tensorflowforpascal the good one?
Title: Re: Pascal for TensorFlow
Post by: jollytall on February 14, 2020, 12:55:31 pm
Oh yes,
Sorry for mistyping it... Good that you found it.
Title: Re: Pascal for TensorFlow
Post by: jollytall on January 19, 2023, 08:06:08 pm
In the last three years Tesnsorflow went from 1.15 to 2.11 and it changed significantly. Now the Pascal interface at https://github.com/zsoltszakaly/tensorflowforpascal (https://github.com/zsoltszakaly/tensorflowforpascal) has been updated to work without error on 2.11 and more examples were added including a simple but complete MNIST solution.
Title: Re: Pascal for TensorFlow
Post by: Root on January 27, 2023, 01:16:32 pm
https://github.com/Pigrecos/TensorFlow.Delphi

this is the implementation of tensorflow and keras that i'm working on occasionally (little free time) without using python etc. based on tensorflow 2.11, but it's mainly written for delphi (version from 10.3 up). The freePascal port is planned (but not a priority at the moment). If anyone would like to help me, they are welcome.
Title: Re: Pascal for TensorFlow
Post by: Thaddy on January 27, 2023, 01:49:07 pm
As far as I can see it already works in {$mode delphi} apart from the dreaded inline variable declarations.
If you agree with me such code is a bad idea, I can provide patches/pull requests against your repository.
Note that bringing the vars to the declaration section will still keep it Delphi compatible.
Title: Re: Pascal for TensorFlow
Post by: Root on January 27, 2023, 03:06:10 pm
As far as I can see it already works in {$mode delphi} apart from the dreaded inline variable declarations.
If you agree with me such code is a bad idea, I can provide patches/pull requests against your repository.
Note that bringing the vars to the declaration section will still keep it Delphi compatible.

yes you are absolutely right, I don't like it either, I don't even like how delphi handles them (especially under debug). But the changes to be made would be considerable. I started with inline variable because it saves a little bit of time, but it wasn't a good idea. :'(
Title: Re: Pascal for TensorFlow
Post by: domasz on January 27, 2023, 03:42:07 pm
I can help fix some inline variables. With just a few people we could get it done by the end of the weekend. Anybody else wants to help?
Title: Re: Pascal for TensorFlow
Post by: Thaddy on January 27, 2023, 04:11:19 pm
Alas there is the rather unfortunate dependence on string4delphi. Will take more than a weekend, I suppose.
Title: Re: Pascal for TensorFlow
Post by: Root on January 27, 2023, 04:54:30 pm
Thanks, any contribution is welcome .

i have limited the use of spring4d. I use it for the Enumerable<TSource> type because it includes practically all the methods of the IEnumerable class of c# and python. then it is used for Tuple and Nullable types.
Title: Re: Pascal for TensorFlow
Post by: domasz on January 27, 2023, 05:51:05 pm
I use it for the Enumerable<TSource> type because it includes practically all the methods of the IEnumerable class of c# and python. then it is used for Tuple and Nullable types.
So perhaps we should just create a replacement for this class in Lazarus and ditch Springs4Delphi.
Title: Re: Pascal for TensorFlow
Post by: domasz on February 03, 2023, 12:13:05 am
I tried but can't compile it under Lazarus because of Springs4Delphi. Anybody knows how to use Springs4Delphi in Lazarus?
Title: Re: Pascal for TensorFlow
Post by: PierceNg on February 03, 2023, 02:44:36 am
I tried but can't compile it under Lazarus because of Springs4Delphi. Anybody knows how to use Springs4Delphi in Lazarus?

There are two separate wrappers mentioned in this thread. Does the other one compile?
Title: Re: Pascal for TensorFlow
Post by: domasz on February 03, 2023, 09:36:11 am
There are two separate wrappers mentioned in this thread. Does the other one compile?
This one? https://github.com/zsoltszakaly/tensorflowforpascal
Yeah, compiles, no problems.

This one https://github.com/Pigrecos/TensorFlow.Delphi uses Spring4Delphi which can't compiled because of types like this:
Code: Pascal  [Select][+][-]
  1. TTimeSpan = TimeSpan.TTimeSpan
  2.   TStopwatch = Diagnostics.TStopwatch;
  3.  
I don't know how to substitute.
Title: Re: Pascal for TensorFlow
Post by: marcov on February 03, 2023, 10:20:09 am
system.Timespan exists in trunk.
Title: Re: Pascal for TensorFlow
Post by: jollytall on April 28, 2023, 05:07:47 pm
I have just found that this thread is still alive and found in it: https://github.com/Pigrecos/TensorFlow.Delphi (https://github.com/Pigrecos/TensorFlow.Delphi). There was a question above on the comparison with https://github.com/zsoltszakaly/tensorflowforpascal (https://github.com/zsoltszakaly/tensorflowforpascal). Now I looked at TensorFlow.Delphi and found that the two are not comparable.
The tensorflowforpascal is only a wrapper around the core Tensorflow functionality, i.e. the elementary operations. In my view it is simple, easy to use, with limited dependency, however it is as basic as basic tensorflow itself is.
The TensorFlow.Delphi on the other hand is a much more sophisticated, complex system, basically implementing the Keras functionality in pascal. As Keras is much more than TensorFlow (we remember that Keras was built as an AI environment over multiple mathematical libraries, and only recently it became TF only), so is its pascal counterpart. It has much more functionality, much more tuned for AI development, but also much more complex and has much more dependencies.
So, if you need something simple in programming, but hard(er) work to make an AI NN, and especially if you want to work under Linux, then I think tensorflowforpascal is a better choice.
On the other hand if you are under Windows/Delphi and are already familiar with the Keras world then TensorFlow.Delphi is a much better choice to migrate Keras to the pascal world.
TinyPortal © 2005-2018