Lazarus

Programming => General => Topic started by: otoien on December 12, 2019, 07:10:00 am

Title: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: otoien on December 12, 2019, 07:10:00 am
OK, let me say up front that I am aware that these kind of less concrete request are difficult to respond to in these fora vs. the ones that can deal directly with code, but I am seeking experiences from those who have made similar efforts in the past, and if there perhaps in hind sight was learning experiences that would have lead to a different strategy:

Introduction: I am in the early process of converting a data acquisition application written in TP 5.5 that I have developed and maintained over more than 30 years up to current days, and that I and my colleagues rely heavily upon for our research. Some work with respect to establishing new file format and translation of old parameter files has already been done (I chose the .ini file format, building upon the IniFiles unit, and expanded and specialized for my purpose. One of the big priorities is stable file formats that can be expanded upon and be able to provide backwards compatibility though import of old binary parameter files). Since then there has been a lapse in the efforts, which I am now funded to continue. I am concerned about maintaining stability, and integrity of this very well tested code with respect to correctness of calculations. The data file format for slow sampling is simple csv-tab-delimited which I see no reason to change except for UTF-8 compatible variable headings and units. The code is about 90,000 program lines so this is not a small task and is going to take a lot of time.

I wrote my own text based and graph based menu systems in TP5.5, so it is not based on any third party user interface libraries. The graphing part is based on a subset of the Turbo Halo library that provides basic graphic primitives (including an excellent world coordinate system that allows mapping to real coordinates of a plot). However code for graphing will have to be replaced, possibly using TAChart, unless I find some obstacles with respect to special features that necessitates translating my old code.

The program collects data and controls hardware (mostly though serial port these days, but also old support for a Labmaster board and parallel ports), will scale data, perform specialized calibrations and calculations in my field of expertise, and plots data in real time. In addition there are a number of routines for specialized imports and  data manipulation. A challenge is some overlap in the use of parameters between these functionalities.

A main point that has caused hesitation is to what degree to "objectify" (i. e create manageable classes) of the code, and at what level of detail. The current code is almost exclusively procedural, but in many cases passing a large number of parameters to the procedures which in some respect can be called an object oriented philosophy of providing an interface. However the parameters for scaling and calibrations, very extensive calculations, and graphing is stored in global dynamic records. I am pretty sure I will have to use multi threading for the data acquisition part. The main data are stored in complicated dynamically allocated array structures (to overcome 64K limitations of DOS memory), also global, that I think can be simplified by using a dynamic array with step wise allocation of new chunks of "cells" as before. (But how to access it in an object oriented way with so many parts of the program manipulating and using it?) 

My current idea is to wrap pretty large portions of code in classes according to functionality that contain its own parameters in each of these areas, and and that each class will also have the code to call the ini file code to save the parameters to a common ini-file (completely rewriting the inifile each time) that goes with the tab delimited text file.

However then I read an old comment by Marco Cantu (which I unfortunately cannot find at the moment as it is a while ago) where he recommended to keep as much code as possible in its original procedural form when converting a DOS program - in other words  not go very far into creating classes of it.

At this early stage I am concerned about not working myself into a corner, as I envision this application to be usable  and maintainable long into the future. So again, are there any experiences to take along on the way?

I have published a couple of papers relating to some functional aspects of the code if anyone get very interested:
Code: Text  [Select][+][-]
  1. Tøien, Ø. (1992) Data acquisition in thermal physiology: measurements of shivering. J. Therm. Biol. 17:357-366.
  2. Tøien, Ø. (2013) Automated open flow respirometry in continuous and long-term measurements: design and principles. J. Appl. Physiol. 114:1094-1107
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: JuhaManninen on December 12, 2019, 08:38:54 am
I would start to design a "perfect" object structure with properties and methods to interact with it.
Actual code should be ported only after it is ready.
It requires lots of changes. Many people would prefer only minimal necessary changes to the procedural code when the file format or libraries etc. so require.
One choice is to port in 2 steps. First minimal changes to get a working implementation. Then "objectify" into well designed classes.
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: marcov on December 12, 2019, 08:57:39 am
I would start working in the existing app to get some form of GUI - business code separation.

Then I would try to attach that business code to a new lazarus/LCL gui with TChart and the rest.

Only then, (and even ...Maybe), would I think about optimizing the structure of the business code.
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: PaulRowntree on December 12, 2019, 09:23:30 am
In my opinion, the biggest challenge is rethinking the front-end UI in terms of what is easily done in Laz, and rethinking of the experimental structure.  What seemed natural using units to divide and conquer may be completely different from a functional object based approach.  Describing your process to a programmer that has no knowledge of the TP5.5 code might be a good step to help re-imagine the structure.
We recently did complete rewrites of 3 major lab codes, moving from Delphi, LabView to Lazarus/FP.  Lots of serial ports (I used LazSerial).  On a previous project (Delphi) I was trying way too hard to be clever with binary and Real48 data structures on disk ... what a mistake!  Now it is all ASCII for me.  Today's machines are so fast, and memory is so cheap, that squeezing wasn't worth the effort.  Our new data approaches, all objectified, are almost future proof because the files are self-documenting, expandable, and text based.  We have adapted the JCAMP-file exchange format to be the primary data storage, and it is wprking well.  We move and load hundreds of 100KB spectra in a few seconds, reprocess data in a blink. I am glad we did it, but it was a long struggle, and the students had to put up with daily new versions for a while.  Part of this was due to programming at home away from the hardware ... makes for a long debug cycle.
I'd be happy to help if you could give an overview of how much data is in memory, acquisition rates, etc.  Most important is how you see the data ... X-Y arrays of data?  Collection of stim/response experiments?What does the experiment look like?
gracie dot rowntree at gmail dot com
Cheers!
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: MarkMLl on December 12, 2019, 09:36:56 am
I would start working in the existing app to get some form of GUI - business code separation.

Then I would try to attach that business code to a new lazarus/LCL gui with TChart and the rest.

Only then, (and even ...Maybe), would I think about optimizing the structure of the business code.

I agree, I'm going through a very similar exercise with a program which collects data via modems at the moment.

The biggest problem I've had was caused by the fact that the program, when initially written in TP5.5, got its commands interactively (i.e. an internal command-line interface). When I converted it to Delphi I discovered that I could make it look the same by doing various things [irrelevant detail omitted] that no longer work in Lazarus, and that I had to go though a major exercise of separating stuff into threads (and then optimising the bottlenecks).

One approach would be to convert the TP program to compile with FPC, using Lazarus as an IDE but not giving it a GUI in the first iteration. That would at least get it to the point where it could target modern OSes, even if the menu system wasn't as pretty as it had been initially.

MarkMLl
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: marcov on December 12, 2019, 10:21:13 am
Last August we got the first prototype of our own main electronics print using ethernet. I still haven't done a project with them, the projects haven't been a match nor the time permitted, but first results are looking good.

The relevant microcontroller does have DMA though, so combined with the hardware socket chip (W5500) the load is really low.

Primarily the benefits are not (yet) so much increase of the primary datastream or latency, rather than improved diagnostic data, easier cabling (length and connecting multiples in star form). Also we had lots of problems getting reliable affordable (sub $50) serial cards in the last few years. Many wch based cards seem to fail if you use both ports at the same time.

The primary application is the fast paced, high tech world of beer bottle diameter measurement.
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: MarkMLl on December 12, 2019, 11:11:06 am
I'm having a bit of a crisis of confidence regarding certain types of modern electronics after the reports of the Tesla failures https://hackaday.com/2019/10/17/worn-out-emmc-chips-are-crippling-older-teslas/ (https://hackaday.com/2019/10/17/worn-out-emmc-chips-are-crippling-older-teslas/). Unfortunately I've not had time to look at FPC's support of AVR yet, but I would like to say that while Eclipse's Arduino stuff works it makes Lazarus look ****ing good: the startup time is around 65 seconds and there's so much functionality that it obscures the basic purpose.

The primary application is the fast paced, high tech world of beer bottle diameter measurement.

Don't tell me: you've got one of those bottling carousels built on a Caterpillar bulldozer bearing :-)

MarkMLl
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: otoien on December 12, 2019, 12:28:07 pm
Wow, lots of friendly responses while I was just out doing the evening cross country ski to exercise the dog I am looking after -  and more as I am typing! Thanks so much all of you! It is getting late here (time zone Alaska) so I will have to digest this further, lots of food for thought. Just to fill in some information:

The program has at a certain level fairly good  UI - business code separation, particularly for the part that performs the calculations and analysis. Other things like the graph screens for sampling  and viewing graphs ended up with some large ugly loops where things are pretty mixed. There are separate units that interact with the hardware though multiple layers.

As a side remark I do also work with pascal on the microcontroller firmware side (MikroPascal dsPIC) and I also  have been getting into electronic and mechanical CAD for hardware designs. I have been more than looking into Ethernet based communication on the hardware design side lately, but it will be a while until I have anything up and running including firmware, it is just one of the things I want to test out. I envision a future expansion into this area would be possible/needed  for my data acquisition application, although the present focus is interfacing with the commercial instruments that we use.

[Note that the funding I was mentioning above is only in that my current job description allows me to spend part of my own time on this, so my intention was not to hire anyone else. Besides that I would have needed to spend a lots of time explaining how the details of the code works to someone unfamiliar with it. Having been developed over so long time, both quality and consistency is quite variable, and my background is actually in physiology, not computer science.]

It is an interesting thought to try to compile some of the code under Lazarus/FreePascal as is. None of the user interface would likely work, as menus are based on code that directly addresses video memory, and I would be very surprised if the Halo graphics library would be able to be linked. But I have had in mind to start out with an application that are able to load the files and perform the calculations and save data back to the files, to check if that aspect of the code works. Except that I would like to simplify the data structure (but perhaps that should be delayed), I suspect some Unicode work will be needed related to unit processing and headings in the files.
 
Attached is a combination of screen shots I use to introduce the main parts of the application in my documentation. Data collected can be envisioned like a spreadsheet with variable #1 containing the time variable and the different variables organized as columns, so nothing really fancy here. There is a specialized set of byte/integer variables (here G1) that contains codes for calibration and gas flow switching status. The calibration screen shows how channels are scaled with units taken into account.

Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: MarkMLl on December 12, 2019, 12:38:52 pm
Nice to hear that there's at least one user in Alaska :-)

Halo with Turbo Pascal- now /that's/ a combination that I've not heard discussed for a while. One possibility would be to take an axe to the code and chop all of the graphics display out and use e.g. a spreadsheet for interim display, that would allow you to focus on the tricky logic first but you might find that that converts comparatively easily.

MarkMLl

p.s. For those of us disliking the long Winter evenings, from today Sunset starts getting a bit later. Anybody who doesn't believe me should check https://en.wikipedia.org/wiki/Equation_of_time (https://en.wikipedia.org/wiki/Equation_of_time)
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: marcov on December 12, 2019, 12:50:10 pm
As a side remark I do also work with pascal on the microcontroller firmware side (MikroPascal dsPIC) and I also  have been getting into electronic and mechanical CAD for hardware designs. I have been more than looking into Ethernet based communication on the hardware design side lately, but it will be a while until I have anything up and running including firmware, it is just one of the things I want to test out.

(actually that ethernet enabled microcontroller of mine is a dspic33E256MU810 (or /814 in some cases) with a Wiznet W5500 socket chip, and yes, that is quite a heavy trajectory for the uninitiated. I work in MP Lab X XC16 though)

 
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: marcov on December 12, 2019, 02:03:15 pm
I'm having a bit of a crisis of confidence regarding certain types of modern electronics after the reports of the Tesla failures https://hackaday.com/2019/10/17/worn-out-emmc-chips-are-crippling-older-teslas/ (https://hackaday.com/2019/10/17/worn-out-emmc-chips-are-crippling-older-teslas/).

Mwhaa, in fast growing industries, corners are frequently cut, intentionally or not. Anyway, it will be hard to duplicate Lockheed Martin's Mars Climate Orbiter fiasco.

Quote
Unfortunately I've not had time to look at FPC's support of AVR yet, but I would like to say that while Eclipse's Arduino stuff works it makes Lazarus look ****ing good: the startup time is around 35 seconds and there's so much functionality that it obscures the basic purpose.

Yup. But my electronics are, as said, dspic based, but that Java contraption is slow too (but Netbeans based)

The primary application is the fast paced, high tech world of beer bottle diameter measurement.

Don't tell me: you've got one of those bottling carousels built on a Caterpillar bulldozer bearing :-)

No, we work for bottle makers, not brewers. Though of course problems of brewers trickle down to us eventually.

We are actually designing a inspection carousel machine, but that is not this product. This simply measures a few dimensions of a bottle, mostly diameters using a camera and a rotating platform in about 10s and 0.02mm precision. It is used to monitor wear of bottle moulds during normal production and some other parameters that can affect dimensions while changing product.
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: sash on December 12, 2019, 04:56:05 pm
I didn't quite understand your whole situation, nor particular decisions.

But instead of (or additionally to) I'd consider not just "Objectification" but a "daemonization" too.
To architecturally separate functional and GUI processes.
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: PascalDragon on December 13, 2019, 09:54:31 am
It is an interesting thought to try to compile some of the code under Lazarus/FreePascal as is. None of the user interface would likely work, as menus are based on code that directly addresses video memory, and I would be very surprised if the Halo graphics library would be able to be linked. But I have had in mind to start out with an application that are able to load the files and perform the calculations and save data back to the files, to check if that aspect of the code works. Except that I would like to simplify the data structure (but perhaps that should be delayed), I suspect some Unicode work will be needed related to unit processing and headings in the files.
Well, you could try to use the 16-bit MS-DOS (https://wiki.freepascal.org/DOS) target as a first step as that is even a bit more compatible with TP than the 32 or 64 bit variants. You could then try to rework your code step by step.
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: otoien on December 13, 2019, 12:22:45 pm

Thanks all for further comments.
To summarize, it looks like the advice so far (as I interpret it) is to isolate the business code with as little changes as possible to begin with (very useful feedback as I had other thoughts on this), get that running (possibly using 16 bit mode to start with) and verify correctness of code in a bare bones environment. Then only slowly refactor it down the line to be more in tune Lazarus. 

In parallel (but likely not at the same time) approach the application from the other side as Juha and Marco suggest, building a simplified object oriented program structure that will perform the different aspects of the application so that I get an understanding and practice how the architecture will work in Lazarus.   Once "perfected" , combine it with the business code. I suspect a lot of the work rebuilding the structure and functionality of the acquisition part and graphing part will take place here, including  practicing multi-threading in the non-user interface code.

With respect to sash comment about "daemonization", I have used a couple of data acquisition programs that took the approach of an acquisition daemon (or server) that was started and stopped separately from the program. It seemed a little "funky" and not very user friendly compared to having the acquisition code integrated in the program. The latter approach should not prevent isolation of the part of the acquisition code that directly interacts with the hardware and run it in a different thread. Perhaps that is what is meant with that comment.
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: MarkMLl on December 13, 2019, 12:32:56 pm
Good summing up. One change that you're likely to see is that you can't cast an integer to a pointer any more, you'll find places where there's ambiguity between pointer and integers to use ptrint or ptruint which are "pointer sized integers".

Where you know that you have mandatory data structures (e.g. to match the format of data blocks arriving from an internal device) it's worth putting an explicit size check in the declaring unit's startup code and making sure that the program halts rather than relying on an assertion (which can be disabled).

MarkMLl
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: otoien on December 13, 2019, 01:02:10 pm
Good summing up. One change that you're likely to see is that you can't cast an integer to a pointer any more, you'll find places where there's ambiguity between pointer and integers to use ptrint or ptruint which are "pointer sized integers".

Where you know that you have mandatory data structures (e.g. to match the format of data blocks arriving from an internal device) it's worth putting an explicit size check in the declaring unit's startup code and making sure that the program halts rather than relying on an assertion (which can be disabled).

MarkMLl

Thanks - one advantage now of being a physiologist that created this code and not a computer scientist, was that I was taking a rather conservative approach and think I did not use too risky coding expressions although they would be more efficient. So I suspect these problems might not surface, but who knows... With respect to memory allocations, I think I was pretty consistent with using new and dispose, which would account for the size of the data structures and and not malloc and free. It is only when reading binary files that data structure size has to be taken into account, but I have already solved reading TP record structures in the parameter files containing 48 bit reals etc.

Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: MarkMLl on December 13, 2019, 01:17:49 pm
I'm /so/ glad that's small-C conservative (sorry, we had an election here yesterday).

When considering data structures etc. the things that will cause you trouble are changes in integer size and changes in alignment (i.e. the risk of padding inserted by the compiler to align 32-bit integers onto 32-bit addresses).

https://www.freepascal.org/docs-html/current/ref/refsu4.html#x26-250003.1.1

https://www.freepascal.org/docs-html/current/prog/progse32.html#x198-2030008.2

MarkMLl
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: otoien on December 14, 2019, 01:17:42 am

Thanks, I will keep my eyes out of these kind of problems.
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: guest64953 on December 14, 2019, 05:57:17 pm
First having it to build with FPC is a success. Keep this original codebase then fork it. Modernize the fork by rewriting it using Object Pascal (both Delphi mode or ObjFPC mode should be fine).
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: MarkMLl on December 14, 2019, 06:38:50 pm
Quote
Unfortunately I've not had time to look at FPC's support of AVR yet, but I would like to say that while Eclipse's Arduino stuff works it makes Lazarus look ****ing good: the startup time is around 35 seconds and there's so much functionality that it obscures the basic purpose.

Yup. But my electronics are, as said, dspic based, but that Java contraption is slow too (but Netbeans based)

Make that 65 seconds for Eclipse to fully initialise: is this really the best that the "mainstream developer community" can do?

I've had an interesting exchange in the Arduino forum about an itch I'd scratched and written up, and am told that the IDE developers are refactoring a whole lot of stuff to make it easier for an arbitrary IDE to call the compilers etc. https://github.com/arduino/arduino-cli (https://github.com/arduino/arduino-cli)

I've been eyeing up the deficiencies of the Arduino IDE for a long time, and it occurs to me that it's so easy to knock a UI together using Lazarus that a moderate amount of work could produce something vastly better than the opposition.

MarkMLl
Title: Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
Post by: otoien on December 14, 2019, 11:12:22 pm
Quote
Unfortunately I've not had time to look at FPC's support of AVR yet, but I would like to say that while Eclipse's Arduino stuff works it makes Lazarus look ****ing good: the startup time is around 35 seconds and there's so much functionality that it obscures the basic purpose.

Yup. But my electronics are, as said, dspic based, but that Java contraption is slow too (but Netbeans based)

Make that 65 seconds for Eclipse to fully initialise: is this really the best that the "mainstream developer community" can do?

I've had an interesting exchange in the Arduino forum about an itch I'd scratched and written up, and am told that the IDE developers are refactoring a whole lot of stuff to make it easier for an arbitrary IDE to call the compilers etc. https://github.com/arduino/arduino-cli (https://github.com/arduino/arduino-cli)

I've been eyeing up the deficiencies of the Arduino IDE for a long time, and it occurs to me that it's so easy to knock a UI together using Lazarus that a moderate amount of work could produce something vastly better than the opposition.

MarkMLl

The mikroPascal compiler/IDE I am using with dsPIC microcontrollers (all from dsPIC 30F4013 to dsPIC33EP512MU814) and PIC24 chips load in about 4 seconds, and has many nice features, particularly abstraction of chip pins, access to the chip definitions, and lots of libraries for different devices built in. There is also a separate visual development environment (VTFT). However bug fixing, especially in the IDE has been close to non-existent in the last few generations, so I am using an IDE a couple of versions back (which did not have some of those bugs) with the latest compiler. Yesterday an implied answer to the lack of bug fixing came: All the resources seems to have been used to create a new IDE (NECTO) based on QT5 and the Katy editor that will milk users for more money. The Pascal versions are several years into the future. How I wish the current compiler had been open source and written in/based on the Lazarus IDE!
TinyPortal © 2005-2018