Recent

Author Topic: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"  (Read 3755 times)

otoien

  • Jr. Member
  • **
  • Posts: 89
OK, let me say up front that I am aware that these kind of less concrete request are difficult to respond to in these fora vs. the ones that can deal directly with code, but I am seeking experiences from those who have made similar efforts in the past, and if there perhaps in hind sight was learning experiences that would have lead to a different strategy:

Introduction: I am in the early process of converting a data acquisition application written in TP 5.5 that I have developed and maintained over more than 30 years up to current days, and that I and my colleagues rely heavily upon for our research. Some work with respect to establishing new file format and translation of old parameter files has already been done (I chose the .ini file format, building upon the IniFiles unit, and expanded and specialized for my purpose. One of the big priorities is stable file formats that can be expanded upon and be able to provide backwards compatibility though import of old binary parameter files). Since then there has been a lapse in the efforts, which I am now funded to continue. I am concerned about maintaining stability, and integrity of this very well tested code with respect to correctness of calculations. The data file format for slow sampling is simple csv-tab-delimited which I see no reason to change except for UTF-8 compatible variable headings and units. The code is about 90,000 program lines so this is not a small task and is going to take a lot of time.

I wrote my own text based and graph based menu systems in TP5.5, so it is not based on any third party user interface libraries. The graphing part is based on a subset of the Turbo Halo library that provides basic graphic primitives (including an excellent world coordinate system that allows mapping to real coordinates of a plot). However code for graphing will have to be replaced, possibly using TAChart, unless I find some obstacles with respect to special features that necessitates translating my old code.

The program collects data and controls hardware (mostly though serial port these days, but also old support for a Labmaster board and parallel ports), will scale data, perform specialized calibrations and calculations in my field of expertise, and plots data in real time. In addition there are a number of routines for specialized imports and  data manipulation. A challenge is some overlap in the use of parameters between these functionalities.

A main point that has caused hesitation is to what degree to "objectify" (i. e create manageable classes) of the code, and at what level of detail. The current code is almost exclusively procedural, but in many cases passing a large number of parameters to the procedures which in some respect can be called an object oriented philosophy of providing an interface. However the parameters for scaling and calibrations, very extensive calculations, and graphing is stored in global dynamic records. I am pretty sure I will have to use multi threading for the data acquisition part. The main data are stored in complicated dynamically allocated array structures (to overcome 64K limitations of DOS memory), also global, that I think can be simplified by using a dynamic array with step wise allocation of new chunks of "cells" as before. (But how to access it in an object oriented way with so many parts of the program manipulating and using it?) 

My current idea is to wrap pretty large portions of code in classes according to functionality that contain its own parameters in each of these areas, and and that each class will also have the code to call the ini file code to save the parameters to a common ini-file (completely rewriting the inifile each time) that goes with the tab delimited text file.

However then I read an old comment by Marco Cantu (which I unfortunately cannot find at the moment as it is a while ago) where he recommended to keep as much code as possible in its original procedural form when converting a DOS program - in other words  not go very far into creating classes of it.

At this early stage I am concerned about not working myself into a corner, as I envision this application to be usable  and maintainable long into the future. So again, are there any experiences to take along on the way?

I have published a couple of papers relating to some functional aspects of the code if anyone get very interested:
Code: Text  [Select][+][-]
  1. Tøien, Ø. (1992) Data acquisition in thermal physiology: measurements of shivering. J. Therm. Biol. 17:357-366.
  2. Tøien, Ø. (2013) Automated open flow respirometry in continuous and long-term measurements: design and principles. J. Appl. Physiol. 114:1094-1107
« Last Edit: December 13, 2019, 12:30:04 am by otoien »
Unless otherwise noted I always use the latest stable version of Lasarus/FPC x86_64-win64-win32/win64

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #1 on: December 12, 2019, 08:38:54 am »
I would start to design a "perfect" object structure with properties and methods to interact with it.
Actual code should be ported only after it is ready.
It requires lots of changes. Many people would prefer only minimal necessary changes to the procedural code when the file format or libraries etc. so require.
One choice is to port in 2 steps. First minimal changes to get a working implementation. Then "objectify" into well designed classes.
« Last Edit: December 12, 2019, 11:28:09 am by JuhaManninen »
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #2 on: December 12, 2019, 08:57:39 am »
I would start working in the existing app to get some form of GUI - business code separation.

Then I would try to attach that business code to a new lazarus/LCL gui with TChart and the rest.

Only then, (and even ...Maybe), would I think about optimizing the structure of the business code.

PaulRowntree

  • Full Member
  • ***
  • Posts: 132
    • Paul Rowntree
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #3 on: December 12, 2019, 09:23:30 am »
In my opinion, the biggest challenge is rethinking the front-end UI in terms of what is easily done in Laz, and rethinking of the experimental structure.  What seemed natural using units to divide and conquer may be completely different from a functional object based approach.  Describing your process to a programmer that has no knowledge of the TP5.5 code might be a good step to help re-imagine the structure.
We recently did complete rewrites of 3 major lab codes, moving from Delphi, LabView to Lazarus/FP.  Lots of serial ports (I used LazSerial).  On a previous project (Delphi) I was trying way too hard to be clever with binary and Real48 data structures on disk ... what a mistake!  Now it is all ASCII for me.  Today's machines are so fast, and memory is so cheap, that squeezing wasn't worth the effort.  Our new data approaches, all objectified, are almost future proof because the files are self-documenting, expandable, and text based.  We have adapted the JCAMP-file exchange format to be the primary data storage, and it is wprking well.  We move and load hundreds of 100KB spectra in a few seconds, reprocess data in a blink. I am glad we did it, but it was a long struggle, and the students had to put up with daily new versions for a while.  Part of this was due to programming at home away from the hardware ... makes for a long debug cycle.
I'd be happy to help if you could give an overview of how much data is in memory, acquisition rates, etc.  Most important is how you see the data ... X-Y arrays of data?  Collection of stim/response experiments?What does the experiment look like?
gracie dot rowntree at gmail dot com
Cheers!
« Last Edit: December 12, 2019, 12:11:52 pm by PaulRowntree »
Paul Rowntree
- coding for instrument control, data acquisition & analysis, CNC systems

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #4 on: December 12, 2019, 09:36:56 am »
I would start working in the existing app to get some form of GUI - business code separation.

Then I would try to attach that business code to a new lazarus/LCL gui with TChart and the rest.

Only then, (and even ...Maybe), would I think about optimizing the structure of the business code.

I agree, I'm going through a very similar exercise with a program which collects data via modems at the moment.

The biggest problem I've had was caused by the fact that the program, when initially written in TP5.5, got its commands interactively (i.e. an internal command-line interface). When I converted it to Delphi I discovered that I could make it look the same by doing various things [irrelevant detail omitted] that no longer work in Lazarus, and that I had to go though a major exercise of separating stuff into threads (and then optimising the bottlenecks).

One approach would be to convert the TP program to compile with FPC, using Lazarus as an IDE but not giving it a GUI in the first iteration. That would at least get it to the point where it could target modern OSes, even if the menu system wasn't as pretty as it had been initially.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #5 on: December 12, 2019, 10:21:13 am »
Last August we got the first prototype of our own main electronics print using ethernet. I still haven't done a project with them, the projects haven't been a match nor the time permitted, but first results are looking good.

The relevant microcontroller does have DMA though, so combined with the hardware socket chip (W5500) the load is really low.

Primarily the benefits are not (yet) so much increase of the primary datastream or latency, rather than improved diagnostic data, easier cabling (length and connecting multiples in star form). Also we had lots of problems getting reliable affordable (sub $50) serial cards in the last few years. Many wch based cards seem to fail if you use both ports at the same time.

The primary application is the fast paced, high tech world of beer bottle diameter measurement.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #6 on: December 12, 2019, 11:11:06 am »
I'm having a bit of a crisis of confidence regarding certain types of modern electronics after the reports of the Tesla failures https://hackaday.com/2019/10/17/worn-out-emmc-chips-are-crippling-older-teslas/. Unfortunately I've not had time to look at FPC's support of AVR yet, but I would like to say that while Eclipse's Arduino stuff works it makes Lazarus look ****ing good: the startup time is around 65 seconds and there's so much functionality that it obscures the basic purpose.

The primary application is the fast paced, high tech world of beer bottle diameter measurement.

Don't tell me: you've got one of those bottling carousels built on a Caterpillar bulldozer bearing :-)

MarkMLl
« Last Edit: December 14, 2019, 06:26:11 pm by MarkMLl »
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

otoien

  • Jr. Member
  • **
  • Posts: 89
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #7 on: December 12, 2019, 12:28:07 pm »
Wow, lots of friendly responses while I was just out doing the evening cross country ski to exercise the dog I am looking after -  and more as I am typing! Thanks so much all of you! It is getting late here (time zone Alaska) so I will have to digest this further, lots of food for thought. Just to fill in some information:

The program has at a certain level fairly good  UI - business code separation, particularly for the part that performs the calculations and analysis. Other things like the graph screens for sampling  and viewing graphs ended up with some large ugly loops where things are pretty mixed. There are separate units that interact with the hardware though multiple layers.

As a side remark I do also work with pascal on the microcontroller firmware side (MikroPascal dsPIC) and I also  have been getting into electronic and mechanical CAD for hardware designs. I have been more than looking into Ethernet based communication on the hardware design side lately, but it will be a while until I have anything up and running including firmware, it is just one of the things I want to test out. I envision a future expansion into this area would be possible/needed  for my data acquisition application, although the present focus is interfacing with the commercial instruments that we use.

[Note that the funding I was mentioning above is only in that my current job description allows me to spend part of my own time on this, so my intention was not to hire anyone else. Besides that I would have needed to spend a lots of time explaining how the details of the code works to someone unfamiliar with it. Having been developed over so long time, both quality and consistency is quite variable, and my background is actually in physiology, not computer science.]

It is an interesting thought to try to compile some of the code under Lazarus/FreePascal as is. None of the user interface would likely work, as menus are based on code that directly addresses video memory, and I would be very surprised if the Halo graphics library would be able to be linked. But I have had in mind to start out with an application that are able to load the files and perform the calculations and save data back to the files, to check if that aspect of the code works. Except that I would like to simplify the data structure (but perhaps that should be delayed), I suspect some Unicode work will be needed related to unit processing and headings in the files.
 
Attached is a combination of screen shots I use to introduce the main parts of the application in my documentation. Data collected can be envisioned like a spreadsheet with variable #1 containing the time variable and the different variables organized as columns, so nothing really fancy here. There is a specialized set of byte/integer variables (here G1) that contains codes for calibration and gas flow switching status. The calibration screen shows how channels are scaled with units taken into account.

Unless otherwise noted I always use the latest stable version of Lasarus/FPC x86_64-win64-win32/win64

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #8 on: December 12, 2019, 12:38:52 pm »
Nice to hear that there's at least one user in Alaska :-)

Halo with Turbo Pascal- now /that's/ a combination that I've not heard discussed for a while. One possibility would be to take an axe to the code and chop all of the graphics display out and use e.g. a spreadsheet for interim display, that would allow you to focus on the tricky logic first but you might find that that converts comparatively easily.

MarkMLl

p.s. For those of us disliking the long Winter evenings, from today Sunset starts getting a bit later. Anybody who doesn't believe me should check https://en.wikipedia.org/wiki/Equation_of_time
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #9 on: December 12, 2019, 12:50:10 pm »
As a side remark I do also work with pascal on the microcontroller firmware side (MikroPascal dsPIC) and I also  have been getting into electronic and mechanical CAD for hardware designs. I have been more than looking into Ethernet based communication on the hardware design side lately, but it will be a while until I have anything up and running including firmware, it is just one of the things I want to test out.

(actually that ethernet enabled microcontroller of mine is a dspic33E256MU810 (or /814 in some cases) with a Wiznet W5500 socket chip, and yes, that is quite a heavy trajectory for the uninitiated. I work in MP Lab X XC16 though)

 

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #10 on: December 12, 2019, 02:03:15 pm »
I'm having a bit of a crisis of confidence regarding certain types of modern electronics after the reports of the Tesla failures https://hackaday.com/2019/10/17/worn-out-emmc-chips-are-crippling-older-teslas/.

Mwhaa, in fast growing industries, corners are frequently cut, intentionally or not. Anyway, it will be hard to duplicate Lockheed Martin's Mars Climate Orbiter fiasco.

Quote
Unfortunately I've not had time to look at FPC's support of AVR yet, but I would like to say that while Eclipse's Arduino stuff works it makes Lazarus look ****ing good: the startup time is around 35 seconds and there's so much functionality that it obscures the basic purpose.

Yup. But my electronics are, as said, dspic based, but that Java contraption is slow too (but Netbeans based)

The primary application is the fast paced, high tech world of beer bottle diameter measurement.

Don't tell me: you've got one of those bottling carousels built on a Caterpillar bulldozer bearing :-)

No, we work for bottle makers, not brewers. Though of course problems of brewers trickle down to us eventually.

We are actually designing a inspection carousel machine, but that is not this product. This simply measures a few dimensions of a bottle, mostly diameters using a camera and a rotating platform in about 10s and 0.02mm precision. It is used to monitor wear of bottle moulds during normal production and some other parameters that can affect dimensions while changing product.

sash

  • Sr. Member
  • ****
  • Posts: 366
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #11 on: December 12, 2019, 04:56:05 pm »
I didn't quite understand your whole situation, nor particular decisions.

But instead of (or additionally to) I'd consider not just "Objectification" but a "daemonization" too.
To architecturally separate functional and GUI processes.
Lazarus 2.0.10 FPC 3.2.0 x86_64-linux-gtk2 @ Ubuntu 20.04 XFCE

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #12 on: December 13, 2019, 09:54:31 am »
It is an interesting thought to try to compile some of the code under Lazarus/FreePascal as is. None of the user interface would likely work, as menus are based on code that directly addresses video memory, and I would be very surprised if the Halo graphics library would be able to be linked. But I have had in mind to start out with an application that are able to load the files and perform the calculations and save data back to the files, to check if that aspect of the code works. Except that I would like to simplify the data structure (but perhaps that should be delayed), I suspect some Unicode work will be needed related to unit processing and headings in the files.
Well, you could try to use the 16-bit MS-DOS target as a first step as that is even a bit more compatible with TP than the 32 or 64 bit variants. You could then try to rework your code step by step.

otoien

  • Jr. Member
  • **
  • Posts: 89
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #13 on: December 13, 2019, 12:22:45 pm »

Thanks all for further comments.
To summarize, it looks like the advice so far (as I interpret it) is to isolate the business code with as little changes as possible to begin with (very useful feedback as I had other thoughts on this), get that running (possibly using 16 bit mode to start with) and verify correctness of code in a bare bones environment. Then only slowly refactor it down the line to be more in tune Lazarus. 

In parallel (but likely not at the same time) approach the application from the other side as Juha and Marco suggest, building a simplified object oriented program structure that will perform the different aspects of the application so that I get an understanding and practice how the architecture will work in Lazarus.   Once "perfected" , combine it with the business code. I suspect a lot of the work rebuilding the structure and functionality of the acquisition part and graphing part will take place here, including  practicing multi-threading in the non-user interface code.

With respect to sash comment about "daemonization", I have used a couple of data acquisition programs that took the approach of an acquisition daemon (or server) that was started and stopped separately from the program. It seemed a little "funky" and not very user friendly compared to having the acquisition code integrated in the program. The latter approach should not prevent isolation of the part of the acquisition code that directly interacts with the hardware and run it in a different thread. Perhaps that is what is meant with that comment.
Unless otherwise noted I always use the latest stable version of Lasarus/FPC x86_64-win64-win32/win64

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Strategy to convert a TP 5.5 application to Lazarus - "Objectification"
« Reply #14 on: December 13, 2019, 12:32:56 pm »
Good summing up. One change that you're likely to see is that you can't cast an integer to a pointer any more, you'll find places where there's ambiguity between pointer and integers to use ptrint or ptruint which are "pointer sized integers".

Where you know that you have mandatory data structures (e.g. to match the format of data blocks arriving from an internal device) it's worth putting an explicit size check in the declaring unit's startup code and making sure that the program halts rather than relying on an assertion (which can be disabled).

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

 

TinyPortal © 2005-2018