Lazarus

Free Pascal => Beginners => Topic started by: Wysardry on August 11, 2022, 08:51:15 am

Title: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 11, 2022, 08:51:15 am
I'm looking to create a domain specific language within standard FreePascal code (rather than modifying the FreePascal compiler).

I'm not 100% certain of the terminology, so I don't know what to search for in the documentation.

Is it possible to replace keywords such as "writeLn" with "writeLine" or "say" or whatever?

How about operators such as ":=" being replaced with "becomes" or "isNow"?
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 11, 2022, 09:00:00 am
Is it possible to replace keywords such as "writeLn" with "writeLine" or "say" or whatever?

WriteLn is not a keyword, it's an intrinsic, a compiler provided function. That means, no, you can't replace it with something else without modifying the compiler (*).

How about operators such as ":=" being replaced with "becomes" or "isNow"?

Again, no, this is not possible without modifying the compiler (*).

* Well, technically you could define macros (e.g. {$define WriteLine := WriteLn} and {$define becomes := :=}), but this might lead to other problems down the road (e.g. if you need to use a preexisting function or type that contains that identifier) and there might be situations where using a macro might not be enough.

Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 11, 2022, 09:24:39 am
Would creating a procedure or function called "writeLine" that contained "writeLn" work?
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 11, 2022, 09:58:32 am
I'm looking to create a domain specific language within standard FreePascal code (rather than modifying the FreePascal compiler).

I'm not 100% certain of the terminology, so I don't know what to search for in the documentation.

Is it possible to replace keywords such as "writeLn" with "writeLine" or "say" or whatever?

How about operators such as ":=" being replaced with "becomes" or "isNow"?

Don't follow that approach. Define the DSL syntax and behaviour you want, then learn how to parse and compile/interpret it.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Thaddy on August 11, 2022, 12:44:44 pm
Quote
WriteLn is not a keyword, it's an intrinsic, a compiler provided function. That means, no, you can't replace it with something else without modifying the compiler (*).
Yes that is perfectly possible to some extend and just like Sarah wrote it does have its limitations. Just for fun:
Code: Pascal  [Select][+][-]
  1. {$macro on}{$define say :=writeln}
  2. program writewhatever;
  3. begin
  4. say('Whatever');
  5. end.
  6.  
We had a lot of fun with such macro's a couple of years ago.. I did a Dutch Pascal, someone else did a German Pascal etc.
See here : https://forum.lazarus.freepascal.org/index.php/topic,36887.msg246289.html#msg246289
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 11, 2022, 01:41:15 pm
Would creating a procedure or function called "writeLine" that contained "writeLn" work?

You can't replicate the functionality of WriteLn one to one. You can replicate single aspects, but not the whole.

Don't follow that approach. Define the DSL syntax and behaviour you want, then learn how to parse and compile/interpret it.

That would be my suggestion as well.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: simone on August 11, 2022, 01:44:24 pm
I don't know exactly what you want to do. However I have implemented a sort of DSL building a transpiler which translates to freepascal. The transpiler itself has been written in freepascal. In this context, the fact that FPC supports a lot of targets is a great advantage in terms of portability.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 11, 2022, 07:30:34 pm
My main goal is to create a simple and easy to use language where the code can be written using Windows, Linux, Raspberry Pi etc. and then compiled for a wider range of platforms including 8-bit retro machines and WebAssembly.

The created programs would be text only, so the graphics capabilities of the target machines would not be an issue.

I don't know of any Pascal compilers that can target multiple retro machines, but I have found a couple of C compilers that should work.

As I personally dislike C syntax and it isn't the easiest language for beginners (who would hopefully eventually use this language) I would need some sort of conversion option.

Pascal is already quite close to what I'm aiming for. It would just need a few minor tweaks to become more accessible to beginners, those using screen readers or anyone that is used to BASIC.

To create an entirely new language from scratch seems like it would be a lot of wasted effort when other established languages are available.

A slightly easier option would be to modify an existing language and compiler. That would also require modifying/creating a transpiler targetting C though.

Even easier would be to create an internal or embedded language, which is an option I have only recently discovered is possible.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 11, 2022, 08:09:30 pm
I don't know of any Pascal compilers that can target multiple retro machines,

https://wiki.freepascal.org/Category:Operating_Systems_and_Platforms

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 11, 2022, 08:31:48 pm
Unless I'm missing something, there are only two retro machines supported: ZX Spectrum and MSX.

z88dk (https://z88dk.org/site/) can compile C code for over a hundred Z80 machines (https://github.com/z88dk/z88dk/wiki/Platform).

I'm not personally interested in compiling for all those platforms, but the ones I am interested in are in that list.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Thaddy on August 11, 2022, 08:54:16 pm
There are many more supported, including Wii which is great fun.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 11, 2022, 09:23:34 pm
In the platform list (https://wiki.freepascal.org/Platform_list) there are just two entries in the classic home computers column.

By "retro machines" I meant 8-bit home computers, such as the Amstrad CPC series, Sinclair Spectrum, Enterprise, Sharp MZ series, MSX etc.

Most of the ones I'm especially interested in had a Z80 processor, but I would like to include other 8-bit machines (such as the Dragon 32 or BBC B) if possible.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Edson on August 11, 2022, 09:29:52 pm
I've developed a compiler/IDE for the 6502 CPU: https://github.com/t-edson/P65Pas. Still in Beta by now.

Support for Commodore 64 and Working in C128. I'll be including support for more 8-bits platforms later.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Handoko on August 12, 2022, 04:46:36 am
How about TurboRascal?

Quote
... one of whom put together a new compiler which targets retro platforms – and it goes by the name Turbo Rascal.

The list of supported platforms is extensive, with Turbo Rascal able to compile highly-optimized binaries for the C64, Amiga 500, BBC Micro, IBM PC, Atari ST, Game Boy, Amstrad, NES, ZX Spectrum, and more.
Source: https://hackaday.com/2021/11/30/turbo-rascal-is-the-retro-pascal-compiler-we-always-wanted/ (https://hackaday.com/2021/11/30/turbo-rascal-is-the-retro-pascal-compiler-we-always-wanted/)

Quote
(Turbo Rascal SE, TRSE) is a complete suite (IDE, compiler, programming language, image sprite level resource editor) intended for developing games/demos for 8 / 16-bit line of computers, with a focus on the MOS 6502, the Motorola 68000, the (GB)Z80 and the X86. TRSE currently supports application development for the C64, C128, VIC-20, PLUS4, NES, Gameboy, PET, ZX Spectrum, TIKI 100, Amstrad CPC 464, Atari 2600, 8086AT, Amiga 500, Atari 800, BBC Micro, Mega65, MSX, Apple II and the Atari ST 520.
Source: https://retrogamecoders.com/introduction-to-trse-programming/
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 12, 2022, 03:41:36 pm
A slightly easier option would be to modify an existing language and compiler. That would also require modifying/creating a transpiler targetting C though.

Then FPC wouldn't be a help for you anyway, cause FPC does not transpile to C. And changing what language FPC supports wouldn't change what targets it supports.

Unless I'm missing something, there are only two retro machines supported: ZX Spectrum and MSX.

z88dk (https://z88dk.org/site/) can compile C code for over a hundred Z80 machines (https://github.com/z88dk/z88dk/wiki/Platform).

I'm not personally interested in compiling for all those platforms, but the ones I am interested in are in that list.

You could just as well port FPC for those platforms. After all I've done the same for MSX...
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 12, 2022, 06:26:37 pm
How about TurboRascal?
I bookmarked that some time ago, but it isn't yet complete enough for my needs.

Unless the support sheet (https://lemonspawn.com/support-sheet/) is out of date, it doesn't yet support the platforms I need and probably won't until most of those on that list are 100% complete.

The source is available, but it's mostly C and C++ unfortunately.

Then FPC wouldn't be a help for you anyway, cause FPC does not transpile to C. And changing what language FPC supports wouldn't change what targets it supports.
None of the third party Pascal to C translators will work with FreePascal?

I haven't looked at any of them closely, as I wasn't sure FreePascal would be suitable for this project.

Quote
You could just as well port FPC for those platforms. After all I've done the same for MSX...
Do you mean create a cross-compiler that runs on modern machines and compiles for the retro machines, or create a native FPC compiler for each retro machine?

Either way, it sounds like it would be complicated.

I did notice that there's a Z80 cross-compiler already in the dev branch, but again I haven't looked into that option as I'm not sure FreePascal meets my other needs.

HiSoft Pascal was available for some of the machines I'm interested in, but getting hold of it legally for them would likely be difficult.

The only other option I have found is to use a language and compiler that already produces C code that can be compiled with SDCC or z88dk.

So far, I've seen Nim and two versions of BASIC that might work. I've heard the former has been used with 8-bit micro boards and 512 bytes of RAM, so that seems the most promising.

There's also The ZX BASIC compiler (https://www.boriel.com/pages/the-zx-basic-compiler.html) written in Python, which targets a wide range of 8-bit platforms, but no modern ones as far as I can tell.

It should be possible to modify the existing BASIC language used (to make it a DSL), but that could mean having to modify a whole bunch of compilation scripts too.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 12, 2022, 06:55:57 pm
(a) Don't write off an FPC port when- by your own admission- you have no understanding of how it's structured (b) you'll find that converting Pascal to C will result in comparatively inefficient and potentially unreliable ** code and (c) I've already told you that your best approach is probably learning how to write a DSL yourself... with care it might be usable with both FPC and TurboRascal (which I admit I'd forgotten about and haven't looked at).

** On account of C's tendency to use the heap heavily, which is unreasonable on a small system.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 12, 2022, 07:39:58 pm
I'm not ruling anything out completely at this point, as I'm not sure what my options are as a relative beginner. That is why I asked for clarification on porting FPC. For all I know, there could be existing code that just needs tweaking.

C might be an inefficient option, but AFAIK the only other simple option is to use whatever version of BASIC is available on each target platform. I would imagine that a C compiler aimed at small devices is going to be reasonably efficient in comparison.

Writing a DSL myself is one of the options I'm looking at, but I'm not sure what existing language to use to write it in, given the target machines I have.

I do know that I want to avoid anything with a C-style syntax, as all those curly braces would sap my will to live before I was halfway done. I'd almost rather work with assembly language.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 12, 2022, 08:09:29 pm
C might be an inefficient option, but AFAIK the only other simple option is to use whatever version of BASIC is available on each target platform. I would imagine that a C compiler aimed at small devices is going to be reasonably efficient in comparison.

Why, and do you mean speed-efficient or space-efficient?

Quote
I do know that I want to avoid anything with a C-style syntax, as all those curly braces would sap my will to live before I was halfway done. I'd almost rather work with assembly language.

ALGOL-derived languages are all much the same: from an implementation POV "begin", "end" and the braces are just tokens (sometimes recognised by a distinct lexer) which are parsed much the same.

And I suggest giving some thought to Alan Kay's comment on Meta-2.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 12, 2022, 08:46:08 pm
Why, and do you mean speed-efficient or space-efficient?
It has supposedly been written with small devices in mind, so should be optimised for such.

As I mentioned previously, Nim has been used to create programs for 8-bit micro boards with 512 bytes of RAM via a C compiler (I don't know which one, but likely SDCC or z88dk).

It should be more efficient in terms of memory and speed, as BASIC was usually interpreted and memory resident on 8-bit machines.
Quote
And I suggest giving some thought to Alan Kay's comment on Meta-2.
I'm afraid you've lost me there.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 12, 2022, 10:57:30 pm
It should be more efficient in terms of memory and speed, as BASIC was usually interpreted and memory resident on 8-bit machines.

Not a safe assumption. Python, Perl and so on are to a large extent interpreted.

Quote
Quote
And I suggest giving some thought to Alan Kay's comment on Meta-2.
I'm afraid you've lost me there.

You can use Google can't you?

Quote
META II is the first documented version of a metacompiler, as it compiles to machine code for one of the earliest instances of a virtual machine.

The paper itself is a wonderful gem which includes a number of excellent examples, including the bootstrapping of Meta II in itself (all this was done on an 8K (six bit byte) 1401!)."—Alan Kay

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: sketch on August 13, 2022, 12:35:47 am
I do know that I want to avoid anything with a C-style syntax, as all those curly braces would sap my will to live before I was halfway done.
Use Racket https://www.racket-lang.org/ (https://www.racket-lang.org/), it avoids the C-style syntax and you don't need to worry about all those curly braces  ;D
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 13, 2022, 07:11:01 pm
Then FPC wouldn't be a help for you anyway, cause FPC does not transpile to C. And changing what language FPC supports wouldn't change what targets it supports.
None of the third party Pascal to C translators will work with FreePascal?

They might work with normal Pascal code, but if you adjust the language to your needs do you really think you wouldn't need to adjust these either?

Quote
You could just as well port FPC for those platforms. After all I've done the same for MSX...
Do you mean create a cross-compiler that runs on modern machines and compiles for the retro machines, or create a native FPC compiler for each retro machine?

Either way, it sounds like it would be complicated.

I did notice that there's a Z80 cross-compiler already in the dev branch, but again I haven't looked into that option as I'm not sure FreePascal meets my other needs.

I mean as a cross-compiler. These targets are too restrained to support FPC itself.

Porting FPC is not even that hard as long as the hardware platform is already supported (in this case Z80 which you noticed is supported in the development version), but you need to know your target system or learn it. The main part is implementing the necessary platform specific RTL functions. For MSX-DOS it essentially boiled down to these commits:

Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 14, 2022, 07:04:32 am
I haven't learned anything about assembly language, but wouldn't object to learning enough about each Z80 system to create new targets based on the existing ones.

Some of the targets I would like to support are quite obscure, but I should be able to find the necessary info.

I could possibly create something similar to what I'm aiming for by building a domain specific library or framework, if I could add enough new Z80 targets.

Would PascalScript be an option if I wanted to be able to test programs without compiling?
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 14, 2022, 10:28:46 am
I haven't learned anything about assembly language, but wouldn't object to learning enough about each Z80 system to create new targets based on the existing ones.

Some of the targets I would like to support are quite obscure, but I should be able to find the necessary info.

If you can't find the info you're screwed in every case. In FPC terms, the compiler generated Z80 code and you might need to adapt the RTL slightly for different targets. However I suggest you examine what early versions of Turbo Pascal did to customise themselves, even if in general they only targeted different variants of CP/M.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 14, 2022, 01:39:52 pm
Some of the targets I would like to support are quite obscure, but I should be able to find the necessary info.

It's not as if FPC doesn't already support obscure targets. ;)

Would PascalScript be an option if I wanted to be able to test programs without compiling?

How do you believe this would help you? If you just want to test programs you can use FPC as well. And for other targets you'd need to port PascalScript first which is rather heavy weight.

I haven't learned anything about assembly language, but wouldn't object to learning enough about each Z80 system to create new targets based on the existing ones.

Some of the targets I would like to support are quite obscure, but I should be able to find the necessary info.

If you can't find the info you're screwed in every case. In FPC terms, the compiler generated Z80 code and you might need to adapt the RTL slightly for different targets. However I suggest you examine what early versions of Turbo Pascal did to customise themselves, even if in general they only targeted different variants of CP/M.

FPC's RTL is already well abstracted so that platform specific code can be easily inserted. It might be a bit more heavy weight than what TP did in a more direct way (after all we want to keep this maintainable), but so far it worked great. :)
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 14, 2022, 02:18:20 pm
FPC's RTL is already well abstracted so that platform specific code can be easily inserted. It might be a bit more heavy weight than what TP did in a more direct way (after all we want to keep this maintainable), but so far it worked great. :)

Yes, but my intended point was that with a minimum of research OP would discover how many variants of early TP versions there were (hint: more than a handful, fewer than a dozen), and the extent to which each variant could be customised for a particular target during installation. And from that he might possibly have the wit to work out /why/ the variants differed, even if he has so far been resistant to looking at the FPC porting documentation and to following our suggestion of learning a bit about compiler writing.

I used stuff to which I have alluded in this thread- and I have no intention of repeating myself- to compile an ALGOL-derivative (which could just as easily have been a Pascal-derivative) to assembler source for various 8- and 16-bit processors (I admit to having been helped by having assemblers available which took a common directive syntax). This was poor on type checking etc., and poor on optimisation... but there is at least one promising derivative which can improve the latter.

But this isn't my project or problem, and while- like any member of the community- I'd do my best to help if he were porting FPC or the RTL I've seen (and admittedly sold or supported) too many dubious development solutions to have much enthusiasm for half-arsed discussion of transpilers via C etc.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 14, 2022, 07:43:22 pm
Okay, a lot of assumptions are being made here, probably because I didn't think it would be helpful to go into a lot of detail about what I have researched so far.

My end goal is to create a system that runs on modern hardware that can produce text-based programs that run on modern and retro hardware.

I want to find the simplest, easiest and least time-consuming route to do that, as I want to live long enough to be able to use that system myself and make it available for others to use.

On and off I have spent the last two years looking into ways this could be done and I'm now suffering from information overload and analysis paralysis. Many others have worked on different parts of the process, but I haven't found a complete system.

The main target audience for this system would be those who own retro 8-bit home computers or use emulators for them (possibly because they no longer own such machines).

Interpreters were common for these devices and compilers were much less so. Including one with the front end of the system seems natural, especially if the process for translating and compiling the code has several stages.

BASIC would have been the most commonly used language on these retro machines, as it was inluded with most of them. Pascal and C compilers were available, but much less widely used.

HiSoft sold versions of BASIC, Pascal and C, for example.

If you owned a Z80 machine with a disk drive (rare in the UK), you could use more languages via CP/M. I'm reasonably sure that this was how Turbo Pascal was made available for 8-bit micros.

The upshot of this is, that more users would know BASIC than any other language, so the system should use something similar. Pascal would be close enough for those used to structured BASIC.

Languages with a C-style syntax would be harder to adjust to. I know it caused me issues when I tried learning it.

So, one end of the process should be a BASIC or Pascal style language and at the other should be 8-bit machine code for various systems.

I know it's possible to create a programming language using FreePascal as there are several in use (Decimal BASIC and Nim are examples), but I'm not sure if creating another would be the most efficient way. Maybe modifying an existing one would be better?

I also know it's possible to create a transpiler (or source to source compiler) using FreePascal as there's one available to convert Decimal BASIC to FreePascal and several mentioned in the Wiki. It's also possible that the Nim to C translation program was written in FreePascal before Nim became self-compiling.

I've found several compilers that target the machines I'm most interested in, plus a site on Z80 assembly programming (https://www.assemblytutorial.com/z80/).

I have at least a hundred sites bookmarked that could potentially be pieces in the puzzle I'm trying to solve, but I'm having problems deciding which ones to use and how to fit them together.

I need help in deciding what to actually focus on going forward. I do know that FreePascal should be involved in the process somewhere, even if it is just as a source to source compiler target.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: y.ivanov on August 14, 2022, 08:40:43 pm
Okay, a lot of assumptions are being made here, probably because I didn't think it would be helpful to go into a lot of detail about what I have researched so far.

My end goal is to create a system that runs on modern hardware that can produce text-based programs that run on modern and retro hardware.

I want to find the simplest, easiest and least time-consuming route to do that, as I want to live long enough to be able to use that system myself and make it available for others to use.
*snip*

And you want it to be BASIC- or Pascal-like one?
IMHO it depends what features you want to include into. Text-mode is one mentioned so far.

For example - in Arduino (https://www.arduino.cc/en/software) a subset of C language is described and the script is cleverly included in a pre-made gcc project. As a result the user thinks he wrote an INO script, but actually there is a full-blown gcc toolchain at the back. That approach is solely IDE oriented.

If something simple is required, it can be written from scratch, in Wirth's book (https://forum.lazarus.freepascal.org/index.php/topic,53821.0.html) a small pascal-like language implementation is described.
 
AFAIK the original sources for the Pascal compiler and P-code interpreter are available in the Internet but I didn't kept the link.

It is not such a big deal to implement (a simple) language since tools for that are freely available, see https://wiki.freepascal.org/Plex_and_Pyacc. Porting FPC to a new target looks to me much more complicated, its getting more and more sophisticated recently (but I can be wrong here).
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 14, 2022, 09:38:07 pm
Yes, I'm leaning towards something similar to BASIC or Pascal syntax.

Something along the lines of ANSI Full BASIC without graphics or sound libraries but possibly with some additional Pascal features such as records or enumeration. It depends on the difficulty in including those features on many platforms.

Even ignoring that many retro machine users will be more familiar with BASIC, I personally find it easier to keep track of where code blocks begin and end when most block types have a different keyword associated with them. i.e. "for" loops end with "next [var]" and "while" loops end with "wend".

I'm guessing that what is easier for me to keep track of is easier for a computer to parse. That is just a guess though.

Edit: At this stage, I also don't want to rule out the possibility of translating to any 8-bit BASIC compilers.

I did find an interesting project called The Amsterdam Compiler Kit (http://tack.sourceforge.net/) that could be useful. It doesn't seem that active, but most of the computers I would be targetting won't have changed much in the last few decades anyway.

I did notice Plex and Pyacc in the Wiki, but didn't explore them too deeply as I'm still undecided on what path to follow.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 14, 2022, 10:26:35 pm
This is likely to be moved to "Other" by a moderator.

Sorry if I was a bit brutal earlier, I was intentionally goading you.

Quoting selectively:

My end goal is to create a system that runs on modern hardware that can produce text-based programs that run on modern and retro hardware. ...

The main target audience for this system would be those who own retro 8-bit home computers or use emulators for them (possibly because they no longer own such machines).

Interpreters were common for these devices and compilers were much less so. Including one with the front end of the system seems natural, especially if the process for translating and compiling the code has several stages.

BASIC would have been the most commonly used language on these retro machines, as it was inluded with most of them. Pascal and C compilers were available, but much less widely used.

HiSoft sold versions of BASIC, Pascal and C, for example.

If you owned a Z80 machine with a disk drive (rare in the UK), you could use more languages via CP/M. I'm reasonably sure that this was how Turbo Pascal was made available for 8-bit micros.

I think that a sufficient proportion of retro users cut their teeth on BASIC machines such as the Spectrum and BBC Micro that selling any other language would be quite a job /unless/ it was as well integrated as the original Turbo Pascal. And for similar reasons that makes using a PC-hosted cross-compiler unrealistic, /unless/ you produced an entire product range of physical and emulated micros with some sort of "tube" arrangement to the host for program download (precedent: https://hackaday.io/Just4Fun for a recently-designed range of boards from the same stable, plus the Arduino IDE (which I'm sure you're aware supports multiple targets) for a way of developing using a slightly-cut-down C++ and downloading over a USB serial link).

I was, for a long time, an outspoken critic of BASIC and C. But the truth is that both language families have inherited a lot of features from Pascal and its successor Modula-2, notably strong type checking and the extension of that across compilation units. And even if BASIC is now, like just about everything else, compiled, the retro community still yearns for something with the immediate response of the classical implementations... I won't go on, you probably know that better than I do.

The only other things that give you that kind of immediacy are environments such as Lisp, Forth or Smalltalk... and quite frankly I think they're non-starters for what you're trying to do. There was at one time an incrementally-compiled Pascal called Mystic Pascal, I think I saw one copy sold and that it ended up on magazine-cover collections.

These days, there are far more people who know how to write a half-decent compiler than there were in the '80s. Unfortunately, they are intent on doing it "properly" by the standard of their computer science courses, and... well, even if they insist that they're using all the right techniques don't expect the result to be compact.

I'm pretty sure that you will not get FPC running on an 8-bit system. I've been trying during the day to remember whether Turbo Pascal on a CP/M-80 system actually switched between editor and compiler using an overlay... without success although I'd throw in that the much later VB for DOS did.

My suspicion is that it's easier to write a Pascal compiler than a BASIC one: recursive descent is adequate, and generally speaking it's possible to avoid having to consider the sort of hoops discussed in the well-regarded https://moam.info/writing-interactive-compilers-and-interpreters-pj-wiley-online-library_59b149bf1723ddd7c686d281.html

However, you need to consider two things: (a) do you need the level of immediacy that only BASIC really gives you and (b) if you decide to use Pascal are you prepared to stick to the language handled by the early versions (short strings, no try-finally construct, no exceptions, no objects and so on).

Oh, and I knew Hisoft and Dave Nutkins fairly well, as well as some of the people who wrote stuff for them.

Quote
The upshot of this is, that more users would know BASIC than any other language, so the system should use something similar. Pascal would be close enough for those used to structured BASIC.

But Structured BASIC is much younger than the 8-bit micros you're talking about: it's a 16-bit thing (QB4?).

Quote
Languages with a C-style syntax would be harder to adjust to. I know it caused me issues when I tried learning it.

There is very little difference between Pascal and C: both are direct descendants of ALGOL-60. I'd suggest that the problem you had with C was the number of idioms you were forced to learn- iterating a string using a pointer etc.- that were quite simply hidden in Pascal.

I'd suggest focussing not on what language you want, but on the degree of immediacy: what sort of IDE, and what sort of debugger. Compared with those, writing a compiler will be trivial.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 15, 2022, 12:07:13 am
And for similar reasons that makes using a PC-hosted cross-compiler unrealistic, /unless/ you produced an entire product range of physical and emulated micros with some sort of "tube" arrangement to the host for program download
I was hoping that anyone with the desire to target retro machines would either own one or find suitable emulators for themselves.

Including links to several of these wouldn't be an issue, but creating new ones or attempting to package some with the tools and keeping on top of updates would be too time consuming.

Quote
My suspicion is that it's easier to write a Pascal compiler than a BASIC one
I actually bought two books on writing an interpreter in Object Pascal and am waiting for the third to be published before diving in. The created language is a variation of BASIC.

Also, Decimal BASIC (http://hp.vector.co.jp/authors/VA008683/english/) is written in FreePascal and has the source code available, so I would have some sort of guidance if I went that route.

Quote
However, you need to consider two things: (a) do you need the level of immediacy that only BASIC really gives you and (b) if you decide to use Pascal are you prepared to stick to the language handled by the early versions (short strings, no try-finally construct, no exceptions, no objects and so on).
I think the desire for immediacy would be sated by any interpreter. Python is a popular mainstream example.

There are going to be some unavoidable restrictions in the language given the number of platforms I would like to support. I'm hoping most users would accept that.

Quote
Oh, and I knew Hisoft and Dave Nutkins fairly well, as well as some of the people who wrote stuff for them.
Are you still in contact with them? I've used the contact form on their site a couple of times asking about legally obtaining HiSoft Pascal for 8-bit machines and haven't had a response.

Quote
But Structured BASIC is much younger than the 8-bit micros you're talking about: it's a 16-bit thing (QB4?).
By "structured BASIC" I meant ANSI Full BASIC or BBC BASIC that allowed for a more modular construction via procedures and provided more looping options. They were available somewhere around the late 80s. IS-BASIC on the Z80-based Enterprise 64 (released in 1985) was a variation of ANSI Full BASIC.

Quote
There is very little difference between Pascal and C: both are direct descendants of ALGOL-60. I'd suggest that the problem you had with C was the number of idioms you were forced to learn- iterating a string using a pointer etc.- that were quite simply hidden in Pascal.
Having to use braces was off-putting, but dealing more directly with memory allocation was also a factor, yes.

Quote
I'd suggest focussing not on what language you want, but on the degree of immediacy: what sort of IDE, and what sort of debugger. Compared with those, writing a compiler will be trivial.
Having a quick way to test code would be a high priority. Maybe that could be made available via a compile and run option on the host machine as the programs shouldn't become too large.

Many errors wouldn't be caught until compile time though, unless each line of code could be assessed by the system in the same way that most retro BASIC interpreters did.

I haven't thought about the IDE too deeply, as I was hoping to allow almost any existing tool to be used. I shall have to reconsider that if I want an integrated interpreter rather than a command line one.

The debugger I haven't considered at all, mostly because of not knowing the direction I will be taking. If I modify or reuse an existing system, it will likely already have one available.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: SymbolicFrank on August 15, 2022, 09:14:02 am
@Wysardry:

When you talk about "a compiler", you're actually talking about three separate things:

1. The actual compiler, that translates your program into machine code.
2. The support libraries, that handle the I/O.
3. The user-interface.

If you have a compiler that compiles to your target CPU architecture, you're done with 1. FPC scores high here. And if you are fine with a character-based UI, as you said, 3 is handled automatically when you have a (G)UI and 2.

What you are talking about is 2. And especially: video, keyboard (, mouse) and storage. Mostly, this is a list of pointers and constants, with some very basic functions. Like, where does the screenbuffer start, how many rows and columns, how do I print a char at that location. Like, an old BIOS. Compiled, it will fit a few KB of code.



Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 15, 2022, 09:20:37 am
Quote
Oh, and I knew Hisoft and Dave Nutkins fairly well, as well as some of the people who wrote stuff for them.
Are you still in contact with them? I've used the contact form on their site a couple of times asking about legally obtaining HiSoft Pascal for 8-bit machines and haven't had a response.

Like other companies of the era, I suspect that the original entity no longer exists (i.e. there might be a successor, but they have no interest in the original business and no longer represent the same software authors etc.).

Quote
I haven't thought about the IDE too deeply, as I was hoping to allow almost any existing tool to be used. I shall have to reconsider that if I want an integrated interpreter rather than a command line one.

The debugger I haven't considered at all, mostly because of not knowing the direction I will be taking. If I modify or reuse an existing system, it will likely already have one available.

That's where you need to start. Writing a compiler is by now well-understood, but a half-decent IDE and debugger is still an art.

Frankly, I think that in effect you're out to reinvent Turbo Pascal or something of that era, and that you'll find it extremely difficult to fit it onto an 8-bit system without writing large parts in carefully-tuned assembler.

With respect to other members of this community, but high-level languages, particularly when you start using their standard libraries etc., have their limitations.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 15, 2022, 10:04:05 am
What you are talking about is 2. And especially: video, keyboard (, mouse) and storage. Mostly, this is a list of pointers and constants, with some very basic functions. Like, where does the screenbuffer start, how many rows and columns, how do I print a char at that location. Like, an old BIOS. Compiled, it will fit a few KB of code.

/If/ the target has a screenbuffer. That was the difference between the MS-DOS and PC-DOS variants of early TP: the PC-DOS one had a memory mapped display (and was what matured into later versions) while the CP/M and MS-DOS ones assumed ASCII+escapes.

Then there was a small number of code generators and RTLs, and a small number of OSes.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: SymbolicFrank on August 15, 2022, 11:00:42 am
What you are talking about is 2. And especially: video, keyboard (, mouse) and storage. Mostly, this is a list of pointers and constants, with some very basic functions. Like, where does the screenbuffer start, how many rows and columns, how do I print a char at that location. Like, an old BIOS. Compiled, it will fit a few KB of code.

/If/ the target has a screenbuffer. That was the difference between the MS-DOS and PC-DOS variants of early TP: the PC-DOS one had a memory mapped display (and was what matured into later versions) while the CP/M and MS-DOS ones assumed ASCII+escapes.

Then there was a small number of code generators and RTLs, and a small number of OSes.

MarkMLl

Certainly. But that doesn't matter. You need to know how many rows and columns it can hold, and you need a function to write a char at that location. Optionally, a function to read the char at a location, but you can emulate that with an array. For a character-based display, everything else is optional, but functions for scrolling and such are nice to have, as those can be hardware-accelerated.

So, you end up with a screenbuffer and the same functions in either case. And that buffer probably holds two locations (bytes) for each char: one for the ASCII code and another one for the attributes (color, blink, underline, etc). Or two buffers, if you want to make it easier to support different displays. If you want it to be generic, you define those.

Or, for FPC, a crt unit.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: avra on August 15, 2022, 03:39:34 pm
If that was my project, I would go a slightly different path. Write a p-code compiler and interpreter.

The main change is that instead of creating different binaries (each time you change sources) for every target with cross compiler, you cross compile once to generate interpreter and compiler for each target (ok, compiler can be just on host if that suits your goal better), and then after each source change you generate single p-code file that will be the same for each target.

Depending on your goals, p-code idea may be better or worse approach to the problem.

Here are some useful links:
https://wiki.freepascal.org/Make_your_own_compiler,_interpreter,_parser,_or_expression_analyzer
https://en.wikipedia.org/wiki/P-code_machine
http://standardpascaline.org/p5.html

Going back to your original idea of just renaming WriteLn() to MyWriteLn(), I think that compared to any other idea it would be more realistic and feasible to just create a single INC file with all your macro renaming aliases that you would just have to include into all your units, and simply continue to use FreePascal with it's full power without limits that a one man solution would bring. You could start immediately writing apps for already supported FPC targets, and later implement missing targets for FPC. That seams to me far less work compared to other alternatives.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 15, 2022, 05:38:17 pm
@Wysardry:

When you talk about "a compiler", you're actually talking about three separate things:
Ideally, I'd like to reuse whatever is already available, to reduce the amount of time and effort it would take, plus a certain number of people would already know the original tools/language/system.

Like other companies of the era, I suspect that the original entity no longer exists (i.e. there might be a successor, but they have no interest in the original business and no longer represent the same software authors etc.).
I rechecked the HiSoft About Us (https://www.hisoft.co.uk/about%20hisoft/) page and it seems David Link still works there, but Dave Nutkins is barely mentioned.

The company now seems to provide services related to web design.

Quote
That's where you need to start. Writing a compiler is by now well-understood, but a half-decent IDE and debugger is still an art.
I think I'll start testing similar software to see what works and what doesn't.

Quote
Frankly, I think that in effect you're out to reinvent Turbo Pascal or something of that era, and that you'll find it extremely difficult to fit it onto an 8-bit system without writing large parts in carefully-tuned assembler.
I'm hoping to avoid creating tools that run on the 8-bit targets. I would much rather reuse what already works, by producing source code to feed into existing 8-bit compilers.

If that was my project, I would go a slightly different path. Write a p-code compiler and interpreter.
I had considered this idea, but discounted it due to the lack of memory on the 8-bit systems.

Interpreters and virtual machines often waste a lot of memory, as they have to contain code for every supported function even if the program it is currently running does not use them all.

Quote
Going back to your original idea of just renaming WriteLn() to MyWriteLn(), I think that compared to any other idea it would be more realistic and feasible to just create a single INC file with all your macro renaming aliases that you would just have to include into all your units...
Yes, storing macros in include files is similar to what I had in mind. I also envisioned creating a library of different files containing functions that could optionally be added if needed.

If this method would interfere with Pascal to other language translators though, that would be a problem. Although many of the 8-bit home computers had a Z80 processor, not all of them did, so FreePascal alone (with its current targets) would not be enough.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: winni on August 15, 2022, 06:40:24 pm

If that was my project, I would go a slightly different path. Write a p-code compiler and interpreter.
I had considered this idea, but discounted it due to the lack of memory on the 8-bit systems.

Interpreters and virtual machines often waste a lot of memory, as they have to contain code for every supported function even if the program it is currently running does not use them all.

Hi!

Just remember:

The whole UCSD operating system including p-machine, editor, compiler, filer did fit into 64 kB of RAM on an Apple ][ with a 6502.  Or cheaper clones.

It was also called Apple Pascal. The Apple version of UCSD was also able to do graphics.

So this interpreter for a p-machine did not waste too much memory ....

Winni

Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: winni on August 15, 2022, 06:53:21 pm
@avra

Maybe interesting for you:

Derek Jones from Knowledge Software wrote the system.interpr(eter) for the UCSD p-system in the 80s in C. Until then it was written in Assembler and had to be adapted on every new processor.

Knowledge Software:  http://knosof.co.uk/ (http://knosof.co.uk/)

Winni



Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 15, 2022, 07:23:44 pm
I have come up with a set of existing open source tools that might work together to form the system that I'm aiming for:-

If I was able to get all these to play together nicely, I could then gradually create the following using FreePascal:-

The last would probably not be needed if the Turbo Rascal translator was done.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Handoko on August 15, 2022, 07:51:14 pm
That seems great. But in case you haven't know, not all open source licenses can be combined with others. For example Decimal BASIC's GPLv2 is incompatible with zxbasic's GPLv3. I'm not a lawyer, but these are what I found:

https://opensource.stackexchange.com/questions/1777/why-is-gplv2-incompatible-with-gplv3

Quote
... the GPLv2 license by itself is not compatible with GPLv3 ...
Source: https://en.wikipedia.org/wiki/License_compatibility#Copyleft_licenses_and_GPL

Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 15, 2022, 08:18:18 pm
This shouldn't be an issue if each tool was kept separate and retained its own license though, right?

I don't have any intention to merge code.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: SymbolicFrank on August 16, 2022, 09:44:18 am
Writing a crt unit for each system is a lot less work then trying to make all those work together. I wouldn't do such a project, it is almost guaranteed to fail.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 16, 2022, 10:10:11 am
I don't have any intention to merge code.

In that case what's the point? You seem to be aiming at a cross-compiled solution with no common download mechanism. You appear to be aiming at similar "look and feel" on different target platforms, but using completely different libraries to achieve that... do you have any idea at all what a maintenance nightmare that's likely to be?

You started the thread asking about a DSL, but what you're now talking about isn't. You're now talking about multiple layers of code translation, but don't appreciate that with every hop you will introduce inefficiency and miss optimisation possibilities.

If you really are prepared to accept a cross-compiler and non-native IDE etc., then start off with FPC or e.g. Turbo-Rascal (don't expect to be able to use both) and after identifying what desirable targets they don't support learn enough to help the project team add the desired functionality.

In any event, you've got a big job on your hands and the first thing you have to do is have a robust description of what /exactly/ you're trying to do: for your own benefit as much as anybody else.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: avra on August 16, 2022, 10:19:05 am
If that was my project, I would go a slightly different path. Write a p-code compiler and interpreter.
I had considered this idea, but discounted it due to the lack of memory on the 8-bit systems.

Interpreters and virtual machines often waste a lot of memory, as they have to contain code for every supported function even if the program it is currently running does not use them all.
Are you aware that almost every mentioned 8-bit system has built in BASIC interpreter as a proof that your claims are wrong? Only essentials are embedded, the rest is provided through external units/ libraries.

Although many of the 8-bit home computers had a Z80 processor, not all of them did, so FreePascal alone (with its current targets) would not be enough.
I've already said what would be enough:
Quote
You could start immediately writing apps for already supported FPC targets, and later implement missing targets for FPC.
If that step is too much for you, then you should not waste time on this direction...
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 16, 2022, 06:05:16 pm
In that case what's the point? You seem to be aiming at a cross-compiled solution with no common download mechanism.
The point would be to provide multiple translation and cross compilation options, depending on what target platforms the end user is targetting. There is no existing direct route for all platforms.

It's a little early to be thinking about how everything would be made available for download.

Quote
You started the thread asking about a DSL, but what you're now talking about isn't. You're now talking about multiple layers of code translation, but don't appreciate that with every hop you will introduce inefficiency and miss optimisation possibilities.
The tools I suggested were for a proof of concept, so that I could learn what would work and what wouldn't.

Decimal BASIC could become the DSL, with the appropriate adjustments to its FreePascal code. The existing BASIC to Pascal translators might also need adjusting, depending on whether I added code to create the DSL or just removed some.

The main reasons that is the start of the testing chain is it has BASIC syntax, an existing interpreter and is written in FreePascal.

Quote
start off with FPC or e.g. Turbo-Rascal (don't expect to be able to use both) and after identifying what desirable targets they don't support learn enough to help the project team add the desired functionality.
It's entirely possible that I might do that in the future, especially if my tests go badly.

Quote
In any event, you've got a big job on your hands and the first thing you have to do is have a robust description of what /exactly/ you're trying to do: for your own benefit as much as anybody else.
What I'm trying to do is create (or reuse) tools to allow text-based programs to be written in a domain specific language and/or library on modern platforms and then compile or cross compile them for as many platforms as practical. Testing should be possible on the modern platforms before cross compilation is attempted to simplify the workflow.

Even if I was creating everything myself from scratch, each of the tools would be independent programs and not every project would use them all.

Are you aware that almost every mentioned 8-bit system has built in BASIC interpreter as a proof that your claims are wrong? Only essentials are embedded, the rest is provided through external units/ libraries.
Are you aware that many of the 64Kb machine had less than 40Kb available to write programs in? Much of the unavailable space was used by the BASIC interpreter (although some was taken by the BIOS of the day).

Quote
I've already said what would be enough:
Quote
You could start immediately writing apps for already supported FPC targets, and later implement missing targets for FPC.
If that step is too much for you, then you should not waste time on this direction...
I've already said that I'd be willing to learn enough to add new Z80 targets as I've been told it's relatively simple due to the work already done by others.

The problem is that creating new targets for unsupported CPUs would be too much for me, so that route would lead to a dead end.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 16, 2022, 07:11:13 pm
It's a little early to be thinking about how everything would be made available for download.

I meant download from the development host to the target. You need to be considering that sort of thing early- very early- since accessibility of debugging symbols etc. is an important issue.

Quote
Are you aware that many of the 64Kb machine had less than 40Kb available to write programs in? Much of the unavailable space was used by the BASIC interpreter (although some was taken by the BIOS of the day).

40K? Luxury!

Note that the term "BIOS" came in with CP/M, in which context it was a part of the OS which was loaded into memory. Most systems either had a full BASIC (with MS copyright notices) or a boot loader which might or might not have remained accessible after the OS had been started... the latter was again more of a CP/M thing but there were still non-CP/M systems which in principle were RAM-only such as the Sharp MZ-80.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 16, 2022, 07:54:53 pm
In most cases, the last tool in the chain would decide the final export format. There are also various existing tools to convert from one format to another.

I'm reasonably sure the "BIOS of the day" varied enough that there wasn't one applicable term, or one that I recall anyway. I do remember "CMOS" being used a lot, but as I understand it that is just a type of chip that can have multiple uses.

I used to own a Sharp MZ-700 and that had a mostly empty memory when first switched on. BASIC was provided on tape or disk and had to be loaded before you could use it.

I think other programming languages could be purchased and loaded instead, but I didn't own any.

I also owned an Enterprise 128, which had BASIC on a cartridge. Again, other languages could be purchased, but I only ever bought a BASIC compiler (on tape).
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 16, 2022, 08:15:42 pm
BIOS was, pre-PC, specifically a CP/M term. At least on CP/M-80, DR shipped a standard BDOS and the machine-specific BIOS was quite simply concatenated to it by the manufacturer... CP/M-86 might have had, and CCP/M-86 definitely did have, a slightly more complex arrangement.

CMOS describes a particular type of electronics rather than to a particular device. Refer to Wikipedia.

And the Sharps were unusual in loading almost everything from tape, I think the ROM was left in the memory map but didn't have much on it. I used one for my MPhil.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: y.ivanov on August 17, 2022, 02:36:41 am
It's a little early to be thinking about how everything would be made available for download.

I meant download from the development host to the target. You need to be considering that sort of thing early- very early- since accessibility of debugging symbols etc. is an important issue.

*snip*

IMHO is the trickiest part of the whole venture. All of the vintage home computers I've had (Apple ][, Oric Atmos, SinclairQL) used some sort of tape cartridges/cassettes or (luxury) floppies with a specific format. Not even a serial port (except for QL).

@Wysardry
If I'm getting you right, the idea is to use BASIC as a primary language, then transpile it to Pascal, use FPC as a cross-compiler to the specific targets (mostly 8-bit vintage CPUs), eventually debug the product in some emulators, and then somehow "magically" send them to the actual items for execution. Correct me if I'm wrong on this.

And in case I'm not wrong, there are few things to consider:

The main BASIC virtue isn't the simplicity of the language, but convenience of use - its interactivity and simultaneous execution, see DTSS (https://en.wikipedia.org/wiki/Dartmouth_Time_Sharing_System). Your workflow is about to nullify those advantages.   

Most if not all of those micros have BASIC included into ROM. Despite it is mostly a toy implementation, it gives the ability to write short programs without the need to load additional executables from the external storage. You don't waste precious RAM, time or effort. In your workflow, again, it requires some sort of downloading and the binary will reside into RAM together with the RTL.

8-bit micros where made with Z80 and 6502 CPUs. While the former is somewhat capable, the latter is a pathetic compiler-target for any HLL - 8-bit registers, just one accumulator. One way it can be used is with some sort of bytecode interpreter written in assembly. Not sure it is a disadvantage, though, since the bytecode can be carved to be more compact than the native one from a compiler. But someone has to write the interpreter in assembly.

Each micro has its own ways to work with the display/keyboard, character codes are weird, sometimes with inverse/blinking bits in them. Any idea how to match, i.e. how to display the same letters for all micros? Graphics mode?

Have you considered/investigated the FORTH system? It exploits things like RPN (https://en.wikipedia.org/wiki/Reverse_Polish_notation), threaded code (https://www.complang.tuwien.ac.at/forth/threaded-code.html) and it is traditionally targeted on restricted environments. IMHO it can be used as in intermediate language or/and interpreter.



Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 17, 2022, 06:26:37 am
IMHO is the trickiest part of the whole venture. All of the vintage home computers I've had (Apple ][, Oric Atmos, SinclairQL) used some sort of tape cartridges/cassettes or (luxury) floppies with a specific format. Not even a serial port (except for QL).

@Wysardry
If I'm getting you right, the idea is to use BASIC as a primary language, then transpile it to Pascal, use FPC as a cross-compiler to the specific targets (mostly 8-bit vintage CPUs), eventually debug the product in some emulators, and then somehow "magically" send them to the actual items for execution. Correct me if I'm wrong on this.
That is one of the possible workflows, yes. Apart from the magical aspect that is.

There are existing tools that allow suitable files to be converted into an audio format. Some retro compilers include the option as standard.

For example, the ZX Basic Compiler (https://www.boriel.com/pages/the-zx-basic-compiler.html) can create files in .bin and .tzx format for use with emulators and in .mp3 and .wav format to use with digital recorders or record to tape.

Some emulators also include an audio export option, and there are third party tools which do the same.

I doubt these options exist for every retro platform, but they do for most of the popular ones.

Quote
The main BASIC virtue isn't the simplicity of the language, but convenience of use - its interactivity and simultaneous execution, see DTSS (https://en.wikipedia.org/wiki/Dartmouth_Time_Sharing_System). Your workflow is about to nullify those advantages.
On a modern machine you do lose the immediate feedback you would get from a retro machine when entering code a line at a time, but you gain a better editing environment.

Including an interpreter should allow you to test programs each time you complete a section of code.

BASIC is still used on modern platforms, possibly by more people than Pascal (if you include all dialects of both languages).

Quote
Most if not all of those micros have BASIC included into ROM. Despite it is mostly a toy implementation, it gives the ability to write short programs without the need to load additional executables from the external storage. You don't waste precious RAM, time or effort. In your workflow, again, it requires some sort of downloading and the binary will reside into RAM together with the RTL.
The BASIC interpreter would only be used on modern platforms with memory and storage space to spare.

At the other end of the toolchain a machine code file would be transferred to the retro machine, which would most likely overwrite any BASIC interpreter in RAM. Most commercial software was sold in machine code form rather than in BASIC (although there were some exceptions).

Quote
Each micro has its own ways to work with the display/keyboard, character codes are weird, sometimes with inverse/blinking bits in them. Any idea how to match, i.e. how to display the same letters for all micros? Graphics mode?
I would hope the differences would be handled by the final compiler, if it includes configuration settings for multiple platforms. Graphics would not be an issue as the system is only intended to be used to create text-based programs.

Quote
Have you considered/investigated the FORTH system?
I have absolutely no experience with FORTH and the only thing I know about it is it was included with the Jupiter Ace, which I have never seen a physical example of.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 17, 2022, 09:28:09 am
There are existing tools that allow suitable files to be converted into an audio format. Some retro compilers include the option as standard.

For example, the ZX Basic Compiler (https://www.boriel.com/pages/the-zx-basic-compiler.html) can create files in .bin and .tzx format for use with emulators and in .mp3 and .wav format to use with digital recorders or record to tape.

Some emulators also include an audio export option, and there are third party tools which do the same.

I doubt these options exist for every retro platform, but they do for most of the popular ones.

Most of these things used audio, i.e. CUTS or something roughly equivalent, so in principle a bit of signal processing and you're there. There might have been one or two which were actually digital (i.e. PE or whatever), but in any case you should find designing something that looks like a cassette pretty easy... I presume you've got a 'scope etc. and know how to use it.

Quote
I have absolutely no experience with FORTH and the only thing I know about it is it was included with the Jupiter Ace, which I have never seen a physical example of.

In that case get off your backside and at least read about how it worked. That sort of ignorance is out of place for somebody trying to do what you are.

But I'm troubled that the use of a multi-layered cross-development platform with diminished immediacy on the target makes this a non-starter: there just isn't enough there to make it attractive to the average retro user who only cares about the computer and software he remembers from his teens. Are you /really/ serious about dedicating a big chunk of your life to doing this?

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: avra on August 17, 2022, 10:57:06 am
Are you aware that almost every mentioned 8-bit system has built in BASIC interpreter as a proof that your claims are wrong? Only essentials are embedded, the rest is provided through external units/ libraries.
Are you aware that many of the 64Kb machine had less than 40Kb available to write programs in? Much of the unavailable space was used by the BASIC interpreter (although some was taken by the BIOS of the day).
In the eighties ZX81 had 8Kb ROM and 1Kb RAM, and it had built in BASIC editor (which takes most space and not needed by you) and interpreter. Screen handling, cassete and cartridge handling and lots of other things also took ROM space but let's be gentle and not count them. ZX Spectrum had 16Kb ROM and 16Kb RAM. I remember that in my early ATMEL AVR days I have even seen a magazine publishing full sources of pascal interpreter for a microcontroller with 1Kb RAM and 8Kb flash. That is way below your available resources so implementation possibility is not in question. Did you look at p-machine implementations at all? P6 is fresh, but first few were built in the seventies when resources were even more limited, so I do not see a problem comming from that side.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 17, 2022, 11:24:04 am
Did you look at p-machine implementations at all? P6 is fresh, but first few were built in the seventies when resources were even more limited, so I do not see a problem comming from that side.

Noting that there are two different usages of "P" here: Wirth's original system described at http://pascal.hansotten.com/px-descendants/ and the UCSD P-system.

I think it's worth noting that chips which are currently considered to be "absolutely miniscule" like the ESP8266 hence barely supportable by Pascal are, in fact, substantially larger than most 1980s home computers.

TBH, I think that OP would do far better throwing his enthusiasm behind either FPC or Turbo Rascal, or identifying some comparable project with a common frontend language and multiple backends.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 17, 2022, 01:34:26 pm
I think it's worth noting that chips which are currently considered to be "absolutely miniscule" like the ESP8266 hence barely supportable by Pascal are, in fact, substantially larger than most 1980s home computers.

I wouldn't consider the ESP8266 even close as “miniscule”. It's simply an embedded platform just like the Pi Pico which Pascal can run very well on. More miniscule are the AVRs, especially the TinyAVRs and even there you can use Pascal ;)
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 17, 2022, 02:20:08 pm
I wouldn't consider the ESP8266 even close as “miniscule”. It's simply an embedded platform just like the Pi Pico which Pascal can run very well on. More miniscule are the AVRs, especially the TinyAVRs and even there you can use Pascal ;)

I'd hardly call the Pico tiny, with 100s of Kb RAM and provision for several Mb Flash. And whenever one maxes-out an AVR one can simply move to the next one along the product range.

However I think it's worth emphasising for OP's benefit that FPC/Lazarus has decent debugging facilities for targets such as the Pico/RP2040, and that it's taken the collective development tool communities hundreds of man-years to get there.

If I had £1 for every suggestion I've seen over the years that a single under-resourced developer of limited experience could rewrite everything from scratch in a finite time, or that "it must be easy" to integrate a few dozen tools from competing suppliers to produce a result that would have well-heeled customers beating a path to his door, I'd be much richer than I am now.

And long-term members of this community will know that I've done some of this stuff myself, with some degree of success. But I have enormous respect for the FPC team- Florian, Jonas, yourself (PascalDragon) and the rest, who by and large manage to keep up with the minutiae and ramifications of an increasingly-large language and ecosystem.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: marcov on August 17, 2022, 02:48:00 pm
Did you look at p-machine implementations at all? P6 is fresh, but first few were built in the seventies when resources were even more limited, so I do not see a problem comming from that side.

Which P6 ?

https://github.com/samiam95124/Pascal-P6     Last commit 4-5 years ago....

http://wirth-dijkstra-langs.org/  Hasn't processed FPC compatibility contributions sent 3 (4?) years ago.

Both seem to be quite stale.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 17, 2022, 03:46:27 pm
Did you look at p-machine implementations at all? P6 is fresh, but first few were built in the seventies when resources were even more limited, so I do not see a problem comming from that side.

Which P6 ?

https://github.com/samiam95124/Pascal-P6     Last commit 4-5 years ago....

http://wirth-dijkstra-langs.org/  Hasn't processed FPC compatibility contributions sent 3 (4?) years ago.

Both seem to be quite stale.

https://sourceforge.net/projects/pascalp5/ appears to have seen action over the last year or so.

Why should they have any interest in FPC compatibility? My recollection is that their priority is to work on (emulations of) larger machines including IBM mainframes, so they might simply have ignored them

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 17, 2022, 07:32:12 pm
In that case get off your backside and at least read about how it worked. That sort of ignorance is out of place for somebody trying to do what you are.
There are plenty of other rabbit holes for me to dive down that would be a more direct route to where I want to end up.

Most likely, Forth is one of the many languages I have discounted because they either don't run on the machines I would be targetting or the syntax looks too different to BASIC or Pascal.

Quote
But I'm troubled that the use of a multi-layered cross-development platform with diminished immediacy on the target makes this a non-starter: there just isn't enough there to make it attractive to the average retro user who only cares about the computer and software he remembers from his teens. Are you /really/ serious about dedicating a big chunk of your life to doing this?
This project wouldn't really be aimed at someone who wants to use a particular computer from their youth, as they can just use BASIC on a physical machine or in an emulator.

This would be for someone who likes or uses more than one machine and wants to produce software for all of them plus modern machines, using one set of code.

The choice of BASIC isn't really for the immediacy aspect, it's to make the system more familiar and to potentially ease translation to other BASIC dialects.

I don't want to dedicate a big chunk of my life to it, no. That's why I'm trying to reuse as many existing tools as possible.

Did you look at p-machine implementations at all? P6 is fresh, but first few were built in the seventies when resources were even more limited, so I do not see a problem comming from that side.
I actually bought a used book on creating text adventures in UCSD Pascal a couple of years back.

I haven't seen any implementations for 8-bit home computers though, so I haven't explored it too thoroughly.

TBH, I think that OP would do far better throwing his enthusiasm behind either FPC or Turbo Rascal, or identifying some comparable project with a common frontend language and multiple backends.
I will definitely be considering adding extra Z80 targets to FPC if it becomes part of the toolchain.

Turbo Rascal would be somewhat of a non-starter because AFAIK it has a small team who likely have their own workflow and wouldn't want a beginner who hates C messing with their existing code.

I could potentially fork it on Github, but my progress would be slow and I'd hate every minute of it due to the amount of C and C++ code I'd have to contend with.

If I actually liked C and knew more about it, I would have more options available to me.

If I had £1 for every suggestion I've seen over the years that a single under-resourced developer of limited experience could rewrite everything from scratch in a finite time, or that "it must be easy" to integrate a few dozen tools from competing suppliers to produce a result that would have well-heeled customers beating a path to his door, I'd be much richer than I am now.
I do know that it will be far from easy. I've been searching for potential solutions on and off for about two years now, trying to find compatible pieces of the puzzle.

The shortest existing toolchain I have found so far is Nim (via its C output) to SDCC or z88dk, a combination which apparently has been successfully used with micro boards with 32Kb ROM and 512 bytes of RAM. There is no existing option to use BASIC with that, but it might be possible to use macros to allow the use of some BASIC and/or domain specific keywords.

Although Nim was originally written in Pascal and there is a Pascal to Nim translator, Nim is now written in Nim and there is no Nim to Pascal translator available that I know of.

I'm not trying to do this because I expect it to sell like hot cakes (it would be free anyway), but rather because it would be a tool that I would find useful and I hope that at least some others would find useful too.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: y.ivanov on August 17, 2022, 08:12:01 pm
*snip*
Most likely, Forth is one of the many languages I have discounted because they either don't run on the machines I would be targetting or the syntax looks too different to BASIC or Pascal.
It was about how someone approached the particular problem with its own DSL. Not for the language itself. BTW BASIC was created in a similar way.

*snip*
This project wouldn't really be aimed at someone who wants to use a particular computer from their youth, as they can just use BASIC on a physical machine or in an emulator.
I can hardly imagine anyone doing it except out of nostalgic reasons.

This would be for someone who likes or uses more than one machine and wants to produce software for all of them plus modern machines, using one set of code.
*snip*
And who would want to do that? Where is the point?
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 17, 2022, 08:51:04 pm
There is already software available that allows people to create code on a modern machine that will run on modern machines and multiple retro platforms, so there must be at least some demand for it.

CROSS LIB (https://github.com/Fabrizio-Caruso/CROSS-LIB) is an example of a game framework for vintage systems that can also be used for modern systems via GCC.

Lantern (http://textadventure.net/) is a similar system for creating text adventures.

Several programming languages with multiple targets have already been mentioned in this thread.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 17, 2022, 10:14:46 pm
There are plenty of other rabbit holes for me to dive down that would be a more direct route to where I want to end up.

Most likely...

Well, yes: you make a good point. But then spoil it with a "most likely" which suggests that you haven't done even basic research.

The important point about Forth is not the (pretty bad) language syntax, but the execution model.

Quote
I actually bought a used book on creating text adventures in UCSD Pascal a couple of years back.

I haven't seen any implementations for 8-bit home computers though, so I haven't explored it too thoroughly.

Right, so you don't know a thing about it.

It was for a long time significantly popular on Apple-II systems... but the important thing is /how/ it worked.

Quote
I will definitely be considering adding extra Z80 targets to FPC if it becomes part of the toolchain.

Although despite my goading you appear to have very little understanding that this is mainly an RTL rather than compiler issue.

Quote
Turbo Rascal would be somewhat of a non-starter because AFAIK it has a small team who likely have their own workflow and wouldn't want a beginner who hates C messing with their existing code.

And you reckon FPC is any different? And you really think you're going to get anywhere if "I hate C" is your overriding criterion?

Quote
I do know that it will be far from easy. I've been searching for potential solutions on and off for about two years now, trying to find compatible pieces of the puzzle.

And I've been doing the same for about 45 years. Need I say more?

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: avra on August 18, 2022, 01:43:15 am
Did you look at p-machine implementations at all? P6 is fresh, but first few were built in the seventies when resources were even more limited, so I do not see a problem comming from that side.

Which P6 ?

https://github.com/samiam95124/Pascal-P6     Last commit 4-5 years ago....

http://wirth-dijkstra-langs.org/  Hasn't processed FPC compatibility contributions sent 3 (4?) years ago.

Both seem to be quite stale.

https://sourceforge.net/p/pascal-p6/code/ci/master/tree/

Last change was in 2020
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 18, 2022, 09:05:14 am
I wouldn't consider the ESP8266 even close as “miniscule”. It's simply an embedded platform just like the Pi Pico which Pascal can run very well on. More miniscule are the AVRs, especially the TinyAVRs and even there you can use Pascal ;)

I'd hardly call the Pico tiny, with 100s of Kb RAM and provision for several Mb Flash. And whenever one maxes-out an AVR one can simply move to the next one along the product range.

Okay, the ESP8266 has around a tenth of the RAM of the Pico, but it can also support several MB of Flash (e.g. here (https://www.sparkfun.com/products/17146)). And at least before the recent announcements of the W variants of the Pico the ability to use WiFi and Bluetooth was quite a selling point (though I personally would go with the ESP32 instead).

Turbo Rascal would be somewhat of a non-starter because AFAIK it has a small team who likely have their own workflow and wouldn't want a beginner who hates C messing with their existing code.

You are aware that FPC also has a relatively small circle of really active core developers? Also with our own workflow... However we are definitely open for merge requests as long as they adhere to a few things (e.g. not changing coding style or providing features that we don't consider in the spirit of Pascal).
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 18, 2022, 09:48:19 am
Turbo Rascal would be somewhat of a non-starter because AFAIK it has a small team who likely have their own workflow and wouldn't want a beginner who hates C messing with their existing code.

You are aware that FPC also has a relatively small circle of really active core developers? Also with our own workflow... However we are definitely open for merge requests as long as they adhere to a few things (e.g. not changing coding style or providing features that we don't consider in the spirit of Pascal).

I must say that looking superficially at the example screenshots e.g. https://lemonspawn.com/wp-content/uploads/2020/08/trse_background.png I'm a little troubled at the extent to which the Turbo Rascal developers have departed from the standard Pascal syntax. For example, they've reverted to Wirth's original /* */ for comments and they're declaring arrays like
Code: Pascal  [Select][+][-]
  1. array[256] of byte
.

It might well be that their requirements were very similar to OP's and that they settled on something "Pascal-like" because it's usable rather than out of affection for the language and its traditions, but there's a risk that at some point in the future somebody will start arguing that FPC ought to have a TurboRascal mode.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 18, 2022, 01:07:11 pm
Turbo Rascal would be somewhat of a non-starter because AFAIK it has a small team who likely have their own workflow and wouldn't want a beginner who hates C messing with their existing code.

You are aware that FPC also has a relatively small circle of really active core developers? Also with our own workflow... However we are definitely open for merge requests as long as they adhere to a few things (e.g. not changing coding style or providing features that we don't consider in the spirit of Pascal).

I must say that looking superficially at the example screenshots e.g. https://lemonspawn.com/wp-content/uploads/2020/08/trse_background.png I'm a little troubled at the extent to which the Turbo Rascal developers have departed from the standard Pascal syntax. For example, they've reverted to Wirth's original /* */ for comments and they're declaring arrays like
Code: Pascal  [Select][+][-]
  1. array[256] of byte
.

It might well be that their requirements were very similar to OP's and that they settled on something "Pascal-like" because it's usable rather than out of affection for the language and its traditions, but there's a risk that at some point in the future somebody will start arguing that FPC ought to have a TurboRascal mode.

There'd probably be a Oxygene mode before that... ::)
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: y.ivanov on August 18, 2022, 02:04:38 pm
There is already software available that allows people to create code on a modern machine that will run on modern machines and multiple retro platforms, so there must be at least some demand for it.

CROSS LIB (https://github.com/Fabrizio-Caruso/CROSS-LIB) is an example of a game framework for vintage systems that can also be used for modern systems via GCC.

Lantern (http://textadventure.net/) is a similar system for creating text adventures.

Several programming languages with multiple targets have already been mentioned in this thread.
I see. My suggestion is that post relates to https://forum.lazarus.freepascal.org/index.php/topic,55335.msg411593.html#msg411593
And it is also about text adventure games, right?

About CROSS LIB - it is a collection of gcc cross compilers and a multi-target library. That is how I would go, but I'm not an opposer of the C language.
About Lantern - it is a GUI of a game builder, it has a script language with {}, not clear to me how the images were built, it is not open-sourced.

If the aim is towards the text adventure games, there is no need for a full programming language. Such games can be completely data driven, i.e. you need the specific game data plus a static interpretative part.

You can write a decent GUI editor (like Lantern) in Lazarus and then assemble a BASIC code with both data and the (game) interpreter. You can use only the common intersection of the available BASIC dialects: namely READ/DATA/RESTORE/PRINT/FOR/NEXT/IF/GOTO. You can even embed the 'scripts' directly into BASIC code - just put line numbers and substitute the 'variables' with a corresponding two-letter names.

That way you can have your beloved BASIC at your disposal and you can use what is given (in ROM) not to waste more RAM for a runtime.



Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 18, 2022, 02:09:58 pm
There'd probably be a Oxygene mode before that... ::)

I've never looked at in an any detail. Are you suggesting that Turbo Rascal might actually be an Oxygene rather than a Turbo Pascal derivative?

Following up https://docs.elementscompiler.com/Oxygene/ makes standard arrays look like ordinary Object Pascal, rather than the stringlike notation Turbo Rascal's using. Can't see anything on their comment format.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 18, 2022, 09:02:07 pm
The important point about Forth is not the (pretty bad) language syntax, but the execution model.
At this point, I don't know what languages, tools, execution model etc. I will be using. Until I do, thorough research on any topic is likely to eat up too much time for no real benefit.

Quote
... but the important thing is /how/ it worked.
Again, at this point that may or may not be relevant.

I'm not currently looking at using an interpreter or virtual machine on the retro targets. If I used one, it would only be on a modern machine for testing purposes.

Quote
And you reckon FPC is any different? And you really think you're going to get anywhere if "I hate C" is your overriding criterion?
As I would mainly be attempting this project for my own satisfaction, I won't be getting paid for it and there may only be a handful of other people that ever use it, I feel perfectly justified in rejecting any language or tool for any reason I wish.

If I dislike what I'm doing, I'll be less likely to want to finish it.

Quote
And I've been doing the same for about 45 years. Need I say more?
As it's unlikely I have 45 years left in me, telling me (and anyone else reading this) which parts were a waste of time would be useful. :)

You are aware that FPC also has a relatively small circle of really active core developers? Also with our own workflow... However we are definitely open for merge requests as long as they adhere to a few things (e.g. not changing coding style or providing features that we don't consider in the spirit of Pascal).
I may be wrong, but I get the impression that adding new Z80 targets to FreePascal would be simpler and involve creating less C code than doing the same with TurboRascal.

IOW, all other things being equal I would prefer the former.

And it is also about text adventure games, right?
It was originally related to creating text adventures and that is still the main focus, yes. However, I have realised that it wouldn't take much more effort to allow other text-based programs to be created with the same system.

Quote
About CROSS LIB - it is a collection of gcc cross compilers and a multi-target library. That is how I would go, but I'm not an opposer of the C language.
CROSS LIB is more complex than I need, as it supports the use of graphics.

Creating something simpler from scratch using FPC or Nim might be quicker than removing the parts I don't need.

Quote
About Lantern - it is a GUI of a game builder, it has a script language with {}, not clear to me how the images were built, it is not open-sourced.
There is some code available here (https://github.com/evancwright?tab=repositories) if you're interested.

There were also some videos available on Youtube, but I haven't checked lately. I think the project was originally used for teaching purposes.

Quote
If the aim is towards the text adventure games, there is no need for a full programming language. Such games can be completely data driven, i.e. you need the specific game data plus a static interpretative part.
There are already several systems that take that approach. I'm looking to create something more flexible.

Quote
You can write a decent GUI editor (like Lantern) in Lazarus and then assemble a BASIC code with both data and the (game) interpreter. You can use only the common intersection of the available BASIC dialects: namely READ/DATA/RESTORE/PRINT/FOR/NEXT/IF/GOTO. You can even embed the 'scripts' directly into BASIC code - just put line numbers and substitute the 'variables' with a corresponding two-letter names.
I was considering a similar system some time ago.

The idea was to create a GUI that allowed the user to create code using blocks and drop down menues similar to the way GameMaker works.

Internally, the code would be stored as an AST in XML or similar. It could then be converted to various program languages (including several BASIC dialects) ready for compilation or direct use.

I put this on the back burner as it didn't seem to be very accessible for screen reader users and I wanted to explore options for the back end of the system.

Quote
That way you can have your beloved BASIC at your disposal and you can use what is given (in ROM) not to waste more RAM for a runtime.
Although I would like to support the use of existing BASIC interpreters on retro targets if possible, I also want users to have other options.

I envision the interpreter as being a set of code libraries that can optionally be included in the primary source code. That way, if someone doesn't need complex maths functions (or whatever) they don't include them and they don't take up space in the final program.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 18, 2022, 10:38:20 pm
Quote
And I've been doing the same for about 45 years. Need I say more?
As it's unlikely I have 45 years left in me, telling me (and anyone else reading this) which parts were a waste of time would be useful. :)

:-) I've been trying to. Let's see if I can knock something out while waiting for my supper...

OK, in no particular order. As an engineer, I didn't have any sort of "formal background" in compilers. I still think that a lot of that stuff is garbage: if a syntax is difficult to parse then it would be better to select an unambiguous and efficient syntax than spending years working on new algorithms. But there's stuff in there which would have been enormously helpful when I was doing my MPhil in the '80s, provided I didn't get enmired in the crap.

When I started off, people weren't programming small computers in high-level languages (other than BASIC, and that was very much pre-structured). As such I had a better grounding in assembler, followed fairly promptly by the innards of Forth, Smalltalk and- slightly later- systems such as UCSD including their applicability to mainframes.

In terms of HLL, Forth, Smalltalk and APL are deadends because while they have interesting execution models their gross departure from expected syntax etc. puts them out on a limb. I'm not saying they're without merit, but I am saying that things like evaluation order are now so deeply ingrained in basic maths that ignoring them is perilous.

Reference here: the first three chapters of https://web.engr.oregonstate.edu/~budd/Books/leda/ are available for download and are worth reading carefully.

In the 1980s, when people were just about starting to take HLLs on micros seriously, I got a reprint of the classic paper on Meta-2 (I quoted an Alan Kay comment on this). I used that for a brief course for staff members at the university I was working at, and have been using a derivative for various tasks ever since. See Wp.

Much more recently, I came across Tree Meta (again, see Wp). That is much more difficult to get to grips with, but it has the potential of being able to specify rules to optimise the intermediate AST as part of the syntax definition. There was an implementation on GitHub which vanished, I got a lead on an ex-HP guy who'd used it in an instrument but he lost everything in one of the California fires :-(

Both of those may be used to generate a compiler using recursive descent, which AIUI is the technique used for FPC.

RD isn't particularly sexy and can barf when fed certain types of syntax. But it's pretty much adequate for Pascal or classic C, and doesn't need a higher degree to understand. And being simple it's compact, and is probably the technique that Borland used to get their early compilers to run natively on CP/M-80.

@PascalDragon: I've seen suggestion that RD isn't as good as some other approaches at error reporting and recovery. Do you have any comment?

To establish terminology, lets say that typical BASICs used tokenised-interpretive execution: each BASIC keyword was typically tokenised to a single byte when entered and the sequence of tokens, variable names etc. was interpreted at runtime. This is fairly concise but grossly inefficient.

Both Metas use a bytecode-interpretive arrangement as does UCSD Pascal, Java etc. In this, the high-level language is compiled beyond the AST stage and saved as a sequence of bytecodes, which may be subsequently interpreted or translated to e.g. assembler source for conversion to an executable binary.

Forth uses a distinct threaded-interpreted scheme, which was originally tightly coupled to the PDP system it was written for. I don't know how well this works on other systems, but the fact that a Forth implementation was used by Sun, IBM, Apple and the "One Laptop Per Child" project suggests that it's got it's good points: it might still be usable as some sort of intermediate notation, and roughly approximates the AST of a compiled language.

If you look at the P.J.Brown book that I cited, you will see that the BASIC-like language described has to gone to a lot of trouble to handle special cases that in the case of an ALGOL-derivative like Pascal "just work". By analogy, when Waychoff wrote an ALGOL compiler on ALGOL a whole lot of things "just worked" that Grau et al. had sweated blood over.

If you are looking at an implementation such as FPC, by and large there is a single x86 compiler, a single Z80 compiler and so on: where you tell the compiler that the target operating system is such-and-such, that just results in hints being passed to the backend ("use this variant of the RTL, use these linker directives..."). So to a very large extent, if there's a compiler targeting a chip you want then you might need to adapt the RTL (mostly written in Pascal) with a particular focus on some equivalent to the Crt unit (I'm not convinced that as it stands it is a good model for an 8-bit home micro).

So, let's look at how /I'd/ be approaching this, if I had the years.

The first thing is to decide on what in Ada terms would be called a workbench, with editor and debugger which assumed a host >16 bits. Borland, Hisoft et al. demonstrated that good work could be done on an 8-bit host, large chunks of what they did was written in assembler /but/ we are told that these days a compiler can produce code which is at least as good so why shouldn't we have a good development workbench to run on a home computer (assuming adequate RAM etc.)?

I like Pascal, although almost everybody agrees that the language as defined had flaws that weren't fixed until Wirth tried to replace it with Modula-2. Most subsequent languages have borrowed heavily from those when it comes to type checking etc.

Use a token such as := for assignment, and one such as == for comparison. In isolation = is always a syntax error.

I admit the importance of established idioms. That includes expression evaluation order, shortcuts such as +=, and accepted practice such as makefiles and decent preprocessor.

The preprocessor should be sufficiently capable that it can implement contentious features such as += without their having to be in the core language. I like the Smalltalk keyword syntax, and it might be possible to use this in conjunction with some sort of list notation to avoid a whole lot of variadic crap.

String handling is important. The zeroeth element of a string should be reserved, and -ve elements should indicate the length format, codepage and so on.

No automatic type conversion: that should be controlled by a trait system or by overloaded assignment. I'd like to see more overloadable operators, but am unsure whether it can be done in a small compiler (it implies feedback from the parser to the lexer, and this would be particularly problematic working across compilation units).

I appreciate objects, the try-finally construct, exception handling and so on, but if these aren't being used in a call chain they should not result in housekeeping overhead. This might make support of precompiled units difficult, particularly if pointers to functions/procedures are supported as callbacks.

FPC's dynamic arrays are great, but I'd rather see a common underlying mechanism for reference-counted (dynamic) arrays and (long, UTF-8) strings. And if reference counting works for those, it should also work for objects- at least at the user level.

Since classical Pascal (etc.) didn't support multiple compilation units, my preferred Pascal-like language would have separate interface and implementation files (like the TopSpeed compilers). In part this is because of the way that copyright etc. laws are moving, but also because it would be desirable to be able to lock-down at the OS level files containing definitions or complex hacks (e.g. the sort of thing that a systems programmer needs to do to get reference counting to work reliably).

I don't know how close to the immediacy of BASIC anything other than persistent environments (Smalltalk, Lisp, Forth) can get. My experience with the extremely good debugging on e.g. the RPi Pico (RP2040, Cortex M0) make me wonder whether that matters... what it can't do is stop at a breakpoint, allow the user to change the next line and then continue without missing a beat (again, xref to Mystic Pascal here since that at least attempted to). It /might/ be possible to arrange for a Pascal-like language to do that if compiled on the target system, it would be much more difficult if the target and host were distinct (OT but slightly relevant: there used to be a mechanism by which the Linux kernel could be patched while running).

I'm running out of steam. Hopefully that's given you something to think about, and plenty for Sven to complain about :-)

Added: xref to an earlier thread https://forum.lazarus.freepascal.org/index.php/topic,53139.msg392670.html although that was in the context of people who proposed "improvements" to Pascal, which I am not.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 19, 2022, 04:40:01 am
Thank you.

I will have to post a longer reply tomorrow as I just spent over 3 hours installing and uninstalling postscript viewers until I found one that would display more than the first page of those .ps files.

I finally used Postscript Viewer (https://postscript-viewer.software.informer.com/) from Software Informer if anyone else using Windows 10 has this problem. It converts to a .pdf file and then opens in your default PDF viewer (possibly Microsoft Edge).
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: sketch on August 19, 2022, 08:27:16 am
I finally used Postscript Viewer (https://postscript-viewer.software.informer.com/) from Software Informer if anyone else using Windows 10 has this problem. It converts to a .pdf file and then opens in your default PDF viewer (possibly Microsoft Edge).

Never heard of Postscript Viewer.  On Windows or Linux, I'd use Evince or GSview (Ghostscript) since they are well known, and I've used them.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 19, 2022, 08:54:36 am
There'd probably be a Oxygene mode before that... ::)

I've never looked at in an any detail. Are you suggesting that Turbo Rascal might actually be an Oxygene rather than a Turbo Pascal derivative?

No, I'm saying that Oxygene might be the more interesting Pascal dialect when compared to Turbo Rascal (at least for me it definitely is) and thus it might be more interesting to support that (if any).

@PascalDragon: I've seen suggestion that RD isn't as good as some other approaches at error reporting and recovery. Do you have any comment?

There are certain places where our error reporting could be better though whether they're due to the recursive descent parser or not I can't say right now. And recovery needs to be taken into account at each location and so depending on the error it's a bit hit and miss. Don't know whether a non-RD parser would really fare better there however...

Since classical Pascal (etc.) didn't support multiple compilation units, my preferred Pascal-like language would have separate interface and implementation files (like the TopSpeed compilers). In part this is because of the way that copyright etc. laws are moving, but also because it would be desirable to be able to lock-down at the OS level files containing definitions or complex hacks (e.g. the sort of thing that a systems programmer needs to do to get reference counting to work reliably).

You might get your wish once ISO Extended Pascal modules are supported as that suggests (though not requires) separate files (though nothing is set in stone there yet as work on that hasn't even remotely started yet). What I don't get however is your last part: what do you mean with “lock-down at the OS level”?

I'm running out of steam. Hopefully that's given you something to think about, and plenty for Sven to complain about :-)

Too busy to really read and digest all that and thus I only picked out the cherries. :P
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 19, 2022, 09:31:36 am
There are certain places where our error reporting could be better though whether they're due to the recursive descent parser or not I can't say right now. And recovery needs to be taken into account at each location and so depending on the error it's a bit hit and miss. Don't know whether a non-RD parser would really fare better there however...

Thanks for that and please note that I definitely wasn't complaining. Noting that I'm only repeating suggestions I've read, my /suspicion/ is that it relates to a compiler's ability to report more than the first syntax error it encounters.

Quote
what do you mean with “lock-down at the OS level”?

Set a file read-only to less-privileged users.

Quote
Too busy to really read and digest all that and thus I only picked out the cherries. :P

You're welcome and I appreciate your comments.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 19, 2022, 09:52:03 am
There are certain places where our error reporting could be better though whether they're due to the recursive descent parser or not I can't say right now. And recovery needs to be taken into account at each location and so depending on the error it's a bit hit and miss. Don't know whether a non-RD parser would really fare better there however...

Thanks for that and please note that I definitely wasn't complaining. Noting that I'm only repeating suggestions I've read, my /suspicion/ is that it relates to a compiler's ability to report more than the first syntax error it encounters.

Well, we do try to develop the compiler in a way that more than one error is reported (e.g. if you compare with pas2js which more often than not does stop after the first error it finds), but some times recovery either isn't easy (especially when it's really the syntax that is off and not simply a missing identifier) or was not implemented correctly or at all.

Quote
what do you mean with “lock-down at the OS level”?

Set a file read-only to less-privileged users.

And what would be the advantage here to only set the implementation to read-only, but not the interface as well? And if you don't want users to manipulate that why not simply provide it binary only? If it's to allow the user to compile for a different platform then what stops them from simply copying all files to a location where it isn't read-only, do their intended changes and go their merry way?
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 19, 2022, 10:47:23 am
Quote
And what would be the advantage here to only set the implementation to read-only, but not the interface as well?

I was assuming they were: I /did/ say "lock-down at the OS level files containing definitions or complex hacks".

Quote
why not simply provide it binary only?

Depends on how the separate compilation works... /if/ it's done at all. Noting that my comments are in the general case rather than pertaining to any particular Pascal compiler or derivative, I suspect that never carrying around baggage relating to exception handling etc. implies that separate compilation would be to the AST level with actual code generation only after all calls into or out of a block had been resolved.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: PascalDragon on August 19, 2022, 02:09:36 pm
Quote
And what would be the advantage here to only set the implementation to read-only, but not the interface as well?

I was assuming they were: I /did/ say "lock-down at the OS level files containing definitions or complex hacks".

And then why the separation if both are locked down anyway?

Quote
why not simply provide it binary only?

Depends on how the separate compilation works... /if/ it's done at all. Noting that my comments are in the general case rather than pertaining to any particular Pascal compiler or derivative, I suspect that never carrying around baggage relating to exception handling etc. implies that separate compilation would be to the AST level with actual code generation only after all calls into or out of a block had been resolved.

What has how the separate compilation is done to do with whether or not you'd provide a binary only module?
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 19, 2022, 02:32:48 pm
And then why the separation if both are locked down anyway?

Because in different contexts you might want to lock different files against unauthorised modification.

For example, interface descriptions once finalised should probably be locked down. For anotherexample, an implementation which contains code which should not be modified by a junior team member should probably be locked down.

In addition, taking into account Google vs Oracle, we're heading for a situation where it might be desirable to hold factual definitions and detailed implementations in different files, because different standards of copyright apply. I don't think anybody's yet tried to apply different copyrights or licenses to different portions of the same file: the situation's bad enough as it is.

Quote
What has how the separate compilation is done to do with whether or not you'd provide a binary only module?

Because if separate compilation targetting assembler cannot express the semantics of the language, then you need to look at a higher-level representation: emitting a unit coded at the AST level, or not compiling to binary at all.

For example, the way that Stallman found that the Pastel language assumed the host was big enough to hold the entire AST in memory until every reference (e.g. to parameterised types) could be resolved. And I suspect that parameterised types are a distant ancestor of generics.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 20, 2022, 08:08:57 am
Due to a crash, I had to run chkdsk on my laptop which took over 12 hours (not sure how long as I stopped checking after that). I have only managed to get halfway through the first sample chapter because of this.

I did start to read the first "Writing an Interpreter in Object Pascal (https://www.objectpascalinterpreter.com/)" book though.

It isn't as detailed as it could be, as whenever the author chooses a particular option he doesn't always say what the alternatives are, just why he chose it. I suspect that those more experienced with Pascal wouldn't need this explained.

He does intend to describe how to include a REPL interactive shell for the BASIC-like language though, allowing programs to be stopped, edited and then continued as many of the 8-bit machines did.

So far, the language syntax seems more like a mix of BASIC and Pascal. Maybe that will change in version 2, which is described in book 3 (not yet published).

Everything is being created from scratch using Delphi or FreePascal, without the use of Yacc or ANTLR. I'm not sure if that's a good thing.

I get the feeling that the entire process will stretch to four books (it was originally only supposed to have been two).
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 20, 2022, 09:14:14 am
So far, the language syntax seems more like a mix of BASIC and Pascal. Maybe that will change in version 2, which is described in book 3 (not yet published).

Yes, because the objective is to illustrate a book rather than write a decent software tool.

But you must understand that tools like YACC weren't available to Wirth or (realistically) Borland in the early days, and you must understand that the compilation technique they use is not necessarily the best for a Pascal-like language. Wirth's early compilers predated recursive descent, and you /really/ don't want to try to make sense of them.

And I'm unconvinced that you can do a REPL for Pascal, since blocks have to be closed before they are meaningful.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 20, 2022, 06:12:49 pm
I haven't reached the point where he explains the REPL in detail, but the source code for all three books is available in his Rhodus Interpreter Repositories (https://github.com/ObjectPascalInterpreter) if you're curious.

The language is named after his dog Rhody.

I imagine it works along the lines of the old BASIC interpreters in that it only does a cursory syntax check when entering each line of code then does a more thorough check at runtime.

BASIC included blocks with opening and closing keywords, but many interpreters would allow you to run an incomplete program and only complain when/if they reached the part with the missing code.

This was a blessing and a curse if you had a program with many conditional branches as you might not find an error (such as having next x and next y in the wrong order) until days after you thought you'd finished a section of code in a long program.

If your version of BASIC didn't support procedures, you often had to add a temporary GOTO or GOSUB to test new code near the end of a program. In the 80s there wasn't much support for separating code into multiple files either.
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: MarkMLl on August 20, 2022, 06:25:32 pm
I haven't reached the point where he explains the REPL in detail, but the source code for all three books is available in his Rhodus Interpreter Repositories (https://github.com/ObjectPascalInterpreter) if you're curious.

I'm not. I've got enough problems of my own, have already seen more than my share of dodgy language implementations, and have already said that my knowledge of the art leads me to believe that a half-decent compiler (or compiler/interpreter etc.) can be more easily written for an ALGOL-derivative than for a BASIC- or FORTRAN-derivative.

MarkMLl
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: y.ivanov on August 21, 2022, 12:41:24 pm
I haven't reached the point where he explains the REPL in detail, but the source code for all three books is available in his Rhodus Interpreter Repositories (https://github.com/ObjectPascalInterpreter) if you're curious.
*snip*
And what REPL has to do in you multi-target bulder idea?

Later BASICs (e.g. QBasic) are structured and much closer to Pascal. Home micro BASICs aren't and thus suitable for being REPL-ed.

Just learn how to use Plex and Pyacc (https://wiki.freepascal.org/Plex_and_Pyacc). They're off the shelf solutions. It will take much less time.   

 
Title: Re: Can FreePascal be used to create an internal or embedded DSL?
Post by: Wysardry on August 21, 2022, 07:52:44 pm
A REPL would be useful for testing programs on the modern front end. The next best thing would be a modern interpreter.

Second generation structured BASIC dialects were available with a REPL on some machines. BBC BASIC is probably the most well known, but IS-BASIC on the Enterprise was an implementation of ANSI Full BASIC.

The books I have so far on writing an interpreter include a REPL as part of the language construction. I doubt it would be a good idea to just skip those parts.

I've never created a language before, so I would need a guide for at least the first attempt. These were the only recent books I could find aimed at using Pascal.

I would much rather modify an existing language if possible. I'm not aiming for anything totally new after all.
TinyPortal © 2005-2018