And I've been doing the same for about 45 years. Need I say more?
As it's unlikely I have 45 years left in me, telling me (and anyone else reading this) which parts were a waste of time would be useful.
:-) I've been trying to. Let's see if I can knock something out while waiting for my supper...
OK, in no particular order. As an engineer, I didn't have any sort of "formal background" in compilers. I still think that a lot of that stuff is garbage: if a syntax is difficult to parse then it would be better to select an unambiguous and efficient syntax than spending years working on new algorithms. But there's stuff in there which would have been enormously helpful when I was doing my MPhil in the '80s, provided I didn't get enmired in the crap.
When I started off, people weren't programming small computers in high-level languages (other than BASIC, and that was very much pre-structured). As such I had a better grounding in assembler, followed fairly promptly by the innards of Forth, Smalltalk and- slightly later- systems such as UCSD including their applicability to mainframes.
In terms of HLL, Forth, Smalltalk and APL are deadends because while they have interesting execution models their gross departure from expected syntax etc. puts them out on a limb. I'm not saying they're without merit, but I am saying that things like evaluation order are now so deeply ingrained in basic maths that ignoring them is perilous.
Reference here: the first three chapters of
https://web.engr.oregonstate.edu/~budd/Books/leda/ are available for download and are worth reading carefully.
In the 1980s, when people were just about starting to take HLLs on micros seriously, I got a reprint of the classic paper on Meta-2 (I quoted an Alan Kay comment on this). I used that for a brief course for staff members at the university I was working at, and have been using a derivative for various tasks ever since. See Wp.
Much more recently, I came across Tree Meta (again, see Wp). That is much more difficult to get to grips with, but it has the potential of being able to specify rules to optimise the intermediate AST as part of the syntax definition. There was an implementation on GitHub which vanished, I got a lead on an ex-HP guy who'd used it in an instrument but he lost everything in one of the California fires :-(
Both of those may be used to generate a compiler using recursive descent, which AIUI is the technique used for FPC.
RD isn't particularly sexy and can barf when fed certain types of syntax. But it's pretty much adequate for Pascal or classic C, and doesn't need a higher degree to understand. And being simple it's compact, and is probably the technique that Borland used to get their early compilers to run natively on CP/M-80.
@PascalDragon: I've seen suggestion that RD isn't as good as some other approaches at error reporting and recovery. Do you have any comment?
To establish terminology, lets say that typical BASICs used tokenised-interpretive execution: each BASIC keyword was typically tokenised to a single byte when entered and the sequence of tokens, variable names etc. was interpreted at runtime. This is fairly concise but grossly inefficient.
Both Metas use a bytecode-interpretive arrangement as does UCSD Pascal, Java etc. In this, the high-level language is compiled beyond the AST stage and saved as a sequence of bytecodes, which may be subsequently interpreted or translated to e.g. assembler source for conversion to an executable binary.
Forth uses a distinct threaded-interpreted scheme, which was originally tightly coupled to the PDP system it was written for. I don't know how well this works on other systems, but the fact that a Forth implementation was used by Sun, IBM, Apple and the "One Laptop Per Child" project suggests that it's got it's good points: it might still be usable as some sort of intermediate notation, and roughly approximates the AST of a compiled language.
If you look at the P.J.Brown book that I cited, you will see that the BASIC-like language described has to gone to a lot of trouble to handle special cases that in the case of an ALGOL-derivative like Pascal "just work". By analogy, when Waychoff wrote an ALGOL compiler on ALGOL a whole lot of things "just worked" that Grau et al. had sweated blood over.
If you are looking at an implementation such as FPC, by and large there is a single x86 compiler, a single Z80 compiler and so on: where you tell the compiler that the target operating system is such-and-such, that just results in hints being passed to the backend ("use this variant of the RTL, use these linker directives..."). So to a very large extent, if there's a compiler targeting a chip you want then you might need to adapt the RTL (mostly written in Pascal) with a particular focus on some equivalent to the Crt unit (I'm not convinced that as it stands it is a good model for an 8-bit home micro).
So, let's look at how /I'd/ be approaching this, if I had the years.
The first thing is to decide on what in Ada terms would be called a workbench, with editor and debugger which assumed a host >16 bits. Borland, Hisoft et al. demonstrated that good work could be done on an 8-bit host, large chunks of what they did was written in assembler /but/ we are told that these days a compiler can produce code which is at least as good so why shouldn't we have a good development workbench to run on a home computer (assuming adequate RAM etc.)?
I like Pascal, although almost everybody agrees that the language as defined had flaws that weren't fixed until Wirth tried to replace it with Modula-2. Most subsequent languages have borrowed heavily from those when it comes to type checking etc.
Use a token such as := for assignment, and one such as == for comparison. In isolation = is always a syntax error.
I admit the importance of established idioms. That includes expression evaluation order, shortcuts such as +=, and accepted practice such as makefiles and decent preprocessor.
The preprocessor should be sufficiently capable that it can implement contentious features such as += without their having to be in the core language. I like the Smalltalk keyword syntax, and it might be possible to use this in conjunction with some sort of list notation to avoid a whole lot of variadic crap.
String handling is important. The zeroeth element of a string should be reserved, and -ve elements should indicate the length format, codepage and so on.
No automatic type conversion: that should be controlled by a trait system or by overloaded assignment. I'd like to see more overloadable operators, but am unsure whether it can be done in a small compiler (it implies feedback from the parser to the lexer, and this would be particularly problematic working across compilation units).
I appreciate objects, the try-finally construct, exception handling and so on, but if these aren't being used in a call chain they should not result in housekeeping overhead. This might make support of precompiled units difficult, particularly if pointers to functions/procedures are supported as callbacks.
FPC's dynamic arrays are great, but I'd rather see a common underlying mechanism for reference-counted (dynamic) arrays and (long, UTF-8) strings. And if reference counting works for those, it should also work for objects- at least at the user level.
Since classical Pascal (etc.) didn't support multiple compilation units, my preferred Pascal-like language would have separate interface and implementation files (like the TopSpeed compilers). In part this is because of the way that copyright etc. laws are moving, but also because it would be desirable to be able to lock-down at the OS level files containing definitions or complex hacks (e.g. the sort of thing that a systems programmer needs to do to get reference counting to work reliably).
I don't know how close to the immediacy of BASIC anything other than persistent environments (Smalltalk, Lisp, Forth) can get. My experience with the extremely good debugging on e.g. the RPi Pico (RP2040, Cortex M0) make me wonder whether that matters... what it can't do is stop at a breakpoint, allow the user to change the next line and then continue without missing a beat (again, xref to Mystic Pascal here since that at least attempted to). It /might/ be possible to arrange for a Pascal-like language to do that if compiled on the target system, it would be much more difficult if the target and host were distinct (OT but slightly relevant: there used to be a mechanism by which the Linux kernel could be patched while running).
I'm running out of steam. Hopefully that's given you something to think about, and plenty for Sven to complain about :-)
Added: xref to an earlier thread
https://forum.lazarus.freepascal.org/index.php/topic,53139.msg392670.html although that was in the context of people who proposed "improvements" to Pascal, which I am not.
MarkMLl