Was the FPC cross-compiler built from scratch?

[anyone]

I should have posted in another 120-day old thread but forum software asks me to start a new topic.

After a few search, I understand that MinGW has nothing to do with FPC though I understand that:

MinGW is a native Windows port of the GNU Compiler Collection (GCC), with freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. Although (currently) offering only a 32-bit compiler suite, all of MinGW's software will execute on the 64bit Windows platforms.

I have no gut to download the source code of FPC to examine myself, because I anticipate its complexities.

So back to my question: Since I am new to Lazarus (not new to FPC though) , I am amazed at the power of the cross-platform compiler.
Was FPC built from scratch? Do the developers of FPC compiler know in-depth about i386 (and x86) Assembly language?
This is no joke. 16-bit ASM is already very complex, let alone using 32-bit and 64-bit CPU registers and memory addressing.

I can see how massive the scope of effort and skill needed to develop a compiler. And if I understand correctly, there are front-end and back-end compiler in which the latter convert to machine opcode.

If it is true that FPC was built from scratch (since it is dated back to 1993), then I believe it is a collaborative work of a team, not a person.

I write ordinary code these days in high-level programming language, so I cannot imagine if the developer of FPC develops end-user apps using FP himself (if any)?

[Thaddy]

FPC compilers are always build with the previous release version.
IOW bootstapped with the version just previous to the latest.
E.g. 3.2.0 is made with 3.0.4.

[marcov]

FPC bootstrapped itself originally on dos from Turbo Pascal. All other targets are crosscompiled at some point. So TP produced a 16-bit protected mode FPC program that would generate 32-bit code. Then that a some point compiled itself.

Note that that bootstrap point is mostly pre 2000, though after 2000 afaik still some 1.0.x debugging was done with TP with specially cut down compilers to make it fit. I never really did anything with TP, and I started with FPC in april 1997

FPC 2.0 and later don't work with TP anymore as these versions use Delphi classes, and bootstrap (originally) with FPC 1.0.x

[MarkMLl]

Reading the question in a slightly different way, it's never used external lexer/parser libraries such as Lex and YACC.

And my understanding is that it's Florian's Pascal Compiler, i.e. it was /originally/ a one-man project.

MarkMLl

[marcov]

Note that afaik no production C compilers actually use LALR parser generators like yacc/lex. Nearly always they use handcrafted recursive descent parsers for speed and quality error message generation.

... like FPC did from the start

[anyone]

Thanks @Thaddy and @marcov for the quick explanation.

FPC bootstrapped itself originally on dos from Turbo Pascal. All other targets are crosscompiled at some point. So TP produced a 16-bit protected mode FPC program that would generate 32-bit code. Then that a some point compiled itself.

Note that that bootstrap point is mostly pre 2000, though after 2000 afaik still some 1.0.x debugging was done with TP with specially cut down compilers to make it fit. I never really did anything with TP, and I started with FPC in april 1997

FPC 2.0 and later don't work with TP anymore as these versions use Delphi classes, and bootstrap (originally) with FPC 1.0.x

This is very interesting. But how could the FPC after bootstrapped itself originally on DOS from TP supports Win32 API later on? Maybe I do not understand how does "bootstrapping" work in production. Hmm...

Reading the question in a slightly different way, it's never used external lexer/parser libraries such as Lex and YACC.

And my understanding is that it's Florian's Pascal Compiler, i.e. it was /originally/ a one-man project.

MarkMLl

 

The parser or lexer are front-end of a compiler design, if I am not mistaken.

----

FPC did not originate from LLVM and GCC, which is cool!

[marcov]

This is very interesting. But how could the FPC after bootstrapped itself originally on DOS from TP supports Win32 API later on? Maybe I do not understand how does "bootstrapping" work in production. Hmm...

It didn't. It went to 32-bit dos first. Keep in mind that a EXE is just a file, a sequence of byte. You can generate any sequence if you work at it hard enough.

Though in reality the initial crosscompile is often to assembler source that is then assembled and linked on target.

Reading the question in a slightly different way, it's never used external lexer/parser libraries such as Lex and YACC.

And my understanding is that it's Florian's Pascal Compiler, i.e. it was /originally/ a one-man project.

MarkMLl

 

The parser or lexer are front-end of a compiler design, if I am not mistaken.

----

FPC did not originate from LLVM and GCC, which is cool!

[MarkMLl]

The parser or lexer are front-end of a compiler design, if I am not mistaken.

Correct. But FPC doesn't use any of the "must-have" tools i.e. YACC etc., and is hardcoded rather than starting off with a syntax description file (I've mentioned Meta-2 and Tree Meta, which do start off with a human-readable syntax description, before).

MarkMLl

[marcov]

The parser or lexer are front-end of a compiler design, if I am not mistaken.

NOT my quote.

[anyone]

Since we are talking about compiler and linker here, actually I have a goal that is to create my own Win32 compiler in its most primitive form, i.e. supports only limited set of commands, and there is no conditional evaluation, no iterative construct, no user-defined variable etc.

I am not sure if any of you heard LTA compiler(my DOS compiler) which generates .COM binary executable supporting 47-commands.
But DOS is DOS, I want to port it to Win32 supporting PE file format.

I am planning to use Free Pascal to write this simple Win32 compiler, and I would expect some hard-coded Assembly instructions.

This going-to-be Win32 compiler of mine is not meant to compete with FPC in any way by any means. No, it is far inferior than FPC. My goal is to learn portable executable format while having a sense of achievement.

Currently, it would support three functions:
WAITKEY (ReadConsoleA, kernel32.dll)
PRINT (WriteConsoleA, GetStdHandle, kernel32.dll)
MSGBOX (MessageBoxA & wsprintfA, user32.dll)
and EXIT (ExitProcess, kernel32.dll)

I know some basic structure of PE file format like Code / Data section and Import Table. RVA, VA and RAW. Also header setting like File Alignment (512 bytes at least).

But I need motivation. Will anyone of you support my idea?

Questions:
-Is there any PE file format spec in Pascal typed record? (winnt.h is in C)
(I know there is one member here who wrote PEDump)

- Is it okay to talk about creating a compiler as hobby project on this forum?
(Look, my proposed 'compiler' is not even near a compiler by standard definition)

 

[MarkMLl]

The parser or lexer are front-end of a compiler design, if I am not mistaken.

NOT my quote.

(re-reads several messages) sorry Marco, but in that case please sort out your formatting.

MarkMLl

[MarkMLl]

-Is there any PE file format spec in Pascal typed record? (winnt.h is in C)

Irrelevant. Generate assembler and run it through an assemble/link stage like almost everybody else does these days, or generate .obj files and use the linker. Or generate an intermediate format that tell the OS to associate an interpreter with it (based on the extension).

MarkMLl

[PascalDragon]

So back to my question: Since I am new to Lazarus (not new to FPC though) , I am amazed at the power of the cross-platform compiler.
Was FPC built from scratch? Do the developers of FPC compiler know in-depth about i386 (and x86) Assembly language?
This is no joke. 16-bit ASM is already very complex, let alone using 32-bit and 64-bit CPU registers and memory addressing.

Yes, especially those developers that work with the backends know quite a bit about assembly language. For example I myself learned M68k assembly to revive the compiler's Motorola 68000 backend a few years back. But a large part of the compiler is also the parser where no real knowledge about assembly is required.

By the way: did you know that FPC also supports 16-bit x86 with its i8086 target.

If it is true that FPC was built from scratch (since it is dated back to 1993), then I believe it is a collaborative work of a team, not a person.

As others have hinted at, it started alone with Florian and then the team grew as the compiler itself grew.

I write ordinary code these days in high-level programming language, so I cannot imagine if the developer of FPC develops end-user apps using FP himself (if any)?

It's rather comfortable to write a compiler in the same language that it compiles.

This going-to-be Win32 compiler of mine is not meant to compete with FPC in any way by any means. No, it is far inferior than FPC. My goal is to learn portable executable format while having a sense of achievement.

Wanting to learn things is never wrong.

Currently, it would support three functions:
WAITKEY (ReadConsoleA, kernel32.dll)
PRINT (WriteConsoleA, GetStdHandle, kernel32.dll)
MSGBOX (MessageBoxA & wsprintfA, user32.dll)
and EXIT (ExitProcess, kernel32.dll)

Isn't that four functions?

I know some basic structure of PE file format like Code / Data section and Import Table. RVA, VA and RAW. Also header setting like File Alignment (512 bytes at least).

As MarkMLI said, easiest would be to simply use an existing assembler and linker (for example GNU AS and LD, Clang (which can be used for sole assembling and linking as well) or Microsoft's MASM and LINK). This way you can concentrate on generating the correct assembly code without having to worry about the other details. Later on you can - if you want - create internal assemblers and linkers. For FPC it's the same: it uses various external assemblers and linkers and only some targets have internal assemblers (e.g. x86, Z80) and linkers (all Windows targets, Z80).

Questions:
-Is there any PE file format spec in Pascal typed record? (winnt.h is in C)
(I know there is one member here who wrote PEDump)

I do have a unit with that if you really want to dive into PE/COFF.

- Is it okay to talk about creating a compiler as hobby project on this forum?
(Look, my proposed 'compiler' is not even near a compiler by standard definition)

Of course. First of it's a Pascal program as you said, so we'd definitely help if you had Pascal specific problems and we do have a Other area after all where you can ask specific questions that are related to compiler development.

[anyone]

Thanks @MarkMLl and @PascalDragon for going extra miles in answering my questions and satisfying my curiosity in the implementation of modern compiler.

@PascalDragon, the last function (Exit) is not considered a "function" from user perspective because it is for program to terminate normally. 

Since assembler and linker like MASM and LINK can take care of everything, I would rather code it in FASM and let it handle the rest, although that is not the reason why I want to propose my own "compiler" in the first place.

I have seen open-source like Golang implements its own PE/COFF file output (like generating checksum and timestamp, give a name to linker name and version, etc) but I am not sure if it uses assembler and linker.

After reading everyone's comments in this thread, I understand far better as to how FPC and the likes work internally. Thank you @PascalDragon, you can keep the PE/COFF Pascal unit with you first.

[MarkMLl]

As MarkMLI said, easiest would be to simply use an existing assembler and linker (for example GNU AS and LD, Clang (which can be used for sole assembling and linking as well) or Microsoft's MASM and LINK). This way you can concentrate on generating the correct assembly code without having to worry about the other details. Later on you can - if you want - create internal assemblers and linkers. For FPC it's the same: it uses various external assemblers and linkers and only some targets have internal assemblers (e.g. x86, Z80) and linkers (all Windows targets, Z80).

There's also the aspect that sooner operating systems are likely to start scrutinising programs and refusing to run anything that hasn't been generated by a known toolchain. I'm not necessarily saying that every program would have to be cryptographically signed, but it's likely that unknown linkers etc. would be unable to "bless" a program into an executable state.

MarkMLl