My simple Win32 Compiler

[FlierMate]

Introducing my simple Win32 compiler:

https://dev.to/bookhanming1/i-ve-had-fun-learning-to-build-my-own-back-end-compiler-1o86

https://dev.to/bookhanming1/i-have-written-a-basic-compiler-what-next-2nam

Satay Compiler supports two commands:

ReadLine
WriteLine

You can use Satay IDE (SATAYIDE.exe) to write code and compile from within, or use any text editor and then run the command-line tool (SATAY.exe):

Code: [Select]

WriteLine Hello World!
WriteLine Press Enter to quit...
ReadLine
Package includes:

Code: [Select]

08/01/2021  07:57 PM           146,870 HEXDUMP.exe
21/01/2021  10:07 PM            96,768 SATAY.exe
25/01/2021  07:30 AM           349,696 SATAYIDE.exe
25/01/2021  07:30 AM             8,798 SATAYIDE.pas
25/01/2021  07:31 AM                65 TEST.txt
25/01/2021  07:36 AM               167 TEST1.txt
21/01/2021  10:07 PM                28 TEST2.txt
              8 File(s)        602,392 bytes

[FlierMate]

I was 'anyone', but that user name can be confusing....

[MarkMLl]

Please could you explain a bit more what you are trying to do here.

MarkMLl

[FlierMate]

Please could you explain a bit more what you are trying to do here.

MarkMLl

Hi, it all started from my conversation in this thread (involving my old username and your goodself, too):
https://forum.lazarus.freepascal.org/index.php/topic,51683.0.html

There are quite a number of expert in PE (especially FPC developer themselves) on this forum who might give constructive feedback to me for my "prototype of reassembler" because I am still learning the reverse-engineering of portable executable.

And since my project is using Pascal, certainly I would post and share my research work on here as well.

As someone who is curious in how operating system works internally, I have great delight to keep improving this project, and I hope you can share your views on it.

[MarkMLl]

Right. So what is a "reassembler"? The name would suggest that it's something that patches a binary.

Your description of a simple language reminds me of pilot... and you /really/ don't want to read ESR's comments on that :-)

MarkMLl

[440bx]

Hi, it all started from my conversation in this thread

From what you posted in that thread, I guess that what you are calling a "reassembler" is actually a bare bones, very simple compiler.  Is this correct ?

As someone who is curious in how operating system works internally,

Just as a comment, there is a lot more (a whole lot more!) to O/S internals than the executable format (PE in the case of Windows.)

Since you are starting, I'd suggest you start by looking at how others have done it.  A particularly interesting project is SuperPascal from Per Brinch Hansen.  It's a hand coded Pascal compiler (produces bytecode), overall fairly easy to understand and, there is a version on github that compiles with FPC which means, you can run it under the debugger in Lazarus and see how it works. 

Once you understand that, which will take some work but, it's not too hard then, you'll be ready to get more ambitious and generate actual machine code (or Assembly code first.) 

Another compiler you may want to look at is one from Vitaly Tereshkov (I hope I got his name right) which you can find at https://forum.lazarus.freepascal.org/index.php/topic,49082.msg354754.html#msg354754  that one generates machine code, is self hosting and can be compiled with FPC as well which means you can see how it works by debugging it.

HTH.

[MarkMLl]

Considering what 440bx has said, I'd add that the Wikipedia page on Meta-2 is extremely useful, including its reference PDF.

Following on from that, http://bitsavers.informatik.uni-stuttgart.de/components/intel/_dataBooks/1983_iAPX_286_Operating_System_Writers_Guide.pdf is a description of how Intel expected an operating system for its chips to be written. Some of it is outdated since it refers to technology elements that Intel chose to deprecate, but I still consider it to be an extremely useful resource.

MarkMLl

[FlierMate]

From what you posted in that thread, I guess that what you are calling a "reassembler" is actually a bare bones, very simple compiler.  Is this correct ?

Yes, that is what I meant. I pick the name 'reassembler' because the word 'compiler' is far more powerful and yet mine is just a simple one.

Just as a comment, there is a lot more (a whole lot more!) to O/S internals than the executable format (PE in the case of Windows.)

I learn very slow, it took more than one year for me to build this simple 'compiler' which does basically nothing.

Since you are starting, I'd suggest you start by looking at how others have done it.  A particularly interesting project is SuperPascal from Per Brinch Hansen.  It's a hand coded Pascal compiler (produces bytecode), overall fairly easy to understand and, there is a version on github that compiles with FPC which means, you can run it under the debugger in Lazarus and see how it works. 

Once you understand that, which will take some work but, it's not too hard then, you'll be ready to get more ambitious and generate actual machine code (or Assembly code first.) 

Another compiler you may want to look at is one from Vitaly Tereshkov (I hope I got his name right) which you can find at https://forum.lazarus.freepascal.org/index.php/topic,49082.msg354754.html#msg354754  that one generates machine code, is self hosting and can be compiled with FPC as well which means you can see how it works by debugging it.

The name of author is Vasiliy Tereshkov. Terrific skills! Where did he or she learn that?
(Although I see he / she also hard-coded the DOS stub)

That is a great achievement, his /her compiler is already matured.

If mine was compared to his /hers, I was like living in 2000 B.C trying to learn science, but he / she would be a time-traveler from few thousands years beyond.

I am speechless...

HTH.

It helps. Thank you.

Right. So what is a "reassembler"? The name would suggest that it's something that patches a binary.

Your description of a simple language reminds me of pilot... and you /really/ don't want to read ESR's comments on that :-)

MarkMLl

In a sense, yes, it is a 'reassembler', but instead of patching a binary program, it patches the only one binary program that has been hard-coded in the 'reassembler' itself.... if I am allowed to use the word 'compiler', I would be more than happy.

Thank you MarkMLI for bringing up this topic of discussion.

[440bx]

I learn very slow, it took more than one year for me to build this simple 'compiler' which does basically nothing.

There are some tutorials online on how to build a compiler.  Jack Crenshaw's is one of the better known ones and it is easy to follow and understand.  Another example worth looking at is JdeHaan's (an FPC forum member) Gear language implemented in FPC.  It's an interpreter but the scanning and parsing parts of it apply to writing a compiler.  It's well documented and fairly easy to understand.  He did an excellent job.  You can find his implementation at https://github.com/jdehaan2014/GearLanguage

IF (note the big IF) you can find it at a reasonable price, the book : Per Brinch Hansen on Pascal Compilers might very well be the best introductory text on compilers ever written but it's out of print and used copies are usually sold for exorbitant prices (don't pay more than $50.00 and a reasonable shipping fee)

The name of author is Vasiliy Tereshkov. Terrific skills! Where did he or she learn that?

Thank you for correcting the name.  FYI, the author is a "he".  As far as where he learned, I surmise he did it like most everyone else did, by reading books and looking at how other people went about implementing a compiler.

It helps. Thank you.

Glad it was helpful and, you're welcome.

HTH.

[MarkMLl]

In a sense, yes, it is a 'reassembler', but instead of patching a binary program, it patches the only one binary program that has been hard-coded in the 'reassembler' itself.... if I am allowed to use the word 'compiler', I would be more than happy.

Thank you MarkMLI for bringing up this topic of discussion.

Please stick to the standard terminology: it's a compiler, and nobody will kick you for doing something crude as a learning exercise (unlike, dare I say it, Python :-)

"Reassembler" is a problematic name, since it really implies something like taking a binary program, patching it, and reassembling it with checksums and signatures as appropriate. Such a thing would be valuable, you're not doing it, so the name is best avoided.

And you're welcome :-)

It can be very difficult for "an outsider"- even an engineer like me- to get into this sort of thing, since so much blatant bullshit is written about it by the computer science priesthood.

The fact is that most of what has been written about compilers focusses on taking a difficult syntax and making sure that it can be compiled without requiring inordinate resources. I have very little time for that, and prefer the philosophy that (by analogy) if you're writing obscure and difficult English it's down to you to improve your presentation, rather than expecting every reader to "get educated".

Most "real" computer languages can be compiled efficiently using a technique called recursive descent, which you will find discussed in that Wp article I pointed you at earlier.

MarkMLl

[mika]

I have included almost everything in the MAKECON.zip, including the CONSOLE.asm (Assembly language), because its output PE is useful for my learning purpose (reverse-engineering).

You are at very beginning. My suggestion:
Search internet for TExeHeader. Replace your ConSec1 and ConSec3 with proper records. Read some documentation about the relevant topics. Play around.
Looking at others code is nice and what not, but true learning is when you put your own code together. Good luck.

[FlierMate]

You are at very beginning. My suggestion:
Search internet for TExeHeader. Replace your ConSec1 and ConSec3 with proper records. Read some documentation about the relevant topics. Play around.
Looking at others code is nice and what not, but true learning is when you put your own code together. Good luck.

Thank you for your feedback. Yes, replacing the DOS Stub and PE COFF header with proper typed record will be in next stage. You said "you are at very beginning" means a lot to me. I will learn more and write better code in the future.   

[MarkMLl]

Point about record types etc. in this context. There's all very well for output, but for input eventually you will be looking at things character-by-character.

I don't know how far you've got in your reading, but generally speaking the sequence runs something like this:

* The lexer reads character-by-character, and assembles individual lexemes (identifiers, numbers, quoted strings and so on). It keeps a record of the character sequence it's processed and what hasn't yet been processed and stops when it finds something unexpected, so if necessary an error message can identify the point in a line where things went wrong.

* The parser knows the type of lexeme it's expecting and what to do in response.

* Variable names etc. go into a symbol table when first parsed, and might accumulate properties which are referred to later.

* When the parser (or a separate code generation stage) outputs code, it might refer back to the symbol table (e.g. to decide whether a variable is to be manipulated as a byte or word).

In a simple compiler, the parser will output code (often assembler source) on the fly. In a more complex one it will build a tree representing an expression, and that tree will be optimised and output.

Even if that lot is beyond you for the moment, just remember that it is a mistake to try to apply high-level text processing commands to the input.

MarkMLl

[avra]

This might interest you: https://wiki.freepascal.org/Make_your_own_compiler,_interpreter,_parser,_or_expression_analyzer

[FlierMate]

I'm sorry, all. I think the Pascal source code itself of my basic compiler was not professional, I have disowned and removed the source code from everywhere.

However, I still keep the EXE, and I introduce it on DEV community which makes the "basic compiler" looks more appealing.

I apologize again if I upset anyone of you.