Lazarus

Programming => General => Topic started by: MarkMLl on July 31, 2021, 12:19:49 pm

Title: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on July 31, 2021, 12:19:49 pm
Is there a way that works with as-installed FPC on Linux of preloading a big block of raw text into a string? For the purpose of the question please assume that this is a multi-line script which includes arbitrary quote characters etc.

If this were Perl or C++ I'd use a here-document. If using Lazarus I have in the past put it into a TStringList component, but the editor is primitive and modal so it's impossible to consult this while working elsewhere in the program.

I've reverted to defining it as a multiline constant, but either have to use the double-' escape or define a substitute and preprocess the text... the latter works out better but is still hardly ideal:

Code: Pascal  [Select][+][-]
  1. const
  2.   qq= LineEnding;                       (* Prefix is same size as 8-spaces tab  *)
  3.  
  4.   syntax='.QUOTE `'
  5. + qq + '`MetaScriptSyntax` <<EOF'
  6. + qq + '.SYNTAX MetaScriptSyntax'
  7. + qq + ''
  8. + qq + '(* This is the master representation of the syntax of the embedded      *)'
  9. + qq + '(* script compiler.                                                     *)'
  10. ...
  11.  

MarkMLl

Title: Re: Old chestnut: including a big block of raw text into a program
Post by: marcov on July 31, 2021, 12:42:05 pm
Simply include the textfile as a resource, and load it from the resource? Afaik resources work with Linux too.

Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on July 31, 2021, 01:35:18 pm
Simply include the textfile as a resource, and load it from the resource? Afaik resources work with Linux too.

https://wiki.freepascal.org/Lazarus_Resources says that doing so needs windres or GoRC, neither of which comes as standard. The current manual concurs, and the examples still have to be quoted.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: Bart on July 31, 2021, 01:50:37 pm
Or LazRes (using old style *.lrs resources)...

Bart
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on July 31, 2021, 02:21:59 pm
Or LazRes (using old style *.lrs resources)...

Except that's not part of FPC.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: Gustavo 'Gus' Carreno on July 31, 2021, 02:25:49 pm
Hey Mark,

Is there a way that works with as-installed FPC on Linux of preloading a big block of raw text into a string? For the purpose of the question please assume that this is a multi-line script which includes arbitrary quote characters etc.

First of all I don't know if you want this on a program you're compiling with the fp console IDE or with the Lazarus IDE, so I'll try and give an answer to both:

1. Using fp

I spent about 20m looking in all the menus for something that would remotely mention resources. Alas I failed.
If there is a way, I need more time to investigate.

2. Using Lazarus

If you go to the project options(CTRL+SHIFT+F11) under the entry Resources on the tree view you can add all types of resources that are automatically linked to your binary.
This way you use the same procedure that creates the *.res file that is usually created and contains, AT LEAST, the default icon for the application when in GUI mode.
And from any normal Linux install, you're able to have that *.res file produced and linked, right?

I'm guessing that even if you're doing a purely console app, Lazarus will still link the resources you've added there, since I think that any binary, GUI or not can have resources linked to it.

If that works for you, do you then need assistance on how to retrieve the saved resource?
I haven't done that in a long while but I do remember that it wasn't hard.

Cheers,
Gus
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: marcov on July 31, 2021, 02:33:59 pm
Simply include the textfile as a resource, and load it from the resource? Afaik resources work with Linux too.

https://wiki.freepascal.org/Lazarus_Resources says that doing so needs windres or GoRC, neither of which comes as standard. The current manual concurs, and the examples still have to be quoted.

Yes, until FPC 3.2.2 and fpcres.

The syntax is

fpcres -i logo.rc -of res  -o logo.res

Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on July 31, 2021, 03:17:16 pm
Yes, until FPC 3.2.2 and fpcres.

The syntax is

fpcres -i logo.rc -of res  -o logo.res

So how does embedded text look? Because if it still has to be quoted there really isn't much point from my POV.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: marcov on July 31, 2021, 03:32:35 pm
Yes, until FPC 3.2.2 and fpcres.

The syntax is

fpcres -i logo.rc -of res  -o logo.res

So how does embedded text look? Because if it still has to be quoted there really isn't much point from my POV.

You store the text in a textfile, and then make a .rc file containing this:

Defaults RCDATA defaults.xml

this makes the contents of the file "defaults.xml" binary (RCDATA is binary) available using the resource name "Defaults". You can include multi files using one .rc file. I do this e.g. with shader source files.

I however load the data over windows unit functions, but it should be possible portably too (Lazarus uses it afaik)
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: lucamar on July 31, 2021, 03:38:27 pm
So how does embedded text look? Because if it still has to be quoted there really isn't much point from my POV.

The text is embedded as-is, and can be retrieved without any conversion. Think of it as if loading a text file into a String, only instead of loading from an external file you get it from the resource in the executable.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: jamie on July 31, 2021, 04:06:12 pm
Code: Pascal  [Select][+][-]
  1.   S := JSonStringTostring('Hello hackers \n');  
  2.  

You should be able to construct your strings this way..
you could use the complete \r\n too.. what ever combo you need.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: Bart on July 31, 2021, 05:33:17 pm
Except that's not part of FPC.

Sorry, you're right of course.

Barr
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on July 31, 2021, 06:13:52 pm
You store the text in a textfile, and then make a .rc file containing this:

Defaults RCDATA defaults.xml

this makes the contents of the file "defaults.xml" binary (RCDATA is binary) available using the resource name "Defaults". You can include multi files using one .rc file. I do this e.g. with shader source files.

Thanks for that. So it's doable, but hardly painless.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: marcov on July 31, 2021, 06:33:02 pm
Thanks for that. So it's doable, but hardly painless.

That is a very opinionated and biassed statement. There are multiple sides to this, and if you have e.g. external editors or viewers for such included files, then suddenly the in-source view is extremely cumbersome and a poor man's solution.

But many languages actively limit themselves to the in source solution,, because they can't standardize tooling outside of the core language due to oppressive requirements for formal language definitions. Then doing it in source becomes the "my only tool is a hammer" kind of solution.

FPC and Delphi don't have that problem.   
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: engkin on July 31, 2021, 06:46:35 pm
I've reverted to defining it as a multiline constant, but either have to use the double-' escape or define a substitute and preprocess the text... the latter works out better but is still hardly ideal:

Code: Pascal  [Select][+][-]
  1. const
  2.   qq= LineEnding;                       (* Prefix is same size as 8-spaces tab  *)
  3.  
  4.   syntax='.QUOTE `'
  5. + qq + '`MetaScriptSyntax` <<EOF'
  6. + qq + '.SYNTAX MetaScriptSyntax'
  7. + qq + ''
  8. + qq + '(* This is the master representation of the syntax of the embedded      *)'
  9. + qq + '(* script compiler.                                                     *)'
  10. ...
  11.  


There is a patch for FPC to add multi-line strings. Discussed here. (https://forum.lazarus.freepascal.org/index.php/topic,46050.0.html)

I don't know why I have to sign in to gitlab to view the bug report %)
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on July 31, 2021, 06:48:50 pm
Thanks for that. So it's doable, but hardly painless.

That is a very opinionated and biassed statement. There are multiple sides to this, and if you have e.g. external editors or viewers for such included files, then suddenly the in-source view is extremely cumbersome and a poor man's solution.

But many languages actively limit themselves to the in source solution,, because they can't standardize tooling outside of the core language due to oppressive requirements for formal language definitions. Then doing it in source becomes the "my only tool is a hammer" kind of solution.

FPC and Delphi don't have that problem.

Well, maybe I'm an opinionated and biased person: after all, I use Pascal and have consistently argued its strengths compared with lesser languages for the last 40 years.

But I note that C++11 introduced inline text storage, which had never been implemented in older revisions or in C itself: whatever we think of the language there's smart people involved, and I'm sure that if they could see a better way they'd have used it.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: marcov on July 31, 2021, 06:53:23 pm
But I note that C++11 introduced inline text storage, which had never been implemented in older revisions or in C itself: whatever we think of the language there's smart people involved, and I'm sure that if they could see a better way they'd have used it.

C and C++ are the archetypical development system of the "my only tool is a hammer" type, since they only standardize compiler and preprocessor, and it is enormously complicated to add another tool and file type to the standard. Going with the in source option is the direction of least resistance, and they probably hope they put the matter to rest for a while

Like us, when we implemented the resource solution.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on July 31, 2021, 06:53:47 pm
There is a patch for FPC to add multi-line strings. Discussed here. (https://forum.lazarus.freepascal.org/index.php/topic,46050.0.html)

I don't know why I have to sign in to gitlab to view the bug report %)

I'd forgotten that debate, and viewed purely as a multiline string issue I think I wasn't enthusiastic... but I can see the overlap now that you've mentioned it.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: avra on July 31, 2021, 08:23:02 pm
It is interesting to know that one embedded pascal I use allows this:

Code: Pascal  [Select][+][-]
  1. const
  2.   // 100 point.x point.y of integer, 3 bytes info
  3.   MuLookUp: array[1..(100 * 4) + 3] of byte = 'AVR Interpol.crvg';
  4.  
  5.   // 32*32 pixels = 128bytes + 2 bytes size info
  6.   MyBitMap: array[1..(32*32 div 8) + 2] of byte = 'logo.pbmp';
  7.  
  8.   // 128 chars = 128 * 7 bytes + 2 bytes size info
  9.   MyCharSet: array[1..(128 * 7) + 2] of byte = 'myCharSet.pchr';

I could not find info that strings could load content from file as arrays above, but anyway...
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on July 31, 2021, 10:03:04 pm
@avra: I see what you're getting at there, and loading pregenerated lookup tables etc. is obviously a fairly common requirement for embedded stuff... not to mention founts for LCD displays. I suppose that we need a preprocessor of some sort, which would mean provision for invoking one as a standard part of the compiler run since makefiles aren't used for most projects.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: Gustavo 'Gus' Carreno on July 31, 2021, 11:12:08 pm
Hey Mark,

I dunno if you still want a bit of code for this particular example.

If not, just ignore my post and All's good.

If you still want it, I've cobbled up a pure RTL version of the resource loading of a very simple shell script into a string and then shoving it into a TMemo, but you'll notice that I have it on a string, so the memo is just for visual aid.

I include the shell script and if you press CTRL+SHIT+F11 and click on the Resources item you'll find where I'm adding the resource and where I define it's name and type.

Hope this helps in any form or shape.

Cheers,
Gus

Edit:
Completely forgot to mention that this is from the stable version of Lazarus/FPC which is 2.0.12/3.2.2 on an Ubuntu 21.04.
My Install is via fpcupdeluxe and I've not added any of the resource linkers mentioned.
But I do have to admit that I don't know what resource builder it uses and if it was installed by fpcupdeluxe or it comes by default with Ubuntu.
I, personally, did nothing to install a resource builder.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: Kays on July 31, 2021, 11:24:27 pm
[…] LCD displays. […]
OMG. Did you just write “liquid crystal display displays”? :o

Is there a way that works with as-installed FPC on Linux of preloading a big block of raw text into a string? […]
No, but since fp-compiler depends on binutils, that means there is an assembler as(1), you could also (ab)use objects files and the {$link} directive (https://freepascal.org/docs-html/current/prog/progsu43.html). First create a wrapper file:
Code: ASM  [Select][+][-]
  1. global words
  2. global words.length
  3.  
  4. section .data
  5.  
  6. words:
  7. incbin '/usr/share/dict/words'
  8.  
  9. words.length:
  10.         dq $ - words
“Assemble” it (sorry, I only know nasm(1), but there definitely is some GNU Assembler equivalent):
Code: Bash  [Select][+][-]
  1. nasm -f elf64 words.asm
And then use it:
Code: Pascal  [Select][+][-]
  1. program foo(input, output, stdErr);
  2. {$link words}
  3. {$modeSwitch unicodeStrings-}{ensure single-Byte `char`}
  4. var
  5.         words: array[1..high(ptrInt)] of char; external name 'words';
  6.         words_length: qWord; external name 'words.length';
  7. begin
  8.         writeLn(words_length, ' Bytes');
  9.         write(words[1..80]);
  10. end.
Evidently that’s not a string constant, yet it’s technically less complicated than using resources, but it’s also just a really dumb character array. Not worth mentioning, using Assembly can be considered nasty.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: marcov on July 31, 2021, 11:35:22 pm
Kays:  bin2obj and data2inc are two pre resource ways for including data.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: Kays on August 01, 2021, 12:30:04 am
Kays:  bin2obj and data2inc are two pre resource ways for including data.
Ah, of course! Why didn’t you mention it earlier? Although I hate typed pseudo-“constants”, data2inc(1) is the way to go.

Evidently that’s not a string constant, […]
To make it effectively a constant, put everything in
Code: ASM  [Select][+][-]
  1. section .rodata
At least the ELF supports non-executable read-only sections, yet any assignments to this “constant” are only detected at run-time. A typed pseudo-constant will still be mutable.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: avra on August 01, 2021, 12:41:48 am
loading pregenerated lookup tables etc. is obviously a fairly common requirement for embedded stuff
Well, I have only shown arrays which are common with standard pascal. That embedded implementation also has tables, which are a specialized 1-dimensional version of arrays containing look-up tables. The table length is limited on power of 2, to allow a very, very fast access:
Code: Pascal  [Select][+][-]
  1. structconst {constant in Rom, at startup copied into Ram}
  2.    Table1: Table[0..3] = (0, $45, $A5, $FF);
  3. var
  4.    tb1: Table[0..15] of char;
  5.    tb2: Table[0..127] of word;
Sometimes tricks like this mean a lot in embedded world...  8)
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: PascalDragon on August 01, 2021, 12:43:24 pm
Simply include the textfile as a resource, and load it from the resource? Afaik resources work with Linux too.

https://wiki.freepascal.org/Lazarus_Resources says that doing so needs windres or GoRC, neither of which comes as standard. The current manual concurs, and the examples still have to be quoted.

windres is contained with the Windows distribution of FPC. On other platforms you need to make sure that your buildchain is set up correctly (just like you need to make sure that as and ld are available).

Though for manual resource compilation you can use fpcres from 3.2.2 and in 3.3.1 you can pass -FF to tell the compiler to use fpcres instead of windres/gorc (though that switch will soon be dropped soon as it's behavior will become the default).

I however load the data over windows unit functions, but it should be possible portably too (Lazarus uses it afaik)

Very easy:

Code: Pascal  [Select][+][-]
  1. program tres;
  2.  
  3. {$mode objfpc}
  4.  
  5. uses
  6.   Classes, resource;
  7.  
  8. {$R test.res}
  9.  
  10. var
  11.   rs: TResourceStream;
  12.   sl: TStringList;
  13. begin
  14.   rs := TResourceStream.Create(HINSTANCE, 'Default', MAKEINTRESOURCE(RT_RCDATA));
  15.   try
  16.     sl := TStringList.Create;
  17.     try
  18.       sl.LoadFromStream(rs);
  19.  
  20.       Writeln(sl.Text);
  21.     finally
  22.       sl.Free;
  23.     end;
  24.   finally
  25.     rs.Free;
  26.   end;
  27. end.

test.rc:

Code: [Select]
Default RCDATA Defaults.ini
Defaults.ini:

Code: [Select]
[test]
a=1
b=2

[foo]
c=3

Works on both Windows and Linux (and also other platforms). In trunk you can use -FF to force the use of fpcres.

If you don't have windres in FPC 3.2.2 or older you need to compile the resource manually using a FPC 3.2.2 fpcres.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: Alextp on August 01, 2021, 12:47:35 pm
@PascalDragon,
that was useful post, so I ask: is it (that info how to use TResourceStream with some text) in the Wiki?
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: PascalDragon on August 01, 2021, 12:53:16 pm
@PascalDragon,
that was useful post, so I ask: is it (that info how to use TResourceStream with some text) in the Wiki?

The already provided link to the Wiki (https://wiki.freepascal.org/Lazarus_Resources) contains that information (though it could be improved a bit).
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on August 01, 2021, 03:06:38 pm
windres is contained with the Windows distribution of FPC. On other platforms you need to make sure that your buildchain is set up correctly (just like you need to make sure that as and ld are available).

I habitually use Debian (mostly AMD64) and build both FPC and Lazarus from scratch. There is no windres on these systems, and Debian has no package of (some variant of) that name in their repositories.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: marcov on August 01, 2021, 03:12:27 pm
I habitually use Debian (mostly AMD64) and build both FPC and Lazarus from scratch. There is no windres on these systems, and Debian has no package of (some variant of) that name in their repositories.

IIRC it is part of cross win32 binutils. But as said better use trunk or 3.2.2 fpcres.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on August 01, 2021, 03:20:12 pm
I habitually use Debian (mostly AMD64) and build both FPC and Lazarus from scratch. There is no windres on these systems, and Debian has no package of (some variant of) that name in their repositories.

IIRC it is part of cross win32 binutils. But as said better use trunk or 3.2.2 fpcres.

I confirm I've got fpcres, and note the comments about improvements... thanks.

MarkMLl
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: Gustavo 'Gus' Carreno on August 01, 2021, 10:04:24 pm
Hey PascalDragon,

In the example (https://forum.lazarus.freepascal.org/index.php/topic,55611.msg413854.html#msg413854) I provided, my code is the logic equivalentl to yours.
There are the obvious differences:
I opted for a GUI and you opted for a CLI and the rest of the differences is to adapt to the necessary units for each case.

The only other diff is that I used the Resource section on the Project Options(CTRL+SHIFT+F11) and in your case one needs to manually compile the linkable objects from the *.rc files.

You then mention the necessary binaries to do so: windres, GoRC or fpcres.

I then had a look into my fpc/bin/x86_64-linux folder on all 3 of my installs: trunk(49634), stable(r49371) and even 3.2.0 and all tree have fpcres.

So my question is: In the background, on my Ubuntu 21.04 64b, fpcupdeluxe installed Lazarus, what is it using to compile the resources that I added on the Project Options', Resources section?

I now ask this cuz, if you read my previous post with the example, I mention that I'm ignorant of the process that occurs in the background, performed by Lazarus and I'm now curious.

Cheer,
Gus
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: PascalDragon on August 02, 2021, 09:30:56 am
The only other diff is that I used the Resource section on the Project Options(CTRL+SHIFT+F11) and in your case one needs to manually compile the linkable objects from the *.rc files.

Please note that the compiler will automatically invoke the resource compiler if the x.res is missing, but a x.rc exists (by default calling windres or gorc and with FPC trunk fpcres if started with -FF (an option that will become default soon)).

So my question is: In the background, on my Ubuntu 21.04 64b, fpcupdeluxe installed Lazarus, what is it using to compile the resources that I added on the Project Options', Resources section?

I now ask this cuz, if you read my previous post with the example, I mention that I'm ignorant of the process that occurs in the background, performed by Lazarus and I'm now curious.

I don't know what Lazarus does exactly only that does indeed prepare the resource file for the project from the settings (resources themselves can be per-unit, e.g. the {$R *.lfm} are handled as such), but in the end it will either use windres/gorc (because up to now fpcres did not support compilation of rc-files) or it will use the code of the fcl-res package (which is also the base for fpcres) and manually create the res-file.
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: BeniBela on August 02, 2021, 11:13:47 am
I habitually use Debian (mostly AMD64) and build both FPC and Lazarus from scratch. There is no windres on these systems, and Debian has no package of (some variant of) that name in their repositories.


binutils-mingw-w64-i686 has i686-w64-mingw32-windres
Title: Re: Old chestnut: including a big block of raw text into a program
Post by: MarkMLl on August 02, 2021, 12:20:37 pm
I habitually use Debian (mostly AMD64) and build both FPC and Lazarus from scratch. There is no windres on these systems, and Debian has no package of (some variant of) that name in their repositories.


binutils-mingw-w64-i686 has i686-w64-mingw32-windres

I know. But noting a core developer's "you need to make sure that your buildchain is set up correctly (just like you need to make sure that as and ld are available)" I'd suggest that this is the first time that I've seen a suggestion that the standard buildchain on Linux should be considered to include Ming.

MarkMLl
TinyPortal © 2005-2018