Writing to file of word not operating as expected

[ajavdk]

Hello all, first time poster here.

I am trying to write a script that can talk to a pascal program. The pascal program creates and write binary files to store compressed data.

The files used in the pascal program are files of "word" (16 bit). Reading and writing works seamlessly. For example, I can write the value 20, then read back the value 20, from such a file.

For example, consider the following code.

Code: Pascal  [Select][+][-]

1. program filetesting;
2. var
3.    f : file of word;
4.    c: integer;
5. begin
6.    Assign(f, 'wordFile.bin');
7.    Rewrite(f);
8.    Write(f, 20);
9.    Close(f);
10.  
11.    assign(f, 'wordFile.bin');
12.    reset(f);
13.    while not eof(f) do
14.    begin
15.      read(f, c);
16.      writeln(c);
17.    end;
18.  
19. end.
20.  

This will accurately read and write 20.

However, when opening the binary file with xxd -b wordFile.bin I get
0001010000000000

which is 20, but bitshifted to the left by 8. This seems like really strange behaviour to me. When I rewrite the same program but with bytes

Code: Pascal  [Select][+][-]

1. program filetesting;
2. var
3.    f : file of Byte;
4.    c: byte;
5. begin
6.    Assign(f, 'wordFile.bin');
7.    Rewrite(f);
8.    Write(f, 20);
9.    Close(f);
10.  
11.    assign(f, 'wordFile.bin');
12.    reset(f);
13.    while not eof(f) do
14.    begin
15.      read(f, c);
16.      writeln(c);
17.    end;
18.  
19. end.
20.  

I get the correct, single byte binary representation for 20, which is 00010100.

Does anyone have more information on how writing to a file of "word" changes our expectations (not just writing purely binary in chunks of 16 bits)?

[Handoko]

Hello ajavdk,
Welcome to the forum.

I haven't tried but I think it has something to do with endian. Read more:

https://www.freepascal.org/docs-html/rtl/system/swapendian.html
http://wiki.lazarus.freepascal.org/Writing_portable_code_regarding_the_processor_architecture
https://en.wikipedia.org/wiki/Endianness

[Leledumbo]

I haven't tried but I think it has something to do with endian. Read more:

With a guess OP is on little endian machine (and it seems so), that might be correct. FPC writes typed files according to machine endian as well. It's easy to test by using hex constant and xxd without -b:

Code: Pascal  [Select][+][-]

1. var
2.    f : file of word;
3. begin
4.    Assign(f, 'endian.bin');
5.    Rewrite(f);
6.    Write(f, $ABCD);
7.    Close(f);
8. end.
9.  

Code: Bash  [Select][+][-]

1. $ xxd endian.bin
2. 00000000: cdab
3.  

As can be seen, the bytes (NOT bits!) are reversed.

[Thaddy]

Isn't it simply:

Code: Pascal  [Select][+][-]

1. program filetesting;
2. var
3.    f: file of word;
4.    c: word; // <----- NOT integer.....maybe smallint is OK. or mode FPC or TP
5. begin
6.    Assign(f, 'wordFile.bin');
7.    Rewrite(f);
8.    Write(f, 20);
9.    Close(f);
10.  
11.    assign(f, 'wordFile.bin');
12.    reset(f);
13.    while not eof(f) do
14.    begin
15.      read(f, c);
16.      writeln(c);
17.    end;
18. end.

Works with any endianness...and in any mode...
Note it can be quite complex: Lazarus chooses a mode for you: mode objfpc. If you would compile from the command-line, usually mode FPC is picked and then your code works as expected. See the following (mode TP or FPC specified):

But I guess you expected this:

Code: Pascal  [Select][+][-]

1. program filetesting;
2. {$mode TP} // 16 bit integers
3. var
4.    f: file of word;
5.    c: integer; // <----- Now that is OK. Integer is 16 bit in TurboPascal mode
6. begin
7.    Assign(f, 'wordFile.bin');
8.    Rewrite(f);
9.    Write(f, 20);
10.    Close(f);
11.  
12.    assign(f, 'wordFile.bin');
13.    reset(f);
14.    while not eof(f) do
15.    begin
16.      read(f, c);
17.      writeln(c);
18.    end;
19. end.

Works in mode TP. See here why: http://wiki.freepascal.org/Integer
 

[Mr.Madguy]

Just little endian rule: lower byte at lower address. Word 20 is actually $0014. But lower byte is $14 and therefore it's stored at lower address - i.e. first. And only then $00. And what we get as result - is $1400, not $0014, as you would expect.

Just to understand this concept:

1) Little endian

Code: Pascal  [Select][+][-]

1. Word = packed record
2.   LowByte, HighByte:Byte;
3. end;
4.  
5. DWord = packed record
6.   LowWord, HighWord:Word;
7. end;
8.  
9. QWord = packed record
10.   LowDWord, HighDWord:DWord;
11. end;
12.  

2) Big endian

Code: Pascal  [Select][+][-]

1. Word = packed record
2.   HighByte, LowByte:Byte;
3. end;
4.  
5. DWord = packed record
6.   HighWord, LowWord:Word;
7. end;
8.  
9. QWord = packed record
10.   HighDWord, LowDWord:DWord;
11. end;
12.  

IMHO: Little endian is more logical. Cuz when you have such declaration

Code: Pascal  [Select][+][-]

1. TDWord = packed record
2.   case Boolean of
3.     False:(DWord:Longword);
4.     True:(Bytes:array[0..3] of Byte);
5. end;
6.  

it's logical to expect MyDWord.Bytes[0] to be the  lowest byte, not the  highest. Cuz 0 is the  lowest number. Same is 0th bit - is the lowest bit.

[Thaddy]

It has nothing to do with endiannes. It has to do with the integer size. As I demonstrated and is the correct answer. <grumpy  >

[Mr.Madguy]

It has nothing to do with endiannes. It has to do with the integer size. As I demonstrated and is the correct answer. <grumpy  >

Please note, that he's using Integer for reading only. So it shouldn't affect contents of file itself.

[engkin]

To confirm the problem has anything to do with endianness, test the output Thaddy's code, where c is of type word, and the output of this code:

Code: Pascal  [Select][+][-]

1. program filetesting;
2.  
3. {$mode objfpc}{$H+}
4.  
5. var
6.    f : file of word;
7.    c: word;
8. begin
9.    Assign(f, 'wordFile.bin');
10.    Rewrite(f);
11.    c := 20;
12.    c := NtoBE(C);
13.    Write(f, c);
14.    Close(f);
15.  
16.    assign(f, 'wordFile.bin');
17.    reset(f);
18.    while not eof(f) do
19.    begin
20.      read(f, c);
21.      c := BEtoN(C);
22.      writeln(c);
23.    end;
24. end.

It forces the file to have big endian format.

[Thaddy]

To confirm the problem has anything to do with endianness, test the output Thaddy's code, where c is of type word, and the output of this code:

Code: Pascal  [Select][+][-]

1. program filetesting;
2.  
3. {$mode objfpc}{$H+}
4.  
5. var
6.    f : file of word;
7.    c: word;
8. begin
9.    Assign(f, 'wordFile.bin');
10.    Rewrite(f);
11.    c := 20;
12.    c := NtoBE(C);
13.    Write(f, c);
14.    Close(f);
15.  
16.    assign(f, 'wordFile.bin');
17.    reset(f);
18.    while not eof(f) do
19.    begin
20.      read(f, c);
21.      c := BEtoN(C);
22.      writeln(c);
23.    end;
24. end.

It forces the file to have big endian format.

That's a bit silly(not to use a more abusive word) . Usually this is not the question to complicate matters....
My code works regardless. That's the point I tried to make..... It is about the mode, not about endiannes.

[Mr.Madguy]

That's a bit silly(not to use a more abusive word) . Usually this is not the question to complicate matters....
My code works regardless. That's the point I tried to make..... It is about the mode, not about endiannes.

This is just unreasonable conflict. I guess, you've simply missed whole point of OP's post. He doesn't complain about his code not working - he says, that it works perfectly. He just wonders, why output is 20 00 instead of string-order 00 20, he has expected. Answer is - it's due to little endian, i.e. lower byte at lower address rule.

[Thaddy]

You can't read : 0001010000000000 is 32 bit.

[engkin]

That's a bit silly(not to use a more abusive word) .

Thank you for not using a more abusive word.

You can't read : 0001010000000000 is 32 bit.

Let me count it:

Code: Pascal  [Select][+][-]

1. 0001010000000000
2. 1234567890123456

It seems to me that 0001010000000000 is 16 bits.

[Bart]

It has nothing to do with endiannes. It has to do with the integer size. As I demonstrated and is the correct answer. <grumpy  >

You're wrong.

I wrote 1 word ($abcd) to using different modes (as reflected by their filenames).

Code: Pascal  [Select][+][-]

1. program test;
2.  
3. var
4.   f: file of word;
5. begin
6.   assign(f,paramstr(1));
7.   rewrite(f);
8.   write(f,$abcd);
9.   close(f);
10. end.

Then inspected their contents, which (as expected) were all the same.

Code: [Select]

C:\Users\Bart\LazarusProjecten\ConsoleProjecten>fpc -Mtp test.pas
..
C:\Users\Bart\LazarusProjecten\ConsoleProjecten>test tp.bin

C:\Users\Bart\LazarusProjecten\ConsoleProjecten>fpc -Mobjfpc test.pas
...
C:\Users\Bart\LazarusProjecten\ConsoleProjecten>test objfpc.bin

C:\Users\Bart\LazarusProjecten\ConsoleProjecten>fpc -Mfpc test.pas
...
C:\Users\Bart\LazarusProjecten\ConsoleProjecten>test fpc.bin

C:\Users\Bart\LazarusProjecten\ConsoleProjecten>fpc -Mdelphi test.pas
..
C:\Users\Bart\LazarusProjecten\ConsoleProjecten>test delphi.bin

C:\Users\Bart\LazarusProjecten\ConsoleProjecten>dir *.bin

29-06-2018  23:13                 2 delphi.bin
29-06-2018  23:13                 2 fpc.bin
29-06-2018  23:13                 2 objfpc.bin
29-06-2018  23:12                 2 tp.bin
               4 File(s)              8 bytes
               0 Dir(s)  110.705.262.592 bytes free

C:\Users\Bart\LazarusProjecten\ConsoleProjecten>hd tp.bin
Offset    Hex                                              ASCII
00000002: CD AB


C:\Users\Bart\LazarusProjecten\ConsoleProjecten>hd fpc.bin
Offset    Hex                                              ASCII
00000002: CD AB


C:\Users\Bart\LazarusProjecten\ConsoleProjecten>hd objfpc.bin
Offset    Hex                                              ASCII
00000002: CD AB


C:\Users\Bart\LazarusProjecten\ConsoleProjecten>hd delphi.bin
Offset    Hex                                              ASCII
00000002: CD AB

Bart

[ajavdk]

Thanks everyone for the replies. Yes, I was mostly interested in why reading the raw binary is in that format. I am on a little-endian machine (as are most people).

I was originally working with strings directly then casting to binary in javascript, but I ended up using the UInt16 typed array (which is analogous to file of word).