dynamic array. Does SetLength realloc memory?

[anna]

As I see in OllyDBG problem is in erasing other variables by SetLength(dynamic array).
I have made demo, to show access violation problem:

Code: [Select]

unit Unit1;

{$mode objfpc}{$H+}

interface

uses
  Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs, StdCtrls;
type TChAr65535 = array[1..65535] of char;
type

  { TForm1 }

  TForm1 = class(TForm)
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
  private
    { private declarations }
  public
    { public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.lfm}

{ TForm1 }

procedure TForm1.Button1Click(Sender: TObject);
Const
  LenAESKey = 16;   // Both for simple and CBC AES samples (May be 16, 24 or 32)
  INT_AES_ENCRYPT = 0;
  INT_AES_DECRYPT = 1;
  procedure aes_cbc_hack(const asource        :TChAr65535;
                           const LenDataCBC     :word;
                           var   adestination   :TChAr65535;
                           var   wdestinationlen:word);
    var KeyBin: array [0..Pred(LenAESKey)] of Byte=(
      $01,$02,$03,$04,$04,$04,$04,$04,$04,$04,$04,$04,$04,$04,$04,$04);
    var DataIn, DataIn2, DataOut: array of Byte;
    var IV: array [0..Pred(16)] of Byte;
    begin
      sleep(7);
     SetLength(DataIn,LenDataCBC);  FillChar(DataIn, LenDataCBC, 0);
      sleep(8);
     SetLength(DataIn2,LenDataCBC);FillChar(DataIn, LenDataCBC, 0);
     SetLength(DataOut,LenDataCBC);FillChar(DataIn, LenDataCBC, 0);
      wdestinationlen:=LenDataCBC and $FFF0;
      // Data, Key and Initial Vector (Just a sample)
      sleep(11);
      move(asource[(LenDataCBC and $000F)+1],DataOut,wdestinationlen); //LenDataCBC must be in form 16*n
      sleep(13);
      FillChar(IV, SizeOf(IV), 0);
      sleep(14);
      // CBC Decrypts encrypted data
      Move(DataOut,DataIn2,wdestinationlen);
      sleep(15);

      //AESCBCCryDec(DataIn2,KeyBin,LenAESKey,INT_AES_DECRYPT,LenDataCBC,IV);
      sleep(16);
      // Results
      Move(DataIn2,adestination,wdestinationlen);
    end;

  var buf2_th   ,
      buf1_th  :TChAr65535;
  NumRead, NumWritten:word;
  f_c68:file;
begin
  system.Assign(f_c68,'~Encoded.zip');
filemode := fmOpenRead;
  Reset (f_c68,1);
    BlockRead (f_c68,buf2_th,Sizeof(buf2_th),NumRead);
  system.close(f_c68);


  aes_cbc_hack(buf2_th,NumRead,buf1_th,NumWritten);
end;

end.



[sam707]

hm , Liftetime of any buffer should be long enough for the buffer to be flushed. It seems that you attempt to fulfill that condition by calling some Sleep , but in fact I think this is a random solution, and may work on a fast enough machine, not on all.

If a buffer is deallocated before any pending write, yup (and it happened to me), you are going to raise external sigsegv errors (violations).

[anna]

hm , Liftetime of any buffer should be long enough for the buffer to be flushed. It seems that you attempt to fulfill that condition by calling some Sleep , but in fact I think this is random.

If a buffer is deallocated before any pending write, yup (and it happened to me), you are going to raise external sigsegv errors.

No problem not in sleep and not in arrays. I got AV on line
 wdestinationlen:=LenDataCBC and $FFF0;
becouse FillChar erases memory for LenDataCBC, and lazarus cannot access LenDataCBC.
On your computer FillChar may erase another variable or may not erase enything, I think.
You can delete all Sleep if you want. I use Sleep to fast find right plase in OllyDBG.

In addition SizeOf(dynamic array) always gives 4. It's bug. So be carefull with SizeOf and SetLength.

[sam707]

well,

TY for the advice.

I used SetLength here :
http://forum.lazarus.freepascal.org/index.php/topic,22276.0.html

on buffers that are AnsiString and didn't experience any issue on both linux Ubuntu 12 and windows 7.

but OK , I use FPC 2.7.1, its maybe fixed , I don't know.

[sam707]

SizeOf would return size of the pointer to dyn array (size of the variable pointer), I think that is preferable to use Length for dyn arrays tho :

SetLength counterpart is Length, not SizeOf, and sounds logical to me. knowing that, there is NO bug in SizeOf 

[sam707]

why using AnsiString as buffers in IO operations is so cool :

1) unlike C strings, they don't have to end with a null character, no no no, their lengths are known by the memory manager

2) they can use SetLength and Length standard FPC calls just like dynamic arrays do

3) you access all AnsiString dedicated functions on your "buffers" to look for characters, substrings, replace parts etc

4) you can convert easily to a C like string using PChar(yourBuffer) this converted string will end at the 1st null char encoutered.

5) the memory manager does not always relocate memories when expanding or shrinking is needed, but in case of it relocates, the already existing data are copied at the right place !

Great Job FPC Compiler ! TY ! 

  C++ erff  ==> NO dynamic I/O buffers without tons of lines of codes and tricks

[Leledumbo]

1) unlike C strings, they don't have to end with a null character, no no no, their lengths are known by the memory manager

Nope, they are still, even guaranteed to be, null terminated. Their lengths are not known by the memory manager, but it's a part of the string structure. Implementation details: length is put at negative offset [-3 .. 0], data from offset 1 onwards. There's also reference counter field. Nowadays, it also has codepage property, etc.

As I see in OllyDBG problem is in erasing other variables by SetLength(dynamic array).

You're not using FillChar against dynamic array correctly and it erases the wrong memory region:

Code: [Select]

SetLength(DataIn,LenDataCBC);  FillChar(DataIn, LenDataCBC, 0);should be:

Code: [Select]

SetLength(DataIn,LenDataCBC);  FillChar(DataIn[0], LenDataCBC, 0);Remember FillChar (and other Fill* routines, also Move, and many others) first parameter is untyped. You shouldn't pass the pointer, but instead the first value.

[anna]

Remember FillChar (and other Fill* routines, also Move, and many others) first parameter is untyped. You shouldn't pass the pointer, but instead the first value.

Thanks. But I never knew that dynamic array is pointer.

[sam707]

var
  buffer : array of something;

begin
  FillChar(buffer, len, #0); // wrong , kills the pointer and part of stack data, program dies
  FillChar(buffer^, len, #0); // correct Pure Pascal Form
  FillChar(buffer[0], len, #0); // correct, young Leledumbo synonym as long as you know 1st element index

by past, only the pure pascal form was allowed because there was a difference between arrays and packed arrays (contiguous storage garanteed or not).

I recomend the pure pascal form as long as you don't need to start a fill at a specific index

So, there's NO bug in SizeOf and NO bug in FillChar, Hope we helped ...

[sam707]

1) unlike C strings, they don't have to end with a null character, no no no, their lengths are known by the memory manager

Nope, they are still, even guaranteed to be, null terminated. Their lengths are not known by the memory manager, but it's a part of the string structure. Implementation details: length is put at negative offset [-3 .. 0], data from offset 1 onwards. There's also reference counter field. Nowadays, it also has codepage property, etc.

.... and all that is known by the memory manager which takes care of allocation and free upon reference counting, almost always transparently to you !

so, sorry , but nope to your nope

nvm ...

[sam707]

assume i have a string S

 s:="hello world";

in memory it ends with a zero, OK no problem

then I do
  s[6]=#0;
 and Length (s); returns the correct 11

I said "they don't have to" end with a null char , I didn't say "they don't".

[marcov]

1) unlike C strings, they don't have to end with a null character, no no no, their lengths are known by the memory manager

Nope, they are still, even guaranteed to be, null terminated. Their lengths are not known by the memory manager, but it's a part of the string structure. Implementation details: length is put at negative offset [-3 .. 0], data from offset 1 onwards. There's also reference counter field. Nowadays, it also has codepage property, etc.

Null terminated to me sounds like they terminate on #0. ansistrings have a #0 after their data, they don't terminate on #0, since they can contain #0. Still there is a #0 after their termination (as determined
by length).

Still get it ? I'm now confusing myself. :-)

[sam707]

@Marcov
you're not confusing yourself at all :

ansistring are guaranteed to be null terminated for sum portability and compatibility reasons (you might want to call a procedure or a function in a "C" written library that needs such null terminated string -in fact, most of these external calls require it-).

but you are free to use null chars inside FPC strings as long as you know what you do and you intend to use this capability only inside pascal code. I find it very useful.

ex:

  suppose I have a string named cmd with null chars inside delimiting script commands

something like :
PChar(cmd[1]) returns a C_compliant null terminated string of the 1st command

PChar (cmd[indexof1stnullchar+1]) returns the next command (null terminated string) , and so on. without bothering me with string extraction functions.

very interesting and useful :

if I took another delimiter than null char , (';' for example) I would need more code => a dedicated procedure or function

[marcov]

ansistring are guaranteed to be null terminated for sum portability and compatibility reasons (you might want to call a procedure or a function in a "C" written library that needs such null terminated string -in fact, most of these external calls requires it-).

Clearly I confused you :-)  You misunderstood my intention;  I have a problem with calling ansistrings nul-terminated because they can contain #0s. (IOW they don't terminate on (first) nul character, and that property is not used by Pascal string handling which always uses the length).

So I only consider the casted raw data as nul terminated in the C style. The string itself not.

P.s. I'm well aware of ansistrings properties. I can remember when they were implemented in FPC in 0.99.10 or so:-)

[sam707]

and me in Delphi 2, C++Builder 3 or sumthing like that around 1999-2001