Recent

Author Topic: [SOLVED] Passing static array of byte by reference... without creating copies  (Read 9881 times)

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
I've read
http://stackoverflow.com/questions/3911432/pass-static-byte-array-in-parameter-in-delphi
and tried to apply it.... without success... or perhaps with other complications.

More eyes welcome

The program recurses through files/directories, and stores details in a Firebird (embedded0 database. Sort of like locate

My goal: have 1 buffer that is used for all md5 calculations instead of having duplicates.
The application is single threaded so 1 buffer should be ok..

In search.pas, the search happens in AddFileToResult
FFileBuffer should create a new array once...
Code: [Select]
  TFileSearch = class
  private
...
    // Buffer that can be used by e.g. md5 hashes
    FFileBuffer: TFileBuffer; //Buffer used e.g. in md5 calculations, defined in flocatecommon as array[1...1024*1024] of byte
...and this is passed to the directoryentry constructor (should be as reference)...
Code: [Select]
procedure TFileSearch.AddFileToResult(const ThisDir: String; item: TSearchRec);
// Finds out properties of file specified in item, adds this information
// to a record, and adds a pointer in the Res list to this record.
var
  DirEntry: TDirectoryEntry;
begin
  assert(ThisDir <> '', 'Directory to be searched may not be empty');
  assert(Item.Name <> '', 'File to be searched must have a filename');
  //Create a new file object and let it figure out all properties of the file.
  DirEntry := TDirectoryEntry.Create(ThisDir + Item.Name,FFileBuffer,FReadFileContents);
... then directoryentry.pas does this
Code: [Select]
  TDirectoryEntry = class(TObject)
  private
...
    FFileBuffer: TFileBuffer; //buffer for e.g. md5 etc can be reused.
... this worries me, AFAIU, this will generate a new array. How can I declare the right variable here?

The constructor then goes on
Code: [Select]
constructor TDirectoryEntry.Create(FullPath: string; var FileBuffer: TFileBuffer; ReadFileContents: boolean);
begin
...
  FFileBuffer:=FileBuffer;

Attached complete code for Win/Linux; it creates a firebird embedded db if it doesn't exist (requires fb dll/lib of course)

Thanks!
« Last Edit: March 10, 2014, 07:05:29 pm by BigChimp »
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

User137

  • Hero Member
  • *****
  • Posts: 1791
    • Nxpascal home
Re: Passing static array of byte by reference... without creating copies
« Reply #1 on: March 10, 2014, 05:18:09 pm »
The format on dynamic and static array are little different, so it's not directly possible i'd assume, without hacks. Easy solution would be to use array pointer
Code: [Select]
type
  PFileBuffer = ^TFileBuffer;
...
private
...
    FFileBuffer: PFileBuffer;

Leledumbo

  • Hero Member
  • *****
  • Posts: 8114
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Passing static array of byte by reference... without creating copies
« Reply #2 on: March 10, 2014, 06:04:39 pm »
Do as User137 suggests. Even if the parameter is declared by reference, assignment between static arrays does a deep copy. Well actually, even if you don't assign anything to TDirectoryEntry.FFileBuffer, when you create TDirectoryEntry instance, each will already occupy the its own 1 MB space. Assignment only copies the values.

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Re: Passing static array of byte by reference... without creating copies
« Reply #3 on: March 10, 2014, 06:37:51 pm »
Thanks a lot, that does seem to work fine!
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

User137

  • Hero Member
  • *****
  • Posts: 1791
    • Nxpascal home
Another solution came to mind, that FileBuffer would be a array property, which uses getter function directly to original static array. It has benefit of having simpler syntax when coding with, no pointer confusions with ^ symbols.

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Ok; however I have to pass on the buffer to an MD5 function anyway (see below), so the pointers actually suit me fine...

@Leledumbo: the md5 function is this one:
Code: [Select]
// Adapted from MDFile in md5 unit
function CustomMD5File(const Filename: String; var Buffer: TFileBuffer): TMDDigest;
const
  Version=MD_VERSION_5; //calculate md5 format hash
var
  F: File;
  Context: TMDContext;
  Count: Cardinal;
  OldFileMode: Longint;
begin
  MDInit(Context, Version);

  Assign(F, Filename);
  {$push}{$i-}
  OldFileMode := FileMode;
  FileMode := 0;
  Reset(F, 1);
  {$pop}

  try
    if IOResult = 0 then
    begin
      repeat
        BlockRead(F, Buffer, Bufsize, Count);
        if Count > 0 then
          MDUpdate(Context, Buffer, Count);
      until Count < BufSize;
      Close(F);
    end;

    MDFinal(Context, Result);
  finally
    FileMode := OldFileMode;
  end;
end;
and you said:
Quote
Even if the parameter is declared by reference, assignment between static arrays does a deep copy.

Do I understand correctly that I should have memory leaks now? -gh/heaptrace doesn't show any memleaks to do with direntry... so perhaps I misunderstand.
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

Leledumbo

  • Hero Member
  • *****
  • Posts: 8114
  • Programming + Glam Metal + Tae Kwon Do = Me
Quote
Even if the parameter is declared by reference, assignment between static arrays does a deep copy.

Do I understand correctly that I should have memory leaks now? -gh/heaptrace doesn't show any memleaks to do with direntry... so perhaps I misunderstand.
No, static arrays are allocated on the stack. So it should never introduce memory leak.

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Thanks, Leledumbo. Couldn't really (quickly) find that documented but well...
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Found that the updated documentation
http://www.freepascal.org/docs-html/ref/refsu18.html#x42-460003.3.1
specifies "When static array-type variables are assigned to each other, the contents of the whole array is copied." Ok.
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified