Recent

Author Topic: Threaded file writer  (Read 731 times)

mas steindorff

  • Sr. Member
  • ****
  • Posts: 437
Threaded file writer
« on: January 14, 2020, 09:46:42 pm »
I'm in need of a threaded file writer that will buffer the output data while the disk is busy with another app (thank you dropbox  :'( ). I think I remember this is already done but I can't find either a unit or a built-in.
Otherwise... The file stuff is ez but whats the best way to pass data from the main program into the threads FIFO?  I also remember reading about some sort of threaded FIFO
Thanks
Mr forgetful ... mas
windows 7/10 - laz 2.0 / 1.2.6 general releases

mas steindorff

  • Sr. Member
  • ****
  • Posts: 437
Re: Threaded file writer
« Reply #1 on: January 16, 2020, 10:01:50 pm »
so I created my own unit but I'm running into some SIGSEGV exceptions when I try and modify my FIFO variables.
I though the issue was my access of the fifo's "count" variable but removed it and I get it when I modify the in variable.
here are the relative snippets of code
Code: Pascal  [Select]
  1. const RawBuffSize = 8192*4;
  2.  
  3. Type
  4.  
  5.  TThreadFileSave = class(TThread)
  6.   private     { private declarations }
  7.      Raw :record
  8.        inpos, outpos:integer;
  9.        count :integer;
  10.        Buff  :Array[-2..RawBuffSize] of byte; // two lead bytes for rap fixes
  11.        end;
  12.  
  13.  
  14. ...
  15.    end;
  16.  
I get the exception when I access raw.inpos with the following after I have copied the new data into my raw.buff[]
Code: Pascal  [Select]
  1. function TThreadFileSave.AddToBuff(p: pbyte; Buflen: integer): Boolean;
  2. var pbuf:pbyte;
  3.     len, limit,count:integer;
  4. begin
  5.   if not assigned(self) then
  6.       exit;
  7.   result := FALSE;   //
  8.   if (Buflen < 1) then begin
  9.      Showstatus('Buffer add of 0 bytes?');
  10.      exit;
  11.      end;
  12. ...
  13.      pbuf := @(raw.Buff[raw.inpos]);
  14.      move(p, pbuf, len);                          // move data into local FIFO works
  15.     count := raw.count;                          // works
  16.     inc(count, len);                              
  17.     if (count >= RawBuffSize) then
  18.         begin
  19.         ShowStatus(' Thread Says buffer overrun.');
  20.         count := RawBuffSize;
  21.         end;
  22.     raw.count = count;                              // <== exception here
  23.     inc(raw.inpos, len);                              // <== or here if I comment out the line above
  24.     if (raw.inpos = RawBuffSize+1) then
  25.         raw.inpos := 0  // this works here due to reduction of len before call
  26.     else if (raw.inpos > RawBuffSize) then begin
  27.         raw.inpos := raw.inpos - RawBuffSize;
  28.         ShowStatus('Warning: was not expecting FIFO to wrap in one pass.');
  29.         end;
  30. result := true;
  31. end;
  32.  
  33.  

I've tried using EnterCriticalsection and LeaveCriticalSection without success. the error hits the on the 1st call. 

I define the thread within the VSD class
Code: Pascal  [Select]
  1.    T_VSD = class
  2.      my : record
  3.         tag               :longint;
  4.         BufFile  : TThreadFileSave;
  5.      ...
  6.    public
  7.      constructor Create;
  8.      destructor Destroy;  override;
  9. end;
  10.  

my thread init code and I am also starting the thread before I make the call to AddToBuff above

Code: Pascal  [Select]
  1.  
  2. constructor vsd.create;
  3. begin
  4. ...
  5.   my.BufFile           := TThreadFileSave.Create(true);  // suspended until file name is set
  6. ...
  7.   my.BufFile.OpenFile('rawTest1.bin');  // which has Start built in.
  8. ...
  9.   my.BufFile.AddToBuff(parameters)          // later after testing that OpenFile was successful
  10.  
  11.  
any ideals?
MAS
windows 7/10 - laz 2.0 / 1.2.6 general releases

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7863
Re: Threaded file writer
« Reply #2 on: January 16, 2020, 10:07:21 pm »
    move(p, pbuf, len);     

If you move pointers you do realize that you copy len bytes from the "p" pointer( the location with the address, not where it points to) to pbuf pointer location.

Unless len <> sizeof(pointer) this will go horribly wrong.

What you want is probably

move(p^,raw.Buff[raw.inpos],len);

or making p a formal parameter.

mas steindorff

  • Sr. Member
  • ****
  • Posts: 437
Re: Threaded file writer
« Reply #3 on: January 16, 2020, 10:27:08 pm »
thanks for the error check.  I've now commented out the move command and retested with the same results.:(
windows 7/10 - laz 2.0 / 1.2.6 general releases

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7863
Re: Threaded file writer
« Reply #4 on: January 16, 2020, 10:40:32 pm »
Hard to say from these fragments. But I would throughly check buff[] indexing. Keep in mind an buff[inpos] with inpos =0 means the THIRD character because of you funky array definiton!

mas steindorff

  • Sr. Member
  • ****
  • Posts: 437
Re: Threaded file writer
« Reply #5 on: January 16, 2020, 10:58:57 pm »
Thanks for looking.  I'll generate a simple program later if I can't track my issue down.
I've done threads that push data into main just fine.  I was thinking and figured this is my first where main pushes data into the thread.  the Addtobuff() function  is called in the main thread and modifies thread variables.  This should be OK AFAIK but is there any hidden memory access issues I've may have forgotten? it's only these inpos, count and outpos variables that are having issues.  I can read them in the same function without any issues.
I guess I'll try moving them out of the "raw" sub record to see if that does anything.
windows 7/10 - laz 2.0 / 1.2.6 general releases

mas steindorff

  • Sr. Member
  • ****
  • Posts: 437
Re: Threaded file writer
« Reply #6 on: January 16, 2020, 11:29:20 pm »
WOW I found the issue
it looks like the thread can not access sub records correctly when accessed from main.  all I did was to change the FIFO variables from
Code: Pascal  [Select]
  1.  TThreadFileSave = class(TThread)
  2.   private     { private declarations }  
  3.     Raw :record
  4.        inpos, outpos:integer;
  5.        count :integer;
  6.        Buff  :Array[-2..RawBuffSize] of byte; // two lead bytes for rap fixes
  7.        end;
  8. ...
  9.  
to
Code: Pascal  [Select]
  1.  TThreadFileSave = class(TThread)
  2.   private     { private declarations }  
  3.      Raw :record
  4.        Buff  :Array[-2..RawBuffSize] of byte; // two lead bytes for rap fixes
  5.        end;
  6.      raw_count :integer;
  7.      raw_inpos,
  8.      raw_outpos :integer;  
  9.  
and it works  (with same name changes done in code).  Is this a bug?

I'm using rev 2.0.0.r60307
windows 7/10 - laz 2.0 / 1.2.6 general releases