Recent

Author Topic: How to stop strings from TMemoryStream from being split by buffering?  (Read 5804 times)

vfclists

  • Hero Member
  • *****
  • Posts: 1013
    • HowTos Considered Harmful?
I am running a command in long running TProcess and copying the Output into a TMemoryStream then into a string lis, but because the intermediate buffer is small, some of the strings get truncated and continued on the next line. How can this be tested for in the stream itself, enabling me to withold the truncated string and adding it to the strings of the next read seamlessly.

The first block of code reads from TProcess.Output and the second block is from the Synchronize procedure. As you can see if the last line gets truncated across  the READ_BYTES buffer size it is divided into two lines and shows that way in a memo field. Is there some way to check the Memory Stream to see if the last line in the buffer was truncated? Would a TStringStream or another kind of stream be better? The method I am using is adapted from the wiki example at  http://wiki.freepascal.org/Executing_External_Programs#Reading_large_output

On the TStringList side of things is there way to know if the last string in a TStringList is truncated in some way?


Code: [Select]
    while FProcess.Running do //process still running on loop entry
    begin
      FMemStream.SetSize(FBytesRead + READ_BYTES);
      FNumBytes := FProcess.Output.Read((FMemStream.Memory + FBytesRead)^, READ_BYTES);
      if FNumBytes > 0 then
        Synchronize(@UpdateOutput)
      else
        break;
    end;

Code: [Select]
  if FNumBytes < 1 then
    Exit;
  FBytesRead := FBytesRead + FNumBytes;
  Inc(FTotalBytesRead, FNumBytes);
  FMemStream.SetSize(FBytesRead);
  pOL := TStringList.Create;
  pOL.LoadFromStream(FMemStream);
  ProcManager.SendStrings(pOL, processOutput);
  pOL.Free;
  FBytesRead := 0;
  FNumBytes :=0;
  FMemStream.SetSize(0);
Lazarus 3.0/FPC 3.2.2

Leledumbo

  • Hero Member
  • *****
  • Posts: 8747
  • Programming + Glam Metal + Tae Kwon Do = Me
I don't know why you design SendStrings to accept TStringList instead of String, in which I would preprocess pOL.Text by trimming extranous new line before sending it.

vfclists

  • Hero Member
  • *****
  • Posts: 1013
    • HowTos Considered Harmful?
I don't know why you design SendStrings to accept TStringList instead of String, in which I would preprocess pOL.Text by trimming extranous new line before sending it.

How do I test the Text or string property to find out if the last line has been split? Can it be checked in the MemoryStream? There are no extraneous new lines in the incoming output, but the conversion to strings obviously appends a new line to the last string which gets split.

The StringList is added to a Memo's Lines property. I guess the main quesiton does appending text to a TStrings or TMemo add a new line character to the end or not? The only thing i can think of is to do someting like Memo.Lines.Text := Memo.Lines.Text + pol.Text
Lazarus 3.0/FPC 3.2.2

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Typically you create a cyclic buffer where you "add" the new read bytes from TProcess stream, and get out complete lines. (by scanning for (#13)#10)

rvk

  • Hero Member
  • *****
  • Posts: 6112
Or send the buffer of the FMemStream (FMemStream.Memory) and receive it on the other end in a FMemStream. TMemoryStream has a ReadAnsiString so you can convert the MemoryStream to directly add to your TMemo.

I think it's best to do as little as possible to your original data. Putting it in a MemoryStream, transfering it to a StringList, Sending it, Receiving it and adding the StringList to the memo seems a bit much.

FMemStream -> Send Stream -> Receive Stream -> add .ReadAnsiString to Memo.
« Last Edit: May 05, 2015, 07:09:53 pm by rvk »

vfclists

  • Hero Member
  • *****
  • Posts: 1013
    • HowTos Considered Harmful?
Is the TMemoryStream a raw byte buffer?

Because if it is then I will try testing of the last characters, ie , whether Size and Size -1 are #13 and or #13#10 and if they are not,  I will know that the last string has been truncated. I can then hold it back and prepend it to the first string of the next read.
Lazarus 3.0/FPC 3.2.2

Leledumbo

  • Hero Member
  • *****
  • Posts: 8747
  • Programming + Glam Metal + Tae Kwon Do = Me
How do I test the Text or string property to find out if the last line has been split?
No need, .Text will always append a newline, so what you have to do is just trim that new line (but don't use Trim(Right), as it may remove ALL whitespace characters from the end).
Can it be checked in the MemoryStream?
Nope, it never happens there.
There are no extraneous new lines in the incoming output, but the conversion to strings obviously appends a new line to the last string which gets split.

The StringList is added to a Memo's Lines property. I guess the main quesiton does appending text to a TStrings or TMemo add a new line character to the end or not? The only thing i can think of is to do someting like Memo.Lines.Text := Memo.Lines.Text + pol.Text
Exactly that's the cause of extranous newline. If you wish, take a look at my project how I do command line application wrapper. See how formmain.pp and ajbridgeprocessthread.pp communicate.

Cyrax

  • Hero Member
  • *****
  • Posts: 836
Or send the buffer of the FMemStream (FMemStream.Memory) and receive it on the other end in a FMemStream. TMemoryStream has a ReadAnsiString so you can convert the MemoryStream to directly add to your TMemo.

I think it's best to do as little as possible to your original data. Putting it in a MemoryStream, transfering it to a StringList, Sending it, Receiving it and adding the StringList to the memo seems a bit much.

FMemStream -> Send Stream -> Receive Stream -> add .ReadAnsiString to Memo.

TMemoryStream.ReadAnsiString expects to read first string length information (saved as 4 byte sequence) from current stream position and after that the whole string. So using it won't be acceptable solution in this situation because there is no string lengths saved in first place. TProcess reads output from console program in raw without processing it further.

rvk

  • Hero Member
  • *****
  • Posts: 6112
No need, .Text will always append a newline, so what you have to do is just trim that new line (but don't use Trim(Right), as it may remove ALL whitespace characters from the end).
No, Memo.Text := Memo.Text + string will not add a newline.

TMemoryStream.ReadAnsiString expects to read first string length information (saved as 4 byte sequence) from current stream position and after that the whole string. So using it won't be acceptable solution in this situation because there is no string lengths saved in first place. TProcess reads output from console program in raw without processing it further.
You're correct. ReadAnsiString is not the correct way (like I initially thought). I thought it would read an AnsiString from the Stream but it doesn't. It reads a typed string from the stream (indeed with the size prefixed like the old pascal string). It's only used in combination with TMemoryStream.WriteAnsiString (which puts the length on the stream too).

I still feel like directly using the TMemoryStream can be used correctly. Look at the following code (Button1, Button2 and a Memo1 on a form):

Button1 just adds "text. " to the memo. It will all be on one line so it demonstrates Text := Text + string does not add newlines.

Button2 puts a "line 1" with linebreak and "line 2" without linebreak in the memo. When using Button2 multiple times you'll see that after "line 2" there is really no linebreak. So you can use this FMemStream as a rawbuffer type. There is absolutely no need to check for #13 anywhere. It just directs all raw output to the memo and there is no issue when that output hasn't a ending linebreak. The next portion will just be added to the previous line.

Code: [Select]
procedure TForm1.Button1Click(Sender: TObject);
begin
  Memo1.Text := Memo1.Text + 'text. ';
end;

function MemoryStreamToString(M: TMemoryStream): AnsiString;
begin
  SetString(Result, PAnsiChar(M.Memory), M.Size);
end;

procedure TForm1.Button2Click(Sender: TObject);
var
  FMemStream: TMemoryStream;
  Str: AnsiString;
begin
  FMemStream := TMemoryStream.Create;
  try

    FMemStream.Clear;
    str := 'lines 1' + #13;
    FMemStream.Write(str[1], Length(str));
    str := 'lines 2';
    FMemStream.Write(str[1], Length(str));

    Str := MemoryStreamToString(FMemStream);

    Memo1.Text := Memo1.Text + Str;

  finally
    FMemStream.Free;
  end;
end;

 

TinyPortal © 2005-2018