[SOLVED] Indy 10 sends more data than expected?

[aidv]

Ok so last couple of weeks I've been on here trying to get some help regarding my project.

I've figured most of it out, but some things remain to be solved.

I use Indy 10 TCP for my communication as I find it to be very reliable, which is good, however I noticed that when I send a TMemoryStream with the size of 47 bytes, the receiver (and XCode project for iOS) receives 56 bytes.

And when I send a memory stream of lets say some 100 thousand bytes, the difference might be up to several tens of thousands of bytes.

What is going on here?

[marcov]

How do you measure this? How does the extra data look like?

[aidv]

I measure it simply by logging the size of the TMemoryStream to be sent and the length of the NSData that has been received.

I can't really know what the extra data looks like as it's difficult to check it with large data.

For smaller TMemoryStreams I don't really have any problems, but with the very large data, it becomes problematic.

I end all my data "packets" with a specific string looking like this: <packetEnd>

and when I extract a substring of my total data packet the very last part of the packet is actually ending with <packetEnd>

and when I extract all data from index 0 up until the beginning of my file-data, everything is OK as well, starting with <packetStart> + some extra human-readable text and finally a ':' character followed by the file-data.

so everything seems to be OK, but still, too much data in it somewhere.

I have checked the size of the file on the server, and all attributes are correct.

So I don't really know whats going on here.

EDIT:

Here is my sending code:

Code: [Select]

procedure THoster.SendData(aData: TMemoryStream; AAddress: string);
  var i, u: integer;
      sendIndex: word;
      idb: TIdBytes;
      ctx : TIdContext;
begin
  aData.Position := 0;
  ReadTIdBytesFromStream(aData,idb,aData.Size,0);
  with fTCP.Contexts.LockList do
  begin
    for i := 0 to Count -1 do
    begin
      ctx := TIdContext(Items[i]);
      if AAddress <> '' then
      begin
        if ctx.Binding.PeerIP = AAddress then
        begin
          ctx.Connection.IOHandler.Write(idb, aData.Size, 0);
          break;
        end;
      end
      else ctx.Connection.IOHandler.Write(idb, aData.Size, 0);
    end;
  end;
end; 

[aidv]

Ok I think I solved the problem by adding the following line to my code:

Code: [Select]

ctx.Connection.IOHandler.WriteBufferFlush;
Now my source looks like this:

Code: [Select]

procedure THoster.SendData(aData: TMemoryStream; AAddress: string);
  var i, u: integer;
      sendIndex: word;
      idb: TIdBytes;
      ctx : TIdContext;
begin
  aData.Position := 0;
  ReadTIdBytesFromStream(aData,idb,aData.Size,0);
  with fTCP.Contexts.LockList do
  begin
    for i := 0 to Count -1 do
    begin
      ctx := TIdContext(Items[i]);
      ctx.Connection.IOHandler.WriteBufferFlush;
      if AAddress <> '' then
      begin
        if ctx.Binding.PeerIP = AAddress then
        begin
          ctx.Connection.IOHandler.Write(idb, aData.Size, 0);
          break;
        end;
      end
      else ctx.Connection.IOHandler.Write(idb, aData.Size, 0);
    end;
  end;
end;
Now the size is correct on both sides, at least for small packets. Hopefully this solves my entire problem.

[derek.john.evans]

Just a couple of things I see:

Why are you reading the stream into a TIdBytes? Why not just write the stream directly?


I also dont see a  Contexts.UnLockList;  // <<<< Important.

[aidv]

Why hello Geepster.

How do I do that? The method asks for a TIdBytes and I found an example on the interwebs where TIdBytes was used, so I assumed that that was the correct way to do it?

You ar right, I forgot to add UnlockList.

And I see now that for very large data, the problem remains.

And oh I just noticed that it seems like parts of the next packet in line to be sent seems to be part of the current packet.

I guess I have the same problem that I had a while back.

[rvk]

And oh I just noticed that it seems like parts of the next packet in line to be sent seems to be part of the current packet.

You don't send any size with your data. How does the receiving end know when the data is complete?? Does it have a timeout? Do you close the connection? Did you use a buffering-option somewhere?

For a buffering problem see here:
http://stackoverflow.com/questions/25438309/indy10-idtcpclients-iohandler-send-additional-data
(where you should use WriteBufferClose but I'm not sure you need buffering at all, see the end of the answer there)

[aidv]

I do send size with my data, it's right after <packetStart>.

I think that I need to make a packet queue, so when I send a new packet, it's put on hold until the client sends a "ready" signal, and then server flushes everything and and sends the next packet.

Is that a good approach?

[derek.john.evans]

I'm a little confused by your design. Are you sending files from a server to multiple clients?

Is that what you were aiming for? I thought you were sending from clients to a server?

TIdIOHandlerSocket (which is available for both clients & servers) has a bunch of Writers including:

Code: Pascal  [Select][+][-]

1. function WriteFile(const AFile: String; AEnableTransferFile: Boolean = False): Int64; override;  
2.  

and

Code: Pascal  [Select][+][-]

1. procedure Write(AStream: TStream; ASize: TIdStreamSize = 0; AWriteByteCount: Boolean = False); overload; virtual;  
2.  

The latter is the one Im talking about, which has AWriteByteCount (ie: sending the size)

But, anyway, by iterating the contexts, your code looks more like a broadcaster.

[aidv]

Well, yes.

My server can receive data from multiple clients, but also send data to multiple clients, if needed that is.

It is not necessary, but it can.

Oh I see, but I have to ask,
is then sending of the size something that is Indy specific, or is it part of the TCP protocol?

and finally: it can be a broadcaster, if the parameter AAddress is left empty, or else it will try to send to the specified address.

[rvk]

I do send size with my data, it's right after <packetStart>.

Yes... but that's not used by the receiving program (is my guess). Or is your receiving program checking the size during the communication?

I thought the third parameter of your Write() should be a boolean (true/false). If true it sends the size of the stream before sending the actual data. If the receiving end uses a similar function to first check the size it needs to receive and then actually receiving that exact number of bytes it's all done automatically.

The sending of the size of the stream is indy/program specific. But if wouldn't be hard to program into the server that you first need to receive the size of a stream and after that the actual stream. The server can complete the package after receiving the exact number of bytes.

I think that I need to make a packet queue, so when I send a new packet, it's put on hold until the client sends a "ready" signal, and then server flushes everything and and sends the next packet.

Did you break up that file you want to send (of hundred MBs) into multiple streams already? How large are those packets/streams? And I don't think you need to send the size of the data in data <packet/start/end>. If your packets are pre-defined (lets say 1MB per packet) the server can just check that 1MB against what it actually received. I thought you wanted to check the validity of the actual received data so you need to send the MD5 (or hash) with the data. But you could do that separately after (or before) sending the package.

[derek.john.evans]

Oh I see, but I have to ask,
is then sending of the size something that is Indy specific, or is it part of the TCP protocol?

It is implemented by indy as a basic package/chunk transfer. If all your data is stored as TStream's, then, its easy to send files using TFileStream, strings using TStringStream, and binary data using TMemoryStream.

Im unsure how you have your chunks sent now, but, a simple binary integer is the way to go, which is what indy can do for TStreams.

Using terminators for chunks is a bad idea, (if thats what you are doing).

[derek.john.evans]

The sending of the size of the stream is indy/program specific. But if wouldn't be hard to program into the server that you first need to receive the size of a stream and after that the actual stream. The server can complete the package after receiving the exact number of bytes.

Hi, rvk. Yep, Indy already has a paired Write/Read for chunked TStreams.

Code: [Select]

procedure TIdIOHandler.ReadStream(AStream: TStream; AByteCount: TIdStreamSize;
  AReadUntilDisconnect: Boolean);   

If AByteCount = cSizeUnknown (ie: -1), then a byte size is read from the connection.

[aidv]

rvk:
The size is used to check if the length of the data is correct.

The size of the packet can be very large, but also very small.

Right now the packets I am testing with are around 300kb in size, just for a test.

And the packets contain all essential information, such as start/endsequences, size of the packet, size of data.

Geepster:
What is a terminator? You mean like my <packetEnd> string at the end of my chunks?

Actually, if a packet is very large, it is automatically sent in several chunks, and I am utilizing that.

Ofcourse if I need to send several MB of data, then I split that data in to independent chunks and then send each chunk as a packet.

[derek.john.evans]

Geepster:
What is a terminator? You mean like my <packetEnd> string at the end of my chunks?
Actually, if a packet is very large, it is automatically sent in several chunks, and I am utilizing that.

Yep. It isn't clear from your SendData code whether you send the size of the TStream or not. If you are, then cool. But, you also said:

I end all my data "packets" with a specific string looking like this: <packetEnd>

I, personally don't like packet terminators which require you to read data, and search for a string, then chop out the chunk, and then continue reading from the end of the last chunk to get the next chunk.

I know some people love that kinda stuff, but, it just does my head in.