Is there an indexable stream object with random block sizes?

[jamie]

To be more specific, is there a readymade class/object that can index non- uniformed sizes of data pointing at a stream?
 
Before I go on a tangent and write my own, I like to first explore what is out there in the libs that comes with the compiler of Lazarus.

  Basically, I supply a stream, be it a file stream or memory stream but stream it is.

  I add nodes for example that has chunks of data, each having its own header attached to the chunk. This data could be anything, but it will go to a stream and the object/Class will generate an index to where these items start in the stream.

 After looking around at some readymade items, I see that I can use a Tlist for example, to store the offset pointers to a stream. would be nice if there was a readymade class that did all of that so I wouldn't have to make it myself, you know reinvent the wheel!

[jamie]

I came up with a test app using a generic embedded within a class.
This basically create a table of my Record that contains the data per chunk of data in the stream.
The actual use case will be a little different of course.

Code: Pascal  [Select][+][-]

1. unit Unit1;
2.  
3. {$mode objfpc}{$H+}
4. {$modeSwitch AdvancedRecords}
5. interface
6.  
7. uses
8.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,fgl;
9.  
10. type
11.  TMyStreamBlob = Class
12.    Type
13.      TField = Record
14.        fDate:TDateTime;
15.        fSize:Integer;
16.        fIndex:Integer;//Position of Stream;
17.        //Some other stuff etc.
18.      class operator =(A,B:TField):Boolean;
19.       End;
20.   TIndex = Specialize TFpgList<TField>;
21.   Private
22.     fStream:TStream;
23.     fownStream:Boolean;
24.    fIndex :TIndex;
25.   public
26.    Function AddBlob(APointer:Pointer;ASize:Integer):Integer;
27.    constructor Create(AStream:Tstream;aOwnit:Boolean=false);Virtual;
28.    destructor  Destroy;override;
29.  end;
30.  
31.   { TForm1 }
32.  
33.   TForm1 = class(TForm)
34.     Button1: TButton;
35.     procedure Button1Click(Sender: TObject);
36.   private
37.  
38.   public
39.  
40.   end;
41.  
42. var
43.   Form1: TForm1;
44.  
45. implementation
46.  
47. {$R *.lfm}
48.  
49. { TForm1 }
50.  
51. procedure TForm1.Button1Click(Sender: TObject); //Test  it.
52. Const ABlobField:String='Test';
53. Var
54.   Blobs:TMyStreamBlob;
55. begin
56.   Blobs := TMyStreamBlob.Create(TMemoryStream.Create, True);
57.   Blobs.AddBlob(@ABlobField,Length(AblobField));//add a blob;
58.   Caption :=Blobs.fIndex.Count.Tostring+' records,'+Blobs.fStream.Size.Tostring+' Bytes';// test data.
59.   Caption := Caption+' At '+DateTimeTostr(Blobs.Findex.Items[0].fDate);
60.   Blobs.Free;
61. end;
62. Function TmyStreamBlob.AddBlob(APointer:Pointer;ASize:Integer):Integer;
63. Var
64.   A:TField;
65. begin
66.   A.fDate := Now;
67.   A.fSize := ASize;
68.   A.fIndex := fStream.Position;
69.   fstream.WriteBuffer(aPointer^,ASize);
70.   fIndex.Add(A);//Add after incase of stream error.
71. end;
72. Destructor TMyStreamBlob.Destroy;
73. Begin
74.   if fOwnStream Then FreeAndNil(fStream);
75.   fIndex.Free;
76.   inherited;
77. end;
78.  
79. Constructor TMySTreamBlob.Create(AStream:TStream;AOwnit:Boolean=false);
80. Begin
81.  fIndex := Tindex.Create;
82.  FStream := AStream;  fOwnStream := AOWnit;
83. End;
84. class operator TMyStreamBlob.Tfield.=(A,B:TField):Boolean;
85. Begin
86.    Result := CompareMem(@A,@B,SizeOf(TField));
87. End;
88. end.
89.  

Ill attach the project, too.

[jamie]

I wanted to ask, maybe a fpc DEV.

I tried to put a reference to the stream within the Constructor.Create, It allow this but when I went to actually call the constructor It would not allow it..

 Constructor Create(Var AStream:TStream);

Later on

  MyBlob := TBlobStream.Create(AStreamElsewhwere);

Would not allow it.

 Clamed to be a type error.

This would be nice because I have a OWNED default in there where I can free the stream inside the blob code and set the stream to NIL at the same time. This way if an attempt with a FREE is executed on that same stream it will just ignore it.

This is in FPC 3.0.4 if that makes any difference.

[af0815]

XML or JSON for this ?! Because it sounds for me Like serialsation of data/components.

[WooBean]

To be more specific, is there a readymade class/object that can index non- uniformed sizes of data pointing at a stream?
 ...

IMO, you are describing a pair of existing  classes (as old as object Pascal is) - TStream and TList. Just use them both.

[cdbc]

Hi
Would it help to take a look at, how TDbf handles ftMemo fields internally?!?Regards Benny

[jamie]

The TEST string was just a data source for the test app.

data will come from various sources like Images, byte blobs. They cannot be stored in text format due to speed and space.

$MS has a component int their .NET framework that is basically a buffered stream of data that is indexed and can be compressed on the fly along with using file caching or memory.

 Their class has a Lock on it so that it can't be used to build descendant classes, but it's designed to allow any form of data.

 The idea is to have a class much like a TImageList but this list will accept any form of data so it's going to be a TBlobList for example.

 I want to be able to actually use the streaming abilities of for example, PNG, JPG, BMP or just plain bytes or from some custom formats and on top of that, I was using a the zstream to compress it as an option.

[jamie]

While I am here.

Any thoughts on the issue that I can't pass a Stream into the constructor.create as a VAR reference so I can gain direct access to the source of the stream within the class?

Code: Pascal  [Select][+][-]

1. Constructor Create(Var AStream:TStream; Owned:Boolean = false);
2. Begin
3.  fInternalStream := AStream;
4.  // Later in the destructor I can do a FreeAndNil and it should result in the source stream getting freeAndNil
5. End;
6.