Recent

Author Topic: Is there an indexable stream object with random block sizes?  (Read 736 times)

jamie

  • Hero Member
  • *****
  • Posts: 6032
Is there an indexable stream object with random block sizes?
« on: December 02, 2023, 10:04:52 pm »
To be more specific, is there a readymade class/object that can index non- uniformed sizes of data pointing at a stream?
 
Before I go on a tangent and write my own, I like to first explore what is out there in the libs that comes with the compiler of Lazarus.

  Basically, I supply a stream, be it a file stream or memory stream but stream it is.

  I add nodes for example that has chunks of data, each having its own header attached to the chunk. This data could be anything, but it will go to a stream and the object/Class will generate an index to where these items start in the stream.

 After looking around at some readymade items, I see that I can use a Tlist for example, to store the offset pointers to a stream. would be nice if there was a readymade class that did all of that so I wouldn't have to make it myself, you know reinvent the wheel!


The only true wisdom is knowing you know nothing

jamie

  • Hero Member
  • *****
  • Posts: 6032
Re: Is there an indexable stream object with random block sizes?
« Reply #1 on: December 03, 2023, 03:37:13 am »
I came up with a test app using a generic embedded within a class.
This basically create a table of my Record that contains the data per chunk of data in the stream.
The actual use case will be a little different of course.

Code: Pascal  [Select][+][-]
  1. unit Unit1;
  2.  
  3. {$mode objfpc}{$H+}
  4. {$modeSwitch AdvancedRecords}
  5. interface
  6.  
  7. uses
  8.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,fgl;
  9.  
  10. type
  11.  TMyStreamBlob = Class
  12.    Type
  13.      TField = Record
  14.        fDate:TDateTime;
  15.        fSize:Integer;
  16.        fIndex:Integer;//Position of Stream;
  17.        //Some other stuff etc.
  18.      class operator =(A,B:TField):Boolean;
  19.       End;
  20.   TIndex = Specialize TFpgList<TField>;
  21.   Private
  22.     fStream:TStream;
  23.     fownStream:Boolean;
  24.    fIndex :TIndex;
  25.   public
  26.    Function AddBlob(APointer:Pointer;ASize:Integer):Integer;
  27.    constructor Create(AStream:Tstream;aOwnit:Boolean=false);Virtual;
  28.    destructor  Destroy;override;
  29.  end;
  30.  
  31.   { TForm1 }
  32.  
  33.   TForm1 = class(TForm)
  34.     Button1: TButton;
  35.     procedure Button1Click(Sender: TObject);
  36.   private
  37.  
  38.   public
  39.  
  40.   end;
  41.  
  42. var
  43.   Form1: TForm1;
  44.  
  45. implementation
  46.  
  47. {$R *.lfm}
  48.  
  49. { TForm1 }
  50.  
  51. procedure TForm1.Button1Click(Sender: TObject); //Test  it.
  52. Const ABlobField:String='Test';
  53. Var
  54.   Blobs:TMyStreamBlob;
  55. begin
  56.   Blobs := TMyStreamBlob.Create(TMemoryStream.Create, True);
  57.   Blobs.AddBlob(@ABlobField,Length(AblobField));//add a blob;
  58.   Caption :=Blobs.fIndex.Count.Tostring+' records,'+Blobs.fStream.Size.Tostring+' Bytes';// test data.
  59.   Caption := Caption+' At '+DateTimeTostr(Blobs.Findex.Items[0].fDate);
  60.   Blobs.Free;
  61. end;
  62. Function TmyStreamBlob.AddBlob(APointer:Pointer;ASize:Integer):Integer;
  63. Var
  64.   A:TField;
  65. begin
  66.   A.fDate := Now;
  67.   A.fSize := ASize;
  68.   A.fIndex := fStream.Position;
  69.   fstream.WriteBuffer(aPointer^,ASize);
  70.   fIndex.Add(A);//Add after incase of stream error.
  71. end;
  72. Destructor TMyStreamBlob.Destroy;
  73. Begin
  74.   if fOwnStream Then FreeAndNil(fStream);
  75.   fIndex.Free;
  76.   inherited;
  77. end;
  78.  
  79. Constructor TMySTreamBlob.Create(AStream:TStream;AOwnit:Boolean=false);
  80. Begin
  81.  fIndex := Tindex.Create;
  82.  FStream := AStream;  fOwnStream := AOWnit;
  83. End;
  84. class operator TMyStreamBlob.Tfield.=(A,B:TField):Boolean;
  85. Begin
  86.    Result := CompareMem(@A,@B,SizeOf(TField));
  87. End;
  88. end.
  89.  

Ill attach the project, too.
The only true wisdom is knowing you know nothing

jamie

  • Hero Member
  • *****
  • Posts: 6032
Re: Is there an indexable stream object with random block sizes?
« Reply #2 on: December 03, 2023, 03:44:14 am »
I wanted to ask, maybe a fpc DEV.

I tried to put a reference to the stream within the Constructor.Create, It allow this but when I went to actually call the constructor It would not allow it..

 Constructor Create(Var AStream:TStream);

Later on

  MyBlob := TBlobStream.Create(AStreamElsewhwere);

Would not allow it.

 Clamed to be a type error.

This would be nice because I have a OWNED default in there where I can free the stream inside the blob code and set the stream to NIL at the same time. This way if an attempt with a FREE is executed on that same stream it will just ignore it.

This is in FPC 3.0.4 if that makes any difference.
The only true wisdom is knowing you know nothing

af0815

  • Hero Member
  • *****
  • Posts: 1279
Re: Is there an indexable stream object with random block sizes?
« Reply #3 on: December 03, 2023, 08:52:42 am »
XML or JSON for this ?! Because it sounds for me Like serialsation of data/components.
regards
Andreas

WooBean

  • Full Member
  • ***
  • Posts: 227
Re: Is there an indexable stream object with random block sizes?
« Reply #4 on: December 03, 2023, 10:08:58 am »
To be more specific, is there a readymade class/object that can index non- uniformed sizes of data pointing at a stream?
 ...

IMO, you are describing a pair of existing  classes (as old as object Pascal is) - TStream and TList. Just use them both.
Platforms: Win7/64, Linux Mint Ulyssa/64

cdbc

  • Hero Member
  • *****
  • Posts: 968
    • http://www.cdbc.dk
Re: Is there an indexable stream object with random block sizes?
« Reply #5 on: December 03, 2023, 10:24:58 am »
Hi
Would it help to take a look at, how TDbf handles ftMemo fields internally?!?Regards Benny
If it ain't broke, don't fix it ;)
PCLinuxOS(rolling release) 64bit -> KDE5 -> FPC 3.2.2 -> Lazarus 2.2.6 up until Jan 2024 from then on it's: KDE5/QT5 -> FPC 3.3.1 -> Lazarus 3.0

jamie

  • Hero Member
  • *****
  • Posts: 6032
Re: Is there an indexable stream object with random block sizes?
« Reply #6 on: December 03, 2023, 04:03:29 pm »
The TEST string was just a data source for the test app.

data will come from various sources like Images, byte blobs. They cannot be stored in text format due to speed and space.

$MS has a component int their .NET framework that is basically a buffered stream of data that is indexed and can be compressed on the fly along with using file caching or memory.

 Their class has a Lock on it so that it can't be used to build descendant classes, but it's designed to allow any form of data.

 The idea is to have a class much like a TImageList but this list will accept any form of data so it's going to be a TBlobList for example.

 I want to be able to actually use the streaming abilities of for example, PNG, JPG, BMP or just plain bytes or from some custom formats and on top of that, I was using a the zstream to compress it as an option.



The only true wisdom is knowing you know nothing

jamie

  • Hero Member
  • *****
  • Posts: 6032
Re: Is there an indexable stream object with random block sizes?
« Reply #7 on: December 03, 2023, 04:16:54 pm »
While I am here.

Any thoughts on the issue that I can't pass a Stream into the constructor.create as a VAR reference so I can gain direct access to the source of the stream within the class?

Code: Pascal  [Select][+][-]
  1. Constructor Create(Var AStream:TStream; Owned:Boolean = false);
  2. Begin
  3.  fInternalStream := AStream;
  4.  // Later in the destructor I can do a FreeAndNil and it should result in the source stream getting freeAndNil
  5. End;
  6.  

The only true wisdom is knowing you know nothing

 

TinyPortal © 2005-2018