Recent

Author Topic: [Solved] TMemDataSet, TBufDataSet file saving problem  (Read 5320 times)

egsuh

  • Hero Member
  • *****
  • Posts: 1273
[Solved] TMemDataSet, TBufDataSet file saving problem
« on: March 29, 2017, 05:33:07 pm »
Hi,

I defined a Memo / Blob type field in TMemDataSet and TBufDataSet, and stored data in following ways. The data to be stored are long text file, e.g. TStrings.Text.

1)     mds1.fieldbyname('memofield').AsString := StringtoStore;
2)     mds1.fieldbyname('memofield').Value := StringtoStore;
3)     (mds1.fieldbyname('memofield') as TBlobField).LoadfromStream (AStream);


All of the above three methods works fine with both mds and bds. I can see the content of the field as I entered. That is, data is stored correctly. 

Problem is that once I save it and load later, the content of memo/blob fields are not contained.

I tried both "Savetofile" + "Loadfromfile", and "savetostream" + "loadfromstream".

I think I had no problem in saving and loading the database file as other fields work fine, only Blob / Memo type fields data are lost --- not sure which one of storing or retrieving is not working correctly.

Can anyone help me?
« Last Edit: April 01, 2017, 06:12:07 pm by egsuh »

wp

  • Hero Member
  • *****
  • Posts: 11854
Re: TMemDataSet, TBufDataSet file saving problem
« Reply #1 on: March 29, 2017, 06:38:44 pm »
Sorry - I am not going to answer your question, but I want to put your attention to another issue with TBufDataset which I learned recently: If you save a few thousand TBufDatset records using its SaveToFile or SaveToStream methods the process gets extremely slow. To avoid this you must call MergeChangeLog always after posting a few hundred records and/or before the call to SaveToFile/Stream. In the following demo, comment the line "TBufDataset(FExportDataset).MergeChangeLog" to see the difference:

Code: Pascal  [Select][+][-]
  1. program project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. {.$DEFINE USE_MEM_DATASET}
  6.  
  7. uses
  8.   SysUtils, classes, db, memds, bufdataset;
  9.  
  10. const
  11.   TABLENAME = 'people'; //name for the database table, extension will be added
  12.   DATADIR = 'data'; //subdirectory where database is stored
  13.  
  14. const
  15.   NUM_RECORDS = 5000;
  16.  
  17. var
  18.   FExportDataset: TDataset;
  19.  
  20. procedure CreateDatabase;
  21. var
  22.   i: Integer;
  23.   fn: String;
  24.   stream: TMemoryStream;
  25.   t: TDateTime;
  26. begin
  27.   ForceDirectories(DATADIR);
  28.  
  29.   fn := DATADIR + DirectorySeparator + TABLENAME + '.db';
  30.   DeleteFile(fn);
  31.  
  32.   {$IFDEF USE_MEM_DATASET}
  33.   FExportDataset := TMemDataset.Create(nil);
  34.   {$ELSE}
  35.   FExportDataset := TBufDataset.Create(nil);
  36.   {$ENDIF}
  37.  
  38.   FExportDataset.FieldDefs.Add('Last name', ftString, 15);
  39.   FExportDataset.FieldDefs.Add('First name', ftString, 10);
  40.   FExportDataset.FieldDefs.Add('City', ftString, 15);
  41.   FExportDataset.FieldDefs.Add('Birthday', ftDate);
  42.   FExportDataset.FieldDefs.Add('Salary', ftCurrency);
  43.   FExportDataset.FieldDefs.Add('Work begin', ftDateTime);
  44.   FExportDataset.FieldDefs.Add('Work end', ftDateTime);
  45.   FExportDataset.FieldDefs.Add('Size', ftFloat);
  46.   {$IFNDEF USE_MEM_DATASET}
  47.   TBufDataset(FExportDataset).CreateDataset;
  48.   {$ENDIF}
  49.  
  50.   FExportDataset.Open;
  51.  
  52.   // Random data
  53.   for i:=1 to NUM_RECORDS do begin
  54.     if (i mod 100 = 0) then begin
  55.       WriteLn(Format('Adding record %d...', [i]));
  56.       {$IFNDEF USE_MEM_DATASET}
  57.       TBufDataset(FExportDataset).MergeChangeLog;
  58.       {$ENDIF}
  59.     end;
  60.     FExportDataset.Insert;
  61.     // just dummy data...
  62.     FExportDataset.FieldByName('Last name').AsString := 'A';
  63.     FExportDataset.FieldByName('First name').AsString := 'B';
  64.     FExportDataset.FieldByName('City').AsString := 'C';
  65.     FExportDataset.FieldByName('Birthday').AsDateTime := 0;
  66.     FExportDataset.FieldByName('Salary').AsFloat := 0;
  67.     FExportDataset.FieldByName('Size').AsFloat := 0;
  68.     FExportDataSet.FieldByName('Work begin').AsDateTime := 0;
  69.     FExportDataSet.FieldByName('Work end').AsDateTime := 0;
  70.     FExportDataset.Post;
  71.   end;
  72.  
  73.   WriteLn('Saving...');
  74.   t := now;
  75.   stream := TMemoryStream.Create;
  76.   try
  77.     {$IFDEF USE_MEM_DATASET}
  78.     TMemDataset(FExportDataset).SaveToStream(stream);
  79.     {$ELSE}
  80.     TBufDataset(FExportDataset).SaveToStream(stream);
  81.     {$ENDIF}
  82.     stream.Position := 0;
  83.     WriteLn('Written to memory stream: ', FormatFloat('0.000 s', (now - t) * SECONDS_PER_DAY));
  84.     stream.SaveToFile(fn);
  85.   finally
  86.     stream.Free;
  87.   end;
  88.   Writeln('Done. Total time needed for saving: ', FormatFloat('0.000 s', (now - t) * SECONDS_PER_DAY));
  89.  
  90.   FExportDataset.Close;
  91.  
  92.   writeLn(Format('Created file "%s" in folder "data".', [
  93.     ExtractFileName(fn), ExtractFileDir(fn)
  94.   ]));
  95.   FExportDataset.Free;
  96. end;
  97.  
  98. begin
  99.   CreateDatabase;
  100.  
  101.   WriteLn;
  102.   WriteLn('Press ENTER to close.');
  103.   ReadLn;
  104. end.

egsuh

  • Hero Member
  • *****
  • Posts: 1273
Re: TMemDataSet, TBufDataSet file saving problem
« Reply #2 on: March 29, 2017, 06:50:08 pm »
Thank you wp,

This is much help, even though I would not make 5,000 record tables.
Anyway does TMemDataSet show no difference in saving time?

Regards,

wp

  • Hero Member
  • *****
  • Posts: 11854
Re: TMemDataSet, TBufDataSet file saving problem
« Reply #3 on: March 29, 2017, 06:54:27 pm »
Sorry, I forgot this: TMemDataset does not have this issue. It saves immediately fast without any further calls. (You can activate the devine USE_MEM_DATASET in the demo to work with TMemDataset instead of TBufDataset).

LacaK

  • Hero Member
  • *****
  • Posts: 691
Re: TMemDataSet, TBufDataSet file saving problem
« Reply #4 on: March 31, 2017, 07:37:03 pm »
Problem is that once I save it and load later, the content of memo/blob fields are not contained.

I tried both "Savetofile" + "Loadfromfile", and "savetostream" + "loadfromstream".

I think I had no problem in saving and loading the database file as other fields work fine, only Blob / Memo type fields data are lost --- not sure which one of storing or retrieving is not working correctly.

TMemDataSet is lightweight in-memory dataset and saving/loading BLOB field is not implemented there.
Use TBufDataset.

egsuh

  • Hero Member
  • *****
  • Posts: 1273
Re: TMemDataSet, TBufDataSet file saving problem
« Reply #5 on: April 01, 2017, 04:52:52 pm »
TMemDataSet is lightweight in-memory dataset and saving/loading BLOB field is not implemented there.
Use TBufDataset.

Thank you. I'll try it again, but TBufDataSet has the same problem. 
SQLite + sqlite3dataset seems good enough as alternative.

egsuh

  • Hero Member
  • *****
  • Posts: 1273
Re: TMemDataSet, TBufDataSet file saving problem
« Reply #6 on: April 01, 2017, 05:15:06 pm »
LacaK,

I did a quick test and TBufDataSet seems to work fine.
I must have mistaken something in previous tests.

Thank you very much.

 

TinyPortal © 2005-2018