Recent

Author Topic: ZMSQL - TBufDataset SQL enhanced in-memory database  (Read 130033 times)

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #60 on: September 24, 2012, 05:23:17 pm »
@Tatamata & zeosmsql users: if you think sdfdataset has bugs, please feel free to add test cases for those bugs in tcsdfdata.pp
see
http://lazarus.freepascal.org/index.php/topic,18224.msg103314.html#msg103314
for details

If these test cases are accepted into FPC trunk, adapting sdfdataset to match these test cases will be justified.


Thanks,
BigChimp

Edit: sdfdataset is probably going to support RFC4180 CSV format instead of SDF. See
http://www.mail-archive.com/fpc-pascal@lists.freepascal.org/msg30087.html
« Last Edit: September 26, 2012, 10:34:37 am by BigChimp »
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

irw10062

  • New Member
  • *
  • Posts: 31
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #61 on: October 05, 2012, 05:21:24 am »
is ZMSQL required library / dependency ??

i see http://wiki.freepascal.org/ZMSQL

Quote
Dependencies / System Requirements
None
Status: Alpha

i test use TDbf and TBufDataset is success compile
but using TZMConnection, TZMQueryDataset this i get error in message arm-linux

Quote
C:\codetyphon\fpc\units\arm-linux\rtl\cprt0.o: In function `_start':
(.text+0x4c): undefined reference to `__libc_start_main'
C:\codetyphon\fpc\units\arm-linux\rtl\cprt0.o: In function `_haltproc_eabi':
(.text+0x88): undefined reference to `_fini'
C:\codetyphon\fpc\units\arm-linux\rtl\cprt0.o: In function `_haltproc_eabi':
(.text+0x90): undefined reference to `_init'
C:\codetyphon\fpc\units\arm-linux\rtl\clocale.o: In function `CLOCALE$_$GETFORMATSETTINGS$TFORMATSETTINGS_$$_GETLOCALECHAR$LONGINT$$CHAR':
clocale.pp:(.text.n_clocale$_$getformatsettings$tformatsettings_$$_getlocalechar$longint$$char+0x10): undefined reference to `nl_langinfo'
C:\codetyphon\fpc\units\arm-linux\rtl\clocale.o: In function `CLOCALE$_$GETFORMATSETTINGS$TFORMATSETTINGS_$$_GETLOCALESTR$LONGINT$$SHORTSTRING':
clocale.pp:(.text.n_clocale$_$getformatsettings$tformatsettings_$$_getlocalestr$longint$$shortstring+0x40): undefined reference to `nl_langinfo'
smartmobile.lpr(22) Error: Error while linking
smartmobile.lpr(22) Fatal: There were 1 errors compiling module, stopping


marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12645
  • FPC developer.
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #62 on: October 05, 2012, 09:58:11 am »

Quote
C:\codetyphon\fpc\units\arm-linux\rtl\cprt0.o: In function `_start':
(.text+0x4c): undefined reference to `__libc_start_main'
C:\codetyphon\fpc\units\arm-linux\rtl\cprt0.o: In function `_haltproc_eabi':
(.text+0x88): undefined reference to `_fini'
C:\codetyphon\fpc\units\arm-linux\rtl\cprt0.o: In function `_haltproc_eabi':
(.text+0x90): undefined reference to `_init'
C:\codetyphon\fpc\units\arm-linux\rtl\clocale.o: In function `CLOCALE$_$GETFORMATSETTINGS$TFORMATSETTINGS_$$_GETLOCALECHAR$LONGINT$$CHAR':
clocale.pp:(.text.n_clocale$_$getformatsettings$tformatsettings_$$_getlocalechar$longint$$char+0x10): undefined reference to `nl_langinfo'
C:\codetyphon\fpc\units\arm-linux\rtl\clocale.o: In function `CLOCALE$_$GETFORMATSETTINGS$TFORMATSETTINGS_$$_GETLOCALESTR$LONGINT$$SHORTSTRING':
clocale.pp:(.text.n_clocale$_$getformatsettings$tformatsettings_$$_getlocalestr$longint$$shortstring+0x40): undefined reference to `nl_langinfo'
smartmobile.lpr(22) Error: Error while linking
smartmobile.lpr(22) Fatal: There were 1 errors compiling module, stopping


These errors are related to your croscompiling setup ( link the proper ARM-linux libraries), not anything database related.

tatamata

  • Hero Member
  • *****
  • Posts: 804
    • ZMSQL - SQL enhanced in-memory database
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #63 on: November 20, 2012, 12:06:33 pm »
Hello guys,
anybody interested to continue development of zmsql package?

tatamata

  • Hero Member
  • *****
  • Posts: 804
    • ZMSQL - SQL enhanced in-memory database
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #64 on: January 13, 2013, 01:45:44 pm »
Hello everybody,
ZMSQL version 0.1.13 is uploaded to Lazarus CCR and available for download. This version works with FPC 2.7.1 in CodeTyphon v 3.10 Lazarus distribution. I didn't test it with older FPC versions, for which it might not work correctly...

Changelog:
ZMSQL version 0.1.13, 13.01.2013: by Zlatko Matić
* Few bug fixes.
* TZMbufdataset replaced with current TBufDataset as ancestor. Tested with current FPC 2.7.1 in CodeTyphon v. 3.10.
If this does not work for your FPC version, then replace TBufDataSet with TZMBufDataset as ancestor.
The TZMBufDataset will stay present latent inside the package, just in case that further development od TBufDataSet goes in direction incompatible with zmsql.
*Added procedure InternalRefresh;override;
TBufDataSet's InternalRefresh does troubles, so it is overriden to do nothing.
It seems that all functionalities which ZMQueryDataset needs are implemented inside TDataSet's Refresh method and that InternalRefresh is not needed at all.                                               

teos

  • Full Member
  • ***
  • Posts: 161
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #65 on: January 19, 2013, 09:17:35 pm »

UPDATE: TZMReferentialKey provides a link between master and slave dataset, enabling referential integrity (update/insert/delete) between them.


As far as I know, you describe the "TMasterdatalink" of TDataset. I wonder if you derived from that of maybe it's wise to consider.

tatamata

  • Hero Member
  • *****
  • Posts: 804
    • ZMSQL - SQL enhanced in-memory database
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #66 on: January 19, 2013, 10:16:39 pm »

UPDATE: TZMReferentialKey provides a link between master and slave dataset, enabling referential integrity (update/insert/delete) between them.


As far as I know, you describe the "TMasterdatalink" of TDataset. I wonder if you derived from that of maybe it's wise to consider.
Hm, interesting.
If I had known for TMasterdatalink existence, I would probably ulitilize it, but no, I was not aware of it at the time I was writing TZMReferentialKey.
Actually, this is the very first time I heard of it.
http://www.freepascal.org/docs-html/fcl/db/tmasterdatalink.html.

So, the answer is that TZMReferentialKey has been written from scratch.
There are probably many possible ways to accomplish master/detail relationship....
You could try to accomplish TMasterdatalink way in zmsql package if you wish.
Anyway, zmsql is free and open-source, so you can freely experiment....and even piyk some ideas to use in your Hashdataset project....

teos

  • Full Member
  • ***
  • Posts: 161
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #67 on: January 19, 2013, 10:52:37 pm »
The Hasdataset is not mine. ;)

I posted my remark because I can imagine that there are standard ways that are wise to be followed to keep everything working like it should. I don't doubt that there are more than one way. Only.. the closer you stay to the "standard" ways, the easier it is for others to follow and to help.

I have my own database project and get that finished first. :)

Vincent Snijders

  • Administrator
  • Hero Member
  • *
  • Posts: 2661
    • My Lazarus wiki user page
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #68 on: February 15, 2013, 06:27:59 pm »
Is this component still in alpha status, as is mentioned on the wiki or has it been matured in the last 18 months?

tatamata

  • Hero Member
  • *****
  • Posts: 804
    • ZMSQL - SQL enhanced in-memory database
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #69 on: February 18, 2013, 09:54:10 pm »
Well, it is quite usable already, I use it for a material requirements planning oftware.
But there are many things that could be improved.
For example, SQL language does not support outer joins, which can be overcomed though by splitting a query in two separated queries and using resultset of the first one as input set for the next one.
Visual query builder might be integrated, parmeters implementation is basically a string replacing in SQL text prior execution, etc, etc.
So, I would conservatively say that the project is still in alpha.

However, I think this project is unique in it's goal and combining flat files storage, in-memory database and SQL.

zbyna

  • Jr. Member
  • **
  • Posts: 63
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #70 on: September 23, 2013, 09:12:21 pm »
Hello everybody,
ZMSQL version 0.1.13 is uploaded to Lazarus CCR and available for download. This version works with FPC 2.7.1 in CodeTyphon v 3.10 Lazarus distribution. I didn't test it with older FPC versions, for which it might not work correctly...

Changelog:
ZMSQL version 0.1.13, 13.01.2013: by Zlatko Matić
* Few bug fixes.
* TZMbufdataset replaced with current TBufDataset as ancestor. Tested with current FPC 2.7.1 in CodeTyphon v. 3.10.
If this does not work for your FPC version, then replace TBufDataSet with TZMBufDataset as ancestor.
The TZMBufDataset will stay present latent inside the package, just in case that further development od TBufDataSet goes in direction incompatible with zmsql.
*Added procedure InternalRefresh;override;
TBufDataSet's InternalRefresh does troubles, so it is overriden to do nothing.
It seems that all functionalities which ZMQueryDataset needs are implemented inside TDataSet's Refresh method and that InternalRefresh is not needed at all.                                             

Hi all,

ZMSQL version 0.1.13 in CodeTyphon v 4.5  - for me works only after replacing TBufDataSet with TZMBufDataset as ancestor.
Wise decesion to keep  TZMBufDataset :)



tatamata

  • Hero Member
  • *****
  • Posts: 804
    • ZMSQL - SQL enhanced in-memory database
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #71 on: December 25, 2013, 02:24:52 pm »
I wanted to implement aautincrementing for ftAutoinc fields in overriden DoNewRecord method;
Code: [Select]
procedure TZMQueryDataSet.DoOnNewRecord;
var i:integer;
begin
  inherited DoOnNewRecord;
 {New behavior}
  //Increase value of autoincrement fields
  for i:=0 to self.FieldCount-1 do begin
   if self.Fields[i].DataType=ftAutoInc then begin
     if self.Fields[i].IsNull then self.Fields[i].AsInteger:=0 else
       self.Fields[i].AsInteger:=self.Fields[i].AsInteger+1;
   end;
 end;
end;       

But this doesn't work, field of type ftAutoInc can' accept integer values this way.
What woulkd be the way to implement ftAutoInc?

Leledumbo

  • Hero Member
  • *****
  • Posts: 8835
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #72 on: December 25, 2013, 04:36:29 pm »
Quote
What woulkd be the way to implement ftAutoInc?
SQLite3 way is to maintain a per autoincrement field counter, which will be incremented upon each insert. Just take a look at their source code.

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #73 on: December 25, 2013, 05:04:29 pm »
well based on the code in the TAutoIncField.SetAsLong you can set the value if and I quote
Code: [Select]
  if not(FDataSet.State in [dsFilter,dsSetKey,dsInsert]) then
    DataBaseError(SCantSetAutoIncFields);

In simple terms that means that onNewRecord might not be the correct event to do what you want to, is the dataset in dsInsert state when the event is fired? Try the same code in OnBeforePost event and see if that is going to help.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

tatamata

  • Hero Member
  • *****
  • Posts: 804
    • ZMSQL - SQL enhanced in-memory database
Re: ZMSQL - TBufDataset SQL enhanced in-memory database
« Reply #74 on: January 01, 2014, 10:18:49 pm »
Hello, everybody.
Please, find attached prerelease of zmsql 0.1.14.
Please, test it and report if you find bugs, before uploaded to ccr.
Main changes:
 * Internal optimizations.
 * Based on TZMBufDataset instead TBufDataset, because problems with current TBufDataset version.
 * ZMQueryDataset now loads from and saves to files with .csv extension, instead of .txt extension.
 * Added ManageFields method, which decides what to do with Fields and FieldDefs.
 * Reworked InspectFields procedure, better deduction of FieldDefs and Fields status.
 * Enabled automatic creation of persistent fields objects from predefined fielddefs.
 * New published properties: DynamicFieldsCreated, PeristentFieldsCreated, MemoryDataSetOpened,
 * New public methods: CreateDynamicFieldsFromFieldDefs, CreatePersistentFieldsFromFieldDefs;

Here are declarations of public methods and published properties (besides those inherited from TBufDataset), along with explainations:

Code: [Select]
  public
    { Public declarations }
    procedure QueryExecute; //Executes SQL query defined in SQL property, on .csv files that are placed in folder defined in ZMConnection property. Resultset of select query is loaded into the the zmquerydataset (self).
    procedure PrepareQuery; //Prepares parameterized queries for execution: replaces parameters with parameter values for parameterized queries.
    procedure EmptyDataSet; //Deletes all records from dataset.
    procedure ClearDataSet; //Deletes records, fields and fielddefs.
    procedure CopyFromDataset (pDataset:TDataSet); //Copies schema and data from any TDataset.
    function SortDataset (const pFieldName:String):Boolean; //Ascending/Descending sorting of memory dataset.
    procedure LoadFromTable; //Loads data (or data and schema) from a .csv file (TableName.csv), set in property TableName, from path specified in ZMConnection property.
    procedure SaveToTable;overload; //Saves data and schema to a .csv file (TableName.csv), set in Tablename property, in path specified in ZMConnection property.
    procedure SaveToTable(pDecimaSeparator:Char);overload; //Saves data and schema to a .csv file (TableName.csv), set in Tablename property, in path specified in ZMConnection property.
    procedure CreateDynamicFieldsFromFieldDefs; // Creates fields from predefined fielddefs. To be used in design-time or run-time for memory dataset creation according to predefined fielddefs.
    procedure CreatePersistentFieldsFromFieldDefs; // Creates PERSISTENT fields from predefined fielddefs. To be used in design-time only.
    procedure MemoryDataSetOpen; //Executes CreateDynamicFieldsFromFieldDefs and set dataset to Active.
    constructor Create(AOwner: TComponent); override;
    destructor  Destroy; override;
    //Properties needed for master/detail and referential integrity
    property MasterDataSetTo:TList read FMasterDataSetTo write SetMasterDataSetTo; // Defines master/detail relationship for purpose of master/detail filtration
    property MasterReferentialKeys:TList read FMasterReferentialKeys write SetMasterReferentialKeys;//Defines master/detail relationship for purpose of referential integrity
    property SlaveReferentialKeys:TList read FSlaveReferentialKeys write SetSlaveReferentialKeys; //Defines master/detail relationship for purpose of referential integrity
    property DisableMasterDetailFiltration:Boolean read FDisableMasterDetailFiltration write SetDisableMasterDetailFiltration; //Master/detail filrtation should be temporarily desabled during bulk inserts or updates...
    property OldRecord:TZMBufDataSet {TBufDataSet} read FOldRecord; //Last delete/insert/edit is preserved in this property.
  published
    { Published declarations }
    property ZMConnection:TZMConnection read FZMConnection write SetConnection; //Defines "database" folder path where .csv tables are placed. Instantiates JanSQL database engine.
    property SQL:TStrings read FSQL write SetSQL; //Unprepared SQL query text.
    property QueryExecuted:Boolean read FQueryExecuted write SetQueryExecuted;  //"True" executes QueryExecute and loads resultset into dataset.
    property TableName:String read FTableName write SetTableName; //Name of .csv file (without extension) from which is data loaded by LoadFromTable and to which is data and schema saved by SaveToTable.
    property TableLoaded:Boolean read FTableLoaded write SetTableLoaded; //"True" executes LoadFromTable and loads resultset into dataset.
    property TableSaved:Boolean read FTableSaved write SetTableSaved; //"True" executes SaveToTable and saves dataset to .csv file defined in TableName property, placed in folder specified by ZMConnection property.
    property DynamicFieldsCreated:Boolean read FDynamicFieldsCreated write SetDynamicFieldsCreated; //"True" executes CreateDynamicFieldsFromFieldDefs, which creates fields from predefined fielddefs.
    property PeristentFieldsCreated:Boolean read FPersistentFieldsCreated write SetPersistentFieldsCreated; //"True" executes CreatePersistentFieldsFromFieldDefs, which creates PERSISTENT fields from predefined fielddefs.
    property MemoryDataSetOpened:Boolean read FMemoryDataSetOpened write SetMemoryDataSetOpened; //"True" executes CreateDynamicFieldsFromFieldDefs and activates dataset for editing.
    property PersistentSave:Boolean read FPersistentSave write SetPersistentSave; //If "True", insert/delete/edit will immediately be written to underlying .csv file. If "False", then dataset is only in-memory.
    property Parameters: TParams read FParameters write SetParameters; //Parameters for parameterized SQL text.
    //Master/detail filtration
    property MasterFields: TStrings read FMasterFields write SetMasterFields; //Fields in masterdatasource, (separated by ";") to be used for master/detail filtration.
    property MasterSource: TDataSource read FMasterSource  write SetMasterSource;//Master datasource for master/detail filtration.                                   
                                                                                                                           

 

TinyPortal © 2005-2018