Lazarus

Announcements => Third party => Topic started by: tatamata on June 06, 2011, 09:49:35 pm

Title: MightyQuery SQL Database Management Software
Post by: tatamata on June 06, 2011, 09:49:35 pm
Hello, I have written a small database management system called "MightyQuery" and I'm looking for interested contributors for further development.

You can download source code and binaries from:

http://www.4shared.com/folder/XL3EZwUR/MightyQuery.html (http://www.4shared.com/folder/XL3EZwUR/MightyQuery.html)

Edit 15.02.2023
Source code (CodeTyphon v. 8.00):
https://matasoft.hr/MightyQuery.7z
Compiled binaries for Linux and Windows:
https://matasoft.hr/MightyQuery_Binary.7z

MightyQuery is a single user SQL database management software working with data stored as comma separated values (.csv, .txt) in plain, flat, textual files. It is serverless, self-contained, simple  SQL DBMS for querying CSV files. In contrast to most database systems, MightyQuery does not require neither server installation, neither external libraries. And is multiplatform (due to Lazarus).

It relies on JanSQL database engine , which original code was written by Jan Verhoeven (http://jansfreeware.com) in Delphi and released under MPL 1.1. license.
As far as I am aware, this might be the only existing free/open-source SQL database engine written in Pascal, thus worthy of preservation and improving.

To enable processing of external data, Mighty Query contains importing utility (due to Zeosdbo), supporting wide range of  SQL database servers, such as: PostgreSQL, Interbase, Firebird, MSSQL, Sybase, SQLite, Oracle, ado, ASA.

MightyQuery uses TDbf for storing metadata about projects (databases), tables, queries, external sources of data. Retrieved data from JanSQL is transfered to TBufDataset for viewing, filtering and searching.

I'm looking forward for your ideas, support and critics.

If someone is interested, we could port this project to SoureForge...

Title: Re: MightyQuery 1.0.4 SQL Database Management Software
Post by: joseme on June 06, 2011, 10:41:25 pm
Interesting. Do you know what are the practical limits of MightyQuery?
Title: Re: MightyQuery 1.0.4 SQL Database Management Software
Post by: tatamata on June 06, 2011, 11:04:25 pm
Well, there are many limits at this moment.
- Some queries with joined tables and few conditions in WHERE clause are terrible slow or even freezes the program. I have found that I can overcome that issue by using ASSIGN TO statement. With ASSIGN TO you can aasign a query dataset to a variable that you can use in next SQL statement. So, I split a query in two queries and then it works.
Obviosly, one of the priorities of the MightyQuery project should be to improve JanSQL database engine.
- JanSQL uses only dot for decimal separator. There should be possibility to take over local system settings.
- JanSQL deals only with textual files, thus no binary fields.
- There are no indexes in janSQL
- MightyQuery currently does not have visual query designer.
- Currently MightyQuery process only one query at time, only one window is open
etc, etc.
There is lot of job to do.
Title: Re: MightyQuery 1.0.4 SQL Database Management Software
Post by: tatamata on June 07, 2011, 09:17:31 am
TODO:
- improve JanSQL to be able to deal with complex joining (WHERE clause), maybe to add JOIN expression.
- Design visual query builder
- make wrapper components for JanSQL, for example: TJanDatabase, TjanTable, TJanActionQuery, TJanSelectQuery...Datasets to be TBufdataset descendants?
- SQL Query Editor: add Code Highlighting and IntelliSense
- separate procedures from the main form unit code into separate unit...
Title: Re: MightyQuery 1.0.4 SQL Database Management Software
Post by: tatamata on July 24, 2011, 10:59:35 pm
MightyQuery version 1.0.5 is out.
You can download both source and binaries from:
http://www.4shared.com/folder/XL3EZwUR/MightyQuery.html (http://www.4shared.com/folder/XL3EZwUR/MightyQuery.html)
Changes from previous version:
Next few days I am going to clean-up whole code, so the next version will be rewritten and based on zmsql package (TZMonnection and TZMQueryDataset components). In future, these two projects (MightyQuery and zmsql) will be synchronized.
Title: Re: MightyQuery 1.0.4 SQL Database Management Software
Post by: fredycc on July 24, 2011, 11:41:35 pm
Cool, very interesting project  ;D

Thanks

Regards.
Title: Re: MightyQuery 1.0.4 SQL Database Management Software
Post by: tatamata on July 25, 2011, 09:08:23 am
In following weeks I am going to focus on zmsql package (http://www.lazarus.freepascal.org/index.php/topic,13821.msg72958.html#msg72958 (http://www.lazarus.freepascal.org/index.php/topic,13821.msg72958.html#msg72958)) that will be basis for future MightyQuery versions.
Namely,
- adding master/detail filtering
- enforcing refferential integrity
- improving JanSQL database engine
- enabling parameterized queries
- to enable autoincrement field type

After that I will focus on the MightyQuery:
-visual query builder
- form generation, reporting
- scripting

Yeah, a long way to go...  8)
Title: Re: MightyQuery 1.0.5 SQL Database Management Software
Post by: BigChimp on July 25, 2011, 09:59:05 am
Tatamata,

Sounds good (especially a visual query designer is something I've wanted for a long time), but yes, a long way to go.

Might be a good idea if you got this and zmsql into Lazarus CCR if possible, or on Sourceforge/Google code/Github/Bitbucket...
In this way, your commits will be visible immediately using the version control system, and many of these services also provide bug trackers etc...
Title: Re: MightyQuery SQL Database Management Software
Post by: tatamata on August 04, 2011, 04:37:19 pm
MightyQuery 1.0.7 is released. This is bug fixes release.
In previous version there was a stupid bug that prevented proper connecting to a "database" when scrolling through "Projects".

Binaries and source code can be downloaded from:
http://www.4shared.com/folder/XL3EZwUR/MightyQuery.html (http://www.4shared.com/folder/XL3EZwUR/MightyQuery.html)
Title: Re: MightyQuery SQL Database Management Software
Post by: bee on March 20, 2018, 09:42:02 am
Hello, @tatamata …

Is this Mighty Query project still alive? I wonder what the latest features you have added into TjanSQL component. I might need it for a little project of mine.

Thank you.
Title: Re: MightyQuery SQL Database Management Software
Post by: tatamata on March 20, 2018, 10:23:01 am
Hi, no it is dead.
You might download source code from 2011 https://www.4shared.com/zip/vTA2CNBS/MightyQuery107.html
or here
https://www.4shared.com/zip/H5pA5CKd/Source.html
Title: Re: MightyQuery SQL Database Management Software
Post by: tatamata on March 20, 2018, 10:35:58 am
I have another, more recen tversion, you can download here:
https://drive.google.com/file/d/1CrSqzxD4WIc-uPmx0kSo7OAEW9b6ZIOF/view?usp=sharing
Title: Re: MightyQuery SQL Database Management Software
Post by: bee on April 16, 2018, 06:07:37 pm
Just got the time to reopen this old library. I barely remember anything about this library. I recalled janSQL library because I want to use it as an engine to query csv files. I modified them a bit just so it can be compiled using pure FPC (v.3.0.2), removing its dependency to Lazarus units.

Here's what I did:
1. Removed all LCL units from the uses clauses. It went without any problems because they are not actually being used by janSQL library.
2. Renamed toString of TTokenOperators (in janSQLTokenizer.pas) into toText because it conflicts with new FPC's string function.

After those little modifications, the library and all of its units can be compiled just fine. But my test code failed on a simple select command. Here's my test code:

Code: Pascal  [Select][+][-]
  1. program jSQLcli;
  2.  
  3. {$MODE OBJFPC}{$H+}{$J-}
  4.  
  5. uses
  6.   janSQL;
  7.  
  8. var
  9.   sampleDB: TjanSQL;
  10.   sqlResult: integer;
  11.  
  12. procedure printResult(resultIndex: integer);
  13. var
  14.   r,c,i,j: integer;
  15. begin
  16.   c := sampleDB.RecordSets[resultIndex].FieldCount;
  17.   r := sampleDB.RecordSets[resultIndex].RecordCount;
  18.   // print column's name
  19.   for i := 0 to c-1 do
  20.     if i < c-1 then
  21.       write(sampleDB.RecordSets[resultIndex].FieldNames[i],' | ')
  22.     else
  23.       writeln(sampleDB.RecordSets[resultIndex].FieldNames[i]);
  24.   // print cell's value
  25.   for i := 0 to r-1 do
  26.     for j := 0 to c-1 do
  27.       if j < c-1 then
  28.         write(sampleDB.RecordSets[resultIndex].Records[i].Fields[j].Value,' | ')
  29.       else
  30.         writeln(sampleDB.RecordSets[resultIndex].Records[i].Fields[j].Value);
  31. end;
  32.  
  33. begin
  34.   sampleDB := TjanSQL.Create;
  35.  
  36.   sqlResult := sampleDB.SQLDirect('select * from authors');
  37.   if sqlResult <> 0 then
  38.   begin
  39.     if sqlResult > 0 then
  40.     begin
  41.       printResult(sqlResult);
  42.       // release recordset
  43.       if not sampleDB.RecordSets[sqlResult].Intermediate then
  44.         sampleDB.ReleaseRecordset(sqlResult);
  45.     end;
  46.   end
  47.   else
  48.     writeln(sampleDB.Error);
  49.  
  50.   sampleDB.Free;
  51. end.
  52.  

The error message is SELECT: can not find table authors. The authors table is from the sample database (authors.txt). I tried both Tatamata's and Theo's version, same error.

Any help? Thank you.
Title: Re: MightyQuery SQL Database Management Software
Post by: Soner on April 16, 2018, 08:25:43 pm
You must call "connect" command before:
Code: Pascal  [Select][+][-]
  1. begin
  2.   sampleDB := TjanSQL.Create;
  3.  
  4.   sampleDB.SQLDirect('CONNECT TO ''C:\MyApp\db'''); //<<-- Folder where the authors.txt exist
  5.   sqlResult := sampleDB.SQLDirect('select * from authors');
  6.   //...
  7. end.
  8.  
  9.  
Title: Re: MightyQuery SQL Database Management Software
Post by: bee on April 17, 2018, 03:44:47 am
Thank you. Sorry for asking a silly question like that. As I said, I barely remember anything about this library. I think I had to dig out the sample program again to know how to use it.

I'm going to experiment with it in the next few days. I assume all the queries in the samples.txt would work. I'll use it to test the library. So, expect some other questions from me again later. Thank you.

Btw… I'm on Mac. I was looking for an app to read .hlp files, to read TjanSQL documentation. Does anyone here know how to read .hlp file on Mac? Or is there an online service to convert .hlp files into .html files? Thank you.
Title: Re: MightyQuery SQL Database Management Software
Post by: bee on April 17, 2018, 05:48:41 pm
I've made some small modifications to janSQL library based on tatamata's version. I removed all unused LCL/VCL units so it can be compiled using only standard pascal units. I put them on my github repo here: https://github.com/git-bee/janSQL

I also have added a simple client program, mainly to test the SQL engine. I've found some problems. Feel free to contribute. Thank you.
Title: Re: MightyQuery SQL Database Management Software
Post by: Soner on April 17, 2018, 07:32:09 pm
...
Btw… I'm on Mac. I was looking for an app to read .hlp files, to read TjanSQL documentation. Does anyone here know how to read .hlp file on Mac? Or is there an online service to convert .hlp files into .html files? Thank you.
I would install Windows Xp in Virtualbox and use it as Hlp-Viewer.
This here is also good help converter http://www.helpscribble.com/ (http://www.helpscribble.com/).

Also I played around with it last autumn and created a gui example.
It is Lazarus port of Jans example. I
I put it as attachment.

Title: Re: MightyQuery SQL Database Management Software
Post by: bee on April 18, 2018, 02:45:23 am
I would install Windows Xp in Virtualbox and use it as Hlp-Viewer.
This here is also good help converter http://www.helpscribble.com/ (http://www.helpscribble.com/).
I don't even have a Windows XP installer to begin with. Would any Windows users here help me convert janSQL's .hlp files into common text format such as .pdf, .hmtl, or .txt? It should be an easy job if you're a Windows user. I'd like to put them in the public repo so everyone could read it. Thank you.

Also I played around with it last autumn and created a gui example.
It is Lazarus port of Jans example. I put it as attachment.
Thank you. Is your version newer than tatamata's? Do all the query examples in the sample.txt file work correctly in your GUI example? I found some of them don't work correctly with tatamata's version. I'm gonna try them with your version. Can you provide some more complex query examples that work with your janSQL version? It'd be helpful to janSQL users, also to show janSQL features.

The original janSQL units put some VCL or LCL units that are not actually being used. They're still there in tatamata's version, I still saw them in yours as well. I would remove them in order to make janSQL doesn't require Lazarus. It's not a GUI library anyway.
Title: Re: MightyQuery SQL Database Management Software
Post by: bee on April 18, 2018, 06:27:25 am
Is your version newer than tatamata's? Do all the query examples in the sample.txt file work correctly in your GUI example?
After comparing the files of tatamata's version and soner's version, I'm pretty sure that soner's is based on tatamata's version, plus some modifications and new features added.

The original janSQL units put some VCL or LCL units that are not actually being used. They're still there in tatamata's version, I still saw them in yours as well.
I was wrong, I might have opened a wrong file. Soner had done it already. So, I think it's safe to use soner's version to start enhancing this janSQL library. I'll update my github repo to use soner's version, after I clean up my mess with these files. [Update: done!]

But, documentation is still a problem as long as I have no access to a Windows machine, to convert the .hlp file.
Title: Re: MightyQuery SQL Database Management Software
Post by: Soner on April 18, 2018, 09:01:43 pm
I took janSQL from tatamatas TZMSQL and I made some changes then I gave my changes to tatamata. I think he integrated my changes.
I comment my modifications every time with my name to follow it.
You can take tatamatas version and when you improve this library (jansql) then you can give your changes to tatamata.

I converted the help-file to doc-file and html-file (copy/paste from original jansql.hlp).
I thought it was small file but it took over an hour to copy it.
Here is it as attachment, have fun:


@tatamata: You can use this doc in your TMZSQL when you want.

Title: Re: MightyQuery SQL Database Management Software
Post by: Soner on April 18, 2018, 09:45:47 pm
..
Do all the query examples in the sample.txt file work correctly in your GUI example? I found some of them don't work correctly with tatamata's version. I'm gonna try them with your version. Can you provide some more complex query examples that work with your janSQL version? It'd be helpful to janSQL users, also to show janSQL features.
..
Yes all examples are working also with tatamatas version. My Version is "same" as tatamatas i copied them only in the same directory to remove tmzsql-dependency.
Some examples like update, insert, alter has no output. You can look at statusbar at the gui example:
  Left: Count of result records.
  Middle: SQL execution time in tick difference.
  Right: Status text. 'OK' when no errors occured else errors as text.

JanSQL is very fast, i think it is good enough for small applications.
Start my example and select from [Menu > Samples > Insert in to select].
Then execute it 4 or 5 time after you get enough data rows. (Attention it can take long time to execute it)
You can count the data rows with this sql:
Code: SQL  [Select][+][-]
  1. SELECT COUNT(*) FROM users
  2.  
Now you can test the engine.
I had 65856 rows in users-Table and the query:
Code: SQL  [Select][+][-]
  1. SELECT * FROM users WHERE productid='6'
  2.  
took only 452 ticks.

When it does not work on your computer then maybe it is mac problem(especially date and time fields). Freepascal on Mac ignores your country settings. I put in my other program this at start:
Code: Pascal  [Select][+][-]
  1.   {$IFDEF DARWIN} //German Dateformat
  2.   FormatSettings.DateSeparator:='.';
  3.   DateSeparator:='.';
  4.   FormatSettings.LongDateFormat:='dddd, d. MMMM yyyy';
  5.   LongDateFormat:='dddd, d. MMMM yyyy';
  6.   FormatSettings.ShortDateFormat:='dd.MM.yyyy';
  7.   ShortDateFormat:='dd.MM.yyyy';
  8.   {$ENDIF}
  9.  
Title: Re: MightyQuery SQL Database Management Software
Post by: bee on April 19, 2018, 04:19:27 am
You can take tatamatas version and when you improve this library (jansql) then you can give your changes to tatamata.
I take your version and use it as the base for my next modification. All of the modification is open to public through my GitHub repo (https://github.com/git-bee/janSQL). Anybody who wants to contribute to the project could simply fork it and make a pull request. This is the new way of how open source projects work.

I converted the help-file to doc-file and html-file (copy/paste from original jansql.hlp). I thought it was small file but it took over an hour to copy it. Here is it as attachment, have fun!
Ha… thank you so much! This is what I need. I'll convert them to GitHub markdown documents and put them in the repo, so everyone could read them. Thank you!

Yes all examples are working also with tatamatas version. My Version is "same" as tatamatas i copied them only in the same directory to remove tmzsql-dependency.
Yes, I've tried your app on Linux and all queries work fine. So, I simply remove tatamata's version and use your version.

JanSQL is very fast, i think it is good enough for small applications.
I know, I've used the original janSQL a very long time ago.

When it does not work on your computer then maybe it is mac problem (especially date and time fields). Freepascal on Mac ignores your country settings. I put in my other program this at start:
Code: Pascal  [Select][+][-]
  1.   {$IFDEF DARWIN} //German Dateformat
  2.   FormatSettings.DateSeparator:='.';
  3.   DateSeparator:='.';
  4.   FormatSettings.LongDateFormat:='dddd, d. MMMM yyyy';
  5.   LongDateFormat:='dddd, d. MMMM yyyy';
  6.   FormatSettings.ShortDateFormat:='dd.MM.yyyy';
  7.   ShortDateFormat:='dd.MM.yyyy';
  8.   {$ENDIF}
Yes, I think it's not right to use local data format settings in the database file because it will make the database not portable. If you bring a database that use '.' as decimal separator –for example– then when you open it on a system that use ',' as decimal separator, janSQL wouldn't able to read the data and arise an error. So, I think the correct approach is to use a consistent data format for the database, then use local data format to display the data. CMIIW.
Title: Re: MightyQuery SQL Database Management Software
Post by: tatamata on February 14, 2023, 08:03:24 am
Hello everybody.
I have resurected MightyQuery program and refactored it a bit with CodeTyphon. You can download source code and compiled binaries here:
Source code (CodeTyphon v. 8.00):
https://matasoft.hr/MightyQuery.7z
Compiled binaries for Linux and Windows:
https://matasoft.hr/MightyQuery_Binary.7z
I intend to work on it in following weeks, I want to add merge/append feature, to be able to combine multiple csv files similar to what can be done in Microsft Power Query...
Title: Re: MightyQuery SQL Database Management Software
Post by: tatamata on February 21, 2023, 08:45:41 pm
Hello everybody.
I have resurected MightyQuery program and refactored it a bit with CodeTyphon. You can download source code and compiled binaries here:
Source code (CodeTyphon v. 8.00):
https://matasoft.hr/MightyQuery.7z
Compiled binaries for Linux and Windows:
https://matasoft.hr/MightyQuery_Binary.7z
I intend to work on it in following weeks, I want to add merge/append feature, to be able to combine multiple csv files similar to what can be done in Microsft Power Query...
Re-uploaded compiled binaries and source code.
In source I have also zipped my pretty substantial modifications of ZMSQL package, but can also be downloaded here: https://matasoft.hr/pl_ZMSQL_v7.6.2.7z

I have done many changes in both MightyQuery and ZMSQL, of which most important changes in ZMSQL are:
      *ZMQueryDataset is now descendant of TRxMemoryData by defualt, but you can always switch back to TBufDataset, by compiler directive.
       Reason for this decision is because TBufDataset cannot load data from large csv files.
      *ZMBufDataset removed from package.
      *Removed dependency on TSdfDataset. TCSVdataset is now used for importing data from CSV files.
      *When importing data from CSV files, via TCSVDataset, FieldDef.Size and Field.Size properties are now determined by inspecting actual values through all rows.
       Rason for this is to decrease memory consumption for loading large csv files.
       Note that TCSVDataset does setup Size property always to Size=255, which is not optimal...
      *CopyFromDataset is renamed to CopyFromADataset, in order not to hide CopyFromDataset coming from TBuFDataset
      *Added function DetermineSeparator(AFileName: String; var HasFieldNames: Boolean): Char; to be used for automatic determination of field delimiter in a csv file.
      *Resolved bug that was preventing multiple consecutive loading of different csv files.
      *Addded procedure GetFieldsListFromCSVFile(List: TStrings);  This is fast method to determine field names in the csv file registered as TableName.
       Corresponding change made in unit vsb_Zmsql, in order for TZMQueryBuilder can load field information fast.
      *Added procedure CopyDataFromDataset(DataSet: TDataSet);
      *Various other minor changes and bugfixes     
Title: Re: MightyQuery SQL Database Management Software
Post by: tatamata on February 24, 2023, 07:37:57 pm
Reuploaded. Implemented changing order of columns in schema definition and merge/append combining of multiple tables.
Next thing to do: columns renaming. This is important for merge/appending, because columns are merged only if have the same name.
I am also thinking on adding row index indicator into RxDBGrids for csv file loading and sql query resultset loading.
I also intend to work on sorting datasets in dbgrids...
Title: Re: MightyQuery SQL Database Management Software
Post by: tatamata on February 26, 2023, 08:45:51 pm
Reuploaded, MightyQuery version 1.2.4
Solved columns renaming, implemented sorting in RxDBGrid for csv dataset and for ZMQueryDataset, I din't have time to do the same for TDBf dataset though.
I couldn't figure out how to add row indicator in the RxDBGrid, problem is that automatic columns are removed if another column is added during runtime, so I quit for now. The cause of trouble is here:
Code: Pascal  [Select][+][-]
  1. function TDBGridColumns.Add: TColumn;
  2. var
  3.   G: TCustomDBGrid;
  4. begin
  5.   {$ifdef dbgDBGrid}
  6.   DebugLn('%s.Add', [ClassName]);
  7.   {$endif}
  8.   G := TCustomDBGrid(Grid);
  9.   if G<>nil then begin
  10.     // remove automatic columns before adding user columns
  11.     if not (gsAddingAutoColumns in G.GridStatus) then
  12.       RemoveAutoColumns;
  13.   end;
  14.   result := TColumn( inherited add );
  15. end;
  16.  
TinyPortal © 2005-2018