Lazarus

Programming => Databases => Topic started by: Чебурашка on November 14, 2022, 02:14:58 pm

Title: Again on SQLite and date times.
Post by: Чебурашка on November 14, 2022, 02:14:58 pm
Hello,

I have a table like this in SQlite

Code: Pascal  [Select][+][-]
  1. CREATE TABLE ExampleT
  2. (
  3.   id integer not null,
  4.   fieldDate datetime
  5. );
  6.  

If I do a TSQLQuery and I try to read the field fieldDate .AsDateTime it succeds.

Instead if I define the table like this:
Code: Pascal  [Select][+][-]
  1. CREATE TABLE ExampleT
  2. (
  3.   id integer not null,
  4.   fieldDate NUMERIC
  5. );
  6.  

(should be equivalent to the previous as DATETIME is treated as NUMERIC in SQlite - https://www.sqlite.org/datatype3.html, also I can see that the data stored in doth cases is the same, using the sqlite3 cli program)

If I do a TSQLQuery and I try to read the field fieldDate .AsDateTime it fails telling: Invalid type conversion to TDateTime in field fieldDate.

I guess that the free pascal code determines the field type using the table field definition.

Is it possible to force a TField to be what one wants without relying only on the table defintion?

Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 14, 2022, 02:56:13 pm
And as i told you in your other thread(s): Don't use those DataTypes!
There is no DateTime DataType, and NUMERIC is not a DataType but an Affinity.
Use the proper Storage-Classes.

And TDateTime wants a double.
"Spot the mistake"
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 14, 2022, 03:25:49 pm
And as i told you in your other thread(s): Don't use those DataTypes!
There is no DateTime DataType, and NUMERIC is not a DataType but an Affinity.
Use the proper Storage-Classes.

And TDateTime wants a double.
"Spot the mistake"

After your reply, I recreated the DB with DATETIME mapped to DOUBLE, in order to follow strictly your indications (unless I did not undestrand them fully).

Code: Bash  [Select][+][-]
  1. SQLite version 3.34.1 2021-01-20 14:10:07
  2. Enter ".help" for usage hints.
  3. sqlite> .schema 'T0041%'
  4. CREATE TABLE T0041_MachineLog (refMachine INTEGER NOT NULL, id UNSIGNED BIG INT NOT NULL, deleted INTEGER NOT NULL, refImport INTEGER NOT NULL, eventDate DOUBLE NOT NULL, eventType INTEGER NOT NULL, eventVersion INTEGER NOT NULL, eventDataType INTEGER, eventData BLOB, CONSTRAINT T0041_MachineLog_PK PRIMARY KEY (refMachine, id), CONSTRAINT T0041_FK_T0003 FOREIGN KEY (refMachine) REFERENCES T0003_ConfMachines(id));
  5. CREATE INDEX T0041_MachineLog_FKI ON T0041_MachineLog(refMachine);
  6.  

The field in question now is:

Code: SQL  [Select][+][-]
  1. eventDate DOUBLE NOT NULL,
  2.  

Hereafter it is a query made on that table, where you can see that it does contain data (=writing seems to work, probably writing doe not guess datatype from db structure)

Code: SQL  [Select][+][-]
  1. sqlite> SELECT * FROM T0041_MachineLog LIMIT 5;
  2. refMachine|id|deleted|refImport|eventDate|eventType|eventVersion|eventDataType|eventData
  3. 1|1|0|1|2455615.89571759|1|1|0|
  4. 1|2|0|1|2455615.89572917|11|1|0|
  5. 1|3|0|1|2455615.89574074|11|1|0|
  6. 1|4|0|1|2455615.89574074|11|1|0|
  7. 1|5|0|1|2455615.89586806|2|1|0|
  8.  

And here are the errors related to the reading of this field (and all others that should be read AsDateTime):

Code: Text  [Select][+][-]
  1. Localization system auto-analysis disabled. To set the compiler directive DEBUG_LOCALIZATION and execute the program
  2. Processing machine "New machine1" with code "" ...
  3. Found 37 source prodlog files
  4. Processed 37 new files and 0 changed filed
  5. Unexpected error while LoadEvents on eventId 1, eventType 1/version 1 form machine 1 (exception: Invalid type conversion to TDateTime in field eventDate)
  6. Unexpected error while ProcessEvents (exception: Invalid type conversion to TDateTime in field eventDate)
  7. Unexpected error while recaching data of machine 1 (exception: Invalid type conversion to TDateTime in field eventDate)
  8. Unexpected error while compiling daily summaries of machine 1 (exception: Invalid type conversion to TDateTime in field eventDate)
  9. Processing machines completed
  10. Unexpected error while loading products in interval (exception: Invalid type conversion to TDateTime in field eventDate)
  11. Unexpected error while loading products in interval (exception: Invalid type conversion to TDateTime in field eventDate)
  12.  


Any ideas at this point?


Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 14, 2022, 03:32:21 pm

After your reply, I recreated the DB with DATETIME mapped to DOUBLE, in order to follow strictly your indications (unless I did not undestrand them fully).

Code: Bash  [Select][+][-]
  1. SQLite version 3.34.1 2021-01-20 14:10:07
  2. Enter ".help" for usage hints.
  3. sqlite> .schema 'T0041%'
  4. CREATE TABLE T0041_MachineLog (refMachine INTEGER NOT NULL, id UNSIGNED BIG INT NOT NULL, deleted INTEGER NOT NULL, refImport INTEGER NOT NULL, eventDate DOUBLE NOT NULL, eventType INTEGER NOT NULL, eventVersion INTEGER NOT NULL, eventDataType INTEGER, eventData BLOB, CONSTRAINT T0041_MachineLog_PK PRIMARY KEY (refMachine, id), CONSTRAINT T0041_FK_T0003 FOREIGN KEY (refMachine) REFERENCES T0003_ConfMachines(id));
  5. CREATE INDEX T0041_MachineLog_FKI ON T0041_MachineLog(refMachine);
  6.  

The field in question now is:

Code: SQL  [Select][+][-]
  1. eventDate DOUBLE NOT NULL,
  2.  
Close. Use REAL instead of Double (but SQLite will map Double to Real in 99.9999% of cases)
Quote
Hereafter it is a query made on that table, where you can see that it does contain data (=writing seems to work, probably writing doe not guess datatype from db structure)

Code: SQL  [Select][+][-]
  1. sqlite> SELECT * FROM T0041_MachineLog LIMIT 5;
  2. refMachine|id|deleted|refImport|eventDate|eventType|eventVersion|eventDataType|eventData
  3. 1|1|0|1|2455615.89571759|1|1|0|
  4. 1|2|0|1|2455615.89572917|11|1|0|
  5. 1|3|0|1|2455615.89574074|11|1|0|
  6. 1|4|0|1|2455615.89574074|11|1|0|
  7. 1|5|0|1|2455615.89586806|2|1|0|
  8.  
Remove the LIMIT 5
I want to see ALL those Dates
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 14, 2022, 03:41:02 pm
Remove the LIMIT 5
I want to see ALL those Dates

Attached
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 14, 2022, 03:49:36 pm
Close. Use REAL instead of Double (but SQLite will map Double to Real in 99.9999% of cases)

Modified DB to REAL, retried.
Same problem.

Any other idea?
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 14, 2022, 04:05:49 pm
Remove the LIMIT 5
I want to see ALL those Dates

Attached
OK, looks consistent.
Will have to check what got borked.

By any chance: Could you give me the INSERT-Statements for those Values? (dump the table).
Otherwise i would have to write a routine to import it to my local sqlite-database
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 14, 2022, 04:17:27 pm
Remove the LIMIT 5
I want to see ALL those Dates

Attached
OK, looks consistent.
Will have to check what got borked.

By any chance: Could you give me the INSERT-Statements for those Values? (dump the table).
Otherwise i would have to write a routine to import it to my local sqlite-database

Attached!
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 14, 2022, 04:41:32 pm
Remove the LIMIT 5
I want to see ALL those Dates

Attached
OK, looks consistent.
Will have to check what got borked.

By any chance: Could you give me the INSERT-Statements for those Values? (dump the table).
Otherwise i would have to write a routine to import it to my local sqlite-database

Attached!
Thx.
Will take a look at it tomorrow.
Am currently swamped with work
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 15, 2022, 02:29:51 pm
Oh, you gotta be kidding me.
From Fields.inc - Line 407 --> Line 513
Code: Pascal  [Select][+][-]
  1. property AsDateTime: TDateTime read GetAsDateTime write SetAsDateTime;    
  2. .
  3. .
  4. .
  5. .
  6. function TField.GetAsDateTime: TDateTime;
  7.  
  8. begin
  9.   raise AccessError(SdateTime);
  10. end;                            
  11.  
Funny enough: AsFloat throws no exception despite the same code as above being in there
and this also didn't throw an exception
Code: Pascal  [Select][+][-]
  1. Var d:TDateTime;
  2. Writeln(Qry.FieldByName('eventDate').AsDateTime); //Exception!
  3. d:=Qry.FieldByName('eventDate').AsFloat;  //No Exception
  4. Writeln(d);   //No Exception
  5.  
So what in blazes is going on?

That said: I apologize to tt
I did get the same Exception.
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 15, 2022, 02:54:00 pm
Oh, you gotta be kidding me.
From Fields.inc - Line 407 --> Line 513
Code: Pascal  [Select][+][-]
  1. property AsDateTime: TDateTime read GetAsDateTime write SetAsDateTime;    
  2. .
  3. .
  4. .
  5. .
  6. function TField.GetAsDateTime: TDateTime;
  7.  
  8. begin
  9.   raise AccessError(SdateTime);
  10. end;                            
  11.  
Funny enough: AsFloat throws no exception despite the same code as above being in there
and this also didn't throw an exception
Code: Pascal  [Select][+][-]
  1. Var d:TDateTime;
  2. Writeln(Qry.FieldByName('eventDate').AsDateTime); //Exception!
  3. d:=Qry.FieldByName('eventDate').AsFloat;  //No Exception
  4. Writeln(d);   //No Exception
  5.  
So what in blazes is going on?

That said: I apologize to tt
I did get the same Exception.

Did you try also havin the eventDate field defined as follows?

eventDate DATETIME NOT NULL

That will work for some criptic reason (feeling is that TField or derived class guesses file type from ddl).
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 15, 2022, 03:20:56 pm

Did you try also havin the eventDate field defined as follows?

eventDate DATETIME NOT NULL

That will work for some criptic reason (feeling is that TField or derived class guesses file type from ddl).
I just did. And you're right: AsDateTime is not throwing an exception.
So basically, it's hardwired between the DataType sent from the DataBase to the FieldType, (Which actually makes sense, since TSQLQuery is DBMS-Agnostic)

And with SQLite you're stuck with what to use: Do it correctly, and use REAL, INTEGER or TEXT for Date(Time), and remember it in the Frontend-Code,
or do it wrong using DATETIME as DataType for the Column during creation, so it gets written into the DDL-Metadata, and HOPE SQLite never changes this.

What baffles me the most: Why is AsDateTime not recognizing that it's basically still a Double (Float)?
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 15, 2022, 04:46:10 pm
And with SQLite you're stuck with what to use: Do it correctly, and use REAL, INTEGER or TEXT for Date(Time), and remember it in the Frontend-Code,
or do it wrong using DATETIME as DataType for the Column during creation, so it gets written into the DDL-Metadata, and HOPE SQLite never changes this.

Due to the exceptionality of SQLite in terms of dynamic typing (they state that they are the only RDBMS with that feature), my feeling is that we should not enforce field/param data type inside the object itself (field/param in this case).

It should be something like

if I have a value with some storage class, I try to represent to the caller according to how the caller asks for.

In this way if value is REAL and I call AsFloat I will get a float, AsDateTime gives datetime.
Similarly if I have a TEXT, AsDateTime gives datetime tring to interpret the TEXT. Or a string if I ask for AsString on that record.

I don't know if this appears senseful, but I see few possibilities of relying on SQLite DDL information, as sometimes it is also possible to define columns with no type at all (https://www.sqlite.org/syntax/column-def.html - it is possible to bypass the type-name).

Moreover, with this dynamic model, I believe is very possible to have, for the same column, different storage types in different rows, so for sure an approach based on the datatype fixed inside the TField object has few chances to work.

 
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 15, 2022, 05:42:18 pm
Yeah, that about sums it up.
The only thing coming to mind would be subclassing TSQLQuery, and override AsDateTime and the others with your own Getter/Setter‘s

EDIT: or maybe a class-helper.
Would have to chew on it.
Title: Re: Again on SQLite and date times.
Post by: LacaK on November 15, 2022, 07:52:24 pm
Regarding to TField.AsDateTime - TField is generic ancestor for all "specialized" field types. So it is expected that getter for TField.AsDateTime raises exception. Because these methods are implemented at TDateTimeField or TDateField level.

Regarding DateTime data type in SQLite3 look at "2.2. Date and Time Datatype" on https://www.sqlite.org/datatype3.html.
You can use DATETIME as "datatype" in CREATE TABLE statement. SQLite3 will perfectly accept it and Lazarus/FPC will recognize it and will map it to ftDateTime field type.
But you must ensure that data written into this column will satisfy conditions from 2.2:
- so either it must be string in format "YYYY-MM-DD HH:MM:SS.SSS"
- or it must be real number in Julian day numbers
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 16, 2022, 08:33:12 am
Regarding DateTime data type in SQLite3 look at "2.2. Date and Time Datatype" on https://www.sqlite.org/datatype3.html.
You can use DATETIME as "datatype" in CREATE TABLE statement. SQLite3 will perfectly accept it and Lazarus/FPC will recognize it and will map it to ftDateTime field type.

Thanks, I agree and it was the way I was working originally. Later I needed to reimplement my code/db in order to support SQLite+PostgreSQL+SQLServer+Oracle and I decided to start using liquibase to manage the various details of the db creation step. At this point I noticed that liquibase was mapping DATETIME datatype to REAL and all this story started.
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 16, 2022, 08:34:25 am
Yeah, that about sums it up.
The only thing coming to mind would be subclassing TSQLQuery, and override AsDateTime and the others with your own Getter/Setter‘s

EDIT: or maybe a class-helper.
Would have to chew on it.

I would be glad to be helpful in some way.
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 16, 2022, 09:49:45 am
Yeah, that about sums it up.
The only thing coming to mind would be subclassing TSQLQuery, and override AsDateTime and the others with your own Getter/Setter‘s

EDIT: or maybe a class-helper.
Would have to chew on it.

I would be glad to be helpful in some way.
After thinking about it, i don't think there is an "easy" solution (nevermind that subclassing/extending TSQLQuery would be the wrong class, since AsDateTime is a property of TField)

Bottom Line: Since you have to "support" multiple DBMS (are you using TSQLConnector?), you'll probably have to write individual code for each connection-type
Kind of like (Pseudo-Code)
If ConnType='SQLite' And FieldType='Float' Then ReadAsFloat

Down-Side: You'd have to know if the Field you want to read is a "real" Float-Value (Say, Money-Balance) or a DateTime-Value
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 16, 2022, 10:48:19 am
Bottom Line: Since you have to "support" multiple DBMS (are you using TSQLConnector?), you'll probably have to write individual code for each connection-type
Kind of like (Pseudo-Code)
If ConnType='SQLite' And FieldType='Float' Then ReadAsFloat

Down-Side: You'd have to know if the Field you want to read is a "real" Float-Value (Say, Money-Balance) or a DateTime-Value

To support different RDBMS I simply create the right (subclass of) TSQLConnection object according database type (SQLITE -> TSQLite3Connection, MSSQL -> TMSSQLConnection, ...). Once this is done TSQLTransation and TSQLQuery are always the same. And from my persective also TParam and TField are always the same, despite actual ones could be subclasses to adapt to actual connection type.

And for queries: given the SQL query, each parameter/field has a fixed and known datatype. So not need to do "If ConnType='SQLite' And FieldType='Float' Then ReadAsFloat".

For example, with this update query:

Code: Pascal  [Select][+][-]
  1.   query.SQL := 'update T0003_ConfMachines'
  2.               + ' set'
  3.               + '   name = :name'
  4.               + ' , deleted = :deleted'
  5.               + ' , machineType = :machineType'
  6.               + ' , machineCode = :machineCode'
  7.               + ' , machineAcquisitionType = :machineAcquisitionType'
  8.               + ' , machineAcquisitionDirectory = :machineAcquisitionDirectory'
  9.               + ' , machineImportAtStartup = :machineImportAtStartup'
  10.               + ' , machineAcquisitionFileName = :machineAcquisitionFileName'
  11.               + ' , machineAcquisitionSerialIdentifier = :machineAcquisitionSerialIdentifier'
  12.               + ' , machineHasEnergyMeasuringDevice = :machineHasEnergyMeasuringDevice'
  13.               + ' , listeningMode = :listeningMode'
  14.               + ' , listeningPort = :listeningPort'
  15.               + ' where id = :id';
  16.  
  17.  

for each parameter I know what is going to be the datatype (regardless of RDBMS), so I'll write

Code: Pascal  [Select][+][-]
  1.   query.ParamByName('name').AsString := 'value';
  2.   query.ParamByName('listeningMode').AsInteger := 4;
  3.   ...
  4.  

Same applies to select. For each field I know pascal datatype so I will simple ask for it.

Code: Pascal  [Select][+][-]
  1. query.SQL := 'select eventDate from T0041_MachineLog where (refMachine = :refMachine) and (id = :id)';
  2. query.ParamByName('refMachine').AsInteger = 4;
  3. query.ParamByName('id').AsLargeInt= 214;
  4.  
  5. query.Open();
  6.  
  7. eventDate := query.FieldByName('eventDate').AsDateTime;
  8.  

Basically my perspective is: in code each field has its own fixed data type, then in each RDBMS it has to be choosen the "datatype" capable of storing/suppyling the correspondent correct datatype. Always very easy and working, except with SQLite that sees datatypes as something less strict than other RDBMS.
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 16, 2022, 12:36:29 pm
Just out of curiosity: How do you handle the different SQL-"Dialects"? (Yeah, nothing to do with the Problem of this Thread)
Title: Re: Again on SQLite and date times.
Post by: Thaddy on November 16, 2022, 12:42:17 pm
Just make sure you store in either Tdatetime (double) or year-month-date-time order and time in UTC.
Handle local date/time in code. But everybody knows that.
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 16, 2022, 01:17:11 pm
Just out of curiosity: How do you handle the different SQL-"Dialects"? (Yeah, nothing to do with the Problem of this Thread)

So far I didn't encounter situations in which I need it, mostly my programs do quite standard sql queryies and if there is some logic involved I prefer to keep it in code, it makes me do more work in some situations, but reduces need to dive into db specific operations (like stored procedures). 

In case it happens it would be something like

Code: Text  [Select][+][-]
  1. if (connection-database-type = SQLite) then
  2.    query.SQL := 'SQL dialect text';
  3.    SQLLite specific processing
  4. end;
  5. if (connection-database-type = MSSQL) then
  6.    query.SQL := 'MSSQL dialect text';
  7.    MSSQL specific processing
  8. end
  9.  

more or less what you suggested.
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 16, 2022, 01:21:53 pm
*snipp* or year-month-date-time order and time in UTC.
Handle local date/time in code. But everybody knows that.
Whatever for?
SQLite does offer a modifier for localtime, IF you use the Built-in Date(Time)-Functions of SQLite's own SQL-"Dialect".
If you INSERT/UPDATE DateTimes, this information comes from the Frontend in 99% of cases, and then it's usually according to system-Date/Time of the Workstation.

Of course, if you use the builtin Date/Time-Functions of SQLite itself to Insert/Update Date/Times, then there is a hideyhole for a serious Bug
kinda like
Code: SQL  [Select][+][-]
  1. INSERT INTO MyTable (ID, SomeDateTime) VALUES (1, DateTime('now'));  //THIS STORES UTC!!!!
  2. INSERT INTO MyTable (ID, SomeDateTime) VALUES (2, DateTime('now','localtime'));
  3.  
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 16, 2022, 01:22:27 pm
Just make sure you store in either Tdatetime (double) or year-month-date-time order and time in UTC.
Handle local date/time in code. But everybody knows that.

I agree.
And, extending the reasonment, in the past I was putting in DB only numeric values in SI unit, so that given a variable it was automatic to know the measure unit. But later I changed my mind and created something that allows me to store values in a somewhat more convenient measure unit system, more usefult during debugging, even if the software during it's internal calculations goes to SI. I am a maniac about these things. If my father understood about these topics I am sure he would tell me too.
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 16, 2022, 01:25:46 pm
Just out of curiosity: How do you handle the different SQL-"Dialects"? (Yeah, nothing to do with the Problem of this Thread)

So far I didn't encounter situations in which I need it, mostly my programs do quite standard sql queryies and if there is some logic involved I prefer to keep it in code, it makes me do more work in some situations, but reduces need to dive into db specific operations (like stored procedures). 

In case it happens it would be something like

Code: Text  [Select][+][-]
  1. if (connection-database-type = SQLite) then
  2.    query.SQL := 'SQL dialect text';
  3.    SQLLite specific processing
  4. end;
  5. if (connection-database-type = MSSQL) then
  6.    query.SQL := 'MSSQL dialect text';
  7.    MSSQL specific processing
  8. end
  9.  

more or less what you suggested.
Ah.... what i thought.
I've left that approach, and worked out a "generic" way to address that specific problem.
So no need anymore for DBMS-specific handling
If you're intested, send me a PM
Title: Re: Again on SQLite and date times.
Post by: Thaddy on November 16, 2022, 01:27:04 pm
@Zvoni
That is what I mean, but how a programmer handles it is often plain wrong. Of course a decent Database adheres to what I wrote. Programmers often make the wrong assumptions about date and time handling. That is all. Easy for you and me, but not for everyone.
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 16, 2022, 02:27:45 pm
@Zvoni
That is what I mean, but how a programmer handles it is often plain wrong. Of course a decent Database adheres to what I wrote. Programmers often make the wrong assumptions about date and time handling. That is all. Easy for you and me, but not for everyone.
Ah... OK
Title: Re: Again on SQLite and date times.
Post by: LacaK on November 17, 2022, 09:05:24 am
At this point I noticed that liquibase was mapping DATETIME datatype to REAL and all this story started.
It is no surprise. SQLite3 in fact stores datetime as REAL (or STRING) as stated in documentation.
So another library may recognize this data (which have real affinity) as real numbers - these real nubers represents Julian days ...
But when you use SQLite3 built-in date and time functions like datetime() on this "real" column you will see valid datetime values ...
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 17, 2022, 09:14:51 am
At this point I noticed that liquibase was mapping DATETIME datatype to REAL and all this story started.
It is no surprise. SQLite3 in fact stores datetime as REAL (or STRING) as stated in documentation.
So another library may recognize this data (which have real affinity) as real numbers - these real nubers represents Julian days ...
But when you use SQLite3 built-in date and time functions like datetime() on this "real" column you will see valid datetime values ...
We know that.
OP's problem was how TSQLQuery (and TField) determine (and set) the Datatype from the Column-MetaData, because if the Metadata of the Column is Real, he cannot use Qry.FieldbyName('SomeDate').AsDateTime
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 17, 2022, 09:56:08 am
Hi guys,
today I made a small test with SQLite related to datatype freedom. Please see what I got.

Code: Bash  [Select][+][-]
  1. user@debian:~/tmp/sqlite-test$ sqlite-cli notypes.db
  2. SQLite version 3.34.1 2021-01-20 14:10:07
  3. Enter ".help" for usage hints.
  4. sqlite> create table NoTypesTable
  5. (
  6.   field_pk integer not null,
  7.   field_int,
  8.   field_string,
  9.   field_datetime,
  10.   field_float
  11. );
  12. sqlite> insert into NoTypesTable (field_pk, field_int, field_string, field_datetime, field_float) values (1, 1, 'hello 1', '2022-11-16 14:51:36.541', 4.32);
  13. sqlite> insert into NoTypesTable (field_pk, field_int, field_string, field_datetime, field_float) values (2, 2, 'hello 2', '2022-11-16 14:52:36.541', 4.36);
  14. sqlite> insert into NoTypesTable (field_pk, field_int, field_string, field_datetime, field_float) values (3, '3', 'hello 3', 'hello 3', '4.36');
  15. sqlite> insert into NoTypesTable (field_pk, field_int, field_string, field_datetime, field_float) values (4, 'hello 4', 'hello 4', 'hello 4', 'hello 4');
  16. sqlite> select * from NoTypesTable;
  17. 1|1|hello 1|2022-11-16 14:51:36.541|4.32
  18. 2|2|hello 2|2022-11-16 14:52:36.541|4.36
  19. 3|3|hello 3|hello 3|4.36
  20. 4|hello 4|hello 4|hello 4|hello 4
  21.  

Sorry if I say a stupid thing, I am old school, but I wonder if we can still call SQLite a database. It seems to me rather a byte array logger.

Apart my uninteresting thoughts, with this example we see that column datatype definition is optional, and even different rows of a table can contain unrelated datatypes.

Is the concept of datatype itself to be senseless here.
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 17, 2022, 10:05:10 am
Sorry if I say a stupid thing, I am old school, but I wonder if we can still call SQLite a database. It seems to me rather a byte array logger.
Apart my uninteresting thoughts, with this example we see that column datatype definition is optional, and even different rows of a table can contain unrelated datatypes.
Is the concept of datatype itself to be senseless here.
It's a clear Yes and No  :D
SQLite offers the flexibility to use it kind of in a "typeless" mode (or weakly typed)
OTOH, there is the STRICT-Qualifier --> https://www.sqlite.org/stricttables.html
Quote
SQLite supports a strict typing mode, as of version 3.37.0 (2021-11-27), that is enabled separately for each table.
.....
.....
If the value cannot be losslessly converted in the specified datatype, then an SQLITE_CONSTRAINT_DATATYPE error is raised. 
Bottom Line: Use the modifier, and you get what you're used to
BUT: As the doc's say
Quote
Every column definition must specify a datatype for that column. The freedom to specify a column without a datatype is removed.

The datatype must be one of following:

    INT
    INTEGER
    REAL
    TEXT
    BLOB
    ANY

No other datatype names are allowed, though new types might be added in future releases of SQLite.
So, no DATETIME, VARCHAR and companions allowed (at least up until now)
Title: Re: Again on SQLite and date times.
Post by: LacaK on November 17, 2022, 06:28:45 pm
..., because if the Metadata of the Column is Real, he cannot use Qry.FieldbyName('SomeDate').AsDateTime

He can use AsDateTime as long as real values stored in such column satisfy: "..., the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic Gregorian calendar. "
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 17, 2022, 08:53:29 pm
..., because if the Metadata of the Column is Real, he cannot use Qry.FieldbyName('SomeDate').AsDateTime

He can use AsDateTime as long as real values stored in such column satisfy: "..., the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic Gregorian calendar. "
No, he can‘t, as long as the returned Fieldtype is not ftDateTime.
Tested and confirmed.
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 18, 2022, 02:31:47 pm
If I understand correctly, this is the place where the problem arises.

In "C:\lazarus\fpc\3.2.2\source\packages\fcl-db\src\sqlite\sqliteds.pas":

Code: Pascal  [Select][+][-]
  1. procedure TSqliteDataset.RetrieveFieldDefs;
  2. var
  3.   ColumnCount, i, DataSize:Integer;
  4.   AType: TFieldType;
  5.   vm: Pointer;
  6.   ColumnNames, ColumnValues:PPAnsiChar;
  7.   ColumnStr: String;
  8. begin
  9.   FieldDefs.Clear;
  10.   FAutoIncFieldNo := -1;
  11.   FReturnCode := sqlite_compile(FSqliteHandle, PAnsiChar(FEffectiveSQL), nil, @vm, nil);
  12.   if FReturnCode <> SQLITE_OK then
  13.     DatabaseError(ReturnString, Self);
  14.   sqlite_step(vm, @ColumnCount, @ColumnValues, @ColumnNames);
  15.   //Prepare the array of pchar2sql functions
  16.   SetLength(FGetSqlStr, ColumnCount);
  17.   // Sqlite is typeless (allows any type in any field)
  18.   // regardless of what is in Create Table, but returns
  19.   // exactly what is in Create Table statement
  20.   // here is a trick to get the datatype.
  21.   // If the field contains another type, may have problems
  22.   for i := 0 to ColumnCount - 1 do
  23.   begin
  24.     DataSize := 0;
  25.     ColumnStr := UpperCase(String(ColumnNames[i + ColumnCount]));
  26.     if (ColumnStr = 'INTEGER') or (ColumnStr = 'INT') then
  27.     begin
  28.       if AutoIncrementKey and
  29.            (UpperCase(String(ColumnNames[i])) = UpperCase(PrimaryKey)) then
  30.       begin
  31.         AType := ftAutoInc;
  32.         FAutoIncFieldNo := i;
  33.       end
  34.       else
  35.         AType := ftInteger;
  36.     end else if Pos('VARCHAR', ColumnStr) = 1 then
  37.     begin
  38.       AType := ftString;
  39.       DataSize := StrToIntDef(Trim(ExtractDelimited(2, ColumnStr, ['(', ')'])), DefaultStringSize);
  40.     end else if Pos('BOOL', ColumnStr) = 1 then
  41.     begin
  42.       AType := ftBoolean;
  43.     end else if Pos('AUTOINC', ColumnStr) = 1 then
  44.     begin
  45.       AType := ftAutoInc;
  46.       if FAutoIncFieldNo = -1 then
  47.         FAutoIncFieldNo := i;
  48.     end else if (Pos('FLOAT', ColumnStr)=1) or (Pos('NUMERIC', ColumnStr) = 1) then
  49.     begin
  50.       AType := ftFloat;
  51.     end else if (ColumnStr = 'DATETIME') then
  52.     begin
  53.       AType := ftDateTime;
  54.     end else if (ColumnStr = 'DATE') then
  55.     begin
  56.       AType := ftDate;
  57.     end else if (ColumnStr = 'TIME') then
  58.     begin
  59.       AType := ftTime;
  60.     end else if (ColumnStr = 'LARGEINT') or (ColumnStr = 'BIGINT') then
  61.     begin
  62.       AType := ftLargeInt;
  63.     end else if (ColumnStr = 'TEXT') then
  64.     begin
  65.       AType := ftMemo;
  66.     end else if (ColumnStr = 'CURRENCY') then
  67.     begin
  68.       AType := ftCurrency;
  69.     end else if (ColumnStr = 'WORD') then
  70.     begin
  71.       AType := ftWord;
  72.     end else
  73.     begin
  74.       AType := ftString;
  75.     end;
  76.     FieldDefs.Add(FieldDefs.MakeNameUnique(String(ColumnNames[i])), AType, DataSize);
  77.     //Set the pchar2sql function
  78.     case AType of
  79.       ftString:
  80.         FGetSqlStr[i] := @Char2SQLStr;
  81.       ftMemo:
  82.         FGetSqlStr[i] := @Memo2SQLStr;
  83.     else
  84.       FGetSqlStr[i] := @Num2SQLStr;
  85.     end;
  86.   end;
  87.   sqlite_finalize(vm, nil);
  88.   {
  89.   if FReturnCode <> SQLITE_ABORT then
  90.      DatabaseError(ReturnString,Self);
  91.   }
  92. end;
  93.  

I was thinking that a possible way is:

1. tell the TSQLQuery that we want it not to try to reconstruct the AType because we will force it.
2. make in a way that this info is passed to this method that will avoid any guessing.
3. set the AtType manually from outside
4. call the As<something> we like.

or event better

1. modify TField so that AType can be forced
2. leave TSQLQuery do all its guessings
3. force for a certain Tfield to have the AType we want
4. call the As<something> we like on that TField
Title: Re: Again on SQLite and date times.
Post by: Zvoni on November 18, 2022, 02:47:41 pm
I was thinking about that, too, this morning when i was driving to work.

I was thinking about maybe a Class Helper for TField (because it's TField offering "AsDateTime"), overriding the Property AsDateTime including its Getter/Setter

But there is still something i don't understand:
https://www.freepascal.org/docs-html/fcl/db/tfield.asdatetime.html
Quote
If the native type of the field is not a date or time value, then an attempt will be made to convert the field value from the native format to a TDateTime value when reading the field's content.
...
Therefor, when reading or writing a field value for a field whose native data type is not a TDateTime-compatible value (dates or string values), an exception may be raised.
That's what i don't understand.
In my tests, if in the SQLite-Table the Column was declared as REAL, AsDateTime failed with the Error-Message from the beginning of the Thread.
I could read it as a Float.

Funny thing: In my test i had two columns, one delcared REAL, the other declared DATETIME (a DataType which officially (!!) doesn't exist in SQLite)
I entered exactly the same floating point values in both columns

But i had no problem to
Code: Pascal  [Select][+][-]
  1. Var
  2.   d:TDateTime;
  3. Begin
  4.   d:=Query.FieldByName('DateAsReal').AsFloat;  //No Exception!
  5.   writeln(d);
  6.   d:=Query.FieldByName('DateAsDateTime').AsDateTime;  //No Exception!
  7.   writeln(d);
  8. End;
  9.  
both times the writeln showed the same value.

That's what threw me off.

Anyone a clou if there is a bug here?
Quote
then an attempt will be made to convert the field value from the native format to a TDateTime
Title: Re: Again on SQLite and date times.
Post by: LacaK on November 22, 2022, 12:15:34 pm
C:\lazarus\fpc\3.2.2\source\packages\fcl-db\src\sqlite\sqliteds.pas is not related to TSQLite3Connection and TSQLQuery
You must look into C:\lazarus\fpc\3.2.2\source\packages\fcl-db\src\sqldb\sqlite\sqlite3conn.pp

According to TField (I wrote it before): do not look at methods of TField but at TDateTimeField.
For given FieldType, in our case ftDateTime is created corresponding Field object. In our case TDateTimeField. So TField does not enters into "game".
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 22, 2022, 12:30:26 pm
For given FieldType, in our case ftDateTime is created corresponding Field object. In our case TDateTimeField. So TField does not enters into "game".

The problem is that in the situation described we do not obtain a FieldType whose value is a ftDateTime.

The only way to have this is to create the table in SQLite indicating the data type "DATETIME".
For all other cases in which is possible to write a datetime (1. REAL, 2.TEXT, 3.NO DATATYPE) the fp library does not understand that the values we are reading are TDateTime, so if you try to call AsDateTime in this case you get a Exeception. The reason why our library does not understand that we are dealing with a TDateTime is that uses the "DATETIME" in the table creation statement to understand what kind of field type is this. In SQLite such a question is more or less without meaning, due to it's type freedom.

Just try creating a table with a REAL field, popolate it with TDateTime double values a read it doing AsDateTime. I will get the Execption.

Title: Re: Again on SQLite and date times.
Post by: LacaK on November 22, 2022, 12:41:08 pm
The problem is that in the situation described we do not obtain a FieldType whose value is a ftDateTime.
The only way to have this is to create the table in SQLite indicating the data type "DATETIME".
Yes this is the only way - use DATETIME as column data type in create table statement.

Just try creating a table with a REAL field, popolate it with TDateTime double values a read it doing AsDateTime. I will get the Execption.
No use REAL as column data type is not a way.
Title: Re: Again on SQLite and date times.
Post by: Thaddy on November 22, 2022, 01:15:55 pm
Yes no workarounds. Stick to that advice.
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 22, 2022, 01:23:04 pm
Yes this is the only way - use DATETIME as column data type in create table statement.
No use REAL as column data type is not a way.

Of course your suggestions are senseful and I could do as you say.

But there are at least two situations in which these are not applicable with the current status to the free pascal SQLite library:

1. Situations where SQLite table creation statements are managed by software tools (in my case was liquibase) and not written directly. For example, liquibase automatically remaps the columns DATETIME to REAL when generating the actual SQLite creation statement. One could force with tricks to stick to DATETIME in the generated SQL, but it is senseful what liquibase does, since the storage class is REAL. I don't think the liquibase guys are stupid to make this remapping automatically.
2. Situations in which I cannot create the table creation statements because they are pre-existing.

My idea is that the free pascal SQLite library should be modified so that:

1. if the column is DATETIME then is read as such (and this is already like this now)
2. if the column is of a type that allows DATETIME to be stored (REAL, TEXT, NOTHING) allow the client of the library to tell that a value is AsDateType even if the column seems be something else (this is what is missing).

I will try to see If I can make it on my own and later to make it publicly available. 

@Thaddy: we should give the same order to the ones dealing with the situations 1 and 2.
Title: Re: Again on SQLite and date times.
Post by: LacaK on November 22, 2022, 07:57:29 pm
Yes I understand.
At client side you can use event handler TField.OnGetText, OnSetText https://www.freepascal.org/docs-html/fcl/db/tfield.ongettext.html to handle content of column and prepare it for DB-aware controls (such as TDBGrid etc.)
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 23, 2022, 09:34:08 am
Yes I understand.
At client side you can use event handler TField.OnGetText, OnSetText https://www.freepascal.org/docs-html/fcl/db/tfield.ongettext.html to handle content of column and prepare it for DB-aware controls (such as TDBGrid etc.)

This is for sure one possible way.

However it does not solve the original problem:

Code: Text  [Select][+][-]
  1. given Table = (dateTimeColumn REAL) // liquibase did that, I asked for DATETIME but he did that, blame on it
  2.  
  3. qwrite = 'insert into Table (dateTimeColumn) values (:dateTimeColumn)';
  4. qwrite.ParamByName('dateTimeColumn').AsDateTime := Today;
  5.  
  6. qread = 'select dateTimeColumn from Table';
  7. value := qread.FiledByName('dateTimeColumn').AsDateTime; // -> Exception is thrown
  8.  
  9.  

Clients (like I was) could be confused when this occurs.
So I still think that is necessary something that solves this problem.
Title: Re: Again on SQLite and date times.
Post by: Thaddy on November 23, 2022, 09:48:54 am
External tools that do no adhere to the standard are not supported on this platform, nor on the Sqlite platforms.
Drop the external tool because it is broken.
Lacak - who is the walking encyclopedia concerning databases and Freepascal - gave you enough info on how to properly handle it.
I am not going to investigate any further, since it is the external tool that is broken, not FPC's bindings and not Sqlite itself.
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 23, 2022, 10:14:58 am
External tools that do no adhere to the standard are not supported on this platform, nor on the Sqlite platforms.
Drop the external tool because it is broken.
Lacak - who is the walking encyclopedia concerning databases and Freepascal - gave you enough info on how to properly handle it.
I am not going to investigate any further, since it is the external tool that is broken, not FPC's bindings and not Sqlite itself.

I also trust that Lacak is a walking encyclopedia databases and Freepascal.

Liquibase broken? Maybe. If you help me, we could try to advance these observation to their forum. Maybe supported by previous statement.

But here things are very simple: the normal (and IMHO very senseful) approach to the db access software based on the concept of field with a "fixed/explicit datatype" does not work. Not because is wrong, just because SQLite does not respect this fixed/explicit datatype model common to all the normal dbs. They say is a feature. I do not discuss this. Of course this requires to write a software that copes with it. And current software does not. As I repeatedly demonstrated with many examples, that everyone can reproduce on it own PC if necessary.

Thanks for investigating so far.
Title: Re: Again on SQLite and date times.
Post by: LacaK on November 25, 2022, 01:44:20 pm
You can create descendant of TSQLQuery where you can override virtual method GetFieldClass() and return own TField descendant which will handle AsDateTime in a special way
(you can create own TSqliteFloatField = class(FloatField) ... define there GetAsDateTime and use this new field type for REAL columns (ftFloat))
For usage you can look at fpjsondataset.
Title: Re: Again on SQLite and date times.
Post by: Чебурашка on November 29, 2022, 10:03:20 pm
You can create descendant of TSQLQuery where you can override virtual method GetFieldClass() and return own TField descendant which will handle AsDateTime in a special way
(you can create own TSqliteFloatField = class(FloatField) ... define there GetAsDateTime and use this new field type for REAL columns (ftFloat))
For usage you can look at fpjsondataset.

This week I am kinda busy, but next I have some more time. This hint will be very useful I am sure.
Thanks
TinyPortal © 2005-2018