Recent

Author Topic: [How To] Designing a Program to support multiple DBMS  (Read 12167 times)

Zvoni

  • Hero Member
  • *****
  • Posts: 3361
[How To] Designing a Program to support multiple DBMS
« on: November 18, 2022, 11:02:18 am »
Hi Folks,

on request of a User i'm going to show/explain how i code Frontends which have to support multiple DBMS, without having to change your code of the Frontend.

Up Front:
1) There will be not that much FPC-Code here. It's more about the concept/systematic behind it.
2) It's mostly about SQL and Database-techniques
3) You can keep any Syntax-Errors you find  :D
3) I don't use visual components. Period.

Peer-Review is welcome. Someone might even point out a pitfall i missed.

Off we go......

Let's say, you've started to code a Database-Program.
You start out with your client (GUI) and, say, SQLite. (e.g. I always start with SQLite)

You often end up with code looking like this
Code: Pascal  [Select][+][-]
  1. Var
  2.     sSQL:String;
  3. Begin
  4.    //Example is: Show me the 10 best paid employees for that Department
  5.    sSQL:='SELECT ID, EmpName, EmpSalary FROM employees WHERE DepartmentID=:DepID ORDER BY EmpSalary DESC LIMIT 10;';
  6.    MyQuery.SQL.Text:=sSQL;  //Or assigning the SQL-Statement directly
  7.    MyQuery.ParamByName('DepID').AsInteger:=6;
  8.    MyQuery.Open;
  9. End;
  10.  
and so on and so forth.
After some time, your source-code becomes littered/cluttered with SQL-Statements.
Then you decide: OK, this is a maintenance nightmare. Let's move the SQL-Statements to a separate area (might even be a separate unit) and declare them as Constants
Code: Pascal  [Select][+][-]
  1. Const
  2.    SELECT_TOP10_EMP_SALARY='SELECT ID, EmpName, EmpSalary FROM employees WHERE DepartmentID=:DepID ORDER BY EmpSalary DESC LIMIT 10;';
  3. .
  4. .
  5. Begin
  6.    MyQuery.SQL.Text:=SELECT_TOP10_EMP_SALARY;  
  7.    MyQuery.ParamByName('DepID').AsInteger:=6;
  8.    MyQuery.Open;
  9. End;
  10.  

Looks familiar?

After some time you think (or your customer decides): Hey, would be a neat idea to support MS-SQL-Server, too.
Let's integrate it.
Most people (at least that's what i found out) end up with code like this:
Code: Pascal  [Select][+][-]
  1. Const
  2.    SELECT_TOP10_EMP_SALARY_SQLITE='SELECT ID, EmpName, EmpSalary FROM employees WHERE DepartmentID=:DepID ORDER BY EmpSalary DESC LIMIT 10;';
  3.    SELECT_TOP10_EMP_SALARY_MSSQL='SELECT TOP(10) ID, EmpName, EmpSalary FROM employees WHERE DepartmentID=:DepID ORDER BY EmpSalary DESC;';
  4. .
  5. .
  6. begin
  7.    Case  Connectiontype Of
  8.         SQLite: MyQuery.SQL.Text:=SELECT_TOP10_EMP_SALARY_SQLITE;
  9.         MSSQL: MyQuery.SQL.Text:=SELECT_TOP10_EMP_SALARY_MSSQL;  
  10.    End;
  11.    MyQuery.ParamByName('DepID').AsInteger:=6;
  12.    MyQuery.Open;
  13. End;
  14.  
You see the difference? It's the SQL-Dialect

What i do:
I store the SQL-Statements themselves in the Database.
Such a table looks like this:
IDStatementConstantConnType
1SELECT ID, EmpName, EmpSalary FROM employees WHERE DepartmentID=:DepID ORDER BY EmpSalary DESC LIMIT 10;1000SQLite
4SELECT TOP(10) ID, EmpName, EmpSalary FROM employees WHERE DepartmentID=:DepID ORDER BY EmpSalary DESC;1000MSSQL

Now before actually firing of the SQL-Statement i want to display the result to the User, i first retrieve the SQL-Statement from the Database according to DBMS
Code: Pascal  [Select][+][-]
  1. //This is usually in its own unit
  2. Interface
  3. //This is it. This Function is a "Write Once" - "Use everywhere". Could even be a Method of a "public" Class or whatever
  4. Function GetSQLStatement(Const AStatement:Integer; Const AConnType:String):String;
  5.  
  6. //i usually have a second TSQLQuery-Object, or even use the "global" one. Here i use a second TSQLQuery-Object
  7. Implementation
  8. Function GetSQLStatement(Const AStatement:Integer; Const AConnType:String):String;
  9. Const
  10.    GetSQL='SELECT Statement FROM tbl_sql WHERE Constant=:pConst AND ConnType:=:pConnType;';  //This is ANSI-SQL!!! Every DBMS should understand this!
  11. Var
  12.    QrySQL:TSQLQuery;
  13. begin
  14.    QrySQL:=TSQLQuery.Create;
  15.    //Connect to Transaction and Database. I'll omit that here
  16.    QrySQL.SQL.Text:=GetSQL;
  17.    QrySQL.ParamByName('pConst').AsInteger:=AStatement;  //First Argument of the Function. Which Statement?
  18.    QrySQL.ParamByName('pConnType').AsString:=AConnType;  //Second Argument of the Function. For which DBMS?
  19.    QrySQL.Open;
  20.    //Sanity-check if recordCount=1?
  21.    Result:=QrySQL.Fieldbyname('Statement').AsString;
  22.    QrySQL.Free;
  23. End;
  24.  
  25. //This is somewhere in another unit. Of course, you have to "use" the unit above
  26. Uses SQLUnit;
  27.  
  28. Const
  29.    SELECT_TOP10_EMP_SALARY=1000;
  30.  
  31. //Again somewhere else in your code-unit
  32. Begin
  33.    MyQuery.SQL.Text:=GetSQLStatement(SELECT_TOP10_EMP_SALARY, MyConnType);
  34.    //MyConnType could be a global value retrieved/set at the beginning, or it could even be the Connection-Type of TSQLConnector!!!
  35.    //That's why i used DataType String/Text for ConnType
  36.    MyQuery.ParamByName('DepID').AsInteger:=6;
  37.    MyQuery.Open;
  38.  

The only thing you should research beforehand is, if the Database-Systems (SQLite, MySQL, MSSQL etc.) you want to support,
support named Parameters, and in which format
I used here the format "... SomeField=:SomeParam..."
There is also the Format "..... SomeField=@SomeParam..." but i haven't used it so far.

Now try to apply this systematic/technique to all the other SQL-Statements (INSERT, UPDATE, DELETE, CREATE TABLE/VIEW etc.) across the different DBMS and their own SQL-Dialects.
Nevermind SQL-Dialects differing in functional areas
e.g. SQLite supports the ROW_NUMBER-Window-Function.
MySQL does, too..... from version 8 up
What to do if customer uses MySQL5.7? You have to use the Workaround to simluate the ROW_NUMBER-Function (there is one!)

Oh, and i'm sure you can see how you can maintain the Statements themselves outside your app with an DB-Admin-Program (DB-browser for SQLite, MySQL-Workbench),
if your customer decides: "Nahh.... change that Statement to show the Top 5 employees" (And no, we're not going to discuss to parametrize that value :)).

You can adjust/maintain the Statements outside your app to your hearts content, as long as the "interface" of the Statement doesn't change.
In my example above, it's 3 Output-Columns (Integer, String/Text, Double) and 1 Input-Value (Integer)

Thoughts?
Opinions?
Reviews?

Feel free to add here.
We might end up with a Wiki-Entry for this
« Last Edit: November 18, 2022, 11:06:40 am by Zvoni »
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

MarkMLl

  • Hero Member
  • *****
  • Posts: 8565
Re: [How To] Designing a Program to support multiple DBMS
« Reply #1 on: November 18, 2022, 11:42:54 am »
I think you're saying that's distinct from any stored procedure facility offered by the servers (the details of which might, of course, be server- and possibly version-specific.

I've been down that route to fairly good effect, although in my case (a) I stored fairly comprehensive scripting in the database, (b) had the details of the connection stored in a .ini file and (c) saved copies of the .ini files in tables whence they could be easily retrieved if setting up another client system (application server).

I settled on PostgreSQL fairly early, but note that Firebird's SQL dialect is very similar and your SQLite fragment is also close. I did at one point have to change date format in some queries when Postgres became more "standards compliant".

I admit to being a little wary of stored procedures (and triggers etc.) since it can be very difficult to reconstruct everything that's needed for workflow. As a result, I recreate almost everything when required from the custom scripting: as such all that's needed when doing a server update is to move the data tables and start running the client-side scripts.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Чебурашка

  • Hero Member
  • *****
  • Posts: 593
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: [How To] Designing a Program to support multiple DBMS
« Reply #2 on: November 18, 2022, 11:55:54 am »
I like the idea of not having in code the specific db depended sql text, and storing it in a "code-extension-db" is nice. I suggest to consider another possible persistence locus, for example in text file(s) of filesystem, after all the db in this case acts only as storage place of something "fixed" because follows what is implemented in code.

I never had this need because my queries are not db dependent, but in case this need arises, this approach is a valid solution.

One question: from what source are the db specific sqls stored in db initially? I mean: in a software solution that makes something one probably starts with some (several) features out of the box. In this case several queries should already exists at first program execution, say after (pay,) dowload and install.
« Last Edit: November 18, 2022, 12:00:22 pm by tt »
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

nummer8

  • Full Member
  • ***
  • Posts: 124
Re: [How To] Designing a Program to support multiple DBMS
« Reply #3 on: November 18, 2022, 11:59:58 am »
Your solution is up for debate. There are always pros and cons for this method.
I am not interested in tha discussion that is a matter of choice in a specific situation.

One thing I want to mention for your method is that I would not use const numbers to identifiy the query.
With a limited amount of query's you know what const belongs to what query. Beyond that amount you have to maintain al ist of const/ query.
A small alteration could be to add an extra field where you write down a decriptive name just like in your first two examples. "SELECT_TOP10_EMP_SALARY"
In this way you have all the information in the DB.

Further more there is the question do you use your DB server as a real server or just as a data container. (Same remark as MArkMli) Meaning: what do you do with the triggers, functions and stored procedures. Do you use the DB- server for that or do you handle that in the software.

Just my 2ct.
 
Nummer8

MarkMLl

  • Hero Member
  • *****
  • Posts: 8565
Re: [How To] Designing a Program to support multiple DBMS
« Reply #4 on: November 18, 2022, 12:10:37 pm »
One thing I want to mention for your method is that I would not use const numbers to identifiy the query.
With a limited amount of query's you know what const belongs to what query.

What is the internationalisation convention for naming phrases?

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 18982
  • Glad to be alive.
Re: [How To] Designing a Program to support multiple DBMS
« Reply #5 on: November 18, 2022, 12:11:35 pm »
I would also mention ODBC and CLI in this context.
Quote
I admit to being a little wary of stored procedures (and triggers etc.)
Stored procedures and triggers can simplify the client code a lot with the chance that you can achieve client code that does not need to change much or at all between dialects. Stored procedures and triggers also contribute to much safer code.
Recovered from removal of tumor in tongue following tongue reconstruction with a part from my leg.

Чебурашка

  • Hero Member
  • *****
  • Posts: 593
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: [How To] Designing a Program to support multiple DBMS
« Reply #6 on: November 18, 2022, 12:18:15 pm »
One extra observation.

In complex SQL queries it might happen that the operations possible in one DB are not possible in another DB (example DB that does not support outer join).
In cases like this there might come up the possibility of postprocessing made at code level in order to obtain the same final result equivalently on the 2 different DBMS.

In a situation like this, even if the statements do not appear in code, code will still show this like

Code: Pascal  [Select][+][-]
  1. if connType = x
  2.     do logic for x
  3. elseif connType = y
  4.     do logic for y
  5.  

But this in not avoidable and does not make the solution proposed less interesting.
« Last Edit: November 18, 2022, 12:20:12 pm by tt »
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

MarkMLl

  • Hero Member
  • *****
  • Posts: 8565
Re: [How To] Designing a Program to support multiple DBMS
« Reply #7 on: November 18, 2022, 12:32:43 pm »
I would also mention ODBC and CLI in this context.
Quote
I admit to being a little wary of stored procedures (and triggers etc.)
Stored procedures and triggers can simplify the client code a lot with the chance that you can achieve client code that does not need to change much or at all between dialects. Stored procedures and triggers also contribute to much safer code.

I don't deny that, I'm pointing out that they can be a management issue.

In complex SQL queries it might happen that the operations possible in one DB are not possible in another DB (example DB that does not support outer join).

Outer joins were certainly an issue at one point, as was the inability of at least one popular database to do a rollback. However a better example might these days be patchy support for analytic (AKA window) functions: I have in the past resorted to APL post-processing to extract e.g. rate-of-change information.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Zvoni

  • Hero Member
  • *****
  • Posts: 3361
Re: [How To] Designing a Program to support multiple DBMS
« Reply #8 on: November 18, 2022, 12:57:07 pm »
Holy smoke!
i didn't expect so many reactions in this short a time.

@Mark
Yes, it's distinct from stored procedures (at least for me), since not all DBMS support stored procedures (SQlite being the prominent one).
Triggers.... Good point. Would have to think on it.

@tt
Yes, there has to be an "initial" catalogue locally somewhere. Could be simple Textfiles (*.sql-Extensions, whatever)
As to persistence locus (fun fact: in germany the word "locus" is often used for "shitter", "toilett" :))
Yes, i was thinking about it. If it's local, you have to provide that File (or whatever) to each client. Might result in maintenance-nightmare again.
If it's part of the Database itself (especially server-based DBMS), then you just rollout the Client, configure connection, and off you go

@nummer8
I declare my consts in the unit they are used in.
It makes no sense (at least to me, no offense) to have a Const for employees available (!!) in a unit where i handle suppliers.

@"Outer Join"
I take it you're talking about Full Outer Join, since i wouldn't know of a single DBMS which doesn't support Left Join (and a Right Join is just a reversed Left Join).
DBMS not supoorting Full outer Join are the two prominent ones : SQLite and MySQL
But there is a Workaround for it.
It was one of my other topics for the "HowTo"'s i asked about in "Other" Thread

Comments welcome
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

Чебурашка

  • Hero Member
  • *****
  • Posts: 593
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: [How To] Designing a Program to support multiple DBMS
« Reply #9 on: November 18, 2022, 01:07:02 pm »
Not being the sql text compiled in the executable, but loaded at runtime from a external source, might also lead to security issues.

What if mr FunnyHacker changes the query (1034, *) to 'drop database';

(I know, in some cases this could be also possible in the case sql text is compiled into the exe, by patching it).
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

Чебурашка

  • Hero Member
  • *****
  • Posts: 593
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: [How To] Designing a Program to support multiple DBMS
« Reply #10 on: November 18, 2022, 01:13:29 pm »
If it's part of the Database itself (especially server-based DBMS), then you just rollout the Client, configure connection, and off you go

Yes I was also thinking about the advantage of using a DB for storing, but it is mandatory that is a centralized db server (for example using SQLite as queries repo is equivalent to using files on fs).
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

Zvoni

  • Hero Member
  • *****
  • Posts: 3361
Re: [How To] Designing a Program to support multiple DBMS
« Reply #11 on: November 18, 2022, 01:20:41 pm »
If it's part of the Database itself (especially server-based DBMS), then you just rollout the Client, configure connection, and off you go

Yes I was also thinking about the advantage of using a DB for storing, but it is mandatory that is a centralized db server (for example using SQLite as queries repo is equivalent to using files on fs).
Yes, but if you use SQLite for Multi-user-access, then there is something wrong from the get go, though it would still be the "central" database (-file)

As for the "Hacker"-Comment: Correct.
But, encrypt your Statements?
reading "zurOBfJHE89376298JjfuUuumdmele" instead of "SELECT Bla FROM" ...... :-)
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

MarkMLl

  • Hero Member
  • *****
  • Posts: 8565
Re: [How To] Designing a Program to support multiple DBMS
« Reply #12 on: November 18, 2022, 01:38:00 pm »
Yes, but if you use SQLite for Multi-user-access, then there is something wrong from the get go, though it would still be the "central" database (-file)

I'm not sure whether this would be better described as "multi-user" or "multi-host"... but whatever it's called, it's best avoided :-)

Just because a server program's been retrofitted for a modicum of network operation- or some equivalent using VMs or containers- doesn't mean that its use in that role is defensible. In the community built (loosely) on top of Borland products we've got the painful example of the number of times Paradox was overextended, and the less said about MySQL or the early days of most databases' clustering the better...

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Zvoni

  • Hero Member
  • *****
  • Posts: 3361
Re: [How To] Designing a Program to support multiple DBMS
« Reply #13 on: November 18, 2022, 01:44:10 pm »
Yes, but if you use SQLite for Multi-user-access, then there is something wrong from the get go, though it would still be the "central" database (-file)

I'm not sure whether this would be better described as "multi-user" or "multi-host"... but whatever it's called, it's best avoided :-)

Just because a server program's been retrofitted for a modicum of network operation- or some equivalent using VMs or containers- doesn't mean that its use in that role is defensible. In the community built (loosely) on top of Borland products we've got the painful example of the number of times Paradox was overextended, and the less said about MySQL or the early days of most databases' clustering the better...

MarkMLl
Correct.

The purpose of this Thread is to display "my way" (to quote Frank Sinatra), how i do what and why, and invite everyone to chip in.
As said in the preamble:
I might learn something i missed
But for someone else it might be the holy grail ("Why have i never thought of that?")

If at the end the result is kind of a "this is the best practice in that scenario" on the wiki, then i would celebrate.
if there is only one single user saying "thanks, never thought of that." i'd consider this thread a capital success

i changed my style of Database-coding completely, after discovering above mentioned systematic/technique.
I even figured out a method to parametrize Column- and Tablenames using above technique (would have been another topic for the "HowTo"'s)
« Last Edit: November 18, 2022, 01:47:38 pm by Zvoni »
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

Чебурашка

  • Hero Member
  • *****
  • Posts: 593
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: [How To] Designing a Program to support multiple DBMS
« Reply #14 on: November 18, 2022, 01:46:18 pm »
if there is only one single user saying "thanks, never thought of that." i'd consider this thread a capital success

I think you can celebrate then, because if somebody has this need, this can anyway can be a good hint about a possible way to go.
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

 

TinyPortal © 2005-2018