Recent

Author Topic: How to combine data from several DBs?  (Read 894 times)

Vodnik

  • Full Member
  • ***
  • Posts: 214
How to combine data from several DBs?
« on: January 08, 2025, 01:45:48 pm »
My application works with Informix DB, creating different reports from it. DB is contolled by Call Center and it is not allowed to make changes in it from outside.
Now I want to improve reports with additional info that is not contained in the main DB, but in huge telephony log files.
For that purpose I've created a tool that get data from log files and put them into SQLite DB.
Tables in both DBs have an ID field that can be used for data linking.
I'm using DBgrid control for reporting aim in my application.

Can you recommend an approach how to deal with two different DBs?

(Well, I have a couple of straightforward ideas, like:
- get data from Informix, put it into TEMP table in SQLite, then select with join from 2 tables by SQLite,
- or built DBGrid from Informix, providing spare column, then fill that column with data from SQLite)

I just don't want to invite a bicycle, missing some good solution that may be already available.

Zvoni

  • Hero Member
  • *****
  • Posts: 2782
Re: How to combine data from several DBs?
« Reply #1 on: January 08, 2025, 02:12:59 pm »
AFAIK, no other way you already described.
I'd prefer Solution 1 (Report based on temp-table in SQLite)
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

TRon

  • Hero Member
  • *****
  • Posts: 3742
Re: How to combine data from several DBs?
« Reply #2 on: January 08, 2025, 02:22:34 pm »
For local databases I use attach database but I do not know if that works icw with informix server.
I do not have to remember anything anymore thanks to total-recall.

Zvoni

  • Hero Member
  • *****
  • Posts: 2782
Re: How to combine data from several DBs?
« Reply #3 on: January 08, 2025, 02:34:25 pm »
For local databases I use attach database but I do not know if that works icw with informix server.
AFAIK, that only works with "other" sqlite-DB's
Don't know if Informix itself offers something like "federated tables" as known from MySQL (which is basically the same with "attach database" in SQLite).
Probably would need a (ODBC) driver
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

Thaddy

  • Hero Member
  • *****
  • Posts: 16300
  • Censorship about opinions does not belong here.
Re: How to combine data from several DBs?
« Reply #4 on: January 11, 2025, 01:58:04 pm »
You can use an informix client locally and use any of the dml sql to create reports over multiple
informix databases. That looks like the best solution. See the informix documentation:
https://www.ibm.com/docs/en/informix-servers/12.10?topic=rrda-sql-statements-that-access-more-than-one-database
That way you do not need to rely on logs or imports and your architecture is not affected.
If that is not an option, then, indeed, your option one is the next best solution, but in a professional setting (which informix always is, it is prohibitatively expensive although there is a "free" developer edition) I would do what I described and keep it all informix. Both informix and oracle have very powerful options to query over multiple databases, even domains and clusters, much more so than other database engines.
I am more familiar with the Oracle family and there it is not much of a question and relatively easy. I suppose it is the same with Informix.
« Last Edit: January 12, 2025, 10:56:40 am by Thaddy »
If I smell bad code it usually is bad code and that includes my own code.

Zvoni

  • Hero Member
  • *****
  • Posts: 2782
Re: How to combine data from several DBs?
« Reply #5 on: Today at 08:13:09 am »
You can use an informix client locally and use any of the dml sql to create reports over multiple
informix databases. That looks like the best solution. See the informix documentation:
This only applies if he's allowed to do "changes" on the Informix-DB.
My guess he's not allowed or he isn't a full dba for the Informix-DB

Quote
That way you do not need to rely on logs or imports and your architecture is not affected.
How do you figure that?
He stated clearly, that he needs Data from some Logs, which apparently is not available anywhere else (guessing here)


Quote
If that is not an option, then, indeed, your option one is the next best solution, but in a professional setting (which informix always is, it is prohibitatively expensive although there is a "free" developer edition) I would do what I described and keep it all informix. Both informix and oracle have very powerful options to query over multiple databases, even domains and clusters, much more so than other database engines.
I am more familiar with the Oracle family and there it is not much of a question and relatively easy. I suppose it is the same with Informix.
Correct. If he can go "all Informix" that's the way to go.
It definitely is NOT a problem to query across multiple Databases/schema, as long as they are on the same server/connection.
I'm doing this every day on our IBM DB2.

That said: If my guess is right (he's not full dba), maybe their IT can give him his own schema with corresponding access-rights (full CRUD).
Then he could import his logs there, and then it would be "all Informix"
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

Thaddy

  • Hero Member
  • *****
  • Posts: 16300
  • Censorship about opinions does not belong here.
Re: How to combine data from several DBs?
« Reply #6 on: Today at 11:44:31 am »
I was guessing too. It is definitely open for interpretation.
To create a report, he only needs read access of course.

@vodnik
Can you report back?
« Last Edit: Today at 11:50:20 am by Thaddy »
If I smell bad code it usually is bad code and that includes my own code.

Vodnik

  • Full Member
  • ***
  • Posts: 214
Re: How to combine data from several DBs?
« Reply #7 on: Today at 02:18:50 pm »
Thank you for ideas.
Well, I have full admin rights over Informix DB. I can create additional tables or even DBs for my own aims, but that breaks the rules of Call Center vendor. They have their own ideas for DB management, replication, backup, etc. They also are very strict with DB load; if I start a SELECT statement that will run for 10 minutes, Call Center operation may degrade. This makes even read-only access dangerous.
So, though pure Informix solution is the best from any point of view, I prefer to stay on a safe position. I'm 99% sure that everything will work fine even if I will add some tables (system is not under heavy load), but, in case of problem, vendor will not provide any support.
Lazarus provide no direct connection to Informix DB, so I work via ODBC.
There is one more idea from Informix guru: set up DB replication (maybe partial, only the tables I need), then work with replica copy. Even if replica becomes broken, this should not affect the Call Center operation. Replication is not prohibited by vendor. But setting up Informix replication is a challenge for me...

Thaddy

  • Hero Member
  • *****
  • Posts: 16300
  • Censorship about opinions does not belong here.
Re: How to combine data from several DBs?
« Reply #8 on: Today at 02:30:11 pm »
There is one more idea from Informix guru: set up DB replication (maybe partial, only the tables I need), then work with replica copy. Even if replica becomes broken, this should not affect the Call Center operation. Replication is not prohibited by vendor. But setting up Informix replication is a challenge for me...
That sounds like the way to go, but note that replication also affects the speed of the main database a bit, not much, but can be noticable.
If I smell bad code it usually is bad code and that includes my own code.

 

TinyPortal © 2005-2018