Recent

Author Topic: [SOLVED] HTTP GET + REQUEST BODY  (Read 3821 times)

Чебурашка

  • Hero Member
  • *****
  • Posts: 568
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
[SOLVED] HTTP GET + REQUEST BODY
« on: January 28, 2022, 03:15:42 pm »
Hello,
I ask for advice (question is at the end).

I have a table "TABLE" like this

TABLE
{
    id int primary key,
    revisionNo int,
    [other useful data for my application]
}


I have "TABLE" on one computer "MASTER".
I have "TABLE" on one computer "SLAVE".

The DB is SQLITE so I cannot do queries from on computer to the other (otherwise
there would be no problem in doing what I need).

"TABLE" on "MASTER" is subject to

1. Creation of new records (when a record is created revisionNo is set to 1)
2. Modification of existing record (when a record is modificed revisionNo is increased by 1)

"TABLE" on "SLAVE" must be periodcally aligned in order to be equal to "MASTER" (no
autonomous modifications on "SLAVE", only synchronization)

My idea was to do this using a WEBSERVICE like this (port 8100):

there is a GET method:

http://MASTER:8100/get_table_delta

The caller of this webservice should provide the following info in the
GET REQUEST BODY (I represent as a json even if my representation would be
different, in the following example I assume the "SLAVE" "TABLE" has following contents)

"SLAVE" "TABLE" contents
id, revisionNo
1,  1
2,  2
3,  1
[1M more records]

JSON sent by "SLAVE"
{
   "records" = [
   {
      "id": "1",
      "revisionNo": "1"
   },
   {
      "id": "2",
      "revisionNo": "2"
   },
   {
      "id": "3",
      "revisionNo": "2"
   },
   <1M more json objects>
   ]
}


when the webservice running on "MASTER" receives this json in GET BODY it checks in his "TABLE"
and finds that
      
"MASTER" "TABLE" contents
id, revisionNo
1,  2
2,  2
3,  1
[1M more records]
1000001,  1

with respect to the one in the "SLAVE"

- item 1 was modified
- item 1000001 was added

therefore, "MASTER" prepares a json (for example) like this

JSON sent back by "MASTER"
{
   "delta" = [
   {
      "id": "1",
      "revisionNo": "2",
      <all other data of record 1>
   },
   <1M more json objects>,
   {
      "id": "1000001",
      "revisionNo": "1"
      <all other data of record 1000001>
   }
   ]
}

Therefore sending only the things updated or new (here one could say that for item 1
we could send only what has actually changed, but in this case I want to make the
things simple and I resend everything).


And finally the question:

Is it ok and supported that I do a HTTP GET operation sending also a REQUEST BODY?
Is it possible to receive such a REQUEST BODY, parse it and use it as I described?

I read many things telling me that GET should be used only to get objects identified
only using the URI, and including the REQUEST BODY it against this recommendation.

So far I have been using fphttpapp on server side, and fphttpclient on client side...

If is not possible or recommended does anyone have some alternative advice?

THANK YOU!!!
« Last Edit: January 28, 2022, 05:12:43 pm by tt »
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: HTTP GET + REQUEST BODY
« Reply #1 on: January 28, 2022, 03:28:41 pm »
Hello,
I ask for advice (question is at the end).
...
If is not possible or recommended does anyone have some alternative advice?

Sure it's possible: like landing on the Moon is possible.

Don't waste your time and find a network-capable database server e.g. PostgreSQL.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Чебурашка

  • Hero Member
  • *****
  • Posts: 568
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: HTTP GET + REQUEST BODY
« Reply #2 on: January 28, 2022, 03:35:02 pm »
Don't waste your time and find a network-capable database server e.g. PostgreSQL.

Thank you for the answer, but there are other reasons that force me to use SQLITE.
The deploy target is not a conventional machine (computer, laptop, server).

Anyway, my question is also a more general question about HTTP GET with or without REQUEST BODY.
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: HTTP GET + REQUEST BODY
« Reply #3 on: January 28, 2022, 04:01:32 pm »
We use a timestamp for each entry in the database to determine changes, and download/upload only the changes neccessary, not all objects.

You should use HTTP.POST to upload, and HTTP.GET to download, using the timestamp filter.

Say you have table1 lastupdated, both in sqlite and in the server. Check against the lastupdated timestamp, upload only the newest edited or added items, that has lastupdated greater than or equal the lastupdated in the table. download the entries greater than or equal the local lastupdated. update the lastupdated to the actual date.

Чебурашка

  • Hero Member
  • *****
  • Posts: 568
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: HTTP GET + REQUEST BODY
« Reply #4 on: January 28, 2022, 04:17:28 pm »
Could you be more specific on POST+GET?
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: HTTP GET + REQUEST BODY
« Reply #5 on: January 28, 2022, 04:26:44 pm »
Post is used to send data to the server and get is used to retrieve data.

nummer8

  • Full Member
  • ***
  • Posts: 108
Re: HTTP GET + REQUEST BODY
« Reply #6 on: January 28, 2022, 04:39:52 pm »
REST API is the term for what you are trying to use.
Every method has its own purpose

GET for receiving data
POST for creating an new record
PUT for updating an existing record
DELETE for deleting a record.

Google is your friend here.



Чебурашка

  • Hero Member
  • *****
  • Posts: 568
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: HTTP GET + REQUEST BODY
« Reply #7 on: January 28, 2022, 04:40:56 pm »

The idea based on the timestamps is nice and senseful, but I often saw these things fail because in industrial field, machine clocks might fail (they can be reset, or wrongly modified) and since they are used to generate the timestamps, these are not reliable, this is why I have choose to use the revisionNo which is under my software exclusive control.
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

rvk

  • Hero Member
  • *****
  • Posts: 6109
Re: HTTP GET + REQUEST BODY
« Reply #8 on: January 28, 2022, 04:41:26 pm »
Quote
Post is used to send data to the server and get is used to retrieve data.
With POST you can also get data back (for example that response data after posting).

You CAN send a request body with a GET (it's allowed) but it doesn't get processed (so it's of no use).

BTW, I think it would be easier to just/only send data from MASTER to SLAVE. MASTER could have a revision_slave besides it's own revision. In that case it only needs to send every record where revision_slave<>revision. After that, and after the slave acknowledged it received and changed X records, you can do SET revision_slave=revision on the table and all would be correct. In that case the SLAVE wouldn't need to send it's millions of revisions to the MASTER to only receive the changes. Because that would be one of a hell of a POST  (big  :o) :P

(Ah, yes, @lainz's suggestion with a timestamp would also work nicely.)

In case something goes wrong the SLAVE could send all it's revisions back to the MASTER and in my example the MASTER could set the revision_slave record correctly so the sync would be ok again.
« Last Edit: January 28, 2022, 04:43:34 pm by rvk »

Чебурашка

  • Hero Member
  • *****
  • Posts: 568
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: HTTP GET + REQUEST BODY
« Reply #9 on: January 28, 2022, 04:43:25 pm »
With POST you can also get data back (for example that response data after posting).

Probably this is the solution I was looking for...

Thank you!
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

rvk

  • Hero Member
  • *****
  • Posts: 6109
Re: HTTP GET + REQUEST BODY
« Reply #10 on: January 28, 2022, 04:44:45 pm »
Did you look at how large the request body would get for 1.000.001 records?

Чебурашка

  • Hero Member
  • *****
  • Posts: 568
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: HTTP GET + REQUEST BODY
« Reply #11 on: January 28, 2022, 04:45:13 pm »
Google is your friend here.

Yes I know this things, but i did not understand how to apply to my case.
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

Чебурашка

  • Hero Member
  • *****
  • Posts: 568
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: HTTP GET + REQUEST BODY
« Reply #12 on: January 28, 2022, 04:49:50 pm »
Did you look at how large the request body would get for 1.000.001 records?

The number 1000001 was just to say something big, it is not going to be like that.

But since I have to make something reliable on big numbers, this is also a problem that I will have to see how to solve.

But the main problem to me was to get the answer you have supplied = I can get data back also for POST requests.

Thank you.
« Last Edit: January 28, 2022, 04:55:09 pm by tt »
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

PierceNg

  • Sr. Member
  • ****
  • Posts: 369
    • SamadhiWeb
Re: [SOLVED] HTTP GET + REQUEST BODY
« Reply #13 on: January 29, 2022, 02:16:44 am »
Some open source network-capable offshoots of SQLite which may suit your use case and could minimize custom programming on your part:

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: [SOLVED] HTTP GET + REQUEST BODY
« Reply #14 on: January 29, 2022, 08:18:31 am »
You forgot mormot... :)
Specialize a type, not a var.

 

TinyPortal © 2005-2018