Lazarus
Programming => Networking and Web Programming => Topic started by: Чебурашка on January 28, 2022, 03:15:42 pm
-
Hello,
I ask for advice (question is at the end).
I have a table "TABLE" like this
TABLE
{
id int primary key,
revisionNo int,
[other useful data for my application]
}
I have "TABLE" on one computer "MASTER".
I have "TABLE" on one computer "SLAVE".
The DB is SQLITE so I cannot do queries from on computer to the other (otherwise
there would be no problem in doing what I need).
"TABLE" on "MASTER" is subject to
1. Creation of new records (when a record is created revisionNo is set to 1)
2. Modification of existing record (when a record is modificed revisionNo is increased by 1)
"TABLE" on "SLAVE" must be periodcally aligned in order to be equal to "MASTER" (no
autonomous modifications on "SLAVE", only synchronization)
My idea was to do this using a WEBSERVICE like this (port 8100):
there is a GET method:
http://MASTER:8100/get_table_delta
The caller of this webservice should provide the following info in the
GET REQUEST BODY (I represent as a json even if my representation would be
different, in the following example I assume the "SLAVE" "TABLE" has following contents)
"SLAVE" "TABLE" contents
id, revisionNo
1, 1
2, 2
3, 1
[1M more records]
JSON sent by "SLAVE"
{
"records" = [
{
"id": "1",
"revisionNo": "1"
},
{
"id": "2",
"revisionNo": "2"
},
{
"id": "3",
"revisionNo": "2"
},
<1M more json objects>
]
}
when the webservice running on "MASTER" receives this json in GET BODY it checks in his "TABLE"
and finds that
"MASTER" "TABLE" contents
id, revisionNo
1, 2
2, 2
3, 1
[1M more records]
1000001, 1
with respect to the one in the "SLAVE"
- item 1 was modified
- item 1000001 was added
therefore, "MASTER" prepares a json (for example) like this
JSON sent back by "MASTER"
{
"delta" = [
{
"id": "1",
"revisionNo": "2",
<all other data of record 1>
},
<1M more json objects>,
{
"id": "1000001",
"revisionNo": "1"
<all other data of record 1000001>
}
]
}
Therefore sending only the things updated or new (here one could say that for item 1
we could send only what has actually changed, but in this case I want to make the
things simple and I resend everything).
And finally the question:
Is it ok and supported that I do a HTTP GET operation sending also a REQUEST BODY?
Is it possible to receive such a REQUEST BODY, parse it and use it as I described?
I read many things telling me that GET should be used only to get objects identified
only using the URI, and including the REQUEST BODY it against this recommendation.
So far I have been using fphttpapp on server side, and fphttpclient on client side...
If is not possible or recommended does anyone have some alternative advice?
THANK YOU!!!
-
Hello,
I ask for advice (question is at the end).
...
If is not possible or recommended does anyone have some alternative advice?
Sure it's possible: like landing on the Moon is possible.
Don't waste your time and find a network-capable database server e.g. PostgreSQL.
MarkMLl
-
Don't waste your time and find a network-capable database server e.g. PostgreSQL.
Thank you for the answer, but there are other reasons that force me to use SQLITE.
The deploy target is not a conventional machine (computer, laptop, server).
Anyway, my question is also a more general question about HTTP GET with or without REQUEST BODY.
-
We use a timestamp for each entry in the database to determine changes, and download/upload only the changes neccessary, not all objects.
You should use HTTP.POST to upload, and HTTP.GET to download, using the timestamp filter.
Say you have table1 lastupdated, both in sqlite and in the server. Check against the lastupdated timestamp, upload only the newest edited or added items, that has lastupdated greater than or equal the lastupdated in the table. download the entries greater than or equal the local lastupdated. update the lastupdated to the actual date.
-
Could you be more specific on POST+GET?
-
Post is used to send data to the server and get is used to retrieve data.
-
REST API is the term for what you are trying to use.
Every method has its own purpose
GET for receiving data
POST for creating an new record
PUT for updating an existing record
DELETE for deleting a record.
Google is your friend here.
-
The idea based on the timestamps is nice and senseful, but I often saw these things fail because in industrial field, machine clocks might fail (they can be reset, or wrongly modified) and since they are used to generate the timestamps, these are not reliable, this is why I have choose to use the revisionNo which is under my software exclusive control.
-
Post is used to send data to the server and get is used to retrieve data.
With POST you can also get data back (for example that response data after posting).
You CAN send a request body with a GET (it's allowed) but it doesn't get processed (so it's of no use).
BTW, I think it would be easier to just/only send data from MASTER to SLAVE. MASTER could have a revision_slave besides it's own revision. In that case it only needs to send every record where revision_slave<>revision. After that, and after the slave acknowledged it received and changed X records, you can do SET revision_slave=revision on the table and all would be correct. In that case the SLAVE wouldn't need to send it's millions of revisions to the MASTER to only receive the changes. Because that would be one of a hell of a POST (big :o) :P
(Ah, yes, @lainz's suggestion with a timestamp would also work nicely.)
In case something goes wrong the SLAVE could send all it's revisions back to the MASTER and in my example the MASTER could set the revision_slave record correctly so the sync would be ok again.
-
With POST you can also get data back (for example that response data after posting).
Probably this is the solution I was looking for...
Thank you!
-
Did you look at how large the request body would get for 1.000.001 records?
-
Google is your friend here.
Yes I know this things, but i did not understand how to apply to my case.
-
Did you look at how large the request body would get for 1.000.001 records?
The number 1000001 was just to say something big, it is not going to be like that.
But since I have to make something reliable on big numbers, this is also a problem that I will have to see how to solve.
But the main problem to me was to get the answer you have supplied = I can get data back also for POST requests.
Thank you.
-
Some open source network-capable offshoots of SQLite which may suit your use case and could minimize custom programming on your part:
- https://github.com/Expensify/Bedrock (https://github.com/Expensify/Bedrock) - networking and distributed transaction layer
- https://github.com/canonical/dqlite (https://github.com/canonical/dqlite) - high availability and automatic failover
- https://github.com/benbjohnson/litestream (https://github.com/benbjohnson/litestream) - streaming replication
- https://github.com/rqlite/rqlite (https://github.com/rqlite/rqlite) - fault tolerance and high availability
-
You forgot mormot... :)
-
And finally the question:
Is it ok and supported that I do a HTTP GET operation sending also a REQUEST BODY?
Is it possible to receive such a REQUEST BODY, parse it and use it as I described?
I read many things telling me that GET should be used only to get objects identified
only using the URI, and including the REQUEST BODY it against this recommendation.
So far I have been using fphttpapp on server side, and fphttpclient on client side...
If is not possible or recommended does anyone have some alternative advice?
THANK YOU!!!
To giva a real answer to these questions, the HTTP/1.1 Specification RFC7231 states in Section 4.3.1 (https://datatracker.ietf.org/doc/html/rfc7231#section-4.3.1)
A payload within a GET request message has no defined semantics;
sending a payload body on a GET request might cause some existing
implementations to reject the request.
It is so called undefined behavior. Implementing it is not wrong, but you can not expect the other side to also implement it. In fact many Client and Server libraries do not implement this. From personal experience I can say that the the HTTP Server and HTTP Client provided by Java standard library for example does not provide this functionality.
Long story short you should not use this.
That said, you can still transfer information via an HTTP GET request, as part of the URI, either as query (the stuff in a URI after the ?), or as Fragment (the stuff in the URI after the #). And it is very common to sometimes transmit even largeer amounts of data via get, e.g. in SAML or OAuth the whole request is encoded as GET request, which can get up to a few Kilobytes in size.
The only problem here is that some server or browser implementations are not capable of handling such large URIs. While the HTTP Standard in RFC 7230 Section 3.1.1 (https://www.rfc-editor.org/rfc/rfc7230#section-3.1.1) recommends that all implementations should support at least 8kbytes, to support older browsers, it is usually recommended to not exceed 2 kbytes.
So while the general rule of thumb for requests is to use GET for receiving data and POST for sending data (as the names where chosen for this), you can also use GET to transmit data, and there are some advantages to that, mainly that you can encode and alter the data within 403 redirects, something that is not allowed for POST data (which is either lost on 301, 302 or 303, or the data must be sent unchanged as per 307 and 308), which is why it is for example used for OAuth or SAML as here redirects are often common.
Long story short, you can send data via GET, and it is sometimes useful, but you should not send it as body bot encode it either as Query and/or Fragment
-
Some open source network-capable offshoots of SQLite which may suit your use case and could minimize custom programming on your part:
- https://github.com/Expensify/Bedrock (https://github.com/Expensify/Bedrock) - networking and distributed transaction layer
- https://github.com/canonical/dqlite (https://github.com/canonical/dqlite) - high availability and automatic failover
- https://github.com/benbjohnson/litestream (https://github.com/benbjohnson/litestream) - streaming replication
- https://github.com/rqlite/rqlite (https://github.com/rqlite/rqlite) - fault tolerance and high availability
Thank you very much for these suggestions.
-
And finally the question:
Is it ok and supported that I do a HTTP GET operation sending also a REQUEST BODY?
Is it possible to receive such a REQUEST BODY, parse it and use it as I described?
I read many things telling me that GET should be used only to get objects identified
only using the URI, and including the REQUEST BODY it against this recommendation.
So far I have been using fphttpapp on server side, and fphttpclient on client side...
If is not possible or recommended does anyone have some alternative advice?
THANK YOU!!!
To giva a real answer to these questions, the HTTP/1.1 Specification RFC7231 states in Section 4.3.1 (https://datatracker.ietf.org/doc/html/rfc7231#section-4.3.1)
A payload within a GET request message has no defined semantics;
sending a payload body on a GET request might cause some existing
implementations to reject the request.
It is so called undefined behavior. Implementing it is not wrong, but you can not expect the other side to also implement it. In fact many Client and Server libraries do not implement this. From personal experience I can say that the the HTTP Server and HTTP Client provided by Java standard library for example does not provide this functionality.
Long story short you should not use this.
That said, you can still transfer information via an HTTP GET request, as part of the URI, either as query (the stuff in a URI after the ?), or as Fragment (the stuff in the URI after the #). And it is very common to sometimes transmit even largeer amounts of data via get, e.g. in SAML or OAuth the whole request is encoded as GET request, which can get up to a few Kilobytes in size.
The only problem here is that some server or browser implementations are not capable of handling such large URIs. While the HTTP Standard in RFC 7230 Section 3.1.1 (https://www.rfc-editor.org/rfc/rfc7230#section-3.1.1) recommends that all implementations should support at least 8kbytes, to support older browsers, it is usually recommended to not exceed 2 kbytes.
So while the general rule of thumb for requests is to use GET for receiving data and POST for sending data (as the names where chosen for this), you can also use GET to transmit data, and there are some advantages to that, mainly that you can encode and alter the data within 403 redirects, something that is not allowed for POST data (which is either lost on 301, 302 or 303, or the data must be sent unchanged as per 307 and 308), which is why it is for example used for OAuth or SAML as here redirects are often common.
Long story short, you can send data via GET, and it is sometimes useful, but you should not send it as body bot encode it either as Query and/or Fragment
Thank you very very very much for this detailed explanation.