Recent

Author Topic: tcp throttling  (Read 2105 times)

mercurhyo

  • Full Member
  • ***
  • Posts: 242
tcp throttling
« on: October 17, 2019, 04:34:37 pm »
I'm on my way to port this
https://doc.qt.io/archives/qq/qq17-ratecontrol.html
to free pascal.
my goal is to make a reusable component able to control data rate on the tcp sockets implemented inside both well known communication package 'LNET" and "SYNAPSE"
https://lnet.wordpress.com
and
http://www.ararat.cz/synapse/doku.php/

if someone has already make a 'TStreamThrottler' or 'TTcpThrottler' general purpose component, I'd like to read how before i start to transpose Qt sources to Free Pascal.
Any suggestion would be very welcome
Thank you
DEO MERCHVRIO - Linux, Win10pro - Ryzen9XT 24threads + Geforce Rtx 3080SUPRIM
god of financial gain, commerce, eloquence (and thus poetry), messages, communication (including divination), travelers, boundaries, luck, trickery and thieves; he also serves as the guide of souls to the underworld

mercurhyo

  • Full Member
  • ***
  • Posts: 242
Re: tcp throttling
« Reply #1 on: October 17, 2019, 04:39:23 pm »
N.B. I debugged Indy's throttling system, but it's also dummy throttler counting 1bit = 1millisecond  :P nowadays it has no more efficiency LOL!!! because of CPUs high speed and fiber transfers. Anyway i don't like Indy's gas factory  ;)
« Last Edit: October 17, 2019, 04:42:46 pm by mercurhyo »
DEO MERCHVRIO - Linux, Win10pro - Ryzen9XT 24threads + Geforce Rtx 3080SUPRIM
god of financial gain, commerce, eloquence (and thus poetry), messages, communication (including divination), travelers, boundaries, luck, trickery and thieves; he also serves as the guide of souls to the underworld

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: tcp throttling
« Reply #2 on: October 20, 2019, 06:45:21 pm »
One word: avoid.

I can't speak for Windows but Linux has its own Traffic Control system which goes hand-in-hand with process-specific queue disciplines.

Also I'm not sure the status of Lnet: I've added a couple of minor bits to it but it was put to me that it was there more to support FPC/Lazarus installation than as a serious set of APIs.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: tcp throttling
« Reply #3 on: October 20, 2019, 06:56:57 pm »
Well, Mark, some agreement here but:
If you need to throttle, better use a dedicated hardware solution. Do not burden your software with that.
That is also what happens in practice. And a box is cheaper than the software.....
Specialize a type, not a var.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: tcp throttling
« Reply #4 on: October 20, 2019, 07:09:33 pm »
You (or rather the OS) can usefully do it at the point where traffic originates, but you need to be able to see the raw packets coming back to the system so that you can slow down as soon as the far end starts indicating that packets have been dropped somewhere along the route (and that doesn't rely on ECN or anything similar). I'd rather not say much more because of a legal problem I've got.

In practical terms, the kernel on modern systems has plenty of capacity for high-rate data flow. As a particular example, a problematic Linux-based router used to accumulate firewall rules (stuff added when an interface came up that wasn't removed when it went down) but there was no significant performance hit until the table concerned had around 12,000 rules in it. And that was on a Pentium-II, with network cards that predated offload engines.
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: tcp throttling
« Reply #5 on: October 20, 2019, 07:12:30 pm »
Mark, if you need to scale beyond a single server the hardware solution is much better and cheaper (I know, been there done that) Do not! do that in code unless you are running a low traffic website.

It takes one month to write such software - if you are really, really good -, a month for testing and a 16 way intelligent switch is just less than 1200 dollar. (Oh my  :o It is Huawei .. :-X )

Apart from that there are multiple sources like Ericson and the likes.
« Last Edit: October 20, 2019, 07:20:59 pm by Thaddy »
Specialize a type, not a var.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: tcp throttling
« Reply #6 on: October 20, 2019, 07:20:17 pm »
Very true. But you can still gain a lot by not generating excessively-fast traffic in the first place.
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: tcp throttling
« Reply #7 on: October 20, 2019, 07:24:19 pm »
Very true. But you can still gain a lot by not generating excessively-fast traffic in the first place.
You usually do that before you offer any data to your connection....
Specialize a type, not a var.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: tcp throttling
« Reply #8 on: October 20, 2019, 08:04:41 pm »
i.e. At or near the application program. But in the general case the application program doesn't get an immediate indication of congestion, so if attempting to maximise throughput it's better left to the kernel with an appropriate (usually non-default) queuing discipline selected.
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

zamronypj

  • Full Member
  • ***
  • Posts: 133
    • Fano Framework, Free Pascal web application framework
Re: tcp throttling
« Reply #9 on: October 20, 2019, 10:31:34 pm »
Agree with above comments, network router hardware has a way to assign quality of service for network traffic. You can configure rate limit to any traffic including tcp.

While not directly related to tcp, on application level protocol, such http, many web application frameworks provides rate limiting request mechanism to control access to an API. For example paid user can have bigger quota than free user. Implementation is different but idea is roughly same.

For example you want to limit a user to access an API max 1000 requests/sec

- identify request (e.g, generate id based on session id)
- read storage for this request
-- if no data for this request, write request id, last accesed time and number of access to storage. Allow request.
-- if data available, compare current time with last accessed time.
---- if it is bigger then 1 second, reset number of access, set last accesed time to current time. Allow request.
---- if it is inside 1 second window, compare number of access with maximum allowable number, if less, inc number of access and allow request. If bigger, block request and send appropriate http header (e.g, 429 Too Many Requests)
« Last Edit: October 20, 2019, 10:33:53 pm by zamronypj »
Fano Framework, Free Pascal web application framework https://fanoframework.github.io
Apache module executes Pascal program like scripting language https://zamronypj.github.io/mod_pascal/
Github https://github.com/zamronypj

mercurhyo

  • Full Member
  • ***
  • Posts: 242
Re: tcp throttling
« Reply #10 on: October 21, 2019, 03:16:34 am »
One word: avoid.


MarkMLl

thanks for being able to be OFFtopic so long. I'm going to yawn forever on a fat pillow of confusions... wana talk around the last fever of my nephew???

the question is not if necessary to trottle or 'avoid' because people needing to trottle communications are ALREADY WELL DEFINED in the article. I gave the link at 1st

https://doc.qt.io/archives/qq/qq17-ratecontrol.html

"""" Who Needs Traffic Control Anyway?
Many of the applications we use every day implement some sort of TCP traffic control: games use traffic control for downloading patches, operating systems use it for downloading security updates, and in web browsers with transfer dialogs, traffic control is absolutely necessary to avoid network congestion.

For applications that download large datasets, limiting a connection's download rate is also very important. Without traffic control, your application might end up stealing most of the bandwidth, slowing down everyone else using the same network link. While fetching a huge email attachment, you could annoy coworkers waiting for an important web transaction to complete.......
""""

the question is "did someone make it with free pascal a cool way, before I translate to pascal the Qt C++ given solution?

I'm afraid that you are alembicing forever just like for "virtual machines" clarity, under 100 meters of fog, on the Qt vs Laz thread LOL
« Last Edit: October 21, 2019, 03:56:36 am by mercurhyo »
DEO MERCHVRIO - Linux, Win10pro - Ryzen9XT 24threads + Geforce Rtx 3080SUPRIM
god of financial gain, commerce, eloquence (and thus poetry), messages, communication (including divination), travelers, boundaries, luck, trickery and thieves; he also serves as the guide of souls to the underworld

mercurhyo

  • Full Member
  • ***
  • Posts: 242
Re: tcp throttling
« Reply #11 on: October 21, 2019, 03:20:25 am »
either i'm getting to old (53, retired for 3 years), eiter I found out a team of muppets show old men behind the backstage  8-) 8-) 8-)

[Applause]

relax please and keep it SIMPLE
« Last Edit: October 21, 2019, 03:33:41 am by mercurhyo »
DEO MERCHVRIO - Linux, Win10pro - Ryzen9XT 24threads + Geforce Rtx 3080SUPRIM
god of financial gain, commerce, eloquence (and thus poetry), messages, communication (including divination), travelers, boundaries, luck, trickery and thieves; he also serves as the guide of souls to the underworld

mercurhyo

  • Full Member
  • ***
  • Posts: 242
Re: tcp throttling
« Reply #12 on: October 21, 2019, 03:41:11 am »

It takes one month to write such software - if you are really, really good -, a month for testing and a 16 way intelligent switch is just less than 1200 dollar. (Oh my  :o It is Huawei .. :-X )


woohooh fellows, I promised you a FREE multiplaya game! uh! bad news, buy a tiny 16 ways cheap switch  :D :D :D :D :D :D :D :D :D

c'mon grumpers! it's only $1200, the game is FREE ! What? you play from Africa, no problem you can pay within 150 years my friend

say no to drugs, .... at least one time LOL  :P

YES: I'm bad! and I love it
« Last Edit: October 21, 2019, 03:54:12 am by mercurhyo »
DEO MERCHVRIO - Linux, Win10pro - Ryzen9XT 24threads + Geforce Rtx 3080SUPRIM
god of financial gain, commerce, eloquence (and thus poetry), messages, communication (including divination), travelers, boundaries, luck, trickery and thieves; he also serves as the guide of souls to the underworld

 

TinyPortal © 2005-2018