Time for a wrap-up, what happened:
There were happening 2 things simultaneously, an increased interest(?) from china and one or two bots looking for SQL injections every second.
This caused a load on the original server configuration, apache using mod_php.
Since the amount of traffic was larger than normal, but still realistic, I decided to use fast-cgi to remove the php overhead from the apache process. The same time I could now use the mpm_event module which can handle way more connections to the server.
After some days of config tweaking everything should be fine..... not.
The server started to work OK until, according to netstat, about 800 sockets were in use after which the server failed to respond. This continued till the amount of sockets dropped to about 50 (due to timeouts etc). Then the server was responsive again and the whole cycle started again. After a lot of googling it appeared that this was caused by this issue in apache
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555Since we're still running 16.04 LTS, our apache was one minor release to old. So I manually upgraded apache to the latest version and the problem was gone