So I looked at a sample request with xdebug profiler in both 3.2 and 3.3 and the time spent in the framework was basically the same.
Today I updated the app to 3.3.14 and same thing happened on the server. So now we're wondering whether it can be related to OPcache or something. When updating the dependencies, we swtich the whole vendor folder, so all the 3rd party code is "new".
When I looked back at the January data (when we first noticed it), the ~50ms delay disappeared (abruptly) after a day or so (more like 18 hours).
So I guess you are safe guys
Sorry for bothering you, thanks for the opinions.