Forums » Linux
"Backgrounded" app and "Screen Saver" ping issues.
I'm cross-posting some of my response to a recent ticket for general awareness. If you "background" the game on desktop Linux, or leave the game running and the screen-saver or sleep modes kick in, the client may become desynchronized from the server, and result in a bad "ping". This is not entirely "real", and will self-correct once you start actively using the game in the "foreground" again at full frame-rate, but it can take a few seconds to recover.
More information here:
The game is based around an internal high-precision synchronized clock (unrelated to your "time of day" system clock), using phase-locked-loops on all clients, that maintain synchronization with the server based on perceived small changes in round-trip-time of packets, assessed from timestamps in the game protocol. Most game clients operate within 1 millisecond of the server's game clock, even across pretty chaotic latencies (players on mobile phones in the jungles of the Philippines, or South Africa, or whatever).
Basically, what seems to happen here is that Linux forces the game down to 1fps whenever it is backgrounded, which is insufficient to maintain time synchronization. This pushes the "ping" completely out of whack, because ping is calculated based on clock differential, and the Linux backgrounding completely skews the clock.
Above and beyond this, Linux also seems to "buffer" incoming network data, in some background or low-power modes. This didn't used to happen, slow systems would simply drop packets resulting in retransmissions, which we could at least "see" from the server. But Linux will now buffer packets internally for sometimes upwards of 10 seconds, or even 30 seconds. Having a series of packets arrive "30 seconds late" causes very undesirable behaviour in a tightly-synchronized network game.
Lastly, Linux has no unified user-space API or central mechanism for an application to "know" when it has been backgrounded. Android, for instance, has an entire "application lifecycle" system, where an application is told when it has been backgrounded, or all kinds of other "states", and can make adjustments accordingly. On Android, we actually disable the renderer when the game is backgrounded, to preserve battery life and optimize energy efficiency. But, Linux doesn't offer any of this. There are some individual window managers that have certain types of features, but they're all different and inconsistent (like KDE vs Gnome), and we aren't sure how "stable" they are either, since no one is really using them. So, we can't actually "tell" if a screen saver has been engaged on a Linux desktop, for instance.
There are hardware processor "sleep states" that could be detected, at a really low level, but that starts to get into information considered "privileged" in some models, requiring elevated permissions that we don't want to require.
The "good news" is that once you return the game to focus, it will re-acquire synchronization pretty quickly, that's something that has improved a lot in the last two months. In most cases it takes under 15 seconds. There are tradeoffs to the way it re-synchronizes, because we can't actually tell the difference between people who have a "bad ping" for genuine network-latency reasons, and people who have a "bad ping" because their computer is forcing the clock to go totally haywire. Those two different issues have totally different needs for mitigation, and "network ping" is a far more frequent problem, across all platforms, where "weird linux clock BS" is.. well.. unique to Linux.
You shouldn't ever need to restart your game, you should be able to (worst-case) just "jump" to another sector and the issue should be resolved. Jumping forces a re-assessment of your current clock state, although there are multiple watchdogs that are doing this to various degrees, one of which is checking every 10 seconds (regardless of what you do).
I don't have an answer beyond that, though. We've already spent a lot of time going down this rabbit-hole, and because it's specific to "Linux devices that are sleeping", it's kind of a niche thing, and a bit of an "unsupported mess" to try and work around. So, for now, I would just suggest that Linux users plan to jump to another sector when they start playing after the game has been backgrounded, or give the game a little time to resynchronize.
More information here:
The game is based around an internal high-precision synchronized clock (unrelated to your "time of day" system clock), using phase-locked-loops on all clients, that maintain synchronization with the server based on perceived small changes in round-trip-time of packets, assessed from timestamps in the game protocol. Most game clients operate within 1 millisecond of the server's game clock, even across pretty chaotic latencies (players on mobile phones in the jungles of the Philippines, or South Africa, or whatever).
Basically, what seems to happen here is that Linux forces the game down to 1fps whenever it is backgrounded, which is insufficient to maintain time synchronization. This pushes the "ping" completely out of whack, because ping is calculated based on clock differential, and the Linux backgrounding completely skews the clock.
Above and beyond this, Linux also seems to "buffer" incoming network data, in some background or low-power modes. This didn't used to happen, slow systems would simply drop packets resulting in retransmissions, which we could at least "see" from the server. But Linux will now buffer packets internally for sometimes upwards of 10 seconds, or even 30 seconds. Having a series of packets arrive "30 seconds late" causes very undesirable behaviour in a tightly-synchronized network game.
Lastly, Linux has no unified user-space API or central mechanism for an application to "know" when it has been backgrounded. Android, for instance, has an entire "application lifecycle" system, where an application is told when it has been backgrounded, or all kinds of other "states", and can make adjustments accordingly. On Android, we actually disable the renderer when the game is backgrounded, to preserve battery life and optimize energy efficiency. But, Linux doesn't offer any of this. There are some individual window managers that have certain types of features, but they're all different and inconsistent (like KDE vs Gnome), and we aren't sure how "stable" they are either, since no one is really using them. So, we can't actually "tell" if a screen saver has been engaged on a Linux desktop, for instance.
There are hardware processor "sleep states" that could be detected, at a really low level, but that starts to get into information considered "privileged" in some models, requiring elevated permissions that we don't want to require.
The "good news" is that once you return the game to focus, it will re-acquire synchronization pretty quickly, that's something that has improved a lot in the last two months. In most cases it takes under 15 seconds. There are tradeoffs to the way it re-synchronizes, because we can't actually tell the difference between people who have a "bad ping" for genuine network-latency reasons, and people who have a "bad ping" because their computer is forcing the clock to go totally haywire. Those two different issues have totally different needs for mitigation, and "network ping" is a far more frequent problem, across all platforms, where "weird linux clock BS" is.. well.. unique to Linux.
You shouldn't ever need to restart your game, you should be able to (worst-case) just "jump" to another sector and the issue should be resolved. Jumping forces a re-assessment of your current clock state, although there are multiple watchdogs that are doing this to various degrees, one of which is checking every 10 seconds (regardless of what you do).
I don't have an answer beyond that, though. We've already spent a lot of time going down this rabbit-hole, and because it's specific to "Linux devices that are sleeping", it's kind of a niche thing, and a bit of an "unsupported mess" to try and work around. So, for now, I would just suggest that Linux users plan to jump to another sector when they start playing after the game has been backgrounded, or give the game a little time to resynchronize.
I had a lightbulb moment shortly after we discussed this. I used to have to use Gamescopes to get the game to cooperate with being able to alt+tab at all. Gamescopes essentially creates its own little "display" environment for a game to run in. I just tested alt+tabbing back and forth with the game run under `gamescope -f -W 3840 -H 2160 -- vendetta` and I'm not seeing any of the ping spikes I have had in the past due to the app "sleeping".
For anyone reading this and potentially experiencing this, I recommend trying to setup Gamescopes to run the game in. For instance: the -W and -H is the width and height of your resolution. -f is for running in a fullscreen mode. Most people play at 1080p, or 1440p. That'd be `gamescope -f -W 1920 -H 1080 -- vendetta`.
https://github.com/ValveSoftware/gamescope for more information.
For anyone reading this and potentially experiencing this, I recommend trying to setup Gamescopes to run the game in. For instance: the -W and -H is the width and height of your resolution. -f is for running in a fullscreen mode. Most people play at 1080p, or 1440p. That'd be `gamescope -f -W 1920 -H 1080 -- vendetta`.
https://github.com/ValveSoftware/gamescope for more information.
In regards to my above post, for GNOME environments, you can use the Unite extension to force removal of the titlebar when maximized, which will automatically take effect when using Gamescope in fullscreen. Extract the unite@hardpixel.eu folder to /home/USERNAME/.local/share/gnome-shell/extensions then reboot or restart gnome-shell.
https://github.com/hardpixel/unite-shell/releases
Update: This seems to negate the alt+tab sleep fix..... Use at your own risk.
https://github.com/hardpixel/unite-shell/releases
Update: This seems to negate the alt+tab sleep fix..... Use at your own risk.
Final update, this one seems to be working as expected.
gamescope -b -W 3840 -H 2160 -w 3840 -h 2160 -o 60 -r 60 -- vendetta
-W/-H is the gamescope defined resolution, -w/-h is the nested resolution, so what the game sees as it's available resolution. -o 60 is 60fps un-focused, -r 60 is 60hz refresh rate for the window. I'm not seeing any throttling in the background, even if I switch workspaces.
gamescope -b -W 3840 -H 2160 -w 3840 -h 2160 -o 60 -r 60 -- vendetta
-W/-H is the gamescope defined resolution, -w/-h is the nested resolution, so what the game sees as it's available resolution. -o 60 is 60fps un-focused, -r 60 is 60hz refresh rate for the window. I'm not seeing any throttling in the background, even if I switch workspaces.