Forums » Bugs

Server side lag?

Sep 19, 2015 bojansplash link
Several players (prom different parts of the globe), me included, noticed a pretty unusual increase in ping and load times today.

My usual 120 +/- 1 ms ping went to steady 175-180 +/- 3.5 ms and loading times after every jump have doubled.
Sep 19, 2015 incarnate link
Ok, but we don't have anything to do with your ping time. Like, the game going awry doesn't alter your ping (or never has, historically). That's just the internet. And since you're several thousand miles away, that's a lot of internet to traverse.

Stuff happens on the internet. Routes change. Much as it probably wasn't us (or even our ISP) who made your ping better recently (instead being a more local routing change), this is probably not related to us either.
Sep 19, 2015 yodaofborg link
I have to agree with Bojan for a change. My ping seems normal, and I do not usually experience any lag. My round-trip time to mirror.cle.vendetta-online.com seems normal; and sector load times are usually so fast that I hardly see the loading screen. Today I am experiencing long load times, and also strange invincible bots.

Sure, it could be outside the data centre that is the problem, but I have noticed some oddities too. Like Sedina B-8 being full of xang-xi bots, when a capship (player or NPC) warps in my system seems to freeze for a moment.

I didn't notice any of this behaviour even when upstream ISP's were having issues. It's like the congentco problem all over again.
Sep 19, 2015 incarnate link
So, I want to help, but you guys would make it a lot easier for us if you could stick to reporting the actual symptom, and not "what you think the problem is".

Because, by reporting the latter, you're correlating a lot of totally-irrelevant data and just confusing the issue.

For instance:

- Bojan opened by complaining in part about ping times. We have nothing to do with ping times. The internet causes ping times, and Bojan is far away. Basically whether it's the data center or not, doesn't matter, because it's out of our control. The game does not cause pings to change.

- Yoda has great latency to mirror.cle.vendetta-online.com, which is the webserver, hosted in Cleveland, and has nothing whatsoever to do with the game, hosted in Chicago. But, hey, great that you have a good ping to the webserver.

It would probably best to just report, without any theories, that:

- Sector load-times seem slower than usual.

- Bots seem more invulnerable than usual.

Because that seems to be what's bothering you the most?
Sep 19, 2015 incarnate link
Also, Bojan, just as a heads-up on the ping thing. Using your last-connected IP as a basis, and then tracerouting to find your immediate upstream node that responds directly to pings (you or your ISP seems to filter ICMP echo-response on your particular IP), I get:

New-server in Chicago: 155ms

Old-server in Milwaukee: 159ms

Generally, that's what's going to be the case. Because Milwaukee has to go to Chicago in order to reach the rest of the world (for the most part), and it usually takes another 4ms to ride the train tracks up to us. So, Chicago will almost always be faster, but the internet can still globally-suck, regardless. Our Milwaukee and Chicago locations have very similar connections, to the same tier-1 providers (actually, the new site in Chicago is much better, overall).

I have no idea why it's 155/159ms instead of 120/124 or whatever, I imagine there may be a trans-atlantic thing going on, or a peering point that's overloaded, or something. But the important thing to take away here is: it has nothing to do with Vendetta Online, in any way.

Also, because I'm seeing 155ms to an upstream point, and you're seeing 175ms downstream, is not noteworthy. There could easily be another 20ms of BS going in on a local municipal cloud that is trying to cope with routing problems and falling back to an over-saturated connection, or any number of other things. The point is.. it's still 155ms upstream of you, and in Milwaukee, it's 4ms worse. So this isn't something that has "changed". It would still be a problem if we hadn't moved the server, and it would still not be our fault.

Now, we're going to continue looking into this slow-load and general slow-performance issue on the server, as that could easily just be a CPU-heavy sector running on the same cluster-server as some other sectors, and making their performance suck or something. Which has nothing to do with ping times, since, as mentioned, the game does not impact ping times. FYI.
Sep 19, 2015 incarnate link
Do you guys have any specific examples of sectors that are slow to load? Like B8 was mentioned as having a lot of bots, but.. that doesn't mean it was slow to load.

Is it specific sectors, or everything?
Sep 19, 2015 yodaofborg link
I'll try to be more specific in future Incarnate, and keep a log of when I have issues including sector/time in UTC. I've been busy chasing and shooting people.
Sep 19, 2015 bojansplash link
Guess I have to live with the cards I have been dealt with (my ISP) and blame Murphy for my ping ups/downs. Thanks for the extra effort put in looking into this Inc.

About slow load... sometimes you have to look at this from the eyes of a player.
You are chasing someone and you jump and your sector loads very slow and you curse because your target is like 4000m away when you finally jump in.
It's hard to be level headed and keep track of when and where that exactly happened (it usually happens in latos/sedina/odia area).
I will try to be more specific next time but can't promise anything.
Sep 20, 2015 incarnate link
We understand the frustration. But this is an internet-based game, and the internet is flawed. So, while we want to address any persistent problems.. if alarm is raised every time someone experiences a little lag, we'll go crazy trying to chase down All Of The Nothing.

Similarly, the game is architecturally based on distributing live sectors within the VO server-cloud based on relative server load (across many individual servers). This means there is no "game-geography" tie between sectors and their server performance. If Sedina B8 is experiencing a slowdown, because the same cloud-server happens to be loaded at the time, and then that instance of Sedina B8 times out (30 seconds empty).. the next time someone flies into it, it will most likely be running on a totally different server and no longer have a related issue.

So, this is why we need to know exactly when, and where, you experienced a problem. We do also monitor things internally, but.. it's hard for us to know if we missed something, and the inherent variability of the internet often means people are reporting "server problems" that aren't actually server problems. We'll be far better equipped to figure out what's wrong with more information.

Anyway, thanks guys.