Forums » General

Server Outage

Mar 04, 2016 incarnate link
We're currently dealing with another server problem. Hopefully relatively brief, but we're going to schedule another restart soon.
Mar 04, 2016 incarnate link
So, yeah.. it has not been very brief. Completely out of our control at this time. We'll bring things back up as soon as we can. Sorry for any inconvenience.
Mar 04, 2016 EmikeT link
Ok I promise to tidy up my inventory now may i come back in?
Mar 04, 2016 Piment link
Why not make a timeout for AFK players who keep sectors online for no bloody reason?

Timeout for being inside a station for too long would be about 2 hours, and outside would be about 2-3?

Just a thought to avoid events like this(assuming that AFK people who keep sectors online for no reason is bad for the server?)
Mar 04, 2016 incarnate link
AFK players are an annoyance for the server.

This is a catastrophic server failure. Not the same thing. The one does not cause the other.

Our provider experienced a major SAN failure, which took down their entire enterprise cloud, far larger than just us.

We're on our way to getting back online now, the machines are back at least. We just need to clean up and check the databases for consistency to bring the game back up cleanly.
Mar 04, 2016 suppush link
I lost 5 mil with the restart (yea it hasnt happened yet..) just saying *grins*
Mar 04, 2016 davejohn link
Well, stuff happens, thanks for sorting it on a friday night devs.

But a general point, vo describes itself as a Massively multiplayer online role-playing game. I spend a fair bit of time flying about, I know all the shops and static spotting bots from assorted guilds by sight, and of course being TGFT by list. About 30 of them in total.

I know its tough running any sort of business these days, but even with the best of intentions 30 AFK static bots cannot be described as a massive number. Imagine thousands of players online at the same time ( which vo hopes to have ) , thats hundreds of folk wandering off in RL at any one time. When they wander off they will be afk, many for long periods.

It has been suggested that you boot AFK players for inactivity, but if numbers of players increase relogging would be a server load.

So, make the stations a separate sector within a sector. A parking sector, since that is where AFK players will be. That way they are not affecting the sector as a whole and are treated as a subset of vo, in the same way that HS or BS missions with no players are a background process that happens when not much else is happening.

Just a thought.
Mar 04, 2016 Dr. Lecter link
So, make the stations a separate sector within a sector.

A side-benefit of which would be inability of station-parked players to observe anything occurring outside the station. Great idea, Ecka!
Mar 04, 2016 davejohn link
But an ec 89 at 20 k visual range could. So paid account spotters would still be effective, but genuine AFK folk wouldn't affect the server.

Anyway, spotting lawyers would be easy, especially if vc is extended ...
Mar 04, 2016 incarnate link
What?

AFK players are not now, nor have they ever been, a significant problem for the game. Nor are "logins".

Thanks for the suggestion, I think? But your suggestion is predicated on a bunch of assumptions, most of which are wrong, making it difficult for me to formulate a response.

And.. even if I could, this really isn't the place to discuss the topic, as it has nothing to do with today's outage, at all, and really shouldn't be posted to an outage thread.

To be clear, AFK players and bots had nothing to do with today's issue. I wrote a newspost about it here an hour ago.
Mar 04, 2016 Kierky link
Ecka's just had way too much to drink.
Mar 05, 2016 Conflict Diamond link
/me pours a not-so-wee dram and tries to catch up with Ecka. Slainte!
Mar 05, 2016 roguelazer link
So what are you going to do to prevent future recurrences? Lean on the vendor to fix their SAN? Change vendors? Go back to instance-local storage?
Mar 05, 2016 incarnate link
So what are you going to do to prevent future recurrences? Lean on the vendor to fix their SAN? Change vendors? Go back to instance-local storage?

Our provider is changing SAN vendors, and keeping me posted on that migration process. This is a pretty massive private cloud, and they have excellent infrastructure otherwise. Stuff can happen to anyone.. I've built ISPs, I'm familiar with that. It's more about how you deal with issues that arise, and they are dealing with it.

Cloud usage is realistically a fact of life now, and all cloud providers have non-local storage, be it Amazon, Google, Azure, or anyone else. I can't really rule out the possibility of this kind of issue, while using the cloud.

We may eventually have a mixed setup of our own hardware, along with additional capacity expansion into the cloud. But, I can't really afford to do that right now.