Level3 Communications Backbone Network Problems

For the last day and a half, the Level3 Communications network has been experiencing major problems in the Midwest area. Although our game is served from four different transit providers, of which Level3 is only one of them, this is still impacting a lot of players. They are a common transit path for many providers, as they've historically been one of the better Tier-1 ISPs.

Level3 is actively working on the issue, but we don't have an ETA at this time. Problems you experience in the game, related to various forms of "lag" are probably going to be related to this.

August Newsletter, Economy Changes, EverSpace Promo

We're pretty excited about some of the new changes coming to our dynamic economy in the near future, and we discuss that and a number of other topics in the latest Vendetta Online Newsletter:

One of the more unusual points to be found in there, is our support for a completely different, unrelated game, the upcoming single-player title EverSpace from RockFish Games. RockFish is a new studio, formed from the original team members who created the Galaxy On Fire series. They're good friends of ours, and we're doing what we can to help their Kickstarter succeed. The game itself looks beautiful, and is already to a playable prototype stage, but they're looking for funds to help complete it.

Thus, we've chosen to make a special offer available: If the EverSpace Kickstarter succeeds, we will give every backer a special Vulture ship-skin in Vendetta Online, and we will randomly choose one lucky backer to receive free a one-year Premium subscription to our game ($90 value)! Check out the above newsletter for details, and be sure to take a look at the EverSpace Kickstarter. If you're a Vendetta Online player, chipping in $5 to help them is probably worth it just for the content we're giving away! (We derive no financial benefit from this, we're just trying to help another indie game company get started).

We think their game looks amazing, and we hope it's something you like as well.

Dramatic Server Migration was Dramatic.

Vendetta Online is now up and running on a completely different server cluster, in a new location. This is actually pretty exciting, and more information about this will be coming in the next Newsletter, due out shortly.

Unfortunately, the process of migrating the server was not without challenges, as shown by the extended downtime over the weekend. We apologize for any inconvenience this caused. The bug was a pretty deep and low-level mix of different things, which all had to randomly happen at the same time in a particular way to create a problem. Unfortunately, it only ever showed up under production usage. No amount of testing ever demonstrated the problem, as some problems actually require a full load before they start to appear (real people behave differently from simulated loads).

Anyway, we tracked down and fixed the bug as quickly as possible, and the game should now be stable again. Please do continue to open Support Tickets and post on the Forums if you should happen to see some sort of major issue.

This new server cluster should open up a lot of gameplay possibilities for the expanded universe, and increased capacity, like the ability to sustain loads from new iPhone and Steam users.

But, for the moment, we're still evaluating things. Thanks for your patience during the transition.

Vendetta Online 1.8.347

VO 1.8.347 includes:

- Updater and game client now use a threaded getaddrinfo to look up the server's IP address; if DNS fails to resolve within 10 seconds, it will fall back to a visible error, instead of using OS timeouts.
- Updater now directly uses the patch cloud hostname, instead of redirecting through
- Hive Stronghold sectors idle timeout is set to 5 minutes, a more reasonable length of time to return and pick up Leviathan drops.
- OpenGL driver restores the window title if it has to be recreated for video option changes.

After the migration last Friday, we hit a bug that only impacted new Android users, which took some time to fully debug, although we had mitigated it on the server-side by Saturday night. A lot of the work this week was to make the updater and installer that much more robust, to prevent this kind of thing from happening again.

Plus, we had been planning to add some of these features for some time. For one thing, it brings VO a little closer to native IPV6 compatibility (although we aren't there yet), and also allows the game to react better to failure cases (with a proper error message), like people who have a misconfigured nameserver, or those experiencing an outage.

The bug, and subsequent fixes, soaked up much of the time intended for the server migration this week. So, that has been tentatively pushed to next week.

Thanks, and have a great weekend, everyone!
05 now More Secure and SPDY!

As of last week, the website is now running from a totally new and updated webserver, with significantly more bandwidth, sixteen times as much ram as the old machine, and four times as many cores.

We used to only encrypt website traffic when you were really using a "secure" area, like viewing important account information. Now, however, you may notice the browser's Lock icon appears constantly: The entire website is 100% encrypted at all times, and non-encrypted traffic is no longer supported. We're also offering some of the more recent cryptographic enhancements, like the Google-supported ChaCha20/Poly1305 cipher, for those on Chrome browsers, as well as protocol enhancements like SPDY, to make the site load considerably faster. Towards the end of the year, we expect to also support the next-generation web protocol HTTP/2 as well. We'll be continuing to tinker with the site over the coming months, as things stabilize, further extending our support for HTTP Strict Transport Security (also pre-loading the site into browsers), and adding HTTP Public Key Pinning.

The upshot of which is: The site should now be more robust, more secure, and faster than ever. It is still built with our same degree of care and heightened security-paranoia, so we hope it will be a great home for a long time to come. The internet has taken a lot of security hits over the last couple of years, from major SSL exploits like Heartbleed, to revelations about eavesdropping, and now the recent Android Stagefright bug. The best anyone in our position can really do, is build the best infrastructure we can imagine, and keep a watchful eye on the horizon. That's part of what this new site and system is about: a new setup to help mitigate potential unforeseen attacks, while also letting us react more quickly to news of brand-new bugs or issues.

Migrating this site took quite a lot of time, as it's a pretty complex endeavor, but we're very happy to have it out of the way. We're still going to be doing more server-migration work over the coming weeks, this time working on the game-server itself, to make for a faster and more scalable game environment. These changes haven't happened yet (we'll post again when they're in place), but we're hoping for faster sector load-times, faster player inventory access, and potentially opening up the ability to experiment with much larger space battles and other content.

We've also had some bugs appear lately in the game-update software, which we want to patch and improve as soon as possible. None of this is really the most exciting stuff to develop, we would rather be working on gameplay; but getting the infrastructural issues out of the way will give us a much better foundation on which to launch Vendetta Online 1.9.. and beyond!

Vendetta Online 1.8.346

VO 1.8.346 includes:

- Changed the 'Route' button in the Mission Info to 'Dock' and displays text saying to dock to the local station to accept the mission when clicked.
- The "PLAYER_DIED" lua event now triggers for anyone who dies within your sensor range (15km). Chat messages are only displayed if you are the victim or the killer.
- After un-docking from your own persistent capship, your capship now retains current velocity and slows to a stop, instead of stopping immediately.
- Jumping into a sector and landing directly into a capship docking bay no longer auto-docks your ship.
- Fixed typo in Training Neutron Blaster description.
- Also, server-side changes from this week: new production database system, new game webserver, etc.

Lots of infrastructural changes took place this week. We're still kind of "settling in" with the new servers and database systems, so please continue to let us know if anything isn't working properly. We'll have more announcements about all of this next week. Have a great weekend, everyone!

Scheduled Maintenance Window

Vendetta Online will be undergoing planned server-side maintenance tonight, July 29th 2015,between 11:59pm and 3am, US Central Time.
Maintenance will impact both the game and the website forums, and downtime could be as long as an hour.
Thank you for your patience while we're working to improve the game.

Scheduled Maintenance Window

Vendetta Online will be undergoing planned server-side maintenance tonight, July 28th 2015,between 11:30pm and 2am, US Central Time.
Maintenance will impact both the game and the website forums, and downtime could be as long as an hour.
Thank you for your patience while we're working to improve the game.

On-going server development.

We've had a long week of very carefully moving database tables from one place to another, and then fixing all the related code. This is pretty delicate, time-consuming and tedious work, but it's critical to the upcoming "server migration" that has been referenced in the last couple of update notices.

For many years, we've run almost entirely out of a single datacenter, operating from hardware we own. But times have changed, we need to be able to scale in the face of possible large player counts from feature-promotions on iPhone, Android, Steam, and the like. "Cloud" computing services allow a degree of elasticity that will strongly benefit the game, letting us spin up a large amount of hardware when briefly needed, and then turn it back off again when the demand goes down.

For instance, this enables options like firing up a huge dedicated server to handle a massive sector battle-event for a few hours, and then shutting it down when the event wraps. The cost of creating that "new" server for a couple of hours would only be a few cents, compared to the big cost of our dropping several grand on a big server (and needing to host it, maintain it, power it, cool it, and having it sit idle when not-needed). By the same token, this same scalable infrastructure can potentially help us absorb hundreds of thousands of new iPhone players for a week, or other difficult-to-predict spikes in player concurrency.

The game architecture itself was always built around this model, even going back to the 90s, as we always wanted to be able to scale the game very quickly. So that part is pretty easy. Some players may even recall that we ran the whole game off of a cloud provider for a week, as a testbed experiment, before pulling it back to our own (less costly) hardware.

But, even with that, there's still some legacy development-baggage from having all your hardware in one place. Right now we're primarily focusing on changing the way the website and the game will communicate, to allow us greater freedom to host the game itself (or parts of it) anywhere, while the website and secure billing data will remain on physical hardware we actually own. This means the connections and services must be robust to potential disconnects from fiber cuts, routing flaps and other unexpected internet outages.

In the meantime, of course, we're also doing a lot of administrative "cleanup", updating and further hardening the security of various systems, while making others more flexible and robust. We originally budgeted a week for this process, but we keep discovering legacy stuff we need to take into consideration with the new design. It's amazing the amount of code cruft and legacy services you end up with, after 13+ years of running an evolving, production MMORPG.

Given that context, a couple of weeks of cleanup is probably not surprising? Still, we're hoping this will mostly wrap up next week, in terms of what is soaking up most of our development time.

After that, we'll be back to patches and release updates, and in a much better place to continue the push for VO 1.9 and beyond!

Vendetta Online 1.8.345

VO 1.8.345 includes:

- Removed older, deprecated method of creating account from within the game client.
- Other background changes to make possible an upcoming server migration.

Not the most exciting update, but we've been really busy with a couple of tasks. One is that we're trying to address a number of recently-reported problems with the MacOS X version of the game, when using the new GL4 renderer. This is proving tricky and time-consuming to debug, as we're finding some surprising and unpredictable OS behaviour.

To our non-Mac users: please have patience. The lack of content updates in the meantime is not that fun, but we really need to squash these kinds of major bugs (across all platforms and versions) prior to the 1.9 release.

Additionally, as mentioned in the release notes above, we're also working our way up to a major server migration. We hope for part of that to wrap next week, which will mark a major improvement in our infrastructure. There'll be an announcement when it goes live, but it's one of those things where.. if all goes well, you won't be able to tell anything changed at all. Nonetheless, take our word for it that this is much-needed (a bit overdue, even), and we'll write more about some of the specific improvements after it goes live.

Have a great weekend, everyone!

Vendetta Online 1.8.343-344

VO 1.8.343-344 include:

- Increased support for anti-aliasing with certain Intel GPUs, and perhaps others.
- Mitigated issue with Mac OS X mouse cursor when cmd-tabbing in fullscreen.
- Windows version now built with a better optimizing compiler, some performance improvements should be evident.
- Fixed intermittent sector crashes.
- Fixed a rare potential issue with /vote mute.
- Fixed issue with joystick input on the Windows version.
- NPCs now require more collision damage to be applied to them before they will consider the attacker a threat.
- New station designs for Sol II H-13.
- Added anti-aliasing and gamma correction to the Mac OpenGL 4 driver. The Linux version already has anti-aliasing.
- Added ability to manually set the screen rotation on Android. The option is in Advanced -> Controls -> Input and has Landscape, Reverse Landscape and Sensor Landscape as the three options.
- NPC capships now target and attack PC capships with their primary weapons.
- Optimizations to multi-threaded rendering.
- Multi-threaded rendering is now default to enabled on DirectX 11 and OpenGL 4 versions. When switching to those drivers, a restart of the client is required to enable multi-threaded rendering. The setting can be turned off by editing the config.ini file and adding multithreaded=0 in the [Vendetta] section.

A variety of infrastructural improvements, in the lead-up to VO 1.9. We also have quite a bit of server-side work going on in the background as well, along with some interesting new gameplay scenery and content. Next week we may start to reveal some of these enhancements, as well as prepare for some significant server migrations, to increase game capacity and performance. Have a great weekend!

Summer Sale, Newsletter, Upcoming 1.9 Release.

We're rapidly approaching the release of Vendetta Online 1.9! This major milestone is a cause for celebration, so we've started our first Summer Sale, with a 33% discount on 1-year Premium Subscriptions in the form of purchasable Keys (which may be applied to any account, or saved indefinitely, or given to friends, etc). Only a limited number of discounted keys will be available, so get them while you can. For more information, check out the recent newsletter:

Vendetta Online 1.8.342

VO 1.8.342 includes:

- New station designs for Itan J11 and Odia M14.
- Six missions updated in the LAR Project tree, substantially improving issues with the Stygian Furies.
- The mission "Rush job! Ores and minerals needed" may now be re-taken (Xang Xi Research locations, Respect standing) to acquire the Xang-Xi Self Propelled Concussion Mine Launcher. Mission is available once per week, per character.
- Concussion Railgun grid power requirements dropped from 12 to 9, allowing dual-usage on the Raptor UDV, etc.
- Rendering optimizations for the new DX11 and GL4 engines. On some machines, that are more CPU-limited, these updates can make a big speed difference. More related improvements next week.

Have a great weekend, everyone!

Vendetta Online 1.8.341

VO 1.8.341 includes:

- Android version now supports both landscape modes. Requires Play Store App Update or sideload update. The Amazon Store version will come out at a later date.
- Added Video driver options to change the number of dynamic lights and toggle shadows in the DirectX 11 and OpenGL 4 drivers.
- Fixed a crash when switching from OpenGL 4 to the OpenGL Reference driver when multithreaded rendering is enabled and the glow effect is on.
- Border Skirmish Teradon firing distance has been lengthened and the average combat distance is shortened by 100 meters.
- Please rate the Android version! The Android version gives the choice to review the app when docking to a non-training station after 10 minutes of being online or after an in-app purchase. You can choose to have it ask again later (one week later) or never.

We're nearing the DX11 and GL4 drivers becoming the production standard, so please continue to help us test them, and report any issues on the Bugs forum, or in a Support Ticket if privacy is needed.

The new rendering-engine becoming "production-ready", as well as some other emerging features, will help mark a major update planned for the near future.

Vendetta Online 1.8.340

VO 1.8.340 includes:

- New station design for Dau L10, more carefully integrating the Capship Hangar components. In the future, there will be more fully-customized stations replacing those with recently added capship hangars.
- Experimental: Fog and radar limitations now impact bot's targeting visibility and aggression. However, this may have undesirable secondary effects, and may require on-going tweaks and changes. Strikeforce NPCs have integrated radar extenders.
- Maximum rent-able cargo capacity of new capship-capable stations now increased to 100,000 cu. Existing cost thresholds for extended cargo space still apply.
- When a Trident pilot switches to a turret view, the local (Trident) ship is now properly visible.
- When launching from a Trident while it is mining, the ore is now properly placed into the Trident's cargo hold.
- Fixed a bug where a persistent Trident may collide with the player's copy during the change-over.

Have a great weekend, please provide feedback on any of these changes, especially if you run across undesirable ramifications (such as the fog/radar situation). Bugs is a great place to report, or open a Support Ticket if something is critical. Thanks much!