About The Future

I've never actually written one of these. I've been super critical in the past of these kind of posts. What kind of skilled Administrator loses a database out of nowhere? You think to yourself, "ha, that'd never happen to me, I have a good backup system". You become complacent in your past accomplishments, and don't test backups maybe as well as you should.

I would like to be clear, things went wrong. There was a series of unforseen events, but I am personally liable, and could of prevented this. I grew complacent, and fully own that. I have broken the trust held in me, and will do my best to regain it. I fully sympathize with those that have lost items unexpectedly. That said, going back to the last working copy of eight months, is completely unacceptable as well, and we will be focusing on the future.

Tl;dr: Unexpected errors in redis reading/writing, led to incorrect backups, and a loss of eight months of data.

In the rest of this post, I'll expand a bit on how the future will look. We had been considering and working on large changes for awhile, so this is not something we had to scramble to put together, rather than had to accelerate.

We as a team remain super grateful to the community, and are super excited to make the best of this bad situation.

We have made significant progress on our plan to finish the client, implement core changes to the gameplay loop, stockpile content for future updates, prepare new launch content, and execute our relaunch.

My next post will be a roadmap of the update and relaunch.

What Happened

Between 3-4 AM EST June 10th, we were alerted to a failure with the host, that resulted in a hard reset. This is normally fine, and happens very rarely.

I got up several hours later to find the game server offline, which was quite odd on it's own, as it should automatically go back up, and retry over several hours.

I put it back online, and a few minutes later was notified that accounts were out of date. I immediately went to the backups, and pulled the one from the evening prior, and restored it. The server went back online, and a few minutes later, got pinged that it was from eight months ago. This was super confusing, so I immediately grabbed a backup and restored it.

We then opened a host ticket, and started a post mortemum internally. I investigated myself, but also called in Orbit as a second set of eyes.

As far as we can tell, our database had both a read & write copy going. We are confused, because this write copy had persisted reboots, but when this host reboot happened, we were back to eight months ago.

Upon review, we found that the last eight months of backups, were the exact same file. I will be fully transparent and admit this is a MASSIVE failing on my behalf. It is confusing, because upon restarting this exact setup, backups worked correctly, and gave us changes, but could of been potentially avoided had I been more due dilligent in checking backup accuracy. I had grown complacent in the fact that the restore I had done at the beginning of the wipe had worked perfectly well, and had checked them shortly after, but didn't check them when player count dwindled.

While this is a super confusing issue, the truth of the matter is that it could of been potentially avoided. As an Administrator, I've always tried my best to be honest and transparent with the community, and felt the need to do so here.

We have implemented new procedures and machine/backup changes to prevent this going forward, and you have my sincere apologies.

What Are You Doing

We will be accelerating the finishing of our client, and taking the opportunity of a fresh database to make drastic changes. The last time we wiped, I was still unsecure with the idea of a wipe and did not.

Rest assured, donators will be compensated, and recent donators will see some form of transfer over. We will reach out personally. We will also be compensating/rewarding Challenge League winners once the server is back online.

As of now, Valor is up with a lot of loot boost, for those interested in playing casually or just dropping a relic or two, while we work on the relaunch. This server will go down in a week or two. If there is an interest in this staying up, I have no issues in moving it to our smaller test box, and keeping it up, but unmaintained until the re-release.

Our new Game Client is discussed more in depth in past blog posts, and we hope to put out more teasers for it very soon, and focus on devblogs as well.

We will launch a cross platform game launcher with it at the same time.

Our website and discord will be updated within the coming week to reflect

Looking For Developers

If you're interested in joining us, please reach out to Arcanuo on discord.

Feedback Form

We are putting out a feedback form, that will collect information. We are mainly focusing on gameplay at this time.

All questions are optional, so feel free to put in as much detail as you would like. We appreciate all feedback.

We're asking these questions to help formalize our timeline, and judge what is important as we make the direction of the game more "Valor", and less random pieces of content.

Donator, Account Changes & Gameplay Loop

I think this one of the bigger failures of our wipe was that we kept accounts as is, and we're afraid to make meaningful changes to the game loop, because it would break accounts.

We will be resetting accounts, there will be compensation for Donators & Challenge League winners.

This wipe will see a fair amount of changes to the gameplay loop, we hope to expand the Realm a bit, as well as better integrate systems such as Trials with the actual game, not have them feeling like add-ons.