Visualizzazioni totali

lunedì 8 aprile 2013

WeekHell

Weeks never end in the world of online gaming, these games are always on and require constant support and backup.
The last two weeks have been so tough that i decided to name this post WeekHell.

But let's start from the beginning.

Everything started with a regular maintenance, the servers were shut down and a new patch was applied, as usual i performed the regular tests and everything seemed to be working fine, so the servers were opened to the public again and everybody started playing. Just a couple of minutes later i recieved the first complaints, two portals to some of the most important dungeons of the game were not working.
I went to check them myself and it was true, the portals were broken, nobody could get into the dungeons.
A bug report was immediately sent to the game developer, wich provided a patch only two days after.
Because of this issue, i had to skip my day off, since i was the only GM available, anyway, i went to work to test the new patch, it took several hours, did all the regular tests we do when a new patch arrives and i had to re-check every single portal in the game to make sure they were all working. Since everything was looking fine, the patch was applied with an emergency maintenance, of course the players were pissed off by this but as long as it was supposed to solve the issue...
Of course with the maintenance the tests had to be performed again, i double checked the stuff that didn't work before and since the issue was removed, the game got launched once again... much to my dismay.
I was starting to relax but after a bit over an hour, i recieve some seriously disturbing complaints, people were not gaining the correct ammount of status and skill points when leveling up.
That couldn't have been possible, i mean, one of the basic tests we perform during EVERY maintenance is in fact to level up and see if we gain these points, and i did gain them correctly when i tetsted.
I logged into the game with a new character and leveled up, i was recieving the points so i didn't bother with a bug report. Went to have lunch and i got called back into the office, the issue was getting bigger, in fact a lot more complaints arrived, and luckily, this time they went into more details, only some charachters in a particular game mode were not recieving the points.
New tests performed, confirmed the bug, but it was too late to send a bug report, i had to contact the higher ups and they had to directly call the developers, but a patch was sent only the day after, meanwhile people kept leveling despite the warnings, unbalancing the game.
With the new bug fix, more tests were required, took the better part of the day to test and i also needed some external help, otherwise i would have never been able to try out everything in a single day.
After making sure that EVERYTHING in the game was working as intended, the patch was applied with another emergency maintenance, but we could not remove the imbalance that had been created so far.
A new and longer maintenance was needed for this, so we waited for the next weekly maintenance.
Unfortunately, the query needed to restore the points of all the affected characters did not work properly, more complaints arrived, and a new solution had to be found.

Doesn't sound like much? Well, it was really a lot, believe me, and it's not even over.
As if it wasn't enough, the company started a security campaign, and due to this, a lot of accounts got blocked, adding even more complaints and recovery requests. In the end, i was about to burst, being the only GM around for 2 whole weeks i couldn't keep up with everything.

I got uber stressed and overworked, spent 9 days straight in the office, but i learnt much about how a gaming company should be directed and organized, this experience showed me how unprepared the company i work for is. I'm not going to do the same mistakes as them working on my own videogame, that's for sure.