Thursday, 27 June 2013

Letters from the Engine Room





Hey guys, 

As Playfire continues to improve and expand, we wanted to start giving you an impression of where the site is heading. I’ve now been on board the Playfire train for around 3-months, give or take, and with Ben having joined last week – see his nice intro post here if you haven’t seen it already - we thought it was a great opportunity to start giving you more of an insight behind Playfire’s hulking metal doors.
To do this we've enslited Lee, GMG's Co-founder and EVP Engineering at Playfire and GMG. He does the tinkering in the system when we want something added, changed or improved, along with his team of Engineers
So without further ado, we’ll pass you over to Lee.

Hey guys,
I wanted the chance to explain to you all the components that make up Playfire on the backend, and what we are doing to improve and, in most cases, completely get rid of the issues you guys have. This particular post is going to focus on our databases.
Playfire has a lot of data. It’s not on the unimaginable scale of Facebook or Twitter, but there’s a considerable amount of stuff here. It’s all split among a number of databases:

- Achievements are stored using a PostgreSQL 9.1 on a single server containing 229 tables containing around 6 million rows each.
- Gaming logs (the “you played game for X and Y hours” stuff) are spread over 4 servers running PostgreSQL 9.1 each containing approximately the same amount.
The rest is stored on a single MySQL server. All of this stuff includes things like social graph data: who follows who, etc. Basically, ‘everything else.’

Ever since we acquired Playfire we’ve wanted to move it to a similar infrastructure that we used to run GMG. They’re technically on the same front end using Python and Django, after all.
But these databases are insanely huge and cumbersome, to say the least. We enlisted some help from the experts – a company called EnterpriseDB – for them to take a look at Playfire’s databases. They tweaked and played with it quite a bit, but the ultimate conclusion was pretty simple: “you need new servers.”
So, the task of moving things begins.

Let’s start with the Achievements, which are one of two reasons why profiles take so long to load for you guys. We have to be very careful when moving this data. We cannot take Playfire offline for 4-5 hours, and doing that could result in any amount of data corruption. It’s unlikely we could transfer it all without experiencing some issues.
To solve it we have to do the following:

1.     Reconfigure the Postgres server to enable replication
2.     Take a hot copy backup of the server to the new location
3.     Restore hot copy at the new location
4.     Set up replication from original server to new one
5.     Let database bed in for a few days
6.     Flip over to reading the achievements from their new home

As it happens, copying that database isn’t proving too fun. We gave up tar’ing it (aka zipping, for the non-Linux users among you) and trying to transfer it. 
We’ve now opted for uploading it to Amazon S3 and downloading it at the other end. This seems to be a bit hit and miss. Sometimes we get to upload at 10MB/s and at others it slows to a crawl at 600KB/s. Fun times.
Once the transfer is done we’ll be able to enable replication. We get to do this without taking anything offline or disrupting the front of Playfire at all. Testing then becomes important; we don’t want to do anything that will slow things down even further. We’ll most likely kick the tyres on it a bit and ensure it’s going to cope with everything we need it to.
After all that is complete, we’ll then transfer the backend server, which gives Playfire the achievements. At that stage we’ll be able to point the website at it and it’ll be far quicker than it is right now, even though the new location is at a data centre in an entirely different part of the world. 
I hope that gives you an idea of where we’re at right now. We’re constantly working to improve the site as best we can. Things take time, but making Playfire a better place for you guys is a huge priority for us.
Cheers,
Lee
Co-Founder GMG, EVP Engineering Playfire & GMG

We couldn’t have put it better ourselves! Lee will be doing regular posts like this one, focusing on other bits of the site that he and his team are constantly working on at Playfire HQ.
Most importantly, we want to pull back the curtains so that you see the site development occurring as we work on it, even if the improvements are behind the scenes and not immediately visible as new features. Hopefully this first post gives you an idea of what we’re aiming towards, and how our vision for Playfire is going to continue to grow as we work our way into the future.
As ever, let us know if you have any questions and both Ben or I will get back to you as best we can.

Thanks for reading guys,
J, Ben and Lee.