Empornium: News

[Len]

Dear pervs,

After being down since Friday afternoon due to a serious hardware problem, we are finally back online. Defective hardware has been replaced and backups have been restored.

There may be some lingering issues with the site, if you encounter any problems (other than what's described under "Data loss" below), please report them in this thread. We also expect that there may be some shorter maintenance sessions in the next couple of days, we will try to announce these in advance.

If you still have questions after reading this post, feel free to ask them, although we tend to not go into specifics with regards to the technical setup of the site other than what's described below.

Thank you all for your patience.


Data loss

A backup was performed approximately 90 minutes before the downtime started. All data entered into the site in that last hour and a half has been lost, including but not limited to new uploads (about 20 of them), forum posts, credit gains, bonus shop purchases and staff PM's.

Unfortunately, as the result of an ongoing problem with our backup scripts (technical explanation below for those interested), we have lost private message content over a much longer period: from April 24th onwards. You will still see the messages appear in your inbox including subject line, but the message content is gone and can not be recovered. If you received a private message from another user that you hadn't had a chance to read yet, please send them a new message asking to re-send whatever they meant to say to you originally. If you received a system notification that you are worried about, contact staff (don't worry about old "security alert" messages). Staff PM's are not affected.


Rumours

As often happens in situations like these, there has been speculation on alternative explanations for our downtime. This speculation is completely baseless. The downtime was the result of technical difficulties only, and we suspect the people voicing these rumours have been waiting for an opportunity like this to speak up. Ignore them, we will always be honest about the reasons for downtime.

Regarding the claim that there are active users on the site with malicious intentions: This is not news. We have had open signups for a long time in the past, and everyone should always assume the worst when it comes to other accounts on the site being abused. Although we make every effort to get rid of such accounts, ultimately it's up to every individual user to take measures against this as they see fit.


Technical explanation

On April 24th, we started encountering problems with the part of our database that stores private messages, causing site backups to fail. Since technical staff availability at that time was limited, we initially excluded the relevant database table from backups so that later backups could succeed, planning to investigate the issue closer at a later time.

A week later, being last week, we did get a chance to investigate, and it was soon determined that the issues resulted from a problem with our RAID storage hardware, exacerbated by an apparent deficiency in our monitoring systems' ability to notice this problem sooner.

Last Friday, still under the assumption that the problem was relatively limited in scope, we performed an unscheduled backup, after which we attempted to move as much of the private message storage as possible to a location where it could be backed up safely. Unfortunately, this ended up leading to a cascade failure of systems that made it necessary to take down the site immediately and declare all new data after the unscheduled backup lost.

Over the weekend, we corresponded with data center staff to have the faulty hardware replaced. After this was done succesfully, we started restoring system configuration and data from backups, which brings us to now.

We perform scheduled backups once daily; if the unscheduled backup had not taken place we would have likely lost approximately 12 hours of new data instead of the current 1,5.