RSS

MDD Hosting Clients Suffering Major Downtime

Sun, 23rd September 2018, 15:16

Clients of MDDHosting have taken to the web to express their outrage, anger and frustration at the extended period of downtime websites are experiencing after some routine maintenance corrupted beyond repair all the data on the company servers.

MDD Hosting’s Michael Denney writing in the community forum states:

“While I was hoping to save some of this for the official RFO [Reason For Outage] - enough people are getting tremendously upset over this that I'm going to spell out what I can now - keeping in mind that I will provide more details when I can.

First and foremost - this failure is not something that we planned on or expected. A server administrator, the most experienced administrator we have, made a big mistake. During some routine maintenance where they were supposed to perform a _file system trim_ they mistakenly performed a _block discard_. The server administrator essentially told our storage platform to drop all data rather than simply dropping data that had been marked as deleted by our servers

So why is restoration taking so long?

Initially the company believed that only the primary operating system partition of the servers were damaged so they worked to bring new machines online to connect to their storage(back-ups) to bring accounts back online. Denney states “Normally we would rely on snapshots in our storage platform - simply mounting a snapshot from prior to the incident and booting servers back up. It would have taken minutes - if maybe an hour. We are not sure as of yet, and will need to investigate, but snapshots were disabled. I wish I could tell you why - and I wish I knew why - but we don't know yet and will have to look into it”.

MDD Hosting is now working to restore client data from cPanel backups from their disaster recovery server of ‘last resort’ in Phoenix Arizona. By the company’s own admission, this is the only copy remaining of all client data.

While hoping they are wrong, the time to undo the damage for all clients is being estimated at 96 hours.

“Do I really expect this to take 96 hours? No. I don't and I hope not - but I don't want to say 24 hours and have 10 people not restored yet get really upset about it. At the end of the day I want to be as accurate as I can be and I don't want to lie or mislead. This situation is really bad for us - the worst thing we've faced since we founded this company in 2007".

The company also admits that backups may not be complete with most servers from restore point that is within the past 24 hours prior to the start of the incident (this can be anywhere from a few hours or up to 24 hours depending on the time the account was backed up). For some less fortunate, a very small number of servers may have restore points going back 48 hours before the outage. While it may be a small consolation, in times past many hosting providers, having faced this many failures, the perfect storm so to speak, have simply lost their clients data entirely and gone out of business!

MDD Hosting Backup Restore Estimates

 

r1    Monday, September 24, 2018 at 11:00:00 PM

p2    Monday, September 24, 2018 at 3:00:00 PM

s2    Tuesday, September 25, 2018 at 1:00:00 AM

r2    Tuesday, September 25, 2018 at 9:00:00 AM

s3    Tuesday, September 25, 2018 at 4:00:00 PM

r3    Wednesday, September 26, 2018 at 12:00:00 AM

s4    Wednesday, September 26, 2018 at 4:00:00 AM

r4    Wednesday, September 26, 2018 at 6:00:00 AM

s5    Wednesday, September 26, 2018 at 12:00:00 PM

s0    Wednesday, September 26, 2018 at 12:00:00 PM