Site update- Technical problems and a tale of woe
As many of you will now be aware, we’ve been offline for a few days and we wanted to keep everyone in the loop about what happened.
On the evening of 5th September, the HD at our server reported a problem and was dying. The result of which meant browsing to any of our site domains didnt work and ended up at a blank screen. Our hosting provider spotted the issue and quickly brought it to our attention.
This is where things went wrong. We *used* to have server-side backup facilities provided by our host but that changed when they got brought by another company and that facility was no longer offered. As such, it was up to customers to arrange their own backup plans and facilities. This is fine. Its their prerogative and they informed all customers of this, so I can’t hold them reponsible.
Sadly, it was something that mentally got put into the ‘yeah I’ll sort that later’ bucket, which, sadly didn’t ever get actioned.
So, with no backup to be able to rely on, we were in a serious situation with members not being able to access the content they’d subscribe to enjoy in good faith.
The tech guys at our host tried a number of options to bring things back to life and with varying degrees of success, but kept me informed of progress and the next options and I can’t fault his approach and thank them for it. The result of which is that on the evening of the 8th September we actually had a server re-installed with 99.9% of the data alive and well.
So, my first thanks have to go to Chris Eastman at AmeriNOC.com Excellent service again and his efforts are very much appreciated.
Nearly there now. The last headache came when trying to log into the members area of the site(s) – this was failing with an error code 500 server misconfiguration error. A difficult one to diagnose because the data had been migrated across ‘as it was’ so to spot what had changed was difficult.
My second thanks have then to go to Bill at phantomFrog.com who took time out to help me diagnose the problem – which, incidentally, is helping with issues that are not his responsibility, nor the result of the excellent software solution he provides for me. Yet he stepped in and used his expertise and insight into helping me out. Far beyond the call of duty and again we really, really appreciate that.
So, to today. We had a quick check with CCBill who helped us get over the last hurdle and we’re up and running again.
The point of all this? Well, two things.
- If you’ve got a website up and running, for God’s sake, make sure you’ve got contingency plans and rescue strategies in place if things go wrong. As Bill explained to me yesterday, “There’s only two people in IT – Ones who have lost data, and ones who haven’t lost data yet”. I’m now in the former category!!
- To thank the people who’ve helped us get over this very trying couple of days.
There’s one last thank you I have to offer and that’s to our site members. Despite a couple of days downtime, you guys took it on the chin and didn’t come marching to my door, pitchforks in hand, ready to bash my door in. For that, Im eternally grateful and restored my faith in ‘Joe Public’ as being a decent, patient guy after all!
Cheers
Robin
