Thursday, February 25, 2010

iServe Update

Hot off the wire from the man himself.

Good afternoon,

Earlier this week a number of our customers experienced issues with some iSERVE services, particularly email and hosting console access.

First allow me to apologise to those affected.

Second, I want to give you a full run down of what happened, what we did to fix it, and what we are doing to ensure that it doesn.t happen again.

What happened?

On Monday we encountered problems with some users accessing hosting consoles and email, which we discovered stemmed from data corruption. Consequently services were offline for 2 hours on Monday evening (between 8 and 10pm) as databases were repaired.

The problem re-appeared on Tuesday morning, and we attempted (unsuccessfully) to repair the databases without further customer impact. Around midday the decision was made to take services offline and carry out a full rebuild from hardware up.

Service was then restored to all customers by 6pm, although you may have noticed email being delayed after this time as the systems worked to clear the backlog of email.

Overnight there were small outages for some services as we made further improvements to the infrastructure so as to ensure that services remain stable.

What have we learnt?

Firstly we have resolved the underlying technical problem and invested heavily in additional processing capacity

In the future we will also be providing more effective and timely communication of the issues, as we are aware that this caused frustration amongst some of our customers.

We realise that some of our customers have suffered as a result of this issue and we sincerely apologise for this.

Kind regards,

Scott M. Bartlett

Nice. Good to see that they have learnt from this failure, unlike others in the same field.

iServe / Orcon has possibly earnt themselves a couple of brownie points. Hopefully next time regular progress reports on the website will be forthcoming.

No comments: