Secret Scotland

If it's secret, and in Scotland…

Murphy’s Law – it never fails, even for servers

Grumpy MurphyExactly a week ago, I was fearing the worst…

Our host had announced a planned outage while our servers were upgraded, and expected normal service to be resumed after something like 6 hours.

I wasn’t going to hold my breath waiting, since they’d done the same few months earlier, and we had nearly a month of repeated outages, with two being extensive.

The final analysis indicated a disk mirroring bug had taken them out the first time – and the second time was down to human error, after the service company forgot to apply the required patch when the  servers were restarted. So after the same period of operation… down they went again.

This time, I posted an alert about the upgrade and the expected duration of the outage – and all went as expected. The upgrade was not even followed by any short outages  while small bugs were fixed.

After a week, I removed the alert – and tempted fate.

The next morning…

Yep… offline went the site the next morning, and had still not returned by the evening.

We now have a server info page provided by our host (previously, we were just like mushrooms, and always in the dark), and it looks as if the host is innocent.

The problem appears to have followed a sustained DDOS attack against two servers in their system, and specifically targeted at their mail services.

Updates reveal that one server has been able to back online – but it’s not ours.

Ours remained under attack, with the host attempting block the offending IPs, but they said it was proving difficult, as these ran into many thousands.

So, it will be back. It just depends on how long it takes to drive the scum back into hiding.

Update

Of course, Murphy was still going strong when I came in this evening and made this post…

According to the logs, the servers came back up a whole 25 minutes after I published it – but the site remained inaccessible. Then went offline again, 9 minutes later.

After that, it was up and down like a yo-yo, but remained inaccessible.

Update II – Wednesday, November 28

The host appears to have mitigated the attack overnight, with a further report of activity levels falling to normal by late morning today.

Unfortunately, while our domain is now accessible, together with the Wiki, current attempts to access the Forum result only in an internal server error message.

The problem with access to the Forum was cleared up during the course of Wednesday afternoon, and so far, operation seems to have settled down fairly quickly.

Advertisements

November 27, 2012 - Posted by | Site News | ,

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: