Recent Web Hosting Issues
Giving Customers a comprehensive explanation for the recent hosting issues is quite tough, more or less due to the lack of information given to us. I’ll break it down.
Sunday evening
Around 4:30pm all FTP servers were suspended – FTP allows you to uploaded files and images to your website, it has no effect on email or the actual viewing of the website – it is just the means used to upload new changes. Work is often carried out at the weekend on servers, so nothing unusual in this.
The major problem was that a client launching their products into 400 Tesco stores the next day needed to get their website up and running – this was solved by moving their website to another hosting provider.
Monday morning
FTP Servers were still offline and at about 11am one web server went offline, this affected about 20 clients and meant that indeed their websites were not working… The problem was fixed about 4 hours later – with I am afraid little in explanation of what went wrong. Servers do go down and it can take a few hours to get them working again, but 4 hours is a long time.
The major problem was that a major Irish IT company was due to launch their website the next day and were trying hard to test the website – not a good start – by about 4pm all services were resumed – including FTP servers.
Tuesday morning
FTP servers were down again and also all the control panels – actual websites were working as was email, but the technicians were making more checks and works started on Sunday seemed to be on going.
Services resumed in full at about 7pm.
Explanation
What appears to have happened is this, and bear in mind I am not full informed, but on Friday last week some websites (none of ours) were hacked, we use shared website servers so we cannot control what websites are on each server. This issue was tackled immediately by the technical team on-site and the process to block hackers was started.
FTP servers were switched off to prevent malicious access until the source could be found. The nature of the hacking too was non-malicious but spam based – it appears HTML code was inserted into website pages linking to spam websites. No passwords, user information or confidential information would have been accessed, only information that was publicly available on your websites. I would hazard to say that only HTML based websites would have been accessed, sites that run of databases would probably be deemed not worth hacking by these hackers.
All the servers were then cleaned by the technicians and special scripts used to go through all websites that made sure all references to the hacking were removed.
The downtime on Monday is most like totally separate issue or because one of these cleaning scripts went wrong and the server had to be taken manually offline.
Finally because the control panel was taken offline, it is possibly thought that the control panel was the weakness. Using an industry standard Windows software to manage website might mean a) security holes existed and b) Spammers knew how to access website via this software. I suspect the control panel software had to be updated to ensure it was secure.
Now – most of this is pure hypothesis – because I’ve been told very little, technician are working, problems are being resolves, inconveniences apologies for and now (you have to read my other post to understand) because everything is back to normal we can all heave a sign of relief and hope nothing goes wrong ever again.
I would like however to reassure you that all through this time people were working on your behalf to put whatever problems they had right. Some messages from technicians came through in the early hours of the morning. I know people have been working very late and very long hours on all of the above. Ultimately I thank them. But moving forwards I leave it to my affected customers to decide what needs to be done and whether this response is acceptable or not. All I can say is carry on reading the post beneath this one.