It seems that the client’s website is down.
No, it’s not a one-off failure. I still have my own monitoring tool from when I administered and developed that website and it has been sending out alarms since 2:20 in the morning.
And it’s already 6:50.
So something has happened with the server.
I try to connect via ftp and it doesn’t respond either.
It’s not just the web server that’s down. It’s the whole host.
Well, it’s not me who administers the website. I’m not the one who administers the website and I don’t have the keys or access to support so I send a message to the company that runs it now and wait for them to check it.
It’s 9:30 and still no response. But in a Telegram group they post a message published a few hours ago on Twitter by the CEO of OVH
We have a major incident at SBG2. Fire broke out in the building. Firefighters were on the scene immediately but were unable to control the fire at SBG2. The entire site has been isolated affecting all services in SGB1-4. We recommend activating your Disaster Recovery Plan.
Octave Klaba @olesovhcom
Oh! Oh!
I imagine SBG2 refers to building 2 in Strasbourg. I don’t know if the website is hosted at SBG2 but I do know that the client’s website is hosted at OVH.
It is not a small local hosting. OVH is one of the most widely used hosting services on the web beyond Amazon’s AWS, Microsoft’s Azure or Google’s Google Cloud. It has 27 data centres, mostly in Europe, but also in Asia and North America. In total, it has around 1.5 million customers.
Immediately the black humour begins.
“It’s hot in there”.
“They must have had the firewall down”.
The downtime of a service is becoming the order of the day. Moreover, the impact is already considerable with many of these services, so they are beginning to be not only twitter news, but also front page news in the press and on TV.
In fact, “Teams and other Microsoft services are down all over the world” is now being highlighted in the Spanish press. This is not unique to Microsoft. The same is happening with Google, Amazon, Facebook, Whatsapp, Twitter, …
So now when our Roomba stops working or the lock of our house doesn’t recognise us and doesn’t want to open the door we have to quickly look at the press.
But going back to our website the serious thing here is that “We recommend activating your Disaster Recovery Plan”.
This is a big word.
Because if a service is down for an hour, we’re going to do something else and that’s it.
But when we are told “We don’t know if we are going to be able to recover it. Go and look for alternatives” a shiver runs down our spine.
Because it’s not going to be one hour and that’s it. It might even be “We’re never going to get yours back”.
It’s like seeing a police car parked on the side of the road just up ahead.
You know you’ve got it all right. But…
It’s the same here. You know you’ve got it all right. But…
You’ve backed everything up pretty recently. You know what you need to do. You need to hire a similar hosting from another provider, change the IP addresses to point the domain to the new server, recover and restore the last backups, check and that’s it.
It’s just a matter of time.
Maybe quite a long time, especially if you have many customers on the same provider.
Plus, tell all the clients that were on that hosting you had recommended what has happened and that during the next day they won’t have the website available. And be ready to answer dozens of Whatsapps during the day from people who have the website down and a lot of time to think.
Luckily by late evening the service was up and running again.
And, once you’ve been able to breathe easy you think “How many thousands of websites are not going to be able to recover from this downtime?”
Backups are like insurance.
It hurts your soul to be paying month after month for something you are not using at all.
And, if you’re lucky, you’re going to pay for it all your life without ever using it.
But, if at some point you have to resort to it, then you’ll be grateful you took it out.
If your websites and applications and, above all, your data are important for your business and you need to be sure that they will still be there in case of crisis, contact me here and we can talk about it.