We are working on an experimental mobile based social photography app called Snaptime and we have an ambitious plan to have it on the various app stores by the 1st of February so that people can start to play with it and help us learn what it does [and should do] for them.
8 hours to try and resolve Plan A
But today we got word from our cloud provider – Digital Ocean -that our server had been sending data out at an alarming rate. For the non-techie a cloud provider doesn’t provide fluffy clusters of water vapour in the sky, they provide computers you can access via the internet that are really quick to set up. Anyway, we use this Digital Ocean – for my other startup Amazemeet. We imagined (or used to) they were our partner in running a reliable service and when some kind of emergency happened, that they would be right with us, working to resolve it.
Well, it turns out this ‘alarming rate’ was 35 million packets of data in a very short time – approximately 18 minutes. That’s more than 220 megabytes of data in a 18 minutes. Their systems detected this and locked down our server and disabled the network connections so the flooding could be contained. They also locked the account.
They both entitled and entirely correct in taking this course of action.
Then they emailed us and asked us to investigate and then explain to them what happened and then they would investigate and consider whether to switch it back on.
Well – so we did – as much as their lock down would allow. Which is not very much – and my 2 emails to them in 8 hours to get assistance went unanswered.
Meanwhile the development is at a standstill. But not for long.
10 minutes to switch to Plan B
Fortunately I keep a backup provider – Vultr.com – for just these kind of situations and the turnaround time to get a new server, set it up and be back online is frankly hard to imagine possible even 5 years ago.
After 8 hours of getting nowhere, I tired of Digital Ocean’s lack of cooperation in this matter – which lets face it is simply a cheap lesson in picking hosting providers – and made the decision to bring the service back up, things happened rather quickly.
I emailed Digital Ocean to tell them since there seemed to be an impasse and frankly radio silence from them, I had no other choice but to destroy the server (so I didn’t continue to pay for it) . Even to destroy it proved impossible until they removed the lock – which they did on request. In fact – they replied faster to me deleting my server and initiating my plan to move all my services away from Digital Ocean than they did to my requests for assistance. I think that is a poor business decision.
It took me less than 10 minutes to get a new server on Vultr.com, change my Cloudflare settings to point to a new IP address and now my developer is going to take anywhere from 30 minutes to 2 hours to put all the software we need back on it and bring it back to full strength.
Mike Gets Philosophical
I have a saying I recite to myself when things don’t go the way I expect them to – “Don’t get angry, get philosophical”.
Given that we have growing subscribers on Amazemeet who are increasingly relying on our service to be robust and reliable. I consider the relatively unimportant downtime on Snaptime as a dress rehearsal for what the experience of the extent of Digital Ocean’s willingness to help me overcome a revenue impacting service. So this was a cheap lesson of an important subject and for that I am a grateful student.
Downtime in the Age of Cloud Computing means that provisioning metal (thats system administrator speak for getting servers setup) is now a fairly simple, quick and inexpensive task. It also means that with the technical complexity resolved, the battle ground for providers is in customer service and recovery partnership.
In this regard I’m disappointed to say Digital Ocean has let me down. I did like them – or my impression of them – young upstarts daring to grab a sandwich from Amazon’s unconquerable AWS service. The underdogs, the cool kids doing cool things for other cool kids. But alas that is not really the case, from this experience my impression is they really couldn’t give a cockroach’s wotsits about my predicament.
But heigh ho – I simply hop on another cloud and carry on my merry way. Lesson learnt, achievement unlocked for fastest Plan B ever.