RE: THE SECRET TO PREVENTING BUSINESS DESTROYING DOWNTIME

My story is more crazy than someone forgetting to pay the bill. It's some years old, but still relevant - it was an eye-opener for me.

I was a "devops" guy at my previous workplace, hired as a developer but eventually ending up doing quite a lot of systems administration. We decided that maintaining a working email server and keeping the spam out was difficult and not a part of our core business, so it was eventually outsourced to Google.

I suppose its a well-known problem among system administrators - there are lots of cronjobs running, some of them as frequently as every minute, and error messages are sent by email by default. Fixing those errors are rarely a priority. With gmail it was also very easy to set up filters to just ignore those emails.

The account had a quota for the amount of gigabytes one could store, but it did not have any quota of the number of emails (I'm such an old-fashioned guy that I still think attachments and HTML mail is evil. With only proper emails in the email folder, some few gigabytes should be sufficient to hold almost a million of emails), but apparently (at least then) there were some hidden limit - after exceeding some hundred thousands emails, things just stopped working.

I did have some few days of intermittent errors and slowness before my account was simply disabled.

People sending me emails would get an error in return that the account had been "deactivated" (an indication that I had been sacked). I would not have access to my email archive. We had also started using the office suite of Google, I no longer could log in and didn't have access neither to my own files nor shared files. Also, I'm a frequent user of "forgot my password"-links, when the email stops working one often finds oneself locked out.

This was a paid account and of course we were complaining to customer support ... but this is Google, they don't really have a customer support. All we got was polite answers from the first line containing phrases like "our engineers are working on the case". I don't know, we probably had a 98% SLA, and Google considered that if 98% of our email accounts were working, everything was fine.

We had to create an account "tobias2" for me. No way Google support could help forwarding emails sent to tobias to the new tobias2-account. Being a system administrator and informing contacts that "... sorry, [email protected] doesn't work anymore, you have to send emails to [email protected]" ... that's just extremely embarrassing. This wasn't the primary reason why I quit that job, but it possibly did play a part. Half a year later (and one week before my termination date), my account was back up and working.