How many returns did you have after the holiday season? It seems that this is all part of the fun after the holidays returning what was not wanted and buying something you do want. Not being overly hard to buy for, I only had 1 item to return. I figured that I would take care of this during my lunch break as I had a few other items to purchase as well as a quick stop to the bank which was in the same shopping center.
When I got to the bank, the ATM was off line – which sometimes happens. So I decided to go inside, which I rarely do. As I walked in, I was greeted with a huge line because the bank itself was "off line". This is one of the biggest banks in the area, and they were not able to provide transactions. I tried to assess how long this was going to take by asking the manager about the problem (network connectivity, local area network issues, etc…). He didn't have any ideas and had the "deer in the headlights look". He mentioned that he had rebooted the "system" twice and nothing had changed. The bank had been down for 25 minutes at this point, and local customers seemed less than impressed with the manager's troubleshooting skills. Since the situation was not improving, I figured I would come back another day and just do my shopping and returns instead. Well, let's just say, the store was more of the same. Long lines and not a lot of happy faces on the customers or the staff. Rather than do my shopping and stand in line to be disappointed, I went to the cashier who also informed me that they were "off line". Soon after, they ended up ushering all the customers out of the store.
Doing a quick technology assessment in my head, it seemed that there was a Wide Area Network (WAN) issue going on in the shopping area, or the LEC was having an issue because that is what would connect the bank and the store. More importantly, I saw that both of the organizations did not have any kind of business continuity plan to keep their systems up and running. They also did not have a disaster recovery plan on their Wide Area Network or any process for troubleshooting procedures for Local Area Network (LAN) or WAN services. The obvious problem with all of this is the disruption that both companies had with achieving the best "Customer Experience" possible. I personally lost time and it was disruptive to my day. What was the total cost of lost sales to both the bank and the store at the end of the day? I am sure it was much more than a secondary bandwidth circuit would cost in a year? For instance, if the store gross profit is $3.5M per year, which is low, then the opportunity cost for lost sales would be somewhere in the neighborhood of $5K to $7K for a 5 hour outage. I think the store was down for 7 hours in total. As you start to make your plans and technology initiatives for 2015, a priority that I suggest is that you look to maximize your network infrastructure. Whether you are supporting only your internal users or trying to make sure your external users / customers are up to an optimum level – you will need to invest in the time to do research to make your network fail proof. For more information on how to start your own process, please feel free to download our "Disaster Recovery Checklist".