Six things to learn from’s holiday disaster

by Neil Moncrief on July 6, 2009

If you’re involved with e-commerce (and you don’t live under a rock), then you’re probably aware that, the biggest, and arguably the best, payment processor around, experienced a failure of Biblical proportions this past Friday morning.  A fire in their Seattle, WA data center brought them (and the 238,000 Internet businesses that depend on them) to their knees.

For the record, I’ve been a fan of for years.  I used another processor in 2000 while first getting started in e-commerce, but we soon changed to  Their service, features, and cost were (and continue to be) superior to their competitors.  I have many e-commerce clients today who rely on their payment gateway.  That being said, dropped the ball on Friday.  What remains to be seen is whether they will fumble or retain possession of the ball.

If your e-commerce business uses, you were certainly affected and probably lost money as a result.  So that your money wasn’t lost in vain, let’s look at what happened and try to learn a thing or two from’s mistakes:

  1. Have a backup plan that includes worst-case scenarios, and test your plan. I don’t know exactly what did wrong.  I don’t know their business any more than I know yours. But I do know that when they shifted their traffic from Seattle to their backup data center(s), something didn’t jive. It may have been due to increased traffic resulting from millions of holiday promo emails. It really doesn’t matter. They knew it would be a busy weekend, and they weren’t prepared to handle what happened.
  2. Have a communications plan, before you need it. If your business is dead in the water, how will you notify those who depend on you? How will you answer their questions? And, perhaps most importantly, how can you be sure your customers will believe you instead of rumors circulated by others? I was in a meeting with a large e-commerce client when I first heard about the outage. He read a Tweet about it online. His business website was down, and the owner learned why through Twitter. That’s amazes me! signed up for a Twitter account, but not until about 11:00 a.m. EST Friday. Within minutes, the Twitter account had more than a thousand subscribers awaiting updates. used Twitter to disseminate information to their merchants, and it was the only reliable source of information throughout the weekend.  I’m still getting updates via Twitter, including one a few minutes ago (7:30 p.m. EST) announcing they are finally back up to full speed.
  3. Answer the phone! I understand that most of us don’t have 238,000 customers calling at the same time. It’s not possible to answer that many calls right after disaster strikes.  But I spoke to many merchants today (3 days after the fact), and none of them had been able to get through to  Make every effort to answer your phone, even if you know the caller is going to be angry.
  4. Post updates on your website. Many of my clients were unable to log in to their accounts. That’s understandable considering what happened. But their homepage was up throughout the weekend.  Knowing that businesses across the country were down, could have at least posted a simple message on their homepage. Any news would have been better than no news at all!
  5. Don’t pat yourself on the back while your clients are still hurting. This one should be obvious; in fact, I’m still shocked that had the guts to do it. Driving home from my client meeting at about 3:00 pm Friday, I received another Tweet from  The author was congratulating everyone on “the team” for the outstanding job they were doing.  There was no mention of how their downtime was affecting businesses. Next time,, apologize for your problem before congratulating yourself for fixing it!
  6. Don’t make promises you can’t keep. Tweeted all weekend about how this was finally working and how that would be working soon.  More often than not, they were wrong and had to retract statements. Remember that your clients are making promises based on the promises you’re making to them. If you’ve already let them down once, down let them down again.

As of 5:00 p.m. EST Monday, was still having problems (three and a half days after the event occurred.) Shopping carts were still timing out, many shoppers were still unable to order, and’s website still posted no information whatsoever. Please learn from this,! Your merchants are watching.

I’m happy to provide this information free of charge.  If you found it helpful, please subscribe to my RSS feed so you’ll be notified of future posts.  You can also follow me on Twitter, where I regularly post short tips.  I promise to never spam you or pressure you.  Please forward this to your friends in business, and feel free to rate my post or leave a comment so I’ll know how to improve. Thanks!

{ 3 comments… read them below or add one }

Tony Newbury July 7, 2009 at 11:32 am

It’s the mgt at that needs to read this most of all. Good post.

Local Coupons July 14, 2009 at 7:31 pm

As a long term customers of, we all have some responsibilities on our own act. We did not find the backup plan as well. However, we can’t believe that has no failover redundancy for all these years.

It’s like putting all our eggs in one basket. We are glad that many other payment solutions are still working, so it’s matter of switching the vendor and never come back.

I guess it’s a warning sign of the company showing they are going down hill.

Amy July 18, 2009 at 10:50 pm

I sure hope they will learn from this and have a better back-up plan next time.
Nice post Neil! Thanks for your support during the “crisis.”

Leave a Comment

Previous post:

Next post: