A lesson in live event crisis management

The first text came at 6:05 a.m.

“Do you know why the website is down?”

The medium for the message belied its urgency. Even so, an edge of panic was detected, or rather, inferred.

The website couldn’t be down on this day, of all days. Spring commencement was a mere three and a half hours away. That gave us three hours and 29 minutes to remedy whatever was wrong so we could broadcast our ceremonies to a global audience.

And most of us hadn’t even rolled out of bed yet.

Nuts & bolts

Let’s not mince words here: Livestreaming can be a pain. Anything can happen. There is the inherent potential for so much to go wrong, with very little response time to troubleshoot. And we at the Department of Digital Strategy at the University of Arkansas at Little Rock, being the ambitious sort, decided to broadcast the biggest event of the year using Google Hangouts on Air.

* We are: Dan Shisler, the man; Jennifer Godwin, content queen; Aaron Baker, SEM/analytics guru; and Meaghan Milliorn, social media maven.

Of course, livestreaming means a lot of setup and testing. Cords here, cables there. We must have a secure, stable internet connection, preferably ethernet. We use firewire to capture the video from the film crew in the production room and then combine that with an audio feed from the sound guy. The convoluted setup required us to be in the same tiny room as the video production crew – a space that feels and operates like a bunker.

Pro tip: Google Hangouts on Air is a fantastic tool for livestreaming if you have a Google+ account. The free service connects to your YouTube account and automatically saves your videos to your channel. We’ve even played around with the cameraman function, which allows for a multi-camera (or laptop, in our case) feed. Those videos can be watched on a computer, a phone, or a tablet. Win win win!

In fact, all had gone swimmingly during our test run the day before commencement. We were feeling good – which if you know anything about foreshadowing – of course meant something bad was inevitable.

* Not our real titles.

6:30 a.m. – 3 hours until showtime

The first text sparked a flurry of responses. Troubleshooting began immediately, with Information Technology Services being brought into our conversation. That team was already working on the issue, having been alerted in a different platform.

Not knowing the root of the issue or the extent of the problem, the digital team began brainstorming solutions for a worst-case scenario, which was no internet for the duration of the day.

Before the advent of livestreaming and social media, no internet at commencement would not have mattered much. But in this day and age, especially at UALR where we have a substantial international student population, the internet is a crucial lifeline for families that could not make it to the ceremony.

Our global audience for commencement included 40 countries

So we knew we had to have the ability to broadcast. We had promised it in several emails, in our social media, and on our homepage leading up to the big day. We could not fail.

But something did fail. Something so out of our control that no amount of planning or Backup Options B, C, or D on our part would have made a dent.

Websites go down for a multitude of reasons. Usually, the downtime is quite imperceptible, and only those in close contact with the web notice. This time was different. No web for a prolonged period. Whatever the problem was, it was more than just a cranky server.

7:30 a.m. – T-minus 2 hours

As we all begin to trickle in to work, word comes around that a vehicle hit a transformer near campus in the middle of the night, touching off a series of unfortunate events. Power was knocked out to the entire datacenter, also causing another failure so abrupt and overwhelming that left the backup generator idle when it normally would have kicked on.

No network = No live stream on ualr.edu.

No good.

We decided then that we needed a communications plan and some alternatives to our network. A message was crafted but not published, just in case. Even with our site down, we could still broadcast to YouTube (provided we had a stable internet connection) and send our audience there from our social media channels.

But for our webcast to take place, remember it must happen in that cramped, bunker-like production room.

8:30 a.m. – T-minus 1 hour

At this point, grads and their friends and family are starting to stream into the Jack Stephens Center, backing up traffic on all roads leading to campus. Later, we are told that the crowd is the biggest ever at the arena. And they all have smartphones*.

Spring 2013 commencement was a full house at Jack Stephens Arena.

That’s a problem, because we are attempting every solution possible to get internet. One of us headed back home to fetch a wi-fi router. Another one’s spouse was driving in with a tethering solution.
Our Facebook fans were kept in the loop about the livestream.
At 8:37 a.m., with options dwindling and concerned tweets just minutes away, we issue a message on Twitter and Facebook: “Our server is currently down but IT is working hard to have it up and running ASAP.”

IT Services was still working hard to get the network back up, but there was no ETA on when that would happen.

*slight exaggeration

9 a.m. – 29 minutes left

At 9 a.m. on Facebook, we explained the situation: “A power line was knocked down last night. Working on an alternative solution. We’ll keep you updated!”

Those alternative solutions had both been stuck in traffic. Once they arrived, we divided up our team to try both.

One patch, connecting to the cable modem via a wi-fi router, wouldn’t work – in that bunker of a production room we couldn’t get a signal through all of that cement and metal. And we didn’t have a CAT 5 cable long enough to work (not to mention it would have been right along the path grads must cross to enter the arena).

That left one option, and it was a gamble: Creating a hotspot and tethering.

We start by trying the wi-fi hotspot. It’s a bust. The laptop just isn’t picking up that signal, no matter what we do.

9:20 a.m. – The show must go on

The grads begin lining up for the processional. As far as they and their families in the arena are concerned, nothing is wrong.

We are still determined to make sure nothing is.

We try the iPad hotspot to get us on the 4G cellular network.

And then there it is, a sight so sweet we nearly tear up: A signal. Not entirely full strength, but not weak, either.

With seconds to go, we scurry to set up the Hangout on Air. Once the Hangout is set up, it takes a minute to process before spitting out a YouTube url and embed code.

It wound up being the longest minute of the morning.

9:29 – And we’re live

The commencement livestream goes up just as our beaming grads take their first steps into the arena.

We are beaming, too, to more than 40 countries around the world.

Commencement livestream for Spring 2013

Shortly thereafter, our website comes back up piecemeal, beginning with the homepage and then extending to the original home for our livestream embed.

We share the link with our grateful online audience.

9:35

Happy sighs of relief are breathed and high fives happen all around. We learned some lessons that morning, but we also realized that we were fleet of foot and brain, which certainly came in handy when the odds were against us.

But we aren’t out of the woods entirely.

This is a livestream, after all, and anything can happen.

Posted in: News

Comments are closed.