This is a long one. I want to capture the emotions of that day and the months that followed. Emotions so strong that I still tear up when I think about them almost 25 years later. I want to capture the heroics and sacrifices of thousands of AT&T people. I am limited to what I saw 1st hand and what I heard 2nd hand. That will be far, far from everything but hopefully a representative sample. I doubt my limited vantage point, my memory, and my writing are up to it. But here goes.
Paradoxically, the 9/11 terrorist attack in Manhattan is the worst tragedy I ever witnessed in my life and the most gratifying experience in my career.
The commitment, heroics, and sacrifice of my AT&T colleagues was extraordinary. On their own initiative, people did things that I would have never asked them to do. Upon reflection, while I may be an infamously demanding leader, I think I spent a too much of my career expecting too little of people. Individuals and teams have unimaginable potential.
The Context.
9/11 was a Tuesday in 2001. The attack happened at 8:41am.
AT&T’s largest local network node was in sub-basement 6 of one of the World Trade Center Towers. The node housed a big switching system and oodles of optical transmission systems. It was a major interconnection point for fiber optic cables in the extraordinarily telecommunications-dense Wall Street area.
When the Towers collapsed, the node was destroyed. Much of the fiber that connected it to Customer locations and other AT&T network nodes was damaged. There were about 20,000 T1-equivalents out of service. By that measure, it was perhaps the largest telecommunications outage in history.
Many Customer locations were just gone, not only those in the World Trade Center Towers but also those in nearby buildings damaged when the Towers collapsed. When you are trying to restore service to a Customer, you know the Customer’s location. Not this time. As Customers scrambled to recover from their losses, they needed to set up operations in new locations. From hotels in Manhattan to corporate training facilities in New Jersey, Customers were scattering everywhere.
About 20 AT&T people reported to work at the node. Most of them proceeded from there to other network nodes or Customer locations. They all got out alive and unharmed. We had verified that early in the day, save one person we could not find. That evening we found her at her Mom’s home in one of New York City’s outer boroughs. She had walked there. That was the best news and the only good news that day.
At the time, I led the team of about 5,000 people who engineered and operated AT&T’s local network in about 100 metro areas all over the US. We formed a team of everyone we needed to re-build the network and restore service in Manhattan as quickly as possible. Within days, that team numbered almost 2,000 people.
I immediately put one of my direct reports, Jerry K, in charge of running the rest of AT&T’s local network. He and the rest of the team all over the United States did a magnificent job. Thanks to them, there was never an issue that distracted us from Manhattan. I am so proud of them … and so thankful for what they did, maintaining the AT&T’s “gold standard” for service, despite being woefully under-staffed.
All my other direct reports and I focused exclusively on Manhattan.
Vignette: Stuck in Miami.
I was in Miami on that Tuesday to give a “speech” to hundreds of AT&T sales leaders from all over the world. I was in my room going over my talk one more time when my boss, Frank I, called. Turn on CNN he said. I watched the 2nd plane crash into the World Trade Center Towers.
I called home. Everyone was OK, but I could hear military jets over our home in New Jersey, only about 25 miles from Wall Street. It was scary; no one knew if the attack was over.
The sales conference ended. We were all stuck in Miami because air travel in the US stopped. My sales colleagues rented cars and even chartered a bus to get back to their home locations to help their teams and their Customers. In the most extreme case, one group drove non-stop to California.
Vignette: Permission to Fly.
I had traveled to Miami on an AT&T corporate jet. Rather than being out of touch in a car, I decided to stay in Miami until our plane could fly. I spent the next 48 hours in my hotel room on conference calls with my team assessing the damage and organizing and starting our restoration efforts. As Ron S said, damage assessment didn’t take long; our network node was just gone.
On Thursday morning, I got word that the AT&T jet would be allowed to fly. Our pilot was skeptical; almost all airports were still closed, and no planes were flying.
Apparently, behind the scenes, the feds had granted an exceptional permission to fly because AT&T’s local network guy – me – was stuck in Miami and I was needed on the scene. I was told that there were 5 non-military planes that flew that day in the whole of the US. Our AT&T plane was one of them.
We van’ed from our hotel to the Miami airport. With no car traffic and no jet engines roaring, it was eerily quiet and seemed deserted. We stuffed about 15 of us into the plane and headed for New Jersey.
Normally, the plane used a small airport in Morristown, NJ. But at the beginning of the flight the pilot said we’d have to land at Newark International Airport because Morristown had not yet been authorized to open. When we got to about Philadelphia, the plane headed northwest into central Pennsylvania rather than the normal northeast route to our destination, presumably because of still restricted air space around New York City. After a bit, we turned east for a straight shot into our destination. About then, the pilot told us Morristown had opened and we would be landing there.
We landed and I hopped into an AT&T limo with my usual driver, Richie C, and headed for home … hugged my wife and daughter, dropped off my suitcase … and drove my car to our AT&T local network services HQ at Jamesburg, near Princeton, NJ.
Vignette: 14-hour, 800-mile Road trip.
Mary M, Tony O, and Bob O’B, 3 key engineering leaders based in Atlanta, decided that they wanted in. And I will be forever glad … and grateful … they did.
At 8:00 pm on 9/11 and on their own initiative, they packed their suitcases, left their families, and drove straight through to Jamesburg. 800+ miles. I don’t know their motivation, but I suspect some combination of patriotism and a huge helping of the Spirit of Service.
They spent the next 2 months away from home, working 7 days a week and literally every waking hour. Doing whatever needed to be done to speed rebuilding the network and restoring service.
Mary M hails from northern Louisiana. She has the most soothing and aristocratic Southern accent and the most calming demeanor. And is as hard as nails … in the nicest way … when she is getting stuff done. She created and maintained a massive excel spreadsheet of all the Customer circuits that needed to be installed, the status of key dependencies (like installing a new switching or optical transmission system), and progress on all the steps required to activate service. Twice a day, every day, Mary hosted an hours long conference call with all the key leaders to review progress on each circuit and who needed to do what and by when. Roadblocks were identified and small teams where chartered to go remove the roadblock and report back at the next call.
Tony O and Bob O’B were general purpose troubleshooters. Simply said, seeing a hard problem, grabbing hold of it, and solving it. I remember Bob pulling an all-nighter to work with a technician to fight their way through the installation of one … one! … really pesky circuit.
Vignette: 2 degrees of separation.
We lived in Monmouth County, NJ. It’s the northern-most county on the Jersey Shore, separated from Manhattan by a few tens-of-miles across the Raritan Bay and the New York Harbor. It then had a population of about 600,000. Lots of people worked in New York City. Our local paper was the Asbury Park Press. It had a full-page showing pictures of almost 100 Monmouth County residents who were killed in the attack. Killed.
It seemed everyone had a story of a near-miss. My AT&T colleague whose daughter was on the PATH train heading from NJ to the World Trade Center terminal. Her train stopped in the tunnel under the Hudson River and she’s alive. Marty was our regular waiter at one of our favorite restaurants. His son had recently changed jobs. His former job was in the World Trade Center. On the morning of 9/11, he was scheduled to meet with his successor to transition his work. He was 15 minutes late for the meeting. He lived. And on and on.
Vignette: Why don’t you have a twitch?
One technician was scheduled to work in the node at the WTC on 9/11. A colleague, on his way to Customer location, saw the 1st plane crash into the first Tower and called him and told him to get out. Note that an AT&T colleague, watching planes crash into the Towers, thought of his colleague, and cared enough to stop in his tracks on his way to safety and warned him. Instead of taking the stairs to get out of the building, the tech took the elevator. Had he taken the stairs, he would have exited the Tower right into the path of falling debris. Instead, he exited the building on a relatively safer side.
When I met him on Saturday, he told me he had been at the World Trade Center node when the 1993 bombing happened, too. Christ, I said, why don’t you have a nervous twitch.
Vignette: One Police Plaza – Thousands of People Were Loved Ones, not Numbers.
On Saturday morning, a few of us went to Ground Zero. We met in Jersey City at a vacant lot now filled with AT&T’s Disaster Recovery trailers, which, by the way, were a very tangible and expensive demonstration of AT&T’s commitment to the Customers, reliability, resilience, and preparedness for “black swan” events. But I digress.
We had 2 AT&T Suburbans and were escorted by a NY City Police car and a NJ State Police car. I have no idea how that happened. Like many things over the next few weeks, some unknown AT&T person had simply figured out what needed to get done and did it.
Neither of the cops were familiar with Manhattan. The NY police officer was from an outer borough and the NJ state trooper was from southern Jersey. So, we put a NY-savvy AT&T guy with each cop to navigate them around lower Manhattan.
And off we went, with light racks flashing, into the Holland Tunnel.
It was a fitting start to a surreal day. Our little 4-car caravan was alone in the Tunnel. Normally, it would have been full of traffic, even on a week-end morning.
Note: There are 3 only ways to get to Manhattan from Jersey, the southern-most Holland Tunnel at Canal Street and closest to Wall Street, the Lincoln Tunnel at about 34th Street in midtown, and the northern-most George Washington Bridge at about 181st Street. The Holland Tunnel was still closed to all traffic. I don’t remember the status of the Lincoln Tunnel or the GW Bridge.
Note: I’m not disclosing specific AT&T locations in this post for security reasons. They are critical infrastructure. Critical with a capital C. Something like 12% of all the traffic in the entire AT&T global network used to pass through one of these places.
When we came out of the Tunnel, we were in what struck me as a war zone. The only vehicular traffic was emergency and constructions trucks. There were roadblocks and cops at every intersection. The only people you saw were cops and firefighters and EMTs, construction and utility company workers, and people in scrubs. There were big fat telecommunications cables laid in the street gutters.
We went to the roof of the high rise “12% node.” We could look right down into the still smoking hole at Ground Zero, the Pit. I can’t adequately describe the scale of destruction. The pile of rubble must have been several stories high; someone remembered 10 stories. At least one tall thin skyscraper near the Towers was visibly leaning. A nearby high-rise Verizon building had holes in its walls. You could see network equipment hanging by its cabling, with the floor under it gone. But unlike my node, it was still there.
We went to our AT&T local network services HQ in NY and got updated by the on-the-ground team and Bill B, our battlefield commander. We had one question: What do you need? We had one answer: You’ve got it.
We then went to One Police Plaza, the NY City Police HQ. We pulled up to the front door. Out of the corner of my eye I noticed a long line of people, 5 or 6 deep, waiting to enter the building.
We met with police brass to make sure they were getting everything they needed. They were and they thanked us.
We then walked out the front door and turned left and walked down the half the length of the building, fully half a city block. I’m now walking by all those people lined up. They aren’t talking. They all seem to have papers in their hands. It then dawns on me that they are there to report a missing person. When you hear that thousands of people have been killed, it’s a number, an abstraction. Even when you try to relate it to something familiar, like 10 times my high school graduating class, my mind … or my heart … couldn’t grasp it. When you look at the faces of hundreds or people who lost a loved one, you get it.
We turned the corner into a parking lot next to One Police Plaza. There sits a small AT&T van with a satellite dish on top and two AT&T people, part of the AT&T Disaster Recovery team. A cable is running from the van into a 3rd or 4th story window. That van and that cable were how the police were getting telephone service.
Vignette: I’m Alive.
We had a #5ESS switching system dedicated to one Customer located in their World Financial Center location, adjacent to the Towers. The building was badly damaged and had been evacuated. Commercial power was off, and the air conditioning wasn’t working. But on Tuesday evening 9/11, our switch was still running on battery back-up … and cooking itself to death in its own heat.
Our AT&T NY field team wanted to go to the building and gracefully power off the switch to “save” this multi-million-dollar asset. Because the cables to the building were cut, we had no telemetry to the switch and couldn’t enter the necessary power-down commands remotely.
I was on a call with the team on Tuesday evening when they explained the situation and asked permission to go to the #5ESS. I was really, really concerned with their safety. I told them they infinitely were more important to me and to AT&T than the switch. But they were insistent. I said OK with one condition: they had to promise me that if they encountered any danger to their personal safety, they would abort. They did and off they went.
In route on foot, they encountered a couple of FDNY firefighters who agreed to allow them access to the building and escorted them. When they got to the building, it was so damaged that the doors frames had been bent askew and the doors were jammed shut. The firefighters used their Halligan bars to pry open the doors. (A Halligan bar, a long, heavy-duty crowbar, was invented by FDNY First Deputy Chief Halligan in 1948.) The wall thermometer in the equipment room was pegged at its maximum, 130 F. Equipment rooms are usually around 70.
When they were finished powering down equipment and exited the building, it’s nightfall, probably 14 hours after the Towers collapsed. The firefighters noticed our AT&T team had satellite phones, the only phones that worked. They asked if they could use them to call home. Of course, yes. Our team overheard the firefighters telling their loved ones, their spouses and kids, their mothers and fathers … for the first time that day … that, unlike hundreds of their colleagues, they were still alive. When I say my telecommunications industry colleagues provide a critical service, these calls put a very, very human face on critical.
Vignette: The good … no, make that great … lawyer.
We found a vacant lot in Jersey City, just across the Hudson River from lower Manhattan. It was close to our fiber route into Manhattan. You don’t find many big vacant lots in Jersey City. It was an ideal place to put our disaster recovery trailers. There are about 100 of these. They contain enough equipment to replace the AT&T network in a large city. The equipment is operational, ready to provide service. There are several hundred volunteers from all over the country who know how to operate it, regularly train on disaster recovery, and practice several times a year.
There was one wrinkle. We had found the owner of the property and approached him with a very generous, short-term rent proposal. He was recalcitrant, apparently afraid we’d be there forever, delaying his sale or lease of the property to someone who wanted to build on it. Us operational types failed to budge him. I asked an AT&T lawyer to work on it. AT&T had wonderful lawyers: big IQs and impressive credentials from the best schools, expert at the law and regulations, and street-smart negotiators. Our guy explained how attractive our offer was … how’d we would iron-clad guarantee to vacate the property in a couple of months with exorbitant financial penalties if we didn’t … and how bad he’d look on New York TV if we publicized how he was delaying recovery from the 9/11 attack. The next day, the trailers rolled in.
Vignette: Cheered on the West Side Highway.
John D is a real-life Radar O’Reilly, Message-to-Garcia guy. He is one of those people who can get anything done.
Our people needed little red wallet cards issued by New York City to gain access to Ground Zero and its surround. I still have mine; it reads: Mayor’s Office of Emergency Management … World Trade Center Emergency … City of New York … Authorized A12209.
Once we understood the need to get cards, how do we get them? John somehow found and jumped on an AT&T truck carrying replacement equipment into lower Manhattan. It went south on the West Side Highway which was lined with hundreds of New Yorkers cheering the passing trucks carrying restoration people and material.
Somehow again, John found out that the cards were being issued at the temporary morgue in lower Manhattan. He went there and got several hundred cards for AT&T. He personally kept the log of each card’s number and who it had been assigned to. (Another company didn’t keep records and all the cards initially issued, including AT&T’s, were voided and new ones issued. So, John got to repeat the miracle.)
Vignette: Going to the Mattresses in Staten Island.
The AT&T local network services unit had a major engineering and customer support office on Staten Island. Among other things, it was there that engineers figured out how many “trunks” and optical transmission systems were needed to interconnect switching systems, designed them, and project managed their implementation. With all the switching systems lost in the attack, we had to install and put in-service a huge number of new trunks and optical transmission systems between new switches. We were, in effect, building a whole new local network … and a big one … in weeks rather than the 6-12 months normally required for a single major network addition.
People were working around the clock. Everyone worked until too tired to work anymore, day-in and day-out for most of 2 months. I never heard one complaint.
At the leadership level, we put in place a buddy system. My buddy was Ron S. He’d keep an eye on me … and I’d keep an eye on him. We’d look for, shall we say, exhaustion-induced losses in mental acuity. In other words, was our buddy acting wacky. If we saw that, we’d force our buddy to go get some sleep. We also kept out buddy up to date on what we were working on. At least one of each pair of buddies was always working so we always had leadership continuity.
Back to the story. Those of you familiar with life in the New York metro will understand that most people routinely commute 1-2 hours each way. So, getting home for some sleep was problematic. Enter the practical problem solver, Greg S. He commandeered the biggest conference room at Staten Island and filled it with beds. When an engineer was so exhausted that they just couldn’t work any longer or risk a commute home, they’d go to the conference room for a nap on a brand-new mattress.
I never heard what became of the beds. Knowing AT&T people, I bet they were donated to a charity or a needy family.
Vignette: The Opening Bell.
One of the early goals after 9/11 was to re-open the New York Stock Exchange as quickly as possible. There were at least 2 reasons: get the global financial markets going again and demonstrate to the world our ability to “shrug off” an attack. The Exchange re-opened on Monday, 9/17. That’s 5 days to restore service to the Exchange and hundreds of back-offices in the financial industry. At the time and to this day, it seems impossible to have done that. But did it we did.
I must recognize John F who worked around the clock at AT&T’s Bedminster, NJ location to get the long distance network part of that done. A bearded PhD in electrical engineering and alumni of Bell Labs who had transitioned to engineering and operations, John is a “one riot, one ranger” guy.
I still remember being in a conference room in Jamesburg doing Mary M’s twice-daily conference call. We paused the call and watched CNN when they rang the bell and the Exchange opened.
According to a retrospective by Bob Pisani of CNBC,
“ … That day, I was there early, standing outside the NYSE with Chairman Richard Grasso as Treasury Secretary Paul O’Neill drove up to help ring the opening bell with firefighters and police officers. They were led by NYC Police Commissioner Bernie Kerik and Fire Department Commissioner Thomas Von Essen, along with Sens. Hillary Rodham Clinton and Charles Schumer, Mayor Rudy Giuliani, New York Gov. George Pataki and SEC Chairman Harvey Pitt …
“ … Before the open, two minutes of silence were observed. Members of the New York City police and fire departments, representing the heroes of 9/11, had come on the floor. The entire room sang “God Bless America,” and then the police and firefighters, along with the dignitaries, rang the opening bell …”
Throughout the Jamesburg building and all around the country at our satellite locations … Denver, Staten Island, Manhattan …. people cheered. We were overwhelmed with pride in our Nation, our industry, our company, our colleagues, and ourselves … relief in having achieved the impossible … and the persistent sadness from the tragedy.
Vignette: Celebrations, Booking the Learnings, & Reunions.
Earlier in his career, Mike K., our PR guy, had been a producer for PBS in Boston. Very soon after 9/11, I knew we were doing something momentous. I asked Mike to capture the people in the restoration effort in a video. I don’t know how many people he interviewed. I just re-watched the video and counted 37 who described their recollections. Mike produced a 21-minute masterpiece. We gave copies to everyone we could think of so they would have a memento of their incredible efforts. (For some reason, the lawyers considered it proprietary and counseled against making it public. I wish I had pushed back; it would have been great marketing for AT&T. Our disaster recovery capability, our resources, our know how … and, most important, our people’s extraordinary and very personal commitment to Customers and our Nation.)
I’m a big believer in celebrating successes. What you choose to celebrate demonstrates what’s important: your values and your results. And celebrations are long remembered. I encouraged everyone to host some kind of celebration for their teams. In early December, I hosted a celebratory dinner for about 50 people from all over the country. It just had to be in Manhattan. We splurged and booked a private room at the Rainbow Room on the 65th floor of 30 Rockefeller Center. First class venue for first class people. Spectacular views of the Manhattan skyline on a crisp, clear Saturday night. Each person got a plaque signed by Rudy Giuliani, the then highly regarded mayor of New York City, and my boss, Frank I, the wonderful and inspiring and visionary guy in charge of AT&T’s network. (In my admittedly biased opinion, the AT&T network and Bell Labs were AT&T’s crown jewels). We had each person photographed getting their plaque from Frank. Credit Mandy Z and Ron S for organizing and executing a simply perfect evening. How in the world did they get Giuliani’s signature?
Getting back to business, we convened a full-day meeting with about 100 people to “book the learnings” from the restoration effort. What did we do right? What could we have done better? What new capabilities were called for if, God forbid, something like this happened again? What process changes were needed? What best practices could we import into normal day-to-day operations? For each learning, who was going to do what by when to make us better, stronger?
This kind of disciplined after-action introspection was SOP at AT&T. It’s how we got better and better. (I now diverge with an unrelated anecdote. One Friday afternoon, we had a cable-cut in a marshy, very hard to get to right-of-way along the Intracoastal Waterway in North Carolina, isolating parts of South Carolina, including Myrtle Beach. It was a bitch to locate and a bitch to get to. But finally, we did. We needed an about 1,000-foot-long piece of fiber optic cable to bypass the damaged cable. We dispatched a technician to retrieve a reel of cable. The reel was in our inventory system and marked as having more than required length of cable. But the marking and inventory system were wrong; it was too short by a lot. So, we had to retrieve another reel, costing our Customers more downtime. This is all playing out Saturday afternoon. Mary Ann C., a mathematician by education, led all of AT&T’s long distance network field people. I’m surprised on Sunday afternoon to learn that she has already had her team test and verify the length of all cable reels in the entire nation. That’s a continuous improvement culture.)
I organize a small reunion dinner every 5 years for some of the key leaders of AT&T’s 9/11 restoration. Our last one was in 2021 and we’ll do another in 2026. We always have it at Il Tinello on West 56th Street in Manhattan, a wonderful place that Mary M, Bob S, and I had all independently discovered in our pre-9/11 personal visits to Manhattan. We have visited the 9/11 memorial and museum in the afternoon before the dinner. Life has scattered our “Band of Brothers … and Sisters” all over the country: CT, GA, MA, MD, NC, PA, TN and, of course, NJ and NY. Some are retired, some working elsewhere, and some still at AT&T. Sadly, but I guess inevitably, a couple of passed. We catch-up and reminisce and glow in that special bond we share.
I think I’ll end this on remembering that glow and that bond with Bob C, Tony F, Tony G, Ricky H, Frank I, George I, Courtney L, Kevin L, Mary M, Kim M, Anthony M, Bob O’B, Tony O, Ron S … and thousands of AT&T’s Angus MacDonald’s.
Discover more from Reed Harrison's
Subscribe to get the latest posts sent to your email.
Thank you for this. During one of our conversations, you had mentioned that you were heavily involved in the restoration of AT&T services directly after 9/11. We didn’t have time then to discuss the full story but I am very glad to have finally heard the details. Hope you are doing well.
Thx, Joe. All good here. Hope same for you all.