12 ways that software engineering is like bridgebuilding

I just finished reading “The Great Bridge: The Epic Story of the Building of the Brooklyn Bridge” by David McCullough. It’s a long, intriguing story that it inspired parts of my new whitepaper about Netflix OSS. My buddy James liked my linkage between bridgebuilding and software engineering.

https://twitter.com/jetpack/status/781919565831954432

To be sure, software teams rarely take fourteen years to deliver a final product and the stakes may be different. But there are some fundamental lessons learned from the construction of the Brooklyn Bridge that apply to us in the software business.

Lesson 1 – You need a strong lead engineer

The original architect of the Brooklyn Bridge was John Roebling. He died before construction started, and his son Washington took over as chief engineer. Washington was also considered an expert in the field, and his experience and attention to detail established him as the authority on the project. However, he also brought on talented lieutenants who managed many of the day-to-day activities. Roebling was extremely hands on and stayed actively engaged, even when he became home-bound due to illness.

On software projects, you also need an accomplished, hands-on engineering leader. I’m all for self-organizing teams, but you also expect that someone rises to the top and helps direct the effort. At the same time, there should be many leaders on a team who can step in for the lead engineer or own sub-activities without any oversight.

Lesson 2- Seek out best practices from your peers

I started off my whitepaper with an anecdote about the lack of knowledge sharing among bridgebuilders. Back then, engineers weren’t inclined to share their best practices and the field was quite competitive. However, Roebling did go to Europe for a year to learn about the latest advances in bridgebuilding, and found a number of willing collaborators. He brought that knowledge home and applied it in ways that were uncommon in the U.S.

Likewise, software teams can’t operate in a vacuum. Now more than ever, there are significant advances in software architecture patterns and technologies, and we need to seek out our peers and learn from their experience. Go to conferences, read blogs, follow people on Twitter. If you only source ideas from in-house folks, you’re in trouble.

Lesson 3 – Visible progress builds momentum

There were MANY parties who objected to the construction of the Brooklyn Bridge. Chiefly, the existing water-based shipping and transportation industries that stood to get disrupted by this hulking bridge crossing the river. People wouldn’t need ferries to get to Brooklyn, and some tall ships might not be able to make it under the bridge. Once the massive bridge towers started taking shape, momentum for the bridge increased. People felt a sense of ownership of this magnificent structure and it was harder to claim that the bridge would be a failure or wasn’t worth doing.

Software can be disruptive. Not just to competitors, but within the company itself. Antibodies within a company swarm to disruptive changes and try to put a halt to them because it upends their comfortable status quo. Software teams can’t get bogged down in lengthy design periods, as the apparent lack of progress threatens to derail their effort. Ship early and often to build momentum and create the perception of unstoppable progress.

Lesson 4 – Respect the physical dangers of your work

As you can imagine, building a bridge is dangerous. Especially 120+ years ago. During construction of the Brooklyn Bridge, a number of people were killed or injured. Some were killed when they ignored safety protocols, and many were injured due to accidents. And dozens experienced temporary, partial paralysis due to “caisson disease” because no one understood the impact of working deep underwater.

Building software has its own dangers. While I doubt you face risks of multi-ton blocks of granite falling on you—unless you work in a peculiar open-office design—there are real health risks associated with modern software development. Listen to John Willis talk about burnout. It’s sobering. As a software industry, we often, stupidly, celebrate how many hours a team worked or how little sleep they got to deliver a project. Putting people in a high-stress environment for weeks or months on end is mentally AND physically dangerous. Recognize that, and take care of yourself and your team!

Lesson 5 – Adapt to changing conditions and new information

You might think that bridgebuilding is all about upfront design. Traditional waterfall projects. However, while the Brooklyn Bridge started out with a series of extremely detailed schematics, the engineers CONSTANTLY adjusted their plan based on the latest information. When the team started digging the New York side of the bridge base, they had no idea how deep they’d have to go. That answer would impact a cascading series of dependencies that had to be adjusted on the fly. The engineers didn’t believe they could factor everything in up front, and instead, stayed closely in tune to the construction process so that they could continually adjust.

Software engineering is the same. There’s a reason why many developers detest waterfall project delivery! Agile isn’t perfect, but at least it’s realistic. Teams have to deliver in small batches and learn along the way. It’s remarkably difficult to design something upfront and expect that NOTHING will change during the subsequent construction.

Lesson 6 – It’s critical to do testing along the way

In reading this book, I was struck by how much math was involved in building a bridge. The engineers constantly calculated every dimension—water pressure, wire strength, tension, and much more—and then re-calculated once pieces went into place. They did rigorous testing of the wire provided by suppliers to make sure that their assumptions were correct.

Software teams are no different. Whether or not you think test-driven development works, I doubt anyone doubts the value of rigorous (automated) testing. That’s one reason I think continuous integration is awesome: test constantly and don’t assume that these pieces will magically fit together perfectly at some future date.

Lesson 7 – Build for unexpected use cases

Roebling didn’t take any chances. He wanted to build a bridge that would last for hundreds of years, and therefore made sure that his bridge was much stronger than it had to be. At nearly every stage, he added cushion to account for the unexpected. When building the caisson that protected the workers as they dug the bridge foundation, Roebling designed it to withstand a remarkable amount of pressure. Same goes for the wire that holds the bridge up. He built it to be 5-10x stronger than necessary. That’s one reason that the introduction of the automobile didn’t require any changes to the bridge. In fact, there were NO major changes needed for the Brooklyn Bridge for fifty years. That’s impressive.

With microservices, we see this principle apply more and more. We don’t exactly know how everything will get used, and tight coupling gets us in trouble. While we need to avoid over-engineering in an attempt to future-proof our services, it is important to recognize that your software will likely be used in ways you didn’t anticipate.

Lesson 8 – Don’t settle for the cheapest material and contractors

“The lowest bidder.” Those are three scary words when you actually care about quality. With the Brooklyn Bridge, the board of directors went against Roebling’s strong advice and chose a wire manufacturer that was the lowest bidder, but led by a suspect individual. This manufacturer ended up committing fraud by exchanging accepted wire for rejected wire because they couldn’t manufacture enough quality material.

When building software that matters, you better use tech that works, regardless of price. Now, with the prevalence of excellent open source software, it’s not always the software cost itself that come into play. But, who builds your software, and who supports it, matters a ton and shouldn’t go to whoever does it the cheapest. Smart companies invest in their people and splurge when necessary to get software they can trust.

Lesson 9 – Invest in your foundation

When building a bridge, the base matters. During the age when Roebling was learning about and building bridges, there were many bridge that collapsed due to poor design. That’s kinda terrifying. Roebling was laser-focused on building a strong foundation for his bridge, and as mentioned above, built his towers to an unheard-of strength.

The foundation of your software is going to directly decide how successful you are. What is it running on? A homegrown hodge-podge of tech that you’re scared to reboot? A ten year old operating system that’s no longer supported? Stop that. Sure, I think Pivotal Cloud Foundry is just about the best foundation you can have, but I don’t care what you choose. Your software foundation should be rock-solid, reliable, and a little boring.

Lesson 10 – Accept that politics are necessary

As you can imagine, embarking on one of the most expensive infrastructure projects in the history of the world required some strange bedfellows. In the case of the Brooklyn Bridge, it meant getting the support of the shady Boss Tweed. Likewise, Roebling and team had to constantly answer to a host of politicians who used the bridge as part of the political platforms.

Software projects aren’t immune from politics. New projects may disrupt the status quo and make enemies within impacted organizations. Unexpected costs could require broad support from financial stakeholders. I’ve learned the hard way that you need a wide set of high-ranking supporters to have your back when projects or products hit a wall. Don’t take an “us-against-the-world” approach if you want to win.

Lesson 11 – Be transparent on progress, but not TOO transparent

Given the costs of the bridge, it’s easy to see why there was a lot of oversight and scrutiny. The engineers were constantly paraded in front of the board of directors to answer questions and share their progress. In some cases, they kept certain internal engineering debates private because they didn’t want non-technical stakeholders freaking out. It was important to openly share major issues, but airing ALL of the challenges would have led to clueless politicians blowing minor issues out of proportion.

This is the same case when building software. I’m sure we’ve all had to roll-up our progress in a series of status reports to the various stakeholders. And even within our software teams, we’ve shared our progress at a morning standup. It’s super important to be honest in these settings. But, there are also issues that are best resolved within the team and not carelessly shared with those that cannot understand the implications. It’s a fine line, but one that good teams know how to straddle.

Lesson 12 – Celebrate success!

When the bridge was finally done, there was a massive party. The President of the United States, along with a whole host of luminaries, led a parade across the Brooklyn Bridge and partied for hours afterwards. The engineers were celebrated, and those that had poured years of their life into this project felt a profound sense of pride.

Software teams have to do this as well. With our modern continuous delivery mindset, there aren’t as many major release milestones to rally around. However, it’d be a mistake to stop the tradition of pausing to reflect on progress, celebrate the accomplishment, and recognize those that worked hard.