September 7, 2026

Long tenure in IT leadership means you accumulate stories. Some are victory laps; most are technological horror stories. The ones that stick with you aren’t the dramatic zero-day ransomware attacks—they’re the quiet, grinding tragedies of institutional amnesia.

Some years ago, I walked into an IT environment that had been stewarded by a predecessor who was—bless his heart—completely out of his depth.

When you do forensic archaeology on an inherited network and ask basic questions like, “Why are we using this vendor?”or “Why did we buy this firewall?”, the answer from the staff was invariably a variation of:

“Because that’s what the guy before him bought.”

There was no paper trail, no feature comparison matrices, no risk assessments. Just a pure, unadulterated application of the IBM Defense: Nobody ever gets fired for buying the same stuff you bought last contract cycle.

Rather than risk disturbing the delicate equilibrium of a system he didn’t understand, the former leader maintained a state of permanent stagnation. The network ran “okay.” Worse, the junior IT staff were all fairly new (2 years for most), and had been trained into a state of learned helplessness. They were treating $4,000 enterprise-grade managed switches like $20 unmanaged desktop hubs from Best Buy. Instead of spending two minutes reassigning a VLAN port on a switch right in front of them, there were 60-foot CAT5e cables across the cable ladders to plug into a port they knew worked.

Whenever anything went sideways, they called a high-priced external consultant—for basic tasks they should have been doing in-house. It was a classic “blind men describing an elephant” scenario. Everyone guarded their own microscopic slice of the infrastructure, completely blind to the macro-topology.

Hardware of Questionable Provenance

When we began mapping the physical layer—a sprawling, Frankenstein-monster collection of fiber backbones and stacked switches—we started cataloging serial numbers.

A few switches were legitimate, under-warranty iron. But a significant chunk of the core network consisted of hardware with “questionable provenance.” They were grey-market ghost switches, likely purchased off-market from overseas brokers to shave a few bucks off a capital budget years prior. No support contracts. No firmware updates. Zero vendor accountability.

Meanwhile, we were staring down the barrel of a massive new building project that required integrating our legacy infrastructure with a brand-new facility.

The building specs were already locked in, yet no one could explain the architectural decisions. In the mid-2020s, the blueprints were still specifying copper runs through underground conduit between the old and new structures. Why? To feed an old, hybrid “Mullet” phone system: VoIP in the front, legacy copper in the back.

I called the original network consulting firm and told them: “Bring your laptops, bring your discovery tools, trace every port, and draw me an absolute, definitive map of this network. Take as much billable time as you need.”

The Forensic Audit

While the consultants spent days drawing vague boxes on our whiteboards, my SysAdmin and I started doing real detective work.

A year prior, the org had bought two modern firewalls configured in failover mode. But the setup had been a completely blind rule-import from an ancient, legacy box—carrying over years of dead, security-hole-riddled policies. To make matters worse, the original consultant — the same consultant who was now tracing ports and cables — had since dropped that firewall brand. Their solution to our configuration issues? “Well, if you rip those out and buy Product X through us, everything will work much better.”

Classic vendor extraction: Sell the client a box, abandon the platform when margins drop, and charge them to migrate to whatever earns a higher commission this quarter.

We fired up our own diagnostics, found a specialized consultant who actually knew the platform, and began stripping out the legacy rule junk. We expanded into a unified ecosystem from a manufacturer that offered integrated managed switches and a Managed Detection and Response (MDR) SOC. For an IT department without the headcount to run a 24/7 internal security operations center, an MDR was a game-changer.

We replaced the grey-market Philippine switches. And it was good.

My office whiteboard became a living crime scene diagram. Working from the firewall outward, we traced, erased, re-traced, and uncovered the true state of the network:

• The Phantom Voice Network: An entire phone VLAN had been provisioned years ago, but never really used. VoIP phones were dumping raw, unprioritized voice traffic across the primary data network.
• The Facilities Shadow IT: Facilities had quietly deployed a massive, unsegmented and undocumented fleet of IoT hardware—security cameras, door access panels, smart irrigation, and wireless thermostatic sensors inside vaccine freezers—all living on the open corporate LAN.

We carved out dedicated VLANs for Facilities and Voice (among others), isolating traffic and insulating the core network from insecure IoT endpoints.

The Big Reveal

A week later, the original high-priced consulting team returned and proudly handed over their final, gold-plated network map.

It was flat-out wrong. Not just slightly inaccurate—it depicted a fantasy network that bore almost no resemblance to physical reality.

I don’t entirely blame the techs. They were trying to map a decade of undocumented digital drift. But it highlighted the fundamental grift of the relationship: they had made a very comfortable living off us by answering questions we didn’t know how to ask, while leaving us with zero institutional knowledge.

I terminated their contract.

Armed with our own accurate map, we tackled the physical layer. We ripped out the tangled, six-foot “spaghetti” patch cables choking the server racks and replaced them with clean 1-foot and 2-foot runs. Suddenly, you could actually see the physical topology without playing operations-roulette every time you pulled a cable.

For the new building expansion, we planned to leverage proper 802.1Q trunking over fiber to publish our newly isolated VLANs (Voice, Data, Facilities, Management) across both sites. When getting close, we’d separate our redundant firewalls physically—placing one in each building with independent network ingresses and distinct gateways, while maintaining automated cross-building failover over fiber.

The Takeaway

As a CIO, your primary job isn’t merely keeping the lights on or buying shiny new boxes. Your job is corporate risk management and institutional sustainability.

That means building a culture where team members understand the system deeply, grow their skills, and actually enjoy coming to work. Retaining the institutional memory.

In the years since, I’ve audited dozens of organizations with variations of this same nightmare. The common threads are never technical limits—they are organizational failures:

1. Zero Succession Planning: Leaving the keys to the kingdom in a single person’s head.
2. No “IT Time Capsule”: Failing to maintain living, mandatory documentation that survives employee turnover.
3. Outsourced Understanding: Allowing vendors to hold the map to your infrastructure, leaving you paying top dollar to navigate your own house.

If your team doesn’t know how the pieces of the elephant fit together, you don’t have an IT infrastructure—you just have an expensive disaster waiting for a trigger.