Back to News
Advertisement
Advertisement

⚡ Community Insights

Discussion Sentiment

41% Positive

Analyzed from 1561 words in the discussion.

Trending Topics

#fiber#backup#faa#same#atc#going#system#paths#https#don

Discussion (96 Comments)Read Original on HackerNews

chasd00•about 2 hours ago
It happens. There use to be a joke during the first big DC build out phase that went like if you're ever going into the wilderness take a 1ft length of fiber optic cable with you. If you get lost bury it and a back hoe operator will appear and sever it within an hour. You can get a ride back with them.
john_strinlai•about 1 hour ago
reminds me of https://www.cbc.ca/news/canada/saskatchewan/stranded-man-cut...

>A man stranded in the bush in northern Saskatchewan was rescued last week after chopping down four power poles — knocking out electricity to surrounding communities. [...]

>But he had an axe and he knew SaskPower would have to check the downed line, so he went to work.

Onavo•about 2 hours ago
I think it happened many times to electrical lines due to lost campers.
cube00•about 4 hours ago
> When it went to flip into the backup, we discovered that the backup fiber had a break

Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.

I wonder how long it was down? Days, weeks, months?

toast0•about 4 hours ago
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
koolba•about 4 hours ago
Put all your backups in one basket, and then pray that nobody crushes the basket.
oasisbob•about 3 hours ago
This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.
zoomablemind•about 3 hours ago
That's very uncharacteristic, especially given all those contingency requirements (backup policies, failovers, testing schedules etc.) imposed on corporations/companies in the wake of 9/11.
krashidov•about 3 hours ago
Could this be sabotage?
kqgnkqgn•about 3 hours ago
That seems more than merely plausible given current tensions and the location/timing connection with NYC and UN.
readthenotes1•about 4 hours ago
A lot of people don't check their backups until they need to restore.

A lot of people are incompetent.

thewebguyd•about 4 hours ago
A backup without a restore test isn't a backup at all
kqgnkqgn•about 3 hours ago
Building multiple diverse paths and monitoring for fiber cuts is not hard. Having 2 fiber paths is insufficient for even only moderately important workloads at a tech company. Overlapping fiber cuts happen. For something with significant economic and safety impact this is just crazy.

Sometimes the level of incompetence / lack of care in organizations like this astounds me. I understand issues like this can be complicated and systemic but it honestly makes me think very poorly of the technologists building these systems in government.

woodruffw•about 3 hours ago
They had two diverse paths, but apparently did not have a (reliable?) process for ensuring the backup path was functional during fail-over. I imagine there will be some soul-searching over that.

Nothing I've ever seen or experienced with FAA indicates a lack of care; the parsimonious explanation is almost always that people who care a great deal are working within complex systems that don't always have a consistent or externally legible set of priorities. Or more intuitively, the failures we see are the "acceptable" ones versus the unacceptable ones (like planes falling out of the sky).

markdown•about 3 hours ago
> Nothing I've ever seen or experienced with FAA indicates a lack of care

Hard to fathom someone still holding that view after the 737MAX disasters.

The NTSB has repeatedly and publicly criticized the FAA for failing to implement their recommendations, including as recently as last week (Amazon Prime crash resulting in 5 deaths). They've explicitly called out the FAA's failure to act as a contributing factor in crashes, multiple times.

Regulatory capture ruined the FAA.

OutOfHere•about 3 hours ago
The problem is with considering the second path a backup path. It is a common problem with things that are considered to be a backup. In contrast, both paths should ideally have been in regular use. As a general policy, for resiliency, one should always exercise all of one's routes/suppliers/vendors, even if some are suboptimal, not just the primary one.
woodruffw•about 3 hours ago
I completely agree. My comment is not to suggest there wasn't a problem or failure (clearly there was one!), but that we shouldn't assume incompetence versus conflicting priorities under limited resources.
dgellow•about 3 hours ago
I’m pretty sure that type of project is contracted to the private sector and isn’t caused by government technologists in any way…
kqgnkqgn•about 3 hours ago
That sounds pretty plausible. But you still have to have enough internal competency to oversee a contractor, and ask the right questions/build proper requirements and verify they are meeting them.
RyJones•about 2 hours ago
I've told this story so many times I've worn the corners off it. BT was giving a presentation to MSN WAN OPS about the new datacenter buildout in London for us. They kept going on and on about physical security, man traps, and that our fiber left the building on each side and didn't get close to each other for so many km away.

BT guy ends that part of the presentation with "the IRA will really have to get shit together to take you off the net"

This was during the troubles. Same planet, different world.

sc68cal•about 4 hours ago
There is also a new ATC system being rolled out, as early as today

https://www.airwaysmag.com/new-post/faa-smart-first-deployme...

whatever1•about 4 hours ago
I suspect it is vibe coded. Contract won in June. Deploying in September.

Good luck to all of us.

macintux•about 3 hours ago
Deploying first to DC? Yeah, that’ll end well.
librasteve•about 1 hour ago
back in the day, i worked for Energis (#3 wholesale telco in the UK) … we had about 2.2m active subscribers and ran a fibre backbone.

one day the backbone (E1 bidirectional ring iirc) was broken north of london … my boss drove out to find that travellers had entered the field, opened the inspection cover in the ground, stood a pole for their clothes line and filled with cement

very savvy, he didn’t let on that we had about £1m revenue per day close to collapse and convinced them to hang their clothes somewhere else (perhaps a few quid changed hands)

atonse•about 4 hours ago
I don't quite understand this sort of thing happening. Wasn't the whole point of the internet to be a self-healing network where we route around severed cables, etc?

Or is it that these ATC networks are their own air-gapped network with less redundancy? That just doesn't add up. Or maybe there was only one line going to the ATC, with no multiple "ISPs" like a datacenter would have?

bri3d•about 4 hours ago
* Yes, TRACON have their own dedicated links. You can look into ASTERIX and STARS to learn some of the cursed ways the data processing and dataflow work.

* There is supposed to be a primary and a secondary link, in this case the primary failed and the fail-over also failed. It's unclear from the reporting if they were damaged in the same incident or if the failover was not tested or monitored adequately.

The Philadelphia TRACON site has been notoriously unreliable and was supposedly improved in 2025, it's also unclear if these issue actually could stem from that implementation.

wrs•about 4 hours ago
I’ve had enough double-redundant links fail - and I mean pretty thoroughly double-redundant, different provider, different media, different physical path - that I’m quite surprised something like the air traffic control system only has two links.
readthenotes1•about 4 hours ago
Do you want air traffic control to be on the general internet?

That seems like an extremely foolhardy thing to do.

ssl-3•about 3 hours ago
As a tertiary backup: Yeah, maybe I do want that.

It seems like it would present a less-chaotic solution than that provided by having no data communications at all.

kqgnkqgn•about 3 hours ago
You certainly don’t need to run this over public Internet to get resiliency over multiple paths.
cmiles8•about 2 hours ago
Finding out your backup fiber has been cut, aparently some time ago, only when you try to use it as the backup isn’t an issue of “aging infrastructure.” It’s an issue of letting completely incompetent people run your tech.
quickthrowman•about 4 hours ago
It has been conjectured that the easiest way to locate a buried fiber optic cable is to start digging.

(The actual easiest method is to install the conduit or direct burial cable with tracer wire)

dboreham•about 4 hours ago
Local people know where they are because once upon a time a D8 drove through pulling a big orange spool of fiber conduit and passers by chatted about how another fiber was going in. Also around here the contractors seeded the disturbed ground with a wildflower mix (presumably to discourage weeds) so you can also look for straps of wildflowers.

And of course there are marker posts that say "buried fiber optic cable do not dig".

dctoedt•about 3 hours ago
Not to be conspiratorial, but about an hour ago I saw this in The Economist: "Russia’s grey-zone attacks on Europe are growing more brazen"

[QUOTE:]

Russia has been staging covert operations and deniable provocations against European countries for many years. Since its full-scale invasion of Ukraine began in 2022 they have increased. These have included incursions by reconnaissance drones, hacking, arson, parcel bombs targeting cargo flights and plots to kill executives of leading European arms firms. In recent months the pace of such shadowy attacks appears to have risen again. The Russian operations have three main goals, according to analysts and security officials: trying to coerce European countries into halting their support for Ukraine; raising the costs of providing that support; and attempting to make NATO look powerless.

[/QUOTE]

And of course there've been "suspicious" cable cuts in the Baltic Sea and other European waters ....

https://www.economist.com/europe/2026/09/21/russias-grey-zon...

tomasphan•about 4 hours ago
ATC seems like a great place to have redundancy with a starlink dish on the roof.
peri-cl•about 3 hours ago
I think this the same system that was the topic of a major contractor dispute a few months ago, between Verizon (incumbent fiber contractor) and SpaceX (lobbying to replace it).

https://news.ycombinator.com/item?id=43205435 ("Starlink to take over $2.4B contract to overhaul air traffic control comms (theverge.com)")

Johnny555•about 2 hours ago
I thought Musk got an FAA telecom contract for Starlink back when he was effectively running the government (or at least running it into the ground)?

https://www.cnn.com/2025/02/25/business/musk-faa-starlink-co...

Advertisement
adamm255•about 3 hours ago
Well.. ATC in the north of the Uk was down today as well. Fun.
cyanydeez•about 3 hours ago
So of course if the FAA is going to put AI into the watchtowers, they're obviously going to use local AI and not rely on some random Amazon cloud infra in some middle eastern country.

right. of course, no ones going to cut corners on the safe use of AI and secure infrastructure in this administration.

cynicalpeace•about 1 hour ago
If/when civilization collapses, flights will be the first thing to go

Just sayin

basisword•about 2 hours ago
The UK ATC system was down again today for the second time this month. I'm no conspiracy theorist but hard to believe those two + this are 'accidents'. I believe the Netherlands has also had several issues recently.
bell-cot•about 2 hours ago
Wasn't the UK ATC system the one where the whole thing core dumped, followed by the backup ATC system core dumping, when someone filed a flight plan with confusing same-name waypoints?

With low-enough engineering quality, you don't actually need saboteurs.

basisword•about 2 hours ago
>> With low-enough engineering quality, you don't actually need saboteurs

While I agree, that’s just what they told us. Not sure I believe it anymore. The fact the first UK one happened immediately after they sanctioned Israel is also just a big coincidence.

dboreham•about 4 hours ago
So much for all that work to ensure path diversity. Perhaps it got deleted by Elon's goons grepping for "diversity".
chresko•about 3 hours ago
Are we sure this wasn't Claude? Has Dario made an announcement yet?
baggachipz•about 3 hours ago
Maybe Sean Duffy was just binge-watching his old reality tv shows and gobbled up the bandwidth?
wnevets•about 2 hours ago
On the same day they interduce AI flight controllers? [1]

Press X for doubt.

[1] https://www.npr.org/2026/09/21/nx-s1-5976816/faa-ai-manage-a...