FR version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
34% Positive
Analyzed from 2311 words in the discussion.
Trending Topics
#infrastructure#internet#plc#should#things#software#need#critical#physical#same
Discussion (51 Comments)Read Original on HackerNews
Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move.
I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant.
This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.
So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard.
Then you layer on top the unfortunate reality that sometimes electricity does weird stuff, people design weird circuits or wire the wrong components in, and firmware tends to have to deal with non-deterministic inputs a lot more often than, say, an API on the web. It's rough.
The pay is also so much worse in my experience.
It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.
I remember a time where I was beating the drums on security and ended up in a meeting with a senior red team member in the company. This person was absolutely convinced we were not running Windows Server 2008 anywhere in the company (the year was 2019 at the time of that meeting). Needless to say, he was very concerned when I showed him the 50+ servers running it globally, all covering critical infrastructure.
I think eventually Ragnarok will happen and things will improve. I just hope it's not as detrimental as it seems setup to be.
They'll often just have The Guy running manual tests instead.
Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?".
Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant.
Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!
I think that's one of the core difficulties with PLC programming. You have to have strong knowledge on traditional science fields like thermal dynamics, material sciences, fluid mechanics, etc., while also understanding the limitations of a 16 bit floating point integer and why overflowing that can be catastrophic.
Same with SCADA: just as bad as what you describe.
I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of these places are run by operating companies, so it’s bidded out (we all know how bids work I think). I’m not surprised when I walk into these places and see the computers are running EOL operating systems and the networking is essentially flat.
I'm kind of worried (probably stupidly) that posting ideas will get me on some list, but it seems like there would be many simpler terrorism opportunities once you have physical access.
The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few billion dollars spent here could save us trillions.
The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.
1. Connected naively to the internet.
2. Behind a hardened VPN endpoint which is on the internet.
3. Has a separate physical private network.
4. Requires physical access.
I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system.
The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.
Folly to think otherwise.
Taking things offline and properly airgapped can also work, but wouldn't the cost of that exceed making specialized things and maintaining them?
We got into this situation due to cost, not ignorance. Both choices are higher cost than putting ancient devices on the internet.
This is a great theory, but practice (over centuries now if not millennia) tells us that critical infrastructure is rarely properly maintained. "If it ain't broke, don't fix it" is the motto of governments and large organizations everywhere when it comes to proper maintenance. As opposed to improper (keep the existing thing running) maintenance, proper maintenance requires being proactive and is expensive, often requiring partial or full replacements of systems while also keeping the old system running until a hand-off time. In order to get a government or corporation to be proactive, they have to see a problem.
No problem, no worry. That it can be hacked is not a problem from their perspective. That it has been hacked might be a problem to them, but only if their constituents find out. More likely, they'll make it the poor engineer's problem, the engineer who had no budget and no staff to address it beforehand.
When it's time to cut costs, proper maintenance is one of the first places organizations look to because it's not a present problem. Then it becomes normal to not do the work, from an organizational perspective, and all those engineers and technicians are just a bunch of Cassandras.
If you need a computer system to be actually secure, rule #0 is absolutely ensure it cannot receive unauthorized inputs of any kind (airgapped, big Faraday cage, JB Weld all the ports, big scary guys with guns, redundant locks, blast doors, etc). Otherwise you've lost against any sufficiently motivated adversary.
And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)
>nsa: what no, stop that