DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
40% Positive
Analyzed from 3148 words in the discussion.
Trending Topics
#automation#more#attention#system#don#traffic#where#makes#driving#pilot
Discussion (77 Comments)Read Original on HackerNews
The entire premise of a system is that it removes the need for careful attention.
system: signal lights tell me whether or not I can pass through an intersection, so that I do not have to attend to potentially high speed traffic from a variety of directions.
system: the side my knife blade sits on my arched guide fingers, so that I do not have to attend to the edge of the blade or the location of my fingers.
etc etc.
I think this premise is flawed or, at best, too narrow. A system is just a logical grouping of items that perform a function. Sometimes that function can be to reduce cognitive burden, but it doesn't have to be. A "vision system" like what humans use does not reduce attention, but increases/enables it, while a autonomic nervous system can reduce attention. The ability to increase/reduce attention is not the central principle of a system.
A vision system does allow you to pay less attention: you don't need to carefully remember how far away the door is, you just need to look! I tried this often as a kid: if you want to navigate a hallway with your eyes closed, you need to pay far more attention to your other senses than you need to pay with your eyes -- where attention here is not the volume of data, but rather the complexity of conscious bookkeeping -- I can (ironically) "play it by ear" with my eyes open, but eyes closed I must plan every step!
It just so happens to be that the ability to pay less attention makes more things possible and hence the demand for attention overall may increase -- if not intrinsically, due to your competitors (who can also see!)
You know I noticed this... I lived in a country where people obey traffic laws, and in a country where they very much don't.
I witnessed many more traffic accidents in the country where people are used to relying on the traffic lights to tell them if it's safe or not.
Whereas in the other country, everyone correctly assumes that the other drivers are completely insane, and so they stay vigilant.
Now I do think the science shows if you design roads and systems to make drivers more thoughtful it can improve outcomes. Size roads for the speed limit, roundabouts, etc. these can make a difference as it balances the system.
The difference here is one of stability: in a developing country, I can just walk across a street (often there is no traffic light) by essentially signalling with my body language -- both I and the drivers are paying attention. And if one party fails, the other has a good chance of catching that mistake.
Now, in a developed country, neither side is paying attention. If I walk across the street, I'm in danger, no matter how clear my body language (I tried it on British streets a few times -- it works in some areas, but usually very poorly!), and no one expects a crazy driver to come barreling through a red light.
The developing countries fall behind because in the crazy * sane intersection, sometimes the sane person is just not fast enough -- whereas the crazy * crazy intersection is extremely dangerius and happens often enough.
On the other hand, a developed country makes every interaction sane * sane regardless of the personalities or moods of those involved -- but God forbid a bit of crazy leaks out!
You can't make that assertion (well you can, it's called "lying with statistics" but that's beside the point) without knowing if the fatalities the result of the accident rate or just a higher conversion ratio as a result of reduced safety equipment, reduced seatbelt usage, more motorcycles, etc, etc, worse emergency services, etc, etc.
INB4 other people start whining on your behalf, I'm not saying those countries aren't less safe to drive, just that you can't do a straight comparison of accident rates and fatalities without considering the conversion ratio.
For example, if you live somewhere where you use the highway more often, that sure as heck can skew the result.
Or if you live(d) somewhere where people tend to hit and run instead of waiting... you're obviously not going to witness them as often.
Also, note that accidents and injuries are not the same thing. You can totally have fewer accidents but more injuries or fatalities.
Without knowing the neighborhoods you've lived in (so people can compare the data for themselves) you're really not going to make a compelling case.
https://en.wikipedia.org/wiki/Risk_compensation
In contrast, most careless driving habits don't actually get anybody to their destination any quicker.
The insane driving country has double the traffic related deaths as the chill, lawful driving country.
People die on road more in countries that conventionally don't follow traffic laws.
No they don't, they tell you and other vehicles to stop. You would fail your driving test if you depend only on the traffic lights and don't bother to verify it is safe to pass yourself.
(If safety/the red phase was the purpose, the intersection would use a roundabout instead.)
Someone learning to fly may be described as paying careful attention: to every little sound, vibration, and sensation. A common tactic by student pilots is overcontrolling the aircraft, e.g., large sudden changes rather than smooth pressures from flying with a light touch.
Automation requires active, intentional attention particularly when flying in clouds. What are my instruments telling me? Are they all telling the same story? Have any failed? Which ones?
A significant part of flight training and testing emphasizes the ability to divide attention between multiple competing needs, being able to correctly prioritize them, and responding promptly and safely in order of priority.
If your task is for example, to pilot a preset route with stable condition and very low surprise, you will fall for the "getting too comfortable" trap and we tend to start to get lazy(or efficient) and offload the mental effort and skills atrophy. A common workaround to this is to have regular training(deliberate practice) that introduce the "tricky" situation to keep the skill up. Problem ofc arises when people don't keep this up.
This can be seen in the diagnostic performance differences between junior and senior doctors, not always in the favor to the senior [3]. If you add a layer of automation but the insight gathered by working on that layer is great (and falls off) then deliberate practice start to become a requirement
[1] https://commoncog.com/putting-mental-models-to-practice/
[2] https://youtu.be/5eW6Eagr9XA?si=Y9exacaW-F4PDOKF
[3] https://pubmed.ncbi.nlm.nih.gov/26375267/
>The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter.
So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied.
In other words, it makes the normal situations safer but the abnormal situations more dangerous.
Any equipment on the aircraft can and will fail. Becoming dependent on autoland — not a worry on most general aviation aircraft — is terrible risk management. Every pilot must maintain hand flying skills. Automation is nice and reduces workload, but the pilot must actively manage it.
Not to mention that they get mandated regular reviews of their ability to fly manually. And even with that, there's still a reason why "children of the magenta line" (i.e. pilots who passively follow automated systems into danger and/or have seriously degraded stick-and-rudder skills) has become a term.
An abnormal landing would be something like trying to land with a broken elevator surface.
If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.
even if the average rate goes up tor net benefit, are organizations prepared for increased carastrophic failures?
So too is what's happening with LLMs: they're writing code that the programmer is increasinging unaware of and the programmer is increasingly not capable of understanding because they dont have the experience of writing and navigating complex codes. So in the event of a permissions prompt, the fallback condition, you have the same race condition between increasing automation removing the knowledge generation of the operator.
So there's obviously a rubicon where automation _has_ to be 100% because no operator can be a fallback.
[1] https://static1.squarespace.com/static/644321e78cd2dd37613af...
This is how I have been doing SRE for the past decade. And the inability to practice this under-the-hood reasoning is the main reason why I don't trust any modern AI based automation systems. They are helpers for some manual work, but they can't be automation.
As far as I know, Places that actually care about safety apply all of them at once.
They add layers of failsafes that don't rely on humans, they make the automation better and better, and for the most critical stuff, they continue training for things that will almost certainly not happen.
And for the rare(depending on personality) cases where we care about the pilot's ability more than the result, just don't automate at all.
There are, of course, many benefits to automation such standardisation, measurability and the list goes on. Plus cuurrently we have this sweet spot where the workforce contains several generations who have experienced both very manual and highly automated processes. This dual experience is invaluable for investigation and continuous improvement. It makes me wonder what will happen when the workforce consists entirely of operators and engineers who simply press start most of the time.
5min to learn. A week of normal driving to get not bad. I wouldn't say it's hard.
It certainly takes more than a week!
Also if you know that automation induced complacency is a thing then it must surely become a target for training, surveillance, and adaptation not mere hand wringing.
In aviation, commercial pilots have very strict and extensive training and monitoring and as a result are generally able to utilize automation effectively while keeping up their manual skills. There are very rarely CFIT incidents in major commercial airlines.
The opposite is true in general aviation (small private Cessnas, etc), where it’s extremely common for pilots to buy more plane than they can handle and then rely on automation to bridge their skill gap. CFIT is much more common in general aviation, along with incorrect actions in response to real system failures that should have been recoverable. Automation complacency regularly kills in general aviation.
A key thing to notice is that automation isn’t outright prohibited in either commercial or general aviation, but there are distinct regulatory frameworks based on potential impact.
We accept looser rules for general aviation because the failures are societally less severe and because the population is much larger so effective training and enforcement would be significantly harder. In commercial airliners where failures are catastrophic, we have much stricter policies and require training and testing regularly to avoid automation complacency.
Will we start to see this practice in software? Probably, but only if/when the societal cost of NOT doing it becomes more clear. We regulated aviation because crashing planes are obviously bad. We license structural engineers because collapsing bridges are obviously bad. Will automation-induced software failures hit a similar tipping point?
I know this is an analogy to AI, but I wish we would dispense with this idea that there’s some appropriate level of machinery which was reached just a hair before right now. There is no appropriate level of machinery, no point at which the nature of the system itself will unambiguously say “that’s enough.”
Actually this could be a side channel to measure the efficiency of automation. If the human operators are just as good in every aspect as before, we know all the money put into automation is wasted.
I wonder what we'd call the children today in hindsight and what line they're chasing now...
[1] https://www.youtube.com/watch?v=5ESJH1NLMLs
Evidence: Look at the most recent article on this blog: https://julienreszka.com/blog/difficult-conversations-don-t-...
"Memory is reconstructive. When someone recalls events differently, it feels like gaslighting. It usually isn't. Document first, then negotiate."
Bleugh