ECC and DDR5
125
DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
58% Positive
Analyzed from 4416 words in the discussion.
Trending Topics
#ecc#memory#errors#ram#data#more#should#cost#error#cpu
Discussion (126 Comments)Read Original on HackerNews
DDR5 on-die ECC detects and corrects one-bit errors. It cannot detect two-bit errors, so it will miscorrect some of them into three-bit errors. My understanding is that the on-die error correction scheme is specifically specially designed such that the resulting three-bit errors are mathematically guaranteed to be detected as uncorrectable two-bit errors by a standard full system-level ECC running on top of the on-die ECC. But I've never found a real authoritative reference that directly says that.
Is anyone here good at navigating patent records? I would be surprised if there isn't something to be revealed in a patent but I don't know how to find it.
One nightmare scenario is a 3-bit error being detected as a correctable 1-bit error upstream, resulting in data corruption. I agree it’d be really nice to know that could never happen!
And 9 times out of 10 (probably much more) it will be faulty storage, not RAM.
I'd say you are way overestimating impact of no-ECC.
Single digit - sometimes none - errors per year in systems with consumer amount of RAM.
Some crappy flash drives might not do this, though. Seems to behave like a floppy disk at times.
I've seen this in DC grade SSDs, NAS grade hard drives, and SSDs sold in high end laptops.
I have DDR4 ECC memory on my home server. Last week I got a crash... logged into my BMC and saw that one of the modules had "uncorrectable ECC". Then I looked over the logs and saw about 200 or so errors for the exact same module but "correctable ECC". They would happen over a short period of time followed by 4 months of no issues. Thanks to ECC and logging it was quick and easy to catch the issue right on the first occurrence, no data corruption. I have 4 modules in total, and only this one was having issues, so clearly a hardware fault.
Now imagine it was a regular non-ECC memory. There won’t be any crashes due to uncorrectable ECC; another 3 or more years would pass before corruption would become frequent enough for the issue to be noticed. At that point, there would be hundreds of corrupted files.
I don't think so. I've got my important data backed up on offline HDDs and SSDs and my most important data lives on a RAID system (ZFS) on a Xeon with ECC RAM but...
... there are simply a huge lot of checks everywhere making it so that bit flips on non ECC systems simply aren't that serious.
We live in a world of checksums and retries at every layer and distributed systems that are just about everywhere.
I went on for decades without ECC memory and nothing bad happened. Yes, I had the occasional bitflips. I did found out faulty mem using memtest (or weirder stuff, like some RAM stick identical to the other three, same chip everywhere, same sticker: but reported by memstick as being some other mem).
But I don't think that the "one bitflip cascading into corruption of your life's data" is something necessary to freak about.
It's not happening. Never happened to anyone.
If you (or other) are really concerned, you can use my scheme:
becomes: which means "that file's Blake3 checksum begins with 73b0835221" (use something else than Blake3 if so desired).About 1/3rd of all my files are "tagged" this way, directly in the filename: this survive different medium and different OSes.
I then have my custom rsync backuping script doing a dry-run and warning me in a "stop the world!" fashion if it detects a bogus checksum (either on the source or on the dest: during the dry-run any modification to a file with a checksum is verified and any checksum mismatch means "stop the world").
I then also have, in addition to my custom backuping script, another script that allows me to verify either all files with that naming convention or a percentage I pick of those files (so basically some random sampling).
It's not mutually exclusive with ZFS / a system with ECC RAM.
> but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time
I hear you but... How much do you really need, years later, a file that you haven't needed for years? That one file is unlikely to be a big loss.
Now I'm not saying you're wrong: I ended up with my naming scheme (appending a partial cryptographic hash to the filename) after finding a series of about 20 pictures that were corrupted. And on all the backups I first checked, they were also corrupted. I ended up finding a very old data CD on which that series of .jpg files were fine. And I decided "never again, I want such SNAFU to be detected in the future".
But truth is: with 40 000 pictures, life without those 20 pictures would still be fine.
Google and Facebook report a few unreliable CPU cores out of thousands of systems. Meaning maybe a 1/10,000 rate of repeatable CPU errors.
https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf
Google reports 8% of DIMMs being affected by errors. While the differences in metrics prevent comparing the two things directly it seems that DRAM errors are more common than CPU errors.
Normal good DIMMs, at least when new, should not have errors more frequently than one error per many months.
Frequent errors may appear with memories that are not seated well in their sockets, or which are old, at least several years old.
Frequent errors may also be caused by more general computer problems, like a bad power supply unit.
I would think most servers and workstations would be RDIMM (Registered DIMM) by now and consumer stuff uses soldered down memory. Memory failing because it's old is definitely a thing, and very possible in this scenario, but I feel like I haven't seen errors due to physical insertion, that were not caught immediately by POST, in years.
But maybe it's just me. Happy to be lucky. :)
With normal sticks I’d not have know that there is a potential issue.
With non-ECC RAM running a memory test and getting any errors means it’s not stable. The memory tests run at maximum bandwidth constantly so a few days of full bandwidth memory testing might produce as many transmissions as your months of regular server use. Seeing bit errors (even if corrected) means something is not right.
The ECC can correct it for a while but I would want to RMA that stick if it’s in warranty.
At some point I should try running it at slightly lower speed i guess. It’s very hard to troubleshoot something that only happens every month or two though
Never seen it fail a memtest but I can see it in the ECC stats since those cover weeks/months.
Boy did I learn that the hard way. I found an ASRock server motherboard that fit my needs for a NAS perfectly. I just assumed all ECC memory were registered so I didn't even bother to check. I had already put everything together minus the memory. I had actually purchased the memory and had to return them because they were RDIMMs and get the ECC UDIMMs for quite a bit more. Luckily this was just before the AI induced price spike.
I hope that my blog posts on these topics will save some people some money.
That's because the on-die ECC cannot detect errors in the transmission of data from the DIMMs to the CPU. It seems like on-die ECC is meant to "hide" some level of errors to make less reliable RAM chips appear good. I have no idea if that's a reasonable way to do it, but it seems like it could be misused. Like TFA suggests, we need to know how the two levels of ECC interact.
On-die ECC was added in DDR5 because the error frequency has increased too much.
Its only effect was that now DDR5, as seen externally, has a reliability similar to that of the older memory generations. On-die ECChas not brought any serious improvement in reliability. It has just prevented the degradation of the reliability.
The errors generated internally are caused mostly by the ionizing radiation from the environment, which discharges the storage capacitors.
However there are many errors that appear during the communication between memories and the CPU, which are caused by the electrical noise from the environment.
These external errors are much less likely to appear for soldered DRAM. Because of this, for soldered LPDDRx memory it is less important to have end-to-end ECC than for socketed modules, which are much more vulnerable to noise.
The DDR5 standard has some options for using some error detection for the communication link, but it is impossible to know whether a given computer or motherboard has implemented such options or if they are enabled by the firmware when the motherboard has the physical support.
The only way to be certain that the memory works fine is to use ECC that covers completely the circuit from the CPU memory controller to the values stored in the DRAM and back to the CPU memory controller and your operating system has the appropriate device driver for obtaining the error reports.
When you do not own a great number of servers, you cannot provide anything else except anecdotal evidence.
Because the vast majority of PCs do not have ECC memory, even the companies that have a great number of PCs have no idea about the reliability of the memory used in those PCs, because it is very difficult to distinguish memory defects from the huge number of software bugs.
Moreover, the frequency of memory errors is proportional with the amount of memory you have, so those with tiny amounts of memory, like 8 GB, are much less likely to encounter memory errors than those who have 32 GB or 64 GB of DRAM in their PCs.
I have tried to always use ECC memory in my computers, whenever possible. Even my laptop is an older Dell Precision, with ECC memory.
But I can also offer only anecdotal testimony.
I have also seen cases like that mentioned in TFA, where the existence of ECC allowed to discover that some modules were not seated well in their sockets, so reseating them prevented the reappearance of errors.
I have also seen a case when a laptop was not used for a long time and it was stored in a rather humid place, so the contacts in the SODIMM sockets had oxidized, which resulted in frequent memory errors. Scrubbing vigorously the contacts and reinserting the memory modules solved the problem.
I have also seen many cases where certain memory modules degraded after many years of use, e.g. 5 or more years of 24/7 use, and they began to have frequent errors, e.g. multiple errors per day, even if when they were new the error rate could have been of one error per year or even less.
In such cases ECC was extremely useful, because the DIMM that had become worn out could be identified and replaced and the server could work fine some more years.
I noticed the difference because in a Hardware forum someone wrote that the framework desktop does not have ECC while an HP G2 Workstation also featuring the Ryzen 395 Max but in the ECC supporting variant paying a premium of +600 bucks.
Please stop repeating this old myth, it is - and always has been - BS and needs to die.
See for example: https://kldload.com/zfs-wiki/myths
It was a pain to select a compatible board and source the RAM but glad I did, in June 2024... the RAM prices went up immediately after.
non-ECC should be a kernel taint in Linux. Newer versions of Windows (12?) should also refuse to boot without ECC. It should be mandatory in DDR6. The insanity must end.
The rest of this article is great. This, however, is absolute garbage and nonsense. The last thing you want is more regulations and "government certified systems". Take a look at the bullshit with age verification bills and operating systems.
Claiming that every issue you've experienced is from seating errors? Come on man, at least deliberately reproduce that scenario once. I've tortured motherboards for fun and never managed that on purpose. We did pry every cap off of an old K6 board while it was on and... it just kept running. It didn't come back after we turned it off
The amount of time it takes to reseat memory, even if the case is already open and ready, is also more than enough time for it to cool back down.
If you have decent memory running at what it's rated for and you're seeing more than a couple per year, good odds are it's getting too hot or the load line calibration is not aggressive enough.
E: I have never, and I mean literally never, seen age or abuse play a factor.
> We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost to society in corrupted data and lost time due to RAM hardware errors. I think that at minimum we need sufficient taxes on non-ECC RAM (and EC4 RAM for DDR5) to make it more expensive when bought new than ECC RAM.
ECC RAM is not needed for every application and mandating it for every gaming rig and machine used to run Chrome will just increase prices unnecessarily. 99% of the time it's also not a "cost to society" but a cost to whoever owns/uses that machine. If they want to eat that cost, it should be up to them.
The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it. No one wants even games or web browsers to crash randomly.
> If they want to eat that cost, it should be up to them.
They need to be given the choice. Most people do not know they even have the choice, and they are not offered it at a reasonable price.
A cost to people in general is a cost to society.
I'm not sure making it illegal to sell consumer grade hardware is the boon for the people you imagine
Restricting risky items is standard practice for all governments. All the "use by" dates on food etc are from government regulation. The current experiments with "raw milk" shows that you can get away with flouting food safety regulations and not have immediate problems.
https://ripe92.ripe.net/programme/meeting-plan/sessions/112/...
AMD doesn't have this segmentation. Where's the hoard of people building systems with ECC ram? At best you have a few people on HN saying they're doing it, but in practice nobody actually does it.
Whether a filesystem could be designed to operate in the face of random memory changes would be an interesting computer science research topic. But for getting working systems having RAM just not fail is a better option.
it is now $500
when the AI/datacenter bubble finally bursts we are going to have some amazing hardware deals
Of course an AI is going to tell you AI isn't a bubble.
Why is it so often Australia with the "We must force unnecessary things upon people" attitude?
RAM without ECC is not faulty. It still does exactly what the fuck it is intended to do - act as a rapidly-modifiable store of referenceable information.
Anyone considering non-ECC RAM to be faulty or defective is a fool that needs to go back to school and re-take some basic Computer Science courses.
Forcing the vendors to sell only products that do what they are advertised cannot be considered in any way as "forcing unnecessary things upon people".
Any computer without ECC memory is a computer that does not compute from time to time and you have no idea when that happens.
Originally, any IBM PC or compatible had memory error detection.
All this problem of the computers without ECC memory has been created by some evil Intel employees, who have conceived around 1994 a scheme for market segmentation between "Pentium" and "Pentium Pro".
So they removed memory error detection support from the Pentium chipsets (Intel Triton), so that this would remain a marketing feature exclusive to the "Pro" Intel products.
Obviously the removal of this feature from "consumer" products was done without any price reduction, which made happy also the memory vendors, because this trick increased their profits.
So in the next years this market segmentation method became the "standard" business practice, preying on the naivety of most computer users.
A consumer protection law should force that any computer or motherboard must support ECC, which adds a completely negligible cost.
The end users may buy non-ECC modules, if they choose so, but non-ECC modules should be sold like cigarettes, with a visible advertisement that using such modules may cause unpredictable errors in your computer.
So are the computers with ECC memory. It's not an absolute protection either.
I hate to break it to you but the computer I had in the 80s didn't come with ECC. Non-ECC computing was a thing WELL BEFORE Intel tried to think of market segmentation. Revisionist history is not welcomed here.
The DDR4 ECC RAM has 72 bits total and 64bits of data which means the same 9/8 ratio of data storage to usable data as the original PC/XT.
The Z80 and 6502 based computers in the 80s typically didn't have parity RAM. The Amiga and Macintosh were higher performance systems in the 80s which also didn't have parity RAM.
We could and should have gone straight from parity RAM to ECC RAM for everything.
That's what we mean when we say that people with your point of view are promoting nanny-statism. It amounts to infantilization of adult consumers and subsequent disempowerment.
Occasionally such legislation can be justified, as when genuinely hazardous products are involved, or products whose use involves wide-ranging externalities, but usually not. People can and should be educated -- and expected -- to make their own decisions about things like whether they need ECC RAM.