Indirect Calling of Nested Functions on GCC Without Executable Stack

inigyou•about 2 hours ago

Self-modifying code is cool. It's a shame we had to disable it for security.

mcculley•about 1 hour ago

Not just security. Instruction caches must also be aware of self-modifying code.

WalterBright•12 minutes ago

Back in the 1980s, text editors had configuration files. The configuration file would be read every time the editor was loaded. This was very slow on a floppy disk system.

I realized that, instead of a configuration file, I could configure the executable instead! So, any changes in configuration meant the editor would patch its own exe file!

This marvelous technique came to an end when attempts to stop malware got folded into the operating system.

inigyou•34 minutes ago

yes but it's easy enough to issue a cache flush when you modify the code.

The overhead of cache flushing means some old school techniques are no longer viable, like modifying a constant in the next instruction. However it is still interesting to write machine code snippets once and execute them many times, like the nested function trampolines. I had a case where I had RGB masks like R=0x00ff0000 etc and wanted to convert 0x00rrggbb to match the mask (so no-op in the common case but not always) which could have involved setting the shift amounts in a series of shift instructions.

mananaysiempre•about 2 hours ago

Related: previous article in the series, https://news.ycombinator.com/item?id=49308685 (103 points, 47 comments)

Dwedit•about 2 hours ago

What do you need the executable stack for? You call using a function pointer, there's no executable read/write memory involved in using a function pointer.

tom_•about 2 hours ago

Nested functions may require a context pointer of some kind, which the caller can't supply. One way of doing this: create a thunk on the stack that provides the context pointer, and use that address as the pointer to the nested function.

But now the stack needs to be executable.

ok123456•34 minutes ago

Doesn't x86 actually support nested call pointers using enter to natively support this in Pascal?

klodolph•16 minutes ago

It’s not a question of ISA support. If you call via a function pointer, how do you supply the pointer to the data? (It has to either be in a separate place from the function code, or the same place. One requires an ABI change, the other an executable, writable section of memory.)

gue5t•about 2 hours ago

At first, I was annoyed by having to read AT&T syntax. Then I was disoriented by realizing the next snippet was in AT&T syntax without the '%' sigil for registers. But the technique is cool.

uecker•about 2 hours ago

Thanks. Sigils fixed (may take a couple of minutes).

Indirect Calling of Nested Functions on GCC Without Executable Stack

⚡ Community Insights

Discussion (12 Comments)Read Original on HackerNews