A Markdown-based test suite

eichin•about 6 hours ago

Anybody remember "cram"? From about 10 years ago, https://bitheap.org/cram/ basically a markdown syntax (making heavy use of code-blocks) for documenting and writing tests as "shell commands and expected output" (with a bunch of the sharp edges filed off, like line endings and partial matches.) Was particularly good for easy-to-write, easy-to-review tests of unix utilities. (It's the kind of thing that you only stumble on if you've been working with doctests but they don't really fit well for shell/unix stuff...)

Saser•about 2 hours ago

I love Cram tests, they can be so readable! Being essentially shell scripts, though, they can be a bit hard to integrate into any given language's regular testing framework though.

I get the sense that "expect tests", also known as "inline snapshots tests", are becoming more and more popular. I think that many, including myself, first learned about it from the Jane Street blog post "What if writing tests was a joyful experience?" (https://blog.janestreet.com/the-joy-of-expect-tests/) because I keep seeing references to it. Indeed, the blog post points to Cram as prior art in this space. I also think Ian Henry's article "My Kind of REPL" (https://ianthehenry.com/posts/my-kind-of-repl/) makes a very convincing case for why expect tests are useful.

Jane Street has published their `ppx_expect` library for OCaml and you can use it today: https://github.com/janestreet/ppx_expect. For other languages, here's an incomplete list of similar libraries I'm vaguely aware of:

- Rust: expect-test, (https://github.com/rust-analyzer/expect-test, quite old), insta (https://insta.rs/) and k9 (https://github.com/boujeepossum/k9)

- JavaScript: Jest has inline snapshot testing: https://jestjs.io/docs/snapshot-testing#inline-snapshots

- Python: inline-snapshot (https://15r10nk.github.io/inline-snapshot/latest/). Also discussed in this post from Pydantic: https://pydantic.dev/articles/inline-snapshot

- Go: https://github.com/hexops/autogold. I like it and use it daily at work. It has some shortcomings around working with strings as the expected value, so I'm fiddling with my own library that will be much more similar to `ppx_expect`.

- Zig: TigerBeetle has an internal `snaptest.zig` library in their codebase (https://github.com/tigerbeetle/tigerbeetle/blob/588123f219f1...), also discussed in their blog post "Snapshot Testing For The Masses" (https://tigerbeetle.com/blog/2024-05-14-snapshot-testing-for...).

thangalin•about 6 hours ago

Speaking of test suites, LLMs cannot quite curl straight quotes correctly. My Markdown editor uses the following suite:

https://repo.autonoma.ca/repo/keenquotes/tree/HEAD/src/test/...

Am curious whether SOTA LLMs can curl the ambiguous cases.

echoangle•about 5 hours ago

Is curl a synonym of grok?

luplex•about 3 hours ago

no, this is just about the shape of the quote.

There are straight quotes: ' or "

and there are curly quotes: ‘ and ’ or “ and ”

"curling" then just means "figuring out whether it's an opening or closing quote"

riffraff•about 7 hours ago

The "test anything protocol" was a text based system for writing tests, I think perl might still use it

I remember using it to implement the test suite for a Shakespeare language interpreter. Fun times.

https://en.wikipedia.org/wiki/Test_Anything_Protocol

abusaidm•about 8 hours ago

Interesting approach, I can see the benefits of such approach where the md is acting like a Jupyter notebook of some sort. You place the content with documentation and it becomes a self standing unit of test.

I wonder if this can be replicated to other notions where the docs/data needed for test add more value to live with the test

pyrolistical•about 4 hours ago

You can also write Gherkin in markdown and have test fixtures for each given, when, then

dhruv3006•about 7 hours ago

I think you will like https://voiden.md/ - some what similar to what you did here.

PS: I am currently working on Voiden.

rubyn00bie•about 7 hours ago

You’re developing that so it would be nice to have the disclosure.

Additionally, I cannot find any obvious examples of what it does, or how it possible relates, and none of the sections seem to offer anything even resembling an example. The entire site seems like a marketing template.

FWIW— if it is similar, or there are examples, you should absolutely put them to the forefront. I literally have no idea what any of the statements/propositions in the pages actually mean.

dhruv3006•about 6 hours ago

yep editing my comment !

A Markdown-based test suite

⚡ Community Insights

Discussion (11 Comments)Read Original on HackerNews