MTG Bench: Testing how well LLMs can play Magic

I actually considered using card forge when I started this. I mostly didn't end up using it because of how much more work it would have been.

But also with a rules engine, you have to manually go though every step, and pass priority after every action.

I think it makes more sense to let an LLM play magic like a person would. On early turns it is acceptable to say "I play a land and pass" without going through every phase. And you can say "I tap all my land and play this card" without having to use a tool call and agent turn for every land tap.

Also card forge would not let you goldfish a deck. You must have opponents.

MTG: Arena uses a rules engine CLIPS (a s-expr expert system based on the RETE engine), which an acquaintance wrote a course for: https://ryjo.codes/tour-of-clips.html and even a declarative chat server: https://ryjo.codes/articles/a-simple-tcp-server-written-in-g...

    (defrule connection
      (connection ?id)
      =>
      (println "User " ?id " connected")
      (printout ?id "Welcome to the chatroom from CLIPS!" crlf)
      (do-for-all-facts ((?f connection)) (neq ?id (nth$ 1 ?f:implied))
          (printout (nth$ 1 ?f:implied) "User " ?id " connected" crlf)))
    
    (defrule say
      (connection ?id)
      ?f <- (message-buffered ?id)
      ?ff <- (message ?id ~/me ?message)
      =>
      (retract ?f ?ff)
      (printout ?id "You: " ?message crlf)
      (do-for-all-facts ((?f connection)) (neq ?id (nth$ 1 ?f:implied))
        (printout (nth$ 1 ?f:implied)
         ?id ": " ?message crlf)))

> because of uncertainty on how it would affect it.

Have the LLM submit a proposed move and either advance the game state or reply "permission denied, try again". Probably also log the number of times it happens since attempted violations seems like a valuable signal as well.

No, I was not aware of that project when I made this.

I'll have to look into that project, but I also have an RTX 5090 and did a lot of testing with Qwen3.6 27B and Gemma 4 31B. I was not able to get it to play legal turns consistently. I had to keep expanding the system prompt and adding rules for edge cases. By the end, the prompt was over 10k tokens, and while it mostly make legal turns, it did not make good turns. And all the heuristics in the prompt degraded the performance and increased the cost for frontier models.

Gotta walk before you can run.

Admittedly, the mulligan phase system prompt is the weakest part of the project. I had to add heuristics to stop the LLMs from mulliganing down to just a few cards looking for a perfect hand. The scoring for the benchmark is mostly based on if the LLM could complete legal turns, not good turns.

https://github.com/CallumFerguson/mtg-auto-deck/blob/a877c08...

Magic is complicated. I looked at doing something like this but the open-ended nature where one specific card will completely change the rules or require a series of followup events or modifications to the rules engine at hand is just tremendous.