ESOBENCH,
its view of a "language-centric" future, driven by the desire of models to write in any language, lower the boundaries between the familiar and the strange, and deliver programs that fulfill the promise of universal computation — plus a report on Run 0001 performance (in 36 languages or less).
(1)vision
The question is no longer whether a language model can write a program. The relevant question is whether it can write one in a language it has barely read.
31 programs × 36 languages = 1,116 tasks per model. One attempt per task. No feedback. Each submission is compiled, run once per test case, and its stdout (and exit status, where the language can set one) is diffed byte for byte against GNU grep, sed, sort, cut and wc, or against a reference solution.
I don't mean a leaderboard of trivia. I don't mean memorized LeetCode. The score is the difference between a model's standard-language pass rate and its esoteric one.
John Palenchar
- LANGUAGES
- 36
- PROGRAMS
- 31
- TEST CASES
- 442
- TASKS PER MODEL
- 1,116
(2)cliff
Same programs, same tests. Only the language changes.
RELATIVE PERFORMANCE INDEX, BEST MODEL
Esoteric pass rate ÷ standard pass rate, same model. 1.00 = no loss.
MODELS EVALUATED
- DeepSeek V4.1 Flash
- Qwen 3.8 Max
- Kimi K3
- GLM 5.3
- Claude Opus 5.5*
(3)results
| MODEL | STANDARD | NICHE | ESOTERIC | OVERALL | INDEX |
|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | — | — | — | — | — |
| Qwen 3.8 Max | — | — | — | — | — |
| Kimi K3 | — | — | — | — | — |
| GLM 5.3 | — | — | — | — | — |
| Claude Opus 5.5* | — | — | — | — | — |
| PASS RATE, % OF TASKS · RUN 0001 IN PROGRESS · * PILOT SCOPE, SEE EXHIBIT IV | |||||
(4)reach
STANDARD12
Python, C, C++, Java, JavaScript, Go, Rust, Ruby, PHP, Perl, Lua, Bash
NICHE16
Common Lisp, Smalltalk, APL, Forth, Scheme, Racket, Haskell, OCaml, Erlang, Tcl, Pascal, Fortran, Ada, Prolog, COBOL, AWK
ESOTERIC8
Brainfuck, Piet, Whitespace, Befunge-93, FALSE, Ook!, Unlambda, INTERCAL
EsoBench has two fundamental missions.
- Measure capability that cannot be recalled, only reasoned out.
- Grade it the way a compiler would: exactly, once, without partial credit.
EXHIBIT II · SCHEDULES
EsoBench Performance Report, Run 0001
The following schedules summarize the programs, languages and terms under which each model was evaluated.
Schedule 1. Unix tools
| (PROGRAM) | (BEHAVIOR REQUIRED) | cases |
|---|---|---|
| grep | Fixed-string search; -v -c -n -i. | 30 |
| sed | s///, addresses, d p q; POSIX BRE subset. | 31 |
| sort | Byte order; -r -n -u. | 16 |
| cut | -f with -d/-s; -c; ranges. | 19 |
| wc | -l -w -c; GNU column padding. | 16 |
| Total | 112 |
Schedule 2. Algorithm problems (26; 330 cases)
| Strings | regex_match, edit_distance, longest_palindrome, longest_unique_substring, longest_common_prefix, group_anagrams, word_frequency, roman_to_integer |
| Arrays | two_sum, max_subarray, product_except_self, trap_water, median_sorted, merge_intervals, spiral_matrix |
| Graphs | shortest_path, course_order, number_of_islands |
| Dynamic prog. | coin_change, lis_length, climbing_stairs |
| Search | n_queens, valid_sudoku |
| Machines | lru_cache, rpn_eval, valid_parentheses |
Schedule 3. Terms
- Attempts per task1
- Feedback before gradingNone
- Pass criterionEvery test case, byte-exact
- Exit status comparedWhere the language can set one
- Wall-clock limit per test case10 s
- Languages without argvArgs framed on stdin
- Oracle, toolsGNU coreutils/grep/sed, LC_ALL=C
- Oracle, algorithmsPython reference, brute-force cross-checked
2
EXHIBIT III · TEST MATRIX
EsoBench Performance Report, Run 0001
Each cell is one task. Shading: number of models, of five, passing it. Results pending.
| STANDARD · 12 | NICHE · 16 | ESOTERIC · 8 | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Python | C | C++ | Java | JavaScript | Go | Rust | Ruby | PHP | Perl | Lua | Bash | Common Lisp | Smalltalk | APL | Forth | Scheme | Racket | Haskell | OCaml | Erlang | Tcl | Pascal | Fortran | Ada | Prolog | COBOL | AWK | Brainfuck | Piet | Whitespace | Befunge-93 | FALSE | Ook! | Unlambda | INTERCAL | |||
| grep | ||||||||||||||||||||||||||||||||||||||
| sed | ||||||||||||||||||||||||||||||||||||||
| sort | ||||||||||||||||||||||||||||||||||||||
| cut | ||||||||||||||||||||||||||||||||||||||
| wc | ||||||||||||||||||||||||||||||||||||||
| regex_match | ||||||||||||||||||||||||||||||||||||||
| edit_distance | ||||||||||||||||||||||||||||||||||||||
| longest_palindrome | ||||||||||||||||||||||||||||||||||||||
| longest_unique_substring | ||||||||||||||||||||||||||||||||||||||
| longest_common_prefix | ||||||||||||||||||||||||||||||||||||||
| group_anagrams | ||||||||||||||||||||||||||||||||||||||
| word_frequency | ||||||||||||||||||||||||||||||||||||||
| roman_to_integer | ||||||||||||||||||||||||||||||||||||||
| two_sum | ||||||||||||||||||||||||||||||||||||||
| max_subarray | ||||||||||||||||||||||||||||||||||||||
| product_except_self | ||||||||||||||||||||||||||||||||||||||
| trap_water | ||||||||||||||||||||||||||||||||||||||
| median_sorted | ||||||||||||||||||||||||||||||||||||||
| merge_intervals | ||||||||||||||||||||||||||||||||||||||
| spiral_matrix | ||||||||||||||||||||||||||||||||||||||
| shortest_path | ||||||||||||||||||||||||||||||||||||||
| course_order | ||||||||||||||||||||||||||||||||||||||
| number_of_islands | ||||||||||||||||||||||||||||||||||||||
| coin_change | ||||||||||||||||||||||||||||||||||||||
| lis_length | ||||||||||||||||||||||||||||||||||||||
| climbing_stairs | ||||||||||||||||||||||||||||||||||||||
| n_queens | ||||||||||||||||||||||||||||||||||||||
| valid_sudoku | ||||||||||||||||||||||||||||||||||||||
| lru_cache | ||||||||||||||||||||||||||||||||||||||
| rpn_eval | ||||||||||||||||||||||||||||||||||||||
| valid_parentheses | ||||||||||||||||||||||||||||||||||||||
3
EXHIBIT IV · MODELS EVALUATED
EsoBench Performance Report, Run 0001
| (MODEL) | (ACCESS) | (SCOPE) | tasks | (STATUS) |
|---|---|---|---|---|
| DeepSeek V4.1 Flash | Fireworks serverless | Full matrix | 1,116 | Pending |
| Qwen 3.8 Max | Fireworks serverless | Full matrix | 1,116 | Pending |
| Kimi K3 | Fireworks serverless | Full matrix | 1,116 | Pending |
| GLM 5.3 | Fireworks serverless | Full matrix | 1,116 | Pending |
| Claude Opus 5.5* | Anthropic API, effort medium | Pilot: two_sum, wc | 72 | Pending |
Notes to the results
1. Every language toolchain is proven before scoring by a program that echoes its arguments and input and sets its exit status.
2. Brainfuck, Piet, Whitespace, Befunge-93, FALSE, Ook! and Unlambda run on reference interpreters written for the benchmark; semantics the language leaves open (end of input, division by zero) are fixed and stated in each prompt.
3. Algorithm problems are well known and are reported separately from the tools. Every task carries a canary string.
4. * Claude Opus 5.5 was run on two programs (two_sum, wc) across all 36 languages only.
4