gemma-4 × xgrammar constrained decoding test

A collection of scripts that verify the modified xgrammar (~/git/xgrammar, style="gemma" / gemma_4 builtin structural tag) works correctly for tool calling with google/gemma-4-E2B-it, and generate a report comparing constrained vs unconstrained behavior.

Usage (Makefile)

make setup     # create uv venv + install deps + local xgrammar editable install (once)
make run       # run all scenarios → generate report.md + print output  ← for PR reports
make smoke     # single-tool smoke test (greedy, fast)
make xgrammar  # reinstall after modifying ~/git/xgrammar
make clean     # remove venv/report

Options: make run SAMPLES=8, XGRAMMAR_DIR=/path/to/xgrammar make setup

The output of make run, report.md, is the artifact to show reviewers — it contains the model/sampling configuration, a per-scenario table of constrained vs unconstrained metrics, and the raw text of failure cases.

Scenarios (test_gemma4_scenarios.py)

	Setup	What it targets
A. complex-schema	Nested objects + array of objects + enum + boolean schema, titles containing quotes	Argument serialization mistakes
B. multiturn-thinking	Two prior rounds of tool call/response already in context and thinking is required, the next call takes a number-typed argument	Skipped/unterminated thinking, type errors
C. adversarial-payload	A string argument whose body must contain JSON/braces/an error log verbatim, an out-of-enum word ("critical") lure, two similarly named distractor tools, temp 1.5 + top_p 1.0	String quoting collapse, enum violations

Metrics (pass/fail per sample):

metric	meaning
well_formed	Every <|tool_call>call:name{...}<tool_call|> block is complete and parseable
valid_name	Only calls tools that actually exist
schema_valid	Parsed arguments pass the JSON schema (parsed via a port of the sglang parser)
stops_at_boundary	Generation doesn't run past <tool_call|> into <|tool_response>
thought_ok	(thinking scenario) Opens the thought channel and closes it before the tool call

Key findings

• gemma-4-E2B-it's format training is very robust, so the tool call argument format itself rarely breaks even at temp 1.5 + top_p 1.0.
• Per the Gemma 4 prompt-formatting spec, the model is only responsible for generating up to <tool_call\|>; <\|tool_response> is appended by the application with the real tool result, and is registered as an additional stop sequence purely as a backstop. This harness has no such stop configured, so unconstrained runs keep generating past that boundary into engine-owned territory (stops_at_boundary failures); required-mode constrained decoding ends the call cleanly at an accept state instead.
• Unconstrained decoding also omits the thought channel entirely in situations where thinking should be enabled; constrained decoding with reasoning=True enforces it (an empty thought <\|channel>thought\n<channel\|> remains legal, matching the spec's no-thinking form).

Files

• test_gemma4_xgrammar.py — basic smoke test (greedy, single tool, alignment check)
• test_gemma4_scenarios.py — scenario runner + report.md generator. --scenario a|b|c|all --samples N --temperature T --model ID --report PATH
• gemma_parser.py — a pure port of sglang's Gemma4Detector parsing logic (for output verification)
• Makefile / report.md