# gemma-4 × xgrammar constrained decoding test

A collection of scripts that verify the modified xgrammar (`~/git/xgrammar`,
`style="gemma"` / `gemma_4` builtin structural tag) works correctly for tool
calling with `google/gemma-4-E2B-it`, and generate a report comparing
constrained vs unconstrained behavior.

## Usage (Makefile)

```bash
make setup     # create uv venv + install deps + local xgrammar editable install (once)
make run       # run all scenarios → generate report.md + print output  ← for PR reports
make smoke     # single-tool smoke test (greedy, fast)
make xgrammar  # reinstall after modifying ~/git/xgrammar
make clean     # remove venv/report
```

Options: `make run SAMPLES=8`, `XGRAMMAR_DIR=/path/to/xgrammar make setup`

The output of `make run`, **`report.md`**, is the artifact to show reviewers —
it contains the model/sampling configuration, a per-scenario table of
constrained vs unconstrained metrics, and the raw text of failure cases.

## Scenarios (`test_gemma4_scenarios.py`)

| | Setup | What it targets |
|---|---|---|
| **A. complex-schema** | Nested objects + array of objects + enum + boolean schema, titles containing quotes | Argument serialization mistakes |
| **B. multiturn-thinking** | Two prior rounds of tool call/response already in context and thinking is required, the next call takes a number-typed argument | Skipped/unterminated thinking, type errors |
| **C. adversarial-payload** | A string argument whose body must contain JSON/braces/an error log verbatim, an out-of-enum word ("critical") lure, two similarly named distractor tools, temp 1.5 + top_p 1.0 | String quoting collapse, enum violations |

Metrics (pass/fail per sample):

| metric | meaning |
|---|---|
| `well_formed` | Every `<\|tool_call>call:name{...}<tool_call\|>` block is complete and parseable |
| `valid_name` | Only calls tools that actually exist |
| `schema_valid` | Parsed arguments pass the JSON schema (parsed via a port of the sglang parser) |
| `stops_at_boundary` | Generation doesn't run past `<tool_call\|>` into `<\|tool_response>` |
| `thought_ok` | (thinking scenario) Opens the thought channel and closes it before the tool call |

## Key findings

- gemma-4-E2B-it's format training is very robust, so the tool call argument
  format itself rarely breaks even at temp 1.5 + top_p 1.0.
- Per the [Gemma 4 prompt-formatting spec](https://ai.google.dev/gemma/docs/core/prompt-formatting-gemma4),
  the model is only responsible for generating up to `<tool_call\|>`;
  `<\|tool_response>` is appended by the application with the real tool
  result, and is registered as an **additional stop sequence** purely as a
  backstop. This harness has no such stop configured, so unconstrained runs
  keep generating past that boundary into engine-owned territory
  (`stops_at_boundary` failures); required-mode constrained decoding ends the
  call cleanly at an accept state instead.
- Unconstrained decoding also **omits the thought channel entirely** in
  situations where thinking should be enabled; constrained decoding with
  `reasoning=True` enforces it (an empty thought `<\|channel>thought\n<channel\|>`
  remains legal, matching the spec's no-thinking form).

## Files

- `test_gemma4_xgrammar.py` — basic smoke test (greedy, single tool, alignment check)
- `test_gemma4_scenarios.py` — scenario runner + report.md generator.
  `--scenario a|b|c|all --samples N --temperature T --model ID --report PATH`
- `gemma_parser.py` — a pure port of sglang's `Gemma4Detector` parsing logic (for output verification)
- `Makefile` / `report.md`