lvl30
search
Prompts That Work

A JSON extraction prompt small local models can actually follow

Getting clean structured output from a 7B or 8B model is mostly about the example object, not the rules. This is the pattern that stopped my local extraction jobs from breaking.

By Adam
Use case
Data extraction (local models)
Works with
Llama 3.1 8B, Qwen3 8B, Mistral 7B
▶ The prompt 
Extract data from the text below into JSON.

Rules:
- Output ONLY the JSON object. No explanation, no markdown fences, nothing
  before or after it.
- Use exactly these keys: {list your keys and their types}
- If a value is not present in the text, use null. Never guess a value.
- Dates as YYYY-MM-DD. Numbers as numbers, not strings.

Example of a correct output:
{a filled-in example object with realistic values}

Text:
{the text to extract from}

Big API models will forgive a sloppy extraction prompt. A quantized 8B running on your own box will not: it wraps the JSON in markdown fences, adds a cheerful “Here’s your data!”, or invents a value for the field it could not find. If you are running extraction in a loop overnight, any of those kills the run.

Two things fixed it for me, and the prompt is only one of them.

Why it works

  • The example object outworks all four rules combined. Small models imitate better than they obey. One realistic filled-in example does more for key names, date formats, and null handling than any amount of instruction text.
  • “Never guess a value” paired with an explicit null policy is what keeps the output honest. Without it, small models treat every key as mandatory and fill blanks with plausible garbage, which is the worst possible failure because it parses fine.
  • Temperature 0. Not in the prompt, but set it. Extraction is not a creativity task, and sampling randomness is where the occasional trailing comma comes from.

What good output looks like

A bare JSON object, byte one to byte last, that parses on the first try, a thousand runs in a row. Anything else is a failure, and you should treat one markdown fence in a hundred runs as a bug, not a quirk.

Where it works (and doesn’t)

Verified on Llama 3.1 8B, Qwen3 8B, and Mistral 7B through Ollama. But here is the honest caveat: if your runtime supports constrained decoding, use it. Ollama’s format parameter and llama.cpp’s GBNF grammars make invalid JSON mechanically impossible, which no prompt can promise. This prompt is for when you cannot constrain (some frontends, some APIs), and as the belt to constrained decoding’s suspenders: the grammar guarantees valid JSON, the prompt keeps the values honest.

The rules make you feel safe. The example does the work.