Juris ranked #2 globally on IBM VAKRA for tool selection
TL;DR. Juris is a deliberately boring agent: it narrows thousands of candidate APIs to a shortlist, lets a ~35B mixture-of-experts model choose one tool, fixes the arguments with a few rules, executes it, and turns the raw payload into the answer with code rather than letting the model rewrite it. We announced it as #2 globally for tool selection on IBM’s VAKRA benchmark. This post opens the whole machine: every stage, every rule, every number in the public run, and what those numbers do and do not prove.
What capability 2 asks of an agent
VAKRA is IBM’s benchmark for tool-using agents: a pool of API-style tools, grouped into databases, and natural-language questions that must be answered by calling the right tool with the right arguments. As described when we announced the result, the full benchmark spans roughly 8,000+ APIs across about 62 domains, and its tasks come in four capabilities. Capability 2 is the “dashboard APIs” setting: each question maps to one of a database’s ready-made endpoints (“which district is the community area Rogers Park grouped into?”), so it is, in the words of the code, “primarily a tool-selection problem”.
That framing explains the whole design. If the question is one lookup away from a single endpoint, then nearly every point is won or lost in three places: choosing the endpoint, getting the arguments exactly right, and returning the value in the shape the scorer expects. Multi-step reasoning is not where the points are, and a free-running agent loop gives opportunities to lose them.
The idea: take the model out after the choice
A standard ReAct agent alternates between thinking and calling tools until it decides it is done. For open-ended tasks that is exactly right. For capability 2 it is a liability: the loop often retrieves the correct tool output and then spends further model turns re-expressing it, and the scorer wants the bare value. The Juris class docstring says this in one sentence: the stock loop “often gets the right tool output, then loses score by paraphrasing the answer.”
Juris therefore splits the work along a hard line. The language model is used for exactly one judgement, which tool and with what arguments, and everything on either side of that judgement is deterministic: code narrows the options before it, and code formats the result after it. The rest of this post walks that line stage by stage.
Stage 0: seven handcrafted overrides
Before any model is involved, the query is tested against seven regular-expression and keyword rules. If one fires, the candidate list collapses to that single tool, and whatever the model says is overridden with the rule’s call.
| # | Trigger in the query | Tool called | Arguments |
|---|---|---|---|
| 1 | “please list the names of the characters in the movie ‹T›” | get_character_names_by_title / …_by_movie | title |
| 2 | “what kind of “wi-fi” does yelp business no.‹N› have” | get_business_attribute_value | business_id = N, attribute_name = “wi-fi” |
| 3 | “which district is the community area ‹X› grouped into” | get_community_area_side_by_name | community_area_name = X |
| 4 | “how many more games were sold on game platform id A than on game platform id B in region id R” | get_sales_difference | platform_id_1, platform_id_2, region_id |
| 5 | “tell the number of games whose publisher id is N” | get_distinct_game_ids | publisher_id = N |
| 6 | the word “borderless” | get_cards_by_border_color | border_color = “borderless” |
| 7 | the phrase “powerful foils” | get_cards_with_foil_and_regular_ids | (none) |
Stage 1: shortlist the tool space
Some domains expose far more endpoints than a prompt can comfortably hold, and a long list is where a model starts to pick “plausible” instead of “correct”. When a domain has more than k = 128 tools, the shortlister keeps the 128 most relevant. It scores every tool two ways and adds the scores.
The embedding half
Each tool is turned into the string name: description and embedded once per domain with all-MiniLM-L6-v2; the query is embedded per question; the two are compared by cosine similarity. Any tool whose name starts with query_ is pinned into the shortlist regardless of score, so generic query endpoints are never dropped.
The lexical half
The lexical score is plain token overlap, |Q ∩ T| / |Q|, not BM25 or TF-IDF, but computed after a hand-written normaliser that folds synonyms (movies, films and titles become movie, title; ids, numbers and “no” become id, number; actors, people and cast become actor, person; and so on) and strips plural “s” from longer tokens. On top of the overlap sit a few hand-tuned bonuses for patterns the benchmark’s questions use heavily.
| Query contains all of | Tool name contains | Bonus |
|---|---|---|
| character + (movie | title) | “character” and (“movie” | “title”) | +0.35 |
| race + year + id | race, year, id | +0.45 |
| driver + nationality + code | driver, nationality, code | +0.45 |
| website + county | “website” | +0.25 |
| “percentage” | “percentage” | +0.20 |
| “count” | “count” | +0.20 |
similarities = (tool_embs_normed @ query_emb_normed.T).squeeze()
lexical_scores = np.array(
[self._lexical_score(query, tool) for tool in candidates], dtype=float)
combined_scores = similarities + (0.9 * lexical_scores)
top_indices = np.argsort(combined_scores)[::-1][:remaining_slots]
return pinned + [candidates[i] for i in top_indices]The 0.9 multiplier is the interesting number. It lets an exact vocabulary match outweigh an embedding that is merely “in the neighbourhood”, which matters in dashboard APIs where two endpoints often share a topic but differ by one noun (races by year versus race results by year and id). The repository contains no recall measurement or with/without ablation of the shortlister, so how much it contributes to the final result is not something we can quote.
Stage 2: force exactly one tool call
The model receives a two-message conversation (a system prompt and the question): no history, no few-shot examples. The prompt is short and prescriptive, and the tool list in it is compressed to one line per tool: name, a 220-character description, and each argument with its type, whether it is required, and a 180-character description.
You are solving VAKRA capability 2 (dashboard APIs).
Choose the single best tool that directly answers the query.
Rules:
- Make at most one tool call.
- Do not answer from memory.
- A direct natural-language answer without a tool call is invalid.
- Do not add explanations, summaries, or prose.
- Use the most specific tool available.
- Match the requested output field exactly. If the user asks for ids, codes,
names, dates, times, counts, or percentages, prefer a tool that returns
that same kind of field.
- Fill arguments exactly from the query.
- Do not call get_data; the universe is already selected.The call itself is a single bind_tools invocation. Only the first returned tool call is ever executed, even if the model emits several.
try:
planner = self._llm.bind_tools(active_tools)
ai_msg = await planner.ainvoke(lc_messages)
tool_calls = getattr(ai_msg, "tool_calls", None) or []
except Exception as exc:
logger.warning("cap2 planner failed, falling back to ReAct: %s", exc)
tool_calls = []
ai_msg = AIMessage(content="")
# only tool_calls[:1] is ever executedIf the model replies in prose instead of calling a tool, which the prompt declares invalid, a second request goes out asking for a bare JSON object {"name": …, "arguments": {…}}. The reply is parsed leniently (fenced or unfenced JSON, or a regex over the text) and accepted only if the named tool is in the shortlist.
Stage 3: canonicalise the arguments
Endpoints are brittle about strings. “Fresno County” returns nothing from an endpoint that stores “Fresno”; “state schools” matches nothing in a column that stores “State School”. Rather than hoping the model remembers each quirk, Juris normalises every string argument after the call is chosen.
| Argument name contains… | Rule | Illustrative input → output |
|---|---|---|
county | strip a trailing “ county” | “Fresno County” → “Fresno” |
district | strip a trailing “ region” | “Bay region” → “Bay” |
educational_option_type | singularise “… schools”; ensure “School”; Title Case | “continuation schools” → “Continuation School” |
nationality | map through a 32-entry alias table | “usa” → “American”, “holland” → “Dutch” |
| (any string) | collapse whitespace | “ New Zealand ” → “New Zealand” |
if "county" in key_lower and clean_value.lower().endswith(" county"):
clean_value = clean_value[:-7]
if "district" in key_lower and clean_value.lower().endswith(" region"):
clean_value = clean_value[:-7]
if "nationality" in key.lower():
mapped = cls._NATIONALITY_ALIASES.get(clean_value.casefold())The nationality table maps 32 spellings and aliases (usa, us, america → American; england, britain → British; holland → Dutch; new zealand → New Zealander, and so on) because Formula 1 style endpoints expect demonyms while questions say countries.
Stage 4: build the answer with code
This is the stage that earns the design its name. After the tool runs, its payload is never shown to the model again. A function decides the answer string from the payload and from a few intents detected in the question:
- Parse the payloadDecode JSON strings and unwrap the benchmark’s one-element
[{"text": …}]envelopes, recursively. - Detect intentCount (“how many”, “tell the number of”), list (“list out”, “what are”, questions starting with “which” or “who”), or identifier (asks for ids, codes, names, dates, urls).
- Unwrap shapesSingle-key dictionaries are unwrapped; a one-row result drops empty cells and, depending on the question word, takes the first cell (“which / who”), the second (“where”), or the last numeric cell (“how old”).
- SerialiseCounts over lists become
len(list); lists become comma-joined first cells; one-element lists become that element; error dictionaries become an empty string; anything else is dumped as sorted-key JSON.
A model asked to “answer in your own words” will happily write “There are 3 schools in Fresno.” when the scorer wanted 3. Code that returns len(result) cannot make that mistake.
When things go wrong: the fallback cascade
- Planner raised an exception → treated as “no tool call” and sent to the JSON-plan fallback.
- No tool call returned → the JSON-plan fallback described above (the metadata tags such answers
capability2_json_fallback). - Tool execution failed in either path → logged, then fall through.
- Last resort → the stock ReAct agent, capped at one iteration; if it made a tool call, its answer is replaced by the code-built serialisation of the tool result.
- Runner level → a hard 300-second wall on every query; a timeout is recorded as an error, and there are no retries.
The cap on the last-resort agent matters for reading the failure log. A ReAct agent with max_iterations = 1 gets a graph recursion limit of 1 × 2 + 1 = 3, which is exactly the message in the five non-timeout failures: “Recursion limit of 3 reached without hitting a stop condition.” So all five are the fallback agent running out of its deliberately tiny budget.
The run, by the numbers
Show the full per-domain table (17 domains)
| Domain | Queries | Succeeded | Failed | Completed |
|---|---|---|---|---|
| california_schools | 89 | 87 | 2 | 97.75% |
| card_games | 182 | 174 | 8 | 95.60% |
| chicago_crime | 78 | 72 | 6 | 92.31% |
| debit_card_specializing | 59 | 55 | 4 | 93.22% |
| european_football_2 | 121 | 120 | 1 | 99.17% |
| financial | 102 | 91 | 11 | 89.22% |
| formula_1 | 152 | 148 | 4 | 97.37% |
| movie | 28 | 28 | 0 | 100.00% |
| movie_3 | 103 | 100 | 3 | 97.09% |
| movielens | 44 | 40 | 4 | 90.91% |
| movies_4 | 77 | 75 | 2 | 97.40% |
| public_review_platform | 73 | 67 | 6 | 91.78% |
| simpson_episodes | 41 | 40 | 1 | 97.56% |
| superhero | 109 | 104 | 5 | 95.41% |
| thrombosis_prediction | 159 | 146 | 13 | 91.82% |
| toxicology | 117 | 108 | 9 | 92.31% |
| video_games | 63 | 62 | 1 | 98.41% |
| Total | 1,597 | 1,517 | 80 | 94.99% |
Two readings are worth making explicit. First, the failures are overwhelmingly timeouts (93.75% of them), not wrong tools or malformed calls: even when everything is routed to a single call, a local 35B model on one 24 GB card occasionally takes more than five minutes on a hard question. Second, the failures are concentrated in a few domains: thrombosis_prediction (13 failures out of 159 queries), financial (11 of 102), toxicology (9 of 117), card_games (8 of 182). Whether that reflects larger catalogues, longer schemas or harder questions cannot be told from totals alone, because the log does not record failure type per domain.
What this does and does not show
Lessons
- Decide where the model is allowed to be wrong. Letting it make one bounded decision, and nothing else, turned a sprawling failure surface into a narrow one that rules could patch.
- Format is part of correctness. A right value in the wrong shape scores zero, and shape is something code can guarantee and a model cannot.
- Pair retrieval signals. Embeddings find the neighbourhood; exact vocabulary picks the house. Weighting them (here 1 : 0.9) is a cheap and legible way to get both.
- Keep the benchmark-specific hacks honest and separate. The seven overrides sit in their own stage so anyone reading the code can see exactly how much of the result rests on template knowledge.
- Budget for the tail. Averages hide the cost: 75 timeouts ate roughly a fifth of the compute. A tighter timeout with a cheaper fallback is the obvious next experiment.
Reproduce and read the code
The submission is public. The files are drop-in replacements for the same-named files in an IBM VAKRA checkout (agents/agent_interface.py, agents/components/tool_shortlister.py, benchmark/benchmark_runner.py), so they need that tree and its MCP configuration to run.
python benchmark_runner.py --capability_id 2 --provider ollama \
--model qwen3.6:35b-a3b --top-k-tools 128 --output output/capability_2_full- Code and metadata: github.com/amethystani/juris-vakra-cap2-submission
- The release archive with all 17 prediction files and the manifest is attached to the repository’s
v1release.


