Juris ranked #2 globally on IBM VAKRA for tool selection

Listen
Share

TL;DR. Juris is a deliberately boring agent: it narrows thousands of candidate APIs to a shortlist, lets a ~35B mixture-of-experts model choose one tool, fixes the arguments with a few rules, executes it, and turns the raw payload into the answer with code rather than letting the model rewrite it. We announced it as #2 globally for tool selection on IBM’s VAKRA benchmark. This post opens the whole machine: every stage, every rule, every number in the public run, and what those numbers do and do not prove.

17 / 17domains covered in capability 2
1,597queries run end to end
94.99%completed without an error or timeout
62.89 saverage time per query

What capability 2 asks of an agent

VAKRA is IBM’s benchmark for tool-using agents: a pool of API-style tools, grouped into databases, and natural-language questions that must be answered by calling the right tool with the right arguments. As described when we announced the result, the full benchmark spans roughly 8,000+ APIs across about 62 domains, and its tasks come in four capabilities. Capability 2 is the “dashboard APIs” setting: each question maps to one of a database’s ready-made endpoints (“which district is the community area Rogers Park grouped into?”), so it is, in the words of the code, “primarily a tool-selection problem”.

That framing explains the whole design. If the question is one lookup away from a single endpoint, then nearly every point is won or lost in three places: choosing the endpoint, getting the arguments exactly right, and returning the value in the shape the scorer expects. Multi-step reasoning is not where the points are, and a free-running agent loop gives opportunities to lose them.

The idea: take the model out after the choice

A standard ReAct agent alternates between thinking and calling tools until it decides it is done. For open-ended tasks that is exactly right. For capability 2 it is a liability: the loop often retrieves the correct tool output and then spends further model turns re-expressing it, and the scorer wants the bare value. The Juris class docstring says this in one sentence: the stock loop “often gets the right tool output, then loses score by paraphrasing the answer.”

Stock ReAct loop versus the Juris single-pass designTwo lanes: a ReAct loop with several model calls ending in a paraphrase, versus Juris with one model call and a code-built answer.Stock ReAct loop (illustrative)LLMpick a tooltoolrun itLLMread resulttoolmaybe againLLMparaphrasefree-form text; the right value can be rewritten awayJuris, capability 2LLMpick ONE tool, fill argstoolexecuted oncecodeserialise the payloadNo second look at the tool output by the model, so nothing to paraphrase.
Figure 1 The design bet. In the repository’s own words the stock loop “often gets the right tool output, then loses score by paraphrasing the answer”, so Juris takes the model out after the choice of tool. Source: github.com/amethystani/juris-vakra-cap2-submission · code/agent_interface.py · Capability2DeterministicAgent docstring

Juris therefore splits the work along a hard line. The language model is used for exactly one judgement, which tool and with what arguments, and everything on either side of that judgement is deterministic: code narrows the options before it, and code formats the result after it. The rest of this post walks that line stage by stage.

Juris end-to-end flowA query passes through a handcrafted override check, a hybrid tool shortlist, a single forced tool call, argument canonicalisation, tool execution and code-built answer serialisation, with a fallback cascade.JURIS · capability 2 · one queryNatural-language query0 Handcrafted override7 regex rules · exact templates1 Shortlist the tool spaceMiniLM cosine + 0.9 × lexical · top 1282 Force ONE tool callbind_tools · one planner call3 Canonicalise argumentscounty · district · school type · nationality4 Execute + serialiseanswer built by code, not by the LLMAnswer stringDirect tool callskips stages 1–2rule hitJSON-plan fallback2nd LLM call · parse JSONno callLast resort: ReAct1 iteration only(recursion limit = 3)errorRunner timeout300 s hard limitno retriesSolid = main path · dashed = fallbacks. One planner call, at most one executed tool call.
Figure 2 The whole agent on one page. The only place a language model is consulted for a decision is stage 2; everything before it narrows the choices and everything after it is plain code. Source: github.com/amethystani/juris-vakra-cap2-submission · code/agent_interface.py, code/tool_shortlister.py

Stage 0: seven handcrafted overrides

Before any model is involved, the query is tested against seven regular-expression and keyword rules. If one fires, the candidate list collapses to that single tool, and whatever the model says is overridden with the rule’s call.

#Trigger in the queryTool calledArguments
1“please list the names of the characters in the movie ‹T›”get_character_names_by_title / …_by_movietitle
2“what kind of “wi-fi” does yelp business no.‹N› have”get_business_attribute_valuebusiness_id = N, attribute_name = “wi-fi”
3“which district is the community area ‹X› grouped into”get_community_area_side_by_namecommunity_area_name = X
4“how many more games were sold on game platform id A than on game platform id B in region id R”get_sales_differenceplatform_id_1, platform_id_2, region_id
5“tell the number of games whose publisher id is N”get_distinct_game_idspublisher_id = N
6the word “borderless”get_cards_by_border_colorborder_color = “borderless”
7the phrase “powerful foils”get_cards_with_foil_and_regular_ids(none)
The seven handcrafted overrides. Each only fires if the named tool exists in the current domain’s catalogue. Source: github.com/amethystani/juris-vakra-cap2-submission · agent_interface.py lines 739–839

Stage 1: shortlist the tool space

Some domains expose far more endpoints than a prompt can comfortably hold, and a long list is where a model starts to pick “plausible” instead of “correct”. When a domain has more than k = 128 tools, the shortlister keeps the 128 most relevant. It scores every tool two ways and adds the scores.

How the hybrid shortlister scores a toolQuery and tool text are embedded and compared by cosine similarity, and separately scored by lexical overlap with synonym folding and bonuses; the two scores are combined as cosine plus 0.9 times lexical.QueryTool text“name: description”MiniLM embeddingper questionMiniLM embeddingonce per domaincosine similarityL2-normalised vectorsQuery + toolnormalised tokens(synonyms folded)Lexical score|Q ∩ T| / |Q|+ hand-tuned bonusesscorecosine + 0.9 × lexical“query_*” tools pinned firstthen top-k by score (k = 128)Shortlisting only runs when a domain has more than k tools; otherwise every tool is shown to the model.
Figure 3 Hybrid shortlisting. The 0.9 weight lets a strong lexical match (e.g. the query says “race year id” and the tool has exactly those words) out-vote a fuzzy embedding match. Source: github.com/amethystani/juris-vakra-cap2-submission · code/tool_shortlister.py lines 22–211

The embedding half

Each tool is turned into the string name: description and embedded once per domain with all-MiniLM-L6-v2; the query is embedded per question; the two are compared by cosine similarity. Any tool whose name starts with query_ is pinned into the shortlist regardless of score, so generic query endpoints are never dropped.

The lexical half

The lexical score is plain token overlap, |Q ∩ T| / |Q|, not BM25 or TF-IDF, but computed after a hand-written normaliser that folds synonyms (movies, films and titles become movie, title; ids, numbers and “no” become id, number; actors, people and cast become actor, person; and so on) and strips plural “s” from longer tokens. On top of the overlap sit a few hand-tuned bonuses for patterns the benchmark’s questions use heavily.

Query contains all ofTool name containsBonus
character + (movie | title)“character” and (“movie” | “title”)+0.35
race + year + idrace, year, id+0.45
driver + nationality + codedriver, nationality, code+0.45
website + county“website”+0.25
“percentage”“percentage”+0.20
“count”“count”+0.20
Hand-tuned lexical bonuses on top of the token overlap. Source: github.com/amethystani/juris-vakra-cap2-submission · tool_shortlister.py lines 104–139
code/tool_shortlister.py · shortlist()
similarities = (tool_embs_normed @ query_emb_normed.T).squeeze()
lexical_scores = np.array(
    [self._lexical_score(query, tool) for tool in candidates], dtype=float)
combined_scores = similarities + (0.9 * lexical_scores)

top_indices = np.argsort(combined_scores)[::-1][:remaining_slots]
return pinned + [candidates[i] for i in top_indices]

The 0.9 multiplier is the interesting number. It lets an exact vocabulary match outweigh an embedding that is merely “in the neighbourhood”, which matters in dashboard APIs where two endpoints often share a topic but differ by one noun (races by year versus race results by year and id). The repository contains no recall measurement or with/without ablation of the shortlister, so how much it contributes to the final result is not something we can quote.

Stage 2: force exactly one tool call

The model receives a two-message conversation (a system prompt and the question): no history, no few-shot examples. The prompt is short and prescriptive, and the tool list in it is compressed to one line per tool: name, a 220-character description, and each argument with its type, whether it is required, and a 180-character description.

code/agent_interface.py · _build_system_prompt (rules block)
You are solving VAKRA capability 2 (dashboard APIs).
Choose the single best tool that directly answers the query.
Rules:
- Make at most one tool call.
- Do not answer from memory.
- A direct natural-language answer without a tool call is invalid.
- Do not add explanations, summaries, or prose.
- Use the most specific tool available.
- Match the requested output field exactly. If the user asks for ids, codes,
  names, dates, times, counts, or percentages, prefer a tool that returns
  that same kind of field.
- Fill arguments exactly from the query.
- Do not call get_data; the universe is already selected.

The call itself is a single bind_tools invocation. Only the first returned tool call is ever executed, even if the model emits several.

code/agent_interface.py · run()
try:
    planner = self._llm.bind_tools(active_tools)
    ai_msg = await planner.ainvoke(lc_messages)
    tool_calls = getattr(ai_msg, "tool_calls", None) or []
except Exception as exc:
    logger.warning("cap2 planner failed, falling back to ReAct: %s", exc)
    tool_calls = []
    ai_msg = AIMessage(content="")
# only tool_calls[:1] is ever executed

If the model replies in prose instead of calling a tool, which the prompt declares invalid, a second request goes out asking for a bare JSON object {"name": …, "arguments": {…}}. The reply is parsed leniently (fenced or unfenced JSON, or a regex over the text) and accepted only if the named tool is in the shortlist.

Stage 3: canonicalise the arguments

Endpoints are brittle about strings. “Fresno County” returns nothing from an endpoint that stores “Fresno”; “state schools” matches nothing in a column that stores “State School”. Rather than hoping the model remembers each quirk, Juris normalises every string argument after the call is chosen.

Argument name contains…RuleIllustrative input → output
countystrip a trailing “ county”“Fresno County” → “Fresno”
districtstrip a trailing “ region”“Bay region” → “Bay”
educational_option_typesingularise “… schools”; ensure “School”; Title Case“continuation schools” → “Continuation School”
nationalitymap through a 32-entry alias table“usa” → “American”, “holland” → “Dutch”
(any string)collapse whitespace“ New Zealand ” → “New Zealand”
Argument canonicalisation. The examples are illustrative inputs run through the rules, not logged queries. Source: github.com/amethystani/juris-vakra-cap2-submission · agent_interface.py lines 595–625
code/agent_interface.py · _canonicalize_tool_args (excerpt)
if "county" in key_lower and clean_value.lower().endswith(" county"):
    clean_value = clean_value[:-7]
if "district" in key_lower and clean_value.lower().endswith(" region"):
    clean_value = clean_value[:-7]
if "nationality" in key.lower():
    mapped = cls._NATIONALITY_ALIASES.get(clean_value.casefold())

The nationality table maps 32 spellings and aliases (usa, us, america → American; england, britain → British; holland → Dutch; new zealand → New Zealander, and so on) because Formula 1 style endpoints expect demonyms while questions say countries.

Stage 4: build the answer with code

This is the stage that earns the design its name. After the tool runs, its payload is never shown to the model again. A function decides the answer string from the payload and from a few intents detected in the question:

  1. Parse the payloadDecode JSON strings and unwrap the benchmark’s one-element [{"text": …}] envelopes, recursively.
  2. Detect intentCount (“how many”, “tell the number of”), list (“list out”, “what are”, questions starting with “which” or “who”), or identifier (asks for ids, codes, names, dates, urls).
  3. Unwrap shapesSingle-key dictionaries are unwrapped; a one-row result drops empty cells and, depending on the question word, takes the first cell (“which / who”), the second (“where”), or the last numeric cell (“how old”).
  4. SerialiseCounts over lists become len(list); lists become comma-joined first cells; one-element lists become that element; error dictionaries become an empty string; anything else is dumped as sorted-key JSON.

A model asked to “answer in your own words” will happily write “There are 3 schools in Fresno.” when the scorer wanted 3. Code that returns len(result) cannot make that mistake.

When things go wrong: the fallback cascade

  1. Planner raised an exception → treated as “no tool call” and sent to the JSON-plan fallback.
  2. No tool call returned → the JSON-plan fallback described above (the metadata tags such answers capability2_json_fallback).
  3. Tool execution failed in either path → logged, then fall through.
  4. Last resort → the stock ReAct agent, capped at one iteration; if it made a tool call, its answer is replaced by the code-built serialisation of the tool result.
  5. Runner level → a hard 300-second wall on every query; a timeout is recorded as an error, and there are no retries.

The cap on the last-resort agent matters for reading the failure log. A ReAct agent with max_iterations = 1 gets a graph recursion limit of 1 × 2 + 1 = 3, which is exactly the message in the five non-timeout failures: “Recursion limit of 3 reached without hitting a stop condition.” So all five are the fallback agent running out of its deliberately tiny budget.

The run, by the numbers

1,517queries returned an answer
80failed executions (5.01%)
75of those: “Agent timed out after 300 seconds”
5of those: recursion limit of 3
Failed executions per domainFailed executions by domain, from thrombosis_prediction with 13 down to movie with 0.thrombosis_prediction13 of 159 queries · 91.8% completed13financial11 of 102 queries · 89.2% completed11toxicology9 of 117 queries · 92.3% completed9card_games8 of 182 queries · 95.6% completed8chicago_crime6 of 78 queries · 92.3% completed6public_review_platform6 of 73 queries · 91.8% completed6superhero5 of 109 queries · 95.4% completed5debit_card_specializing4 of 59 queries · 93.2% completed4formula_14 of 152 queries · 97.4% completed4movielens4 of 44 queries · 90.9% completed4movie_33 of 103 queries · 97.1% completed3california_schools2 of 89 queries · 97.8% completed2movies_42 of 77 queries · 97.4% completed2european_football_21 of 121 queries · 99.2% completed1simpson_episodes1 of 41 queries · 97.6% completed1video_games1 of 63 queries · 98.4% completed1movie0 of 28 queries · 100.0% completed0
Figure 4 Where it fails. Three domains (thrombosis_prediction, financial, toxicology) account for 33 of the 80 failures. Failure type per domain is not recorded; only the totals (75 timeouts, 5 recursion-limit errors) are. Source: github.com/amethystani/juris-vakra-cap2-submission · submission_metadata.json · per_domain
Where the wall-clock time wentA single bar split into time spent on timed-out queries and time on everything else.all other queries · ≈21.6 h22%≈ 27.9 h of model time for 1,597 queries (62.89 s × 1,597)75 timeouts × 300 s = 22,500 s ≈ 6.25 hDerived from the run metrics (arithmetic, not logged). It shows what a single hard timeout costs, not a measured profile.
Figure 5 About a fifth of all compute went to the 75 queries that hit the 300 s ceiling, even in a one-tool design. The long tail, not the average, is the cost. Source: derived from submission_metadata.json
Show the full per-domain table (17 domains)
DomainQueriesSucceededFailedCompleted
california_schools8987297.75%
card_games182174895.60%
chicago_crime7872692.31%
debit_card_specializing5955493.22%
european_football_2121120199.17%
financial102911189.22%
formula_1152148497.37%
movie28280100.00%
movie_3103100397.09%
movielens4440490.91%
movies_47775297.40%
public_review_platform7367691.78%
simpson_episodes4140197.56%
superhero109104595.41%
thrombosis_prediction1591461391.82%
toxicology117108992.31%
video_games6362198.41%
Total1,5971,5178094.99%

Two readings are worth making explicit. First, the failures are overwhelmingly timeouts (93.75% of them), not wrong tools or malformed calls: even when everything is routed to a single call, a local 35B model on one 24 GB card occasionally takes more than five minutes on a hard question. Second, the failures are concentrated in a few domains: thrombosis_prediction (13 failures out of 159 queries), financial (11 of 102), toxicology (9 of 117), card_games (8 of 182). Whether that reflects larger catalogues, longer schemas or harder questions cannot be told from totals alone, because the log does not record failure type per domain.

What this does and does not show

Lessons

  1. Decide where the model is allowed to be wrong. Letting it make one bounded decision, and nothing else, turned a sprawling failure surface into a narrow one that rules could patch.
  2. Format is part of correctness. A right value in the wrong shape scores zero, and shape is something code can guarantee and a model cannot.
  3. Pair retrieval signals. Embeddings find the neighbourhood; exact vocabulary picks the house. Weighting them (here 1 : 0.9) is a cheap and legible way to get both.
  4. Keep the benchmark-specific hacks honest and separate. The seven overrides sit in their own stage so anyone reading the code can see exactly how much of the result rests on template knowledge.
  5. Budget for the tail. Averages hide the cost: 75 timeouts ate roughly a fifth of the compute. A tighter timeout with a cheaper fallback is the obvious next experiment.

Reproduce and read the code

The submission is public. The files are drop-in replacements for the same-named files in an IBM VAKRA checkout (agents/agent_interface.py, agents/components/tool_shortlister.py, benchmark/benchmark_runner.py), so they need that tree and its MCP configuration to run.

inside an IBM/vakra checkout (inferred from the runner’s CLI and the metadata)
python benchmark_runner.py --capability_id 2 --provider ollama \
    --model qwen3.6:35b-a3b --top-k-tools 128 --output output/capability_2_full
All articles

Research and building cool stuff

© 2026, Animesh Mishra

GitHub|LinkedIn

New Delhi, India