arxiv:2611.05036

ToolSpace: Scaling Tool-Centric Environment Synthesis for Long-Horizon Agents

Published on Nov 3

· Submitted by

KABI on Nov 4

Renmin University of China

Upvote

Authors:

Guanting Dong ,

Xin Deng ,

Jiejun Tan ,

Abstract

ToolSpace constructs realistic multi-environment tool-use benchmarks by mining execution traces, clustering tool compositions, and validating tasks through replayable state-aware random walks.

Generated by thinkingmachines/Inkling-Small

Large language models are deployed as long-horizon agents that compose diverse tools across MCP services into executable workflows. Existing MCP environment-scaling methods are data-centric: they first synthesize a database and scenario, then derive tools and tasks from them. As a result, tool interfaces may not match real ones, tool compositions are not grounded in execution, and tasks rarely span more than one environment. We propose ToolSpace, a tool-centric framework that reverses this order. ToolSpace mines data, state, and retry dependencies from adjacent calls in execution traces of real MCP services and open agent corpora, using deterministic rules rather than LLM judgment, and clusters the resulting weighted tool-composition graph into environments. Databases are synthesized only for state-mutating environments, while query and stateless tools replay recorded returns. ToolSpace then performs state-aware random walks with rollback and decompiles each chain into a natural-language task with assertion-based checks, keeping only samples that replay successfully from the initial state. Applied to AutomationBench and tau3-Bench Banking, the pipeline produces harder Pro versions on which 7 frontier models drop by up to 38.2 points. Across seven long-horizon tool-use benchmarks, ToolSpace raises Qwen3.6-35B-A3B from 23.0 to 41.0 average score (43.2 with RL) and GLM-5.2 from 44.6 to 56.6, on par with Claude Opus 5. Ablations show that execution-witnessed edges clearly outperform LLM-judged ones, and performance keeps improving with data and environment scale.

View arXiv page View PDF Project page GitHub 34 Add to collection

Community

dongguanting

Paper author Paper submitter 2 days ago

Happy to share our latest work on agent environment synthesis:

🌟 ToolSpace: Scaling Tool-Centric Environment Synthesis for Long-Horizon Agents

Most existing environment synthesis methods are data-centric: they first build the underlying database, then synthesize tools and tasks. Tool interfaces easily drift from reality, and it is hard to form tasks that span environments.

ToolSpace takes the opposite approach. It first collects tools from real call traces, then chains real tool sets into a tool universe, and finally synthesizes the underlying database on demand. This preserves authenticity while flexibly generating verifiable tasks across any environment × any tool.

💡 Key methods
1️⃣ Tool universe exploration: mine tool dependencies from real traces and link them into a tool-world graph; synthesize databases only for environments that change state.
2️⃣ Verifiable task synthesis: walk the graph with state-aware traversal and backtracking, decompile executable tool chains into tasks and verification scripts, and keep environment state continuous. The same chain can yield high-quality SFT data and mixed-environment RL data.
3️⃣ Harder evaluation benchmarks: ToolSpace seamlessly upgrades two frontier benchmarks into AutomationBench Pro and τ³-Bench Banking Pro. Strong existing models all score below 40, setting a harder bar for the field.

📊 Experiments
✅ We evaluate ToolSpace-8B, 35B, and 744B on 7 long-horizon tool benchmarks. ToolSpace-744B averages 56.6, on par with Claude Opus 5.
✅ ToolSpace-744B: AutomationBench 54.7, Harvey LAB 27.5, SkillsBench 67.8, OfficeQA Pro 67.1, JobBench 57.5.
✅ Agent performance scales as environment and data scale grow.

We welcome discussion and feedback. Code and models are open-sourced 👏👏👏

📄 Paper: https://arxiv.org/pdf/2611.05036
🌐 Project page: https://toolspace-agent.github.io/
💻 GitHub: https://github.com/dongguanting/ToolSpace
🤗 Hugging Face: https://huggingface.co/collections/dongguanting/toolspace

librarian-bot

1 day ago

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

kutomanovhennadii

23 minutes ago

Hello. I was interested in the part of your paper where you compare the standard and Pro versions of AutomationBench and τ³-Bench Banking. On Pro the models make noticeably more tool calls, the quality of all seven models drops, and you attribute this to longer and more frequent tool orchestration.

If the drop comes from a growing load on the model, that load should leave a characteristic signature. While the load is below a model's limit, quality holds, and the models differ above all in their strength. Once the load exceeds the limit of each of them, all models start losing quality and sink towards the same floor, which is now set by the task itself rather than by the model. A strong model then has far to fall, while a weak one has almost nowhere left to go, so the strong models lose more and the gap between the models narrows. This, it seems to me, is exactly what your Table 2 shows for AutomationBench Pro. On the standard version the strongest and the weakest models are about 31 points apart, and on Pro about 8: the best model on the standard version loses 38 points, the weakest only 11, and all seven end up in a band from 18 to 26. This matters for a benchmark. When the results of all models lie this close together, a difficulty level like this distinguishes the models less well than the standard version does, and it shows that the difficulty has gone beyond the models' limits rather than which of them is stronger. Of course, convergence does not prove overload but is only consistent with it: the floor may also come from partial credit for easy assertions or from environments unfamiliar to the models. Nor would I read anything into the changes in ranking: the order of the models on Pro is within random variation, and the standard results are taken from the public leaderboard.

Such a limit can be described as the capacity of a model. A study of the behavior and degradation of language models under controlled load (arXiv:2610.36095) varies the load separately along several axes: the length of the task, the number of simultaneous instructions and their density, that is, how constantly they have to be applied as the work proceeds. It shows that, in the measured model, quality falls further as each of these loads grows, and that each axis has its own law. Capacity is derived from these laws as well: for example, instruction capacity is the number of simultaneous instructions that a model can follow at a given level of quality.

In your work these axes seem to be directly visible. The length of the tool-call chain corresponds to the axis of task length. The number of environments and distinct tools corresponds to the axis of the number of simultaneous instructions: each environment adds its own requirements. Density is high in your tasks too, because almost every step depends on earlier ones: it takes values from their outputs or requires the environment to be in the right state. On AutomationBench Pro it is mainly the chain length that grew, and the results converged. On τ³-Bench Banking Pro the number of calls rose only a little, yet the drop is large as well, and I would suppose that there the number of environments is what makes itself felt. This is only a conjecture: in the study mentioned above the laws were measured on simple word-substitution rules, and their transfer to tool-call chains has not been shown. But your generator sets the chain length and the number of environments separately and could test this.