# How we scaled Helena's tools to tens of thousands | Enrich Labs > Four architectures we tried for an AI agent's tool surface, what broke each one, and where we ended up: connection filtering, entity consolidation, progressive loading with skills, and meta-tools for the long tail. _Source: https://www.enrichlabs.ai/engineering/scaling-agent-tools_ --- ## The problem: too many tools Helena, our AI marketing agent, can run ad campaigns on Meta, schedule posts on Instagram, pull analytics from Google, send emails through Klaviyo, and manage content across CMS platforms. Today she has access to a couple hundred native tools across 50+ integrations, and through third-party connectors, tens of thousands more. As we added more integrations, the number of tools we shipped to the model on every chat turn kept growing. **Tool selection accuracy was degrading. Latency and cost were climbing.** At the same time, users kept asking us to add more integrations. This is the story of four architectures we tried, what broke each one, and where we ended up. Phase 1 ## One tool per action, filtered by connection When we started on our agent harness back in mid-2025, we did the obvious thing. Every action has its own tool: `get_shopify_products`, `create_meta_ad_campaign`, `update_meta_campaign_status`, and so on. Clean and focused. As we added Google Ads, TikTok, Klaviyo, HubSpot, and more, it grew past fifty tools within a few months. **That’s when we started noticing the agent picking the wrong tools more often.** A user would ask “how are my Meta campaigns performing?” and the model would call `get_meta_ad_performance` instead of `get_meta_campaign_info`. The tools were all named clearly, but at scale, “clearly named” wasn’t enough. The model was drowning in options. The first fix was straightforward: check which platforms the user has actually connected via OAuth, and strip out tools for everything else. A Shopify-only user went from 50+ tools to ~25. But this only fixed the floor, not the ceiling. Power users, the ones who had legitimately connected many integrations, still had far too many tools. Connection filtering alone couldn’t solve this; we needed to reduce the tools themselves. Phase 2 ## Consolidate by entity In early 2026, we shifted from one tool per CRUD verb to one tool per entity: // Before (5 tools) // After (1 tool) create\_meta\_ad\_campaign manage\_meta\_ad\_campaign get\_meta\_campaign\_info action: create | get | update | list update\_meta\_campaign\_status delete\_meta\_campaign get\_meta\_ad\_performance We applied this across the board; today, **64 of our 192 tool files follow the `manage_*` pattern.** **But consolidation has a sweet spot.** Too granular multiplies tool count. Too coarse bloats schemas and confuses the model. The rule: **group by entity, not by platform, not by verb.** `manage_meta_ad_campaign` is one entity with all its operations. `manage_meta_ad_set` is a separate entity, separate tool. Don’t collapse them further. Entity consolidation helped, but power users still had more tools than the model could reliably select from. Phase 3 ## Progressive loading with skills Even after connection filtering and entity consolidation, our power users still had too many tools. When we measured across our paid accounts, the top 10% had 51+ plan-filtered tools, with a max of 88. The rest sat at ~24. We noticed tool selection accuracy degrading roughly past ~50 tools, which became our rule of thumb. This was the shift that actually solved the problem. But first, some context on skills. ### Skills: composable capabilities Before we built progressive loading, we’d already introduced a **skills** system following the [agentskills.io](https://agentskills.io/specification) spec. A skill is a markdown file (YAML frontmatter + instructions + optional assets) that the agent can load on demand via `load_skill('skill-name')`. We use skills for reusable content workflows: SEO article writing, image editing, motion graphics, and so on. Skills let us extend the agent’s capabilities without shipping new tool code. This turned out to be the foundation for progressive loading. ### The insight On any given turn, the user is usually talking about one integration. If they ask “analyze my Shopify orders from last week,” they need Shopify tools, not the other 40 for Meta, Google, TikTok, and Klaviyo. Why send all of them? When a user’s plan-filtered tool count exceeds 50, we start with **core tools only** (~24: web search, image generation, content calendar, etc). Connected integrations are listed in the context as a menu: AVAILABLE INTEGRATIONS (call load\_skill to activate): - shopify: \[get\_shopify\_products, shopify\_orders, manage\_shopify\_blogs, ...\] - metaAds: \[create\_meta\_ad\_campaign, get\_meta\_ad\_performance, ...\] - klaviyo: \[manage\_klaviyo\_campaigns, klaviyo\_performance\_report, ...\] When the model needs Shopify, it calls `load_skill('shopify')`. The tools are added to the session and become callable **within the same turn**, not the next one. The conversation loop re-reads session metadata at the top of each iteration, widening the tool array before the next model call. Once loaded, integrations persist for the session. Because we’d already built `load_skill` for content skills, we didn’t need a new mechanism. Integration tool groups are just another thing to load on demand. One tool, two uses: load\_skill('shopify') // loads integration tools load\_skill('seo-article-writing') // loads skill instructions A user with 70 plan-filtered tools now starts a turn with 24. **First-turn schema overhead dropped by roughly 60% for power users.** We ran A/B quality evals to verify that progressive loading didn’t hurt response quality. It improved it, because the model was choosing from 24 focused tools instead of 70 scattered ones. Phase 4 ## Meta-tools for the long tail Everything above deals with our ~50 native integrations. But users kept asking: “Can Helena work with my Linear board?” “What about Dropbox?” “Can you connect to my CRM?” Building a native integration for each (OAuth flow, API wrapper, tool definitions, tests) takes at least half a day per platform, not counting the time for manual app reviews. That doesn’t scale to the long tail of thousands of integrations users actually use. We solved this by integrating third-party integration platforms like Pipedream and Composio. These platforms provide OAuth flows and API wrappers for thousands of apps out of the box. For each third-party app a user connects, we dynamically generate and register exactly two meta tools: integration\_trello\_find\_tools(query: "create card") → searches actions, returns top 8 with typed schemas integration\_trello\_call\_tool(tool\_name: "TRELLO\_CREATE\_CARD", arguments: {...}) → executes the action Two tools per integration, not one per entity. A user with 5 third-party integrations adds 10 tools, manageable within our budget. Unlike native integration tools (which load via `load_skill`), meta-tools are dynamically generated and registered at session start based on which third-party apps the user has connected. Under progressive loading, they’re included alongside the core tools when appropriate, so they don’t bypass the loading threshold. A meta-tool call takes two model round-trips where a native call takes one: ![Meta-tool call flow: user message to find\_tools to action discovery to call\_tool to proxy through the provider to result, compared against the shorter native tool flow that skips discovery](cs-assets/diagram-metatool-flow.png) A meta-tool call takes two model round-trips (find, then call); a native tool takes one direct call. `find_tools` ranking is deliberately simple: lexical token overlap, not embeddings. We tokenize the query and each action’s key, name, and description, weight a key/name hit above a description hit, and return the top 8 with their schemas. Tokenizing (rather than substring matching) is what lets “list boards” find `trello-find-boards` even though the words aren’t contiguous. It’s not semantic, but token overlap over a few-hundred-action catalog has been good enough to ship, with no index to build or keep in sync as the provider’s catalog changes. ### Why per-app meta-tools, not a single tool search? An alternative approach, similar to Anthropic’s tool search tool pattern, would be a single `tool_search` tool that searches across all available actions. We chose per-app meta-tools instead for two reasons: **Schema transfer cost.** A tool search tool requires uploading all available tool schemas to the provider upfront so it can index and search them. With thousands of actions across a user’s connected apps, that’s a large payload on every request. Per-app meta-tools avoid this: the schemas live on our side, and `find_tools` returns only the top matches for one app at a time. **Precision.** Even scoped to a user’s connected apps (not all apps globally), a single search across hundreds of actions from multiple integrations returns noisy results. Per-app scoping means the model decides which app first, then searches within it. The result set is small and relevant. It also keeps us provider-neutral, since tool search is a provider-specific feature. ## The status quo of third-party tool quality While we worked our way through integrating third-party platforms, we noticed gaps in how their tools are designed for AI agents. **Too many tools.** This is the biggest issue. These platforms expose hundreds of actions per app (as reported by their APIs): Exhibit 1 * * * Actions exposed per app, by platform Integration Composio tools Pipedream tools GitHub 823 36 HubSpot 304 100 Trello 345 31 Mailchimp 271 30 Action counts as reported by each platform’s API. Source: Enrich Labs internal data, July 2026. Enrich Labs * * * **Missing schemas.** Some platforms’ actions lack typed input schemas entirely. The model has to guess field names, and guesses wrong often enough to matter. **Machine-generated names.** Names like `GITHUB_ISSUES_LIST_COMMENTS_FOR_A_REPOSITORY_BELONGING_TO_THE_AUTHENTICATED_USER` are verbose and hard for models to parse. Compare our native equivalent: `manage_github_issues` with action `list`. **Inconsistent parameters.** One app uses `board_id`, another `boardId`, another `id_board`. The model can’t transfer patterns across integrations. Well-designed tool definitions are the difference between an agent that works reliably and one that guesses. This is why we maintain hand-crafted native integrations for our core platforms and route only the long tail through meta-tools. Native tools have concise descriptions, tight schemas, and consistent naming. ## Where we are now The system works in layers, trading polish for breadth. Exhibit 2 * * * Three layers, from hand-crafted to long-tail Layer Integrations Tools Accuracy When it loads Core tools N/A ~24 Highest Always Native integration tools ~50 platforms A couple hundred High On demand via `load_skill` Third-party meta-tools 3,000+ apps Tens of thousands Good enough Per connected app Source: Enrich Labs internal data, July 2026. Enrich Labs * * * ![Helena's three tool layers stacked as a pyramid: core tools at the narrow top (~24, always present), native integration tools in the middle (~50 platforms, 192 tools, loaded on demand), and third-party meta-tools at the wide base (3,000+ apps, tens of thousands of actions). Polish increases toward the top, breadth toward the bottom.](cs-assets/diagram-tool-layers.png) Polish increases toward the top of the stack; breadth increases toward the bottom. 1. **Core tools** are always present: web search, image generation, content calendar, memory. 2. **Native integration tools** load on demand when the model needs them. Hand-crafted schemas, consistent naming, high selection accuracy. Skills (the `load_skill` mechanism from Phase 3) are the utility that makes this layer work. 3. **Meta-tools** provide long-tail coverage. Broader but rougher: the model finds and calls actions through a search-then-execute pattern. ## What’s next We’re currently working on improving meta-tool search ranking so the model finds the right action faster, and collaborating with integration platforms on better tool schemas. We’re also exploring ways to let the agent learn which tools it uses most often per user and pre-load them. The layered approach (core, native, meta-tool, skill) has held up well as we’ve scaled from 16 tools to tens of thousands, and **the next frontier is making each layer smarter, not just broader.** _We’re [Enrich Labs](https://enrichlabs.ai). Helena is our AI marketing agent. If building AI agents that work in real-world environments is the kind of problem that excites you, we’d love to talk._