Semantic Tool Filtering Policy¶
Overview¶
The Semantic Tool Filtering policy dynamically filters the tools provided in an API request based on their semantic relevance to the user query. It uses vector embeddings and cosine similarity to select the most relevant tools, optimizing the request before it reaches the LLM. This policy supports both JSON and text-based tool definitions and works with embedding providers such as OpenAI, Mistral and Azure OpenAI.
Features¶
- Semantic similarity matching between user query and tool definitions
- Flexible selection modes: By Rank (top-N) or By Threshold (similarity score)
- Multiple embedding provider support
- Configurable parameters for selection and extraction
How It Works¶
- Extracts the user query and tools from the request using JSONPath.
- Generates embeddings for the query and each tool (using description).
- Calculates cosine similarity between the query and each tool.
- Selects tools based on the configured selection mode (top-N or threshold).
- Replaces the original tools array with the filtered subset before forwarding to the LLM.
How to Use¶
Follow these steps to integrate the Semantic Tool Filtering policy into your AI API:
- Open the API Publisher Portal (
https://<host>:<port>/publisher) - Select your AI API
- Go to Develop > API Configurations > Policies
- Expand Common Policies in the Policy List
- Drag and drop the Semantic Tool Filtering policy into your request flow
- Fill in the required parameters
- Save and Deploy the AI API
Note: An embedding provider must be configured for semantic matching to work. See examples for more details.
Example Policy Configuration¶
Click to expand configuration steps.
Note: An embedding provider must be configured. Add one of the following to $APIM_HOME/repository/conf/deployment.toml:
Generic template:
[apim.ai.embedding_provider]
type = "openai | mistral | azure-openai"
[apim.ai.embedding_provider.properties]
embedding_endpoint = "<embedding-endpoint>"
apikey = "<api-key>"
embedding_model = "<embedding-model>"
MistralAI example:
[apim.ai.embedding_provider]
type = "mistral"
[apim.ai.embedding_provider.properties]
apikey = "<api-key>"
embedding_endpoint = "https://api.mistral.ai/v1/embeddings"
embedding_model = "mistral-embed"
Policy parameter configurations for text based and JSON formatted tools list.¶
Tools as JSON list¶
JSON formatted Tool List Policy parameters Configuration
| Field | Example |
|---|---|
Policy Name |
semantic-tool-filtering |
selectionMode |
By Rank |
Limit |
2 |
queryJSONPath |
$.contents[0].parts[0].text |
toolsJSONPath |
$.tools[0].function_declarations |
userQueryIsJson |
true |
toolsIsJson |
true |
Request Body Example (Tool List as JSON)
{
"contents": [
{
"role": "user",
"parts": [
{
"text": "I'm planning a corporate retreat in Denver for next weekend. Can you find the weather forecast, book a conference room for 15 people, find a highly-rated catering service that offers vegan options, and then email the itinerary to my assistant at [email protected]?"
}
]
}
],
"tools": [
{
"function_declarations": [
{
"name": "get_weather",
"description": "Get current weather and 7-day forecast for a location.",
"parameters": {
"type": "OBJECT",
"properties": {
"location": { "type": "string", "description": "The city and state, e.g. Denver, CO" }
},
"required": ["location"]
}
},
{
"name": "book_venue",
"description": "Reserve a conference room or meeting space.",
"parameters": {
"type": "OBJECT",
"properties": {
"location": { "type": "string" },
"capacity": { "type": "integer", "description": "Number of people" },
"date": { "type": "string", "description": "ISO date format" }
},
"required": ["location", "capacity", "date"]
}
},
{
"name": "find_restaurants",
"description": "Locate dining options based on cuisine and dietary needs.",
"parameters": {
"type": "OBJECT",
"properties": {
"location": { "type": "string" },
"dietary_options": { "type": "array", "items": { "type": "string" }, "description": "e.g. ['vegan', 'gluten-free']" }
},
"required": ["location"]
}
},
{
"name": "calendar_add",
"description": "Create a new event on the user's primary calendar.",
"parameters": {
"type": "OBJECT",
"properties": {
"summary": { "type": "string" },
"start_time": { "type": "string" },
"end_time": { "type": "string" }
},
"required": ["summary", "start_time"]
}
},
{
"name": "send_email",
"description": "Send an email to a specific recipient.",
"parameters": {
"type": "OBJECT",
"properties": {
"recipient": { "type": "string", "description": "Email address" },
"subject": { "type": "string" },
"body": { "type": "string" }
},
"required": ["recipient", "body"]
}
}
]
}
]
}
After filtering, only the top 2 most relevant tools (e.g., get_weather, book_venue) are included in the forwarded request.
Text-Based Tool List¶
Text-based Tool List Policy parameters Configuration
| Field | Example |
|---|---|
Policy Name |
semantic-tool-filtering |
selectionMode |
By Threshold |
Threshold |
0.9 |
queryJSONPath |
$.contents[0].parts[0].text |
toolsJSONPath |
$.contents[0].parts[0].text |
userQueryIsJson |
false |
toolsIsJson |
false |
Request Body Example (Text-Based Tool List)
Note: After extraction, any inline tags used to mark tools or user queries (for example, <toolname>, <tooldescription>, and <userq>) are removed from the text before forwarding.
{
"contents": [
{
"parts": [
{
"text": "## Role: Executive Logistics Orchestrator ... <toolname>get_weather</toolname> (<tooldescription>Get current weather and 7-day forecast for a location</tooldescription>) ... <userq>I'm planning a corporate retreat in Denver for next weekend. ...</userq>"
}
]
}
]
}
Tools and query are extracted from tagged text. Only tools with similarity above 0.9 are included.