> For the complete documentation index, see [llms.txt](https://ai-security-docs.akto.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai-security-docs.akto.io/akto-argus-agentic-ai-security-for-homegrown-ai/connectors/ai-agent-security/litellm.md). # LiteLLM ## Overview LiteLLM is a unified interface for calling 100+ LLM APIs in a consistent format. This integration enables monitoring of API traffic from a LiteLLM proxy and ensures AI-powered applications maintain security standards. The Akto LiteLLM connector provides the following capabilities: * Validates AI requests and responses against security policies * Detects PII, prompt injection, and policy violations * Ingests traffic into Akto for monitoring and analysis * Blocks malicious requests (sync mode) or logs violations (async mode) * Creates per-agent collections based on agent identity ## Prerequisites Before integrating Akto with LiteLLM, ensure the following are in place: * An existing LiteLLM proxy installation (running or ready to configure) * An Akto guardrails service endpoint (URL and authentication token) ## Steps to Connect {% stepper %} {% step %} **Download the Custom Hook** Download the `custom_hooks.py` file to the LiteLLM configuration directory: ```bash # Navigate to the LiteLLM config directory cd /path/to/your/litellm/config # Download the hook file curl -O https://raw.githubusercontent.com/akto-api-security/akto/master/apps/mcp-endpoint-shield/litellm/custom_hooks.py ``` {% endstep %} {% step %} **Configure Environment Variables** Add the following environment variables to the LiteLLM environment (`.env` file or system environment): ```bash # URL of this LiteLLM proxy instance (used as the default collection host) LITELLM_URL=http://your-litellm-instance-url # Akto's Data Ingestion Service URL DATA_INGESTION_SERVICE_URL=http://data-ingestion-service-url # Akto API token used to authenticate to the Data Ingestion Service AKTO_API_TOKEN= # Optional: Operation Mode SYNC_MODE=true # true = block violations, false = async logging only, default true # Optional: timeout (in seconds) for calls to the Data Ingestion Service TIMEOUT=5 # default 5 ``` The connector reads these variables (`custom_hooks.py`): | Variable | Required | Default | Description | | ---------------------------- | -------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `DATA_INGESTION_SERVICE_URL` | Yes | — | Akto Data Ingestion Service endpoint the hook sends traffic and validation requests to. | | `AKTO_API_TOKEN` | Yes | empty | Token sent in the `Authorization` header to the Data Ingestion Service. Obtain from **Akto Argus → Connectors → Setup Guardrail**. | | `LITELLM_URL` | Yes | `http://localhost:4000` | This proxy's URL; its host is used as the default collection name when no agent identity is present. | | `SYNC_MODE` | No | `true` | `true` blocks violations before the LLM call; `false` validates asynchronously (log only). | | `TIMEOUT` | No | `5` | Timeout in seconds for HTTP calls to the Data Ingestion Service. | {% hint style="warning" %} **Note** `SYNC_MODE` determines behavior: * `SYNC_MODE=true`: Requests are validated before being sent to the LLM. Violations block the request immediately. * `SYNC_MODE=false`: Requests proceed immediately. Validation occurs in the background. {% endhint %} {% endstep %} {% step %} **Update LiteLLM Configuration** Edit the `config.yaml` to enable the custom hook: ```yaml model_list: # Existing models litellm_settings: callbacks: [custom_hooks.proxy_handler_instance] # ← Add this line drop_params: true set_verbose: false request_timeout: 600 num_retries: 2 # ... rest of the config ... ``` The required change is adding `callbacks: [custom_hooks.proxy_handler_instance]` to activate the Akto guardrails hook. {% endstep %} {% step %} **Ensure Hook File is Accessible** {% tabs %} {% tab title="Using LiteLLM Directly" %} Ensure `custom_hooks.py` is in the same directory as `config.yaml`, then start LiteLLM with the environment variables from the previous step set: ```bash litellm --config config.yaml ``` {% hint style="info" %} The Akto API Token can be obtained from **Akto Argus -> Connectors -> Setup Guardrail**.\ ![](/files/NW2TeOXo0U8K3EsWqYsn) {% endhint %} {% endtab %} {% tab title="Using Docker" %} Mount the hook file in the docker compose configuration: ```yaml services: litellm: image: docker.litellm.ai/berriai/litellm:main-stable volumes: - ./config.yaml:/app/config.yaml - ./custom_hooks.py:/app/custom_hooks.py environment: - LITELLM_URL=${LITELLM_URL} - DATA_INGESTION_SERVICE_URL=${DATA_INGESTION_SERVICE_URL} - AKTO_API_TOKEN=${AKTO_API_TOKEN} - SYNC_MODE=${SYNC_MODE} # ... rest of config ... ``` {% endtab %} {% endtabs %} {% endstep %} {% step %} **Start LiteLLM** ```bash # Using Docker Compose docker compose restart litellm # Using Docker run docker restart litellm-container # Running directly (stop with Ctrl+C, then restart) litellm --config config.yaml ``` {% endstep %} {% step %} **Verify Integration** Confirm that LiteLLM starts successfully with the hook: ```bash # Check logs for hook initialization docker compose logs litellm | grep GuardrailsHandler # Expected output: # GuardrailsHandler initialized | sync_mode=True ``` **Send a test request:** ```bash curl -X POST http://localhost:4000/chat/completions \ -H "Authorization: Bearer YOUR_LITELLM_MASTER_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4", "messages": [{"role": "user", "content": "Hello!"}] }' ``` **Verify in the Akto dashboard:** * Log into the Akto dashboard * Navigate to the Collections section * Confirm that requests from LiteLLM are appearing {% endstep %} {% endstepper %} ## Per-Agent Collections By default, all LiteLLM traffic is grouped into a single collection named after the proxy host. The connector supports creating separate collections per agent, allowing each agent's API traffic to be tracked independently in the Akto dashboard. ### How Collections are Created The connector extracts the agent identity from request metadata and uses it as the collection name. The following sources are checked in order of priority: 1. **`metadata.agent_name`** — explicitly provided by the user in the request body 2. **`key_alias`** — the human-readable name assigned to the LiteLLM virtual key 3. **`team_alias`** — the human-readable name assigned to the team the key belongs to If none of the above are available, all traffic is grouped into a single collection named after the LiteLLM proxy host. ### Using Metadata Users can specify an `agent_name` in the request metadata. This approach is supported across all SDKs: {% tabs %} {% tab title="OpenAI SDK" %} ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:4000", api_key="sk-...") response = client.chat.completions.create( model="gpt-4", messages=[{"role": "user", "content": "Hello!"}], extra_body={ "metadata": {"agent_name": "chatbot-agent"} } ) ``` {% endtab %} {% tab title="LiteLLM SDK" %} ```python import litellm response = litellm.completion( model="litellm_proxy/gpt-4", messages=[{"role": "user", "content": "Hello!"}], extra_body={ "metadata": {"agent_name": "chatbot-agent"} }, api_base="http://localhost:4000", api_key="sk-...", ) ``` {% endtab %} {% tab title="curl" %} ```bash curl -X POST http://localhost:4000/chat/completions \ -H "Authorization: Bearer sk-..." \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4", "messages": [{"role": "user", "content": "Hello!"}], "metadata": {"agent_name": "chatbot-agent"} }' ``` {% endtab %} {% endtabs %} The above creates a collection named **chatbot-agent** in the Akto dashboard. ### Using key\_alias If the LiteLLM virtual keys have a `key_alias` configured (e.g., `"search-agent"`), the connector identifies the agent automatically. All traffic associated with that key is grouped into a collection named after the alias. No changes to the end user's code are required. ### Using team\_alias If the key belongs to a team with a `team_alias` configured (e.g., `"search-agents"`), the connector uses it as the collection name. This groups all traffic from keys within that team into a single collection. No changes to the end user's code are required. {% hint style="info" %} **Priority** The resolution order is: `metadata.agent_name` (highest) → `key_alias` → `team_alias` (lowest). When multiple sources are available, the highest priority value is used. {% endhint %} ## Session-Based Guardrails The connector supports session tracking, which lets the Akto guardrails service correlate multiple requests belonging to the same conversation or user session. This enables session-aware policies such as malicious-session detection and session-summary injection. To enable this, send an `x-session-id` header on the request to the LiteLLM proxy. When present, the connector captures it and forwards it to the Akto guardrails service, which groups requests sharing the same session ID. {% tabs %} {% tab title="OpenAI SDK" %} ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:4000", api_key="sk-...") response = client.chat.completions.create( model="gpt-4", messages=[{"role": "user", "content": "Hello!"}], extra_headers={"x-session-id": "session-abc-123"} ) ``` {% endtab %} {% tab title="curl" %} ```bash curl -X POST http://localhost:4000/chat/completions \ -H "Authorization: Bearer sk-..." \ -H "Content-Type: application/json" \ -H "x-session-id: session-abc-123" \ -d '{ "model": "gpt-4", "messages": [{"role": "user", "content": "Hello!"}] }' ``` {% endtab %} {% endtabs %} {% hint style="info" %} **Note** Session tracking is optional. Requests without an `x-session-id` header are processed normally and are simply not associated with a session. Session-based features on the guardrails service are controlled by the `SESSION_ENABLED` setting (enabled by default). {% endhint %} ## How It Works ### Request Flow (SYNC\_MODE=true) ``` 1. Client → LiteLLM Proxy 2. Hook intercepts the request (pre-call hook) 3. Request is sent to the Akto Data Ingestion Service API 4. Data Ingestion Service validates against policies ├─ If BLOCKED: Error returned to the client (LLM is not called) └─ If ALLOWED: Continue to step 5 5. Request is forwarded to the LLM provider 6. LLM response is received 7. Hook intercepts the response (post-call hook) 8. Response is sent to the Akto Data Ingestion Service API for display in the dashboard ``` ### Request Flow (SYNC\_MODE=false) ``` 1. Client → LiteLLM Proxy 2. Hook initiates a background validation task (non-blocking) 3. Request is immediately forwarded to the LLM provider 4. Response is returned to the client 5. Validation completes in the background (violations are logged only) ``` ## Get Support for your Akto setup There are multiple ways to request support from Akto. We are 24X7 available on the following: 1. In-app `intercom` support. Message us with your query on intercom in Akto dashboard and someone will reply. 2. Join our [discord channel](https://www.akto.io/community) for community support. 3. Contact `help@akto.io` for email support. 4. Contact us [here](https://www.akto.io/contact-us). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://ai-security-docs.akto.io/akto-argus-agentic-ai-security-for-homegrown-ai/connectors/ai-agent-security/litellm.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.