> For the complete documentation index, see [llms.txt](https://ai-security-docs.akto.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai-security-docs.akto.io/akto-argus-agentic-ai-security-for-homegrown-ai/connectors/ai-agent-security/litellm.md).

# LiteLLM

## Overview

LiteLLM is a unified interface for calling 100+ LLM APIs in a consistent format. This integration enables monitoring of API traffic from a LiteLLM proxy and ensures AI-powered applications maintain security standards.

The Akto LiteLLM connector provides the following capabilities:

* Validates AI requests and responses against security policies
* Detects PII, prompt injection, and policy violations
* Ingests traffic into Akto for monitoring and analysis
* Blocks malicious requests (sync mode) or logs violations (async mode)
* Creates per-agent collections based on agent identity

## Prerequisites

Before integrating Akto with LiteLLM, ensure the following are in place:

* An existing LiteLLM proxy installation (running or ready to configure)
* An Akto guardrails service endpoint (URL and authentication token)

## Steps to Connect

{% stepper %}
{% step %}
**Download the Custom Hook**

Download the `custom_hooks.py` file to the LiteLLM configuration directory:

```bash
# Navigate to the LiteLLM config directory
cd /path/to/your/litellm/config

# Download the hook file
curl -O https://raw.githubusercontent.com/akto-api-security/akto/master/apps/mcp-endpoint-shield/litellm/custom_hooks.py
```

{% endstep %}

{% step %}
**Configure Environment Variables**

Add the following environment variables to the LiteLLM environment (`.env` file or system environment):

```bash
# URL of this LiteLLM proxy instance (used as the default collection host)
LITELLM_URL=http://your-litellm-instance-url

# Akto's Data Ingestion Service URL
DATA_INGESTION_SERVICE_URL=http://data-ingestion-service-url

# Akto API token used to authenticate to the Data Ingestion Service
AKTO_API_TOKEN=<your-akto-guardrail-service-token>

# Optional: Operation Mode
SYNC_MODE=true              # true = block violations, false = async logging only, default true

# Optional: timeout (in seconds) for calls to the Data Ingestion Service
TIMEOUT=5                   # default 5
```

The connector reads these variables (`custom_hooks.py`):

| Variable                     | Required | Default                 | Description                                                                                                                        |
| ---------------------------- | -------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `DATA_INGESTION_SERVICE_URL` | Yes      | —                       | Akto Data Ingestion Service endpoint the hook sends traffic and validation requests to.                                            |
| `AKTO_API_TOKEN`             | Yes      | empty                   | Token sent in the `Authorization` header to the Data Ingestion Service. Obtain from **Akto Argus → Connectors → Setup Guardrail**. |
| `LITELLM_URL`                | Yes      | `http://localhost:4000` | This proxy's URL; its host is used as the default collection name when no agent identity is present.                               |
| `SYNC_MODE`                  | No       | `true`                  | `true` blocks violations before the LLM call; `false` validates asynchronously (log only).                                         |
| `TIMEOUT`                    | No       | `5`                     | Timeout in seconds for HTTP calls to the Data Ingestion Service.                                                                   |

{% hint style="warning" %}
**Note**

`SYNC_MODE` determines behavior:

* `SYNC_MODE=true`: Requests are validated before being sent to the LLM. Violations block the request immediately.
* `SYNC_MODE=false`: Requests proceed immediately. Validation occurs in the background.
  {% endhint %}
  {% endstep %}

{% step %}
**Update LiteLLM Configuration**

Edit the `config.yaml` to enable the custom hook:

```yaml
model_list:
  # Existing models

litellm_settings:
  callbacks: [custom_hooks.proxy_handler_instance]  # ← Add this line
  drop_params: true
  set_verbose: false
  request_timeout: 600
  num_retries: 2

# ... rest of the config ...
```

The required change is adding `callbacks: [custom_hooks.proxy_handler_instance]` to activate the Akto guardrails hook.
{% endstep %}

{% step %}
**Ensure Hook File is Accessible**

{% tabs %}
{% tab title="Using LiteLLM Directly" %}
Ensure `custom_hooks.py` is in the same directory as `config.yaml`, then start LiteLLM with the environment variables from the previous step set:

```bash
litellm --config config.yaml
```

{% hint style="info" %}
The Akto API Token can be obtained from **Akto Argus -> Connectors -> Setup Guardrail**.\
![](/files/NW2TeOXo0U8K3EsWqYsn)
{% endhint %}
{% endtab %}

{% tab title="Using Docker" %}
Mount the hook file in the docker compose configuration:

```yaml
services:
  litellm:
    image: docker.litellm.ai/berriai/litellm:main-stable
    volumes:
      - ./config.yaml:/app/config.yaml
      - ./custom_hooks.py:/app/custom_hooks.py
    environment:
      - LITELLM_URL=${LITELLM_URL}
      - DATA_INGESTION_SERVICE_URL=${DATA_INGESTION_SERVICE_URL}
      - AKTO_API_TOKEN=${AKTO_API_TOKEN}
      - SYNC_MODE=${SYNC_MODE}
    # ... rest of config ...
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}
**Start LiteLLM**

```bash
# Using Docker Compose
docker compose restart litellm

# Using Docker run
docker restart litellm-container

# Running directly (stop with Ctrl+C, then restart)
litellm --config config.yaml
```

{% endstep %}

{% step %}
**Verify Integration**

Confirm that LiteLLM starts successfully with the hook:

```bash
# Check logs for hook initialization
docker compose logs litellm | grep GuardrailsHandler

# Expected output:
# GuardrailsHandler initialized | sync_mode=True
```

**Send a test request:**

```bash
curl -X POST http://localhost:4000/chat/completions \
  -H "Authorization: Bearer YOUR_LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

**Verify in the Akto dashboard:**

* Log into the Akto dashboard
* Navigate to the Collections section
* Confirm that requests from LiteLLM are appearing
  {% endstep %}
  {% endstepper %}

## Per-Agent Collections

By default, all LiteLLM traffic is grouped into a single collection named after the proxy host. The connector supports creating separate collections per agent, allowing each agent's API traffic to be tracked independently in the Akto dashboard.

### How Collections are Created

The connector extracts the agent identity from request metadata and uses it as the collection name. The following sources are checked in order of priority:

1. **`metadata.agent_name`** — explicitly provided by the user in the request body
2. **`key_alias`** — the human-readable name assigned to the LiteLLM virtual key
3. **`team_alias`** — the human-readable name assigned to the team the key belongs to

If none of the above are available, all traffic is grouped into a single collection named after the LiteLLM proxy host.

### Using Metadata

Users can specify an `agent_name` in the request metadata. This approach is supported across all SDKs:

{% tabs %}
{% tab title="OpenAI SDK" %}

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="sk-...")

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={
        "metadata": {"agent_name": "chatbot-agent"}
    }
)
```

{% endtab %}

{% tab title="LiteLLM SDK" %}

```python
import litellm

response = litellm.completion(
    model="litellm_proxy/gpt-4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={
        "metadata": {"agent_name": "chatbot-agent"}
    },
    api_base="http://localhost:4000",
    api_key="sk-...",
)
```

{% endtab %}

{% tab title="curl" %}

```bash
curl -X POST http://localhost:4000/chat/completions \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "metadata": {"agent_name": "chatbot-agent"}
  }'
```

{% endtab %}
{% endtabs %}

The above creates a collection named **chatbot-agent** in the Akto dashboard.

### Using key\_alias

If the LiteLLM virtual keys have a `key_alias` configured (e.g., `"search-agent"`), the connector identifies the agent automatically. All traffic associated with that key is grouped into a collection named after the alias. No changes to the end user's code are required.

### Using team\_alias

If the key belongs to a team with a `team_alias` configured (e.g., `"search-agents"`), the connector uses it as the collection name. This groups all traffic from keys within that team into a single collection. No changes to the end user's code are required.

{% hint style="info" %}
**Priority**

The resolution order is: `metadata.agent_name` (highest) → `key_alias` → `team_alias` (lowest). When multiple sources are available, the highest priority value is used.
{% endhint %}

## Session-Based Guardrails

The connector supports session tracking, which lets the Akto guardrails service correlate multiple requests belonging to the same conversation or user session. This enables session-aware policies such as malicious-session detection and session-summary injection.

To enable this, send an `x-session-id` header on the request to the LiteLLM proxy. When present, the connector captures it and forwards it to the Akto guardrails service, which groups requests sharing the same session ID.

{% tabs %}
{% tab title="OpenAI SDK" %}

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="sk-...")

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={"x-session-id": "session-abc-123"}
)
```

{% endtab %}

{% tab title="curl" %}

```bash
curl -X POST http://localhost:4000/chat/completions \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -H "x-session-id: session-abc-123" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Note**

Session tracking is optional. Requests without an `x-session-id` header are processed normally and are simply not associated with a session. Session-based features on the guardrails service are controlled by the `SESSION_ENABLED` setting (enabled by default).
{% endhint %}

## How It Works

### Request Flow (SYNC\_MODE=true)

```
1. Client → LiteLLM Proxy
2. Hook intercepts the request (pre-call hook)
3. Request is sent to the Akto Data Ingestion Service API
4. Data Ingestion Service validates against policies
   ├─ If BLOCKED: Error returned to the client (LLM is not called)
   └─ If ALLOWED: Continue to step 5
5. Request is forwarded to the LLM provider
6. LLM response is received
7. Hook intercepts the response (post-call hook)
8. Response is sent to the Akto Data Ingestion Service API for display in the dashboard
```

### Request Flow (SYNC\_MODE=false)

```
1. Client → LiteLLM Proxy
2. Hook initiates a background validation task (non-blocking)
3. Request is immediately forwarded to the LLM provider
4. Response is returned to the client
5. Validation completes in the background (violations are logged only)
```

## Get Support for your Akto setup

There are multiple ways to request support from Akto. We are 24X7 available on the following:

1. In-app `intercom` support. Message us with your query on intercom in Akto dashboard and someone will reply.
2. Join our [discord channel](https://www.akto.io/community) for community support.
3. Contact `help@akto.io` for email support.
4. Contact us [here](https://www.akto.io/contact-us).