LiteLLM
Connect Akto with LiteLLM
Overview
Prerequisites
Steps to Connect
1
# Navigate to the LiteLLM config directory
cd /path/to/your/litellm/config
# Download the hook file
curl -O https://raw.githubusercontent.com/akto-api-security/akto/master/apps/mcp-endpoint-shield/litellm/custom_hooks.py2
LITELLM_URL=http://your-litellm-instance-url
# Akto's Data Ingestion Service URL
DATA_INGESTION_SERVICE_URL=http://data-ingestion-service-url
# Optional: Operation Mode
SYNC_MODE=true # true = block violations, false = async logging only, default true3
model_list:
# Existing models
litellm_settings:
callbacks: [custom_hooks.proxy_handler_instance] # ← Add this line
drop_params: true
set_verbose: false
request_timeout: 600
num_retries: 2
# ... rest of the config ...4
# Set environment variables
export LITELLM_URL=http://your-litellm-instance-url
export DATA_INGESTION_SERVICE_URL=http://data-ingestion-service-url
export SYNC_MODE=true
# Start LiteLLM
litellm --config config.yamlservices:
litellm:
image: docker.litellm.ai/berriai/litellm:main-stable
volumes:
- ./config.yaml:/app/config.yaml
- ./custom_hooks.py:/app/custom_hooks.py
environment:
- LITELLM_URL=${LITELLM_URL}
- DATA_INGESTION_SERVICE_URL=${DATA_INGESTION_SERVICE_URL}
- SYNC_MODE=${SYNC_MODE}
# ... rest of config ...5
# Using Docker Compose
docker compose restart litellm
# Using Docker run
docker restart litellm-container
# Running directly (stop with Ctrl+C, then restart)
litellm --config config.yaml6
# Check logs for hook initialization
docker compose logs litellm | grep GuardrailsHandler
# Expected output:
# GuardrailsHandler initialized | sync_mode=Truecurl -X POST http://localhost:4000/chat/completions \
-H "Authorization: Bearer YOUR_LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello!"}]
}'Per-Agent Collections
How Collections are Created
Using Metadata
Using key_alias
Using team_alias
How It Works
Request Flow (SYNC_MODE=true)
Request Flow (SYNC_MODE=false)
Get Support for your Akto setup
Last updated