Hugging Face

Overview

This guide explains how to integrate Akto AI Agent Gateway with a Hugging Face Private Inference Endpoint used by customers to run private LLM inference. The gateway sits between the end user and the agent application (Option B) to monitor, enforce guardrails, and log model invocation traffic without modifying internal client code.

Akto AI Agent Gateway provides:

• Guardrail enforcement on both requests and responses

• Sensitive data redaction

• Security guardrail detection

Hugging Face’s Private Inference Endpoint provides a dedicated, managed model endpoint accessible only via AWS PrivateLink from within a VPC. Hugging Face does not automatically log full prompt & response conversations like AWS Bedrock, so Akto must capture this upstream.

Prerequisites

Before integrating Akto Gateway:

1. A working Hugging Face Private Inference Endpoint configured with PrivateLink.

2. AWS VPC where the endpoint service is reachable.

3. The AI agent and Akto Gateway deployed in the same VPC or with network access to the PrivateLink interfaces.

4. Access credentials for Hugging Face inference (API token).

---

Architecture Diagram

1. End user calls the AI agent API.

2. Akto Gateway intercepts requests (guardrail enforcement).

3. Gateway forwards to HF Private Inference Endpoint (via PrivateLink).

4. Responses pass back through Akto Gateway.

5. Akto logs, analyzes and optionally redacts or blocks results.

Setup Steps

1

Configure Hugging Face Private Inference Endpoint

Ensure the endpoint is set up with:

• Model deployed

• PrivateLink enabled

• Correct AWS account and region

• VPC interface endpoint created in your VPC

Hugging Face does not log full request/response content by itself. You must capture it upstream.

2

Deploy Akto AI Agent Gateway

Deploy the gateway in the same VPC where:

• End user traffic enters

• The AI agent application runs

• The PrivateLink interface to HF endpoint exists

3

Configure Gateway Environment

Here is an example config for the gateway:

export AKTO_API_TOKEN=<YOUR_AKTO_PROXY_TOKEN>
export AKTO_API_BASE_URL=<AKTO_API_BASE_URL>
export APP_URL=<HUGGING_FACE_PRIVATELINK_ENDPOINT_URL>
export LOG_LEVEL=INFO

• AKTO_API_TOKEN: Akto ingestion token (go to Akto Argus → Connectors → Setup Guardrail card to obtain it)

• AKTO_API_BASE_URL: Akto gateway ingestion server. Follows the format https://<account_id>-guardrails.akto.io; contact the Akto support team to get the URL for your account.

• APP_URL: Upstream target (the HF Private Inference Endpoint URL)

• LOG_LEVEL: Logging verbosity

4

Adjust Endpoint URL in Agent App

Update the AI agent’s inference call configuration:

• Set model base URL to the Akto Gateway endpoint

• Pass Hugging Face authentication headers through gateway

For example:

AI_AGENT_INFERENCE_URL=https://akto-proxy.internal.svc
HF_AUTHORIZATION=Bearer <HF_TOKEN>

This ensures:

• Traffic flows through Akto Gateway

• Akto captures all inference calls

5

Validate Integration

Verify end-to-end flow:

1. Send an inference request from the user

2. Akto Gateway receives and logs the call

3. Gateway enforces any guardrails

4. Gateway forwards to HF Private Endpoint

5. Response returns through Akto Gateway

6. Logs appear in Akto dashboard

Look for:

• Request/response pairs in gateway logs

• Guardrail hits (if configured)

• Redaction results

Security & Guardrails

Akto Gateway supports:

• Request guardrails (input sanitization)

• Response guardrails (filtering outputs)

• Redaction of sensitive tokens or PII

• Rate limiting and anomaly detection

Use our policy packs or define custom rules based on:

• Content patterns

• Risk categories

• Endpoint sensitivity

Logging & Monitoring

Hugging Face Private Endpoints offer:

• Operational logs (status, errors)

• Metrics (latency, throughput)

They do not log conversation content by default.

Akto Gateway will log:

• Full request and response traces

• Guardrail decision events

• Alerts and incidents

• Metadata for analytics

Troubleshooting

• Gateway cannot reach HF Endpoint: Check PrivateLink and VPC routing.

• Auth failures: Verify Hugging Face API token headers are passed by gateway.

• No logs in Akto: Confirm AKTO_API_TOKEN and ingestion config.

• Guardrail not triggering: Validate rule pack configuration.

Summary

By integrating Akto AI Agent Gateway in front of a Hugging Face Private Inference Endpoint:

• You achieve guardrail enforcement without modifying the client code

• You capture and monitor model invocation traffic

• You gain observability of conversation logging

Akto Gateway becomes the enforcement and observability layer for private HF model usage.