AI Gateway

AI Gateway lets people and managed agents call centrally configured models. Provider API keys remain in the private engine. People use their Tunnex login; managed agents use short-lived credentials tied to their existing identity.

The gateway is available in Community and starts disabled for each organization. Installing it or saving a model does not grant anyone access. This guide covers the updated AI console; use a matching control-plane and CLI release.

The sidebar’s AI section contains three workspaces:

Manage people and directory groups under Users & Groups → Groups. Manage machine membership under AI Agents → Agent groups. These are different group types. Network grants remain under Access Policies.

Install the private engine using the matching release’s deploy/ai-gateway/compose.yml and managed-provider overlays, or the optional aiGateway settings in the Helm chart. Follow docs/AI-gateway-setup.md in that release for image pins, private networking, encryption, persistent volumes, and bridge configuration. Keep engine administration and inference ports private.

Saved-credential tests and additional provider protocols need the matching apps/ai-engine extension and LiteLLM bridge. The supplied compose-litellm.yml overlay configures these components. An unmodified engine cannot perform the saved-key test operation.

Preserve the engine encryption key and its configuration/accounting volumes through upgrades. The current installation uses a single control-plane API process and one private engine; this guide does not establish high availability.

An organization owner or administrator can assign roles in Users & Groups → Roles. A person may hold multiple roles.

• ai-admin manages model credentials, models, gateway settings, and model grants without receiving general VPN or organization administration.
• ai-view reads AI configuration and usage but cannot change it.
• member can use models explicitly granted to their user groups.

AI roles do not grant access to every model. A user still needs an applicable group grant, and the organization must enable AI access. AI-only roles cannot invite users, change membership roles, or administer agent lifecycle. See Users, invitations, and roles.

1. Open AI Gateway → LLM credentials → Add Credentials.
2. Select the provider and enter a model or deployed model name. Suggestions appear as you type. If the deployment has a custom name, enter that exact name.
3. Enter Upstream API Base, the provider key, and a recognizable credential name such as Engineering Azure.
4. Choose the model’s Mode, then select Test Connect.
5. On a successful response, the toast shows HTTP 200. Select Create credentials and wait for applied status.

Test Connect makes a small inference request and can incur a provider charge. It tests the first selected model and mode only. Results expire after five minutes; changing the model, mode, endpoint, key, or saved credential revision requires a fresh test. Catalog checks do not generate model tokens and do not prove deployment access.

Failure feedback distinguishes an upstream HTTP response from a gateway error, timeout, incomplete response, or an endpoint that could not be reached. A network failure has no provider HTTP response code. For a private Azure deployment, configure private DNS and network reachability from the gateway installation; your browser being able to open Azure does not establish that path.

Choose Azure AI Foundry for Azure-hosted models. Use your deployment name, which may differ from the catalog suggestion. The catalog includes supported Foundry model families, including Claude, and is not limited to GPT models. Suggestions are reference data, not discovery of your private deployments.

Accepted public-cloud resource hosts include services.ai.azure.com, openai.azure.com, and cognitiveservices.azure.com.

• For OpenAI-compatible deployments, paste https://<resource>.services.ai.azure.com/openai/v1/.
• For Claude, paste https://<resource>.services.ai.azure.com/anthropic/v1/messages.
• Supported Azure portal deployment URLs can also be pasted. The form extracts the deployment and operation and uses the canonical v1 endpoint. It does not forward a legacy api-version query unchanged.

Use the endpoint actually offered by your deployment. Not every model supports every protocol or mode. Public HTTPS endpoints are checked automatically; private or internal endpoints need installation network configuration. No separate per-endpoint approval is needed for supported public destinations.

The provider list comes from the installation’s supported registry. A SageMaker connection uses the configured bridge and AWS-side endpoint/credential setup; its availability depends on that installation. Catalog entries alone do not add new transport support.

1. Open Models & endpoints → Add Model.
2. Choose Existing Credentials. Saved credentials select their provider automatically; endpoint and API-key inputs disappear because those values are reused securely.
3. Select a suggested model or enter the exact deployed model name.
4. Select Mode, run Test Connect, then Add Model.
5. Wait for applied status and use the exact API model ID shown in the list.

A saved credential can test a new model before it is added. Testing does not change its serving scope. Saving expands the credential’s model scope; callers still need a separate group grant or agent policy and assignment.

Mode options are Chat, Completion, Embedding, Audio speech, Audio transcription, Image generation, Video generation, and Rerank. Choose the operation the upstream supports. Selecting an option does not make an incompatible model support that operation. Model names are exact identifiers, not public aliases.

1. Open Users & Groups → Groups, create or open the people group Engineering, and add its four users. Directory-managed membership remains controlled by the directory integration.
2. In AI Gateway → Settings, explicitly enable organizational AI access.
3. In Models & endpoints, select Grant access on the model, or open Model access directly.
4. Choose Engineering, select the saved credential and exact model, enable the grant, then save. Wait for the grant to be applied.
5. Ask a group member to open AI Gateway → My models. The model and its endpoint should be listed there.

Users receive no provider key. They authenticate to Tunnex, and the gateway injects the saved provider credential only on the authorized upstream call. Removing membership or disabling the grant refuses new requests; already accepted requests may finish within their deadline.

Install a CLI release that includes the ai commands. Sign in once, list your permitted model IDs, and call the chosen chat model:

tunnex login --server 'https://control-plane.example.com'
tunnex ai models --org '<organization-uuid>'
tunnex ai chat --org '<organization-uuid>' \
  --model '<exact-API-model-ID-from-the-list>' --prompt 'Hello'

The CLI uses the saved Tunnex login. Browser calls use the signed-in session. API clients use the same Tunnex login credential; they do not send the Azure or other provider key. My models → Copy endpoint gives the concrete URL.

The user chat endpoint is:

POST /api/v1/organizations/{orgId}/ai-gateway/inference/v1/chat/completions

Other configured modes use their corresponding operation below the same inference/v1 base. The browser Call model form and tunnex ai chat are for chat. The My models inventory only shows currently granted models.

1. Enroll an agent with an accountable human owner.
2. Enable the Agent groups feature when available, then create a group under AI Agents → Agent groups and add the agent.
3. Open AI Agents → Model access. Select the policy team, exact models, and applied provider connections. Save the team policy.
4. Select the agent, choose its AI team, and enable its assignment. An empty model override inherits the team set; a populated override can only narrow it.
5. Wait for applied status with matching desired/applied revisions. Retry reconciliation after correcting a pending or failed assignment.

An agent can have several ordinary memberships but selects one explicit AI team. People-group grants do not substitute for an agent assignment.

Exchange the current runtime credential at POST /api/v1/agent/runtime/ai-credential for a five-minute Tunnex AI token. Use that token on POST /ai/v1/chat/completions or the qualified POST /ai/anthropic/v1/messages route. Do not put a bootstrap token, runtime credential, user session, or provider key on these agent inference routes.

The qualified chat/Messages payload accepts string text only. Images, audio content blocks, Claude content arrays, tool calls, and additional message controls are rejected before forwarding. Streaming is supported. Output defaults to 1,024 tokens and is limited to 4,096; the request duration is bounded to 30 seconds. Selecting a provider with richer APIs does not expand this gateway payload subset.

Under AI Agents → Model access → Team model policy, set Daily USD soft threshold (optional). Leave it blank for no monetary threshold, or enter a positive USD amount up to 100000; for example, 2.50 means USD 2.50. Save and reopen the policy to confirm the value.

The comparison uses observed usage since midnight UTC. Once recorded cost reaches the threshold, new requests are refused. Concurrent requests and work already accepted can exceed it, so it is a soft threshold, not a strict spend cap. Editing the policy does not reset observed spending. Unknown pricing or unavailable/incomplete accounting also refuses new requests when a monetary threshold applies. No threshold notification delivery is provided.

The agent credential boundary returns a generic HTTP 403 policy refusal. A 403 alone does not identify a threshold event: inspect policy scope, usage, pricing, and synchronization too. Native engine accounting and control-plane admission are distinct checks.

AI Gateway → Usage & cost shows observed estimates, including daily, model, team, and agent breakdowns. Native estimates are not provider invoices. The supplied configuration retains accounting metadata for seven days and disables prompt/response content logging. A longer query cannot recover expired records.

Editing or disabling credentials affects models using them. Deletion refuses retained policy references: remove the applicable grants first. Preserve engine configuration, accounting volumes, and encryption material during maintenance. After recovery, verify applied policy and an independently authorized caller.

Model access does not replace VPN access policies or MCP tool controls. Configure those separately.