---
title: "Solving LLM Tool Calling Incompatibilities: A Python Middleware Auth Proxy for LiteLLM & Cloud Run"
date: "2026-09-23T10:00:00+02:00"
tags: ["Python", "LiteLLM", "Open WebUI", "Cloud Run", "OIDC", "Proxy", "vLLM"]
categories: ["AI & LLM Ops", "Homelab & Containers"]
---

![Solving LLM Tool Calling Incompatibilities with Python Middleware Auth Proxy][1]

When connecting modern web interfaces like **Open WebUI** or **LiteLLM** to remote inference engines on **Google Cloud Run**, you quickly run into two major technical hurdles:
1. **Tool-choice Schema Conflicts:** Open WebUI automatically injects `tool_choice: "auto"` and `tools: [...]` into every OpenAI-compatible request. Some specialized vLLM builds and distilled models reject these parameters with `400 Bad Request`.
2. **Dynamic OIDC Authentication:** Securing Cloud Run behind IAM requires generating fresh Google OIDC identity tokens, which standard OpenAI client libraries do not natively handle.

Here is how I solved both problems by building a lightweight **Python Auth & Filtering Proxy**.

<!--more-->

## The Problem Breakdown

When an end-user chats in Open WebUI:
```json
{
  "model": "qwen3.8-27b",
  "messages": [{"role": "user", "content": "Hello!"}],
  "tool_choice": "auto",
  "tools": []
}
```

If the backend vLLM instance does not have auto-tool-choice enabled for that specific model branch, vLLM returns:
`{"error": "tool_choice is not supported with the current parser"}`.

Furthermore, hitting a private Cloud Run endpoint requires an `Authorization: Bearer <GCP_OIDC_TOKEN>` header that expires every 60 minutes.

---

## The Solution: `llm-auth-proxy`

We placed a lightweight Python service between LiteLLM and Cloud Run:

```text
[Open WebUI] ──► [LiteLLM] ──► [llm-auth-proxy (Port 8000)] ──► [Google Cloud Run (vLLM)]
```

### The Proxy Implementation (`proxy.py`)

```python
import os
import requests
from flask import Flask, request, Response
from google.auth.transport.requests import Request as GoogleRequest
from google.oauth2 import id_token

app = Flask(__name__)
CLOUD_RUN_URL = os.environ.get("CLOUD_RUN_URL")

def get_oidc_token(audience):
    auth_req = GoogleRequest()
    return id_token.fetch_id_token(auth_req, audience)

@app.route("/v1/chat/completions", methods=["POST"])
def proxy_completions():
    req_json = request.get_json(silent=True)
    
    # 1. Sanitize JSON payload
    if req_json and isinstance(req_json, dict):
        req_json.pop("tool_choice", None)
        req_json.pop("tools", None)
        
    # 2. Fetch fresh Google OIDC token
    token = get_oidc_token(CLOUD_RUN_URL)
    headers = {
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"
    }
    
    # 3. Forward stream to Cloud Run
    resp = requests.post(
        f"{CLOUD_RUN_URL}/v1/chat/completions",
        json=req_json,
        headers=headers,
        stream=True
    )
    
    return Response(
        resp.iter_content(chunk_size=1024),
        status=resp.status_code,
        content_type=resp.headers.get("Content-Type", "application/json")
    )

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)
```

---

## Configuring LiteLLM

In `litellm_config.yaml`, LiteLLM routes queries to the local proxy instead of directly to GCP:

```yaml
model_list:
  - model_name: Qwen-3.8-27B-BF16
    litellm_params:
      model: openai/qwen3.8-27b
      api_base: http://llm-auth-proxy:8000/v1
      api_key: proxy-key
```

This decoupling ensures that any quirks in frontend payloads are cleaned up before reaching production inference endpoints!

[1]: /images/uploads/2026/09/python-auth-proxy-litellm-cloud-run.webp