Solving LLM Tool Calling Incompatibilities: A Python Middleware Auth Proxy for LiteLLM & Cloud Run

Wed, 23 Sep 2026 10:00:00 +0200

When connecting modern web interfaces like Open WebUI or LiteLLM to remote inference engines on Google Cloud Run, you quickly run into two major technical hurdles:

Tool-choice Schema Conflicts: Open WebUI automatically injects tool_choice: "auto" and tools: [...] into every OpenAI-compatible request. Some specialized vLLM builds and distilled models reject these parameters with 400 Bad Request.
Dynamic OIDC Authentication: Securing Cloud Run behind IAM requires generating fresh Google OIDC identity tokens, which standard OpenAI client libraries do not natively handle.

Here is how I solved both problems by building a lightweight Python Auth & Filtering Proxy.

The Problem Breakdown

When an end-user chats in Open WebUI:

{
  "model": "qwen3.8-27b",
  "messages": [{"role": "user", "content": "Hello!"}],
  "tool_choice": "auto",
  "tools": []
}

If the backend vLLM instance does not have auto-tool-choice enabled for that specific model branch, vLLM returns: {"error": "tool_choice is not supported with the current parser"}.

Furthermore, hitting a private Cloud Run endpoint requires an Authorization: Bearer <GCP_OIDC_TOKEN> header that expires every 60 minutes.

The Solution: `llm-auth-proxy`

We placed a lightweight Python service between LiteLLM and Cloud Run:

`1`	`[Open WebUI] ──► [LiteLLM] ──► [llm-auth-proxy (Port 8000)] ──► [Google Cloud Run (vLLM)]`

The Proxy Implementation (`proxy.py`)

import os
import requests
from flask import Flask, request, Response
from google.auth.transport.requests import Request as GoogleRequest
from google.oauth2 import id_token

app = Flask(__name__)
CLOUD_RUN_URL = os.environ.get("CLOUD_RUN_URL")

def get_oidc_token(audience):
    auth_req = GoogleRequest()
    return id_token.fetch_id_token(auth_req, audience)

@app.route("/v1/chat/completions", methods=["POST"])
def proxy_completions():
    req_json = request.get_json(silent=True)
    
    # 1. Sanitize JSON payload
    if req_json and isinstance(req_json, dict):
        req_json.pop("tool_choice", None)
        req_json.pop("tools", None)
        
    # 2. Fetch fresh Google OIDC token
    token = get_oidc_token(CLOUD_RUN_URL)
    headers = {
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"
    }
    
    # 3. Forward stream to Cloud Run
    resp = requests.post(
        f"{CLOUD_RUN_URL}/v1/chat/completions",
        json=req_json,
        headers=headers,
        stream=True
    )
    
    return Response(
        resp.iter_content(chunk_size=1024),
        status=resp.status_code,
        content_type=resp.headers.get("Content-Type", "application/json")
    )

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

Configuring LiteLLM

In litellm_config.yaml, LiteLLM routes queries to the local proxy instead of directly to GCP:

model_list:
  - model_name: Qwen-3.8-27B-BF16
    litellm_params:
      model: openai/qwen3.8-27b
      api_base: http://llm-auth-proxy:8000/v1
      api_key: proxy-key

This decoupling ensures that any quirks in frontend payloads are cleaned up before reaching production inference endpoints!

OIDC on Wojciech Marusiak IT Blog

Solving LLM Tool Calling Incompatibilities: A Python Middleware Auth Proxy for LiteLLM & Cloud Run

The Problem Breakdown

The Solution: llm-auth-proxy

The Proxy Implementation (proxy.py)

Configuring LiteLLM

The Solution: `llm-auth-proxy`

The Proxy Implementation (`proxy.py`)