Prompt Caching

By storing the repeated prefix of a prompt, Prompt Caching cuts both latency and input-token cost when you call large language model APIs.

How Caching Works

1. Hit condition: Only an exactly matching prompt prefix qualifies as a hit. Supplying prompt_cache_key can help steer routing and may improve the hit rate.

2. Routing: The hash value computed over the prompt prefix determines which specific server handles each request.

3. Lookup outcome: A matching prefix means the response draws on cached results. Without a match, the platform processes the complete request and stores the prefix so that later calls can reuse it.

Privacy and Billing

1. Organization isolation: Cached data never crosses organization boundaries — it stays visible only to members of the organization that created it.

2. Outputs unchanged: Caching operates on the input prompt alone and has no effect on the content the model generates.

3. Rate-limit accounting: Even requests served from the cache still count toward your TPM limit.

Usage Example

import requests

url = "https://api.watt-cloud.com/v1/chat/completions"

payload = {
    "model": "${MODEL_NAME}",
    "messages": [
        {
            "role": "system",
            "content": "You are a senior clinical physician with 20 years of internal medicine experience, skilled in symptom analysis and differential diagnosis. Diagnosis process: 1) Chief complaint extraction 2) Symptom correlation analysis 3) Initial differential diagnosis (list 3-5 possibilities) 4) Recommended examination items 5) Medical advice. Important note: Only for reference, not a final diagnosis. Patients must visit a regular hospital."
        },
        {
            "role": "user",
            "content": "I've had recurring headaches for the past week, accompanied by nausea and photophobia. What could be the problem?"
        }
    ],
    "max_tokens": 512,
    "prompt_cache_key": "medical_symptom_analysis_v1"
}

headers = {
    "Authorization": "Bearer ${API_KEY}",
    "Content-Type": "application/json"
}

response = requests.request("POST", url, json=payload, headers=headers)
print(response.text)

Monitoring Metrics

To track the hit rate and overall cache performance, check the usage.prompt_tokens_details.cached_tokens field in API responses, or open the monitoring dashboard.

{
  "usage": {
    "completion_tokens": 512,
    "prompt_tokens": 105,
    "prompt_tokens_details": {
      "cached_tokens": 80
    },
    "total_tokens": 617
  }
}