Chat Completion

​

Tool calling

​

Request parameters

• tools (optional array): Each entry must be { "type": "function", "function": { "name": string, "description"?: string, "parameters"?: JSON-Schema object } }. Only function tools are accepted. The serialized tools payload is limited to 200 KB (overrides via TOOL_SPEC_MAX_BYTES); violating the shape or size yields a 400 with tool_spec_too_large, invalid_tool_spec, or invalid_tool_spec_parse.
• tool_choice (optional string or object): Defaults to auto. Set "none" to guarantee no tool calls (the server also drops the tools payload upstream), "required" to force the next response to be a tool call, or { "type": "function", "function": { "name": "your_function" } } to pin the exact function.
• parallel_tool_calls (optional boolean): When true the flag is forwarded to providers that support issuing multiple tool calls in a single turn. Models that ignore the flag fall back to sequential calls.
• messages[].tool_calls (assistant role): Persist the tool call metadata returned by the model so future turns can see which functions were invoked. Each item uses the OpenAI shape { id, type: "function", function: { name, arguments } }.
• messages[] with role: "tool": Respond to the model by sending { "role": "tool", "tool_call_id": "<assistant tool_calls id>", "content": "<JSON or text payload>" }. The server drops any tool response that references an unknown tool_call_id, so keep the IDs in sync.
• Validation behavior: If you send tool_choice: "none" with a tools array the request is accepted but the tools are omitted before hitting the model; invalid schemas or oversize payloads return the error codes above.

​

Example request

POST /api/v1/chat/completions
{
  "model": "gpt-4o-mini",
  "messages": [
    { "role": "user", "content": "What's the temperature in San Francisco right now?" }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "lookup_weather",
        "description": "Fetch the current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {
            "city": { "type": "string" },
            "unit": { "type": "string", "enum": ["c", "f"] }
          },
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto",
  "parallel_tool_calls": true
}

​

Example assistant/tool turn

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_abc123",
      "type": "function",
      "function": {
        "name": "lookup_weather",
        "arguments": "{\"city\":\"San Francisco\",\"unit\":\"f\"}"
      }
    }
  ]
}

{
  "role": "tool",
  "tool_call_id": "call_abc123",
  "content": "{\"city\":\"San Francisco\",\"temperatureF\":58,\"conditions\":\"foggy\"}"
}

​

Overview

​

Sampling & Decoding Controls

​

Temperature & Nucleus

​

Length & Stopping

​

Penalties & Repetition Guards

​

Logit Shaping & Determinism

​

Usage notes

• Parameters can be combined (e.g., temperature + top_p + top_k), but overly narrow settings may lead to early stops.
• Invalid ranges yield a 400 before reaching the provider.
• Provider defaults apply to any omitted field.

​

Example request

curl -X POST https://nano-gpt.com/api/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "Write a creative story about space exploration"}
    ],
    "temperature": 0.8,
    "top_p": 0.9,
    "top_k": 40,
    "tfs": 0.8,
    "typical_p": 0.95,
    "mirostat_mode": 2,
    "mirostat_tau": 5,
    "mirostat_eta": 0.1,
    "max_tokens": 500,
    "frequency_penalty": 0.3,
    "presence_penalty": 0.1,
    "repetition_penalty": 1.1,
    "stop": ["###"],
    "seed": 42
  }'

​

Web Search

​

Option A: model suffixes

• :online (default web search, standard depth)
• :online/linkup (Linkup, standard)
• :online/linkup-deep (Linkup, deep)
• :online/tavily (Tavily, standard)
• :online/tavily-deep (Tavily, deep)
• :online/exa-fast (Exa, fast)
• :online/exa-auto (Exa, auto)
• :online/exa-neural (Exa, neural)
• :online/exa-deep (Exa, deep)
• :online/kagi (Kagi, standard, search)
• :online/kagi-web (Kagi, standard, web)
• :online/kagi-news (Kagi, standard, news)
• :online/kagi-search (Kagi, deep, search)

​

Option B: request body configuration

• enabled (boolean, required to activate web search)
• provider (string): linkup | tavily | exa | kagi
• depth (string):
  • Linkup/Tavily/Kagi: standard or deep
  • Exa: fast, auto, neural, deep (use standard if you want auto)
• search_context_size or searchContextSize (string): low | medium | high (default: medium)
• kagiSource or kagi_source (string, Kagi only): web | news | search

​

Examples

import requests
import json

BASE_URL = "https://nano-gpt.com/api/v1"
API_KEY = "YOUR_API_KEY"

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

# Suffix-based standard web search
data = {
    "model": "chatgpt-4o-latest:online",
    "messages": [
        {"role": "user", "content": "What are the latest developments in AI?"}
    ]
}

response = requests.post(
    f"{BASE_URL}/chat/completions",
    headers=headers,
    json=data
)

# Request-body configuration (Exa neural)
data_linkup = {
    "model": "chatgpt-4o-latest",
    "messages": [
        {"role": "user", "content": "Provide a comprehensive analysis of recent AI breakthroughs"}
    ],
    "linkup": {
        "enabled": True,
        "provider": "exa",
        "depth": "neural",
        "search_context_size": "medium"
    }
}

​

Image Input

​

Supported Forms

• Remote URL: {"type":"image_url","image_url":{"url":"https://..."}}
• Base64 data URL: {"type":"image_url","image_url":{"url":"data:image/png;base64,...."}}

• Prefer HTTPS URLs; some upstreams reject non‑HTTPS. If in doubt, use base64 data URLs.
• Accepted mime types: image/png, image/jpeg, image/jpg, image/webp.
• Inline markdown images in plain text (e.g., ![alt](data:image/...;base64,...)) are auto‑normalized into structured parts server‑side.

​

Message Shape

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What is in this image?" },
    { "type": "image_url", "image_url": { "url": "https://example.com/image.jpg" } }
  ]
}

​

cURL — Image URL (non‑streaming)

curl -sS \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -X POST https://nano-gpt.com/api/v1/chat/completions \
  --data '{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Describe this image in three words."},
          {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/3/3f/Fronalpstock_big.jpg"}}
        ]
      }
    ],
    "stream": false
  }'

​

cURL — Base64 Data URL (non‑streaming)

curl -sS \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type": "application/json" \
  -X POST https://nano-gpt.com/api/v1/chat/completions \
  --data '{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is shown here?"},
          {"type": "image_url", "image_url": {"url": "data:image/png;base64,...BASE64..."}}
        ]
      }
    ],
    "stream": false
  }'

​

cURL — Streaming SSE

curl -N \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -X POST https://nano-gpt.com/api/v1/chat/completions \
  --data '{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Two words only."},
          {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
        ]
      }
    ],
    "stream": true,
    "stream_options": { "include_usage": true }
  }'

​

Prompt Caching (Claude Models)

Note: NanoGPT’s automatic failover system ensures high availability but may occasionally cause cache misses. If you’re seeing unexpected cache misses in your usage logs, see the “Cache Consistency with stickyProvider” section below.

headers = {
  "Authorization": "Bearer YOUR_API_KEY",
  "Content-Type": "application/json",
  "anthropic-beta": "prompt-caching-2024-07-31"
}

payload = {
  "model": "claude-3-5-sonnet-20241022",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "Reference handbook + rules of engagement.",
          "cache_control": {"type": "ephemeral", "ttl": "5m"}
        }
      ]
    },
    {"role": "user", "content": "Live request goes here"}
  ]
}

requests.post("https://nano-gpt.com/api/v1/chat/completions", headers=headers, json=payload)

• Each cache_control marker caches the full prefix up to that block. Place them on every static chunk (system messages, tool definitions, large contexts) you plan to reuse.
• Anthropic currently supports TTLs of 5m (1.25× one-time billing) and 1h (2× one-time). Replays show the discounted tokens inside usage.prompt_tokens_details.cached_tokens.
• The anthropic-beta: prompt-caching-2024-07-31 header is mandatory on requests that include caching metadata.

await client.chat.completions.create(
  {
    model: 'claude-3-5-sonnet-20241022',
    messages: [
      { role: 'system', content: 'Static rubric lives here.' },
      { role: 'user', content: 'Additional reusable context.' },
      { role: 'user', content: 'This turn is not cached.' },
    ],
    prompt_caching: {
      enabled: true,
      ttl: '1h',
      cut_after_message_index: 1,
    },
  },
  {
    headers: { 'anthropic-beta': 'prompt-caching-2024-07-31' },
  },
);

​

Cache Consistency with stickyProvider

{
  "model": "claude-sonnet-4-5-20250929",
  "messages": [...],
  "prompt_caching": {
    "enabled": true,
    "ttl": "5m",
    "stickyProvider": true
  }
}

• stickyProvider: false (default) — If the primary service fails, NanoGPT automatically retries with a backup service. Your request succeeds, but the cache may be lost (you’ll pay full price for that request and need to rebuild the cache).
• stickyProvider: true — If the primary service fails, NanoGPT returns a 503 error instead of failing over. Your cache remains intact for when the service recovers.

• You have very large cached contexts where cache misses are expensive
• You prefer to retry failed requests yourself rather than pay for cache rebuilds
• Cost predictability is more important than request success rate

• You prefer requests to always succeed when possible
• Occasional cache misses are acceptable
• You’re using shorter contexts where cache rebuilds are inexpensive

{
  "error": {
    "message": "Service is temporarily unavailable. Fallback disabled to preserve prompt cache consistency. Switching services would invalidate your cached tokens. Remove stickyProvider option or retry later.",
    "status": 503,
    "type": "service_unavailable",
    "code": "fallback_blocked_for_cache_consistency"
  }
}

​

Troubleshooting

• 400 unsupported image: ensure the image is a valid PNG/JPEG/WebP, not a tiny 1×1 pixel, and either HTTPS URL or a base64 data URL.
• 503 after fallbacks: try a different model, verify API key/session, and prefer base64 data URL for local or protected assets.
• Missing usage events: confirm include_usage is true in the payload or that prompt caching is enabled.

​

Context Memory

• Append :memory to any model name
• Or send header memory: true
• Can be combined with web search: :online:memory
• Retention: default 30 days; configure via :memory-<days> (1..365) or header memory_expiration_days: <days>; header takes precedence

import requests

BASE_URL = "https://nano-gpt.com/api/v1"
API_KEY = "YOUR_API_KEY"

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

# Suffix-based
payload = {
    "model": "chatgpt-4o-latest:memory",
    "messages": [{"role": "user", "content": "Keep our previous discussion in mind and continue."}]
}
requests.post(f"{BASE_URL}/chat/completions", headers=headers, json=payload)

​

Custom Context Size Override

• Parameter: model_context_limit (number or numeric string)
• Default: Derived from the selected model’s context size
• Minimum: Values below 10,000 are clamped internally
• Scope: Only affects memory compression; does not change the target model’s own window

# Enable memory via header; use model default context size
curl -s -X POST \
  -H "Authorization: Bearer $NANOGPT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "memory: true" \
  https://nano-gpt.com/api/v1/chat/completions \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Briefly say hello."}],
    "stream": false
  }'

# Explicit numeric override
curl -s -X POST \
  -H "Authorization: Bearer $NANOGPT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "memory: true" \
  https://nano-gpt.com/api/v1/chat/completions \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Briefly say hello."}],
    "model_context_limit": 20000,
    "stream": false
  }'

# String override (server coerces to number)
curl -s -X POST \
  -H "Authorization: Bearer $NANOGPT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "memory: true" \
  https://nano-gpt.com/api/v1/chat/completions \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Briefly say hello."}],
    "model_context_limit": "30000",
    "stream": false
  }'

​

Reasoning Streams

​

Endpoint variants

• https://nano-gpt.com/api/v1/chat/completions — default endpoint that streams internal thoughts through choices[0].delta.reasoning (and repeats them in message.reasoning on completion). Recommended for apps like SillyTavern that understand the modern response shape.
• https://nano-gpt.com/api/v1legacy/chat/completions — legacy contract that swaps the field name to choices[0].delta.reasoning_content / message.reasoning_content for older OpenAI-compatible clients. Use this for LiteLLM’s OpenAI adapter to avoid downstream parsing errors.
• https://nano-gpt.com/api/v1thinking/chat/completions — reasoning-aware models write everything into the normal choices[0].delta.content stream so clients that ignore reasoning fields still see the full conversation transcript. This is the preferred base URL for JanitorAI.

​

Streaming payload format

data: {
  "choices": [{
    "delta": {
      "reasoning": "Assessing possible tool options…"
    }
  }]
}
data: {
  "choices": [{
    "delta": {
      "content": "Let me walk you through the solution."
    }
  }]
}

​

Showing or hiding reasoning

curl -X POST https://nano-gpt.com/api/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet-20241022",
    "messages": [{"role": "user", "content": "What is 2+2?"}],
    "reasoning": {"exclude": true}
  }'

{
  "choices": [{
    "message": {
      "content": "The answer is 4.",
      "reasoning": "The user is asking for a simple addition. 2+2 equals 4."
    }
  }]
}

{
  "choices": [{
    "message": {
      "content": "The answer is 4."
    }
  }]
}

​

Reasoning Effort

​

Parameter: reasoning_effort

​

Usage

curl -X POST https://nano-gpt.com/api/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "o3-mini",
    "messages": [
      {"role": "user", "content": "Explain quantum entanglement step by step"}
    ],
    "reasoning_effort": "high",
    "max_tokens": 4096
  }'

{
  "model": "o3-mini",
  "messages": [{"role": "user", "content": "Solve this complex math problem..."}],
  "reasoning": {
    "effort": "high"
  }
}

​

Combining effort with exclude

{
  "model": "deepseek-reasoner",
  "messages": [{"role": "user", "content": "..."}],
  "reasoning": {
    "effort": "high",
    "exclude": false
  }
}

​

Model suffix: :reasoning-exclude

• Equivalent to sending { "reasoning": { "exclude": true } }
• Only the :reasoning-exclude suffix is stripped before the request is routed; other suffixes remain active
• Works for streaming and non-streaming responses on both Chat Completions and Text Completions

{
  "model": "claude-3-5-sonnet-20241022:reasoning-exclude",
  "messages": [{ "role": "user", "content": "What is 2+2?" }]
}

​

Combine with other suffixes

• :thinking (and variants like …-thinking:8192)
• :online and :online/linkup-deep
• :memory and :memory-<days>

• claude-3-7-sonnet-thinking:8192:reasoning-exclude
• gpt-4o:online:reasoning-exclude
• claude-3-5-sonnet-20241022:memory-30:online/linkup-deep:reasoning-exclude

​

Legacy delta field compatibility

{
  "model": "openai/gpt-4o-mini",
  "messages": [...],
  "reasoning": {
    "delta_field": "reasoning_content"
  }
}

​

Notes and limitations

• GPU-TEE models (phala/*) require byte-for-byte SSE passthrough for signature verification. For those models, streaming cannot be filtered; the suffix has no effect on the streaming bytes.
• When assistant content is an array (e.g., vision/text parts), only text parts are filtered; images and tool/metadata content are untouched.

​

YouTube Transcripts

​

Defaults

• Parameter: youtube_transcripts (boolean)
• Default: false (opt-in)
• Opt-in: set youtube_transcripts to true (string "true" is also accepted) to fetch transcripts
• Limit: Up to 3 YouTube URLs processed per request
• Injection: Transcripts are added as a system message before your messages
• Billing: $0.01 per transcript fetched

​

Enable automatic transcripts

curl -X POST https://nano-gpt.com/api/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "Summarize this: https://youtu.be/dQw4w9WgXcQ"}
    ],
    "youtube_transcripts": true
  }'

​

Notes

• Web scraping is separate. To scrape non‑YouTube URLs, set scraping: true. YouTube transcripts do not require scraping: true.
• When not requested, YouTube links are ignored for transcript fetching and are not billed.
• If your balance is insufficient when enabled, the request may be blocked with a 402.

​

Performance Benchmarks

​

Important Notes

• Web search increases input token count, which affects total cost
• Models gain access to real-time information published less than a minute ago
• Internet connectivity can provide up to 10x improvement in factuality
• All models support web search - append a suffix or send a linkup object