Azure openai patterns

Azure OpenAI Patterns

Rate limiting, function calling, error handling, and token optimization for Azure OpenAI API.

Staleness Watch: See EXTERNAL-API-REGISTRY.md for source URLs and recheck cadence

Version: 1.1.0 | Last validated: April 2026 (GPT-5.x, Responses API, Structured Outputs)

---

Rate Limiting: The Dual System

Azure OpenAI uses dual rate limits: Tokens Per Minute (TPM) and Requests Per Minute (RPM). The ratio is typically 6 RPM per 1000 TPM.

TPM vs RPM Relationship

Model	Tier	TPM	RPM	Notes
gpt-5.4-mini	Default	2M	12K	Latest flagship (mini)
gpt-5.2	Default	1M	6K	Reasoning model
gpt-4.1	Default	1M	6K	1M context, structured outputs
gpt-4.1-mini	Default	2M	12K	Cost-efficient 1M context
o4-mini	Default	200K	1.2K	Reasoning (o-series)
o3	Default	200K	1.2K	Advanced reasoning
gpt-4o	Default	450K	2.7K	Legacy — prefer gpt-4.1+
gpt-4o-mini	Default	2M	12K	Legacy — prefer gpt-4.1-mini

Migration: gpt-4o → gpt-4.1 (same API, larger context, better quality). gpt-4o-mini → gpt-4.1-mini or gpt-4.1-nano (cost savings).

How TPM is Calculated

TPM is estimated before processing based on:

1. Prompt text character count (converted to estimated tokens)
2. max_tokens parameter setting
3. best_of parameter setting (if used)

The rate limit estimate is NOT the same as actual token consumption for billing.

Burst vs Sustained Limits

RPM is enforced over small time windows (1-10 seconds):

600 RPM deployment = max 10 requests per second
If you send 15 requests in 1 second → 429 error
Even though 15/min < 600/min

---

Responses API (New Standard)

GPT-5.x and newer models support the Responses API alongside Chat Completions. The Responses API is the recommended API for new development — it supports stateful conversations, built-in tools (web search, code interpreter, file search, computer use), and simpler multi-turn management.

// Responses API — simpler multi-turn
const response = await client.responses.create({
  model: "gpt-5.2",
  input: "Summarize Q3 results from the uploaded file",
  tools: [{ type: "file_search" }],  // built-in tools
});

// Multi-turn: pass previous response ID
const followUp = await client.responses.create({
  model: "gpt-5.2",
  input: "What were the key risks mentioned?",
  previous_response_id: response.id,
});

When to use which: Responses API for new projects. Chat Completions API remains fully supported and required for fine-tuned models and some legacy patterns.

Structured Outputs

Request guaranteed JSON Schema conformance with strict: true. Eliminates JSON parsing failures.

const response = await client.chat.completions.create({
  model: "gpt-4.1",
  messages: [{ role: "user", content: "Extract name and age from: John is 30" }],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "person",
      strict: true,
      schema: {
        type: "object",
        properties: {
          name: { type: "string" },
          age: { type: "integer" }
        },
        required: ["name", "age"],
        additionalProperties: false
      }
    }
  }
});
// Guaranteed valid JSON matching the schema — no parsing errors

Supported by gpt-4.1, gpt-4o, gpt-5.x, and o-series models. Also works with function calling (strict: true in function definitions).

Reasoning Models (o-series, GPT-5.x)

o3, o4-mini, and GPT-5.x models support reasoning_effort (low, medium, high). GPT-5.1+ defaults reasoning_effort to none — you must explicitly set it if you want reasoning.

const response = await client.chat.completions.create({
  model: "o4-mini",
  messages: [{ role: "user", content: "Prove the Pythagorean theorem" }],
  reasoning_effort: "high",  // low | medium | high
});

Reasoning models do NOT support temperature, top_p, presence_penalty, or frequency_penalty. Remove these when migrating from GPT-4o to reasoning models.

---

Function Calling Patterns

Pattern 1: Exponential Backoff with Status Callback

async function chatWithTools(
  messages: ChatCompletionMessage[],
  tools: Tool[],
  onStatusUpdate?: (status: string) => void
): Promise<ChatCompletionResponse> {
  const maxRetries = 5;

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    const response = await fetch(apiUrl, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${token}`,
      },
      body: JSON.stringify({ messages, tools, tool_choice: 'auto' }),
    });

    if (response.ok) {
      return response.json();
    }

    if (response.status === 429 && attempt < maxRetries) {
      const waitTime = Math.pow(2, attempt);
      onStatusUpdate?.(`Rate limited. Waiting ${waitTime}s...`);
      await new Promise(resolve => setTimeout(resolve, waitTime * 1000));
      continue;
    }

    throw new Error(`API error: ${response.status}`);
  }
}

Pattern 2: Optimize max_tokens

// Bad: Set unnecessarily high max_tokens — uses full quota even for short responses
const badRequest = { messages: [...], max_tokens: 4000 };

// Good: Set appropriate max_tokens for expected response length
const goodRequest = { messages: [...], max_tokens: 500 };

Pattern 3: Tool Result Batching

// Bad: Send one request per tool result (consumes RPM quota)
for (const toolCall of toolCalls) {
  const result = await executeFunction(toolCall);
  await sendToolResult(result);
}

// Good: Collect all results and send once
const results = await Promise.all(
  toolCalls.map(tc => executeFunction(tc))
);
await sendToolResults(results); // Single request

---

Response Headers to Monitor

const headers = response.headers;
const remainingRequests = headers.get('x-ratelimit-remaining-requests');
const remainingTokens = headers.get('x-ratelimit-remaining-tokens');
const resetRequests = headers.get('x-ratelimit-reset-requests');
const resetTokens = headers.get('x-ratelimit-reset-tokens');
const retryAfter = headers.get('Retry-After'); // Only on 429

---

Function Design Best Practices

1. Minimize Token Consumption

// Bad: Return entire resource objects
{ name: 'get_resources', description: 'Get all Azure resources' }
// Returns: huge JSON with all properties

// Good: Return only necessary fields
{ name: 'get_resources', description: 'Get Azure resource summary' }
// Returns: { name, type, status } only

2. Use parallel_tool_calls

const request = {
  messages,
  tools,
  parallel_tool_calls: true, // Default: true in newer models
};
// Model may call multiple tools in one response, reducing round trips

3. Request Queuing for High Volume

class RequestQueue {
  private queue: Array<() => Promise<void>> = [];
  private processing = false;
  private minDelayMs = 100; // 10 req/sec max

  async enqueue<T>(request: () => Promise<T>): Promise<T> {
    return new Promise((resolve, reject) => {
      this.queue.push(async () => {
        try { resolve(await request()); }
        catch (e) { reject(e); }
        await this.delay(this.minDelayMs);
      });
      this.process();
    });
  }

  private async process() {
    if (this.processing) return;
    this.processing = true;
    while (this.queue.length > 0) {
      const next = this.queue.shift();
      await next?.();
    }
    this.processing = false;
  }

  private delay(ms: number) {
    return new Promise(r => setTimeout(r, ms));
  }
}

---

Error Codes and Handling

Code	Meaning	Action
429	Rate limited	Exponential backoff, check Retry-After
400	Invalid request	Check request format, content filter
401	Authentication error	Refresh token
403	Quota exceeded	Wait or upgrade tier
500	Server error	Retry with backoff
503	Service unavailable	Retry with longer backoff

Content Filter Handling

if (response.status === 400) {
  const error = await response.json();
  if (error.error?.code === 'content_filter') {
    return { message: 'Content was filtered by safety policy.', filtered: true };
  }
}

---

Recommended Settings

Setting	Value	Rationale
max_tokens	500-2000	Sized for expected response
temperature	0.3-0.7	Lower for tool calling, higher for creative
retry attempts	5	Handles transient rate limits
base delay	2000ms	Start at 2s for backoff
max delay	60000ms	Cap at 1 minute

---

References

• Azure OpenAI quotas and limits
• Manage Azure OpenAI quota
• Best practices for function calling

---

Activation Patterns

Trigger	Response
"azure openai", "rate limit", "429"	Full skill activation
"function calling", "tool calling"	Function Calling Patterns section
"token optimization", "max_tokens"	Pattern 2 + Recommended Settings
"retry", "backoff"	Pattern 1 + Error Codes
"request queue", "high volume"	Pattern 3