Frappe impl integrations

Frappe Integrations

Step-by-step workflows for OAuth, Webhooks, Payment Gateways, Data Import/Export, and external API calls.

Version: v14/v15/v16

---

Decision Tree: Which Integration Pattern?

WHAT ARE YOU INTEGRATING?
│
├─► External service needs to call YOUR Frappe site?
│   ├─► On document events → Webhook (push to external)
│   ├─► External sends data to you → Whitelisted API endpoint
│   └─► External needs user auth → OAuth 2.0 Provider
│
├─► YOUR Frappe site calls an external service?
│   ├─► Needs user-level OAuth consent → Connected App
│   ├─► Server-to-server with API key → make_request / requests
│   └─► Recurring sync → Scheduler + API calls
│
├─► Bulk data in/out?
│   ├─► Import CSV/XLSX → Data Import DocType
│   ├─► Export data → Report Builder / export-csv / API
│   └─► Programmatic bulk → frappe.get_doc().insert()
│
├─► Payment processing?
│   └─► Payment Request + Payment Gateway controller
│
└─► Real-time vs batch?
    ├─► Real-time → Webhook or API endpoint
    ├─► Near real-time → frappe.enqueue() after event
    └─► Batch → Scheduler task (hourly/daily)

---

Workflow 1: OAuth 2.0: Frappe as Provider

Use when external applications need "Sign in with Frappe" or API access on behalf of users.

Step 1: Configure OAuth Provider Settings

Navigate to Setup > Integrations > OAuth Provider Settings:

• Force: ALWAYS asks user for confirmation
• Auto: Asks only if no active token exists

Step 2: Create OAuth Client

Navigate to Setup > Integrations > OAuth Client:

Field	Value
App Name	External app identifier
Scopes	Space-separated (e.g., openid all)
Redirect URIs	Space-separated callback URLs
Default Redirect URI	Primary callback URL
Grant Type	Authorization Code (RECOMMENDED) or Implicit
Response Type	Code (for Auth Code) or Token (for Implicit)
Skip Authorization	Check for trusted first-party apps only

Step 3: Use the Generated Endpoints

Endpoint	URL
Authorize	/api/method/frappe.integrations.oauth2.authorize
Token	/api/method/frappe.integrations.oauth2.get_token
Profile	/api/method/frappe.integrations.oauth2.openid_profile

Step 4: Configure External App

# Example: Grafana generic_oauth config
client_id = <generated_client_id>
client_secret = <generated_client_secret>
auth_url = https://your-frappe.com/api/method/frappe.integrations.oauth2.authorize
token_url = https://your-frappe.com/api/method/frappe.integrations.oauth2.get_token
api_url = https://your-frappe.com/api/method/frappe.integrations.oauth2.openid_profile
scopes = openid all

Critical Rules

• NEVER use Implicit grant type for server-side apps — use Authorization Code
• ALWAYS use HTTPS in production for all OAuth endpoints
• NEVER expose client_secret in client-side JavaScript

---

Workflow 2: Connected App: Frappe as OAuth Consumer

Use when your Frappe instance needs to access external services (Google, Microsoft, etc.) on behalf of users.

Step 1: Create Connected App DocType

Field	Purpose
Name	Identifier for the connection
OpenID Configuration URL	Auto-fetches endpoints (e.g., /.well-known/openid-configuration)
Authorization URI	Consent screen URL (auto-filled from OpenID)
Token URI	Token exchange URL (auto-filled from OpenID)
Redirect URI	Auto-generated — copy this to external provider
Client ID	From external provider
Client Secret	From external provider
Scopes	Permissions needed (e.g., https://mail.google.com/)

Step 2: Register Redirect URI with Provider

Copy the auto-generated Redirect URI and register it in the external provider's OAuth console.

Step 3: Add Extra Parameters (if needed)

access_type=offline    # Google: enables refresh tokens
prompt=consent         # Google: forces re-consent for refresh token

Step 4: Use in Code

import frappe

connected_app = frappe.get_doc("Connected App", "My Google App")
# Initiates OAuth flow — user clicks "Connect to..." button
# After consent, tokens are stored automatically

# Making authenticated calls:
session = connected_app.get_oauth2_session()
response = session.get("https://www.googleapis.com/gmail/v1/users/me/messages")

Critical Rules

• ALWAYS add access_type=offline for Google APIs to get refresh tokens
• NEVER store tokens manually — Connected App manages token lifecycle
• ALWAYS handle TokenExpiredError — call session.refresh_token() or reconnect

---

Workflow 3: Webhooks: Push Notifications to External Services

Step 1: Create Webhook DocType

Navigate to Integrations > Webhook:

Field	Value
DocType	Target document type
Doc Event	on_update, after_insert, on_submit, on_cancel, on_trash
Request URL	External endpoint
Request Method	POST (default)
Conditions	Optional Jinja filter (e.g., doc.status == "Approved")
Enabled	Check to activate

Step 2: Configure Headers

Add custom headers for authentication:

Authorization: Bearer <api_token>
Content-Type: application/json

Step 3: Configure Data: Choose Format

Form URL-encoded: Select specific fields from a table.

JSON: Use Jinja templates for structured payloads:

{
  "id": "{{ doc.name }}",
  "total": "{{ doc.grand_total }}",
  "items": {{ doc.items | tojson }},
  "event": "{{ event }}"
}

Step 4: Enable Webhook Secret (HMAC Verification)

Set a Webhook Secret — Frappe adds X-Frappe-Webhook-Signature header with base64-encoded HMAC-SHA256 hash of the payload.

Receiver verification (Python example):

import hmac, hashlib, base64

def verify_webhook(payload_body, secret, signature_header):
    expected = base64.b64encode(
        hmac.new(secret.encode(), payload_body, hashlib.sha256).digest()
    ).decode()
    return hmac.compare_digest(expected, signature_header)

Critical Rules

• ALWAYS enable Webhook Secret for production webhooks
• NEVER rely on webhooks for guaranteed delivery — implement idempotency on the receiver
• ALWAYS use | tojson filter for child table data in JSON payloads
• Webhook logs are created for every delivery — check Webhook Request Log for debugging

---

Workflow 4: External API Calls from Frappe

Using frappe.integrations.utils

from frappe.integrations.utils import make_get_request, make_post_request

# GET request
response = make_get_request(
    "https://api.example.com/data",
    headers={"Authorization": "Bearer token123"}
)

# POST request
response = make_post_request(
    "https://api.example.com/submit",
    data={"key": "value"},
    headers={"Content-Type": "application/json"}
)

Using requests Library Directly

import requests
import frappe

def sync_to_external():
    try:
        response = requests.post(
            "https://api.example.com/endpoint",
            json={"data": "value"},
            timeout=30
        )
        response.raise_for_status()
        return response.json()
    except requests.exceptions.RequestException as e:
        frappe.log_error(f"API call failed: {e}", "Integration Error")
        raise

Critical Rules

• ALWAYS set a timeout on external requests (30s recommended)
• ALWAYS wrap external calls in try/except and log errors with frappe.log_error()
• NEVER call external APIs inside validate or before_save — use on_update + frappe.enqueue()
• ALWAYS use frappe.enqueue() for slow external calls to avoid blocking the web request

---

Workflow 5: Data Import

Via UI (Data Import DocType)

1. Navigate to Home > Data Import > New
2. Select DocType and Import Type (Insert or Update)
3. Download template CSV/XLSX
4. Fill in data following the template format
5. Upload and preview
6. Start Import

CSV Format Rules

ID,Item Name,Item Group,Stock UOM
,Widget A,Products,Nos
,Widget B,Raw Material,Kg

• First row: field labels or API field names
• Leave ID/name empty for Insert (auto-generated)
• For Update: ID column MUST contain existing document names
• Child tables: repeat parent row data, add child fields as extra columns

Programmatic Import

import frappe
from frappe.core.doctype.data_import.data_import import DataImport

# Create Data Import document
di = frappe.get_doc({
    "doctype": "Data Import",
    "reference_doctype": "Item",
    "import_type": "Insert New Records",
    "import_file": "/path/to/file.csv"
})
di.insert()
di.start_import()

Critical Rules

• ALWAYS download and use the template — column order and names must match exactly
• NEVER import more than 5,000 rows at once — split into batches
• ALWAYS test with 5-10 rows first before bulk import
• ALWAYS check Import Log for row-level errors after import completes

---

Workflow 6: Data Export

Via Report Builder

1. Open any DocType list view
2. Apply filters
3. Menu > Export (CSV/Excel)

Via CLI

bench --site mysite export-csv "Sales Invoice"
bench --site mysite export-doc "Sales Invoice" "INV-001"
bench --site mysite export-json "Sales Invoice" "INV-001"
bench --site mysite export-fixtures --app myapp

Programmatic Export

import frappe

# Export filtered data
data = frappe.get_all("Sales Invoice",
    filters={"status": "Paid", "posting_date": [">", "2024-01-01"]},
    fields=["name", "customer", "grand_total", "posting_date"],
    order_by="posting_date desc",
    limit_page_length=0  # No limit
)

# Convert to CSV
import csv, io
output = io.StringIO()
writer = csv.DictWriter(output, fieldnames=["name", "customer", "grand_total", "posting_date"])
writer.writeheader()
writer.writerows(data)
csv_content = output.getvalue()

---

Workflow 7: Frappe REST API Authentication

API Key + Secret (Server-to-Server)

# Generate via User > API Access > Generate Keys
curl -H "Authorization: token api_key:api_secret" \
  https://your-site.com/api/resource/Sales%20Invoice

OAuth Bearer Token

curl -H "Authorization: Bearer access_token" \
  https://your-site.com/api/resource/Sales%20Invoice

Session-Based (Login)

# Login first
curl -X POST https://your-site.com/api/method/login \
  -d "[email protected]&pwd=password"
# Subsequent requests use session cookie

---

Integration Patterns: Sync vs Async

Pattern	When to Use	Implementation
Synchronous	Response needed immediately	Direct API call in controller
Async (enqueue)	External call > 5s	frappe.enqueue("myapp.api.sync_record", doc_name=doc.name)
Webhook	Push on event	Webhook DocType configuration
Scheduled sync	Periodic batch	scheduler_events in hooks.py
Real-time	Live updates	Socket.IO + frappe.publish_realtime()

Retry Pattern

import frappe
from frappe.utils.background_jobs import get_jobs

def sync_with_retry(doc_name, retry_count=0, max_retries=3):
    try:
        result = call_external_api(doc_name)
        frappe.db.set_value("Sales Invoice", doc_name, "sync_status", "Success")
        frappe.db.commit()
    except Exception as e:
        if retry_count < max_retries:
            frappe.enqueue(
                "myapp.integrations.sync_with_retry",
                doc_name=doc_name,
                retry_count=retry_count + 1,
                queue="short",
                enqueue_after_commit=True
            )
        else:
            frappe.log_error(f"Sync failed after {max_retries} retries: {e}")
            frappe.db.set_value("Sales Invoice", doc_name, "sync_status", "Failed")
            frappe.db.commit()

---

Version Differences

Feature	V14	V15	V16
Webhook DocType	Yes	Yes	Yes
Connected App	Yes	Yes	Yes
OAuth 2.0 Provider	Yes	Yes	Yes
Data Import (new UI)	Yes	Yes	Yes
Print Designer	No	Yes	Yes
make_get_request	Yes	Yes	Yes
Webhook HMAC	Yes	Yes	Yes

---

Reference Files

File	Contents
workflows.md	Complete integration workflow patterns
examples.md	Working code examples for all integration types
anti-patterns.md	Common integration mistakes and fixes
decision-tree.md	Extended decision trees for integration choice