Offlineaid pack builder

Offline Pack Builder

Build portable, FTS5-searchable SQLite knowledge packs from any data source for on-device AI agents that work without internet.

When to use this skill

Use this when the user wants to:

• Create an offline knowledge pack for a country, city, or scenario
• Package government open data, POI data, emergency info, or any structured data into a portable SQLite database
• Build data packs for mobile apps, edge devices, or offline-first agents
• Convert scraped web data, API responses, CSV/JSON files, or MCP tool output into a searchable offline pack

Architecture

A knowledge pack is a single SQLite .db file containing:

Required:

1. pack_metadata — key-value metadata (name, version, country, scenario, vector_method…)
2. chunks — text + JSON data rows with token estimates for LLM context budgeting
3. fts_chunks — FTS5 virtual table, content-synced to chunks (no data duplication)

Optional: 4. chunk_vectors — corpus-derived vectors (LSA/TF-IDF SVD); method in metadata, not per-row 5. geo_points — lat/lon points for spatial queries 6. layers — organisational grouping with tier classification

Three-tier data classification

Every data layer is classified into one of three tiers:

Tier	Name	Description	Example
1	static_reference	Rarely changes, safe to cache for months	Hospital locations, bus routes, embassy addresses
2	periodic_snapshot	Changes weekly/monthly, refresh when online	Fuel prices, A&E wait times, weather forecasts
3	realtime_cache	Changes constantly, cache only briefly	Live traffic, bus ETAs, stock prices

Tiers 1 and 2 go into the pack. Tier 3 data is noted in layers as realtime-only (not packaged, but the agent knows about it).

How to build a pack

Step 1: Create a pack manifest

Create a JSON manifest describing what the pack should contain:

{
  "name": "au-scams",
  "version": "1.0.0",
  "country": "AU",
  "scenario": "Anti-scam and consumer safety",
  "layers": [
    {
      "name": "scam_categories",
      "tier": "static_reference",
      "source": {"type": "mcp", "server": "open-data", "tool": "search_datasets", "args": {"query": "scam consumer protection", "countries": ["au"]}},
      "description": "ACCC Scamwatch scam categories and red flags"
    },
    {
      "name": "report_channels",
      "tier": "static_reference",
      "source": {"type": "url", "url": "https://www.scamwatch.gov.au/report-a-scam", "format": "json"},
      "description": "How and where to report scams in Australia (Scamwatch, AFP, AFCA, ACMA)"
    },
    {
      "name": "consumer_rights",
      "tier": "periodic_snapshot",
      "source": {"type": "file", "path": "./data/au-consumer-rights.csv", "format": "csv"},
      "description": "Refund, chargeback, and banking-dispute paths under Australian Consumer Law",
      "publisher": "ACCC",
      "license": "CC-BY-4.0-AU",
      "source_url": "https://www.accc.gov.au/...",
      "reviewed_at": "2026-04-28T00:00:00Z",
      "expires_at": "2026-07-28T00:00:00Z",
      "language": "en-AU"
    },
    {
      "name": "emergency_phrases",
      "tier": "static_reference",
      "source": {"type": "inline", "data": [
        {"phrase_en": "I think I've been scammed", "phrase_local": "我觉得我被骗了", "category": "report"},
        {"phrase_en": "Please freeze my account", "phrase_local": "الرجاء تجميد حسابي", "category": "banking"}
      ]},
      "description": "Multilingual anti-scam phrases (EN / 简体中文 / العربية)"
    }
  ]
}

Step 2: Gather data for each layer

For each layer in the manifest, fetch data using the appropriate method based on source.type:

mcp — Query an MCP server tool

Use any available MCP tool. The agent calls the tool, receives structured data, and passes it to the packager.

Example: call open-data search_datasets with query="hospital" countries=["hk"]
Then: call open-data get_dataset_details for each result
Then: call open-data get_data_preview to get actual rows

This is not limited to open-data. Any MCP server works — Overpass for OSM data, fuel-server for prices, a custom server for proprietary data, etc.

url — Fetch from a URL (API or download)

# JSON API
curl -s "https://data.gov.au/api/3/action/package_show?id=accc-scamwatch-scam-reports" | python3 -m json.tool

# CSV download
curl -sL "https://example.com/data.csv" -o ./data/layer.csv

The agent can also use web scraping tools (browser, fetch, etc.) if the data isn't available via clean API.

file — Read a local file

The user may provide CSV, JSON, GeoJSON, or other files. The packager accepts these directly:

python3 {baseDir}/packager.py add-layer \
  --pack ./output/au-scams.db \
  --name hospitals \
  --tier static_reference \
  --file ./data/hospitals.csv \
  --format csv \
  --description "Hospital locations" \
  --publisher "ACCC" \
  --license "CC-BY-4.0-AU" \
  --source-url "https://www.scamwatch.gov.au/..." \
  --reviewed-at "2026-04-28T00:00:00Z" \
  --language "en-AU"

Provenance flags (optional, all per-layer)

Provenance is captured per-source so a multi-source pack attributes each chunk to the correct publisher in the on-device nutrition-label UI:

Flag	Purpose
--publisher	Source publisher (e.g. ACCC, NASC, Department of PM&C)
--license	Licence id, prefer SPDX-style (e.g. CC-BY-4.0-AU)
--source-url	Canonical URL of the upstream document
--reviewed-at	ISO-8601 UTC timestamp of last human review
--cultural-sensitivity	Free-text sensitivity flag (e.g. indigenous-protocols)
--expires-at	ISO-8601 UTC; UI may warn after this date
--language	BCP-47 tag of the source content (e.g. en-AU, zh-Hans, ar)

The same fields are accepted as keys on each layer block in the build manifest (see manifest example above) — they round-trip into the pack's layers table.

scrape — Web scraping output

If the user scraped a webpage, save the extracted data as JSON or CSV first, then use the file source type. The agent should structure scraped data before packaging:

[
  {"category": "Phishing email", "red_flag": "Urgency + bank-detail request", "action": "Do not click; report at scamwatch.gov.au"},
  ...
]

inline — Data provided directly in the manifest

Small datasets (phrases, emergency numbers, metadata) can be embedded directly in the manifest JSON.

Step 3: Package into SQLite

Use the packager CLI to create or add to a pack:

# Initialize a new pack
python3 {baseDir}/packager.py init \
  --output ./output/au-scams.db \
  --name "au-scams" \
  --version "1.0.0" \
  --country "AU" \
  --scenario "Anti-scam and consumer safety"

# Add a layer from a JSON file
python3 {baseDir}/packager.py add-layer \
  --pack ./output/au-scams.db \
  --name hospitals \
  --tier static_reference \
  --file ./data/hospitals.json \
  --format json \
  --description "Hospital and clinic locations"

# Extract geo points for the layer (geo_fields are NOT a flag on add-layer;
# use the separate add-geo command or declare geo_fields in the build manifest)
python3 {baseDir}/packager.py add-geo \
  --pack ./output/au-scams.db \
  --layer hospitals \
  --lat-field lat \
  --lon-field lon \
  --name-field name

# Add a layer from CSV
python3 {baseDir}/packager.py add-layer \
  --pack ./output/au-scams.db \
  --name bus_routes \
  --tier static_reference \
  --file ./data/bus-routes.csv \
  --format csv \
  --description "Bus route stops and schedules"

# Add a layer from inline JSON (piped)
echo '[{"phrase_en":"I think I have been scammed","phrase_local":"我觉得我被骗了"}]' | \
  python3 {baseDir}/packager.py add-layer \
  --pack ./output/au-scams.db \
  --name emergency_phrases \
  --tier static_reference \
  --format json \
  --description "Anti-scam phrases" \
  --from-stdin

# Add geo points for a layer
python3 {baseDir}/packager.py add-geo \
  --pack ./output/au-scams.db \
  --layer hospitals \
  --lat-field lat \
  --lon-field lon \
  --name-field name

# Build FTS5 index
python3 {baseDir}/packager.py build-index \
  --pack ./output/au-scams.db

# Optional: generate corpus-derived vectors (LSA, zero deps)
python3 {baseDir}/packager.py vectorise \
  --pack ./output/au-scams.db \
  --method lsa \
  --dimensions 64

# Optional: load pre-computed vectors from file
python3 {baseDir}/packager.py vectorise \
  --pack ./output/au-scams.db \
  --method file \
  --file ./data/vectors.json

# Show pack summary
python3 {baseDir}/packager.py info \
  --pack ./output/au-scams.db

# Query the pack (test)
python3 {baseDir}/packager.py query \
  --pack ./output/au-scams.db \
  --search "hospital emergency" \
  --layer hospitals \
  --limit 5

Verify a pack archive

Any receiver can check structural and cryptographic integrity of a .oapack.zip without writing Python:

python3 {baseDir}/packager.py verify ./output/scam-resilience-au.oapack.zip

Prints OK with pack name, sha256, format version, and builder version — or FAIL <reason> and exits 1. Wraps the same _validate_archive_contract used at archive time.

Compiler Agent Tool Contract

The offlineaid-compiler model is trained to drive the full pack-building pipeline via these five PydanticAI tools defined in agent.py:

Tool	Purpose	Arguments
extract_chunks	Adds markdown text to pack	text, pack_path, source_file, language
translate_chunk	Translates a single chunk	chunk (text), target_lang, pack_path
derive_provenance	Extracts source metadata	source_url, pack_path
propose_geo_points	Extracts lat/lon points	text, region, pack_path
validate_pack	Verifies archive integrity	pack_path

Usage in Agent Loop

The agent receives pre-extracted markdown from Stage 1. It iterates through sections, calling extract_chunks for each. It may optionally call translate_chunk for multilingual packs or propose_geo_points if coordinates are detected. Finally, it must call validate_pack before emission.

All tools take pack_path (the .db file) as an explicit argument to remain stateless.

Step 4: Build from manifest (all-in-one)

If the manifest has all layers with file or inline sources resolved, build the entire pack at once:

python3 {baseDir}/packager.py build \
  --manifest ./manifests/au-scams.json \
  --output ./output/au-scams.db

For layers with mcp or url sources, the agent must first resolve them to local files, then run build.

Pack spec (.db contract)

Required tables:

Table	Purpose
chunks	Core data store — text + JSON per row, with token estimates
fts_chunks	FTS5 virtual table, content-synced to chunks (no data duplication)
pack_metadata	Key-value metadata (name, version, country, scenario, vector_method, etc.)

Optional tables:

Table	Purpose
chunk_vectors	Corpus-derived vectors (method in metadata, NOT per-row)
geo_points	Lat/lon points for spatial queries
layers	Organisational grouping with tier classification

Schema SQL

-- Required: pack metadata (key-value)
CREATE TABLE pack_metadata (
    key   TEXT PRIMARY KEY,
    value TEXT NOT NULL
);
-- Keys: name, version, country, scenario, created_at, builder_version, sources
-- Vector keys (when present): has_vectors, vector_method, vector_dimensions

-- Required: text chunks
CREATE TABLE chunks (
    id       INTEGER PRIMARY KEY AUTOINCREMENT,
    text     TEXT NOT NULL,       -- flattened searchable text
    source   TEXT NOT NULL,       -- layer name
    section  TEXT DEFAULT '',     -- sub-section (optional)
    data     TEXT,                -- full JSON object
    tokens   INTEGER DEFAULT 0   -- estimated token count for LLM context budgeting
);

-- Required: FTS5 index (content-synced to chunks, no data duplication)
CREATE VIRTUAL TABLE fts_chunks USING fts5(
    text, source, section,
    content='chunks', content_rowid='id'
);

-- Optional: corpus-derived vectors
-- Method and dimensions stored in pack_metadata, NOT per-row.
-- No neural model reference. User/agent decides at build time.
CREATE TABLE chunk_vectors (
    chunk_id   INTEGER PRIMARY KEY REFERENCES chunks(id),
    vector     BLOB NOT NULL      -- float32[] as little-endian bytes
);

-- Optional: geo points
CREATE TABLE geo_points (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    name       TEXT NOT NULL,
    name_local TEXT,
    category   TEXT NOT NULL,
    lat        REAL NOT NULL,
    lon        REAL NOT NULL,
    address    TEXT,
    address_local TEXT,
    district   TEXT,
    metadata   TEXT,              -- JSON: opening hours, phone, etc.
    chunk_id   INTEGER REFERENCES chunks(id),
    layer      TEXT REFERENCES layers(name)
);

-- Optional: organisational layers
CREATE TABLE layers (
    name        TEXT PRIMARY KEY,
    tier        TEXT NOT NULL CHECK(tier IN ('static_reference','periodic_snapshot','realtime_cache')),
    description TEXT,
    row_count   INTEGER DEFAULT 0,
    source_type TEXT,
    source_info TEXT,
    added_at    TEXT NOT NULL
);

Example: full agent workflow

Here's how to build a pack end-to-end — whether you're an agent runtime or a human at a terminal:

1. User says: "Build me an offline pack for Australian anti-scam guidance"
2. Agent discovers data sources:
   • Checks if open-data MCP server is available → uses search_datasets to find AU consumer-protection data
   • Checks if overpass-server is available → uses query_pois_in_bbox for local reporting offices
   • Falls back to web scraping (ACCC Scamwatch, MoneySmart) if no MCP server available
3. Agent fetches data from each source, saves as JSON files
4. Agent creates manifest listing each layer with source type, tier, description
5. Agent runs packager via the CLI commands above
6. Agent reports pack size, layer count, entry count to user

The agent adapts to whatever tools are available. MCP servers are preferred (structured data), but the skill works with raw HTTP, scraping, or user-provided files too.

Tips

• Keep packs under 100MB for mobile deployment. Split by scenario if needed.
• Use static_reference for most data. Only use periodic_snapshot for data that genuinely changes weekly/monthly.
• Always add geo points for location-based data — on-device agents use bounding-box queries for spatial search.
• Test with query before shipping — make sure FTS search returns sensible results.
• The packager is a standalone Python script with zero dependencies beyond the Python standard library (sqlite3, json, csv are all built-in).