Oms cocoindex

Skill armelhbobdad/oh-my-skills/skills/oms-cocoindex/0.3.37/oms-cocoindex

Use when builds data transformation flows on top of cocoindex — a Python framework with a Rust engine for ultra-performant, incremental data indexing (ETL, RAG ingestion, knowledge graphs, vector search). Covers the flow-building public API: FlowBuilder, DataScope, DataSlice, Flow, FlowLiveUpdater, sources, targets (Postgres/Qdrant/Neo4j/Pinecone/LanceDB/etc.), functions (SplitRecursively, SentenceTransformerEmbed, ExtractByLlm, EmbedText, ColPali), LlmSpec, index defs, settings, and runtime lifecycle. Use for authoring indexing flows, chunking+embedding pipelines, LLM extraction, and live updates. Do NOT use for authoring custom Rust engine components — users write flows in Python via pyo3 bindings.From its SKILL.md

Install

npx -y skills add armelhbobdad/oh-my-skills --skill oms-cocoindex

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

7 stars7 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

35.3 KB, ~9.5k tokens by cl100k_base, as published. Nobody here has run it

oms-cocoindex

Overview

cocoindex is a Python ETL framework with a Rust engine for building incremental data indexes (embeddings, knowledge graphs, vector search, LLM extraction). Users author flows in Python; the Rust engine handles incremental recomputation and target state management.

Source: cocoindex-io/cocoindex @ v0.3.37 (commit 87c5dbf0)
Forge tier: Deep — AST structural extraction + QMD temporal/docs enrichment
Exports documented: 102 public exports (T1 AST-verified) across flow, lib, index, llm, setting, auth_registry, query_handler, typing, op, sources, targets, functions, cli, utils
Confidence distribution: T1 = 102, T2 = 15, T3 = 10 (docs), T1-low = 0

Note on stability: cocoindex is Development Status 3 — Alpha. This skill is pinned to tag v0.3.37; upstream has since moved to v1.0.0-alpha*. Re-forge for newer versions.

Quick Start

End-to-end text-embedding flow — read markdown files, chunk, embed with SentenceTransformer, export to Postgres + pgvector:

import cocoindex

@cocoindex.flow_def(name="TextEmbedding")
def text_embedding_flow(
    flow_builder: cocoindex.FlowBuilder,
    data_scope: cocoindex.DataScope,
):
    data_scope["documents"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(path="markdown_files")
    )
    doc_embeddings = data_scope.add_collector()

    with data_scope["documents"].row() as doc:
        doc["chunks"] = doc["content"].transform(
            cocoindex.functions.SplitRecursively(),
            language="markdown", chunk_size=2000, chunk_overlap=500,
        )
        with doc["chunks"].row() as chunk:
            chunk["embedding"] = chunk["text"].transform(
                cocoindex.functions.SentenceTransformerEmbed(
                    model="sentence-transformers/all-MiniLM-L6-v2"
                )
            )
            doc_embeddings.collect(
                filename=doc["filename"],
                location=chunk["location"],
                text=chunk["text"],
                embedding=chunk["embedding"],
            )

    doc_embeddings.export(
        "doc_embeddings",
        cocoindex.targets.Postgres(),
        primary_key_fields=["filename", "location"],
        vector_indexes=[
            cocoindex.VectorIndexDef(
                field_name="embedding",
                metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY,
            )
        ],
    )

Run with: cocoindex update main.py (one-time) or cocoindex update main.py -L (live). Requires COCOINDEX_DATABASE_URL env var. Source: [AST:python/cocoindex/init.py:L1] · [EXT:https://cocoindex.io/docs/getting_started/quickstart]

Common Workflows

Define and register a flow (recommended): @cocoindex.flow_def(name="...") wraps a function (FlowBuilder, DataScope) -> None and registers it. [AST:python/cocoindex/flow.py:L1011]

Explicit flow registration (dynamic names): cocoindex.open_flow(name, fl_def) returns a Flow. Use when the name is computed at runtime. [AST:python/cocoindex/flow.py:L986]

One-time vs live update: demo_flow.update(print_stats=True) — finishes when target is fresh. [AST:python/cocoindex/flow.py:L771] cocoindex.FlowLiveUpdater(demo_flow) with start()/wait() or with context manager — continuous change capture. [AST:python/cocoindex/flow.py:L603]

Setup / drop target backends: Per flow: demo_flow.setup() / demo_flow.drop() (+ _async variants). [AST:python/cocoindex/flow.py:L855] [AST:python/cocoindex/flow.py:L868] All flows at once: cocoindex.setup_all_flows() / cocoindex.drop_all_flows(). [AST:python/cocoindex/flow.py:L1300] [AST:python/cocoindex/flow.py:L1309]

Transient in-memory transform (no target): @cocoindex.transform_flow() on a function (DataSlice[T], ...) -> DataSlice[U], then call .eval(...) / .eval_async(...). [AST:python/cocoindex/flow.py:L1251]

LLM structured extraction: cocoindex.functions.ExtractByLlm(llm_spec=LlmSpec(...), output_type=MyDataclass, instruction=...) → transforms text into a typed struct. [AST:python/cocoindex/functions/_engine_builtin_specs.py:L64]

Key API Summary

Export	Kind	Purpose	Provenance
`FlowBuilder`	class	Flow authoring helper (`add_source`, `transform`, `declare`)	[AST:python/cocoindex/flow.py:L495]
`DataScope`	class	Scope container for fields + collectors (top-level, per-row)	[AST:python/cocoindex/flow.py:L302]
`DataSlice`	class	Typed reference to scope field; `.row()`, `.transform()`, `.call()`	[AST:python/cocoindex/flow.py:L212]
`Flow`	class	Pipeline handle: `update`, `setup`, `drop`, `close`, `evaluate_and_dump`	[AST:python/cocoindex/flow.py:L705]
`flow_def(name=None)`	decorator	Wraps a flow function and registers it	[AST:python/cocoindex/flow.py:L1011]
`open_flow(name, fl_def)`	function	Explicit flow registration, returns `Flow`	[AST:python/cocoindex/flow.py:L986]
`transform_flow()`	decorator	Wraps a transient in-memory transform function	[AST:python/cocoindex/flow.py:L1251]
`FlowLiveUpdater`	class	Live updater with `start`/`wait`/`abort`/`next_status_updates`	[AST:python/cocoindex/flow.py:L603]
`init(settings=None)`	function	Explicit cocoindex initialization	[AST:python/cocoindex/lib.py:L58]
`start_server(settings)`	function	Start HTTP server (used by `cocoindex server` CLI)	[AST:python/cocoindex/lib.py:L67]
`LlmSpec`	dataclass	LLM target config: `api_type`, `model`, `address`, `api_config`	[AST:python/cocoindex/llm.py:L55]
`VectorIndexDef`	dataclass	`field_name`, `metric`, `method` (Hnsw/IvfFlat)	[AST:python/cocoindex/index.py:L33]
`FtsIndexDef`	dataclass	Full-text-search index (LanceDB only currently)	[AST:python/cocoindex/index.py:L44]
`add_auth_entry(key, value)`	function	Register stable-key auth (target backend identity)	[AST:python/cocoindex/auth_registry.py:L31]
`add_transient_auth_entry(value)`	function	Register ephemeral auth (sources/functions)	[AST:python/cocoindex/auth_registry.py:L25]
`Settings`	dataclass	Library settings (`database`, `app_namespace`, ...)	[AST:python/cocoindex/setting.py:L75]

Migration & Deprecation Warnings

cocoindex.add_flow_def(name, fl_def) is DEPRECATED — use cocoindex.open_flow(name, fl_def) instead. Identical behavior; the old name is kept for compatibility only. [AST:python/cocoindex/flow.py:L997]
cocoindex.remove_flow(fl) is DEPRECATED — use fl.close() instead. [AST:python/cocoindex/flow.py:L1004]
cocoindex.storages is a DEPRECATED alias for cocoindex.targets — use cocoindex.targets in new code. [AST:python/cocoindex/init.py:L12]
cocoindex.utils.get_target_storage_default_name is DEPRECATED — use get_target_default_name. [AST:python/cocoindex/utils.py:L18]
Retired Kuzu target — cocoindex.targets.Kuzu, KuzuConnection, KuzuDeclaration are backward-compat aliases for Ladybug. New code should use Ladybug. [AST:python/cocoindex/targets/_engine_builtin_specs.py:L205]
Alpha stability: cocoindex is Development Status 3 — Alpha. The public API may change across minor versions; this skill is pinned to v0.3.37. [QMD:oms-cocoindex-temporal:releases.md]

See Full API Reference for migration details.

Key Types

# Index metrics
VectorSimilarityMetric.COSINE_SIMILARITY  # "CosineSimilarity"
VectorSimilarityMetric.L2_DISTANCE        # "L2Distance"
VectorSimilarityMetric.INNER_PRODUCT      # "InnerProduct"

# LLM API types (enum)
LlmApiType.OPENAI, AZURE_OPENAI, OLLAMA, GEMINI, VERTEX_AI,
ANTHROPIC, VOYAGE, LITE_LLM, OPEN_ROUTER, VLLM, BEDROCK, NOVITA

# Generated field markers
GeneratedField.UUID  # auto-generated UUID, stable across unchanged inputs

# Annotated typing aliases (attach schema info to python types)
Int64, Float32, Float64, LocalDateTime, OffsetDateTime, Range, Json
Vector[np.float32, Literal[384]]  # dimension-parameterized vector

Provenance: [AST:python/cocoindex/index.py:L6] [AST:python/cocoindex/llm.py:L7] [AST:python/cocoindex/flow.py:L359] [AST:python/cocoindex/typing.py:L37]

Architecture at a Glance

Flow authoring (Python): flow.py — FlowBuilder, DataScope, DataSlice, DataCollector, Flow, FlowLiveUpdater, TransformFlow, flow_def / open_flow / transform_flow decorators.
Runtime lifecycle: lib.py — init, start_server, stop; implicit auto-init on first flow call.
Sources (built-in): sources/_engine_builtin_specs.py — LocalFile, GoogleDrive, AmazonS3, AzureBlob, Postgres.
Targets (built-in): targets/_engine_builtin_specs.py — Postgres, Qdrant, Pinecone, Neo4j, FalkorDB, Ladybug (+ retired Kuzu alias). Extended target modules: targets/chromadb.py (ChromaDB), targets/lancedb.py (LanceDB), targets/doris.py (DorisTarget), targets/turbopuffer.py (Turbopuffer), targets/pinecone.py (connector for built-in Pinecone spec).
Functions (built-in): functions/_engine_builtin_specs.py — ParseJson, DetectProgrammingLanguage, SplitRecursively, SplitBySeparators, EmbedText, ExtractByLlm; plus functions/sbert.py (SentenceTransformerEmbed) and functions/colpali.py (ColPaliEmbedImage, ColPaliEmbedQuery).
Operation spec framework: op.py — FunctionSpec, SourceSpec, TargetSpec, TargetAttachmentSpec, DeclarationSpec base classes; @op.function, @op.executor_class, @op.source_connector, @op.target_connector decorators for custom ops.
Auth, indexes, settings, typing: auth_registry.py, index.py, setting.py, typing.py — orthogonal support modules.
CLI: cli.py — cocoindex click entry point with ls, show, setup, drop, update, evaluate, server subcommands.

The Rust engine (rust/ directory) is intentionally out of scope — users author in Python via pyo3-compiled bindings exposed as cocoindex._engine.

CLI

The cocoindex CLI (click group [AST:python/cocoindex/cli.py:L122]):

cocoindex -e .env -d app_dir ls [APP_TARGET]                    # List flows
cocoindex show APP_FLOW_SPECIFIER [--verbose]                   # Show flow spec
cocoindex setup APP_TARGET [-f] [--reset]                       # Setup backends
cocoindex drop APP_TARGET [FLOW_NAME...] [-f]                   # Drop backends
cocoindex update APP_FLOW_SPECIFIER [-L] [--reexport]           # Build/update index
cocoindex evaluate APP_FLOW_SPECIFIER --output-dir ./eval [--no-cache]  # Dry-run
cocoindex server APP_TARGET [-a ADDR] [-L] [--reload] ...       # Start HTTP server

APP_TARGET = path/to/app.py or installed_module. APP_FLOW_SPECIFIER = APP_TARGET or APP_TARGET:FLOW_NAME. Provenance: [AST:python/cocoindex/cli.py:L143] [AST:python/cocoindex/cli.py:L195] [AST:python/cocoindex/cli.py:L337] [AST:python/cocoindex/cli.py:L364] [AST:python/cocoindex/cli.py:L456] [AST:python/cocoindex/cli.py:L522] [AST:python/cocoindex/cli.py:L642]

Full API Reference

Flow authoring — `cocoindex.flow`

See references/flow-api.md for the complete API.

FlowBuilder [AST:python/cocoindex/flow.py:L495] — wraps _FlowBuilderState. Core methods:

add_source(spec, /, *, name=None, refresh_interval=None, max_inflight_rows=None, max_inflight_bytes=None) -> DataSlice[T] [AST:python/cocoindex/flow.py:L511] — import from a source. spec must be a subclass of op.SourceSpec. refresh_interval (datetime.timedelta) triggers periodic list-and-diff in live mode. max_inflight_rows/max_inflight_bytes bound concurrent processing per source.
transform(fn_spec, *args, **kwargs) -> DataSlice[Any] [AST:python/cocoindex/flow.py:L548] — apply a function spec to one or more data slices.
declare(spec: op.DeclarationSpec) -> None [AST:python/cocoindex/flow.py:L566] — register a target declaration out-of-band (e.g., graph-target node labels referenced by relationships).

DataScope [AST:python/cocoindex/flow.py:L302]:

__getitem__(field_name) -> DataSlice[T] / __setitem__(field_name, value: DataSlice[T]) — get/add fields; cannot override existing fields.
add_collector(name=None) -> DataCollector [AST:python/cocoindex/flow.py:L345]
Context manager: __enter__ / __exit__ (used by DataSlice.for_each / row()).

DataSlice[T] [AST:python/cocoindex/flow.py:L212]:

row(*, max_inflight_rows=None, max_inflight_bytes=None) -> DataScope [AST:python/cocoindex/flow.py:L232] — per-row child scope; use as with slice.row() as row_scope:.
for_each(f, *, max_inflight_rows=None, max_inflight_bytes=None) -> None [AST:python/cocoindex/flow.py:L253] — apply a function to each row (uses row() internally).
transform(fn_spec, *args, **kwargs) -> DataSlice[Any] [AST:python/cocoindex/flow.py:L270] — transform this slice, passing it as first positional argument.
call(func, *args, **kwargs) -> S [AST:python/cocoindex/flow.py:L291] — apply a host-side callable (for composition).
__getitem__(field_name) — select a sub-field (struct type).

DataCollector [AST:python/cocoindex/flow.py:L367]:

collect(**kwargs) -> None [AST:python/cocoindex/flow.py:L381] — record an entry. Values may be DataSlice or GeneratedField.UUID.
export(target_name, target_spec, /, *, primary_key_fields, attachments=(), vector_indexes=(), fts_indexes=(), vector_index=(), setup_by_user=False) -> None [AST:python/cocoindex/flow.py:L402] — export collected data to a target. Must be called at top level. target_name identifier must remain stable across runs — renaming causes drop+recreate. vector_index is a legacy alias; use vector_indexes in new code.

Flow [AST:python/cocoindex/flow.py:L705]:

name / full_name (properties) [AST:python/cocoindex/flow.py:L758] [AST:python/cocoindex/flow.py:L765] — full_name is {app_namespace}.{name}.
update(*, reexport_targets=False, full_reprocess=False, print_stats=False) -> _engine.IndexUpdateInfo [AST:python/cocoindex/flow.py:L771] — one-time update. Async: update_async(...) [AST:python/cocoindex/flow.py:L791].
setup(report_to_stdout=False) -> None / setup_async(...) [AST:python/cocoindex/flow.py:L855] [AST:python/cocoindex/flow.py:L861]
drop(report_to_stdout=False) -> None / drop_async(...) [AST:python/cocoindex/flow.py:L868] [AST:python/cocoindex/flow.py:L879] — drops backends; the Flow instance remains valid.
close() -> None [AST:python/cocoindex/flow.py:L886] — remove from current process; instance invalid afterward. Does NOT touch backends.
evaluate_and_dump(options: EvaluateAndDumpOptions) -> _engine.IndexUpdateInfo [AST:python/cocoindex/flow.py:L815] — run transformations to disk without updating target.
add_query_handler(name, handler, /, *, result_fields=None) -> None [AST:python/cocoindex/flow.py:L898] — attach a named query handler (called via server).
query_handler(name=None, result_fields=None) [AST:python/cocoindex/flow.py:L935] — decorator form.

FlowLiveUpdater [AST:python/cocoindex/flow.py:L603]:

__init__(fl: Flow, options: FlowLiveUpdaterOptions | None = None) [AST:python/cocoindex/flow.py:L612]
start() / start_async() — begin live update. [AST:python/cocoindex/flow.py:L632] [AST:python/cocoindex/flow.py:L638]
abort() — stop the updater. [AST:python/cocoindex/flow.py:L677]
wait() / wait_async() — block until updater finishes. [AST:python/cocoindex/flow.py:L646] [AST:python/cocoindex/flow.py:L652]
next_status_updates() / next_status_updates_async() → FlowUpdaterStatusUpdates [AST:python/cocoindex/flow.py:L658] [AST:python/cocoindex/flow.py:L667]
update_stats() -> _engine.IndexUpdateInfo [AST:python/cocoindex/flow.py:L683]
Supports sync (with) and async (async with) context-manager usage (calls start/abort+wait on enter/exit). [AST:python/cocoindex/flow.py:L616] [AST:python/cocoindex/flow.py:L624]

Module-level helpers:

cocoindex.flow_def(name=None) [AST:python/cocoindex/flow.py:L1011] — decorator wrapping a flow function.
cocoindex.open_flow(name, fl_def) -> Flow [AST:python/cocoindex/flow.py:L986]
cocoindex.add_flow_def(name, fl_def) -> Flow — DEPRECATED alias for open_flow. [AST:python/cocoindex/flow.py:L997]
cocoindex.remove_flow(fl) -> None — DEPRECATED alias for fl.close(). [AST:python/cocoindex/flow.py:L1004]
cocoindex.update_all_flows_async(options: FlowLiveUpdaterOptions) -> dict[str, _engine.IndexUpdateInfo] [AST:python/cocoindex/flow.py:L1068]
cocoindex.setup_all_flows(report_to_stdout=False) -> None [AST:python/cocoindex/flow.py:L1300]
cocoindex.drop_all_flows(report_to_stdout=False) -> None [AST:python/cocoindex/flow.py:L1309]
cocoindex.transform_flow() -> Callable[[Callable[..., DataSlice[T]]], TransformFlow[T]] [AST:python/cocoindex/flow.py:L1251]

Dataclasses exported to users:

FlowLiveUpdaterOptions(live_mode=True, reexport_targets=False, full_reprocess=False, print_stats=False) [AST:python/cocoindex/flow.py:L574]
FlowUpdaterStatusUpdates(active_sources: list[str], updated_sources: list[str]) [AST:python/cocoindex/flow.py:L591]
EvaluateAndDumpOptions(output_dir: str, use_cache: bool = True) [AST:python/cocoindex/flow.py:L696]
GeneratedField(Enum) — member UUID = "Uuid". [AST:python/cocoindex/flow.py:L359]

Library lifecycle — `cocoindex.lib`

cocoindex.settings(fn=None) [AST:python/cocoindex/lib.py:L35] — decorator registering a function () -> Settings as the settings provider. Overrides env vars. Warns on re-registration.
cocoindex.init(settings: Settings | None = None) -> None [AST:python/cocoindex/lib.py:L58] — explicit init. Optional — cocoindex auto-inits on first flow method call. Prefer explicit init at startup so errors surface early.
cocoindex.start_server(settings: ServerSettings) -> None [AST:python/cocoindex/lib.py:L67] — starts HTTP server (used by cocoindex server). Ensures all flows are built first.
cocoindex.stop() -> None [AST:python/cocoindex/lib.py:L73]

Settings & auth — `cocoindex.setting`, `cocoindex.auth_registry`

See references/settings-auth.md for full field tables.

Settings(ignore_target_drop_failures=False, database=None, db_schema_name=None, app_namespace="", global_execution_options=None) [AST:python/cocoindex/setting.py:L75] — classmethod Settings.from_env().
DatabaseConnectionSpec(url, user=None, password=None, max_connections=25, min_connections=5) [AST:python/cocoindex/setting.py:L29]
GlobalExecutionOptions(source_max_inflight_rows=1024, source_max_inflight_bytes=None) [AST:python/cocoindex/setting.py:L43]
ServerSettings(address="127.0.0.1:49344", cors_origins=None) [AST:python/cocoindex/setting.py:L149]
get_app_namespace(*, trailing_delimiter=None) -> str [AST:python/cocoindex/setting.py:L12]
AuthEntryReference[T] — dataclass with key: str, subclass of TransientAuthEntryReference[T]. [AST:python/cocoindex/auth_registry.py:L21]
add_auth_entry(key: str, value: T) -> AuthEntryReference[T] [AST:python/cocoindex/auth_registry.py:L31]
add_transient_auth_entry(value: T) -> TransientAuthEntryReference[T] [AST:python/cocoindex/auth_registry.py:L25]
ref_auth_entry(key: str) -> AuthEntryReference[T] [AST:python/cocoindex/auth_registry.py:L37]

Indexes — `cocoindex.index`

VectorSimilarityMetric(Enum) — COSINE_SIMILARITY, L2_DISTANCE, INNER_PRODUCT. [AST:python/cocoindex/index.py:L6]
HnswVectorIndexMethod(kind="Hnsw", m=None, ef_construction=None) [AST:python/cocoindex/index.py:L13]
IvfFlatVectorIndexMethod(kind="IvfFlat", lists=None) [AST:python/cocoindex/index.py:L22]
VectorIndexDef(field_name: str, metric: VectorSimilarityMetric, method: VectorIndexMethod | None = None) [AST:python/cocoindex/index.py:L33]
FtsIndexDef(field_name: str, parameters: dict[str, Any] | None = None) — LanceDB only. [AST:python/cocoindex/index.py:L44]
IndexOptions(primary_key_fields: Sequence[str], vector_indexes=(), fts_indexes=()) [AST:python/cocoindex/index.py:L57]

LLM — `cocoindex.llm`

LlmApiType(Enum) — 12 members (OPENAI, OLLAMA, GEMINI, VERTEX_AI, ANTHROPIC, LITE_LLM, OPEN_ROUTER, VOYAGE, VLLM, BEDROCK, AZURE_OPENAI, NOVITA). [AST:python/cocoindex/llm.py:L7]
LlmSpec(api_type, model, address=None, api_key=None, api_config=None) [AST:python/cocoindex/llm.py:L55]
VertexAiConfig(project, region=None) [AST:python/cocoindex/llm.py:L25]
OpenAiConfig(org_id=None, project_id=None) [AST:python/cocoindex/llm.py:L35]
AzureOpenAiConfig(deployment_id, api_version=None) [AST:python/cocoindex/llm.py:L45]

See references/sources-targets-functions.md for source / target / function specs and [EXT:https://cocoindex.io/docs/ai/llm] for integration details per provider.

Sources — `cocoindex.sources`

LocalFile(path: str, binary=False, included_patterns=None, excluded_patterns=None, max_file_size=None) — KTable with filename (key), content. [AST:python/cocoindex/sources/_engine_builtin_specs.py:L10]
GoogleDrive(service_account_credential_path, root_folder_ids, binary=False, included_patterns=None, excluded_patterns=None, max_file_size=None, recent_changes_poll_interval=None) [AST:python/cocoindex/sources/_engine_builtin_specs.py:L30]
AmazonS3(bucket_name, prefix=None, binary=False, included_patterns=None, excluded_patterns=None, max_file_size=None, sqs_queue_url=None, redis=None, force_path_style=False) [AST:python/cocoindex/sources/_engine_builtin_specs.py:L61]
AzureBlob(account_name, container_name, prefix=None, binary=False, included_patterns=None, excluded_patterns=None, max_file_size=None, sas_token=None, account_access_key=None) [AST:python/cocoindex/sources/_engine_builtin_specs.py:L77]
Postgres(table_name, database=None, included_columns=None, ordinal_column=None, notification=None, filter=None) [AST:python/cocoindex/sources/_engine_builtin_specs.py:L110]
RedisNotification(redis_url, redis_channel) / PostgresNotification(channel_name=None) — change-capture config. [AST:python/cocoindex/sources/_engine_builtin_specs.py:L52] [AST:python/cocoindex/sources/_engine_builtin_specs.py:L102]

Targets — `cocoindex.targets`

Built-in specs:

Postgres(database=None, table_name=None, schema=None, column_options=None) — pgvector for fixed-dim float vectors; jsonb otherwise. [AST:python/cocoindex/targets/_engine_builtin_specs.py:L20]
PostgresSqlCommand(name, setup_sql, teardown_sql=None) — attachment for arbitrary setup/teardown SQL. [AST:python/cocoindex/targets/_engine_builtin_specs.py:L29]
Qdrant(collection_name, connection=None) / QdrantConnection(grpc_url, api_key=None) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L46]
Pinecone(index_name, connection, namespace="", cloud="aws", region="us-east-1", batch_size=100) / PineconeConnection(api_key, environment=None) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L62]
Neo4j(connection, mapping) / Neo4jConnection(uri, user, password, db=None) / Neo4jDeclaration(connection, nodes_label, primary_key_fields, vector_indexes=()) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L136]
FalkorDB(connection, mapping) / FalkorDBConnection(uri, graph=None) / FalkorDBDeclaration(...) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L163]
Ladybug(connection, mapping) / LadybugConnection(api_server_url) / LadybugDeclaration(connection, nodes_label, primary_key_fields) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L188]
Kuzu, KuzuConnection, KuzuDeclaration — retired, aliases for Ladybug*. [AST:python/cocoindex/targets/_engine_builtin_specs.py:L205]

Graph-mapping types:

Nodes(label: str) / Relationships(rel_type: str, source: NodeFromFields, target: NodeFromFields) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L101] [AST:python/cocoindex/targets/_engine_builtin_specs.py:L110]
NodeFromFields(label, fields: list[TargetFieldMapping]) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L84]
ReferencedNode(label, primary_key_fields, vector_indexes=()) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L92]
TargetFieldMapping(source: str, target: str | None = None) [AST:python/cocoindex/targets/_engine_builtin_specs.py:L74]
Backward-compat aliases: NodeMapping = Nodes, RelationshipMapping = Relationships, NodeReferenceMapping = NodeFromFields. [AST:python/cocoindex/targets/_engine_builtin_specs.py:L121]

Extended target modules:

cocoindex.targets.chromadb.ChromaDB(collection_name, client_type=ClientType.PERSISTENT, path=None, host=None, port=None, ssl=False, api_key=None, tenant=..., database=..., hnsw_config=None, document_field=None) [AST:python/cocoindex/targets/chromadb.py:L34] · ClientType, HnswConfig helpers.
cocoindex.targets.lancedb.LanceDB(db_uri, table_name, db_options=None, num_transactions_before_optimize=50) [AST:python/cocoindex/targets/lancedb.py:L84] · DatabaseOptions(storage_options=None).
cocoindex.targets.doris.DorisTarget(fe_host, database, table, fe_http_port=8080, query_port=9030, username="root", password="", ...) [AST:python/cocoindex/targets/doris.py:L105] — with DorisError hierarchy (DorisConnectionError, DorisAuthError, DorisStreamLoadError, DorisSchemaError) and RetryConfig.
cocoindex.targets.turbopuffer.Turbopuffer(namespace_name, api_key, region="gcp-us-central1") [AST:python/cocoindex/targets/turbopuffer.py:L36]
cocoindex.targets.pinecone — provides the connector for the built-in Pinecone spec (no new public spec class).

Functions — `cocoindex.functions`

ParseJson() — parses text to JSON. [AST:python/cocoindex/functions/_engine_builtin_specs.py:L10]
DetectProgrammingLanguage() — detects language from filename. [AST:python/cocoindex/functions/_engine_builtin_specs.py:L23]
SplitRecursively(custom_languages=[]) — hierarchical text splitter. Spec fields only; runtime args (chunk_size, min_chunk_size, chunk_overlap, language) are passed through transform(...). Returns KTable with location, text, start, end. [AST:python/cocoindex/functions/_engine_builtin_specs.py:L27]
SplitBySeparators(separators_regex=[], keep_separator="NONE", include_empty=False, trim=True) [AST:python/cocoindex/functions/_engine_builtin_specs.py:L33]
EmbedText(api_type, model, address=None, output_dimension=None, expected_output_dimension=None, task_type=None, api_config=None, api_key=None) [AST:python/cocoindex/functions/_engine_builtin_specs.py:L51]
ExtractByLlm(llm_spec: LlmSpec, output_type: type, instruction: str | None = None) [AST:python/cocoindex/functions/_engine_builtin_specs.py:L64]
CustomLanguageSpec(language_name, separators_regex, aliases=[]) [AST:python/cocoindex/functions/_engine_builtin_specs.py:L15]
SentenceTransformerEmbed(model, args=None) — requires pip install 'cocoindex[embeddings]'. [AST:python/cocoindex/functions/sbert.py:L12]
SentenceTransformerEmbedExecutor — gpu+cache+batched executor (internal but exported for custom builds). [AST:python/cocoindex/functions/sbert.py:L38]
ColPaliEmbedImage(model) / ColPaliEmbedImageExecutor — requires pip install 'cocoindex[colpali]'. [AST:python/cocoindex/functions/colpali.py:L99]
ColPaliEmbedQuery(model) / ColPaliEmbedQueryExecutor [AST:python/cocoindex/functions/colpali.py:L178]

Op framework — `cocoindex.op`

For authoring custom sources/functions/targets. See references/op-framework.md.

FunctionSpec, SourceSpec, TargetSpec, TargetAttachmentSpec, DeclarationSpec — dataclass-like spec base classes via SpecMeta. [AST:python/cocoindex/op.py:L75] [AST:python/cocoindex/op.py:L71] [AST:python/cocoindex/op.py:L79] [AST:python/cocoindex/op.py:L83] [AST:python/cocoindex/op.py:L87]
OpCategory Enum — FUNCTION, SOURCE, TARGET, DECLARATION, TARGET_ATTACHMENT. [AST:python/cocoindex/op.py:L40]
ArgRelationship Enum — EMBEDDING_ORIGIN_TEXT, CHUNKS_BASE_TEXT, RECTS_BASE_IMAGE. [AST:python/cocoindex/op.py:L126]
OpArgs(gpu=False, cache=False, batching=False, max_batch_size=None, behavior_version=None, timeout=None, arg_relationship=None) [AST:python/cocoindex/op.py:L135]
executor_class(**args) — decorator turning a class into a function executor. Expects a spec: YourSpecClass class attribute with type annotation. [AST:python/cocoindex/op.py:L448]
function(**args) — decorator wrapping a plain function as a CocoIndex function op. Converts snake_case to CamelCase for op_kind. [AST:python/cocoindex/op.py:L489]
source_connector(*, spec_cls, key_type=Any, value_type=Any) — decorator registering a source connector class (create, list, get_value, optional provides_ordinal). [AST:python/cocoindex/op.py:L746]
target_connector(*, spec_cls, persistent_key_type=Any, setup_state_cls=None) — decorator registering a target connector class. [AST:python/cocoindex/op.py:L1073]
SourceReadOptions(include_ordinal=False, include_content_version_fp=False, include_value=False) [AST:python/cocoindex/op.py:L521]
PartialSourceRowData[V] — dataclass with fields value, ordinal, content_version_fp (all default None). [AST:python/cocoindex/op.py:L556]
PartialSourceRow[K, V] — dataclass with fields key: K, data: PartialSourceRowData[V]. [AST:python/cocoindex/op.py:L571]
TargetStateCompatibility Enum — COMPATIBLE, PARTIALLY_COMPATIBLE, NOT_COMPATIBLE. [AST:python/cocoindex/op.py:L811]
EmptyFunctionSpec — placeholder for @op.function-decorated bare functions. [AST:python/cocoindex/op.py:L478]

Query handlers — `cocoindex.query_handler`

QueryHandlerResultFields(embedding=[], score=None) — metadata for CocoInsight query result introspection. [AST:python/cocoindex/query_handler.py:L11]
QueryInfo(embedding=None, similarity_metric=None) [AST:python/cocoindex/query_handler.py:L31]
QueryOutput[R] — dataclass with fields results: list[R], query_info: QueryInfo (default QueryInfo()). [AST:python/cocoindex/query_handler.py:L44]

Schema typing — `cocoindex.typing`

Annotated aliases used on dataclass fields and function return types to attach CocoIndex schema information:

Int64 = Annotated[int, TypeKind("Int64")]
Float32 = Annotated[float, TypeKind("Float32")]
Float64 = Annotated[float, TypeKind("Float64")]
Range = Annotated[tuple[int, int], TypeKind("Range")]
Json = Annotated[Any, TypeKind("Json")]
LocalDateTime = Annotated[datetime, TypeKind("LocalDateTime")]
OffsetDateTime = Annotated[datetime, TypeKind("OffsetDateTime")]
Vector[T, Dim] — alias for Annotated[NDArray[T] | list[T], VectorInfo(dim=Dim)]. Usage: Vector[np.float32], Vector[np.float32, Literal[384]]. [AST:python/cocoindex/typing.py:L49]

Utils — `cocoindex.utils`

get_target_default_name(flow: Flow, target_name: str, delimiter: str = "__") -> str [AST:python/cocoindex/utils.py:L5] — {app_namespace}{delimiter}{flow.name}{delimiter}{target_name}.
get_target_storage_default_name — DEPRECATED alias. [AST:python/cocoindex/utils.py:L18]

Full Type Definitions

See references/flow-api.md, references/settings-auth.md, and references/sources-targets-functions.md for the complete type reference.

Full Integration Patterns

LLM-aware extraction pipeline (source → split → embed → collect → export): This is the canonical cocoindex pattern, repeated across most examples: flow_builder.add_source() → DataSlice.row() (per-document scope) → doc["content"].transform(SplitRecursively(), ...) (per-chunk KTable) → chunk["text"].transform(SentenceTransformerEmbed(...)) → collector.collect(...) → top-level collector.export(target_spec, primary_key_fields=..., vector_indexes=[...]). [EXT:https://cocoindex.io/docs/getting_started/quickstart]

LLM structured extraction pattern: collector.collect(..., extracted=doc["text"].transform(ExtractByLlm(llm_spec=LlmSpec(api_type=LlmApiType.OPENAI, model="gpt-4o"), output_type=MyDataclass, instruction="..."))). Define clear dataclasses — output_type schema is fed to the LLM. [EXT:https://cocoindex.io/docs/ops/functions#extractbyllm]

Graph target pattern (Neo4j / FalkorDB / Ladybug): Two collectors — one exports Nodes(label="Document"), another exports Relationships(rel_type="MENTIONS", source=NodeFromFields(label="Document", fields=[...]), target=NodeFromFields(label="Entity", fields=[...])). Use flow_builder.declare(Neo4jDeclaration(connection=..., nodes_label="Entity", primary_key_fields=[...], vector_indexes=[...])) to configure referenced nodes that aren't produced by any specific collector. Connection via cocoindex.add_auth_entry("my_graph_conn", Neo4jConnection(...)). [AST:python/cocoindex/targets/_engine_builtin_specs.py:L136]

Live update loop pattern:

with cocoindex.FlowLiveUpdater(demo_flow, FlowLiveUpdaterOptions(live_mode=True, print_stats=True)) as updater:
    while True:
        updates = updater.next_status_updates()
        for source in updates.updated_sources:
            run_downstream(source)
        if not updates.active_sources:
            break

Sources with change capture (Postgres logical-replication, S3 event notifications, Google Drive polling, or refresh_interval) continuously push updates; one-time sources complete after the initial update. [EXT:https://cocoindex.io/docs/core/flow_methods]

Custom function (two forms):

Standalone — @cocoindex.op.function(cache=True, behavior_version=1) on a typed function.
Spec + executor — class FooSpec(cocoindex.op.FunctionSpec) with parameter fields, plus @cocoindex.op.executor_class(cache=True, behavior_version=1, gpu=True) on a class with spec: FooSpec, optional prepare(self), and required __call__(self, ...) -> .... [EXT:https://cocoindex.io/docs/custom_ops/custom_functions]

Attachment pattern (Postgres SQL hooks): collector.export("doc_embeddings", Postgres(table_name="doc_embeddings"), primary_key_fields=["id"], attachments=[PostgresSqlCommand(name="fts", setup_sql="CREATE INDEX IF NOT EXISTS ... USING GIN (to_tsvector('english', text));", teardown_sql="DROP INDEX IF EXISTS ...;")]) — idempotent setup/teardown SQL for capabilities not native to the target spec. [EXT:https://cocoindex.io/docs/targets/postgres]

What ships with it: 6 files

65.1 KB alongside SKILL.md

references/

context-snippet.md754 B
metadata.json2.8 KB

One thing to look at

SKILL.md

oms-cocoindex

Overview

Quick Start

Common Workflows

Key API Summary

Migration & Deprecation Warnings

Key Types

Architecture at a Glance

CLI

Full API Reference

Flow authoring — cocoindex.flow

Library lifecycle — cocoindex.lib

Settings & auth — cocoindex.setting, cocoindex.auth_registry

Indexes — cocoindex.index

LLM — cocoindex.llm

Sources — cocoindex.sources

Targets — cocoindex.targets

Functions — cocoindex.functions

Op framework — cocoindex.op

Query handlers — cocoindex.query_handler

Schema typing — cocoindex.typing

Utils — cocoindex.utils