agentsclimarketplace

Spice data connector

Skill spiceai/skills/skills/spice-data-connector

Configure individual data source connectors in Spice — PostgreSQL, MySQL, S3, Databricks, Snowflake, DuckDB, GitHub, Kafka, and 25+ more. Use this skill whenever the user wants to add a dataset, connect to a specific database or data source, load data from S3 or files, configure connector-specific parameters, understand file formats (Parquet, CSV, PDF, DOCX), or set up hive partitioning. This skill is the reference for the `from:` and `params:` fields in dataset configuration. For cross-source federation, views, and catalogs, see spice-connect-data.From its SKILL.md

Install
npx -y skills add spiceai/skills --skill spice-data-connector

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

7.9 KB, as published. Nobody here has run it

Spice Data Connectors

Data Connectors enable federated SQL queries across databases, data warehouses, data lakes, and files. Spice connects directly to your existing data sources and provides a unified SQL interface — no ETL pipelines required. The query planner (built on Apache DataFusion) optimizes and routes queries, including filter pushdown and column projection.

Cross-Source Federation

Query across multiple heterogeneous sources in one SQL statement:

datasets:
  - from: postgres:customers
    name: customers
    params:
      pg_host: db.example.com
      pg_user: ${secrets:PG_USER}
  - from: s3://bucket/orders/
    name: orders
    params:
      file_format: parquet
  - from: snowflake:analytics.sales
    name: sales
-- Query across all three sources in one statement
SELECT c.name, o.order_total, s.region
FROM customers c
  JOIN orders o ON c.id = o.customer_id
  JOIN sales s ON o.id = s.order_id
WHERE s.region = 'EMEA';

Without acceleration, each query fetches data directly from the underlying sources with optimized filter pushdown.

Basic Dataset Configuration

datasets:
  - from: <connector>:<identifier>
    name: <dataset_name>
    params:
      # connector-specific parameters
    acceleration:
      enabled: true # optional: enable local materialization

Supported Connectors

Databases

ConnectorFrom FormatStatus
PostgreSQLpostgres:schema.tableStable (native WAL CDC; also Amazon Redshift)
MySQLmysql:schema.tableStable
DuckDBduckdb:database.tableStable
DynamoDBdynamodb:tableStable (with Streams)
Azure Cosmos DBcosmosdb:database.containerRelease Candidate
MS SQL Servermssql:db.tableBeta
MongoDBmongodb:collectionAlpha (Change Streams)
ClickHouseclickhouse:db.tableAlpha
Oracleoracle:schema.tableAlpha
ScyllaDBscylladb:tableAlpha

Data Warehouses

ConnectorFrom FormatStatus
Databricks (Delta Lake)databricks:catalog.schema.tableStable
Snowflakesnowflake:db.schema.tableRelease Candidate
Sparkspark:db.tableBeta

Data Lakes & Object Storage

ConnectorFrom FormatStatus
S3s3://bucket/path/Stable
Delta Lakedelta_lake:/path/to/delta/Stable
File (local)file:./path/to/dataStable
Icebergiceberg:tableRelease Candidate (read+write)
DuckLakeducklake:tableBeta
Azure BlobFSabfs://container/path/Alpha
Google Cloud Storagegs://bucket/path/Alpha
AWS Glueglue:db.tableAlpha

Other Sources

ConnectorFrom FormatStatus
Spice.aispice.ai:path/to/datasetStable
Dremiodremio:source.tableStable
GitHubgithub:github.com/owner/repo/issuesStable
GraphQLgraphql:endpointRelease Candidate
ADBCadbc:tableRelease Candidate
FlightSQLflightsql:queryBeta
ODBCodbc:connectionBeta (Spice.ai Enterprise)
SharePointsharepoint:site/pathBeta
FTP/SFTPsftp://host/path/Alpha
HTTP/HTTPShttps://url/path/data.csvAlpha
Kafkakafka:topicAlpha
Debezium CDCdebezium:topicAlpha
Elasticsearchelasticsearch:indexAlpha (Spice.ai Enterprise)
IMAPimap:mailboxAlpha
localpodlocalpod:datasetAlpha
SMBsmb://host/share/path/Alpha
NFSnfs://host/path/Alpha (Spice.ai Enterprise)

Common Examples

PostgreSQL

datasets:
  - from: postgres:public.users
    name: users
    params:
      pg_host: localhost
      pg_port: 5432
      pg_user: ${ env:PG_USER }
      pg_pass: ${ env:PG_PASS }
    acceleration:
      enabled: true

S3 with Parquet

datasets:
  - from: s3://my-bucket/data/sales/
    name: sales
    params:
      file_format: parquet
      s3_region: us-east-1
    acceleration:
      enabled: true
      engine: duckdb

GitHub Issues

datasets:
  - from: github:github.com/spiceai/spiceai/issues
    name: spiceai.issues
    params:
      github_token: ${ secrets:GITHUB_TOKEN }
    acceleration:
      enabled: true
      refresh_mode: append
      refresh_check_interval: 24h
      refresh_data_window: 14d

Local File

datasets:
  - from: file:./data/sales.parquet
    name: sales

File Formats

Connectors reading from object stores (S3, ABFS, GCS) or network storage (FTP, SFTP) support:

Formatfile_formatStatusType
Apache ParquetparquetStableStructured
CSVcsvStableStructured
MarkdownmdStableDocument
TexttxtStableDocument
PDFpdfAlphaDocument
Microsoft WorddocxAlphaDocument

Document Formats

Document files (md, txt, pdf, docx) produce a table with location and content columns:

datasets:
  - from: file:docs/decisions/
    name: my_documents
    params:
      file_format: md
SELECT location, content FROM my_documents LIMIT 5;

Hive Partitioning

datasets:
  - from: s3://bucket/data/
    name: partitioned_data
    params:
      file_format: parquet
      hive_partitioning_enabled: true
SELECT * FROM partitioned_data WHERE year = '2024' AND month = '01';

Dataset Naming

  • name: foo creates spice.public.foo
  • name: myschema.foo creates spice.myschema.foo
  • Use . to organize datasets into schemas

Documentation

Keep looking

Skills are one crate of 609,856. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.