Matlab connect databricks spark

MATLAB Interface for Databricks — Spark (Databricks Connect)

Read data from Databricks into MATLAB via Databricks Connect. This skill covers first-time setup, authentication, and using Spark DataFrames from desktop MATLAB.

When to Use

• First-time setup of the MATLAB Interface for Databricks package
• Configuring .databrickscfg and authentication (OauthU2M, OauthM2M, PAT)
• Reading large tables via Spark with server-side filtering before pulling data locally
• Creating a Spark session from desktop MATLAB (getDatabricksSession)
• Converting Spark DataFrames to MATLAB tables or writing MATLAB tables back to Spark

When NOT to Use

• SQL queries on small-to-medium datasets via JDBC/ODBC — use /matlab-connect-databricks-jdbc
• Deploying compiled MATLAB code to Databricks clusters (Job workflow)
• Databricks Files API operations
• REST API management (Clusters, Jobs, Unity Catalog admin)

Workflow

0. Obtain the package — Ask the user if they already have the MATLAB Interface for Databricks downloaded locally. If not, direct them to download it from https://www.mathworks.com/solutions/partners/databricks.html
1. Setup — Run setup from the package directory to configure settings, .databrickscfg, and install the Databricks Connect library via pip
2. Verify — Create a Spark session and run spark.range(1) to confirm connectivity
3. Read — Use spark.read().table() to get a DataFrame handle (data stays on cluster)
4. Filter — Apply .filter(), .withColumn(), .select() server-side
5. Collect — Convert to MATLAB table with T = table(DF) only after reducing data

Key Functions

Function	Purpose
setup	Interactive first-time configuration: settings, .databrickscfg, and Databricks Connect library download
startup	Adds package paths to MATLAB (run once per session, required after setup)
matlab.databricks.setup.configureDBC()	Re-run Databricks Connect library download independently (supports alternativeRepo for internal Artifactory)
getDatabricksSession()	Creates a databricks.PySparkSession connected to Databricks (classic compute)
getDatabricksSession(serverless=true)	Creates a serverless session (no cluster needed, 10-min timeout)
updateClusterId(id)	Updates the cluster ID in .databrickscfg (pass "serverless" to switch to serverless mode)
spark.range(n)	Creates a DataFrame with n rows (useful for testing)
spark.read().table("catalog.schema.table")	Reads a Unity Catalog table as a DataFrame
DF.filter(expr)	Server-side row filtering
DF.withColumn(name, col)	Adds/transforms a column server-side (col must be a matlab.pyspark.sql.column.Column)
DF.limit(n)	Server-side row limiting (returns first n rows as a new DataFrame)
matlab.pyspark.sql.functions.col(name)	Creates a Column reference by name
matlab.pyspark.sql.functions.lit(value)	Creates a Column containing a literal constant
DF.select(cols)	Server-side column selection
DF.show(n)	Displays n rows (stays on cluster)
DF.printSchema	Prints DataFrame schema
table(DF)	Converts DataFrame to a MATLAB table (pulls data locally)
spark.sql(query)	Executes a SQL query and returns a DataFrame
spark.table("catalog.schema.table")	Shortcut for spark.read().table()
matlab.sparkutils.table2dataset(T, spark)	Converts a MATLAB table back to a Spark DataFrame
DF.write.mode(m).format(f).save(path)	Writes DataFrame to a file path
DF.write.mode(m).format(f).saveAsTable(name)	Writes DataFrame to a Unity Catalog table

Patterns

First-Time Setup

Run setup from the package's Software/MATLAB directory. It is interactive — follow the prompts to configure the host URL, authentication method, cluster ID, and Databricks Connect library installation.

The Databricks Connect library is not shipped with the package — setup downloads it via pip (from PyPI or an internal Artifactory) into a Python virtual environment under Software/MATLAB/Connect/<version>/venv/. This requires Python 3.10–3.12 and pip to be available.

If the Databricks Connect library download fails repeatedly, do not retry with modified arguments — ask the user to check with their IT team whether the required version of databricks-connect is available in their organization's Python package repository.

cd('/path/to/databricks-package/Software/MATLAB')
setup

After setup completes, MATLAB's pyenv must point to the venv Python — not the system Python. If getDatabricksSession fails with "databricks.connect package is not installed", switch pyenv:

terminate(pyenv);
pyenv(Version="/path/to/databricks-package/Software/MATLAB/Connect/17.3/venv/bin/python");

Warning: If Python has already been loaded in-process (ExecutionMode is "InProcess"), terminate(pyenv) will fail. A full MATLAB restart is required to switch pyenv in this case. Do not attempt to switch pyenv mid-session after Python has been used — it will not work and may disconnect the MCP server. Instead, ask the user to restart MATLAB, then set pyenv to the venv path before calling any Python-dependent functions.

Then run startup to add the package paths to MATLAB. This must also be run at the beginning of each new MATLAB session:

startup

Then verify connectivity:

spark = getDatabricksSession();
T = table(spark.range(1));
disp(T)

If this returns a 1-row table with column id = 0, authentication and compute are working.

Authentication Configuration

Authentication uses the .databrickscfg file — an INI-format configuration file located in the user's home directory (~/.databrickscfg). It stores the Databricks workspace URL, authentication credentials, and cluster ID. The file supports multiple named profiles so you can switch between workspaces or auth methods.

OauthU2M (recommended for interactive use) — requires only host. Note: OauthU2M opens a browser for login and requires pasting a redirect URL back into MATLAB. It does not work in non-interactive contexts (scripts, batch mode, MCP tools). Use OauthM2M or PAT for automation:

[DEFAULT]
host = https://<workspace-url>
cluster_id = <cluster-id>

OauthM2M (for service principals / automation) — requires host, client_id, client_secret:

[M2M]
host = https://<workspace-url>
client_id = <service-principal-client-id>
client_secret = <service-principal-secret>
cluster_id = <cluster-id>

PAT (simple but may be disabled by admins) — requires host and token:

[PAT]
host = https://<workspace-url>
token = <personal-access-token>
cluster_id = <cluster-id>

Specifying the auth method: When authMethod is not passed, getDatabricksSession uses a chain that tries methods in order until one succeeds — this may not pick what the user expects (e.g., it may trigger an OauthU2M browser prompt even if a PAT token is configured). Always ask the user which auth method they want and pass it explicitly:

spark = getDatabricksSession(authMethod="PAT");
spark = getDatabricksSession(authMethod="OauthU2M");
spark = getDatabricksSession(authMethod="OauthM2M");

To use a named profile other than [DEFAULT]:

spark = getDatabricksSession(profileName="M2M", authMethod="OauthM2M");

Environment variables override config file values:

Variable	Overrides
DATABRICKS_HOST	host
DATABRICKS_TOKEN	token
DATABRICKS_CLUSTER_ID	cluster_id
DATABRICKS_CLIENT_ID	client_id
DATABRICKS_CLIENT_SECRET	client_secret

Create a Spark Session (Classic Compute)

spark = getDatabricksSession();

Uses the default profile's cluster_id. The cluster starts automatically if stopped (takes 3-5 minutes for a cold start).

To target a specific cluster:

spark = getDatabricksSession(cluster="0812-091301-zntwkr4b");

Create a Spark Session (Serverless Compute)

Serverless sessions don't require a cluster_id — they start instantly with a 10-minute inactivity timeout:

spark = getDatabricksSession(serverless=true);

To make serverless the default for a profile, remove the cluster_id and set serverless_compute_id:

updateClusterId("serverless");

Or configure .databrickscfg directly:

[DEFAULT]
host = https://<workspace-url>
serverless_compute_id = auto

Serverless requires Python 3.11–3.12 and Databricks Connect client >= 15.4.

Read and Filter a Table

Data stays on the cluster until explicitly collected. Filter server-side first, then collect.

spark = getDatabricksSession();

DF = spark.read().table("catalog.schema.sensor_readings");

filtered = DF.filter("temperature > 100 AND event_date > '2024-01-01'");

filtered.printSchema;
filtered.show(5);

T = table(filtered);

Filter and Limit Rows

Use .filter() for conditional filtering and .limit() for taking the first N rows:

spark = getDatabricksSession();
DF = spark.read().table("catalog.schema.large_table");

filtered = DF.filter("status = 'active' AND created_date > '2024-01-01'");
limited = filtered.limit(1000);
T = table(limited);

Add Computed Columns with withColumn

withColumn requires a matlab.pyspark.sql.column.Column object — not a scalar or string. Use col() to reference existing columns and lit() for constants:

import matlab.pyspark.sql.functions.col
import matlab.pyspark.sql.functions.lit

DF = spark.read().table("catalog.schema.measurements");

DF2 = DF.withColumn("temp_fahrenheit", col("temp_celsius") * lit(9/5) + lit(32));
DF3 = DF2.withColumn("source", lit("sensor_array"));

Do not pass raw MATLAB scalars or strings to withColumn — it will error. Always wrap values in lit().

Read Files from Unity Catalog Volumes

Use spark.read.format().load() for CSV, Parquet, or JSON files stored on Volumes:

spark = getDatabricksSession();

% CSV with header
DF = spark.read.format("csv").option("header", "true").load("/Volumes/catalog/schema/volume/data.csv");

% Parquet
DF = spark.read.format("parquet").load("/Volumes/catalog/schema/volume/data.parquet");

% JSON
DF = spark.read.format("json").load("/Volumes/catalog/schema/volume/data.json");

After loading, use the same filter/select/collect workflow as with tables.

Convert MATLAB Table Back to Spark DataFrame

DF_new = matlab.sparkutils.table2dataset(T, spark);

Write to a Volumes path as Parquet:

DF_new.write.mode("overwrite").format("parquet").save("/Volumes/catalog/schema/volume/output_data");

Write to a Unity Catalog table:

DF_new.write.mode("overwrite").format("delta").saveAsTable("catalog.schema.output_table");

Always specify .mode() — without it, writes fail if the target already exists. Options: "overwrite", "append", "ignore", "error" (default).

Handle Stale Sessions

If a session becomes unresponsive after inactivity:

clear spark
spark = getDatabricksSession(forceNewSession=true);

Python Version Compatibility

The local Python version must match the Databricks runtime's Python version:

Runtime	Python Version	Notes
18.x	3.12	Supports serverless and classic
17.3	3.12	Supports serverless and classic
16.4 LTS	3.12	Supports serverless (>= 16.4.1) and classic
15.4 LTS	3.11	Supports serverless (>= 15.4.10) and classic
14.3 LTS	3.10	Classic only
13.3 LTS	3.10	Classic only

Check with:

pe = pyenv;
disp(pe.Version)

Conventions

• Always filter DataFrames server-side before calling table(DF) — pulling millions of unfiltered rows wastes bandwidth and memory
• Use three-level names for Unity Catalog tables: "catalog.schema.table"
• Use getDatabricksSession() — never construct Spark sessions manually or mimic PySpark builder patterns
• Use forceNewSession=true when a session becomes stale, not clear all
• Never hardcode tokens or secrets in MATLAB code — use .databrickscfg or environment variables
• Only Databricks Connect v2 is supported (v1 is not supported)

Common Mistakes

Mistake	Why It's Wrong	Correct Approach
Manually building JDBC URLs with Simba driver params	Bypasses the package entirely; fragile and insecure	Use getDatabricksSession() for Spark or databricks.JDBCConnection for SQL
Inventing PySpark-like builder: databricks.spark.Session.builder().remote()	These classes do not exist in MATLAB	Use getDatabricksSession() which returns databricks.PySparkSession
Hardcoding tokens in source code	Security risk; tokens expire and rotate	Store credentials in .databrickscfg or environment variables
Calling table(DF) on unfiltered large tables	Transfers entire dataset to local MATLAB memory	Apply .filter() / .select() / .limit() first, then collect
Passing scalars to withColumn: DF.withColumn("x", 2)	Second argument must be a Column object	Use DF.withColumn("x", lit(2)) or DF.withColumn("x", col("y") * lit(2))
Writing without .mode(): DF.write.save(path)	Fails if target exists (default mode is "error")	Use .mode("overwrite") or .mode("append") before .save() or .saveAsTable()
Using databricks.SCIM.me to verify auth	Deprecated API	Verify with spark = getDatabricksSession(); table(spark.range(1))
Calling getDatabricksSession() without authMethod when user has PAT configured	Default chain tries methods in order and may trigger OauthU2M browser prompt instead of using PAT	Ask the user which auth method they want and pass authMethod="PAT" explicitly
Passing a DataFrame as an argument to a chained method: DF.unionAll(DF2)	The MATLAB Spark wrapper's bracket method only supports dot-chaining, not methods that take DataFrame arguments	Use spark.sql() with a SQL UNION ALL query instead, or if the data is small enough, collect both DataFrames to MATLAB tables, manipulate locally, and push back with table2dataset
Mismatched Python version	Session creation fails with version error	Match local Python to runtime version (e.g., 3.12 for runtime 17.3)
PAT token missing scopes	403 "does not have required scopes: clusters"	Regenerate the PAT with All access or at minimum the clusters scope enabled
cluster_id not set error	.databrickscfg has no cluster_id for the active profile	Run updateClusterId("your-cluster-id") or add cluster_id to the profile manually

---

Copyright 2026 The MathWorks, Inc.

---