Spark
Apache Spark distributed computing. Use for big data processing.From its SKILL.md
npx -y skills add G1Joshi/Agent-Skills --skill sparkAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 12 stars12 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
1.1 KB, 266 tokens by cl100k_base, as published. Nobody here has run it
Apache Spark
Spark is the king of Big Data. v4.0 (2024/2025) makes Spark Connect the default, allowing thin clients (like VS Code) to connect to massive clusters easily.
When to Use
- Data Engineering: ETL at Petabyte scale.
- Streaming: Structured Streaming for real-time analytics.
- Legacy ML:
spark.ml(though mostly replaced by XGBoost/Torch).
Core Concepts
Spark Connect
Decouples client (your laptop) from server (the cluster). Allows using Spark from Go/Rust/TypeScript.
Catalyst Optimizer
Optimizes your SQL/DataFrame queries before execution.
RDD
The low-level API. Almost never used directly in modern Spark.
Best Practices (2025)
Do:
- Use PySpark: It is now a first-class citizen with Python UDF profiling.
- Use Delta Lake / Iceberg: Spark works best with modern table formats.
- Use
pandas_udf: For vectorized Python UDFs.
Don't:
- Don't use
rdd.map: It is slow (Python serialization). Use DataFrames.
References
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.