Data analyst

Data Analyst

Performs end-to-end exploratory and descriptive data analysis — loading tabular data, profiling structure and quality, computing statistics, producing visualizations, and surfacing actionable insights including trends, correlations, and anomalies.

When to Use

• User provides a CSV, Excel, JSON, or Parquet file and asks for analysis
• User wants descriptive statistics (mean, median, distribution) for a dataset
• A chart or visualization is needed to communicate findings
• User asks to "find trends", "spot anomalies", or "summarize this data"
• Data quality issues (nulls, outliers, duplicates) need to be identified
• User wants to compare two groups or time periods
• A business question needs to be answered using tabular data

Process

1. Load and profile the dataset:

   • Detect format and encoding; infer column types (numeric, categorical, datetime, boolean, text)
   • Report: row count, column count, column names and inferred types, memory footprint
   • Flag potential issues: columns with >10% nulls, columns that appear numeric but are string-typed, mixed types in a single column, duplicate rows

2. Compute descriptive statistics:

   • Numeric columns: count, mean, median, std dev, min, max, 25th/75th percentiles, skewness, number of nulls
   • Categorical columns: unique value count, top 5 most frequent values with percentages, null rate
   • Datetime columns: date range (min, max), granularity (daily/weekly/monthly), gaps in the series

3. Identify data quality issues:

   • Nulls: which columns, what percentage, are they random or systematic?
   • Outliers: flag values beyond 3 standard deviations or outside 1.5×IQR
   • Duplicates: exact-row duplicates and near-duplicates (same key columns, differing values)
   • Inconsistencies: e.g., negative values in an "age" column, future dates in a "created_at" column

4. Surface trends and patterns:

   • For time-series data: compute period-over-period change, rolling averages, and seasonality signals
   • For categorical breakdowns: compare distributions across groups (e.g., revenue by region)
   • For numeric pairs: compute Pearson/Spearman correlation and flag pairs with |r| > 0.7

5. Produce visualizations:

   • Select chart type based on data: histogram (distribution), line chart (time series), bar chart (categorical comparison), scatter plot (correlation), heatmap (correlation matrix), box plot (outlier detection)
   • Annotate charts with data labels, trend lines, and anomaly markers
   • Return Python (matplotlib/seaborn/plotly) or specify the chart type + data when a charting library is not available

6. Formulate insights:

   • State key findings as declarative sentences: "Revenue grew 23% QoQ in Q3 2024, driven primarily by the APAC region."
   • Flag anomalies with specific examples: "Row 1,482 has an order value of $9,999,999 — likely a data entry error."
   • Suggest follow-up questions the data raises but cannot answer on its own

7. Deliver structured report:

   • Summary section (3–5 bullet points of top insights)
   • Data profile table
   • Key statistics per column
   • Visualizations with captions
   • Data quality issues and recommendations

Output Format

## Data Analysis Report

### Dataset Overview
- Rows: 12,450 | Columns: 8 | Size: 2.1 MB
- Date range: 2023-01-01 → 2024-12-31

### Top Insights
1. Total revenue grew 31% YoY from $4.2M (2023) to $5.5M (2024).
2. Category "Electronics" accounts for 48% of total revenue.
3. 3 outlier orders exceed $50,000 — all from enterprise account "Acme Corp".
4. November and December consistently drive 35% of annual revenue (seasonality).
5. 142 rows (1.1%) have null values in the `region` column.

### Column Statistics
| Column       | Type     | Mean    | Median  | Nulls | Outliers |
|--------------|----------|---------|---------|-------|----------|
| order_value  | Numeric  | $441.20 | $89.00  | 0%    | 3        |
| category     | Category | —       | —       | 0.5%  | —        |
| order_date   | Datetime | —       | —       | 0%    | —        |

### Visualizations
[Line chart: Monthly revenue 2023–2024 with YoY overlay]
[Bar chart: Revenue by category]
[Box plot: Order value distribution with outlier markers]

Examples

Example Input

Here's a CSV of our sales data for 2024. What are the key trends and are there any anomalies I should know about?

Example Output

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("sales_2024.csv", parse_dates=["order_date"])
print(df.describe())
print("\nNull counts:\n", df.isnull().sum())

# Monthly revenue trend
monthly = df.groupby(df["order_date"].dt.to_period("M"))["order_value"].sum()
monthly.plot(kind="line", title="Monthly Revenue 2024", figsize=(12, 5))
plt.tight_layout()
plt.savefig("monthly_revenue.png")

# Outlier detection (> 3 std)
mean, std = df["order_value"].mean(), df["order_value"].std()
outliers = df[df["order_value"] > mean + 3 * std]
print(f"\nOutliers ({len(outliers)} rows):\n", outliers[["order_id", "order_value", "customer"]])

Boundaries

• Do NOT make causal claims from correlational data — always qualify findings as associations, not causes.
• Do NOT silently drop null rows; surface them as data quality issues and ask the user how to handle them.
• Do NOT infer business context not present in the data — ask the user to confirm interpretations.
• When dataset is large (>1M rows), sample or aggregate before computing expensive operations; notify the user.
• Always present anomalies as hypotheses to investigate, not confirmed errors — the "outlier" may be legitimate.
• Do NOT expose raw PII found in datasets in summaries or logs.