Csv data handling

CSV Data Handling with D3.js

Overview

D3.js provides built-in CSV parsing. Understanding type coercion and data transformation is essential for visualization.

1. Loading CSV Files

Basic CSV Load

d3.csv("data.csv").then(data => {
  console.log(data);  // Array of objects
  // [{ key1: value1, key2: value2 }, ...]
});

With Error Handling

d3.csv("data.csv")
  .then(data => {
    console.log("Data loaded:", data.length, "rows");
    processData(data);
  })
  .catch(error => {
    console.error("Error loading CSV:", error);
  });

With Type Accessor

D3 can automatically convert types:

d3.csv("data.csv", row => {
  return {
    ticker: row.ticker,
    marketCap: +row.marketCap,  // convert to number
    sector: row.sector,
    value: parseFloat(row.value)
  };
}).then(processData);

2. Data Transformation

Parsing Numeric Values

data.forEach(d => {
  d.marketCap = +d.marketCap;  // unary + operator
  d.employees = parseInt(d.employees, 10);
  d.yield = parseFloat(d.yield);
});

Handling Missing Values

data = data.filter(d => {
  // Keep only rows with required data
  return d.marketCap && d.sector;
});

// Or replace missing with default
data.forEach(d => {
  d.marketCap = d.marketCap || 0;
  d.website = d.website || "N/A";
});

Filtering and Sorting

// Filter by sector
const tech = data.filter(d => d.sector === "Information Technology");

// Sort by market cap
data.sort((a, b) => b.marketCap - a.marketCap);

// Top 50 by market cap
const top50 = data.sort((a, b) => b.marketCap - a.marketCap).slice(0, 50);

3. Loading Multiple Files

Sequential Loading

Promise.all([
  d3.csv("companies.csv"),
  d3.csv("prices.csv")
]).then(([companies, prices]) => {
  // Both loaded
  const merged = mergeData(companies, prices);
  visualize(merged);
});

Loading Individual Stock Data

// Load main data
d3.csv("data/stock-descriptions.csv").then(stocks => {
  // For each stock, load price history
  const pricePromises = stocks.map(stock =>
    d3.csv(`data/indiv-stock/${stock.ticker}.csv`)
      .then(prices => ({
        ticker: stock.ticker,
        prices: prices
      }))
  );

  return Promise.all(pricePromises);
}).then(allData => {
  // Process combined data
  visualize(allData);
});

4. Aggregation & Grouping

Group by Category

const bySetor = d3.group(data, d => d.sector);
// Map { sector: [stocks...], ... }

// Or convert to array
const sectorGroups = Array.from(bySetor, ([sector, stocks]) => ({
  sector,
  count: stocks.length,
  totalCap: d3.sum(stocks, d => d.marketCap)
}));

Nesting (Hierarchical Grouping)

const nested = d3.nest()
  .key(d => d.sector)
  .entries(data);

// Returns: [{ key: "sector1", values: [stocks...] }, ...]

5. Formatting Numbers for Display

Format for Visualization

// Market cap formatter
const capFormatter = d3.format(".2s");  // "1.6T"
const capFormat = (value) => {
  const sizes = ['', 'K', 'M', 'B', 'T'];
  let sizeIndex = 0;
  let num = value;
  while (num >= 1000 && sizeIndex < sizes.length - 1) {
    num /= 1000;
    sizeIndex++;
  }
  return num.toFixed(2) + sizes[sizeIndex];
};

console.log(capFormat(1641026945024));  // "1.64T"

Format Percentages

const percentFormatter = d3.format(".2%");
console.log(percentFormatter(0.28806));  // "28.81%"

6. Data Validation

Check for Required Fields

function validateData(data) {
  return data.every(d => {
    return d.ticker &&
           d.sector &&
           d.marketCap !== undefined;
  });
}

if (!validateData(data)) {
  console.error("Invalid data structure");
}

7. Copying Data for Performance

Deep Copy to Avoid Mutations

// Shallow copy (references still point to original objects)
const copy1 = [...data];
const copy2 = data.slice();

// Deep copy (complete independence)
const copy3 = JSON.parse(JSON.stringify(data));

// Selective copy
const lightweightData = data.map(d => ({
  ticker: d.ticker,
  sector: d.sector,
  marketCap: d.marketCap
}));

Common Patterns

Parse and Enrich

d3.csv("stocks.csv")
  .then(stocks => {
    // Type conversion
    stocks.forEach(d => {
      d.marketCap = +d.marketCap;
      d.employees = +d.employees;
    });

    // Filter out invalid
    stocks = stocks.filter(d => d.marketCap > 0);

    // Add computed fields
    stocks.forEach(d => {
      d.capFormatted = capFormatter(d.marketCap);
      d.category = categorizeByMarketCap(d.marketCap);
    });

    return stocks;
  })
  .then(processedData => visualize(processedData));

Tips & Gotchas

• CSV string "1000000" stays a string until converted
• Empty cells become empty strings "", not null
• Use d => d.value to access object properties, not d["value"]
• D3's CSV parser handles quoted fields with commas automatically
• Use Promise.all() for loading multiple files in parallel
• Always validate data structure before visualizing