Data ingestion

Data Ingestion Skill

This skill is responsible for fetching and converting time-series data from different sources into a standardized JSON format for other skills to consume.

Inputs:

• source_type: String indicating the data source (e.g., "csv", "cloud_monitoring").
• source_details: A dictionary or list containing the necessary information to access the data source.
  • For source_type: "csv": A list of file paths.

Output:

A JSON string in the standardized format (see README.md for details).

Workflow:

1. Validate Inputs: Check if source_type and source_details are provided.
2. Route to Parser: Based on source_type:
   • If source_type is "csv":
     • Ensure source_details is a list of file paths.
     • Create a list of temporary file names for intermediate JSON outputs (e.g., ~/.gemini/tmp/user/parsed_0.json, ...).
     • Parse Each CSV: Iterate through the input file paths:
       • Execute python ./skills/data-ingestion/parse_csv.py <input_csv_path> > <temp_json_path> using run_shell_command (with venv activation).
       • Check for errors.
     • Merge JSONs: Execute python ./skills/data-ingestion/merge_timeseries.py <temp_json_path_1> <temp_json_path_2> ... using run_shell_command (with venv activation).
     • Capture the stdout from merge_timeseries.py as the final result.
     • Clean up: Remove the temporary JSON files.
     • Handle any errors during script executions.
   • If source_type is not supported, return an error message.
3. Return JSON: Output the standardized JSON string.