Azure ml dataset creator

Azure ML Dataset Creator

Generate synthetic datasets using Azure AI Foundry simulators for evaluation and fine-tuning—replacing manual data collection with automated simulation.

Two simulator types:

1. Simulator — Non-adversarial task-specific conversations from text, indexes, or custom prompts
2. AdversarialSimulator — Safety evaluation datasets with jailbreak attacks and harmful content

Use this skill when:

• Building evaluation or training datasets without production data
• Testing application responses to varied user queries
• Red-teaming for safety evaluation
• Creating multi-turn conversation datasets
• Need cost-effective synthetic data generation

Prerequisites

• Azure AI Foundry hub-based project (not Foundry)
• Azure OpenAI deployment (GPT-5-mini recommended for cost)
• Packages: azure-ai-evaluation, azure-identity
• For adversarial: Project in East US 2, France Central, UK South, or Sweden Central

Template Files

These are templates in examples/ directory. Copy and adapt them for your project:

examples/
  ├── generate_qa_from_text.py              # Template: Q&A from Wikipedia/documents
  ├── generate_conversation.py               # Template: Multi-turn conversations
  ├── generate_adversarial.py                # Template: Safety evaluation datasets
  ├── generate_jailbreak_attacks.py          # Template: UPIA/XPIA attack simulation
  ├── generate_with_custom_prompty.py        # Template: Custom simulator behavior
  ├── utils.py                               # Template: Utility functions
  └── custom_simulator_prompty/
      ├── user_override.prompty              # Template: Custom user behavior
      └── query_generator.prompty            # Template: Custom Q&A generation

Do NOT reference these files directly. Copy and adapt them for your project structure.

Quick Start

Generate Q&A from Text

1. Copy examples/generate_qa_from_text.py and examples/utils.py to your project
2. Run: python generate_qa_from_text.py
3. Outputs: training_data.jsonl in chat completion format
   • Extracts text from Wikipedia
   • Generates Q&A with multiple personas
   • Ready for SFT fine-tuning

Generate Multi-Turn Conversations

1. Copy examples/generate_conversation.py and examples/utils.py to your project
2. Run: python generate_conversation.py
3. Outputs: conversation_data.jsonl
   • Predefined conversation starters
   • Multi-turn dialogue (up to 5 turns)
   • User simulator with configurable behavior

Generate Safety Evaluation Data

1. Copy examples/generate_adversarial.py and examples/utils.py to your project
2. Run: python generate_adversarial.py
3. Outputs: adversarial_qa.jsonl, adversarial_conversation.jsonl, adversarial_summarization.jsonl
   • Tests responses to harmful/unsafe prompts
   • Covers: hate, sexual, violence, self-harm
   • Designed for safety evaluator benchmarking

Generate Jailbreak Attacks

1. Copy examples/generate_jailbreak_attacks.py and examples/utils.py to your project
2. Run: python generate_jailbreak_attacks.py
3. Outputs: direct_attack_baseline.jsonl, direct_attack_jailbreak.jsonl, indirect_attack.jsonl
   • UPIA: Direct user prompt injection
   • XPIA: Context/document injection
   • Baseline + attack variants for comparison

Custom Simulator Behavior

1. Copy examples/generate_with_custom_prompty.py, examples/utils.py, and examples/custom_simulator_prompty/ to your project
2. Run: python generate_with_custom_prompty.py
3. Outputs: custom_prompty_data.jsonl
   • Override user mood/persona (e.g., "professional")
   • Control response diversity (temperature, top_p)
   • Custom query-response generation logic

Data Formats

Chat Completion (for SFT fine-tuning)

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "What is Azure ML?"},
    {"role": "assistant", "content": "Azure Machine Learning is..."}
  ]
}

Q&A Format (for evaluation)

{"query": "What is Azure ML?", "response": "Azure Machine Learning is..."}

See examples/generate_qa_from_text.py for output conversion patterns.

Adversarial Scenarios

Scenario	Enum	Max Samples	Content Types
Q&A	ADVERSARIAL_QA	1,384	Hate, sexual, violence, self-harm
Conversation	ADVERSARIAL_CONVERSATION	1,018	Hate, sexual, violence, self-harm
Summarization	ADVERSARIAL_SUMMARIZATION	525	Hate, sexual, violence, self-harm
Search	ADVERSARIAL_SEARCH	1,000	Hate, sexual, violence, self-harm
Rewrite	ADVERSARIAL_REWRITE	1,000	Hate, sexual, violence, self-harm
Ungrounded Content	ADVERSARIAL_CONTENT_GEN_UNGROUNDED	496	Hate, sexual, violence, self-harm
Grounded Content	ADVERSARIAL_CONTENT_GEN_GROUNDED	475	All + jailbreak
Protected Material	ADVERSARIAL_PROTECTED_MATERIAL	306	Copyright detection

Integration with Training

Generated JSONL files can be uploaded to Azure ML for fine-tuning. Use azureml:// URI paths with azure-ml-llm-trainer skill for SFT/DPO/RL.

See examples/generate_qa_from_text.py for Azure ML data asset creation patterns.

Customization

User Simulator Parameters

Control response diversity and behavior with simulator kwargs. See examples/generate_with_custom_prompty.py for implementation.

Multi-Language Support

Adversarial simulators support multiple languages: Spanish, Italian, French, Japanese, Portuguese, Chinese (Simplified), German. Check example files for language parameter usage.

Callback Pattern

Target application must be defined as async callback accepting messages dict and optional parameters. See examples/generate_qa_from_text.py or examples/generate_conversation.py for callback implementation patterns.

Notes

• Synthetic data validation: Always review generated samples before production use
• Token costs: Monitor Azure OpenAI quota; use GPT-5-mini for cost efficiency
• Context limits: Keep text inputs under 5,000 characters for optimal results
• Reproducibility: Set randomization_seed for consistent results across runs
• Regional availability: Adversarial simulators require supported regions (see Prerequisites)
• Ethical use: Adversarial scenarios for testing/evaluation only; not for malicious use

Common Patterns

• Fine-Tuning Dataset: See examples/generate_qa_from_text.py for SFT data generation from text
• Safety Benchmarking: See examples/generate_adversarial.py and examples/generate_jailbreak_attacks.py for multi-scenario safety data
• Multi-Turn Evaluation: See examples/generate_conversation.py for conversation dataset generation

References

• Azure AI Foundry Simulator
• Evaluation Datasets Guide
• Adversarial Scenarios