agentsclimarketplace

Dspy adapters multimodal

Skill OmidZamani/dspy-skills/skills/dspy-adapters-multimodal

Collection of Claude Skills for DSPy framework - program language models, optimize prompts, and build RAG pipelines systematically

Install
npx -y skills add OmidZamani/dspy-skills --skill dspy-adapters-multimodal

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

What its author says it does

Copied from the file, not written here

Use for DSPy adapter selection, JSONAdapter, XMLAdapter, ChatAdapter, native function calling, structured outputs, and multimodal inputs like dspy.Image or dspy.Audio.

SKILL.md

3.4 KB, as published. Nobody here has run it

DSPy Adapters and Multimodal I/O

Goal

Choose an adapter deliberately and model image, audio, and file inputs with DSPy's typed primitives.

Adapter Selection

AdapterUse it for
dspy.ChatAdapter()Default, human-readable field markers, broad model compatibility
dspy.JSONAdapter()Structured JSON output and native function calling where supported
dspy.XMLAdapter()XML-tagged fields when XML is easier for the target LM to follow
dspy.TwoStepAdapter()A separate extraction pass when parsing needs extra help

Configure globally or for a limited scope:

import dspy

dspy.configure(
    lm=dspy.LM("openai/gpt-4o-mini"),
    adapter=dspy.JSONAdapter(),
)

with dspy.context(adapter=dspy.XMLAdapter()):
    result = dspy.Predict("question -> answer")(question="What is DSPy?")

Native Function Calling

JSONAdapter enables native function calling by default. ChatAdapter keeps text parsing by default. Override either behavior explicitly:

chat_native = dspy.ChatAdapter(use_native_function_calling=True)
json_manual = dspy.JSONAdapter(use_native_function_calling=False)

DSPy falls back to manual parsing when the configured LM does not support native function calling.

Image Inputs

class DescribeImage(dspy.Signature):
    image: dspy.Image = dspy.InputField()
    description: str = dspy.OutputField()

describe = dspy.Predict(DescribeImage)
result = describe(image=dspy.Image("./diagram.png"))

Pass a local path, HTTP URL, bytes, PIL image, or existing data URI directly to dspy.Image(...).

Audio and File Inputs

class SummarizeAudio(dspy.Signature):
    audio: dspy.Audio = dspy.InputField()
    summary: str = dspy.OutputField()

audio = dspy.Audio.from_file("./meeting.wav")
summary = dspy.Predict(SummarizeAudio)(audio=audio)

class SummarizeFile(dspy.Signature):
    file: dspy.File = dspy.InputField()
    summary: str = dspy.OutputField()

document = dspy.File.from_path("./research.pdf")
summary = dspy.Predict(SummarizeFile)(file=document)

Provider capabilities vary. Verify that the selected model accepts the media type before deployment.

Best Practices

  1. Start with ChatAdapter; switch only for a measured reason.
  2. Use typed signatures for structured output.
  3. Test adapter behavior against the exact production model.
  4. Avoid deprecated Image.from_file() and Image.from_url() helpers; call dspy.Image(...).
  5. Keep local file handling and uploaded file IDs within provider policy.

Related Skills

Official Documentation

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.