agentsclimarketplace

Multimodal rag architecture

Skill mouadja02/skills/skills/llm-tooling/multimodal-rag-architecture

A curated collection of agent skills for your AI agents - engineering craft, prompt engineering, design, growth marketing, ...

Install
npx -y skills add mouadja02/skills --skill multimodal-rag-architecture

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 8 stars8 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use when designing or auditing RAG over PDFs, images, tables, charts, equations, video frames, or heterogeneous documents where text-only chunking loses important evidence.

SKILL.md

3.2 KB, as published. Nobody here has run it

Multimodal RAG Architecture

Use this skill when a knowledge base contains more than plain text. Treat images, tables, equations, layout, captions, and cross-page structure as first-class evidence instead of stripping everything into text chunks.

When Text-Only RAG Fails

Switch to multimodal RAG when:

  • Important answers live in tables, charts, screenshots, diagrams, figures, or equations
  • PDFs have layout-dependent meaning such as forms, invoices, manuals, or scientific papers
  • The same concept appears across text, image, and table regions
  • Long documents cause retrieval to miss sparse visual evidence
  • Users ask for answers that require comparing visual and textual context

Architecture

  1. Parse by modality

    • Text blocks
    • Tables
    • Figures and images
    • Equations
    • Captions
    • Page and section layout

  2. Create multimodal evidence nodes

    • Preserve source document, page, bounding box, modality, caption, extracted text, and raw asset pointer.
    • Attach normalized text summaries for retrieval.
    • Keep original media accessible for answer verification.

  3. Build relationships

    • Figure-to-caption
    • Table-to-section
    • Equation-to-explanation
    • Cross-page continuation
    • Visual element-to-text mention

  4. Retrieve in stages

    • Query rewrite into text, table, and visual intents.
    • Hybrid lexical/vector retrieval over summaries and extracted text.
    • Graph traversal to pull adjacent evidence.
    • Optional visual reranking for image-heavy answers.

  5. Generate with provenance

    • Cite document, page, modality, and region.
    • Distinguish extracted facts from model interpretation.
    • Re-open raw assets when the answer depends on visual detail.

Design Rules

  • Do not OCR everything and discard layout.
  • Do not embed raw image summaries without retaining the image.
  • Do not answer from captions alone when the figure itself matters.
  • Prefer smaller modality-specific indexes over one overloaded index.
  • Keep chunk boundaries aligned to document structure, not fixed token counts.
  • Record extraction confidence for OCR, table parsing, and visual descriptions.

Helper Script

Use rag_modality_audit.py to scan a folder and estimate whether a corpus needs multimodal handling:

python scripts/rag_modality_audit.py ./docs

References

Read architecture-checklist.md when implementing or reviewing a multimodal RAG pipeline.

External grounding:

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.