Embedding attacks

Embedding Attacks

Attack Surface

Embedding models convert text to dense vectors for semantic search, similarity comparison, and classification. Attacks against embedding models affect:

• RAG retrieval ranking (what gets retrieved)
• Embedding-based input filters (safety classifiers, topic filters)
• Semantic deduplication (bypass dedup to inject duplicate malicious content)
• User identity / session binding based on semantic similarity

Attack Variants

1. Nearest-Neighbour Poisoning

Craft text that embeds close to a target document's vector without semantic similarity to a human reader.

# Adversarial suffix method (GCG-style)
# Append a learned suffix to arbitrary text to move its embedding toward the target

import torch
from transformers import AutoTokenizer, AutoModel

def find_adversarial_suffix(model, tokenizer, source_text, target_embedding, steps=500):
    """Find a suffix that moves source_text's embedding toward target_embedding."""
    suffix = torch.randn(20, model.config.hidden_size, requires_grad=True)
    optimizer = torch.optim.Adam([suffix], lr=0.01)
    for step in range(steps):
        source_emb = embed(model, tokenizer, source_text + decode(suffix))
        loss = cosine_distance(source_emb, target_embedding)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return decode(suffix)

Use cases:

• Make a malicious document retrieve instead of a legitimate one
• Cause benign queries to retrieve adversarial documents

2. Semantic Collision Attack

Find two texts that are semantically dissimilar to humans but have similar embeddings:

# Search for natural-language semantic collisions
def find_collision(model, tokenizer, target_text, candidate_pool):
    target_emb = embed(model, tokenizer, target_text)
    for candidate in candidate_pool:
        candidate_emb = embed(model, tokenizer, candidate)
        if cosine_similarity(target_emb, candidate_emb) > 0.95:
            print(f"Collision found: {candidate}")

Use cases:

• Bypass embedding-based safety filters (craft text that is semantically similar to "safe" content but contains harmful meaning)
• Access-controlled document retrieval: craft a query that retrieves restricted documents without triggering access control checks

3. Embedding Inversion

Recover approximate original text from an embedding vector:

# Gradient-based inversion (vec2text approach)
# Requires white-box access to the embedding model

def invert_embedding(target_vector, model, tokenizer, steps=1000):
    """Approximately recover text from embedding vector."""
    # Start from random tokens
    tokens = torch.randint(0, tokenizer.vocab_size, (1, 50))
    tokens.requires_grad_(True)
    optimizer = torch.optim.Adam([tokens], lr=0.1)
    for _ in range(steps):
        emb = embed(model, tokenizer, tokens)
        loss = mse_loss(emb, target_vector)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return tokenizer.decode(tokens.argmax(-1))

Use cases:

• Recover approximate content from embedding-only storage (where text is discarded)
• Reconstruct PII that was "anonymized" to embeddings

4. Filter Bypass via Embedding-Space Routing

Identify the decision boundary of embedding-based classifiers:

def find_bypass(classifier, safe_text, unsafe_target, steps=100):
    """Find text that is classified as safe but semantically close to unsafe content."""
    current = safe_text
    for _ in range(steps):
        # Perturb current text toward unsafe target in embedding space
        perturbed = perturb(current, direction=unsafe_target)
        if classifier.predict(perturbed) == "safe":
            current = perturbed
        if semantic_distance(current, unsafe_target) < threshold:
            return current  # Found bypass
    return None

Tools

• vec2text — embedding inversion
• TextFooler — adversarial text generation
• OpenAttack — adversarial NLP framework
• ART — adversarial robustness

OWASP LLM Top 10 Mapping

• LLM01:2025 — Prompt Injection (via embedding-space manipulation of retrieved content)
• LLM02:2025 — Sensitive Information Disclosure (via embedding inversion)

MITRE ATLAS Mapping

• AML.T0015 — Evade ML Model
• AML.T0005 — Create Proxy ML Model (used to develop attacks)

Notes

Embedding attacks typically require either white-box access to the model or a significant query budget for black-box attacks. Embedding inversion against commercial APIs (OpenAI ada-002) has been demonstrated in academic research. Report successful inversions immediately as they may expose PII.