Reverse engineering custom c2 protocols

Reverse Engineering Custom C2 Protocols

When to Use

• A sample uses a non-standard protocol (or a custom layer over HTTP/TCP) and you must decode its traffic.
• You need to recover message framing, the encryption/encoding scheme, and the command set.
• You want to build a standalone decoder or emulator to parse captured beacon traffic.

Do not use this for documented protocols handled by existing dissectors — use the protocol analyzer directly instead of reversing from scratch.

Prerequisites

• A disassembler/debugger and a network capture (PCAP) of the sample's traffic where possible.
• The crypto/obfuscation skills for recovering keys and encodings.

Workflow

Step 1: Locate the network routines

Find send/recv/WSASend/WinHTTP calls (or the framework wrappers) and the buffers they operate on. These bracket the serialization and crypto.

Step 2: Recover framing

Trace how outbound buffers are built: magic bytes, length prefixes, sequence/ID fields, and type/opcode fields. Document the header layout.

[ magic(4) ][ len(4 LE) ][ opcode(1) ][ flags(1) ][ payload(len) ]

Step 3: Recover crypto/encoding

Identify the transform applied before send / after recv (XOR with embedded key, RC4, AES, base64). Recover keys from the binary or from the key-setup routine.

Step 4: Reconstruct the command set

Map opcodes to handlers (shell, download, upload, sleep/jitter, exit) by following the dispatch switch. Build a table of command → behavior.

Step 5: Build a decoder and validate

Implement a parser/decryptor and run it against captured traffic; confirm it yields coherent commands and recovers the beacon's config.

python scripts/analyst.py decode capture.bin --key 0x5a --magic 4d5a4331

Validation

• The decoder parses every message in a real capture without desync.
• Decrypted payloads are coherent (printable commands / structured config).
• The opcode table matches the dispatch logic observed in the binary.

Pitfalls

• Assuming fixed keys when the protocol negotiates a session key in the first exchange.
• Missing a length/sequence field and desynchronizing after the first message.
• Confusing transport (TLS) with the inner custom layer; decode the inner layer.

References

• See references/api-reference.md for the framing decoder.
• MITRE ATT&CK T1071 and T1573 (linked in frontmatter).