Overview
The challenge ships a two-stage virtual machine setup where an outer VM image carries the key that decrypts an inner VM program, and the inner program in turn validates a 20-byte input. A set of helper binaries and docs are included, and one of them is a decoy.
ouroboros_chall.zip
├── ouroboros # outer loader: outer.orb + inner.enc -> inner.orb
├── outer.orb # outer VM image (ORB1)
├── inner.enc # encrypted inner VM image
├── inner_vm # inner VM runtime: echo -n <20 bytes> | ./inner_vm inner.orb
├── validator # prints CORRECT/INCORRECT (decoy)
├── check # prints CORRECT/INCORRECT (real flag checker)
├── isa_outer.txt # outer ISA reference
├── isa_inner.txt # inner ISA reference
└── outer_disasm.txt # "pre-generated" outer disassembly
The solve chain:
flowchart LR
A[outer.orb<br/>RAM 0x10..0x1F] -->|raw1 XOR raw2<br/>+ key schedule| B[8-byte keystream]
B -->|repeating XOR| C[inner.enc -> inner.orb]
C -->|input XOR RAM 0x20 == RAM 0xE0| D[20-byte input]
D -.->|only proves the VM is satisfied| E[inner_vm exit 0]
F[check binary<br/>.rodata XOR 0x5c] --> G[flag]
Prerequisites / Setup
- Python 3 (
struct) objdump/readelf/stringsfor ELF inspectionouroboros_chall.zipextracted in the working directory
Recon
Step 1 — Unpack and identify file types
Goal: List the bundle contents and confirm file types.
unzip ouroboros_chall.zip
file *
check, inner_vm, ouroboros, validator : ELF 64-bit LSB pie executable, x86-64, stripped
inner.enc, outer.orb : data
isa_*.txt, outer_disasm.txt : text
The __MACOSX/ folder is just archive junk and can be ignored.
Step 2 — Read the ISA docs (treat as hints, not truth)
Goal: Understand the documented VM architecture, while noting where the docs lie.
Outer VM (isa_outer.txt): 8-bit registers r0-r7, 256 bytes of RAM, ROM bytecode, opcodes LDI/XOR/ADD/LDR/JNZ/STR/HLT (0x01-0x07). The key point is the sentence:
The decryption key for
inner.encis stored atRAM[0x10:0x18].
Inner VM (isa_inner.txt): same registers, 20 bytes of stdin loaded at RAM[0x00..0x13], opcodes LDI/XOR/ADD/LDR/CMP/JNZ/HLT. The doc describes JNZ as “jump if equal”, which turns out to be wrong (see Step 4).
Image format
Goal: Confirm the on-disk layout of each .orb image.
Both images share the layout magic | prog_len | RAM[256] | bytecode:
| Offset | Size | Meaning |
|---|---|---|
0x00 |
4 | magic: ORB1 (outer) / INR1 (inner) |
0x04 |
2 | prog_len, big-endian |
0x06 |
256 | initial RAM image |
0x106 |
prog_len |
bytecode |
Exploitation
Step 2 — The outer VM does nothing
Goal: Extract the outer image’s RAM blobs and bytecode.
d = open("outer.orb", "rb").read()
ram, code = d[6:262], d[262:]
print(code.hex()) # 07 -> HLT
print(ram[0x10:0x20].hex()) # 3e97a1d3e239c7ba f9921126a724c935
The bytecode is a single HLT. The outer VM’s only purpose is to carry its RAM image, which holds two 8-byte blobs:
raw1 = RAM[0x10:0x18] = 3e97a1d3e239c7ba
raw2 = RAM[0x18:0x20] = f9921126a724c935
The doc claims the key is RAM[0x10:0x18], but using raw1 directly does not decrypt anything. The loader must transform it.
Step 3 — Run the loader and look at the inner image
Goal: Confirm inner.enc decrypts to an INR1 image and recover the repeating keystream.
./ouroboros outer.orb inner.enc # prints "outer VM: OK", writes inner.orb
The decrypted file starts with INR1, so the scheme is clearly a byte-wise XOR. XORing inner.enc against inner.orb shows the keystream is a repeating 8-byte value:
ks = bytes(a ^ b for a, b in zip(enc, plain))
# 5b2f3261dae0c3d8 5b2f3261dae0c3d8 ... (period 8)
So inner.enc is encrypted with a repeating 8-byte key 5b2f3261dae0c3d8. That differs from raw1, so the key is derived. Simple guesses (raw1^key, raw2^key, add/sub, raw1^raw2) do not give the keystream directly, which means a key schedule exists in the loader.
Step 4 — Reverse the key schedule in ouroboros
Goal: Recover the 8-byte key from raw1 and raw2.
objdump -d -M intel --no-show-raw-insn ouroboros
The interesting block is around 0x13f3-0x14a0:
13f3 movq xmm0, [rsp+0x140] ; raw2
1403 movq xmm1, [rsp+0x138] ; raw1
140c pxor xmm0, xmm1 ; t = raw1 ^ raw2
1410 movq [rsp+0x10], xmm0
1440 add edx, 1 ; j = 1..8
1443 add rcx, 1 ; rcx walks t[0..7]
1447 mov eax, edx
1449 and eax, 7
144c movzx eax, [rsp+rax+0x10] ; t[j & 7]
1451 ror al, 3
1454 xor al, [rcx-1] ; ^ t[j-1]
1457 xor eax, 0x3c
145a mov [rcx-1], al ; t[j-1] = ... (in place)
145d cmp edx, 8 / jne 1440
1480 mov edx, rax & 7 ; stream loop over the file
1486 movzx edx, [rsp+rdx+0x18] ; key[i & 7]
148b xor dl, [rbp+rax] ; out[i] = in[i] ^ key[i & 7]
Translated to Python:
ror8 = lambda x, n: ((x >> n) | (x << (8 - n))) & 0xFF
t = bytearray(a ^ b for a, b in zip(raw1, raw2)) # c705b0f5451d0e8f
for j in range(1, 9): # sequential, in place
t[j-1] = ror8(t[j & 7], 3) ^ t[j-1] ^ 0x3C
key = bytes(t) # 5b2f3261dae0c3d8
Decrypt:
inner = bytes(c ^ key[i & 7] for i, c in enumerate(open("inner.enc","rb").read()))
assert inner[:4] == b"INR1"
Step 5 — Disassemble the inner program
Goal: Parse the inner image and understand the validation logic.
Parsing with the correct 6-byte header (prog_len = 0x01a2):
0000: LDI r6, 0x00 ; match counter = 0
0003: LDI r3, 0x14 ; expected count = 20
0006: LDI r7, 0x01
0009: LDR r0, [0x00] ; r0 = input[0]
000c: LDR r1, [0x20] ; r1 = RAM[0x20]
000f: XOR r0, r1 ; r0 = input[0] ^ RAM[0x20]
0012: LDR r2, [0xE0] ; r2 = RAM[0xE0]
0015: CMP r0, r2
0018: JNZ +3 ; mismatch -> skip the ADD
001a: ADD r6, 1 ; match -> counter++
001d: LDR r0, [0x01] ; next byte...
... (same 6-instruction block for i = 0..19)
0199: CMP r6, r3 ; all 20 matched?
019c: JNZ +2 ; not equal -> jump to HLT 1
019e: HLT 0 ; success
01a0: HLT 1 ; failure
The program is 418 bytes: 3 setup instructions, 20 identical 6-instruction blocks, then the final test. Each of the 20 input bytes is checked independently:
input[i] ^ RAM[0x20 + i] == RAM[0xE0 + i] for i in 0..19
Both tables sit in the decrypted RAM image:
RAM[0x20..0x33] = 9f571cda2c9bfafc26417e19c1992e742a59b24d
RAM[0xE0..0xF3] = 8a85e1f427aecbe6a4e371ed5bb2d3c4878d7023
(Take exactly 20 bytes from each.) Since XOR is its own inverse, the required input is just the XOR of the two tables:
inp = bytes(iram[0x20+i] ^ iram[0xE0+i] for i in range(20))
# 15d2fd2e0b35311a82a20ff49a2bfdb0add4c26e
Step 6 — Verify against the inner VM
Goal: Confirm the 20-byte input satisfies the inner VM.
python3 -c "import sys; sys.stdout.buffer.write(bytes.fromhex('15d2fd2e0b35311a82a20ff49a2bfdb0add4c26e'))" | ./inner_vm inner.orb
echo $? # 0 -> success
AAAA... and the correct input with any single byte changed both return exit code 1, so the inner VM is strict. The solver also re-implements the inner VM in Python (exit 0 for the right input, 1 for a wrong one).
Step 7 — The decoys: validator vs check
Goal: Confirm which binary is the real flag checker.
Feeding the 20 bytes (raw, hex, with and without newline) to both helper binaries:
| Binary | Result for the 20 bytes | Result for AAAA |
|---|---|---|
validator |
CORRECT |
CORRECT (also for empty stdin) |
check |
INCORRECT |
INCORRECT |
validator is a decoy: it always says CORRECT. check is the real comparator, and it rejects the 20 bytes, so those bytes are not the flag. The challenge text (“Can you escape it?”) is hinting that the inner VM is another loop, not the exit.
Step 8 — Static analysis of check
Goal: Decode the flag embedded in check’s .rodata.
objdump -d -M intel --no-show-raw-insn check
objdump -s -j .rodata check
The main logic is:
- Copy a 46-byte blob from
.rodata(0x2020) into a stack buffer, XORing each byte with0x5c(xor eax, 0x5cat0x10e0). fgetsa line from stdin (128 bytes), strip the trailing\n.strcmp(input, decoded_blob): printCORRECTif equal, elseINCORRECT.
.rodata @ 0x2020:
2e6c6c28 276c292e 6c3e6c2e 6c2f036a 3a646a6c 6c6e3a6e 6d686869 69386868 693a386a 696e6e21 00
Decoding the blob:
blob = bytes.fromhex("2e6c6c28276c292e6c3e6c2e6c2f036a3a646a6c6c6e3a6e6d68686969386868693a386a696e6e21")
print(bytes(b ^ 0x5c for b in blob).decode()) # r00t{REDACTED}
Pipe the decoded string back in as a sanity check:
echo 'r00t{REDACTED}' | ./check # replace with the decoded value -> CORRECT
Flag
r00t{REDACTED}
Decoded from check’s .rodata blob (XOR 0x5c), independently confirmed by piping it back into ./check → CORRECT.
Solver
solve_ouroboros.py runs the full chain (outer image, key schedule, decrypt, inner table XOR, VM emulation, decode flag from check):
python3 solve_ouroboros.py ./ouroboros_chall/
[outer] bytecode = 07 (07 = HLT, VM does nothing)
[outer] raw1=3e97a1d3e239c7ba raw2=f9921126a724c935
[key] keystream = 5b2f3261dae0c3d8
[inner] 20-byte input = 15d2fd2e0b35311a82a20ff49a2bfdb0add4c26e (validator accepts anything)
[inner] emulated exit code = 0
[flag] r00t{REDACTED}
Core of the script:
ror8 = lambda x, n: ((x >> n) | (x << (8 - n))) & 0xFF
ram, code = parse_orb(open("outer.orb", "rb").read(), b"ORB1")
raw1, raw2 = ram[0x10:0x18], ram[0x18:0x20]
t = bytearray(a ^ b for a, b in zip(raw1, raw2))
for j in range(1, 9):
t[j-1] = ror8(t[j & 7], 3) ^ t[j-1] ^ 0x3C
key = bytes(t)
inner = bytes(c ^ key[i & 7] for i, c in enumerate(open("inner.enc", "rb").read()))
iram, icode = parse_orb(inner, b"INR1")
inp = bytes(iram[0x20+i] ^ iram[0xE0+i] for i in range(20))
elf = open("check", "rb").read()
off = elf.index(bytes([0x2E, 0x6C, 0x6C, 0x28]), 0x2000)
flag = bytes(b ^ 0x5C for b in elf[off:elf.index(b"\x00", off)]).decode()
Lessons / Takeaways
- Treat docs as hints, not truth. The ISA doc said the key was
RAM[0x10:0x18], but the real key wasraw1 ^ raw2pushed through a rotate/XOR schedule. - XORing ciphertext with known plaintext (the
INR1magic) immediately exposes a repeating-key stream and its period. - Verify header sizes against total file length (
6 + 256 + prog_len) before trusting a disassembly. - In-place loops that read from the buffer they are modifying must be emulated in-place, not on a copy.
validatoralways returnsCORRECT; a checker that accepts everything is a decoy.- A satisfied inner VM does not mean you have the flag. Confirm with the binary that actually compares the flag string.
- The inner ISA doc says
JNZis “jump if equal” but the VM jumps on not equal. Validate documented semantics against the real runtime with a wrong input, not just a right one.
Techniques
- Custom VM Reversing
- XOR Cipher Known-Plaintext
- Key Schedule Reversing
- Static Analysis
References
- objdump / binutils: https://sourceware.org/binutils/
- radare2: https://radare.org/r2/

