flagchecker3: A Bytecode VM That Decrypts Itself Only If You Already Know the Flag

flagchecker3 was a reversing challenge at VSLCTF 2026: one Android APK, a PIN gate, and a 75-character flag. The Kotlin side is scaffolding — two activities, two native declarations, and nothing else worth reading. Everything that matters is in libveilcore.so, and the interesting half of it is a bytecode VM whose own instructions are encrypted under a key that the comparison results keep changing.

That design is meant to stop you from dumping the program. It does the opposite: it hands you an oracle.

The comparison result is an input to the decryption of the next instruction. Satisfy the comparisons in order and the program keeps running; miss one and it stops immediately.

Getting to the native code

apktool d -f app-release.apk -o apktool_out

The launcher is PinActivity, and both activities load the same library and declare one native method each:

# PinActivity
private final native checkPin(Ljava/lang/String;)Z

# MainActivity
private final native validateFlag(Ljava/lang/String;)Ljava/lang/String;

The APK ships four ABIs. The x86_64 build is the one to open on a desktop, since it disassembles natively and the logic is identical across all of them:

file apktool_out/lib/x86_64/libveilcore.so
strings -n 4 apktool_out/lib/x86_64/libveilcore.so \
  | grep -E 'Java_com_vsl_flagchecker_(PinActivity_checkPin|MainActivity_validateFlag)'
llvm-objdump -d -M intel --start-address=0x6bd0 --stop-address=0x6f90 \
  apktool_out/lib/x86_64/libveilcore.so

validateFlag also scans /proc/self/maps for frid1 and gum-js-loop, which is an anti-instrumentation check aimed at Frida. It is worth noticing and then ignoring: nothing in the solution below runs the app.

The PIN: a 6×6 matrix over a finite field

checkPin demands exactly six characters, requires each to be a digit, mixes them, and compares the result against a six-byte target. The mixing is immediately recognisable if you have read an AES implementation — multiply-by-two is (x << 1) ^ 0x1b when the high bit was set, which is Rijndael's xtime with its reduction polynomial, and the coefficients are the same small constants (1, 2, 3, 5) that appear in MixColumns.

So the check is a matrix–vector product over $GF (2^{8})$ :

M \cdot p = t, M \in GF (2^{8})^{6 \times 6}, t \in GF (2^{8})^{6} .

Both $M$ and $t$ sit in .rodata, so they can be lifted straight out of the file rather than transcribed from a disassembly listing. Inverting the matrix would work, but it is not worth writing: the input is constrained to six decimal digits, so the whole space is $10^{6}$ candidates. A brute force over a million six-digit strings is instant and cannot be got wrong.

PIN = 251005

TIP

Recovering constants by parsing the binary beats copying them out of a disassembler window. It takes about the same time, it is reproducible by anyone who runs the script, and it does not silently transpose two bytes of a 36-entry matrix — a mistake that produces no error, just a brute force that finds nothing.

The flag: a VM, and a key that listens to the comparisons

validateFlag checks that the input is 0x4B = 75 bytes long, then interprets an embedded bytecode blob. The dispatch is a jump table over a handful of opcodes — INB, PUSH32, XOR, ADD, AND, ROL8, CMPEQ, JIF, INLEN, RET — which is an ordinary enough stack machine. Two details make it not ordinary.

Every byte is encrypted before it is interpreted. Each fetched byte is XORed with the low eight bits of a 32-bit key. The key starts at 0x64390ce0.

The key is updated from the execution state, including the comparison result. After each instruction:

key = (key >> 3) ^ cmp_flag ^ (stack_size & 0xff) ^ 0xA5A5A5A5

cmp_flag is the outcome of the most recent CMPEQ. That single bit is the whole trick. The author encrypted the bytecode along the path a correct flag takes, so the stream only decrypts into valid instructions while the comparisons keep coming out the way the author assumed. One wrong byte flips cmp_flag, the key diverges on that step and never recovers, and the next fetch decrypts to 0x09 — not an opcode — and the VM stops.

This also means the blob does not need to be at a hardcoded offset to be found. Scanning the file for a position whose first ten bytes decrypt, under the initial key, into the expected prologue —

INLEN
PUSH32 0x4B
CMPEQ
JIF +3

— identifies it uniquely. The encryption that was supposed to hide the program is specific enough to act as its signature.

The per-character constraint

Emulating the VM shows all 75 checks share one shape:

INB idx
PUSH32 c1      ; XOR
PUSH32 add     ; ADD
PUSH32 0xff    ; AND
PUSH32 shift   ; ROL8
PUSH32 0xaa    ; XOR
PUSH32 expected ; CMPEQ

which is

expected = {rol}_{8} (((input [idx] \oplus c_{1}) + add) mod 256, shift) \oplus 0xaa

and every operation in it is invertible:

byte = (ror8(expected ^ 0xAA, shift) - add) & 0xFF
byte ^= c1

The constants differ per index, so there are 75 distinct little equations — but each is solved in one line once its operands are known, and the operands are sitting on the stack right before the CMPEQ that uses them.

The solver

The two features combine into something simpler than either: the VM stops at the first wrong byte, so emulating it tells us exactly which index is wrong, and the stack at that moment holds everything needed to compute the right value.

Start from any 75-byte guess.
Emulate until the first failing CMPEQ.
Read idx, c1, add, shift, expected off the preceding instructions.
Invert the formula and patch input[idx].
Repeat.

Each pass fixes exactly one character and gets strictly further than the last, so it terminates in 75 passes with no searching at all:

python3 solve.py app-release.apk

PIN: 251005
FLAG: VSL{d0nt_trust_ops_even_wh3n_th3y_pretend_to_be_s4f3_2026_edfefa180f0ca9a7}

What the obfuscation got wrong

The self-decrypting instruction stream is a real technique and it achieves a real goal: at no point in a legitimate run does the whole bytecode exist in cleartext, so a memory dump gets you fragments. Against a static emulator, though, it backfires in a way worth being precise about.

A checker's resistance to brute force comes from how little it tells you about a wrong guess. A 75-character flag has an astronomical space, but only if a wrong answer and a nearly-right answer look the same from outside. This one does the reverse — it reports where the first error is. Three independent properties were needed for that, and removing any one of them would have mattered:

The comparisons run in index order. Checking bytes in a scrambled, input-dependent order would make the failure position much less informative.
Execution halts at the first failure. A checker that evaluates all 75 constraints and ANDs the results together leaks one bit, not a position. Here the halt is the leak, and the rolling key is what causes the halt.
Each constraint touches exactly one input byte. Every equation is independently invertible, so knowing where the failure is means knowing what the byte must be. A constraint over a hash of the whole input, or over several bytes at once, would leave nothing to invert.

The last one is the real lesson, and it is not specific to CTFs. Any flag check that compares derived bytes one position at a time is a position oracle, whether the derivation is a rolling-key VM, a loop with an early return false, or a plain strcmp whose timing is measurable. The fix is not more obfuscation on top; it is to make the check atomic — compare one hash of the entire input against one stored digest, so a wrong answer is wrong everywhere and says nothing about where.

Obfuscation makes a program expensive to read. It cannot make a leaky design tight, and layering it over one can turn a halt into the very signal the design was trying not to give.