Imagine a developer who is worried that someone might read the password validation code.

So they encrypt the validation functions, decrypt them at runtime, put them into freshly mapped memory, add a custom virtual machine, shuffle the password characters around, and throw some arithmetic on top.

At this point the code looks intimidating enough to deserve its own movie trailer.

Then you remember one small problem: the client still has to run the checker.

That means the machine still has to reveal the instructions, the values being compared, and the rules used to produce the final decision. Obfuscation can make reversing slower. It does not magically turn a client into a trusted oracle.

This article walks through a generic example of a heavily obfuscated password validator, from the first ptrace() call to the custom VM, runtime decryption, dynamic tracing, and finally the much less exciting mathematics that actually decides whether the password is correct.

What Is Client-Side Password Validation, Anyway?

A client-side password validator is any scheme where the client itself decides whether an entered secret is correct.

That can be as simple as:

if password == "correct-horse-battery-staple":
    print("Correct!")

or as dramatic as:

Encrypted code
     |
     v
Runtime decryption
     |
     v
Executable memory
     |
     v
Custom VM
     |
     v
Password permutation
     |
     v
Arithmetic
     |
     v
Comparison

The second version looks much harder to reverse.

It is.

It is also still a client.

If the answer depends on a secret value embedded in the client, an analyst who controls the client can eventually inspect the same information the client uses.

That is the core problem.

Obfuscation vs. Security

Obfuscation is useful when the goal is to increase attacker effort.

It is not a replacement for a security boundary.

A good way to think about it is putting a combination lock inside a glass box. The box can have smoke machines, three warning labels, and a laser turret. The combination is still physically sitting inside the box.

Why Should You Care?

This design shows up whenever developers want to hide a verification secret from casual inspection.

Common cases include:

Desktop license checks
Client-side activation systems
Game key validation
Reverse engineering challenges
Proprietary protocol clients
Offline feature gates
Mobile applications with embedded logic

The security impact depends on what is being protected.

Design	Main risk
Plaintext comparison	Secret recovery is trivial
Hash comparison in the client	Secret may still be brute-forced or the check may be patched
Encrypted validation code	Runtime analysis can recover the plaintext
Custom VM	More reversing work, same trust boundary
Server-side verification	Secret can remain outside the client

The important distinction is between raising cost and creating a real security boundary.

A custom VM raises cost.

A server-side verifier creates a boundary.

Those are very different jobs.

What Does the Obfuscated Validator Actually Do?

A generic validator can be modeled as six stages.

1. Anti-debugging
2. Runtime decryption
3. VM initialization
4. Input preparation
5. Mathematical validation
6. Final success/failure branch

The trick is that these stages are mixed together.

The first few minutes of analysis can feel like trying to understand a washing machine by staring at the instruction manual for the toaster.

That is normal.

You do not need to understand everything at once.

Stage 1: Start With Runtime Behavior

Before opening the binary in a decompiler, run it under a syscall or library tracer.

For example:

ltrace ./validator

A typical output might reveal:

malloc(...)
mmap(...)
mmap(...)
mmap(...)
ptrace(...)
exit(...)

The mmap() calls are immediately interesting because the program is repeatedly creating new memory regions.

The combination of:

mmap(...)
mmap(...)
mmap(...)
...
ptrace(...)

is worth investigating before touching the hundreds of lines of decompiler output.

Why `ptrace()` Matters

One common anti-debugging trick is:

ptrace(PTRACE_TRACEME, 0, NULL, NULL);

If the process is already being traced, the call can fail.

The program then exits.

That makes normal debugging annoying, which is presumably the point.

The important part for the analyst is not "this is impossible to debug."

It is:

We found a branch whose only job is to stop us from debugging.

That is something we can isolate and neutralize.

For example, in a controlled reversing environment, the return value can be modified in the debugger so execution continues normally.

Stage 2: Find the Runtime-Decrypted Code

Now open the binary in Ghidra, IDA, Binary Ninja, or your preferred disassembler.

Even with symbols stripped, the control flow eventually leads to a function responsible for setting up the runtime environment.

The interesting pattern is usually:

void *region = mmap(
    NULL,
    size,
    PROT_READ | PROT_WRITE | PROT_EXEC,
    MAP_PRIVATE | MAP_ANONYMOUS,
    -1,
    0
);

Then some data is written into region.

That combination is suspicious for two reasons:

The memory is executable.
The contents did not exist there in their final form at process start.

That strongly suggests runtime code generation, decryption, unpacking, or some other form of self-modifying behavior.

The first useful question is therefore:

What does each mapped region contain after initialization?

Dumping the Mapped Regions

A small GDB helper can capture the contents of memory immediately after each mmap() returns.

The generic idea is:

break mmap

commands
    silent

    set $len = $rsi
    finish

    set $addr = $rax

    printf "mapped %p, size %lu\n", $addr, $len

    dump memory
        /tmp/region.bin
        $addr
        $addr+$len

    continue
end

For repeated regions, keep the addresses and lengths in arrays and dump each one to a separate file.

Once those files are available, disassemble them independently.

You may find that every region begins with a valid function prologue or endbr64, followed by ordinary machine code.

That is the key moment.

The program is not hiding an unknown language forever.

It is hiding normal machine code until runtime.

Stage 3: Realizing You Are Looking at a VM

Eventually, another function starts looking suspiciously like a VM dispatcher.

A simplified version might look like this:

for (;;) {
    uint8_t opcode = *ip;

    if (opcode == VM_EXIT)
        break;

    handlers[opcode](
        &ip,
        &sp,
        &stack,
        &state
    );
}

This pattern is extremely recognizable.

You have:

An instruction pointer
An opcode
A dispatch table
A set of handler functions
A loop
A termination opcode

Congratulations.

Someone built a tiny CPU inside a normal CPU.

Because apparently x86 needed a roommate.

Finding the VM Bytecode

Once the dispatcher is understood, identify the array or memory region that contains the VM instructions.

It may look like random bytes:

03 00 00 00
12 17 00 00
19 8A 13 00
05 00 00 00
...

Random-looking is exactly what bytecode tends to look like when you do not know the instruction format.

A useful clue is the instruction width.

If you see repeated four-byte records, try splitting the stream into 4-byte instructions.

A generic parser might be:

def split_instructions(words, width=4):
    return [
        words[i:i + width]
        for i in range(0, len(words), width)
    ]

Then map the last field, first field, or another likely byte to the opcode.

You can confirm the guess by matching values against the dispatcher.

Build an Opcode Table

Do not try to understand the entire VM at once.

Build a tiny table.

For example:

Opcode	Meaning
`0x00`	Get input length
`0x01`	Shift left
`0x02`	Modulo
`0x03`	Load user input
`0x04`	Divide
`0x05`	Add
`0x06`	Multiply
`0x07`	Anti-debugging wrapper
`0x08`	OR
`0x0B`	Jump
`0x0D`	Password loop
`0x10`	Copy value
`0x11`	Inner loop
`0x12`	Load password byte
`0x13`	Compare
`0x16`	Conditional branch
`0x17`	Relative move
`0x18`	Subtract
`0x19`	Load immediate
`0x1A`	Shift right
`0x1C`	Return

You do not need a perfect emulator yet.

You just need enough semantics to understand where the password bytes go.

Stage 4: Follow the Password Bytes

The most useful instruction in this kind of validator is usually the one that reads a character from the original input.

A simplified handler might be:

uint8_t value = password[index];
push(value);

From the VM's perspective, this may happen in a strange order:

password[24]
password[14]
password[27]
password[15]

password[11]
password[7]
password[12]
password[4]

...

That is not random.

It is a permutation.

The VM is effectively saying:

Give me character 24, then 14, then 27, then 15, and I will call that a chunk.

At this point the scary part of the validator starts becoming boring.

That is good.

Boring code is much easier to solve.

Stage 5: Recover the Final Arithmetic

Suppose four password characters are packed into one 32-bit integer:

def pack_chunk(chars):
    return (
        chars[0]
        | (chars[1] << 8)
        | (chars[2] << 16)
        | (chars[3] << 24)
    )

Then the VM applies a rotate:

encoded = rol32(chunk, rotation)

and compares it with a constant:

if encoded != expected:
    fail()

That looks complicated until you remember that bit rotations are reversible.

If:

encoded = ROL32(chunk, rotation)

then:

chunk = ROR32(encoded, rotation)

That is the whole trick.

Reversing the Check

The inverse operation is:

MASK32 = 0xffffffff

def ror32(value, count):
    count &= 31

    if count == 0:
        return value & MASK32

    value &= MASK32

    return (
        (value >> count)
        |
        (value << (32 - count))
    ) & MASK32

Given:

expected = 0xDEADBEEF
rotation = 11

we calculate:

chunk = ror32(expected, rotation)

Then unpack it:

chars = bytes([
    chunk & 0xff,
    (chunk >> 8) & 0xff,
    (chunk >> 16) & 0xff,
    (chunk >> 24) & 0xff,
])

That gives us the original four password bytes.

Repeat this for all chunks and put the recovered bytes back into their original positions.

A Generic Solver

The final solver does not need to emulate every instruction once the final checks are known.

For example:

CHECKS = [
    ([24, 14, 27, 15], 7,  0x12345678),
    ([11,  7, 12, 4],  14, 0x87654321),
    # ...
]

password = bytearray(b"?" * 32)

for positions, rotation, target in CHECKS:
    chunk = ror32(target, rotation)

    chars = bytes([
        chunk & 0xff,
        (chunk >> 8) & 0xff,
        (chunk >> 16) & 0xff,
        (chunk >> 24) & 0xff,
    ])

    for position, char in zip(positions, chars):
        password[position] = char

print(password.decode())

Notice what happened.

The custom VM is no longer important.

The encrypted functions are no longer important.

The anti-debugging code is no longer important.

Once the validation equations are recovered, the rest is just bit manipulation and bookkeeping.

The giant security theater collapsed into four bytes at a time.

What If the Math Is More Complicated?

Sometimes the final operation is not just a rotate.

You may see:

x = ((x * A) + B) ^ C
x = rol32(x, R)
x = x mod N

You have three options.

1. Reverse the operations manually

If every operation is invertible, walk backward.

final
  |
  v
inverse XOR
  |
  v
inverse rotate
  |
  v
subtract B
  |
  v
multiply by inverse(A)

2. Use Z3

SMT solving is useful when the equations become unpleasant.

from z3 import *

x = BitVec("x", 32)

solver = Solver()

solver.add(
    RotateLeft(x, 7) == 0x12345678
)

if solver.check() == sat:
    print(solver.model()[x])

3. Emulate the VM

When there are loops, branches, and state changes, writing a small emulator is often faster than manually rewriting a page of decompiler output.

That is exactly why VM challenges are interesting.

The VM itself becomes the thing you reverse.

A Better Way to Debug the VM

A full debugger trace can become unreadable very quickly.

Instead, log only the state that explains the current instruction.

For example:

ip=0132 opcode=12 password[24] -> 0x41
ip=0135 opcode=19 push 0x00000007
ip=0138 opcode=05 add
ip=0142 opcode=1A shr 0x03
ip=0146 opcode=13 compare

This gives you a high-level execution trace without forcing you to mentally simulate registers, stack slots, and every unrelated instruction.

A tiny trace function is often enough:

def trace(ip, opcode, **state):
    print(
        f"ip={ip:04x} "
        f"opcode={opcode:02x} "
        f"{state}"
    )

When reversing an unfamiliar VM, observability is worth more than elegance.

Why the Custom VM Does Not Save the Design

A custom VM is a speed bump.

It can:

Hide intent from static analysis
Increase analyst workload
Slow automated scanners
Make casual patching harder
Complicate debugging

It cannot:

Keep secrets away from the client
Make client-side decisions trustworthy
Prevent runtime observation
Stop someone from instrumenting the verifier

The client needs the information to perform the check.

An analyst can observe the information while the check is happening.

That is the uncomfortable part.

What Would a Safer Design Look Like?

Do not put the secret validation decision entirely on the client.

A safer pattern is:

Client
  |
  | password / credential proof
  v
Server
  |
  | validation
  v
Protected secret / database

For a desktop application that needs offline functionality, use signed capabilities, asymmetric cryptography, short-lived authorization tokens, or another design where possession of the client binary does not reveal the protected secret.

The exact design depends on the product.

The principle does not.

If the client can independently prove the secret, the client probably knows enough to be studied.

Testing and Audit Points

When reviewing a suspicious password checker, ask:

Does the binary contain the expected value or a reversible transformation of it?
Is validation performed locally?
Does the program create executable memory at runtime?
Are functions decrypted immediately before use?
Is there a dispatch table based on attacker-observable instruction values?
Does the input get rearranged before validation?
Are comparisons performed against constants embedded in the client?
Can the validation branch be patched or traced?
Does the application rely on obfuscation as the primary protection?
Would moving verification server-side remove the secret from the client?

If the answers start sounding like "yes, yes, yes, and unfortunately yes," you probably found an expensive password checker whose biggest achievement is making your debugger sweat.

Defensive Design Checklist

Category	Action	Goal
Trust boundary	Move sensitive verification server-side	Keep secrets off the client
Credentials	Never embed plaintext secrets	Prevent direct extraction
Authorization	Use signed or server-issued tokens	Separate identity from local logic
Obfuscation	Treat it as defense-in-depth	Slow analysis, do not rely on it
Debug resistance	Do not consider anti-debugging a secret	It only raises effort
Binary review	Search for embedded constants and comparison logic	Find accidental secret storage
Runtime review	Monitor `mmap`, decryption, and generated code	Detect unpacking and VM execution

Common Myths

"The code is encrypted, so nobody can read it."

The CPU cannot execute encrypted instructions forever.

At some point they become instructions.

That is the point where you stop asking the file what it contains and start asking memory.

"The password is only stored as a hash."

If the client contains enough information to decide whether your guess is correct, you may still be able to brute-force the local check or patch the decision.

A hash is not automatically a security boundary.

"Anti-debugging makes it impossible to reverse."

It makes debugging annoying.

Those are not the same sentence.

"A custom VM is basically encryption."

No.

A VM is an execution mechanism.

Unless the actual secret is kept outside the client, you are still protecting the same information with more moving parts.

Final Thoughts

The interesting lesson from this class of binary is not how to write a fancier password checker.

It is how quickly the apparent complexity collapses once you identify the right abstraction.

At the beginning, you have:

anti-debugging
encrypted functions
dynamic memory
custom bytecode
jump tables
scrambled input
arithmetic

At the end, you have:

ROL32
   |
   v
ROR32
   |
   v
four bytes

That is reverse engineering in miniature.

Do not fight the whole binary at once. Find the state transition that actually decides the answer, then work backward from there.

A client can hide the path.

It cannot hide the destination from the machine that has to reach it.

What Is Client-Side Password Validation, Anyway?

Obfuscation vs. Security

Why Should You Care?

What Does the Obfuscated Validator Actually Do?

Stage 1: Start With Runtime Behavior

Why ptrace() Matters

Stage 2: Find the Runtime-Decrypted Code

Dumping the Mapped Regions

Stage 3: Realizing You Are Looking at a VM

Finding the VM Bytecode

Build an Opcode Table

Stage 4: Follow the Password Bytes

Stage 5: Recover the Final Arithmetic

Reversing the Check

A Generic Solver

What If the Math Is More Complicated?

1. Reverse the operations manually

2. Use Z3

3. Emulate the VM

A Better Way to Debug the VM

Why the Custom VM Does Not Save the Design

What Would a Safer Design Look Like?

Testing and Audit Points

Defensive Design Checklist

Common Myths

"The code is encrypted, so nobody can read it."

"The password is only stored as a hash."

"Anti-debugging makes it impossible to reverse."

"A custom VM is basically encryption."

Final Thoughts

Why `ptrace()` Matters