Mobile App Security

DEX Virtualization: Turning Bytecode into a Custom VM

Android apps ship as DEX bytecode — a public, fully documented format that decompilers like jadx-gui turn back into readable source in seconds. This article shows exactly why that works, proves it against Google’s own spec, and explains how DEX virtualization breaks the assumption by replacing the standard instruction set with a private one only your own interpreter understands.

By · · 5 Min

DEX Virtualization: Turning Bytecode into a Custom VM

Reverse Engineering Is a First-Class Attack Surface

Every Android app you publish also ships everything needed to understand it. An attacker pulls the APK, loads the compiled code into a decompiler, and within seconds is reading a reconstruction of your source — logic, string constants, endpoints, and the exact location of security checks. From there they repackage, clone, lift proprietary algorithms, or neutralize the controls meant to stop them. With Android banking-trojan attacks up 56% year over year (Kaspersky, 2025), that readability is a direct fraud and IP risk. The practical question is simply: why is a shipped app so easy to read back into source, and what can be done about it? To answer it, we start with DEX.

02 — FOUNDATIONSWhat a DEX File Is, and How Your Code Becomes One

DEX stands for Dalvik Executable — the bytecode format Android apps compile into. It is the Android counterpart of a Java .class file, but redesigned for mobile: a single classes.dex holds all your classes in one compact, shared-constant structure.

From source to DEX: the toolchain

How the device runs DEX: ART

On the device, DEX is executed by ART (the Android Runtime), a hybrid that ahead-of-time (AOT) compiles via dex2oat, interprets what isn’t precompiled, and uses a JIT to promote hot methods. The part that matters here is the interpreter’s decode-dispatch loop: read the next instruction, decode its opcode, jump to that opcode’s handler, execute, advance, repeat. A custom DEX virtual machine is, in essence, your own private copy of exactly this loop.

Opcodes, operands, and a lot of metadata

A DEX file is a structured container: a header and map list, a string pool, type/prototype/ field/method tables, class definitions, and the data section holding method bytecode. Each instruction is an opcode (what to do) plus its operands (which registers, literal, or table index to act on). Dalvik defines roughly 230 opcodes, and every one is documented publicly. Note too that Dalvik is register-based (operands are registers like v0, v1), unlike the stack-based JVM.

Example A — the simplest method

Calculator.java

public int add(int a, int b) {
    return a + b;
}

add() — DEX disassembly (smali)

    add-int v0, p1, p2   # v0 = a + b
    return v0

add-int is the opcode; v0, p1, p2 are its operands (destination, then two sources). That is the entire method. Load the APK into jadx-gui and it hands back essentially the original Java. Simple methods reconstruct perfectly — but so do complex ones.

Example B — a medium-complex method

Now something with real logic: a multiply, a constant, a conditional branch, a division by a literal, and a subtraction.

Pricing.java

public int computeDiscount(int price, int qty) {
    int total = price * qty;
    if (total > 1000) {
        total = total - (total / 10);   // 10% off large orders
    }
    return total;
}

computeDiscount() — DEX disassembly (smali)

.method public computeDiscount(II)I
    .registers 5        # locals v0,v1 ; params p0=this=v2, p1=price=v3, p2=qty=v4

    mul-int        v0, p1, p2    # v0 = price * qty       (total)
    const/16       v1, 0x3e8      # v1 = 1000
    if-le          v0, v1, :cond_0  # if total <= 1000, skip discount
    div-int/lit8   v1, v0, 0xa    # v1 = total / 10
    sub-int        v0, v0, v1    # total = total - v1
    :cond_0
    return         v0
.end method

Six instructions, six different opcodes, and a mix of operand kinds — registers (v0, v1), literals (0x3e8, 0xa), and a branch target (:cond_0). This is representative of the instruction density in real methods.

Proof the instruction set is public

Take just the first instruction, mul-int v0, p1, p2. With .registers 5, that is mul-int v0, v3, v4. In the compiled classes.dex, it is a run of raw bytes. Now compare those bytes to the entry in Google’s official Dalvik bytecode reference:

In your compiled app — classes.dex

92 00 03 04
└─ opcode 0x92
   dest v0, src v3, src v4

In the public spec — source.android.com

92 23x  mul-int vAA, vBB, vCC
  A: destination register
  B: first source register
  C: second source register
  Multiply the two source
  registers into the dest.

The same byte, 0x92, means the same thing in your app and in Google’s documentation. The mapping isn’t secret — it’s a standard.

This is the root of the problem: anyone — including automated tooling — can look up every opcode in your app, because it’s the same public table the runtime itself uses.

03 — THE TOOLINGHow jadx-gui Reconstructs Your Source

jadx doesn’t “crack” anything — it reads the file the way the runtime does, then works upward to Java. Briefly, it:

Run it against Example B and the output is essentially the original method back again:

jadx-gui — decompiled output for computeDiscount()

public int computeDiscount(int price, int qty) {
    int total = price * qty;
    if (total > 1000) {
        total -= total / 10;
    }
    return total;
}

The gut punch: from a shipped, compiled APK, a free tool reconstructed the logic — multiplication, the threshold check, the discount — not a fragment, the whole method. (Without debug info the local names may become i, i2, but the behavior is fully recovered.)

The insight that sets up the fix

Decompilation is easy because the instruction set is public and semantically transparent. If a method’s instructions were instead private and semantically opaque — encoded in a scheme no off-the-shelf tool has ever seen — jadx would have nothing to decode against. That is exactly what code virtualization does.

04 — THE SOLUTIONDEX Virtualization: Turning Bytecode into a Custom VM

Code virtualization is a protection technique borrowed from desktop software protectors (VMProtect, Themida) and adapted to Android. Instead of shipping a method as standard DEX instructions any decompiler understands, you translate it into a custom instruction set — a private bytecode of your own design — and bundle a small custom interpreter (a virtual machine) inside the app that knows how to execute it. The original method body is replaced by a stub that hands control to your VM.

That is the idea in one sentence. But how does a Dalvik method actually become something jadx can no longer read? Let’s walk it through, step by step, using the same computeDiscount method from before.

Step 1 — Design a VM and define your own opcodes

There is no law that says logic must be expressed in Dalvik opcodes. The only component that needs to understand your encoding is the interpreter you ship. So you design a small virtual machine and invent your own instruction set from scratch: you choose the mnemonics, you choose the byte values, and you write a handler for each one. None of these values exist in the Dalvik specification — they are private to your VM.

For our discount method, a private instruction set might look like this:

Your private opcode set — illustrative; you choose the bytes and the meaning

0x2C  VMUL         r[A] = r[B] * r[C]
0x3F  VCONST       r[A] = imm16
0x47  VCMP_LE_JMP  if r[A] <= r[B] then jump to offset
0x52  VDIV_IMM     r[A] = r[B] / imm8
0x63  VSUB         r[A] = r[B] - r[C]
0x7E  VRET         return r[A]

Crucially, there is no public document mapping 0x2C to “multiply.” That mapping lives only inside your interpreter. This is the exact opposite of the situation we proved earlier, where 0x92 was published in Google’s spec for anyone to look up.

Step 2 — Translate the method into your custom bytecode

Next, the build pipeline takes the original DEX instructions of computeDiscount and re-encodes the same logic into your private opcodes. Each Dalvik instruction is expressed using your VM’s operations, producing an opaque byte array:

computeDiscount() re-encoded as your custom VM bytecode (program blob)

[00] 2C 00 03 04   ; VMUL        r0 = r3 * r4      (price * qty)
[04] 3F 01 E8 03   ; VCONST      r1 = 1000
[08] 47 00 01 14   ; VCMP_LE_JMP if r0 <= r1 jump to [14] (VRET)
[0C] 52 01 00 0A   ; VDIV_IMM    r1 = r0 / 10
[10] 63 00 00 01   ; VSUB        r0 = r0 - r1
[14] 7E 00         ; VRET        return r0

The original method body is then stripped out and replaced with a small stub whose only job is to hand this blob (plus the incoming arguments) to your VM. The discount algorithm no longer exists anywhere as Dalvik instructions — it exists only as this array of private bytes.

Step 3 — What jadx-gui sees now

Reopen the protected APK in jadx-gui and decompile the same method. Because the body is now just a call into the VM, there is no arithmetic, no threshold, no discount left to reconstruct. What comes back is roughly:

jadx-gui — decompiled output AFTER virtualization

public int computeDiscount(int price, int qty) {
    return VmRuntime.run(PROGRAM_7A3F, new int[]{ price, qty });
}

(If the interpreter is implemented in native code — the common choice for extra stealth — jadx shows even less: a bare native method declaration whose body lives inside a .so library.) Either way, the byte array it references is just data. jadx has no opcode table for 0x2C, 0x52, and the rest, so it cannot turn those bytes back into logic. Compare this to the pre-virtualization output, where the entire method reappeared.

Same method, same behavior on the device — but the logic has vanished from the decompiler’s view. That is the whole point of virtualization.

Step 4 — How the VM executes your instruction set

At runtime, control reaches the stub, which enters your interpreter. The VM runs its own decode-dispatch loop — the same pattern ART’s interpreter uses, but over your opcodes and your virtual registers, driven by your handler table:

Inside the VM — a representative decode-dispatch loop

int vm_run(const uint8_t *prog, int *args) {
    int32_t r[16];                 // the VM's own virtual registers
    r[3] = args[0]; r[4] = args[1];// load price, qty
    size_t vip = 0;                // virtual instruction pointer

    for (;;) {
        uint8_t op = prog[vip++];  // fetch YOUR opcode
        switch (op) {            // YOUR handler table
            case 0x2C: { int a=prog[vip++],b=prog[vip++],c=prog[vip++]; r[a]=r[b]*r[c]; break; } // VMUL
            case 0x3F: { int a=prog[vip++]; int imm=prog[vip]|(prog[vip+1]<<8); vip+=2; r[a]=imm; break; } // VCONST
            case 0x47: { int a=prog[vip++],b=prog[vip++],off=prog[vip++]; if(r[a]<=r[b]) vip=off; break; } // VCMP_LE_JMP
            case 0x52: { int a=prog[vip++],b=prog[vip++],imm=prog[vip++]; r[a]=r[b]/imm; break; } // VDIV_IMM
            case 0x63: { int a=prog[vip++],b=prog[vip++],c=prog[vip++]; r[a]=r[b]-r[c]; break; } // VSUB
            case 0x7E: { int a=prog[vip++]; return r[a]; }                              // VRET
        }
    }
}

Trace it against the blob from Step 2 and it reproduces the original method exactly: VMUL computes price * qty, VCONST loads 1000, VCMP_LE_JMP jumps past the discount when the total is small, otherwise VDIV_IMM and VSUB apply the 10% cut, and VRET returns the result.

Here is the key mental shift: ART never executes the discount logic as recognizable instructions. To ART, this is just an ordinary function running a loop. The real algorithm lives in two pieces — the data (your private byte array) and the handlers (the switch cases inside your VM). Neither is meaningful without the other, and neither corresponds to anything in the public Dalvik instruction set. To recover the logic, an attacker must reverse-engineer the interpreter itself and work out what every custom opcode does — by hand.

Why this defeats decompilation

The trade-offs (an honest accounting)

Virtualization is powerful but not free, which is why it is applied selectively:

The point in one line

Virtualization removes the shared, public knowledge that decompilers depend on — turning a one-second automated decompile into days or weeks of specialist manual work. For the handful of methods that matter most, that shift in cost is exactly the goal.