Key takeaways
- A packer wraps malware in a small stub that decrypts the real payload into memory at runtime, leaving static scanners with no signature to match.
- The stub's VirtualAlloc call requesting RWX memory is the first and most reliable behavioral indicator a packer produces.
- Catching the original entry point and dumping process memory produces a clean executable that can be analysed and used to write YARA signatures.
- Import reconstruction with Scylla, an open-source IAT repair tool, restores the API calls the packed binary masked and makes the payload's capabilities visible.
- An adversary simulation that exercises packed payloads tells you whether your EDR detects the allocation pattern before the payload executes, not only whether it recognises the file on disk.
What a packer does to a binary
A static scanner works by reading a file on disk and comparing its bytes against a database of known patterns. A packer defeats this by making the file contain almost nothing a scanner can read. The original executable, the payload, is compressed or encrypted and stored as a data blob inside the packer's own binary. What sits on disk is a small decryption stub and a block of high-entropy bytes. The stub carries no payload code, so no signature matches it.
Entropy is the most visible indicator. A standard Windows executable has code sections at moderate entropy and string sections at low entropy. A packed binary inverts that: its data section reads near entropy 8.0 because encrypted bytes resemble random bytes. Tools like Detect It Easy score entropy directly and name the packer family when they recognise the stub. Software packing is documented under MITRE ATT&CK T1027.002 as a defense evasion sub-technique and is used by both commodity malware and advanced operators. During a red team engagement, a packed implant routinely passes an email gateway and an endpoint's on-access scanner without modification, because neither control reads the encrypted payload that the stub will unpack at runtime.
The stub: allocate, decrypt and transfer control
When the packed binary runs, the stub executes first. Its first task is memory: it calls VirtualAlloc with MEM_COMMIT | MEM_RESERVE and sets protection to PAGE_EXECUTE_READWRITE. That single call, requesting a block of memory that is simultaneously writable and executable, is the clearest behavioral indicator a packer produces. A well-configured endpoint should alert before any payload code runs. A standard penetration test that delivers only known-signature tools never reaches this moment and never measures whether that alert fires.
Once memory is allocated, the stub reads its encrypted data blob and writes the decrypted payload bytes into that region. A simple XOR loop suffices for commodity packers; more capable implementations chain multiple transforms or derive a key from an environment variable to complicate sandbox emulation. When the stub finishes writing, it transfers execution to the payload's original entry point, the OEP, which is the first instruction of the real program. From that point, the payload runs as if no packing had occurred. The on-disk file and the running process no longer match. That is the state an assumed breach exercise begins from when the operator delivers a post-exploitation implant that has already evaded the perimeter.
Catching the original entry point under a debugger
Finding the OEP means catching the moment the stub hands execution to the payload. The method is to let the stub do its own work and set a trap at the transfer point. In WinDbg, a breakpoint on VirtualAlloc captures the allocation call. Recording the returned address and placing a hardware execute breakpoint on it stops execution at the OEP before the payload's initialisation code has run. The code block below shows the sequence.
x64dbg, with the Scylla plugin loaded, offers the same capture through 'Find OEP by Section Hop', which steps over the stub's own code section in one action. Either method gives the analyst an intact snapshot of the payload at its entry point. The adversary simulation exercise applies the same logic in reverse: the operator confirms that the endpoint's behavioral controls fire on the VirtualAlloc call, before the payload's first instruction executes, rather than only on static file inspection at the gate. A control that fires after execution has transferred to the OEP has already lost the window.
; Set breakpoint on VirtualAlloc and run
0:000> bp kernelbase!VirtualAlloc
0:000> g
Breakpoint 0 hit
kernelbase!VirtualAlloc:
00007ffa`3e2a1d80 mov [rsp+8], rbx
; Step out to caller and read the returned base address from rax
0:000> gu
0:000> r rax
rax=0000027a`8c400000
; Place a hardware execute breakpoint on the allocated region
0:000> ba e1 0000027a`8c400000
0:000> g
Breakpoint 1 hit: execution has transferred to the unpacked payload at the OEP
Dumping the image and recovering the import table
Once stopped at the OEP, the analyst dumps the running process. Microsoft's ProcDump captures all committed memory regions in a minidump. PE-sieve, a purpose-built scanner from hasherezade, walks the running process's memory looking for PE images that differ from their on-disk counterparts and exports them directly. The exported image is the payload, but its import address table may still point into the stub's in-memory space rather than into resolved Windows API addresses.
Scylla, an open-source import address table reconstructor maintained under the NtQuery project, repairs this. It reads the OEP, walks the in-memory IAT to identify which Windows API each entry resolves to, and writes a corrected import directory into the dumped binary. The result is a fully-linked executable that a disassembler can load without stub noise. Behavioral analysis of this kind is within the technical security testing scope described by NIST SP 800-115, which defines host-based examination as a core element of technical security assessments.
# Full memory dump while paused at the OEP
procdump.exe -ma <pid> C:\analysis\dump.dmp
# Extract the packed PE directly from the running process
pe-sieve.exe /pid <pid> /out C:\analysis\unpacked\
# Output: C:\analysis\unpacked\<pid>_dump.exe (the extracted payload image)
# Run Scylla from x64dbg plugin menu to repair the import address table:
# Plugins > Scylla > IAT Autosearch > Get Imports > Fix Dump| Tool | Role | Where to obtain |
|---|---|---|
| WinDbg / x64dbg | Attach debugger, set breakpoints on VirtualAlloc, capture OEP | Microsoft Store / x64dbg.com |
| PE-sieve | Scan running process and export PE regions that differ from disk | hasherezade / GitHub |
| ProcDump | Full process memory dump when PE-sieve extraction is incomplete | Sysinternals (Microsoft) |
| Scylla | Walk in-memory IAT, identify resolved API calls, write corrected import directory | NtQuery / GitHub |
Writing detections from the unpacked sample
An unpacked sample is readable. Strings that the packer scrambled are visible in a disassembler. API calls that resolved through the stub's loader appear in the corrected import table. A YARA rule written against the unpacked binary's code section can be applied to future variants of the same family by scanning process memory at runtime, and to other artefacts the payload dropped during execution. Static rules written against the packed file catch only that exact build; rules written against the unpacked code section survive repacking.
Behavioral signatures cover what static rules cannot. The sequence that catches any packer family is a process allocating RWX memory from a non-image-backed region, writing into it, and then transferring execution from a different base address into that region. An EDR or XDR platform that records and alerts on this sequence catches a packer before the payload runs, regardless of what the payload is. When evaluating endpoint detection platforms, ask specifically whether the product alerts on cross-region execute transfers following an RWX allocation from a non-image-backed region, and ask for a demonstration against a benign packer stub rather than a product slide. A purple teaming session that exercises this allocation pattern against your actual deployment gives the SOC a chance to tune the detection rule before a live adversary simulation runs the same sequence against production.
Observe
Entropy check
DIE or PEStudio flags near-8.0 entropy data section
Header inspection
Import table lists one or two API calls only
Unpack
Attach debugger
Break on VirtualAlloc, record returned base address
Catch OEP
Hardware execute breakpoint on allocated region
Recover and detect
Dump and rebuild
ProcDump captures memory; Scylla corrects the IAT
Write rules
YARA against unpacked code bytes; behavioral rule on RWX allocation chain
Adversary simulation and packer detection in the UAE
Adversary simulation engagements in the UAE routinely include packed payloads, because static-only endpoint controls remain common across financial, government and healthcare sectors in both Abu Dhabi and Dubai. Under CBUAE supervision, financial institutions operating in the UAE are expected to demonstrate that their security controls detect threats that evade signature-based detection. An engagement that delivers only known-signature test tools does not satisfy that expectation. One that delivers a custom-packed implant and measures whether the endpoint fires before the OEP is reached satisfies it, because it tests the specific control layer that commodity and targeted malware both depend on bypassing.
DESC, which sets security requirements for Dubai government entities and their technology suppliers, requires technical controls to be tested against realistic attack patterns rather than theoretical ones. A packed executable is realistic: post-exploitation frameworks and commodity malware both use it as a standard delivery format. RedTeam Security, licensed in Dubai Silicon Oasis, builds packed payload exercises into adversary simulation scopes that align with EU Red Teaming standards such as TIBER-EU and ART. Organisations in Abu Dhabi and Dubai that complete this type of engagement gain a measured answer to whether their EDR detects in-memory execution before the payload runs, rather than relying on a product claim that was never tested against a real operator technique.
Frequently asked questions
What is software packing in the context of malware?
Software packing is a technique where a program's real code is compressed or encrypted and stored inside a wrapper binary. A small stub handles the decryption at runtime and transfers execution to the recovered code. It is documented under MITRE ATT&CK T1027.002 as a defense evasion sub-technique and is used by both commodity malware and advanced operators to evade static signature detection at the file layer.
How does a software packer differ from a standard file compressor?
A standard compressor like UPX reduces file size and produces well-known byte patterns; most antivirus engines unpack and scan it automatically. A malware packer is designed so that the payload cannot be read without executing the stub, and the stub itself carries no recognisable payload bytes. The difference is intent: compression optimises file size, packing optimises detection evasion.
What is the original entry point in a packed binary?
The original entry point, or OEP, is the first instruction of the real unpacked payload: the address execution reaches after the stub finishes decrypting. It is distinct from the binary's nominal entry point, which points into the stub. Identifying the OEP is the central task in manual unpacking, because it is the moment at which the full payload sits in memory and can be captured cleanly.
Why do static antivirus scanners miss packed malware?
Static scanners match byte patterns in a file on disk against a signature database. A packed binary stores the real payload encrypted; on-disk bytes are randomised by the encryption and match no known pattern. The scanner sees only the stub, which is either unknown or too small to match, and passes the file. The payload appears in memory only after execution, which static analysis never reaches.
What does an RWX memory allocation indicate during runtime analysis?
A call to VirtualAlloc that requests PAGE_EXECUTE_READWRITE creates a memory region that can be written and then executed, which is the mechanism a packing stub uses to load its payload. Legitimate applications rarely need this combination. It is a strong behavioral indicator of packing, shellcode staging or code injection, and an EDR that monitors and alerts on this allocation pattern can interrupt the chain before the payload runs.
What is Scylla and how does it help with import table reconstruction?
Scylla is an open-source import address table reconstructor used in manual malware unpacking. When a packing stub decrypts a payload and transfers control to the OEP, the payload's IAT entries may still point into the stub's memory space rather than into resolved Windows API addresses. Scylla reads the OEP, identifies which API each IAT entry resolves to, and writes a corrected import directory into the dumped binary, producing a clean PE that a disassembler can load and analyse normally.
How are YARA rules written for unpacked malware effective against repacked variants?
YARA rules targeting the unpacked code section scan process memory at runtime rather than files at rest, catching the payload the moment it is decrypted regardless of what outer packing layer was applied. Distinctive bytes from the payload's initialisation routine or characteristic API call sequences are reliable anchors because they change slowly across variants, unlike the encrypted outer blob which changes with every repack.
How does adversary simulation use software packing as a test case?
An adversary simulation operator that delivers a packed implant is testing whether the organisation's endpoint controls detect the packing behavior, specifically the RWX memory allocation and the cross-region execute transfer, before the payload runs. A detection that fires only on the payload's content after decryption has already lost the window. The simulation measures whether the control fires at the allocation stage, which is the only point at which the chain can be stopped reliably.
What should I ask an EDR vendor about packed malware detection?
Ask whether the product records process memory allocations and whether it alerts when a process allocates RWX memory from a non-image-backed region and then executes into it from a different base address. Ask for a demonstration against a benign packer stub rather than a reference to a product capability sheet. A vendor that cannot demonstrate the detection in a controlled environment is not answering the question you need answered.
