Unhelpful IP Helper: A Handle Leak Story

After my post about a CancelIoEx bug, I decided to cover another defect in Windows system components—this time in IP Helper (the part of the Windows API responsible for network statistics and connection tables).

Among other things, this API lets you map packets intercepted at the network level to local processes. You’d think it’s a battle-tested mechanism running under the hood of many tools and network filters. But while testing WireSock Secure Connect in process-based split tunneling mode, we hit a leak that can exhaust the system’s handle limit in just a few minutes.

The investigation started with a message in our WireSock Telegram support group: one user noticed abnormally fast growth in the number of open process handles. The problem reproduced reliably under load and disappeared if we switched to IP-based filtering. That was the first clue that eventually led us to a bug in IP Helper’s implementation.

Special thanks to @dno5iq, who spotted the issue, reversed GetOwnerModuleFromPidAndInfo, and helped confirm the defect in its implementation.

Getting to the root cause

We debugged it right in chat, as a mini-investigation. It quickly became clear that handle leaks occurred only in process-based split tunneling; with IP-based filtering (AllowedIPs/DisallowedIPs), everything worked fine. That ruled out the networking logic and let us focus on the code that resolves a process name by PID.

In WireSock, mapping an intercepted packet to a local process is done via the connection tables you can obtain through IP Helper. We use that API to derive the executable module name from a PID.

WireSock itself doesn’t open process handles directly for this. However, the IP Helper functions GetOwnerModuleFromTcpEntry, GetOwnerModuleFromTcp6Entry, GetOwnerModuleFromUdpEntry, and GetOwnerModuleFromUdp6Entry can internally open a handle to the owning process to retrieve module info. Those handles aren’t returned to the caller—Windows is supposed to close them internally. Under load, something clearly went wrong.

All of these functions call the same internal routine – GetOwnerModuleFromPidAndInfo. After decompiling it, we found that it indeed opens a process handle and then performs two checks:

1. Whether the handle was obtained successfully.
2. Whether GetModuleFileNameExW succeeds in retrieving the path to the executable.

If the first check passes (e.g., the process has technically exited but still lingers thanks to remaining references) and the second fails, the function returns without closing the previously opened handle. Under load, that turns into a steadily growing leak.

Simplified pseudocode

HANDLE hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, pid);

if (hProcess) {
        // Пытаемся получить путь к модулю процесса
        if (!GetModuleFileNameExW(hProcess, NULL, pathBuffer, bufferSize)) {
            // Выход без CloseHandle — утечка гарантирована
            return ERROR;
        }
     // Дальнейшая работа с именем процесса
     CloseHandle(hProcess);
}

Our workaround

Once it was clear the leak lived inside IP Helper, expecting a quick Microsoft fix seemed optimistic. Bugs like this can live for years, and even if a patch arrives, it will only cover supported Windows versions. For some users, that’s moot – many still aren’t rushing to leave Windows 7.

As a temporary (practically permanent) solution, I rewrote the process resolver so we no longer rely on GetOwnerModuleFromPidAndInfo to obtain the process name. Now:

• We open and close handles explicitly in our code.
• We added request caching to speed up frequent lookups.
• Performance in our scenario is actually better than with the original IP Helper path.

The source for the new resolver is in the ProxiFyre repository and is used in both ProxiFyre and WireSock Secure Connect. These changes significantly improved stability and responsiveness in high-churn networking scenarios where connections are constantly created and torn down. Many apps fit this pattern, but the effect is especially noticeable with torrent clients: faster peer establishment, fewer disconnects, and higher overall throughput during file sharing.

The fix has been included in WireSock Secure Connect since v2.4.18. Early builds and discussion are available in our Telegram support group: https://t.me/wiresock.

Takeaways

Even system APIs that developers rely on can harbor defects that show up only under specific conditions. Finding them requires reproducible load tests, isolating suspicious areas, and, if necessary, digging into internal implementations.

In this case, we sidestepped the bug by replacing the problematic call with our own implementation—eliminating the leak and making the application faster and more reliable.