Can a WinpkFilter application handle Gigabit traffic in user mode without a noticeable slowdown? I am asked this often, and my answer is usually “it depends.” If maximum performance matters, putting packet-processing code in the driver may be worth considering. Before choosing that route, I look at three factors:
- First, and the most important one, is CPU performance. A faster CPU helps because fetching a packet from the kernel, processing it and reinjecting it into the stack consumes CPU cycles
- Application architecture. Even a fast CPU cannot compensate for an application design that slows packet processing
- Network latency and TCP windows. Round-trip time (RTT) is the interval between sending data and receiving its acknowledgment. In an idealized steady state, one TCP flow is limited to roughly min(congestion window, advertised receive window) / RTT bytes per second when those windows are measured in bytes. Path capacity, packet loss, application behavior and processing overhead can lower throughput further. Filtering adds CPU work and may delay packet delivery or acknowledgments, so a slow processing path can reduce the throughput of a single session. See RFC 5681 for the congestion-window rules.
I therefore tested what a user-mode pass-through filter could handle on my two-workstation setup. The work also led to performance optimizations in the WinpkFilter 3.2.7 driver used for these 2016 measurements.
Prerequisites
Hardware
The test used two older mobile workstations with built-in Gigabit Ethernet, connected by UTP Cat 5e cables through a Gigabit home router:
- Core i7-2760QM 2.40 GHz 16GB RAM Windows 10 x64 Pro
- Core i7-Q720 1.60 GHz 16GB RAM Windows 10 x64 Pro
Software
I used iPerf3 to measure transfers between the hosts, including tests with multiple TCP sessions. NetCPS was another tool I used at the time, but its original download page is no longer available. For the filter, I needed a WinpkFilter pass-through application with basic counters. PassThru and PackThru demonstrate the API, but they print packet headers to the console, which adds substantial I/O overhead. I wrote the smaller PerfTest workload shown below to avoid that overhead.
The listing below adapts the linked PerfTest source for readability and safer counting. This adapted excerpt adds argument bounds checks and uses an aligned LONG64 with InterlockedIncrement64 for packet counting. The packet forwarding algorithm is unchanged. The performance table records the original 2016 run; I did not rerun it using this edited listing. Consult the repository for the original source. This excerpt is not a complete production application.
#include "stdafx.h"
//
// Maximum number of packets to fetch from the driver per single operation
//
#define MAX_PACKET_CHUNK 256
//
// Some global definitions
//
TCP_AdapterList g_AdList;
DWORD g_iIndex;
CNdisApi g_api;
HANDLE g_hEvent;
BOOLEAN g_bIsRunning = TRUE;
__declspec(align(8)) volatile LONG64 g_llPacketFiltered = 0;
volatile LONG g_dwReadOps = 0;
A dedicated worker thread reads and forwards packets instead of doing that work in main(). This also lets us compare one worker with two workers on the same interface. MAX_PACKET_CHUNK is the maximum number of packets requested in a single driver read. The bulk ReadPackets, SendPacketsToAdapter and SendPacketsToMstcp calls reduce user/kernel transitions compared with their single-packet counterparts. I chose a batch limit of 256 experimentally for this test; an application can tune its batch size for its workload.
//
// Worker routine. The command line selects the number of worker threads.
//
unsigned __stdcall WorkingThread(void * index)
{
PETH_M_REQUEST ReadRequest;
PETH_M_REQUEST ToMstcpRequest;
PETH_M_REQUEST ToAdapterRequest;
INTERMEDIATE_BUFFER PacketBuffer[MAX_PACKET_CHUNK];
unsigned dwMaxRead = 0;
ULONG_PTR dwThreadIndex = reinterpret_cast<ULONG_PTR>(index);
printf("Thread: %I64d started\n", (ULONGLONG)dwThreadIndex);
//
// Initialize Request
//
ReadRequest = (PETH_M_REQUEST)malloc(sizeof(ETH_M_REQUEST) +
sizeof(NDISRD_ETH_Packet)*(MAX_PACKET_CHUNK - 1));
ToMstcpRequest = (PETH_M_REQUEST)malloc(sizeof(ETH_M_REQUEST) +
sizeof(NDISRD_ETH_Packet)*(MAX_PACKET_CHUNK - 1));
ToAdapterRequest = (PETH_M_REQUEST)malloc(sizeof(ETH_M_REQUEST) +
sizeof(NDISRD_ETH_Packet)*(MAX_PACKET_CHUNK - 1));
ZeroMemory(ReadRequest, sizeof(ETH_M_REQUEST) +
sizeof(NDISRD_ETH_Packet)*(MAX_PACKET_CHUNK - 1));
ZeroMemory(ToMstcpRequest, sizeof(ETH_M_REQUEST) +
sizeof(NDISRD_ETH_Packet)*(MAX_PACKET_CHUNK - 1));
ZeroMemory(ToAdapterRequest, sizeof(ETH_M_REQUEST) +
sizeof(NDISRD_ETH_Packet)*(MAX_PACKET_CHUNK - 1));
ZeroMemory(&PacketBuffer, sizeof(INTERMEDIATE_BUFFER)*MAX_PACKET_CHUNK);
ReadRequest->hAdapterHandle = (HANDLE)g_AdList.m_nAdapterHandle[g_iIndex];
ToMstcpRequest->hAdapterHandle = (HANDLE)g_AdList.m_nAdapterHandle[g_iIndex];
ToAdapterRequest->hAdapterHandle = (HANDLE)g_AdList.m_nAdapterHandle[g_iIndex];
ReadRequest->dwPacketsNumber = MAX_PACKET_CHUNK;
for (unsigned i = 0; i < MAX_PACKET_CHUNK; ++i)
{
ReadRequest->EthPacket[i].Buffer = &PacketBuffer[i];
}
while (g_bIsRunning)
{
WaitForSingleObject(g_hEvent, INFINITE);
// Reset event, as we don't need to wake up all working threads at once
if (g_bIsRunning)
ResetEvent(g_hEvent);
// Start reading packet from the driver
while (g_api.ReadPackets(ReadRequest))
{
InterlockedIncrement(&g_dwReadOps);
if (ReadRequest->dwPacketsSuccess > dwMaxRead)
{
dwMaxRead = ReadRequest->dwPacketsSuccess;
printf(
"Thread: %I64d New read operation maximum %d packets from driver\n",
(ULONGLONG)dwThreadIndex,
dwMaxRead
);
}
for (unsigned i = 0; i < ReadRequest->dwPacketsSuccess; ++i)
{
InterlockedIncrement64(&g_llPacketFiltered);
if (PacketBuffer[i].m_dwDeviceFlags == PACKET_FLAG_ON_SEND)
{
ToAdapterRequest->EthPacket[ToAdapterRequest->dwPacketsNumber].Buffer = &PacketBuffer[i];
ToAdapterRequest->dwPacketsNumber++;
}
else
{
ToMstcpRequest->EthPacket[ToMstcpRequest->dwPacketsNumber].Buffer = &PacketBuffer[i];
ToMstcpRequest->dwPacketsNumber++;
}
}
if (ToAdapterRequest->dwPacketsNumber)
{
g_api.SendPacketsToAdapter(ToAdapterRequest);
ToAdapterRequest->dwPacketsNumber = 0;
}
if (ToMstcpRequest->dwPacketsNumber)
{
g_api.SendPacketsToMstcp(ToMstcpRequest);
ToMstcpRequest->dwPacketsNumber = 0;
}
ReadRequest->dwPacketsSuccess = 0;
}
}
free(ReadRequest);
free(ToMstcpRequest);
free(ToAdapterRequest);
return 0;
}Each worker reads a batch, separates incoming from outgoing packets, and reinjects them into the appropriate path. It also counts packets and ReadPackets calls to report an average batch size.
int main(int argc, char* argv[])
{
if (argc < 3)
{
printf ("Command line syntax:\
\n\tPerfTest.exe index threads\
\n\tindex - network interface index.\
\n\tthreads - number of working threads.\
\n\tYou can use ListAdapters to determine correct index.\n");
return 0;
}
int requestedIndex = atoi(argv[1]);
int requestedThreads = atoi(argv[2]);
if (requestedIndex < 1 || requestedThreads < 1 || requestedThreads > MAXIMUM_WAIT_OBJECTS)
{
printf("Provide a positive adapter index and 1 to 64 threads.\n");
return 1;
}
g_iIndex = requestedIndex - 1;
unsigned concurentThreadsSupported = requestedThreads;
if(!g_api.IsDriverLoaded())
{
printf ("Driver not installed on this system of failed to load.\n");
return 0;
}
g_api.GetTcpipBoundAdaptersInfo ( &g_AdList );
if ( g_iIndex + 1 > g_AdList.m_nAdapterCount )
{
printf("There is no network interface with such index on this system.\n");
return 0;
}
ADAPTER_MODE Mode;
Mode.dwFlags = MSTCP_FLAG_SENT_TUNNEL|MSTCP_FLAG_RECV_TUNNEL;
Mode.hAdapterHandle = (HANDLE)g_AdList.m_nAdapterHandle[g_iIndex];
// Create notification event
g_hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
// Set event for helper driver
if ((!g_hEvent) ||
(!g_api.SetPacketEvent((HANDLE)g_AdList.m_nAdapterHandle[g_iIndex], g_hEvent)))
{
printf ("Failed to create notification event or set it for driver.\n");
return 0;
}
g_api.SetAdapterMode(&Mode);
HANDLE* phThreads = new HANDLE[concurentThreadsSupported];
for (unsigned i = 0; i < concurentThreadsSupported; ++i)
{
phThreads[i] =
(HANDLE)_beginthreadex (
NULL,
0,
WorkingThread,
reinterpret_cast<void*>((ULONG_PTR)i),
0,
NULL
);
}
printf("Press any key to stop packet filtering... \n");
_getch();
g_bIsRunning = FALSE;
SetEvent(g_hEvent);
WaitForMultipleObjects(concurentThreadsSupported, phThreads, TRUE, INFINITE);
for (unsigned i = 0; i < concurentThreadsSupported; ++i)
{
CloseHandle(phThreads[i]);
}
printf(
"Filtered %I64d packets read in %d operations. Packets per read average:%I64d \n ",
g_llPacketFiltered,
g_dwReadOps,
g_dwReadOps ? g_llPacketFiltered / g_dwReadOps : 0
);
//
// Although we exit application and all resources will be cleaned up
// automatically let's release everything before we exit
//
Mode.dwFlags = 0;
Mode.hAdapterHandle = (HANDLE)g_AdList.m_nAdapterHandle[g_iIndex];
// Set NULL event to release previously set event object
g_api.SetPacketEvent(g_AdList.m_nAdapterHandle[g_iIndex], NULL);
// Close Event
if (g_hEvent)
CloseHandle(g_hEvent);
// Set default adapter mode
g_api.SetAdapterMode(&Mode);
// Empty adapter packets queue
g_api.FlushAdapterPacketQueue(g_AdList.m_nAdapterHandle[g_iIndex]);
return 0;
}The console program accepts an interface index and a worker count, starts the requested workers and waits for termination. For the historical comparison I used WinpkFilter 3.2.7, the release available for this 2016 test; its single-session TCP performance had been improved.
Performance tests
I ran iPerf3 server mode on both workstations, then used the other machine as client to measure incoming and outgoing throughput with one and 16 simultaneous TCP sessions. I repeated those transfers under these configurations:
- WinpkFilter driver not installed
- WinpkFilter installed but not used
- WinpkFilter installed and perftest.exe (utility we created above) is started on Ethernet interface having single packet processing thread
- WinpkFilter installed and perftest.exe (utility we created above) is started on Ethernet interface having two packet processing threads.
| Configuration | Receive 1 TCP stream (Mbits/sec) | Send 1 TCP stream (Mbits/sec) | Receive 16 TCP streams (Mbits/sec) | Send 16 TCP streams (Mbits/sec) |
|---|---|---|---|---|
| NOT INSTALLED | 939 | 938 | 940 | 939 |
| INSTALLED | 941 | 940 | 940 | 933 |
| PERFTEST 1 THREAD | 937 | 935 | 934 | 939 |
| PERFTEST 2 THREADS | 648 | 880 | 923 | 934 |
Conclusion
In this 2016 test, one PerfTest worker stayed near the roughly 940 Mbit/s baseline in all four measurements. Two workers did not: receive throughput for one TCP stream fell to 648 Mbit/s and send throughput fell to 880 Mbit/s. More workers were not automatically faster on this setup. CPU load was 17-20% with perftest.exe running and 7-8% without it. For this pass-through workload, the results gave me no reason to move the processing into a kernel driver. The second worker sometimes changed packet order within a TCP stream, producing duplicate ACKs and retransmissions. If multiple workers or another processing path can reorder packets from one interface, the application must preserve the order when reinjecting them.
These figures describe the two Windows 10 workstations, WinpkFilter 3.2.7 and the test configuration above; they are not a current throughput guarantee. The listing illustrates the 2016 benchmark and still needs ordinary production checks for allocation, API errors and thread creation.
PerfTest project source code is available on GitHub