Case study: Fraps
One of the topics that I often find myself bluffing through on GameDev is Direct3D hooking. In particular, how to display an overlay of your own on the window of another Direct3D program, often a commercial game. It’s pretty clear that the simplest method would involve somehow hooking the call to IDirect3DDevice8/9/10::Present, but the details are a little sketchy, particularly when you throw anti-hack systems into the mix. To be quite honest, I wasn’t sure I’d have been able to write a scalable hook that wouldn’t cause any incompatibilities – at least not without doing some very immoral things. So when I found out that Fraps has been doing exactly this for years, and that it somehow manages to avoid angering PunkBuster and other such systems, I decided to investigate.
What is Fraps?
Simply put, it’s a profiling and video-capture tool for PC games. As well as providing the ability to capture a video stream from any Direct3D 8/9/10, DirectDraw or OpenGL program, it can display real-time performance statistics (frame-rate and such) by means of an overlay on the game’s window (or full-screen display).
Remarkably, Fraps seems to handle the hacky side of all this automatically, with no fuss. It will dig its claws into any compatible game, whether it was started before or after Fraps. Yet if you investigate the state of DirectX and OpenGL’s DLLs on disk (in system32), they remain untouched at all times.
So how does it work?
Well it could be a whole lot worse, but I wasn’t thrilled to find out that every process running on my system had an instance of Fraps.dll loaded. It’s not so much the 106kB footprint that bothers me, but the performance and stability concerns. Anyway, my suspicions that this DLL was ‘infecting’ all processes via a system-wide hook were soon confirmed when OllyDbg caught Fraps.exe making a call to SetWindowsHookEx.
SetWindowsHookEx(WH_CBT, &FrapsProcCBT, hModuleFrapsDLL, 0);
So in one fell swoop, Fraps guarantees that Windows will load a copy of Fraps.dll into every process currently running, as well as those created in future. Moreover, the function FrapsProcCBT will be called by that process each time it attempts to create, destroy, show, hide, move or resize a window. Now this may seem like the perfect solution to a difficult problem, but it’s rather wasteful considering that most processes run window-driven interfaces, and only very few involve DirectX or OpenGL.
How should it work?
Now I haven’t tested this, but a much cleaner way to achieve the same goal would be for Fraps.exe to periodically poll EnumProcesses and EnumProcessModules, so as to determine the processes that actually need hooking. Installing the hooks into these specific processes would require no more work but would save the OS some effort and limit the worst-case-scenario disaster-zone to DirectX and OpenGL applications, which is a considerably smaller domain than almost everything. In Fraps’s defence, the code makes extensive use of IsBadReadPtr and suchlike, and I’ve never heard of it causing any trouble, but nevertheless, the best way to prevent your DLL from crashing someone else’s program is to make sure it never gets loaded.
What does the hook do?
All events but window activation and focus-acquisition (HCBT_ACTIVATE, HCBT_SETFOCUS) fall through the hook chain (CallNextHookEx). But in either of these cases, Fraps.dll goes on to look for a supported graphical interface.
Rather achronologically, the first thing it does at this point is a bunch of string-processing to capitalise and isolate the executable image’s file name. Presumably, the writers would have used GetProcessImageFileName, but bringing psapi.dll along to the party for this reason alone would be borderline-criminal.
Next, GetModuleHandleA is called on opengl32.dll, d3d8.dll, d3d9.dll, dxgi.dll and ddraw.dll. If all return NULL, then there is no work to do and the function returns. But if any of these modules are found, Fraps.dll gets straight to installing its function hooks. The hooks are simply JMP operations assembled ad-hoc at the beginning of IDirect3DDevice9::Release and Present (and presumably the equivalent functions belonging to the other APIs). Now, I was rather surprised to find that PunkBuster has no problem with such crude, unsubtle behaviour, but it’s possible that there is some agreement between the two developers.
That’s almost everything, but one problem remains. Nothing will be drawn to the screen unless d3d9!Present is called, and installation of the patch renders the original functions useless. It is for this reason IAT hooking is preferable to the patch method being used here, but from what I gather PunkBuster periodically ‘fixes’ the IAT of its client process, so that’s no-go. Fraps gets around this little inconvenience in the messy, but reliable way that you’d expect: each time the proxy Present function fires, it removes the patch, calls the original function, and restores the patch.
Here’s some untested C++ concept-code I threw together for the IDirect3DDevice9::Present case. The unbraced snippet installs a patch at address_d3d9_Present (use GetProcAddress) to redirect it to PresentHook. I’ve omitted the patch-removal code, along with a whole load of sanity-checking that really shouldn’t be left out in such a risky situation. Don’t use my laziness as an excuse.
// Calculate offset DWORD from_int = reinterpret_cast <DWORD> (address_d3d9_Present); DWORD to_int = reinterpret_cast <DWORD> (&PresentHook); // This version of the JMP instruction takes an address relative // to the current address, and it is 5 bytes long // So the relative offset is 'to - from - 5' // Don't worry about the unsigned DWORD underflowing DWORD offset = to_int - from_int - 5; // Assemble the patch at the beginning of Present const unsigned char jmp = 0xE9; // The opcode for a 32-bit rel JMP unsigned char* ip = reinterpret_cast <unsigned char*> (address_d3d9_Present); *ip = jmp; *(reinterpret_cast <DWORD*> (ip + 1)) = offset; HRESULT __cdecl PresentHook(const RECT* pSourceRect, const RECT* pDestRect, HWND hDestWindowOverride, const RGNDATA* pDirtyRegion) { IDirect3dDevice9* device; __asm MOV device, ECX; // Do anything that needs to be done before Present gets called // Remove the Present hook HRESULT return_value = Present(pSourceRect, pDestRect, hDestWindowOverride, pDirtyRegion); // Reinstall the Present hook // Do anything that needs to be done after Present gets called return return_value; }
[...] moment is DLL hooking, and so I thought I’d present an applied example. I already explained how Fraps works, and since I’ve recently been roped into writing a similar tool for a stranger, I thought [...]
Hi Greg,
and thanks again for an invalueble source of knowledge. I tried to run fraps in ollydbg, but it got stuck. i tried to search for the call IsDebuggerPresent (or something like that) to set there a breakpoint, but i think the freeze happens before
any tricks? i’m a complete newbie to ollydbg, so if you could pin-point me it would be just great.
again, thanks a lot
Joel
Hi again Joel.
I’m pretty sure this is new behaviour. I just downloaded the latest Fraps, gave it a spin and saw Olly crash just like you did. Perhaps the makers of Fraps didn’t take too kindly to my recent analysis, or maybe it’s just a coincidence, but this is an anti-debug measure targetting OllyDbg.
Attaching a JIT debugger to the failed instance of OllyDbg (after Fraps has crashed it) reveals a very suspicious looking string not too far down the stack. It turns out that OllyDbg uses an unsanitised call to sprintf in its console logging. Consequently, if the debuggee causes it to log a string containing formatting tokens, undefined behaviour ensues. In this case, a simple call to OutputDebugStringA("%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s...") will cause a Olly to crash with certainty.
Normally I’d patch the offending call out of the executable file, but Fraps seems to be packed, so the quickest way to proceed is to simply short-circuit OutputDebugStringA itself, in the target Fraps process before allowing execution to commence.
So to sidestep this nasty behaviour, load up Fraps, wait for Olly to pause at WinMain, then in the disassembly pane, ‘Go to’ OutputDebugStringA, and assemble (space-bar) ‘RETN 4′ at its first instruction. Now any calls to this function will return immediately, and execution may resume normally.
Unfortunately, this will need to be done each time you restart the target process, which is a bit of a pain, but if you’re feeling ambitious you could either automate the process with an OllyScript or attempt to unpack the executable on disk and patch out the offending OutputDebugStringA calls (PEiD doesn’t positively identify any packer, so I couldn’t say how difficult this process will be).
wow, amazing! what a find… and, the solution worked! I had a couple of illegal and privileged instructions but shift-f9 did solve it. now fraps is up and running, and i can continue my journey.
btw, a general question – I wondered why the 4 in RETN 4? I know what the 4 does (increment the stack pointer by a further 4 bytes after popping the return address), just wondered why did you use it here
thanks much
J
Glad it worked out for you.
The RETN 4 is there to clean up the stack before returning. If you look at OutputDebugStringA’s declaration, you’ll see it’s a stdcall taking a single pointer. This calling convention dictates that the callee function must remove the arguments from the stack, and since a single pointer is four bytes long (on x86), RETN 4 is the quickest way to do it.
[...] into every running process?" i think it only targets directX and openGL apps. Wrong It targets ANY application, regardless of API. [...]
[...] into every running process?" i think it only targets directX and openGL apps. Wrong It targets ANY application, regardless of API. Which is wrong and ineffective. [...]
Hey Greg, another solution to enumproc is to load a very small dll into every process that does ONE, and only one thng: Check if the real fraps dll needs loading.
Hello there!
It’s very nice of you to publish sample of D3D9 hook. Thanks.
Although I was searching the web for weeks, reading lots of tutorials and articles, I’m still confused with DirectX and D3D. And I still have questions unsolved.
So the problem is next. I have a DirectX application and I want to get pixel color from certain coords. I’m doing this using C++ GetPixel(HDC, X, Y). It works with DirectX window, but it won’t do when it’s minimized/hidden. And I’m wondering how FRAPS does this? I mean, I can capture a screenshot using FRAPS even if window is minimized and/or hidden. I know it inject fraps.dll in a window’s process, but when I inject the same dll (using RemoteDll http://www.novell.com/coolsolutions/tools/17354.html)nothing happens.
So I want to ask what function(s) do I have to hook in order to be able to take a pixel color from some coords even if the DirectX window is minimized and/or hidden? I.e. I want to make DirectX application to draw itself in HDC or anywhere else (not a .bmp or .jpg or anything like that), even if it’s minimized hidden.
Thanks in advance.
Kindest Regards, AP.
Hi Greg,
1st, a thanks “from France
” for this source of knowledge…
2nd, is this trick is still valid ?
Tnx a lot