1st, a thanks “from France ” for this source of knowledge…
2nd, is this trick is still valid ?

Tnx a lot

It’s very nice of you to publish sample of D3D9 hook. Thanks.

Although I was searching the web for weeks, reading lots of tutorials and articles, I’m still confused with DirectX and D3D. And I still have questions unsolved.

So the problem is next. I have a DirectX application and I want to get pixel color from certain coords. I’m doing this using C++ GetPixel(HDC, X, Y). It works with DirectX window, but it won’t do when it’s minimized/hidden. And I’m wondering how FRAPS does this? I mean, I can capture a screenshot using FRAPS even if window is minimized and/or hidden. I know it inject fraps.dll in a window’s process, but when I inject the same dll (using RemoteDll http://www.novell.com/coolsolutions/tools/17354.html)nothing happens.

So I want to ask what function(s) do I have to hook in order to be able to take a pixel color from some coords even if the DirectX window is minimized and/or hidden? I.e. I want to make DirectX application to draw itself in HDC or anywhere else (not a .bmp or .jpg or anything like that), even if it’s minimized hidden.

Thanks in advance.

Kindest Regards, AP.

The RETN 4 is there to clean up the stack before returning. If you look at OutputDebugStringA’s declaration, you’ll see it’s a stdcall taking a single pointer. This calling convention dictates that the callee function must remove the arguments from the stack, and since a single pointer is four bytes long (on x86), RETN 4 is the quickest way to do it.

btw, a general question – I wondered why the 4 in RETN 4? I know what the 4 does (increment the stack pointer by a further 4 bytes after popping the return address), just wondered why did you use it here

thanks much
J

I’m pretty sure this is new behaviour. I just downloaded the latest Fraps, gave it a spin and saw Olly crash just like you did. Perhaps the makers of Fraps didn’t take too kindly to my recent analysis, or maybe it’s just a coincidence, but this is an anti-debug measure targetting OllyDbg.

Attaching a JIT debugger to the failed instance of OllyDbg (after Fraps has crashed it) reveals a very suspicious looking string not too far down the stack. It turns out that OllyDbg uses an unsanitised call to sprintf in its console logging. Consequently, if the debuggee causes it to log a string containing formatting tokens, undefined behaviour ensues. In this case, a simple call to OutputDebugStringA("%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s...") will cause a Olly to crash with certainty.

Normally I’d patch the offending call out of the executable file, but Fraps seems to be packed, so the quickest way to proceed is to simply short-circuit OutputDebugStringA itself, in the target Fraps process before allowing execution to commence.

So to sidestep this nasty behaviour, load up Fraps, wait for Olly to pause at WinMain, then in the disassembly pane, ‘Go to’ OutputDebugStringA, and assemble (space-bar) ‘RETN 4′ at its first instruction. Now any calls to this function will return immediately, and execution may resume normally.

Unfortunately, this will need to be done each time you restart the target process, which is a bit of a pain, but if you’re feeling ambitious you could either automate the process with an OllyScript or attempt to unpack the executable on disk and patch out the offending OutputDebugStringA calls (PEiD doesn’t positively identify any packer, so I couldn’t say how difficult this process will be).

and thanks again for an invalueble source of knowledge. I tried to run fraps in ollydbg, but it got stuck. i tried to search for the call IsDebuggerPresent (or something like that) to set there a breakpoint, but i think the freeze happens before

any tricks? i’m a complete newbie to ollydbg, so if you could pin-point me it would be just great.

again, thanks a lot
Joel