Drawing on another Direct3D program’s viewport

Update: See the post for the new version.

The theme of the moment is DLL hooking, and so I thought I’d present an applied example. I already explained how Fraps works, and since I’ve recently been roped into writing a similar tool for a stranger, I thought I’d share the wealth. There isn’t much new material here, but people like examples with source code, so you can download the DLL source (C++) from the project page.

If you don’t know how to inject this DLL into a foreign process, then you’ll need to read my previous post or wait for the injection framework I’m working on. But once it’s injected call the Initialise method, via CreateRemoteThread or otherwise, to install the hooks. It works with any program that uses IDirect3DDevice9::Present (or IDirect3DSwapChain::Present) to render, which is probably all of the DirectX 9 games. Similarly, invoke Release to remove the hooks. The source is fairly self-explanatory, with a few exceptions.

• It’s not safe for 64-bit consumption, though this should be obvious.
• While there’s no reason it can’t be made to work with Unicode, I’ve written everything in ASCII, for simplicity.
• By default, the DLL will increase its own reference count to prevent it being unloaded prior to termination of the host process. This is because there is a small risk of the DLL being unloaded by one thread, while a hooked function in another returns to the now dead memory. I figured that it’s best to waste a little bit of everybody’s memory than to crash unnecessarily.
• The d3d9.dll function addresses (and prologues) are hard-coded, or at least their offsets are. While this may look very unprofessional and rather risky, I can assure you that it’s quite safe. The alternative would be to hack up some virtual-function tables and that’s a whole other story for a whole other post.
• You may notice that the compiled DLL is dependent upon D3DX. This isn’t necessary for the hook itself, but I used ID3DXFont in my example for demonstrative purposes. The only reason I mention this is that there is no way to guarantee the existence of any D3DX DLLs on a DirectX 9 machine, and distributing them yourself is in violation of the DirectX Runtime EULA. So if you happen to need to distribute this code, you’ll either need to carry the huge runtime installer around, or avoid using D3DX altogether.

Update:

• The soft-hooks used here will cause problems with PunkBuster if applied to any of its monitored functions. If you need to do this then you’ll have to be a bit cleverer.
• The source assumes that the graphics device will never become invalid. If you suspect that this isn’t the case (which will be true for any full-screen game at a minimum) then you’ll need to add the appropriate sanity checks (see IDirect3DDevice9::TestCooperativeLevel) before attempting to render anything, lest you want to crash and burn.

Posted in Game Programming, RCE | 35 Comments »

Case study: Fraps

One of the topics that I often find myself bluffing through on GameDev is Direct3D hooking. In particular, how to display an overlay of your own on the window of another Direct3D program, often a commercial game. It’s pretty clear that the simplest method would involve somehow hooking the call to IDirect3DDevice8/9/10::Present, but the details are a little sketchy, particularly when you throw anti-hack systems into the mix. To be quite honest, I wasn’t sure I’d have been able to write a scalable hook that wouldn’t cause any incompatibilities - at least not without doing some very immoral things. So when I found out that Fraps has been doing exactly this for years, and that it somehow manages to avoid angering PunkBuster and other such systems, I decided to investigate.

What is Fraps?

Simply put, it’s a profiling and video-capture tool for PC games. As well as providing the ability to capture a video stream from any Direct3D 8/9/10, DirectDraw or OpenGL program, it can display real-time performance statistics (frame-rate and such) by means of an overlay on the game’s window (or full-screen display).

Remarkably, Fraps seems to handle the hacky side of all this automatically, with no fuss. It will dig its claws into any compatible game, whether it was started before or after Fraps. Yet if you investigate the state of DirectX and OpenGL’s DLLs on disk (in system32), they remain untouched at all times.

So how does it work?

Well it could be a whole lot worse, but I wasn’t thrilled to find out that every process running on my system had an instance of Fraps.dll loaded. It’s not so much the 106kB footprint that bothers me, but the performance and stability concerns. Anyway, my suspicions that this DLL was ‘infecting’ all processes via a system-wide hook were soon confirmed when OllyDbg caught Fraps.exe making a call to SetWindowsHookEx.

SetWindowsHookEx(WH_CBT, &FrapsProcCBT, hModuleFrapsDLL, 0);

So in one fell swoop, Fraps guarantees that Windows will load a copy of Fraps.dll into every process currently running, as well as those created in future. Moreover, the function FrapsProcCBT will be called by that process each time it attempts to create, destroy, show, hide, move or resize a window. Now this may seem like the perfect solution to a difficult problem, but it’s rather wasteful considering that most processes run window-driven interfaces, and only very few involve DirectX or OpenGL.

How should it work?

Now I haven’t tested this, but a much cleaner way to achieve the same goal would be for Fraps.exe to periodically poll EnumProcesses and EnumProcessModules, so as to determine the processes that actually need hooking. Installing the hooks into these specific processes would require no more work but would save the OS some effort and limit the worst-case-scenario disaster-zone to DirectX and OpenGL applications, which is a considerably smaller domain than almost everything. In Fraps’s defence, the code makes extensive use of IsBadReadPtr and suchlike, and I’ve never heard of it causing any trouble, but nevertheless, the best way to prevent your DLL from crashing someone else’s program is to make sure it never gets loaded.

What does the hook do?

All events but window activation and focus-acquisition (HCBT_ACTIVATE, HCBT_SETFOCUS) fall through the hook chain (CallNextHookEx). But in either of these cases, Fraps.dll goes on to look for a supported graphical interface.

Rather achronologically, the first thing it does at this point is a bunch of string-processing to capitalise and isolate the executable image’s file name. Presumably, the writers would have used GetProcessImageFileName, but bringing psapi.dll along to the party for this reason alone would be borderline-criminal.

Next, GetModuleHandleA is called on opengl32.dll, d3d8.dll, d3d9.dll, dxgi.dll and ddraw.dll. If all return NULL, then there is no work to do and the function returns. But if any of these modules are found, Fraps.dll gets straight to installing its function hooks. The hooks are simply JMP operations assembled ad-hoc at the beginning of IDirect3DDevice9::Release and Present (and presumably the equivalent functions belonging to the other APIs). Now, I was rather surprised to find that PunkBuster has no problem with such crude, unsubtle behaviour, but it’s possible that there is some agreement between the two developers.

That’s almost everything, but one problem remains. Nothing will be drawn to the screen unless d3d9!Present is called, and installation of the patch renders the original functions useless. It is for this reason IAT hooking is preferable to the patch method being used here, but from what I gather PunkBuster periodically ‘fixes’ the IAT of its client process, so that’s no-go. Fraps gets around this little inconvenience in the messy, but reliable way that you’d expect: each time the proxy Present function fires, it removes the patch, calls the original function, and restores the patch.

Here’s some untested C++ concept-code I threw together for the IDirect3DDevice9::Present case. The unbraced snippet installs a patch at address_d3d9_Present (use GetProcAddress) to redirect it to PresentHook. I’ve omitted the patch-removal code, along with a whole load of sanity-checking that really shouldn’t be left out in such a risky situation. Don’t use my laziness as an excuse.

// Calculate offset
DWORD from_int = reinterpret_cast <DWORD> (address_d3d9_Present);
DWORD to_int = reinterpret_cast <DWORD> (&PresentHook);
 
// This version of the JMP instruction takes an address relative
// to the current address, and it is 5 bytes long
// So the relative offset is 'to - from - 5'
// Don't worry about the unsigned DWORD underflowing
DWORD offset = to_int - from_int - 5;
 
// Assemble the patch at the beginning of Present
const unsigned char jmp = 0xE9; // The opcode for a 32-bit rel JMP
 
unsigned char* ip = reinterpret_cast <unsigned char*> (address_d3d9_Present);
*ip = jmp;
*(reinterpret_cast <DWORD*> (ip + 1)) = offset;
 
HRESULT __cdecl PresentHook(const RECT* pSourceRect, const RECT* pDestRect, HWND hDestWindowOverride, const RGNDATA* pDirtyRegion) {
    IDirect3dDevice9* device;
    __asm MOV device, ECX;
 
    // Do anything that needs to be done before Present gets called
 
    // Remove the Present hook
    HRESULT return_value = Present(pSourceRect, pDestRect, hDestWindowOverride, pDirtyRegion);
    // Reinstall the Present hook
 
    // Do anything that needs to be done after Present gets called
    return return_value;
}

Posted in Game Programming, RCE | 8 Comments »

RST decomposition of a general skew-free 3D transformation

First of all, I refer you to D3DMatrixDecompose. If you want to break a standard 3D transformation matrix into its rotational, translational and scaling parts, without caring how it’s done, then look no further. If your needs are a little more specific and you’re sure you aren’t reinventing this wheel, then read on.

There is nothing clever about this decomposition, but it’s a question that comes up more often than I’d expect, so here’s the lowdown. I assume that your matrix is in DirectX-standard row-vector representation (so v_out = v_inM) and that the skew components are zero (M₁₄ = M₂₄ = M₃₄ = 0). You should visualise the matrix like this:

First, we extract the translation:

v_translation = (M₄₁, M₄₂, M₄₃)

Then the scaling factors:

s_x = √(M₁₁² + M₂₁² + M₃₁²)
s_y = √(M₁₂² + M₂₂² + M₃₂²)
s_z = √(M₁₃² + M₂₃² + M₃₃²)

The rotation matrix is then the upper-left 3×3 minor, after scaling back to unity.

M^rotation = M;

// Remove translation
M^rotation₄₁ = 0;
M^rotation₄₂ = 0;
M^rotation₄₃ = 0;

// Normalise
M^rotation₁₁ /= s_x;
M^rotation₂₁ /= s_x;
M^rotation₃₁ /= s_x;

M^rotation₁₂ /= s_y;
M^rotation₂₂ /= s_y;
M^rotation₃₂ /= s_y;

M^rotation₁₃ /= s_z;
M^rotation₂₃ /= s_z;
M^rotation₃₃ /= s_z;

Posted in Game Programming | No Comments »

LDR tone-mapping and how to do it properly

I’m a huge fan of post-processing in games. It seems that no matter what I’m writing, I can’t resist the temptation to install an over-the-top bloom effect and some tone-mapping. And that’s me being conservative. The great thing about tone-mapping is that you can throw it on the end of just about any rendering pipeline and instantly glitz up the visuals, giving it that ‘digitally remastered’ feel.

Tone-Mapping

So what is tone-mapping? Well, it’s a post-processing effect that remaps the render’s colour-dynamics to change the overall appearance of the game. Some of the more common tone-mapping operations are contrast & brightness, saturation and HDR exposure. HDR tone-mapping is an art unto itself and it can only be used for HDR pipelines (with floating-point render-targets and HDR textures), so I’ll restrict the conversation to the more universal LDR tone-mapping.

The Conjugate Transform

If you plan on doing anything interesting in a tone-mapping pass, then it’s rather necessary, for the sakes of performance, readability and maintainability, to convert to a more suitable colour-space than RGB. The first such space that springs to mind is HSL, and indeed tone-mapping in HSL is like a gentle walk in the park, but it’s wise to look a little further afield to YCC. But why YCC? Sure, it does offer a luma component for brightness & contrast mapping, but the saturation is tied up in the two chroma components. Granted, this is a bad thing, but it’s not nearly as bad as the cost of a full RGB->HSL->RGB conversion.

The Problem With HSL

I spent a fair while trying to optimise this HSL-detour code in HLSL, hoping that I could make it viable for small shaders, but came out rather disappointed. Despite the availability of vector SIMD instructions, the piecewise-linear nature of the transformation demands a worryingly large number of conditional branches, and unless you have the luxury of Shader Model 4’s true branching, this amounts to a horror story of register-juggling and lerp operations. I didn’t try too hard, but believe it’s impossible to complete the transformation-and-back through HSL in under 100 ps_3_0 operations, which immediately rules out the possibility of assembling on a Shader Model 2 target platform.

YCC To The Rescue

Contrast this with the simplicity of the truly linear RGB->YCC->RGB transformation. If there’s one thing that the GPU does best, it’s vector-matrix multiplication, and that’s exactly what this boils down to:

float4x4 RGBToYCC = 
{ 0.299,  0.587,  0.114,  0.000,
  0.701, -0.587, -0.114,  0.000,
 -0.299, -0.587,  0.886,  0.000,
  0.000,  0.000,  0.000,  1.000};
 
float4x4 YCCToRGB = 
{ 1.000,  1.000,  0.000,  0.000,
  1.000, -0.509, -0.194,  0.000,
  1.000,  0.000,  1.000,  0.000,
  0.000,  0.000,  0.000,  1.000};
 
float4 PS_LDRToneMap(float4 tex_coord : TEXCOORD) : COLOR
{
    float4 RGBA = tex2D(linear_sampler, tex_coord);
    float4 YCCA = mul(RGBToYCC, RGBA);
 
    // Work goes here
 
    RGBA = mul(YCCToRGB, YCCA);
    return saturate(RGBA);
}

This assembles to a handsome 9 instructions, leaving plenty of room even with the arcane ps_1_4’s instruction limit.

The Prize

My current project makes use of this code to ramp the contrast and saturation up and down, according to the scene. The code is simple, and the results rather dramatic.

// Contrast
YCCA.x -= contrast_midpoint;
YCCA.x *= contrast_gain;
YCCA.x += contrast_midpoint;
 
// Chroma
YCCA.y -= chroma_red_midpoint;
YCCA.y *= chroma_red_gain;
YCCA.y += chroma_red_midpoint;
 
YCCA.z -= chroma_blue_midpoint;
YCCA.z *= chroma_blue_gain;
YCCA.z += chroma_blue_midpoint;

Posted in Game Programming | No Comments »

Compatible X-file HLSL-based vertex-blending with D3DX

On the whole, D3DX does a great job of making our lives easier, us Direct3D 9 programmers. But one topic that has generated a lot of confusion yet very little documentation is the correct usage of the BLENDINDICES shader semantic.

If you’re having trouble getting your CPU and GPU to communicate blend-indices correctly, the first thing to do is make sure that your source data is valid. If you’re using a custom mesh-loading routine then you’re on your own, but certainly under D3DXLoadMeshHierarchyFromX you can save yourself a lot of potential grief by prototyping with a reliable X-file. Support for X-file exporting is very shaky for most 3D modelling suites, but the SDK’s tiny.x sample is as close as you’ll get to a standard model. Once Tiny is happily running around in her fighter-pilot’s uniform you can start thinking about trying out your own models.

Nobody seems to be sure just what went wrong when the late ARB drafted up the usage semantics of blending parameters, but many current video cards lack hardware support for the UBYTE data type needed to store the indices. For this reason, developers have found themselves hammering unsigned-byte-quadruplets into the shape of D3DCOLORs, lying about the data content and praying that the vertex pipeline will have magically transformed the output into something that can be jimmied back into a tuple of UBYTEs. As far as I can see, this is the mother of all DirectX hacks, but it also seems to be the universal standard. That’s the only reason I’ll sleep at night after posting this.

After successfully calling ID3DXSkinInfo::ConvertToIndexBlendedMesh, the blend-index data is all present and correct. The challenge is to pipe it to the shader without misinterpretation. For that, I found it necessary to set all index data in the vertex declaration to be passed as D3DCOLOR data (DWORDS in disguise). By the way, if you are still using FVFs for something as finicky as GPU skinning then you don’t deserve any help: bite the bullet and write out those overly verbose, but oh-so-flexible vertex declarations.

// Fix UBYTE4 Support
D3DVERTEXELEMENT9 decl[MAX_FVF_DECL_SIZE];
skinned_mesh->GetDeclaration(decl);
{
    int i = 0;
    while (decl[i].Method != 0xFF)
    {
        if (decl[i].Usage == D3DDECLUSAGE_BLENDINDICES)
        {
            decl[i].Type = D3DDECLTYPE_D3DCOLOR;
            break;
        }
        ++i;
    }
}
skinned_mesh->UpdateSemantics(decl);

This takes care of the program code. All that remains is to interpret the data accordingly in the shader. For this, Microsoft generously provided us with the D3DCOLORtoUBYTE4 macro. HLSL (as of SM3) doesn’t have true integer support, let alone unsigned byte support, so we use their emulated int4 vector.

VS_OUTPUT VS_SkeletalBlend
(float3 pos           : POSITION0,
 float1 blend_weight  : BLENDWEIGHT0,
 float4 blend_indices : BLENDINDICES0,
 float3 normal        : NORMAL0,
 float2 tex_coord     : TEXCOORD0)
{
    int4 indices = D3DCOLORtoUBYTE4(blend_indices);
 
    float weight0 = blend_weight;
    float weight1 = 1 - weight0;
    int ind0 = indices.x;
    int ind1 = indices.y;
 
    // ...
}

Notice that the semantic here is BLENDINDICES, even though the vertex declaration says D3DCOLOR. Note further that the type becomes, somewhat wastefully, a float4 as all colour values are passed as such. The sample here is for a two-weights-per-vertex animation, but it’s easy enough to scale this up to four. If you need more than four weights then you need to have a word/fight with your artist, but the idea is the same, only you pass another parameter just like the first. Now I’m not certain that this is the best configuration - it certainly looks like a horrid mess - but it’s the only combination I know to work, after hours and hours of painful Googling. Suggestions for improvement are welcome.

Posted in Game Programming | No Comments »