{"id":817,"date":"2012-09-10T06:51:21","date_gmt":"2012-09-10T06:51:21","guid":{"rendered":"http:\/\/coherent-labs.com\/?p=817"},"modified":"2017-11-13T12:29:17","modified_gmt":"2017-11-13T12:29:17","slug":"debugging-undebuggable-applications-with-pix","status":"publish","type":"post","link":"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/","title":{"rendered":"Debugging undebuggable applications with PIX"},"content":{"rendered":"

Developers can ask DirectX 9 not to allow PIX to debug their application by calling D3DPERF_SetOptions(1)<\/span>. I knew that and encountered several commercial applications using it. One day, I was fooling around with Portal 2<\/i> and wanted to feed my curiosity on how some stuff is done but when I started PIX all I got was \u201cDirect3D Analysis Disabled<\/i>\u201d and I knew it was the time to find a way to circumvent this little peculiarity. So, let\u2019s see how can we convince DirectX to ignore the request of the said developers.<\/p>\n

I started with a simple application:<\/p>\n

 <\/p>\n

Let\u2019s check out the disassembly of the D3DPERF_SetOptions(1) call:<\/p>\n

_D3DPERF_SetOptions@4:<\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
72EC7402 8B FF<\/span><\/td>\nmov<\/span><\/td>\nedi,edi<\/span><\/td>\n<\/tr>\n
72EC7404 55<\/span><\/td>\npush<\/span><\/td>\nebp<\/span><\/td>\n<\/tr>\n
72EC7405 8B EC<\/span><\/td>\nmov<\/span><\/td>\nebp,esp<\/span><\/td>\n<\/tr>\n
72EC7407 83 EC 18<\/span><\/td>\nsub<\/span><\/td>\nesp,18h<\/span><\/td>\n<\/tr>\n
72EC740A A1 50 92 FC 72<\/span><\/td>\nmov<\/span><\/td>\neax,dword ptr [___security_cookie (72FC9250h)]<\/span><\/td>\n<\/tr>\n
72EC740F 33 C5<\/span><\/td>\nxor<\/span><\/td>\neax,ebp<\/span><\/td>\n<\/tr>\n
72EC7411 89 45 FC<\/span><\/td>\nmov<\/span><\/td>\ndword ptr [ebp-4],eax<\/span><\/td>\n<\/tr>\n
72EC7414 A1 54 74 EC 72<\/span><\/td>\nmov<\/span><\/td>\neax,dword ptr [string “DirectX Direct3D SO” (72EC7454h)]<\/span><\/td>\n<\/tr>\n
72EC7419 89 45 E8<\/span><\/td>\nmov<\/span><\/td>\ndword ptr [ebp-18h],eax<\/span><\/td>\n<\/tr>\n
72EC741C 8B 0D 58 74 EC 72<\/span><\/td>\nmov<\/span><\/td>\necx,dword ptr ds:[72EC7458h]<\/span><\/td>\n<\/tr>\n
72EC7422 89 4D EC<\/span><\/td>\nmov<\/span><\/td>\ndword ptr [ebp-14h],ecx<\/span><\/td>\n<\/tr>\n
72EC7425 8B 15 5C 74 EC 72<\/span><\/td>\nmov<\/span><\/td>\nedx,dword ptr ds:[72EC745Ch]<\/span><\/td>\n<\/tr>\n
72EC742B 89 55 F0<\/span><\/td>\nmov<\/span><\/td>\ndword ptr [ebp-10h],edx<\/span><\/td>\n<\/tr>\n
72EC742E A1 60 74 EC 72<\/span><\/td>\nmov<\/span><\/td>\neax,dword ptr ds:[72EC7460h]<\/span><\/td>\n<\/tr>\n
72EC7433 89 45 F4<\/span><\/td>\nmov<\/span><\/td>\ndword ptr [ebp-0Ch],eax<\/span><\/td>\n<\/tr>\n
72EC7436 8B 0D 64 74 EC 72<\/span><\/td>\nmov<\/span><\/td>\necx,dword ptr ds:[72EC7464h]<\/span><\/td>\n<\/tr>\n
72EC743C 89 4D F8<\/span><\/td>\nmov<\/span><\/td>\ndword ptr [ebp-8],ecx<\/span><\/td>\n<\/tr>\n
72EC743F C6 45 ED 44<\/span><\/td>\nmov<\/span><\/td>\nbyte ptr [ebp-13h],44h<\/span><\/td>\n<\/tr>\n
72EC7443 8B 4D FC<\/span><\/td>\nmov<\/span><\/td>\necx,dword ptr [ebp-4]<\/span><\/td>\n<\/tr>\n
72EC7446 33 CD<\/span><\/td>\nxor<\/span><\/td>\necx,ebp<\/span><\/td>\n<\/tr>\n
72EC7448 E8 D3 A1 F5 FF<\/span><\/td>\ncall<\/span><\/td>\n@__security_check_cookie@4 (72E21620h)<\/span><\/td>\n<\/tr>\n
72EC744D 8B E5<\/span><\/td>\nmov<\/span><\/td>\nesp,ebp<\/span><\/td>\n<\/tr>\n
72EC744F 5D<\/span><\/td>\npop<\/span><\/td>\nebp<\/span><\/td>\n<\/tr>\n
72EC7450 C2 04 00<\/span><\/td>\nret<\/span><\/td>\n4<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Some movs, xors, runtime security check and that\u2019s it, nothing with the actual value we passed to D3DPERF_SetOptions… well that was big nothing.<\/p>\n

Ok, take two – let\u2019s first start the application with PIX and then attach.
\nWe\u2019ll have to add some code to give us time to attach:<\/p>\n

 <\/p>\n

Use something like this, or just a Sleep() for enough time. Now what do we have with the new setup:<\/p>\n

_D3DPERF_SetOptions@4:<\/span>
\n72EC7402 E9 46 77 AA E8 \u00a0\u00a0 jmp \u00a0 \u00a0\u00a0 HookedD3DPERF_SetOptions (5B96EB4Dh)<\/p>\n

All right, the sneaky PIX has modified d3d9.dll’s<\/i> memory and now it has a jmp in the beginning! The function it now executes takes us inside PIXHelper.dll<\/i>:<\/p>\n

HookedD3DPERF_SetOptions:<\/span><\/p>\n\n\n\n\n\n\n\n
5BF6EB4D 8B FF<\/span><\/td>\nmov<\/span><\/td>\nedi,edi<\/span><\/td>\n<\/tr>\n
5BF6EB4F 55<\/span><\/td>\npush<\/span><\/td>\nebp<\/span><\/td>\n<\/tr>\n
5BF6EB50 8B EC<\/span><\/td>\nmov<\/span><\/td>\nebp,esp<\/span><\/td>\n<\/tr>\n
5BF6EB52 83 7D 08 01<\/span><\/td>\ncmp<\/span><\/td>\ndword ptr [ebp+8],1<\/span><\/td>\n<\/tr>\n
5BF6EB56 75 0B<\/span><\/td>\njne<\/span><\/td>\nHookedD3DPERF_SetOptions+16h (5BF6EB63h)<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

We\u2019ve only had one push so far (for the value we passed) and we push ebp, so that\u2019s 2 pushes. After “mov ebp,esp” ebp <\/span> is the same as the stack pointer, so dword ptr [ebp + 8] would be exactly the value we passed to D3DPERF_SetOptions. We compare that to 1 and if it\u2019s equal some procedures are invoked that stop the execution. If it isn\u2019t – we follow the jump specified by jne. What we have to do is make that jump unconditional – i.e. always execute the jump, regardless of the value passed. We don\u2019t care about the code that pops the message for disabled analysis so we have plenty of bytes to play with; However, we don\u2019t need them as we can see in \u201cIntel\u00ae 64 and IA-32 Architectures Software Developer Manuals<\/i>\u201d – what we\u2019re looking for is the EB cb variant of jmp, the exact same amount of bytes as the used jne instruction. Now all that\u2019s left is open PIXHelper.dll<\/i> with a hex editor (I used Notepad++ with hex-editor plugin), search for some of the bytes (try \u201c8B EC 83 7D 08 01 75 0B<\/span>\u201d – I found it only once) and change the 75 to EB. Voila! Now you won\u2019t see that annoying warning anymore.<\/p>\n

 <\/p>\n

Follow Nick on Twitter:\u00a0@Nikxio<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Developers can ask DirectX 9 not to allow PIX to debug their application by calling D3DPERF_SetOptions(1). I knew that and encountered several commercial applications using it. One day, I was fooling around with Portal 2 and wanted to feed my curiosity on how some stuff is done but when I started PIX all I got […]<\/p>\n","protected":false},"author":10,"featured_media":9891,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[408],"tags":[168],"yoast_head":"\nDebugging undebuggable applications with PIX - Coherent Labs<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Debugging undebuggable applications with PIX - Coherent Labs\" \/>\n<meta property=\"og:description\" content=\"Developers can ask DirectX 9 not to allow PIX to debug their application by calling D3DPERF_SetOptions(1). I knew that and encountered several commercial applications using it. One day, I was fooling around with Portal 2 and wanted to feed my curiosity on how some stuff is done but when I started PIX all I got […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/\" \/>\n<meta property=\"og:site_name\" content=\"Coherent Labs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/CoherentLabs\/\" \/>\n<meta property=\"article:published_time\" content=\"2012-09-10T06:51:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-11-13T12:29:17+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/coherent-labs.com\/posts\/wp-content\/uploads\/2012\/09\/stencil.blog-post-feature-29.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Nick\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@CoherentLabs\" \/>\n<meta name=\"twitter:site\" content=\"@CoherentLabs\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nick\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/\",\"url\":\"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/\",\"name\":\"Debugging undebuggable applications with PIX - Coherent Labs\",\"isPartOf\":{\"@id\":\"https:\/\/coherent-labs.com\/posts\/#website\"},\"datePublished\":\"2012-09-10T06:51:21+00:00\",\"dateModified\":\"2017-11-13T12:29:17+00:00\",\"author\":{\"@id\":\"https:\/\/coherent-labs.com\/posts\/#\/schema\/person\/0f7e122b7a2587b2ccaf7fe8f9964f94\"},\"breadcrumb\":{\"@id\":\"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/coherent-labs.com\/posts\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Debugging undebuggable applications with PIX\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/coherent-labs.com\/posts\/#website\",\"url\":\"https:\/\/coherent-labs.com\/posts\/\",\"name\":\"Coherent Labs\",\"description\":\"Cross-Platform Game UI Middleware\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/coherent-labs.com\/posts\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/coherent-labs.com\/posts\/#\/schema\/person\/0f7e122b7a2587b2ccaf7fe8f9964f94\",\"name\":\"Nick\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/coherent-labs.com\/posts\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cf9bc3aa3b82bb10f4cc082fad0faa59?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cf9bc3aa3b82bb10f4cc082fad0faa59?s=96&d=mm&r=g\",\"caption\":\"Nick\"},\"url\":\"https:\/\/coherent-labs.com\/posts\/author\/nick\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Debugging undebuggable applications with PIX - Coherent Labs","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/","og_locale":"en_US","og_type":"article","og_title":"Debugging undebuggable applications with PIX - Coherent Labs","og_description":"Developers can ask DirectX 9 not to allow PIX to debug their application by calling D3DPERF_SetOptions(1). I knew that and encountered several commercial applications using it. One day, I was fooling around with Portal 2 and wanted to feed my curiosity on how some stuff is done but when I started PIX all I got […]","og_url":"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/","og_site_name":"Coherent Labs","article_publisher":"https:\/\/www.facebook.com\/CoherentLabs\/","article_published_time":"2012-09-10T06:51:21+00:00","article_modified_time":"2017-11-13T12:29:17+00:00","og_image":[{"width":1200,"height":600,"url":"https:\/\/coherent-labs.com\/posts\/wp-content\/uploads\/2012\/09\/stencil.blog-post-feature-29.jpg","type":"image\/jpeg"}],"author":"Nick","twitter_card":"summary_large_image","twitter_creator":"@CoherentLabs","twitter_site":"@CoherentLabs","twitter_misc":{"Written by":"Nick","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/","url":"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/","name":"Debugging undebuggable applications with PIX - Coherent Labs","isPartOf":{"@id":"https:\/\/coherent-labs.com\/posts\/#website"},"datePublished":"2012-09-10T06:51:21+00:00","dateModified":"2017-11-13T12:29:17+00:00","author":{"@id":"https:\/\/coherent-labs.com\/posts\/#\/schema\/person\/0f7e122b7a2587b2ccaf7fe8f9964f94"},"breadcrumb":{"@id":"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/coherent-labs.com\/posts\/debugging-undebuggable-applications-with-pix\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/coherent-labs.com\/posts\/"},{"@type":"ListItem","position":2,"name":"Debugging undebuggable applications with PIX"}]},{"@type":"WebSite","@id":"https:\/\/coherent-labs.com\/posts\/#website","url":"https:\/\/coherent-labs.com\/posts\/","name":"Coherent Labs","description":"Cross-Platform Game UI Middleware","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/coherent-labs.com\/posts\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/coherent-labs.com\/posts\/#\/schema\/person\/0f7e122b7a2587b2ccaf7fe8f9964f94","name":"Nick","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/coherent-labs.com\/posts\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/cf9bc3aa3b82bb10f4cc082fad0faa59?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cf9bc3aa3b82bb10f4cc082fad0faa59?s=96&d=mm&r=g","caption":"Nick"},"url":"https:\/\/coherent-labs.com\/posts\/author\/nick\/"}]}},"_links":{"self":[{"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/posts\/817"}],"collection":[{"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/comments?post=817"}],"version-history":[{"count":1,"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/posts\/817\/revisions"}],"predecessor-version":[{"id":9500,"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/posts\/817\/revisions\/9500"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/media\/9891"}],"wp:attachment":[{"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/media?parent=817"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/categories?post=817"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/coherent-labs.com\/posts\/wp-json\/wp\/v2\/tags?post=817"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}