MITHUN SHANBHAG’S BLOG

On my team, we currently have two open SDET (software development engineer for test) positions that we are actively looking to fill. The job description is already posted here so I’ll not rehash the  details.

So what exactly do we do? –

I’ll give you a very high level description. We are a part of the Common Language Runtime (CLR) team and are responsible for producing, testing the debugging and profiling engines. These APIs/engines are consumed by the Visual Studio Debugger, WinDbg and F1 profiler and other ISV tools. As an SDET you’ll be primarily responsible for testing these APIs. This means writing new debuggers/profilers, extending functionality of existing ones, writing automation tools etc.

What we require you to have –

-          Interest in low level technologies and bit-twiddling.

-          Strong scripting skills.

-          Proficiency in debugging (I cannot emphasize this enough!) and fixing bugs.

-          4-6 years of development experience.

-          Proficiency in C, C++ and/or any managed language.  

Icing on the cake –

-          Experience with implementing diagnostic tools like debuggers, profilers or dis-assemblers. Bonus points if they actually work J.

-          Knowledge of MSIL and/or native assembly.

-          Familiarity with the internals of the CLR – Garbage Collector, Loader, Type System, JIT/CodeGen, Exception handling, BCL, hosting, Security etc. 

Please let me know if you (or someone you know) are interested. I encourage you to apply via the Microsoft career website.

Windbg/Ntsd unfortunately does not support a breakpoint hit-counter (I searched through Windbg’s help file but couldn’t find one). It would be very useful to know how many times a particular breakpoint is triggered. 
 

Fortunately for us, we can cook up our own BP hit-counter using pseudo registers. According to the help file, $t0, $t1, …, St19 are available as user-defined pseudo registers. We can use these to store the hit-count. 
 

Bp <module>!<class>::<function> “r $t0 = @$t0 +1; r $t0; g”

Bp <address> “r $t0 = @$t0 +1; r $t0; g”

 Please let me know if there is a simpler way to do this.
 

Struggling with regular expressions? Maybe this might be of some help - http://regexlib.com/CheatSheet.aspx
 

If you are into writing custom prologs and epilogs for functions with naked calling convention, then you need to -

1. Get a life! No seriously.
2. Know about the __LOCAL_SIZE compiler symbol.
    

This is how a function prolog looks like -

    __asm

{

push ebp

mov ebp, esp // Set up our base-pointer frame

sub esp, <size of stack variables>

pushad // Save all general purpose registers.

}

So how do you calculate the size required for stack variables? Till date, I used to compute it by hand. Luckily today I discovered the existence of the __LOCAL_SIZE compiler symbol, the value of which is decided by the compiler.

MSDN documentation says -

The compiler determines the value of __LOCAL_SIZE. Its value is the total number of bytes of all user-defined local variables and compiler-generated temporary variables. __LOCAL_SIZE can be used only as an immediate operand; it cannot be used in an expression. You must not change or redefine the value of this symbol.
 

So now my function prolog simply looks like this –

    __asm

{

push ebp

mov ebp, esp // Set up our base-pointer frame

sub esp, __LOCAL_SIZE

pushad // Save all general purpose registers.

}

Quick and easy, huh?
 
 
 
 
 

All Windows aficionados are eagerly looking forward to reading the books listed below (I myself haven’t read them yet but these are up there on my list of things to do). Incase you have read them already, let me know how they are -

	Title: Advanced Windows Debugging Authors: Mario Hewardt, Daniel Pravat ISBN: 0321374460
	Title: Windows via C/C++ Authors: Jeffrey Richter, Christophe Nasarre ISBN: 0321374460 Notes: This is an update to Richter’s famous book on Windows Programming.

 

 

Image File Execution options provides you with a mechanism to always launch an executable directly under the debugger. This is extremely useful if you ever need to investigate issues in the executable’s startup code (services especially). You can set the IFEO options directly via the registry or indirectly using the Gflags tools (available with the Window debugging toolkit).

 

You need to create a registry key and populate it with a value as follows -

Key	“HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\<executable>“
Value	Debugger : REG_SZ : <full-path to your favorite debugger>

 

You do not need the full path to the application, only the exe name will suffice. However you do need the full path to the debugger.  As an example, we look at launching notepad under ntsd, you would be creating the following -

Key	“HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe”
Value	Debugger : REG_SZ : “c:\dbg\ntsd.exe -g”

 

 

You can also use Gflags to set IFEO too -

 



 

 

 

How does IFEO work?

 

Kernel32!CreateProcess when called without the DEBUG_PROCESS or DEBUG_ONLY_THIS_PROCESS creation flags, checks the registry to see if IFEO has been set on the executable that it is launching. If yes, then it simply prepends the debugger path to the executable name, effectively getting the executable to launch under the debugger. If you do not specify the correct path to the debugger, then you’ll probably get greeted with a “file not found” error. In our notepad/ntsd example above, Kernel32!CreateProcess ends up invoking -

“c:\dbg\ntsd.exe -g  notepad.exe”

 

Now ntsd eventually launches notepad under the debugger by calling Kernel32!CreateProcess with one of the following creation flags - DEBUG_PROCESS or DEBUG_ONLY_THIS_PROCESS. The presence of any of these creation flags forces Kernel32!CreateProcess to bypass IFEO options this time around (else we would have been running into an endless loop) and actually launch the executable under the debugger.

 

 

IFEO and 64 bit -

 

A word of caution - For 32 bit executable running in the WOW on X64 machines, your natural tendency might be to create the registry key in the syswow node -

“HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\<executable>“

 

However Gregg Miskelly notes that you should set the IFEO corresponding to the bit-ness to the application calling into kernel32!CreateProcess to launch the executable -

 

“On Win 64, there are two copies of HKEY_LOCAL_MACHINE\Software (one for 32-bit apps, and one for 64-bit apps), and therefore there are two copies of these options. However, where the operating system looks isn’t dependant on the bit-ness of the application that is going to be debugged (which is what you would probably expect). Instead, it is dependant on the bit-ness of the application that called CreateProcess.”

 

 

Other IFEO caveats - 

 

Raymond Chen notes the following caveat in his blog entry  -

 

“If you passed special parameters via the STARTUPINFO structure, those parameters get passed to the debugger. And the PROCESS_INFO that is returned by the CreateProcess function describes the debugger, not the process being debugged.”

 

 

IFEO and Managed debuggers -

 

IFEO can only be used for native or interop debugging, but not for managed debugging. Mike Stall has an excellent blog entry that describes in great detail exactly why. The gist is this - Managed debuggers like Mdbg/cordbg/VS.NET use ICorDebug::CreateProcess to launch managed executables under the debugger. However for managed debugging, the debugger should call ICorDebug::CreateProcess without the DEBUG_PROCESS or DEBUG_ONLY_THIS_PROCESS creation flags (this is publicly documented). This API internally ends up calling Kernel32!CreateProcess without the DEBUG_PROCESS or DEBUG_ONLY_THIS_PROCESS creation flags. This leads to the endless loop that I described above. Is this an ICorDebug API design flaw?  Not really. Just an oversight in my opinion - The API designers missed one scenario. Maybe in the next version of the CLR this will be fixed (I do not know for sure).

 

 

Some excellent resources on IFEO -

 

[MSDN]	Launching the debugger automatically (How to launch VS debugger using IFEO)
[Mike Stall]	IFEO and managed debugging
[Raymond Chen]	Image File Execution options just inserts the debugger in front of the command line
[Gregg Miskelly]	Inside ‘Image File Execution Options’ debugging
[Junfeng Zhang]	Image File Execution Options (talks about the other not-so-well documented IFEO options)

 

 

 

Exercise for reader -

 

Does IFEO work with other Win32 APIs like ShellExecute, CreateProcessAsUser, CreateProcessWithLogonW and CreateProcessWithTokenW?

 

 

TIP of the day -

 

Question - System services can launch before the user has a chance to log on. So how do you debug the startup code of these system services?

 

Answer - Put the machine under kernel Debugger (KD), use IFEO to launch the service under NTSD (use ntsd’s  “-d” option to pipe the ntsd output to KD) and reboot the machine. When the system service launches, it will be launched under ntsd.  The ntsd debugger will automatically cause it break into KD when it encounters the initial loader breakpoint. The debugging session will begin in user mode automatically (yipee!). After you are done debugging, switch control to KD by issuing “.breakin” command.

 

 

 

NOTE - I wrote this up more than a year back and for some mysterious reasons forgot to post it! Much of what is written here applies to Vista UAC accounts as well.

QUESTION - Under Windows Server 2003, can a “limited user” impersonate an “administrator”?

ANSWER –  As you are aware, there are four impersonation levels, each of which indicates the degree to which one user can impersonate another – SecurityAnonymous, SecurityIdentification, SecurityImpersonation, and SecurityDelegation. A limited user on Windows Server 2003 (or UAC admin on vista) needs SE_IMPERSONATE_PRIVILEGE enabled in order to impersonate the context of an “administrator” at any level above “SecurityIdentification”.

Consider this snippet of code executed in the context of a limited user (non-admin). An admin’s token is first obtained and duplicated to create an impersonation token at security level “SecurityImpersonation“. The limited user then attempts to impersonate the admin. If impersonation succeeds, then it tries some operation that requires admin privileges (In this case, creating a key under HKLM\Software\Microsoft with write access).  

void wmain ()

{

HANDLE hPrivToken ;

HANDLE hPrivTokenImperson ; 

if ( 0 == LogonUser(

L”administrator”,

L”MyMachine”,

L”MyPassWord”,

LOGON32_LOGON_INTERACTIVE,

LOGON32_PROVIDER_DEFAULT, 

& hPrivToken ) )

{

wprintf( L”Failed to logon admin. Last Error = %d\n”, GetLastError() ) ;

return ;

} 

if ( 0 == DuplicateTokenEx(

hPrivToken,

TOKEN_QUERY | TOKEN_IMPERSONATE,

NULL, 

SecurityImpersonation,

TokenImpersonation, 

& hPrivTokenImperson ) )

{

wprintf( L”Failed to get impersonation token. Last Error = %d\n”, GetLastError());

return ;

} 

if ( 0 == SetThreadToken( NULL, hPrivTokenImperson ) )

{

wprintf( L”Failed to impersonate admin. Last Error = %d\n”, GetLastError());

return ; 

} 

// Do something that requires admin privileges………… //

if ( ERROR_SUCCESS != RegCreateKeyEx(

HKEY_LOCAL_MACHINE,

L”Software\\Microsoft\\RandomKey”,

0,

NULL, 

REG_OPTION_NON_VOLATILE,

KEY_READ | KEY_WRITE,

NULL,

& hKey,

& dwDisposition ) )

{

wprintf( L”RegCreateKeyEx FAILED. Last Error = %d\n”,  GetLastError());

}

else

{

wprintf( L”RegCreateKeyEx SUCCEEDED.\n” ) ;

} 

SetThreadToken( NULL, NULL ) ; // revert back

}

Under Windows XP, the limited user is successfully able to impersonate the admin at level SecurityImpersonation. However on Windows 2003 and Vista things are a bit different because of the security changes related to SE_IMPERSONATE_PRIVILEGE that I described above. Although the first call to SetThreadToken() succeeds, the impersonation is being sneakily carried out at level SecurityIdentification. As a result of which your privileged operation (writing to HKLM in this case) fails with an access-denied error. Sweet! On Windows Server 2003 and above, the ability to impersonate at SecurityImpersonation and above is given only to Network Service, Local Service, Local System and Administrators (On Vista you need to be running elevated as well to have SE_IMPERSONATE_PRIVILEGE enabled in the token).

DllImportAttribute.PreserveSig -

• Mainly used in PInvoke Signatures
• By default, while using DllImportAttribute to p-invoke, the DllImportAttribute.PreserveSig is set to true.
• If false: Converts returned HRESULTS from unmanaged signature to managed exceptions if they are not S_OK.
• If true: Does not convert returned HRESULT to exceptions.

PreserveSigAttribute (pseudo custom attribute)

• This is mainly used in COM interface definitions in managed code.
• By default, for COM interface methods, the CLR does HRESULT to exception transformations (opposite behavior to that of pinvoke).
• If attribute applied: Does not convert returned HRESULT to exceptions.
• If attribute not applied: Default behavior of HRESULT to exception transformations.

My short-term memory is bad and I tend to forget things as soon as I learn them. So I jotted this down while I was reading Adam Nathan’s excellent manuscript on COM interop. Hope you find this tidbit useful as well.

Since I have no energy to type a long post today (sigh), I’ll spare you the gory details on the cause of this error. Here are the main reasons why you would be seeing this error -

• While expecting to debug a v2.0.xxxx managed app, you have forgotten to implement ICorDebugManagedCallback2 in your managed callback handler. The managed callback handler must implement both ICorDebugManagedCallback and ICorDebugManagedCallback2 in such cases.

class ManagedEngine

: public ICorDebugManagedCallback,

  public ICorDebugManagedCallback2

{

} 

• While implementing QueryInterace(..), you probably forgot to account for ICorDebugManagedCallback2.

HRESULT __stdcall ManagedEngine::QueryInterface(const IID & iid, void** ppv)

{

if (iid == IID_IUnknown)

* ppv = static_cast<ICorDebugManagedCallback*>(this);

 

else if (iid == IID_ICorDebugManagedCallback)

* ppv = static_cast<ICorDebugManagedCallback*>(this);

 

else if (iid == IID_ICorDebugManagedCallback2)

* ppv = static_cast<ICorDebugManagedCallback2*>(this);

 

else

{

* ppv = NULL;

return E_NOINTERFACE;

}

 

reinterpret_cast<IUnknown*>(*ppv)->AddRef();

 

return S_OK;

}

Ever seen this run-time failure? - “The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention“

Whenever I get sloppy, I tend to write awful code as shown below -

FARPROC pDllFunc = NULL;

HRESULT (* pDllGetReqRuntimeVer) (LPWSTR, LPWSTR, DWORD, DWORD*);

HMODULE hModuleMscoree = LoadLibrary(L“mscoree.dll”); 

if(NULL != hModuleMscoree)

{

pDllFunc = GetProcAddress(hModuleMscoree, “GetRequestedRuntimeVersion”);

if(NULL != pDllFunc)

{

pDllGetReqRuntimeVer = (HRESULT(*)(LPWSTR, LPWSTR, DWORD, DWORD*))pDllFunc;

hr = pDllGetReqRuntimeVer(

thrParam->cmdLineSw.pwszImage,

wszDebuggeeVer,

sizeof(wszDebuggeeVer)/sizeof(wszDebuggeeVer[0]),

& dws);

:  

:  

:

}

}

When sloppy, I forget to use the correct calling convention while declaring function pointers. Then I run into the above run-time error while calling into various functions exported by dlls (mscoree!GetRequestedRuntimeVersion in above example). Most Win32 APIs are declared as __stdcall which expects the “callee” to cleanup the stack. I compile with /Gd (uses __cdecl as default convention). Because I forgot of mark the function pointer with __stdcall, the run-time is now expecting the “caller” to cleanup the stack. Fortunately I’m bailed out by the above run time check.

To correct this the, function pointer must be declared and used as follows -

FARPROC pDllFunc = NULL;

HRESULT (__stdcall * pDllGetReqRuntimeVer) (LPWSTR, LPWSTR, DWORD, DWORD*);

HMODULE hModuleMscoree = LoadLibrary(L“mscoree.dll”); 

if(NULL != hModuleMscoree)

{

pDllFunc = GetProcAddress(hModuleMscoree, “GetRequestedRuntimeVersion”);

if(NULL != pDllFunc)

{

pDllGetReqRuntimeVer = (HRESULT(__stdcall *)

(LPWSTR, LPWSTR, DWORD, DWORD*))pDllFunc;

hr = pDllGetReqRuntimeVer(

thrParam->cmdLineSw.pwszImage,

wszDebuggeeVer,

sizeof(wszDebuggeeVer)/sizeof(wszDebuggeeVer[0]),

& dws);

:

:

:

}

}

Given below are the native disassemblies (function prolog and epilog) when you use __cdecl and __stdcall respectively. As you can see, for the function marked as __cdecl, the “caller” is cleaning up the stack. While the “callee” does the needful when __stdcall is used.

When using __cdecl 

00412DB1 mov esi,esp

00412DB3 lea eax,[dws]

00412DB9 push eax

00412DBA push 80h

00412DBF lea ecx,[ebp-168h]

00412DC5 push ecx

00412DC6 mov edx,dword ptr [ebp-48h]

00412DC9 mov eax,dword ptr [edx+10h]

00412DCC push eax

00412DCD call dword ptr [ebp-61Ch]

00412DD3 add esp,10h ß———– stack is cleaned by caller!

00412DD6 cmp esi,esp

00412DD8 call @ILT+840(__RTC_CheckEsp) (41134Dh) ß—— The run time check

00412DDD mov dword ptr [ebp-18h],eax

When using __stdcall 

00412DB1 mov esi,esp

00412DB3 lea eax,[dws]

00412DB9 push eax

00412DBA push 80h

00412DBF lea ecx,[ebp-168h]

00412DC5 push ecx

00412DC6 mov edx,dword ptr [ebp-48h]

00412DC9 mov eax,dword ptr [edx+10h]

00412DCC push eax

00412DCD call dword ptr [ebp-61Ch]

00412DD3 cmp esi,esp ß——— At this stage callee has already cleaned up stack

00412DD5 call @ILT+840(__RTC_CheckEsp) (41134Dh) ß—— The run time check

00412DDA mov dword ptr [ebp-18h],eax

I should be paying more attention to the calling conventions and instead of relying on this run-time check. But it is nice to have this safety net.