Calling managed code from unmanaged win32 assembly dll - crash
- by JustGreg
I'm developing a serial port dll in win32 assembly (MASM32). It has its own thread checking multiple events and at a specified buffer treshold it'd notify the managed main application by calling a callback function. It just a call with no arguments/return value.
At startup the main application stores the callback function's address by calling a function in the dll:
pCallBackFunction dd 0
SetCallBackPointer proc pcb:DWORD
mov eax, pcb
mov pCallBackFunction, eax
call DWORD ptr pCallBackFunction ; verify it immediately
ret
SetCallBackPointer endp
The upper function immediately calls back the managed application callback routine for verification purposes. It is working fine.
However, when I place the call instruction to other functions in the dll it crashes the application. It doesn't matter if the call is in a simple function or in the threadproc of the dll. For example:
OpenPort proc pn:byte,br:dword, inputbuffersize: dword, outputbuffersize:dword, tresholdsize: dword
LOCAL dcb: DCB
LOCAL SerialTimeOuts: COMMTIMEOUTS
call DWORD ptr pCallBackFunction
xor eax, eax
mov al, pn
mov [com_port+3],al
etc. etc.
will crash at call DWORD ptr pCallBackFunction always. Since I call SetCallBackPointer first to store a valid address in pCallBackFunction, it should have a valid address.
My managed app is written in C# and the relevant part is:
public partial class Form1 : Form
{
public delegate void CallBackDelegate();
public static CallBackDelegate mydelegate;
[DllImport("serialport.dll")]
private static extern void SetCallBackPointer(CallBackDelegate Delegate);
[DllImport("serialport.dll")]
public static extern int OpenPort(byte com, uint br, uint inbufsize, uint outbufsize, uint treshsize);
public Form1()
{
InitializeComponent();
mydelegate =new CallBackDelegate(CallbackFunction);
SetCallBackPointer(mydelegate);
unsafe
{
int sysstat;
int hResult;
hResult = OpenPort(Convert.ToByte('5'), 9600, 306, 4, 4);
}
}
public static void CallbackFunction()
{
MessageBox.Show( "CallBack Function Called by Windows DLL");
}
The VS debugger reported that the dll had tried to read/write from/to a protected memory address. But when calling SetCallBackPointer there is no such problem. What am I doing wrong here?
Any tips would be great!
