After enabling MPIO for ISCSI one of our HyperV Clustered Windows 2008 R2 Datacenter x64 servers has been unstable, crashing 3-5 times daily. I've had to take it out of service and after analyzing the dump file it seems there is an issue with msiscsi.sys and overlapping drivers 'tssecsrv' and 'tdtcp.sys' This all started occuring after we enabled MPIO to connect to our Promise Vessraid ISCSI using MultiPath I/O over iSCSI. We are connecting using two intel network cards to 2 gigabit ports on the VessRaid. Other than the Intel network drivers (v14.7 package version 11.0.103 7/14/2009) and Intel Matrix Storage Array 10R drivers, everything is Microsoft (OS, patches, iscsi initiator, HyperV, MPIO) Any ideas and where to start?
Analyze -v below:
Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [c:\windows\memory.dmp]
Kernel Summary Dump File: Only kernel address space is available
Symbol search path is: SRV*n:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: n:\i386
Windows 7 Kernel Version 7600 MP (16 procs) Free x64
Product: LanManNt, suite: TerminalServer DataCenter SingleUserTS
Built by: 7600.16385.amd64fre.win7_rtm.090713-1255
Machine Name:
Kernel base = 0xfffff800`0180c000 PsLoadedModuleList = 0xfffff800`01a49e50
Debug session time: Mon Dec 14 20:12:27.351 2009 (GMT-8)
System Uptime: 0 days 18:44:22.000
Loading Kernel Symbols
...............................................................
................................................................
...................
Loading User Symbols
Loading unloaded module list
.............
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck A, {4, 2, 1, fffff8000188fac1}
Probably caused by : msiscsi.sys ( msiscsi!iSpProcessWMIRequestTimeout+71e )
Followup: MachineOwner
---------
8: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 0000000000000004, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: fffff8000188fac1, address which referenced memory
Debugging Details:
------------------
OVERLAPPED_MODULE: Address regions for 'tssecsrv' and 'tdtcp.sys' overlap
WRITE_ADDRESS: 0000000000000004
CURRENT_IRQL: 2
FAULTING_IP:
nt!IoReleaseRemoveLockEx+21
fffff800`0188fac1 f0834304ff lock add dword ptr [rbx+4],0FFFFFFFFh
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
BUGCHECK_STR: 0xA
PROCESS_NAME: System
TRAP_FRAME: fffff8800233d180 -- (.trap 0xfffff8800233d180)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
rdx=fffffa8055864210 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8000188fac1 rsp=fffff8800233d310 rbp=0000000000000000
r8=0000000000000020 r9=000000000000000c r10=fffff8000180c000
r11=fffff8800233d2e0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei ng nz ac pe cy
nt!IoReleaseRemoveLockEx+0x21:
fffff800`0188fac1 f0834304ff lock add dword ptr [rbx+4],0FFFFFFFFh ds:00000000`00000004=????????
Resetting default scope
LAST_CONTROL_TRANSFER: from fffff8000187d469 to fffff8000187df00
STACK_TEXT:
fffff880`0233d038 fffff800`0187d469 : 00000000`0000000a 00000000`00000004 00000000`00000002 00000000`00000001 : nt!KeBugCheckEx
fffff880`0233d040 fffff800`0187c0e0 : fffffa80`5718f020 00000000`00000000 00000000`00000000 fffff880`03a82366 : nt!KiBugCheckDispatch+0x69
fffff880`0233d180 fffff800`0188fac1 : fffffa80`58de9030 fffff800`01a215f8 fffffa80`24f54040 fffffa80`24f54148 : nt!KiPageFault+0x260
fffff880`0233d310 fffff880`036b72b6 : fffffa80`2c662a20 fffffa80`2c662a20 00000000`00000002 fffffa80`55864210 : nt!IoReleaseRemoveLockEx+0x21
fffff880`0233d380 fffff880`036b668b : fffffa80`26a72018 fffff880`03616200 00000000`00000000 00000000`00000000 : msiscsi!iSpProcessWMIRequestTimeout
+0x71e
fffff880`0233d420 fffff800`01868def : 00000000`00000000 00000000`00000002 00000000`00000001 fffffa80`254debd8 : msiscsi!iSpTickHandler+0x11f
fffff880`0233d460 fffff800`01889fa6 : 00000000`00000002 fffff880`0233d618 00000000`00000000 00000000`00000005 : nt!IopTimerDispatch+0x132
fffff880`0233d570 fffff800`01889326 : fffffa80`263a6440 fffffa80`263a6440 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x66
fffff880`0233d5e0 fffff800`01889e7e : 0000009d`127c5d6d fffff880`0233dc58 00000000`0041fc6b fffff880`023182e8 : nt!KiProcessExpiredTimerList+0xc6
fffff880`0233dc30 fffff800`01889697 : 0000009d`62ca30cb fffff880`0041fc6b fffffa80`57195bf0 00000000`0000006b : nt!KiTimerExpiration+0x1be
fffff880`0233dcd0 fffff800`018866fa : fffff880`02315180 fffff880`023202c0 00000000`00000002 fffff880`00000000 : nt!KiRetireDpcList+0x277
fffff880`0233dd80 00000000`00000000 : fffff880`0233e000 fffff880`02338000 fffff880`0233dd40 00000000`00000000 : nt!KiIdleLoop+0x5a
STACK_COMMAND: kb
FOLLOWUP_IP:
msiscsi!iSpProcessWMIRequestTimeout+71e
fffff880`036b72b6 33d2 xor edx,edx
SYMBOL_STACK_INDEX: 4
SYMBOL_NAME: msiscsi!iSpProcessWMIRequestTimeout+71e
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: msiscsi
IMAGE_NAME: msiscsi.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bcad7
FAILURE_BUCKET_ID: X64_0xA_msiscsi!iSpProcessWMIRequestTimeout+71e
BUCKET_ID: X64_0xA_msiscsi!iSpProcessWMIRequestTimeout+71e
Followup: MachineOwner
---------
Btw Intel 82576/82574L v11.0.103 (7/14/2009) used