Hi,
We have a DFSR hub and spoke topology with 50 members across 45 sites connected with a mixture of IPSEC VPN & MPLS. 1 Hub server running Server 2012 Standard, all others are either Server 2008 R2 Standard or Enterprise.
All servers have the latest patchlevel
All servers are in one domain
Theres no problem with normal AD-Replication
No obvious errors from dcdiag or netdiag
1 Namespace
61 Replicated Folders containing a mixture of office documents, images and videos
Problem:
There are large backlogs reported by DFSR diag
Large files replicate extremely slowly or not at all (for example a 984MB file created 32 days ago has still not reached 40 out of 50 sites.)
Many DFSR Event ID 5014 warnings are logged on all members ('Error: 1726 (The remote procedure call failed.)
The hub server's DFSR debug logs contain many entries stating 'Failed to reserve file transfer server context, no more contexts are available'. These messages seem to corrospond with the DFSR event 5014 messages.
Example debug log:
20131201 18:03:46.908 3696 SRTR 2772 InitializeFileTransferAsyncState::ProcessIoCompletion 0000003401A87920
20131201 18:03:46.908 3696 RPCN 1889 [WARN] NetworkGlobals::ReserveFileTransfer Failed to reserve file transfer server context, no more contexts available. updateName:removedfilename.ods uid:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1665723 gvsn:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1665723
connId:{1F655F11-0101-4833-805E-680361C778FE} csId:{ACF6E46C-09CE-4E7D-85F5-10206356C231} totalContexts:128
20131201 18:03:46.908 3696 SRTR 3011 [WARN] InitializeFileTransferAsyncState::ProcessIoCompletion Failed to initialize a file transfer. connId:{1F655F11-0101-4833-805E-680361C778FE} rdc:1 uid:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1665723 gsvn:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1665723
completion:0 ptr:0000003401A87920 Error:[Error:9078(0x2376) InitializeFileTransferAsyncState::ProcessIoCompletion servertransport.cpp:2884 3696 C All server file transfer contexts are currently busy]
20131201 18:03:46.908 3696 SRTR 2701 InitializeFileTransferAsyncState::~InitializeFileTransferAsyncState this:0000003401A87920
20131201 18:03:46.908 3696 SRTR 2772 InitializeFileTransferAsyncState::ProcessIoCompletion 0000003401A85D00
20131201 18:03:46.908 3696 RPCN 1889 [WARN] NetworkGlobals::ReserveFileTransfer Failed to reserve file transfer server context, no more contexts available. updateName:removedfilename.xls uid:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1608783 gvsn:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1608783
connId:{1F655F11-0101-4833-805E-680361C778FE} csId:{ACF6E46C-09CE-4E7D-85F5-10206356C231} totalContexts:128
20131201 18:03:46.908 3696 SRTR 3011 [WARN] InitializeFileTransferAsyncState::ProcessIoCompletion Failed to initialize a file transfer. connId:{1F655F11-0101-4833-805E-680361C778FE} rdc:1 uid:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1608783 gsvn:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1608783
completion:0 ptr:0000003401A85D00 Error:[Error:9078(0x2376) InitializeFileTransferAsyncState::ProcessIoCompletion servertransport.cpp:2884 3696 C All server file transfer contexts are currently busy]
20131201 18:03:46.908 3696 SRTR 2701 InitializeFileTransferAsyncState::~InitializeFileTransferAsyncState this:0000003401A85D00
20131201 18:03:46.908 3696 SRTR 2772 InitializeFileTransferAsyncState::ProcessIoCompletion 0000003401A87530
20131201 18:03:46.908 3696 RPCN 1889 [WARN] NetworkGlobals::ReserveFileTransfer Failed to reserve file transfer server context, no more contexts available. updateName:removedfilename.pdf uid:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1665542 gvsn:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1665542
connId:{1F655F11-0101-4833-805E-680361C778FE} csId:{ACF6E46C-09CE-4E7D-85F5-10206356C231} totalContexts:128
20131201 18:03:46.908 3696 SRTR 3011 [WARN] InitializeFileTransferAsyncState::ProcessIoCompletion Failed to initialize a file transfer. connId:{1F655F11-0101-4833-805E-680361C778FE} rdc:1 uid:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1665542 gsvn:{80EBA14C-E68B-438F-AC8F-22D9B32BF51D}-v1665542
completion:0 ptr:0000003401A87530 Error:[Error:9078(0x2376) InitializeFileTransferAsyncState::ProcessIoCompletion servertransport.cpp:2884 3696 C All server file transfer contexts are currently busy]
20131201 18:03:46.908 3696 SRTR 2701 InitializeFileTransferAsyncState::~InitializeFileTransferAsyncState this:0000003401A87530
20131201 18:03:46.908 3696 SRTR 2772 InitializeFileTransferAsyncState::ProcessIoCompletion 0000003401A83510
20131201 18:03:46.908 3696 RPCN 1889 [WARN] NetworkGlobals::ReserveFileTransfer Failed to reserve file transfer server context, no more contexts available. updateName:removedfilename.doc uid:{1DA51289-C37E-4A07-A625-90933A0F808D}-v1412159 gvsn:{1DA51289-C37E-4A07-A625-90933A0F808D}-v1412159
connId:{4344F679-9507-4D15-9730-DFC73EFAD456} csId:{ACF6E46C-09CE-4E7D-85F5-10206356C231} totalContexts:128
20131201 18:03:46.924 3696 SRTR 3011 [WARN] InitializeFileTransferAsyncState::ProcessIoCompletion Failed to initialize a file transfer. connId:{4344F679-9507-4D15-9730-DFC73EFAD456} rdc:1 uid:{1DA51289-C37E-4A07-A625-90933A0F808D}-v1412159 gsvn:{1DA51289-C37E-4A07-A625-90933A0F808D}-v1412159
completion:0 ptr:0000003401A83510 Error:[Error:9078(0x2376) InitializeFileTransferAsyncState::ProcessIoCompletion servertransport.cpp:2884 3696 C All server file transfer contexts are currently busy]
20131201 18:03:46.924 3696 SRTR 2701 InitializeFileTransferAsyncState::~InitializeFileTransferAsyncState this:0000003401A83510
20131201 18:03:46.924 3696 SRTR 2772 InitializeFileTransferAsyncState::ProcessIoCompletion 0000003401A864E0
I have tried the following solutions:
Installed latest NIC-drivers on each server
Installed latest DFSR binaries
No teaming at the NICs
Disabled task offloading
Increased the size of the staging folder to 16GB
Excluded DFSRs.exe from Antivirus
Excluded 'System Volume Information' directory from Antivirus
No backups are running
Increased Bandwidth allocation
Tried RDC disabled & Enabled - no difference
Removed any FSRM screens
All members are pre-staged using the Robocopy method
reviewed http://blogs.technet.com/b/askds/archive/2007/10/05/top-10-common-causes-of-slow-replication-with-dfsr.aspx
reviewed http://support.microsoft.com/default.aspx?scid=kb%3bEN-US%3b958802 to get the latest DFS-Files
Any suggestions please?