Environment:
2012 R2 hypervisor, three VMs (SBS 2011, 2012 R2, 2008 R2 respectively).
All VHDs and VMs stored on HV's D: drive.
Arcserve Backup 17.5 SP1
Background:
SBS 2011 server blue-screened, seems to be due to corruption in/with VHD for system drive. Boot order was lost, now fixed and server is running normally.
Issue:
SBS server seems to be experiencing issues with VSS writers & event 51s for errors during paging operations. Backups failing as a result. Arcserve leverages Windows VSS writers to take snapshots which are then backed up.
Steps taken so far:
Usual VSS troubleshooting - restart associated services, restart server, re-register DLLs.
CHKDSK /R & SFC /SCANNOW on affected server.
Shut down server, removed and re-attached VHDs.
Events/logs:
DISKPART> list vol
Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 E DVD-ROM 0 B No Media
Volume 1 System Rese NTFS Partition 100 MB Healthy System
Volume 2 C NTFS Partition 99 GB Healthy Boot
Volume 3 D Data NTFS Partition 449 GB Healthy
PS C:\Windows\system32> vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Waiting for responses.
These may be delayed if a shadow copy is being prepared.
Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error
Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error
Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error
Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {0fd2f208-866a-4633-b2a2-7509c646dbc0}
State: [1] Stable
Last error: No error
Writer name: 'SharePoint Services Writer'
Writer Id: {da452614-4858-5e53-a512-38aab25c61ad}
Writer Instance Id: {84b03184-534f-4715-a379-bf082a56c336}
State: [1] Stable
Last error: No error
Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {54ef1739-2650-4e34-9c40-100ba005f7ca}
State: [1] Stable
Last error: No error
Writer name: 'FRS Writer'
Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
Writer Instance Id: {a5b29068-d666-4d71-a870-e008fd9422bf}
State: [1] Stable
Last error: No error
Writer name: 'FSRM Writer'
Writer Id: {12ce4370-5bb7-4c58-a76a-e5d5097e3674}
Writer Instance Id: {171f9fe1-ba82-4c03-b26c-14b8118dcfcb}
State: [1] Stable
Last error: No error
Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {866b92e2-bfe3-4d41-80a7-5c0d9195cb25}
State: [1] Stable
Last error: No error
Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {166b7390-bc7a-4b41-8cec-23995a4c973c}
State: [1] Stable
Last error: No error
Writer name: 'MSSearch Service Writer'
Writer Id: {cd3f2362-8bef-46c7-9181-d62844cdc0b2}
Writer Instance Id: {b3d7ddf7-38d6-4d3c-9bf2-b0ba11ff48c4}
State: [10] Failed
Last error: Timed out
Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {1812c8f8-af21-44ab-a38d-ef7f5b89acc6}
State: [1] Stable
Last error: No error
Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {65e8d950-69af-4f63-bed0-0875ccc9f22e}
State: [1] Stable
Last error: No error
Writer name: 'TS Gateway Writer'
Writer Id: {368753ec-572e-4fc7-b4b9-ccd9bdc624cb}
Writer Instance Id: {3cf3256a-5a12-4fe0-989a-25f4c3b892e9}
State: [10] Failed
Last error: Timed out
Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {de5764c6-6181-465f-a7b2-46a1266adfd8}
State: [10] Failed
Last error: Timed out
Writer name: 'Dhcp Jet Writer'
Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}
Writer Instance Id: {517b3010-6b88-4dc0-b609-753bb57a2147}
State: [10] Failed
Last error: Non-retryable error
Writer name: 'NTDS'
Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Instance Id: {fd4ae2b1-2821-4f53-997e-f75d65969b79}
State: [10] Failed
Last error: Non-retryable error
Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {5ded8fd7-7860-43aa-89a4-6c9dd7cbf98c}
State: [1] Stable
Last error: No error
Writer name: 'Certificate Authority'
Writer Id: {6f5b15b5-da24-4d88-b737-63063e3a1f86}
Writer Instance Id: {023b1fcb-ab11-482a-bea6-40c35c1cf28c}
State: [1] Stable
Last error: No error
Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {548d95cd-84a9-499c-95e7-d5b7a29e1c57}
State: [1] Stable
Last error: No error
Writer name: 'NPS VSS Writer'
Writer Id: {35e81631-13e1-48db-97fc-d5bc721bb18a}
Writer Instance Id: {b26bf05e-f738-4614-9f66-7a0ca282d6f1}
State: [10] Failed
Last error: Timed out
Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {1e102557-f63a-4b1f-aa1c-73032fa4eccb}
State: [1] Stable
Last error: No error
Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {8d5311e4-cb6f-4a40-8f2c-a97c4b75deef}
State: [1] Stable
Last error: No error
Errors on HV:
* AE0603 2018-08-02 22:14:56 Failed to create VSS shadow copy for the VM on the Hyper-V host machine. (Hyper-V Host Machine=[HV], VM=[SBS])
* Checkpoint operation for '[SBS]' failed. (Virtual machine ID 4D9E428E-8CDB-42E1-9443-0538DB6681C0)
* Could not create backup checkpoint for virtual machine '[SBS]': The shadow copy provider had an unexpected error while trying to process the specified operation. (0x8004230F). (Virtual machine ID 4D9E428E-8CDB-42E1-9443-0538DB6681C0)
* VSS writers inside virtual machine '[SBS]' failed to perform BackupComplete to its shadow copy (VSS snapshot) set: A function call was made when the object was in an incorrect state for that function (0x80042301). (Virtual machine ID 4D9E428E-8CDB-42E1-9443-0538DB6681C0)
Errors on VM:
System
* An error was detected on device \Device\Harddisk2\DR2 during a paging operation.
* An error was detected on device \Device\Harddisk3\DR3 during a paging operation.
* The shadow copies of volume F: were aborted during detection because a critical control file could not be opened.
* The shadow copies of volume H: were aborted during detection because a critical control file could not be opened.
Application:
* Not all the shadow volumes arrived in the guest operating system. [vmicvss, 19]
* Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details PreFinalCommitSnapshots({8c1560b1-750c-41c3-9e5c-61e076c925d7}, 3) [hr = 0x800705b4,
This operation returned because the timeout period expired.]. [VSS, 12293] Operation: Executing Asynchronous Operation Context: Current
State: DoSnapshotSet
* Volume Shadow Copy Service warning: ESENT ERROR {b2014c9e-8711-4c5c-a5a9-3cf384484757} NTDS (ditto for Dhcp Jet Writer and Certificate Authority): -2404. hr = 0x00000000, The operation completed successfully.
Operation:
OnAbort event
Aborting Writer
Context:
Execution Context: Jet Writer
Writer Class Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Name: NTDS (ditto for Dhcp Jet Writer and Certificate Authority)
Writer Class Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Name: NTDS (ditto for Dhcp Jet Writer and Certificate Authority)
Writer Instance ID: {fd4ae2b1-2821-4f53-997e-f75d65969b79}
Any help would be much appreciated! Need to get this server backed up so that I can sleep soundly again :)