W2K12-R deduplication optimization job stuck at 100% for days

Hello,

I've W2K12-R server (VM, 4 vCPU, 14 GB vRAM) with a 7 TB D: drive, deduplication is enabled for this drive. I use this drive for backing up virtual machines, at the moment 3,6 TB are used. Each night ~2 TB new data is backed up, only the C: drives of VMs, so there is a very high dedup rate, these new 2 TB are usually deduplicated during daytimes.

Capacity                 : 6.64 TB
FreeSpace               : 3.01 TB
UsedSpace              : 3.63 TB
UnoptimizedSize        : 110.74 TB
SavedSpace             : 107.1 TB
SavingsRate             : 96 %

Since a few weeks I've the problem that ThroughputOptimization jobs (all Optimization jobs) do not finish. The job is able to free space but it stays at 100% for days.

Although I've set the DurationHours to 6 or 9 hours, the scheduled job does not end and thus no new Optimization job starts. That leads to a completely filled D drive because the space that gets filled by backup jobs during the night  is not freed during the planned time frame.

Event Log from the start of the job:

I don't see further information about job 39D260C4-32D0-4511-8982-598DA53DB423 in event log. In resource monitor I see that fsdmhost has very little disk activity (600 Bytes/s), but a cpu load of 25% (which is one of the four vCPUs).

I stopped the backups this week and started a manual full Optimization job, which stays at 100% since more than one day now.

Any ideas what to check? Any patch I can apply?

Some details:

23.07.014 - 8:40 Uhr


Get-DedupStatus

FreeSpace    SavedSpace   OptimizedFiles     InPolicyFiles      Volume
---------    ----------   --------------     -------------      ------
3.01 TB      107.1 TB     12232              12232              D:


Get-DedupStatus | fl


Volume                             : D:
VolumeId                           : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
Capacity                           : 6.64 TB
FreeSpace                          : 3.01 TB
UsedSpace                          : 3.63 TB
UnoptimizedSize                    : 110.74 TB
SavedSpace                         : 107.1 TB
SavingsRate                        : 96 %
OptimizedFilesCount                : 12232
OptimizedFilesSize                 : 110.58 TB
OptimizedFilesSavingsRate          : 96 %
InPolicyFilesCount                 : 12232
InPolicyFilesSize                  : 110.58 TB
LastOptimizationTime               : 23.07.2014 04:44:59
LastOptimizationResult             : 0x8056533D
LastOptimizationResultMessage      : The operation was cancelled.
LastGarbageCollectionTime          : 20.07.2014 05:26:12
LastGarbageCollectionResult        : 0x00000000
LastGarbageCollectionResultMessage : The operation completed successfully.
LastScrubbingTime                  : 20.07.2014 16:06:29
LastScrubbingResult                : 0x00000000
LastScrubbingResultMessage         : The operation completed successfully.


Get-DedupVolume

Enabled            UsageType          SavedSpace           SavingsRate          Volume
-------            ---------          ----------           -----------          ------
True               Default            107.1 TB             96 %                 D:


Get-DedupVolume |fl


Volume                   : D:
VolumeId                 : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
Enabled                  : True
UsageType                : Default
DataAccessEnabled        : True
Capacity                 : 6.64 TB
FreeSpace                : 3.01 TB
UsedSpace                : 3.63 TB
UnoptimizedSize          : 110.74 TB
SavedSpace               : 107.1 TB
SavingsRate              : 96 %
MinimumFileAgeDays       : 0
MinimumFileSize          : 32768
NoCompress               : False
ExcludeFolder            :
ExcludeFileType          :
ExcludeFileTypeDefault   : {edb, jrs}
NoCompressionFileType    : {asf, mov, wma, wmv...}
ChunkRedundancyThreshold : 100
Verify                   : False
OptimizeInUseFiles       : False
OptimizePartialFiles     : False



Get-DedupMetadata | fl


Volume                         : D:
VolumeId                       : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
StoreId                        : {1E87DD60-0265-4726-9EDD-81FCE4E9D70A}
DataChunkCount                 : 151954987
DataContainerCount             : 3983
DataChunkAverageSize           : 24.35 KB
DataChunkMedianSize            : 0 B
DataStoreUncompactedFreespace  : 0 B
StreamMapChunkCount            : 95398
StreamMapContainerCount        : 1944
StreamMapAverageDataChunkCount :
StreamMapMedianDataChunkCount  :
StreamMapMaxDataChunkCount     :
HotspotChunkCount              : 1048520
HotspotContainerCount          : 54
HotspotMedianReferenceCount    :
CorruptionLogEntryCount        : 0
TotalChunkStoreSize            : 3.6 TB



Get-DedupSchedule

Enabled    Type               StartTime              Days               Name
-------    ----               ---------              ----               ----
True       Optimization                                                 BackgroundOptimization
False      GarbageCollection  07:30                  {Monday, Tuesda... DailyGarbageCollection
True       Optimization       09:00                  {Sunday, Monday... ThroughputOptimization
False      Optimization       14:30                  {Sunday, Monday... ThroughputOptimization-2
True       GarbageCollection  16:00                  Saturday           WeeklyGarbageCollection
True       Scrubbing          03:45                  Sunday             WeeklyScrubbing


Get-DedupSchedule | fl


Name                     : BackgroundOptimization
Enabled                  : True
Type                     : Optimization
Days                     :
Start                    :
DurationHours            :
StopWhenSystemBusy       : True
Memory                   : 25 %
Priority                 : Low
InputOutputThrottleLevel : Low
ScheduledTask            : \Microsoft\Windows\Deduplication\BackgroundOptimization
Full                     :
ReadOnly                 :

Name                     : DailyGarbageCollection
Enabled                  : False
Type                     : GarbageCollection
Days                     : {Monday, Tuesday, Wednesday, Thursday...}
Start                    : 17.06.2014 07:30:00
DurationHours            : 0
StopWhenSystemBusy       : False
Memory                   : 50 %
Priority                 : Normal
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\DailyGarbageCollection
Full                     : False
ReadOnly                 : False

Name                     : ThroughputOptimization
Enabled                  : True
Type                     : Optimization
Days                     : {Sunday, Monday, Tuesday, Wednesday...}
Start                    : 14.07.2014 09:00:00
DurationHours            : 6
StopWhenSystemBusy       : False
Memory                   : 40 %
Priority                 : Normal
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\ThroughputOptimization
Full                     : False
ReadOnly                 : False

Name                     : ThroughputOptimization-2
Enabled                  : False
Type                     : Optimization
Days                     : {Sunday, Monday, Tuesday, Wednesday...}
Start                    : 14.07.2014 14:30:00
DurationHours            : 9
StopWhenSystemBusy       : False
Memory                   : 40 %
Priority                 : Normal
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\ThroughputOptimization-2
Full                     : False
ReadOnly                 : False

Name                     : WeeklyGarbageCollection
Enabled                  : True
Type                     : GarbageCollection
Days                     : Saturday
Start                    : 20.06.2014 16:00:00
DurationHours            : 26
StopWhenSystemBusy       : True
Memory                   : 50 %
Priority                 : High
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\WeeklyGarbageCollection
Full                     : True
ReadOnly                 : False

Name                     : WeeklyScrubbing
Enabled                  : True
Type                     : Scrubbing
Days                     : Sunday
Start                    : 17.05.2014 03:45:00
DurationHours            : 0
StopWhenSystemBusy       : True
Memory                   : 50 %
Priority                 : Normal
InputOutputThrottleLevel : None
ScheduledTask            : \Microsoft\Windows\Deduplication\WeeklyScrubbing
Full                     : True
ReadOnly                 : False


Get-DedupJob

Type               ScheduleType       StartTime              Progress   State                  Volume
----               ------------       ---------              --------   -----                  ------
Optimization       Scheduled                                 0 %        Queued                 D:
Optimization       Manual             16:20                  100 %      Running                D:


Get-DedupJob |fl


Volume                   : D:
VolumeId                 : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
Type                     : Optimization
ScheduleType             : Scheduled
StartTime                :
Progress                 : 0 %
State                    : Queued
Id                       : {556DC9B2-068C-4CAE-9BE7-4104A0963F49}
StopWhenSystemBusy       : True
Memory                   : 25 %
Priority                 : Low
InputOutputThrottleLevel : Low
ProcessId                : 0
Full                     : False
ReadOnly                 : False

Volume                   : D:
VolumeId                 : \\?\Volume{27fe6999-df3b-4864-81fe-853126e2c9cc}\
Type                     : Optimization
ScheduleType             : Manual
StartTime                : 21.07.2014 16:20:04
Progress                 : 100 %
State                    : Running
Id                       : {39D260C4-32D0-4511-8982-598DA53DB423}
StopWhenSystemBusy       : False
Memory                   : 60 %
Priority                 : High
InputOutputThrottleLevel : None
ProcessId                : 4672
Full                     : True
ReadOnly                 : False