When backing up 1 Windows MA, the file sync is failing trying to create VSS snapshots.  Windows event logs report VSS failure with "system reserved" volume.  If I turn off backing up the system state for that MA, the backup of the MA filesystem run fine.  I've restarted the services, and then the server.  Did not make a difference. 

This all started coincidently after removing a disk (actually a jbod) from the MA on Monday, and rebooting the server at that time. 

Running vssadmin list writers, shows the System Writer and WMI Writer in a failed state. 

Again, I can restart the services and/or reboot the machien, and the writers all show as "ok". 

They only go into the failed state when I try doing a backup of the MA, which includes saving the system state.

Running V11 SP16

Any hints?

Windows event log:

Event ID 12289

Unexpected error DeviceIoControl.... The parameter is incorrect

Sample of messages in filescan.log

10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::DoShadowSet() - Call asyncShadow.WaitUntilDone()  [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO
10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::CreateSnapshotSet() - Call DoShadowSet()  [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO
10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::Snap() - Call CreateSnapshotSet()  [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO
10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::DeleteSnapshot() - Shadow not created!
10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::BackupComplete() - Deleted snapshot set
10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::BackupComplete() - Aborting backup!

Are you able to run a successful system state backup using the native windows backup application?

It is possible that the missing JBOD disk had a piece of VSS data, that broke the VSS chain. In that case, delete all VSS data and re-create VSS and start again. Yes that will delete all old snapshots but if it is corrupted, it is not good any more.

The event ID 12289 should have an error code like 0x800xxxxxxx, which can give you more clue.

Not sure how to do that.

One thing I notice in looking at the other MAs in our environmemt.  If I look at the windows "Configure shadow copies" screen, it is only this one particular MA that shows the "reserved system" volume.  And if I look at the settings for that, it shows "total disk space 350MB, Free space 72MB".  I'm wondering why this is the only MA with this volume and if the reason it is breaking, is that there is no room to create a snapshot, and whether it even needs one.

I did clear out the old snapshots (were for the C: drive), and tried a backup with the "system state backup" re-enabled, but that failed

I went into Windows, and changed (at least I think I did ), where the snapshot is created for the "system reserved" partition, re-ran the backup and it apparrently succeeded. Looks like it did not have enough space to create the snapshot in the original location.  Will hold off on declaring victory for a while. 

As mentioned, I looked at my other MAs, and they do not show the "system reserved" volume under Shadow Copies.

Strange.

Still having this issue with VSS snapshots failing on random Windows 2012R2 servers.  When this occurs, much of the time the file scan will fail, retry, fail,retry...etc and eventually succeed .  The rest of the time the backup will ultimately fail.  If I turn off system state backups, the jobs succeed.

i can get a report showing the fails, but is there a report I can run that would list jobs where the file scan failed one or more times before the backup succeeded?  Am try to see if I can localize the problem.  The options "complete with errors" does not do this.

Also, now when looking at the vss writers that fail, it is not always the same ones.

System writer, WMI writer, BITS Writer most always, but occasionally others as well.

These are all OS issues being reported. You should have corresponding errors in the system\application logs for the client. When CommVault reports provider_veto it's the VSS writer reporting it can't create a snapshot. 

This additional setting will make sure CV calls out the Windows VSS writer, regardless if there are additional writers installed.

https://documentation.commvault.com/additionalsetting/details?name=nUseVSSSoftwareProvider

Tried the setting, but unfortunately it did not work.

OK, you'll have to look at it from the OS side. The same command is sent everytime from CommVault to have VSS create a snapshot through the writers. The resulting 'VSS_E_PROVIDER_VETO' means the OS had an issue and wasn't able to create the snapshot. The CommVault software does not have a way to workaround the OS failure. 

Thanks.

Really don't care for intermittent problems.  Nothing fails consistantly.  Will work completely fine at times, And when it does break, most often recover after a scan failue or two.  Only rarely does the backup fail completely after about ten scan attempts.

Can't pin it down to the client being busy, as they're showing no cpu or storage related issues.  There is a mix of VM's and a handful of physical machines that both show the problem.  And the VM's are using different storage systems.  The only consistancy is that it seems to more affect some systems than others.  There are a couple that seem to have problems fairly consistantly.  A few that only do it once a week or longer.  And most never have the problem.

Will just have to keep picking at it.