iFind Vss issue with system state

Last post 08-13-2020, 12:09 PM by deb. 10 replies.
Sort Posts: Previous Next
  • iFind Vss issue with system state
    Posted: 01-29-2020, 12:52 PM

    When backing up 1 Windows MA, the file sync is failing trying to create VSS snapshots.  Windows event logs report VSS failure with "system reserved" volume.  If I turn off backing up the system state for that MA, the backup of the MA filesystem run fine.  I've restarted the services, and then the server.  Did not make a difference. 

    This all started coincidently after removing a disk (actually a jbod) from the MA on Monday, and rebooting the server at that time. 

    Running vssadmin list writers, shows the System Writer and WMI Writer in a failed state. 

    Again, I can restart the services and/or reboot the machien, and the writers all show as "ok". 

    They only go into the failed state when I try doing a backup of the MA, which includes saving the system state.

    Running V11 SP16

    Any hints?

  • Re: iFind Vss issue with system state
    Posted: 01-29-2020, 4:01 PM

    Windows event log:

    Event ID 12289

    Unexpected error DeviceIoControl.... The parameter is incorrect

    Sample of messages in filescan.log

    10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::DoShadowSet() - Call asyncShadow.WaitUntilDone()  [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO
    10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::CreateSnapshotSet() - Call DoShadowSet()  [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO
    10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::Snap() - Call CreateSnapshotSet()  [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO
    10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::DeleteSnapshot() - Shadow not created!
    10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::BackupComplete() - Deleted snapshot set
    10400 1f28  01/29 01:52:09 477182 CsSnapRequestor::BackupComplete() - Aborting backup!

  • Re: iFind Vss issue with system state
    Posted: 01-30-2020, 11:46 AM
    • Aplynx is not online. Last active: 01-05-2021, 3:13 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,906

    Are you able to run a successful system state backup using the native windows backup application?

  • Re: iFind Vss issue with system state
    Posted: 01-30-2020, 7:30 PM

    It is possible that the missing JBOD disk had a piece of VSS data, that broke the VSS chain. In that case, delete all VSS data and re-create VSS and start again. Yes that will delete all old snapshots but if it is corrupted, it is not good any more.

    The event ID 12289 should have an error code like 0x800xxxxxxx, which can give you more clue.

  • Re: iFind Vss issue with system state
    Posted: 01-31-2020, 12:08 PM

    Not sure how to do that.

    One thing I notice in looking at the other MAs in our environmemt.  If I look at the windows "Configure shadow copies" screen, it is only this one particular MA that shows the "reserved system" volume.  And if I look at the settings for that, it shows "total disk space 350MB, Free space 72MB".  I'm wondering why this is the only MA with this volume and if the reason it is breaking, is that there is no room to create a snapshot, and whether it even needs one.

    I did clear out the old snapshots (were for the C: drive), and tried a backup with the "system state backup" re-enabled, but that failed

  • Re: iFind Vss issue with system state
    Posted: 01-31-2020, 12:26 PM

    I went into Windows, and changed (at least I think I did Smile), where the snapshot is created for the "system reserved" partition, re-ran the backup and it apparrently succeeded. Looks like it did not have enough space to create the snapshot in the original location.  Will hold off on declaring victory for a while. 

    As mentioned, I looked at my other MAs, and they do not show the "system reserved" volume under Shadow Copies.

    Strange.

  • Re: iFind Vss issue with system state - continuing
    Posted: 07-20-2020, 1:38 PM

    Still having this issue with VSS snapshots failing on random Windows 2012R2 servers.  When this occurs, much of the time the file scan will fail, retry, fail,retry...etc and eventually succeed .  The rest of the time the backup will ultimately fail.  If I turn off system state backups, the jobs succeed.

    i can get a report showing the fails, but is there a report I can run that would list jobs where the file scan failed one or more times before the backup succeeded?  Am try to see if I can localize the problem.  The options "complete with errors" does not do this.

    Also, now when looking at the vss writers that fail, it is not always the same ones.

    System writer, WMI writer, BITS Writer most always, but occasionally others as well.

  • Re: iFind Vss issue with system state - continuing
    Posted: 08-12-2020, 11:16 AM
    • Aplynx is not online. Last active: 01-05-2021, 3:13 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,906

    These are all OS issues being reported. You should have corresponding errors in the system\application logs for the client. When CommVault reports provider_veto it's the VSS writer reporting it can't create a snapshot. 

     

    This additional setting will make sure CV calls out the Windows VSS writer, regardless if there are additional writers installed.

     

    https://documentation.commvault.com/additionalsetting/details?name=nUseVSSSoftwareProvider

  • Re: iFind Vss issue with system state - continuing
    Posted: 08-13-2020, 11:29 AM

    Tried the setting, but unfortunately it did not work.

    Attachment: CV-Vss-0813.txt
  • Re: iFind Vss issue with system state - continuing
    Posted: 08-13-2020, 11:36 AM
    • Aplynx is not online. Last active: 01-05-2021, 3:13 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,906

    OK, you'll have to look at it from the OS side. The same command is sent everytime from CommVault to have VSS create a snapshot through the writers. The resulting 'VSS_E_PROVIDER_VETO' means the OS had an issue and wasn't able to create the snapshot. The CommVault software does not have a way to workaround the OS failure. 

  • Re: iFind Vss issue with system state - continuing
    Posted: 08-13-2020, 12:09 PM

    Thanks.

    Really don't care for intermittent problems.  Nothing fails consistantly.  Will work completely fine at times, And when it does break, most often recover after a scan failue or two.  Only rarely does the backup fail completely after about ten scan attempts.

    Can't pin it down to the client being busy, as they're showing no cpu or storage related issues.  There is a mix of VM's and a handful of physical machines that both show the problem.  And the VM's are using different storage systems.  The only consistancy is that it seems to more affect some systems than others.  There are a couple that seem to have problems fairly consistantly.  A few that only do it once a week or longer.  And most never have the problem.

    Will just have to keep picking at it.

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2021 Commvault | All Rights Reserved. | Legal | Privacy Policy