After SP12 Upgrade network bindings on Mediaagents don't work

Last post 08-10-2018, 11:51 AM by oalexis. 5 replies.
Sort Posts: Previous Next
  • After SP12 Upgrade network bindings on Mediaagents don't work
    Posted: 07-31-2018, 11:27 PM

    Folks,

    Upgraded our Commserv and Mediaagents to SP12 and it hasn't been fun. First the install would fail on the mediaagents but after multiple attempts we got it. First backup test failed and things went south after that.

    The main issue I have right now is mediaagents seem to have lost their network bindings for port 8403 and jobs are failing because of that. In our environment we have tons on DIPs configured. Our MAs have a default interface and many sub interfaces. The only interface that seem to work is the default interface. Clients are configured to use a local interface on their subnet via specific DIPs. Any backup job with a client that resides on one of the sub interfaces does not work. In some cases I can get cvping on port 8403 and some I can't. But either way the backup job fails. One MA was resolved as support discovered it was a Windows update that broke the bindings. That culprit update is not installed on all MAs.

    Attached below is the error that I get when trying to cvping on port 8403

     

    I confirmed the MA is listening on port 8403 on all interfaces. There is no Windows filewall in place to block the connection. This was all working before SP12 install. I've rebooted the MAs countless times. I've disabled and reenabled interfaces and nothing works.

    I have this case all the way in Dev support right now. but any help is greatly appreciated.

     

    Thank you

     

     


    Attachment: cvping.png
  • Re: After SP12 Upgrade network bindings on Mediaagents don't work
    Posted: 08-01-2018, 9:28 AM
    • Aplynx is not online. Last active: 12-12-2018, 2:21 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,455
    There are two known connectivity issues at the moment for SP12.
     
    After installing Service Pack 12, a Client or MediaAgent may lose connectivity in the CommCell. Firewall tunnels will fail to establish, the MediaAgents may go offline and backups\restores will result in network errors.

    In Service Pack 12, development split handling of firewall connections out of CVD and into a new service/executable called cvfwd.exe; similar to how firewalls are handled on Unix clients. Many customers rely on the default exclusion that are configured on install, which adds all executables in the base directly to the Windows Firewall Exclusions.

    When upgrading to the new service pack, the installer doesn't add programs from CommVault Base folder to the Firewall Exclusions list. New installs using SP12 are unaffected as the firewall batch file will include the new executable.

    Resolution

    The workaround is to manually add the exclusion, or add a new port-based rule rather than program/executable rule (which is preferred). You can also run the Base\AddFWExclusions.bat http://documentation.commvault.com/commvault/v11/article?p=7378.htm file to refresh the firewall exclusions and add the new 'cvfwd' service into the list.

    Hotfix 2877 http://documentation.commvault.com/commvault/v11_sp12/article?p=SP12_2018-07-26.htm 'Add cvfwd.exe to firewall exclusion list for upgrades' will address this issue.

    OR

    If the above July 10th Windows Update is installed and the services on the client are recycled, then the same symptoms as documented in https://support.microsoft.com/en-us/help/4345424/improvements-and-fixes-windows-8-1-and-server-2012-r2 affect the CommVault software. The following Windows Updates have been released to address the issue:

    For Windows 7 and Windows 2008 R2: July 18th Preview of Monthly Rollup Pack https://support.microsoft.com/en-us/help/4338821 or KB4345459https://support.microsoft.com/en-us/help/4345459

    For Windows 8.1 and Windows 2012 R2: July 18th Preview of Monthly Rollup Pack https://support.microsoft.com/en-us/help/4338831 or [KB4345424]https://support.microsoft.com/en-us/help/4345424
     
    For Windows 10 and Windows Server 2016 Released July 16th, 2018:
    Build 1607 - KB4345418 https://support.microsoft.com/en-us/help/4345418
    Build 1703 - KB4345419 https://support.microsoft.com/en-us/help/4345419
    Build 1709 - KB4345420 https://support.microsoft.com/en-us/help/4345420
    Build 1803 - KB4345421 https://support.microsoft.com/en-us/help/4345421
     
     
     
     
  • Re: After SP12 Upgrade network bindings on Mediaagents don't work
    Posted: 08-01-2018, 12:19 PM

    Thank you so much!! Will take a look. I discovered a work around that seems to work:

    • Disabled the DIP config on the client
    • Pushed network config
    • Checked for client readiness
    • Deleted the sub interface on the MAs
    • Recreated the sub interfaces on the MAs
    • Re-ip’d interfaces
    • Recreated DIP on the client level and not group level
    • Pushed network config to client and MAs
    • Checked client readiness…success
    • Checked cvping on port 8403… success
    • Run backup up job….success

     

    Will be doing some additional testing on other subnets. May have rinse and repeat the above steps. I still have to create DIP on group level.

    Windows firewall is set to allow all traffic.

  • Re: After SP12 Upgrade network bindings on Mediaagents don't work
    Posted: 08-10-2018, 10:54 AM

     

    Did you also install July 26 or Aug 2 hotfix pack.  They both contain hotfix 2877 that has some fix in this regard.

     

    http://documentation.commvault.com/commvault/v11_sp12/article?p=SP12_2018-07-26.htm

     

     

  • Re: After SP12 Upgrade network bindings on Mediaagents don't work
    Posted: 08-10-2018, 11:04 AM
    • Aplynx is not online. Last active: 12-12-2018, 2:21 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,455

    Here's a more legible version of the windows updates:

    http://kb.commvault.com/article/WFS0021

    Also SP12 added an executable called cvfwd.exe and it isn't automatically added to the firewall exclusions list until 2877. This only effects updates to SP12 and not new installs. Running addfwexclusions.bat file in the base directory of the client will add it again. 

    http://documentation.commvault.com/commvault/v11/article?p=7378.htm

  • Re: After SP12 Upgrade network bindings on Mediaagents don't work
    Posted: 08-10-2018, 11:51 AM

    Thanks Liam. Turns out that is what it was. Added the exclusion that sure enough that fixed the MAs. Thanks again!

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2018 Commvault | All Rights Reserved. | Legal | Privacy Policy