Media mount and usage errors - going on months

Last post 09-10-2019, 6:24 PM by Wwong. 7 replies.
Sort Posts: Previous Next
  • Media mount and usage errors - going on months
    Posted: 09-09-2019, 5:23 PM

    I have opened several Incidents with Support but we cannot seem to find a resolution. I keep getting the Media Mount and Usage error alerts on all 3 media agents and pretty much all paths over the past several weeks/months.

    The latest SE noticed the following message:

    [TYPE] Information [TIME] 2019/09/09 01:06:43  [SOURCE] Galaxy [COMPUTER] xxx.xxx.local[DESCRIPTION] Failed in acquiring the lock on file [E:\Program Files\Commvault\ContentStore\Log Files\PerformanceMetrics.log] Module [PerformanceMetrics] Error[0x80070306:{CQiLogger::LockLockfile(1326)} + {CQiLogger::LockLockfile(1320)/W32.774.(One or more errors occurred while processing the request.)-Timed out acquiring mutex on Global\QiLogger_PerformanceMetrics}]

    Has anyone else encountered several Media Mount and Usage alerts and or know what the "mutex on Global" even means?

    Thanks. 

  • Re: Media mount and usage errors - going on months
    Posted: 09-09-2019, 7:19 PM

    Is the CommServe or are the MediaAgents clustered?

  • Re: Media mount and usage errors - going on months
    Posted: 09-09-2019, 8:48 PM

    The errors are showing up on our Media agents. We have 2 on one campus and a single media agent at a different campus. They are not clustered and the OS is Win2k12 R2 and the hardware is HPE Proliant 380DL Gen9. Each campus location has its own dedicated Storage (Netapp e2700 series). So single Commcell environment. 

  • Re: Media mount and usage errors - going on months
    Posted: 09-09-2019, 9:13 PM

    Given that it is happening on the three mediaagents, it is possible that the problem is coming from the CommServe.  What version+servicepack of Commvault are you running, and what is the Build Number of the SQL Server on the CommServe?

  • Re: Media mount and usage errors - going on months
    Posted: 09-10-2019, 8:29 AM

    I am running v11 sp15 HPK21. The SQL build number on the CommServe is Server 2014 v12.0.2269.0 (X64) Jun 10 2015.

    So there is no Anti-Virus on the media agents but I happened to notice that SEP is installed but disabed on the CommServe. Should it be completely uninstalled? 

  • Re: Media mount and usage errors - going on months
    Posted: 09-10-2019, 10:00 AM

    Ok thanks. That is an old revision of SQL server 2014. Can you see errors in the Program Files\Microsoft SQL Server\MSSQL.n\MSSQL\LOG\ERRORLOG file?

  • Re: Media mount and usage errors - going on months
    Posted: 09-10-2019, 12:47 PM

    I checked the SQL ERRORLOG files on both commserve servers (active & standby) and I didn't see any errors around the date/time the media mount and usage alert came out. 

    Support is seeing this when they parse the most recent logs:

    redatacted\cvd

    1760  2150  09/09 01:27:50 1611889 6700604-# [DM_BASE    ] **ERROR** MountActiveVolume Failed to mount volume 0  MediaGroupId = 205 MA hostname- redacted MediaManager returned error [752], errorString [A mount path already exists at the given location. Cannot create a file when that file already exists. ]

    1760  2150  09/09 01:27:50 1611889 6700604-# [DM_BASE    ] Going to check whether the mount should be retried for the above mediamanager error

     

    1760  2150  09/09 01:27:50 1611889 6700604-# [DM_BASE    ] **ERROR** MountActiveVolume Failed to mount volume 0  MediaGroupId = 205 MA hostname- redacted MediaManager returned error [752], errorString [A mount path already exists at the given location. Cannot create a file when that file already exists. ]

     

     

    The error code 183 returned by the OS indicates that the file already exists:

    C:\>net helpmsg 183

    Cannot create a file when that file already exists.

     

    This might happen if the device did not respond, timed out and CV eventually retried the same file.

    The health check on the dedicated storage device showed no issues. 

  • Re: Media mount and usage errors - going on months
    Posted: 09-10-2019, 6:24 PM

    Hi wsimps01

    The original error looks to be an issue when updating logs, and could be related to slowness on the Disk where MediaAgent Binary is installed. 

    The subsequent mount errors looks like volume exist in the current mount path and this type of issue could be exhibited due to:

    • Slow storage at the time of writing, Commvault acknowledges the volume is already created or
    • A DR restore on the Commserve is done, when it attempts to re-write the same volume, as the volume already exist on Disk it will report the error and skip writing. 
    As you have mentioned you are using NetApp Arrays, so I presume you will be using CIF presented volume for the Disk Storage. 
     
    Can we also check the Network aspect to confirm whether there are TCP reset or packet drops resulting in these intermittent errors?
     
    Regards
     
    Winston 
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy