I'm getting some errors in the event viewer every day while performing aux copies, but so far I've been unable to figure out the problem. I've tried to open several tickets on this as well, and so far have not been successful in pinpointing the cause. This is our environment. We have a Commserve host running on a VM in my office. The media agents are also VM's, each one in a different geographic location. The aux copies we run consist of copying the backed up data from that site to either another site, or to our OracleCloud container. One site seems to be giving us the lions share of the errors. The errors read as follows:
"Unable to communicate with CommServe on host [cvmaster]. Please check the network connectivity between the Client and CommServe, and verify this product's Communication Service is running on the CommServe."
Support most recently told us to make sure that certain paths etc were excluded from scanning by antivirus software. So just to be sure as a test, we actually uninstalled all antivirus from the commserve, and the media agents, and let the jobs run again. We continue to get the same errors daily on the same jobs. We've noticed though that the jobs start running at 6 am, and usually around 7 am or soon after, that's when we start to get the errors. It is sporadic up until about 12:30 pm, and the errors taper off for the day. In some cases, the job actually goes into a pending state for a while and later restarts automatically. In some cases the errors are accompanied by these types of errors:
"Error occurred while processing chunk  in media [V_42648] for storage policy [SP-SC_Main] copy [SG-Offsite-Primary]: Backup job . Unable to send the stop data transfer control message to the tail. ."
We thought that antivirus might have been responsible for interrupting something on the disk library, but this error actually occured AFTER the antivirus had been removed. I thought it might be a network issue, but we've tried to pair up errors from the log file with time stamps of errors on the network, and could not find anything that matched up. Firewalls on the network are open for Commvault to communicate freely. What else can we look at?