Exchange Mailbox agent Errors and Throughput

Last post 11-03-2020, 6:11 PM by meb_. 10 replies.
Sort Posts: Previous Next
  • Exchange Mailbox agent Errors and Throughput
    Posted: 10-29-2020, 6:16 PM

    Installed the new exchange mailbox agent for exchange and although I can see mailboxes and start backups, some of the mailboxes I added fail right away and the ones that do start have a really slow transfer rate.

    Here is a snipped of what I see as far as troughput. 

    And the error for the two mailboxes in the coordinator log

    Mailbox status info XML string to be updated in CS [<?xml version="1.0" encoding="UTF-8" standalone="no" ?><JobManager_ExchMbJobInfo commcellId="2" includeAll="0" jobId="251"><SourceMailboxStats AgentName="access node_Remote0" DisplayName="User A" FailedMessages="0" FailureReason="The mailbox failed." GUID="5EAD296BXB7BCX4F47X99C2X8B8C8FD795E8" SMTP="User.A@domain.com" SuccessfulMessages="0" status="1"/><SourceMailboxStats AgentName="access node_Remote1" DisplayName="User B" FailedMessages="0" FailureReason="The mailbox failed." 

    I dont see anything that it is obvious to me. Ideas? The receive-as permissions and throttling policy were created as the doc asks.

    Thanks

  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 10-30-2020, 2:44 AM

    I've seen >2TB/day with the new Exchange Mailbox Agent, perhaps you can share a little more about your configuration?

  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 10-30-2020, 3:52 AM

    Hi meb_,

    That throughput looks bad even for single streamed MAPI!!

    Is this On-Prem or Online? I am assuming On-Prem since you created the throttling policy. Which Exchange version are you running here? Which Commvault Feature Release is rolled out here as well?

     

    David

  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 10-30-2020, 4:21 AM

    In relation to the errors, we'd need to check the CVMailBackup.log on the Access Node where it ran into errors for more information.

     

    David

  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 10-30-2020, 10:37 AM

    Thank you both for getting back to me. A little bit about the environment;

    Fresh new install of CV SP17.57

    On Prem Exchange 2016 users

    Two Access Nodes (Outlook 2013 x64 installed)

    Indexing Server dedicated for Exchange Indexing Jobs

    WebServer dedicated for Exchange

    Follow the documentation, meaning that all the exchange permissions to service accounts were applied.

     


  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 10-30-2020, 11:23 PM

    Try forcing Exchange 2016 to use EWS over MAPI.  MAPI backups relies on Outlook 2013 SP1 which is out of support by Microsoft, and with EWS Outlook 2013 is only needed for PST restores.  EWS is faster

    Try adding this Additional Settting on the Access Nodes (cycle the services on them).

    Name: ewsemail

    Category: MSExchangeMBAgent

    Type: STRING

    Value: service.account@yourdomain.com.au

    and on the CommServe (cycle the services).

    Name: benablesmtpaddress

    Category: CommServDB.Console

    Type: BOOLEAN

    Value: True

  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 10-31-2020, 4:17 PM

    Both keys added and will monitor througout the week and report back. A different question related to exchange, is there a workflow or report that shows me the health status or kind of check list of important points between Commvault and exchange? For example, I know that as per the documentation the service accounts used to backup need to be part of the organization management, would there be a report that could tell me whether the service accounts arent part of that group? Or even if the throttle policy exists, if the receive as permissions are applied or not? This is not the first exchange 2016 I configure for commvault and not the first time I get issues like this meaning that even though you follow the documentation each and every line, commvault doesnt seem to behave as it should and evert time I need to add a additional key, it almost seems like a "bandaid".

  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 10-31-2020, 5:11 PM
    I echo those concerns. That transition to EWS was the result of a very long support case for an incorrectly functioning Hybrid Exchange 2016 backup configuration. In my case I had to use EWS for on premises because a recent Exchange security update seemed to break MAPI and as such we implemented the Exchange 2019 solution (check this for more info https://documentation.commvault.com/commvault/v11/article?p=112073_1.htm, I forgot to link before sorry). The other good thing about EWS is that service account does not need to have as many permissions.
  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 11-02-2020, 10:00 AM

    Yes, I have a second CommCell running the same exchange version on-prem, configured the same way (except the CommCell Version is SP18) and yet the throughput on that one is flying. No exactly sure what it is wrong with this one though (SP17). I did add the keys you recommended and it didnt seem to make any difference.

    As for the logs, collected some of the failed messages and this is what I frequently see;

    .11/02 03:00:56 JOBID Failed message retry count  = [3]
    .11/02 03:01:03 JOBID ClientHives::ClientHives(145) - Failed to get key handle to [SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\HARDWARE] in profile list, error=2
    .11/02 03:01:03 JOBID ClientHives::ClientHives(145) - Failed to get key handle to [SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\SYSTEM] in profile list, error=2
    .11/02 03:01:03 JOBID ClientHives::ClientHives(145) - Failed to get key handle to [SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\BCD00000000] in profile list, error=2
    .11/02 03:01:03 JOBID ClientHives::ClientHives(145) - Failed to get key handle to [SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\SOFTWARE] in profile list, error=2
    .11/02 03:01:03 JOBID ClientHives::ClientHives(145) - Failed to get key handle to [SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\.DEFAULT] in profile list, error=2
    .11/02 03:01:03 JOBID ClientHives::ClientHives(145) - Failed to get key handle to [SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\SECURITY] in profile list, error=2
    .11/02 03:01:03 JOBID ClientHives::ClientHives(145) - Failed to get key handle to [SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\SAM] in profile list, error=2
    .11/02 03:01:03 JOBID ClientHives::ClientHives(155) - Failed to query the CentralProfile from [SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\S-1-5-20], error=2
    .11/02 03:01:03 JOBID Failed items file path: \\JobResultsPath\backup.out

    11/02 03:01:12 JOBID Failed items retry feature is enabled
    11/02 03:01:12 JOBID Read 0 items from the previous job's failed items file
    11/02 03:02:05 JOBID Mailbox [USER] Archive Statistics: Failed Folders : [0], Successful Folders : [0], Failed Items : [0], Successful Items (including stubs) : [0], Successful Stubs : [0], Time To Process : [62.118360] seconds, Bytes Processed : [0], SenderSMTP check time : [0.000000] seconds

    .11/02 03:02:47 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Calendar]
    .11/02 03:02:48 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Contacts]
    .11/02 03:02:48 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Contacts]
    .11/02 03:02:51 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Contacts\{06967759-274D-40B2-A3EB-D7F9E73727D7}]
    .11/02 03:02:51 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Contacts\{06967759-274D-40B2-A3EB-D7F9E73727D7}]
    .11/02 03:02:52 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Contacts\GAL Contacts]
    .11/02 03:02:52 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Contacts\GAL Contacts]
    .11/02 03:02:54 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Contacts\Recipient Cache]
    .11/02 03:02:54 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Contacts\Recipient Cache]
    .11/02 03:02:56 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Conversation Action Settings]
    .11/02 03:02:56 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Conversation Action Settings]
    .11/02 03:02:57 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\CV Test]
    .11/02 03:02:57 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\CV Test]
    .11/02 03:02:59 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Deleted Items]
    .11/02 03:02:59 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Deleted Items]
    .11/02 03:03:01 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Drafts]
    .11/02 03:03:01 JOBID Finished archiving failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\Drafts]
    .11/02 03:03:02 JOBID Starting to archive failed items in folder [\MB\{80BF2865X3357X4A6DXA78AXAFA2D6AF518F}\ExternalContacts]
     
  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 11-03-2020, 5:56 PM

    Hi _meb. In my case I was running FR11.21, and as those keys are not listed in the official Additional Settings so they may not work with SP17.  One way you can check if it is using EWS though is to look through the logs and see if it is still attempting to use Outlook Profiles.  EWS does not need to use a local administrator (and can be removed from the Exchange Backup config) on the proxy and everything is done over HTTPS.  If it is using MAPI, perhaps it is worth experimenting with just one access node.  Ideally though, all Exchange backups need to get off MAPI asap and it may end up being a support call that may recommend an update.

  • Re: Exchange Mailbox agent Errors and Throughput
    Posted: 11-03-2020, 6:11 PM

    Updated to SP18.37 today, will see how it jobs will perform going forward.

    Thanks

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy