Getting the Most Bandwidth out of your Media Agents

Last post 08-19-2010, 7:24 AM by Renaat. 14 replies.
Sort Posts: Previous Next
  • Getting the Most Bandwidth out of your Media Agents
    Posted: 08-03-2010, 4:19 PM

    I am attempting to get as much bandwidth out of our MA's as possible.  We have a 10GB backbone with 1 GB per individual NIC connection.  I have four NIC's per media agent, and all four are teamed for load-balancing to one IP address with Broadcom's teaming application.

    So far I have not seen throughput get past 1.6 gbps while manually monitoring the media agents, and am wondering if any of you have had more success than this using either software teaming (like us) or other methods.  Theoretically, I was hoping to see upwards of 4 gbps with the four nics all running under the load-balanced team.

    Also, on a side-note...  Is there a simple tool that I can actively test the maximum throughput from a media agent to a server?  A utility that performs a stress test would be nice.

    Thanks much,

    Paul

  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-03-2010, 6:43 PM
    Mortepa For the performance tool - browse the Resource Pack, please, on Maintenance Advantage https://services.commvault.com/mainadv/resource_pack.asp?dt=3&fv=8.0 DiskRead This utility can be used to validate backup speeds by measuring the following: Performance of hard disk reads. Measure the network read and write performance. With those teamed NICs - you might try the following: 1. Adjust Bindings on NIC - CommVault will communicate through the card bound to the top of the order. a. There are known issues with Broadcomm NIC's -There is a known Microsoft issue with these NIC's. If you are running a Broadcom NIC, please turn off the TCP/IP offloading and try the job again. Also if the MA platform is 2003 consider the URL here http://support.microsoft.com/kb/912222/en-us
  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 2:02 PM

    a. There are known issues with Broadcomm NIC's -There is a known Microsoft issue with these NIC's. If you are running a Broadcom NIC, please turn off the TCP/IP offloading and try the job again.

     

    Please provide links to these known issues.  Why turn off the TCP/IP offloading?

     

    To the original poster - verify your storage can stream the data.  Are you using CV dedupe?

  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 2:24 PM

    Hello bolsen,

     

    Here is an article refering to the chimney stack http://support.microsoft.com/kb/945977

    I believe this issue may have been addressed with the latest driver but don't qoute me.

     

     

  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 2:41 PM

    If I understand correctly, your recommendations only apply to Windows 2003 MAs (at least that's what I get from the MS KB)?

  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 2:56 PM

    Yes. In respect to that article and the server being a Media Agent this is refering to Windows Server 2003 with Scalable Networking Pack.

     

  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 2:57 PM

    Hi bolsen,

    although the article only references W2k3, we have seen issues with 2k8 as well.  recent example we had a customer getting 9GB/hour with offload enabled on the team.  Once Offload was disabled (I think reboot was required) we then observed a drastic increase in throughput (close to 90GB/hour). This is not too say that everyone will see this much improvment, but it is worthwhile to try the setting and if there is no improvement it can always be enabled again.


    Thanks,

    Craig T.
  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 3:49 PM

    Where should the changes be made?  Only on the media agent?  How about the index cache and dedupe servers?

  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 4:04 PM

    Just the Media Agent should be sufficient (unless you are seeing alot of indexing errors/disconnects).


    Thanks,

    Craig T.
  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 4:53 PM

    Can you clarify which fix you recommend?  Is it enough to disable the offload within the NIC properties or do you have to run the netsh.exe tool?

  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-04-2010, 5:50 PM
    I would run the netsh command and also turn off on the NIC properties. If you have the broadcom teaming utility/gui it is easiest to turn off there.
    Thanks,

    Craig T.
  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-06-2010, 11:54 AM

    What spec is the server?, one of the biggest perfomance increases we have seen was by adding extra vCPU's and memory. I would start simple and use iperf http://sourceforge.net/projects/iperf/ to test bandwidth between the MA's and server(s)

  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-17-2010, 10:39 AM

    Just a follow-up.  Our Dell servers with Windows 2008 using the Broadcom NIC teaming software had the Chimney Offload State = Off by default.

     

     

     

     


  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-18-2010, 10:28 AM

     Hi bolsen,

    At this point you will want to use the CommVault TestPort Utility to get a baseline avg. throughput.  This utility  can be found on the resource pack (can also be downloaded from the Resource Pack on the CommVault Maintenance Advantage Website; (Downloads and Packages>>Browse the Simpana 8.0 Resource Pack>>(Version(32bit, x64, etc))>>Network and Foundation).  Keeping in mind that there is no overhead with this test (no reads on client and no writes on Media Agent), this will give  you a base line throughput.  If your average backup throughput differs greatly from your baseline throughput, then more than likely a ticket would need to be opened with support to determine why.  Keep screen shots to provide support as these will help greatly to avoid duplicate testing, etc.

    To test this setup, you will use the following Syntax:

    On the Media Agent: Copy the appropriate Version of TestPort (x86, x64, etc) to the CV base Directory.
    From a command prompt, navigate to the Base Directory.
    Type:
    Testport.exe -server -useipv4 -bindname <Resolvable name of the Media Agent>  


    This will setup a listener on the Media Agent on a random port.  Take Note of the Listening Port as this will be needed for the next step (on the Client)

    On a Client: Copy the appropriate Version of TestPort (x86, x64, etc) to the CV base Directory.
    From a command prompt, navigate to the Base Directory.
    Type:
    Testport.exe -Client -useipv4 -bindname < Resolvable name of the Client> -srvname <Enter the Resolvable name of the Media Agent from the (Media Agent) steps above> -srvport <Enter the listening port the Media Agent is listening on) -buffers 1000000

    This will send 1,000000 16k buffers over the network (does not write any data) and will return an average throughput value when it completes.


    *NOTE:  you can replace -bindname with -bindip <IP> and srvname with srvip <ip>

     

     

     


    Thanks,

    Craig T.
  • Re: Getting the Most Bandwidth out of your Media Agents
    Posted: 08-19-2010, 7:24 AM

    Hi,

    You did not specify which OS and architecture you were using...

    You are aware that 64-bit is highly advised for decent throughputs, and that 2008 64-bit  is even better that 2003 64-bit ? That kernel was made for throughput!

    Also, what is your CPU doing when backing up?  If you're running up to 100% you're just choking the network flow.

    Also, "Optimize for concurrent LAN backups" is enabled on your media agents, right? I have seen 20% decrease in CPU usage when enabled

    Cheers

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy