Block settings for DDBs, policies and data paths for AWS S3

Last post 10-13-2019, 7:45 PM by Watson. 1 replies.
Sort Posts: Previous Next
  • Block settings for DDBs, policies and data paths for AWS S3
    Posted: 03-29-2019, 11:03 AM

    We've been running Commvault 11 for a few years and recently got an Aux Copy to AWS S3 working and tested so we can recover in a full DR scenario.  Now I'm trying to figure out the best settings to make sure we're getting an efficient data transfer and storage.  I've read through many articles and forum posts but nothing that ties everything together and am hoping posting here will help organize what I've read.

    Our environment is two sites with one media agent local to each that handles backups for VMs, file agents and DB agents. Every day the previous night's backups are sync copies to the other site.  We use dedupe locally and Aux copies are DASH.  A single CommServe controls everything and all data is AES encrypted.

    For the copy to AWS S3, all data is sent from media agent A. It sends jobs from the Primary copy for storage policies that use it's deduped disk library, and uses the Secondary copies for jobs that come in from the remote site media agent B.  That works well over our 1Gb synchronous internet connection.

    What I'm struggling to iron out are the detailed settings like block sizes and how to get the most efficient dedupe and job thoughput.

    The different block level dedupe factors we have set are:

    Local global dedupe DBs: 128KB

    AWS S3 global dedupe DB (on media agent A): 512KB

    Storage policies: 128KB

    The block sizes on the copies' data paths are:

    Primary and Secondary copies to disk libaries (on site): Use media type settings (64KB)

    AWS S3 copy: 1024 (automatically set when creating the copy)

    Based on the last bullet point in link 1 below, should I change the storage policies to have a block level of 512KB or is the default 128KB like link 2 shows OK?

    Similarly, should the data path block size for the S3 cloud lirbary be set to 512KB instead of 1024?  I only found reference to the 1024 in link 3 for tuing 10Gb+ connections.

    My plan was to leave storage policy block sizes at 128KB and set the cloud library data path to 512KB. Any advise is welcome. Thanks!

    Note: The S3 cloud copies are DASH Network Optimized and Space optomized Aux Copy is checked.

    Link 1:

    Link 2:

    Link 3:

  • Re: Block settings for DDBs, policies and data paths for AWS S3
    Posted: 10-13-2019, 7:45 PM
    When performing a Dash Copy with a mismatched block size (source is 128KB and destination is 512KB) we recommend using network optimized not disk optimized.  With network optimized we read the data and generate a signature based on the destination DDB block size.  
    With disk optimized we read the signature from the chunk_metadata based on the source block size 128KB and check the destination DDB where the block size is 512KB. We will not find the signature in the destination DDB because the block sizes are mismatached. We then need to read the data and generate a signature back on the destination DDB block size.
    You can use disk optimized but only if you set the block size needs to be the same on the source and destination DDB.  When the block size is the same and we use disk read optimized we can read the signature from the chunk_metadata and do not need to read/rehydate the data.
    You can setup the source and destination to use the same block size and use disk optimized.  
    If your source library uses a 512KB block size, there will be less space savings on your source library.
    If your destination library uses a 128KB block size, pruning and read performance will suffer.
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy