Error: Cannot Allocate Pipeline Buffer on FS backup

Last post 05-28-2020, 12:13 PM by Liam. 7 replies.
Sort Posts: Previous Next
  • Error: Cannot Allocate Pipeline Buffer on FS backup
    Posted: 05-13-2020, 9:55 AM

    Hi to all, I'm having an issue performing FileSystem backup, I keep recieving this error: 


    Cannot allocate Pipeline Buffer.
    Source: ----, Process: clBackup

    My client is on the latest SP anche hotfix as the Commserve, and when performing the Check Readiness I can see that my client is ready. 

    I also tried to add manually the data pair interface Client proprieties --> Advanced --> Job configuration and the client is again ready, but I keep getting the same error. 

    Can anyone help me try to solve this issue? 

    Any help is appreciated

    Let me know if you need  other logs. 

    Thank you so much 

  • Re: Error: Cannot Allocate Pipeline Buffer on FS backup
    Posted: 05-13-2020, 7:32 PM

    Hi there!

    Allocating pipeline buffers refers to the ability to push data into memory in order to transfer it across the network to your MediaAgent.

    Can you make sure that there is free memory on the client and it's not writing a lot of requests to page file?

    It could also be an issue with the pipeline between the client and the MediaAgent. Do you have an Network Configuration Rules set up between the client and MA in CV?

    I would suggest creating a Persistent Tunnel between the client and MA to see whether this assists here. If there are still problems you could check the cvfwd.log to see whether there are any ungraceful terminations being thrown.

    To set up the persistent tunnel, right-click the client and select Properties, go to the Network section and Network Route Configuration tab. If it's not checked, check the box to enable the feature. On the Incoming Connections tab, select the MediaAgent from the entity dropdown and the connection state to BLOCKED. This will force the client to reach out on the MA's configured tunnel port to create a tunnel, then periodically send a keep-alive packet to ensure that it is open.

    If this doesn't work, it could be that something is stopping the data transfer between client and MA. I would then recommend trying to use the CvNetworkTestTool to send data between the client and MA and make sure that it is able to transfer.

    Run on MediaAgent side of the tool in Server Mode: http://documentation.commvault.com/commvault/v11/article?p=7584.htm

    Then Run the tool on the Client in Client Mode:
    http://documentation.commvault.com/commvault/v11/article?p=7590.htm

    This will hopefully assist in determining whether data transfer is possible between the machines.

    I hope this assists.

     

    David

  • Re: Error: Cannot Allocate Pipeline Buffer on FS backup
    Posted: 05-13-2020, 7:34 PM

    Is it possible you could attach the clBackup.log?  Also it would be useful to see the \Base\FwConfig.txt for both the client and MediaAgent.

  • Re: Error: Cannot Allocate Pipeline Buffer on FS backup
    Posted: 05-15-2020, 6:08 AM

    Hi David, 

    thank you so much for the tips and the quick response. 

    I tried what you suggested up here, but unfortunatly it didn't work. 

    During backup I didn't find any aborted connection log on the cvfwd.log log file on media agent.

    After that I tried to run the CvNetworkTestTool from the client to media and from media to client but it gave these errors: 

    From Media agent                          
    SERVER STARTED
    ---------------------------------------------------------------------
    18552 Trying to bind to [--------] at port 25000
    18552 Bind failed!!. [0x80072741:{CQiSocket::Bind(250)/W32.10049.(The requested address is not valid in its context.)}]
    ---------------------------------------------------------------------
    SERVER STOPPED
    ---------------------------------------------------------------------

    From the client :
    ---------------------------------------------------------------------
    CLIENT STARTED
    ---------------------------------------------------------------------
    22116 Attempting connection to server cv001ma08b.mo-services.it:25000
    22116
    Attempting firewalled connect to (cv001ma08b.mo-services.it:cv001ma08b/25000)
    22116 Failed to connect to server (cv001ma08b/cv001ma08b.mo-services.it), error=(-1/Couldn't complete client connection to 127.0.0.1:25000)
     
    Does this mean that there is a communication problem between these two servers? 
    By the way, we do have a restricted envirement, so we managed to create the Network Topologies for client communication.
    Anyway, I want to point out that the backups were running fine 4 days ago, then it started to fail. 
    Let me know if you need any other information.
  • Re: Error: Cannot Allocate Pipeline Buffer on FS backup
    Posted: 05-15-2020, 6:32 AM

    Hi Anthony, 

    I'll attach to the reply 3 files, clBackup.log FwConfig Client and FwConfig Media.

    I have tried several combination, but nothing worked. 

    Let me know if you find anything that can help me. 

    Thank you so much 

  • Re: Error: Cannot Allocate Pipeline Buffer on FS backup
    Posted: 05-15-2020, 7:31 AM

    Correct me if I am wrong, and I know you may have tried many combinations but it looks like you have a BLOCKED rule on both client and MediaAgent to each other. If so this won't work but setting them to RESTRICTED should (this is a two-way on-demand firewall rule where both can be an initiator).

    CLIENT

    [outgoing]
    rrgpsmexc16 cv001ma08b remote_guid=6CC83BDB-806F-4ABD-93A0-2C2144BEA9E2 type=persistent local_iface=172.30.164.72 proto=httpsa cvfwd=172.30.164.78:8403
    rrgpsmexc16 * proxy=cv001ma08b

    MA

    [outgoing]
    cv001ma08b rrgpsmexc16 remote_guid=F693B93E-92F0-4E13-AAB4-BC2E337B317C type=persistent local_iface=172.30.164.78 proto=http cvfwd=172.30.164.72:8403
    cv001ma08b rrgtorexc16 remote_guid=05053518-F61D-4C69-B820-334C433D1ECC type=passive

    Perhaps instead of using the Firewall Topology wizard, try you hand at the Advanced Firewall between the client and media agent, then set them both to restricted.  https://documentation.commvault.com/commvault/v11/article?p=7233.htm

    Also check if the Firewall is enabled for the unidentified DIP network, and if you cannot turn it off run \Base\AddFWExclusions.bat on both client and MediaAgent.

  • Re: Error: Cannot Allocate Pipeline Buffer on FS backup
    Posted: 05-16-2020, 8:39 AM

    I agree with Anthony here, both sides of the tunnel here are showing as persistent which is not correct. It looks like there is another client here which has a tunnel setup with the MA (see rrgtorexc16) and it is configured to create a persistent tunnel tothe MA. This is why MA FwConfig is showing cvfwd=passive for that client connection. 

    Further to this, checking the clBackup.log it seems like there is a problem connecting to the Commserve and therefore an issue getting the encryption keys which is leading to the allocate buffer error. 

    28928 75fc  05/15 11:14:53 9806745 FetchKeysByEncryptArgs() - Fetching Enc keys [1/1] for AFile 40221089, copyId 1304
    28928 75fc  05/15 11:14:54 9806745 CvFwClient::_connect() - ERROR: Unable to resolve remote host CV001CS.MO-SERVICES.IT to IPv4 family: 0x80072AF9:{CQiIPAddr::GetAllByName(216)/W32.11001.(No such host is known.)}
    28928 75fc  05/15 11:14:54 9806745 CvException::init(): func_name=Cvc::decryptDataWithCRC32, file_name=unknown, message='Failed to decrypt data: wrong decryption key.'
    28928 75fc  05/15 11:14:54 9806745 SdtBase::encDataBuffer() - Failed to decrypt data: wrong decryption key.
    28928 75fc  05/15 11:14:54 9806745 SdtBase::procDataBufWrkr: Buffer stage [3], generated an error [103][Failed to decrypt data: wrong decryption key.]. RCId [15557559]
    28928 75fc  05/15 11:14:54 9806745 SdtBase::setLastErr: Setting last err [103][Failed to decrypt data: wrong decryption key.] RCId [15557559]
    28928 55ec  05/15 11:14:54 9806745 SdtBase::allocateBuffer(): error_set found RCId [15557559]
    28928 55ec  05/15 11:14:54 9806745 SdtSignature::AllocateBuffer [ERROR]Failed to allocate SDT buffer. SDT Error [Failed to decrypt data: wrong decryption key.]
    28928 55ec  05/15 11:14:54 9806745 CDeDupCache::AllocateCallerBufferAndSend():[ERROR]Allocate CallerBuffer Failed
    28928 55ec  05/15 11:14:54 9806745 CDeDupCache::MoveBuffer():[ERROR]AllocateCallerBufferAndSend Failed
    28928 55ec  05/15 11:14:54 9806745 CDeDupCache::DispatchTagList():[ERROR]ConstructAndSendBuffer3 Failed
    28928 60f8  05/15 11:14:54 9806745 SdtBase::allocateBuffer(): error_set found RCId [15557559]

    It looks like there is a wildcard rule here to push traffic to * via cv001ma08b as proxy, I would recommend switching the state of the incoming connection from rrgpsmexc16 on the Network Route Configuration settings of cv001ma08b from BLOCKED to RESTRICTED and seeing whether this is enough to route traffic to the Commserve. If it still fails after this change, then I would recommend to also add and Outgoing Route on the client to route the connection to the CS via the proxy (cv001ma08b).

    David

  • Re: Error: Cannot Allocate Pipeline Buffer on FS backup
    Posted: 05-28-2020, 12:13 PM
    • Aplynx is not online. Last active: 06-30-2020, 12:58 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,846
    Unless you are using a proxy to get back to the CommServe, the client is having an issue resolving the name 'CV001CS.MO-SERVICES.IT'

    Ping -a CV001CS.MO-SERVICES.IT should result in same. Hostfile, DNS entry, DIP or changing the CS hostname in the client properties to the IP should correct.

    28928 75fc 05/15 11:14:54 9806745 CvFwClient::_connect() - ERROR: Unable to resolve remote host CV001CS.MO-SERVICES.IT to IPv4 family: 0x80072AF9:{CQiIPAddr::GetAllByName(216)/W32.11001.(No such host is known.)}

    One way firewall from CS to client would skip name resolution coming back.
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy