I can't backup some virtual machines working on VMware with SAN transport mode.

Last post 12-03-2020, 1:25 AM by ibrahimemrekaya. 8 replies.
Sort Posts: Previous Next
  • I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 10-12-2020, 9:30 AM

    Hello,

    We updated Commvault to v11.20.17 last week. After the update we couldn't backup some virtual machines with san transport mode. Our previous version is v11.19.18.

    VixDiskLib.log;

    11116 32dc 10/12 13:53:07 70402 Opening file [] /.vmdk (vpxa-nfc://[] /.vmdk@.local:902)
    11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.590+03:00 error -[00300] [Originator@6876 sub=BlockList] Other error encountered: Operation timed out
    11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.590+03:00 error -[00300] [Originator@6876 sub=Default] San transport error: Unspecified error.
    11116 12c 10/12 13:53:09 70402 DISKLIB-LIB : RWv failed ioId: #4102 (318) (62) .
    11116 12c 10/12 13:53:09 70402 VixDiskLib: Detected DiskLib error 318 (Unknown error).
    11116 12c 10/12 13:53:09 70402 VixDiskLib: VixDiskLib_Read: Read 1 sectors at 7317504 failed. Error 1 (Unknown error) (DiskLib error 318: Unknown error) at 6151.
    11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.637+03:00 error -[00300] [Originator@6876 sub=BlockList] Other error encountered: Operation timed out
    11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.637+03:00 error -[00300] [Originator@6876 sub=Default] San transport error: Unspecified error.
    11116 12c 10/12 13:53:09 70402 DISKLIB-LIB : RWv failed ioId: #4147 (318) (62) .
    11116 12c 10/12 13:53:09 70402 VixDiskLib: Detected DiskLib error 318 (Unknown error).
    11116 12c 10/12 13:53:09 70402 VixDiskLib: VixDiskLib_Read: Read 2 sectors at 7317504 failed. Error 1 (Unknown error) (DiskLib error 318: Unknown error) at 6151.
    11116 29c4 10/12 13:53:09 70402 2020-10-12T13:53:09.637+03:00 warning -[10692] [Originator@6876 sub=Default] Resolve failed for (null): Operation has been cancelled (The operation completed successfully)
    11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.652+03:00 error -[00300] [Originator@6876 sub=BlockList] Other error encountered: Operation timed out
    11116 cf0 10/12 13:53:09 70402 2020-10-12T13:53:09.652+03:00 warning -[03312] [Originator@6876 sub=Default] Resolve failed for (null): Operation has been cancelled (The operation completed successfully)
    11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.652+03:00 error -[00300] [Originator@6876 sub=Default] San transport error: Unspecified error.
    11116 12c 10/12 13:53:09 70402 DISKLIB-LIB : RWv failed ioId: #4149 (318) (62) .
    11116 12c 10/12 13:53:09 70402 VixDiskLib: Detected DiskLib error 318 (Unknown error).
    11116 12c 10/12 13:53:09 70402 VixDiskLib: VixDiskLib_Read: Read 1 sectors at 7317504 failed. Error 1 (Unknown error) (DiskLib error 318: Unknown error) at 6151.
    11116 32dc 10/12 13:53:10 70402 VixDiskLib: VixDiskLib_QueryAllocatedBlocks: Query allocated blocks.
    11116 32dc 10/12 13:53:10 70402 NBD_ClientOpen: attempting to create connection to vpxa-nfc://[] /.vmdk@.local:902
    11116 32dc 10/12 13:53:10 70402 Started up WSA
    11116 32dc 10/12 13:53:10 70402 CnxOpenTCPSocket: Couldn't set early TCP timeout connecting to server .local:902: 19
    11116 32dc 10/12 13:53:11 70402 Setting NFC log level to 1
    11116 32dc 10/12 13:53:11 70402 Setting NFC log level to 1
    11116 32dc 10/12 13:53:11 70402 NFC Async IO session is established with log level 1.
    11116 32dc 10/12 13:53:11 70402 Opening file [] /.vmdk (vpxa-nfc://[] /.vmdk@.local:902)
    11116 2c3c 10/12 13:53:14 70402 2020-10-12T13:53:14.058+03:00 warning -[11324] [Originator@6876 sub=Default] Resolve failed for (null): Operation has been cancelled (The operation completed successfully)
    11116 12c 10/12 13:53:14 70402 2020-10-12T13:53:14.058+03:00 error -[00300] [Originator@6876 sub=BlockList] Other error encountered: Operation timed out
    11116 12c 10/12 13:53:14 70402 2020-10-12T13:53:14.058+03:00 error -[00300] [Originator@6876 sub=Default] San transport error: Unspecified error.
    11116 12c 10/12 13:53:14 70402 DISKLIB-LIB : RWv failed ioId: #4314 (318) (62) .
    11116 12c 10/12 13:53:14 70402 VixDiskLib: Detected DiskLib error 318 (Unknown error).
    11116 12c 10/12 13:53:14 70402 VixDiskLib: VixDiskLib_Read: Read 2048 sectors at 2099200 failed. Error 1 (Unknown error) (DiskLib error 318: Unknown error) at 6151.
    11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_Close: Close disk.
    11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_Disconnect: Disconnect.
    11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
    11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_Cleanup: Remove previous mount points and clean up .
    11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_Connect: Establish connection.
    11116 32dc 10/12 13:53:15 70402 VixDiskLib: Resolve host.
    11116 2c3c 10/12 13:53:16 70402 2020-10-12T13:53:16.043+03:00 warning -[11324] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
    11116 32dc 10/12 13:53:16 70402 2020-10-12T13:53:16.043+03:00 warning -[13020] [Originator@6876 sub=vimaccess] cannot get thumbprint: SSL error code '151441516', exception: 'Wrong X.509 Certificate format'
    11116 32dc 10/12 13:53:16 70402 VixDiskLib: VixDiskLib_Disconnect: Disconnect.
    11116 32dc 10/12 13:53:16 70402 VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
    11116 12c 10/12 13:53:16 70402 VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
    11116 32dc 10/12 13:53:31 70402 VDDK_PhoneHome: VDDK PhoneHome succeed to exit
    11116 32dc 10/12 13:53:31 70402 VixDiskLib: VixDiskLib_Exit: Unmatched Init calls so far: 1.
    11116 32dc 10/12 13:53:32 70402 OBJLIB-LIB: ObjLib cleanup done.
     

    vsbkp.log;
    11116 12c 10/12 13:53:11 70402 VSBkpWorker::BackupVMDisk() - Extent Size [1048576] bytes [2048] sectors [512] bytes per sector
    11116 12c 10/12 13:53:11 70402 VSBkpWorker::BackupVMDisk() - Backing up disk/volume [.vmdk] [64,424,509,440] bytes
    11116 2110 10/12 13:53:11 70402 VSBkpWorker::ArchiveExtentProcessor() - Send volume or disk data to MA thread
    11116 12c 10/12 13:53:14 70402 CVMDiskInfo::ReadDisk() - VixDiskLib_Read failed Err=1 Unknown error
    11116 12c 10/12 13:53:14 70402 VSBkpWorker::BackupVMDisk() - Failed to read from virtual disk [.vmdk] block [1025], [0x80070306:{CArchiveVirtualDiskFile::GetNextBlock(1423)} + {CVmdkVcb4ExtentReader::ReadBlock(690)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-Failed to ReadCurrVmdkFile}]
    11116 12c 10/12 13:53:14 70402 VSBkpWorker::BackupVMDisk() - Waiting for request(s) to complete...
    11116 2110 10/12 13:53:14 70402 VSBkpWorker::ArchiveExtentProcessor() - Done...
     
    I deleted server names etc. from logs.
     
    I created new subclient with failed virtual machines. I can't backup with auto or san transport mode but I can with nbd.
     
    Anyone can help me with this situation?
     
    BR,
    İbrahim
     

    Best Regards,
    İbrahim
  • Re: I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 10-15-2020, 9:08 AM

    Hi İbrahim,

     

    Can you open a ticket with support?

    I beleive this is an issue with VDDK version 7 on the proxies and they can help you downgrade to 673.

    are you on Vcenter Version 7 ?


    Gary Seibak
    Technical Account Manager - Commvault
  • Re: I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 10-15-2020, 9:15 AM

    Hi Gary,

    Thank you for your reply.

    I didn't want to open a call because maybe it's a simple problem.

    Our vCenter is 6.7


    Best Regards,
    İbrahim
  • Re: I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 10-15-2020, 10:00 AM

    I opened a case.

    I will update when it is done.


    Best Regards,
    İbrahim
  • Re: I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 10-16-2020, 2:12 AM

    Hello,

    I'm sharing Commvault Engineer's reply;

    "SP20 is using VDDK 7.0 and we have several other cases in development escalation for the same issue. From what I can see, there's an issue with VMware's VDDK 7.0 with SAN transport and our developers are working with VMware on it."

    I think they will publish hotfix for this issue.


    Best Regards,
    İbrahim
  • Re: I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 11-05-2020, 8:52 AM

    Hi,

     

    we have get the following info from support:

     

    Our engineering team is still working on fix for this issue.

     

    In the meantime you can add the following 2 additional settings to your VSA proxies to leverage VDDK 673

     

    Name: nVDDKVersionPath
    Category: VirtualServer
    Type: Integer
    Value: 673

     

    Name: bLockVDDKVersionPath
    Category: VirtualServer
    Type: Boolean
    Value: True

     

    Adding Additional Settings:
    https://documentation.commvault.com/commvault/v11/article?p=8681.htm

     

    in our case it helps

     

  • Re: I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 11-10-2020, 11:37 AM
    • Aplynx is not online. Last active: 12-04-2020, 12:53 PM Liam
    • Top 10 Contributor
    • Joined on 05-04-2010
    • New Jersey
    • Master
    • Points 1,890

    We've escalated the issue direct to VMware and our Development Team is working with thiers to resolve the issue. For now please use the workaround to downgrade the VDDK.

     

    Name: nVDDKVersionPath
    Category: VirtualServer
    Type: Integer
    Value: 673

     

    Name: bLockVDDKVersionPath
    Category: VirtualServer
    Type: Boolean
    Value: True

     

    Adding Additional Settings:
    https://documentation.commvault.com/commvault/v11/article?p=8681.htm

  • Re: I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 11-10-2020, 12:08 PM

    Hello,

    Yes, i opened a case at 15 October and they said me we are working with vmware for solution. I changed transport mode to nbd on subclients. We can backup without an issue. Customer know this situation and they don't bother me for this. We can wait hotfix etc. for that issue. But if you want me to try thies additional settings, i can do them for you.

    BR,

    İbrahim


    Best Regards,
    İbrahim
  • Re: I can't backup some virtual machines working on VMware with SAN transport mode.
    Posted: 12-03-2020, 1:25 AM

    Hello,

    Commvault engineer sent to me a patch. I updated Commvault to v11.20.27 and installed patch on VSA machine.

    I changed subclient's transport mode to auto. All backups working last night were successfully completed. SAN transport mode working now.

    Thank you all.

    Have a healthy day.


    Best Regards,
    İbrahim
The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2020 Commvault | All Rights Reserved. | Legal | Privacy Policy