Hello,

We updated Commvault to v11.20.17 last week. After the update we couldn't backup some virtual machines with san transport mode. Our previous version is v11.19.18.

VixDiskLib.log;

11116 32dc 10/12 13:53:07 70402 Opening file [] /.vmdk (vpxa-nfc://[] /.vmdk@.local:902)
11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.590+03:00 error -[00300] [Originator@6876 sub=BlockList] Other error encountered: Operation timed out
11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.590+03:00 error -[00300] [Originator@6876 sub=Default] San transport error: Unspecified error.
11116 12c 10/12 13:53:09 70402 DISKLIB-LIB : RWv failed ioId: #4102 (318) (62) .
11116 12c 10/12 13:53:09 70402 VixDiskLib: Detected DiskLib error 318 (Unknown error).
11116 12c 10/12 13:53:09 70402 VixDiskLib: VixDiskLib_Read: Read 1 sectors at 7317504 failed. Error 1 (Unknown error) (DiskLib error 318: Unknown error) at 6151.
11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.637+03:00 error -[00300] [Originator@6876 sub=BlockList] Other error encountered: Operation timed out
11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.637+03:00 error -[00300] [Originator@6876 sub=Default] San transport error: Unspecified error.
11116 12c 10/12 13:53:09 70402 DISKLIB-LIB : RWv failed ioId: #4147 (318) (62) .
11116 12c 10/12 13:53:09 70402 VixDiskLib: Detected DiskLib error 318 (Unknown error).
11116 12c 10/12 13:53:09 70402 VixDiskLib: VixDiskLib_Read: Read 2 sectors at 7317504 failed. Error 1 (Unknown error) (DiskLib error 318: Unknown error) at 6151.
11116 29c4 10/12 13:53:09 70402 2020-10-12T13:53:09.637+03:00 warning -[10692] [Originator@6876 sub=Default] Resolve failed for (null): Operation has been cancelled (The operation completed successfully)
11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.652+03:00 error -[00300] [Originator@6876 sub=BlockList] Other error encountered: Operation timed out
11116 cf0 10/12 13:53:09 70402 2020-10-12T13:53:09.652+03:00 warning -[03312] [Originator@6876 sub=Default] Resolve failed for (null): Operation has been cancelled (The operation completed successfully)
11116 12c 10/12 13:53:09 70402 2020-10-12T13:53:09.652+03:00 error -[00300] [Originator@6876 sub=Default] San transport error: Unspecified error.
11116 12c 10/12 13:53:09 70402 DISKLIB-LIB : RWv failed ioId: #4149 (318) (62) .
11116 12c 10/12 13:53:09 70402 VixDiskLib: Detected DiskLib error 318 (Unknown error).
11116 12c 10/12 13:53:09 70402 VixDiskLib: VixDiskLib_Read: Read 1 sectors at 7317504 failed. Error 1 (Unknown error) (DiskLib error 318: Unknown error) at 6151.
11116 32dc 10/12 13:53:10 70402 VixDiskLib: VixDiskLib_QueryAllocatedBlocks: Query allocated blocks.
11116 32dc 10/12 13:53:10 70402 NBD_ClientOpen: attempting to create connection to vpxa-nfc://[] /.vmdk@.local:902
11116 32dc 10/12 13:53:10 70402 Started up WSA
11116 32dc 10/12 13:53:10 70402 CnxOpenTCPSocket: Couldn't set early TCP timeout connecting to server .local:902: 19
11116 32dc 10/12 13:53:11 70402 Setting NFC log level to 1
11116 32dc 10/12 13:53:11 70402 Setting NFC log level to 1
11116 32dc 10/12 13:53:11 70402 NFC Async IO session is established with log level 1.
11116 32dc 10/12 13:53:11 70402 Opening file [] /.vmdk (vpxa-nfc://[] /.vmdk@.local:902)
11116 2c3c 10/12 13:53:14 70402 2020-10-12T13:53:14.058+03:00 warning -[11324] [Originator@6876 sub=Default] Resolve failed for (null): Operation has been cancelled (The operation completed successfully)
11116 12c 10/12 13:53:14 70402 2020-10-12T13:53:14.058+03:00 error -[00300] [Originator@6876 sub=BlockList] Other error encountered: Operation timed out
11116 12c 10/12 13:53:14 70402 2020-10-12T13:53:14.058+03:00 error -[00300] [Originator@6876 sub=Default] San transport error: Unspecified error.
11116 12c 10/12 13:53:14 70402 DISKLIB-LIB : RWv failed ioId: #4314 (318) (62) .
11116 12c 10/12 13:53:14 70402 VixDiskLib: Detected DiskLib error 318 (Unknown error).
11116 12c 10/12 13:53:14 70402 VixDiskLib: VixDiskLib_Read: Read 2048 sectors at 2099200 failed. Error 1 (Unknown error) (DiskLib error 318: Unknown error) at 6151.
11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_Close: Close disk.
11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_Disconnect: Disconnect.
11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_Cleanup: Remove previous mount points and clean up .
11116 32dc 10/12 13:53:15 70402 VixDiskLib: VixDiskLib_Connect: Establish connection.
11116 32dc 10/12 13:53:15 70402 VixDiskLib: Resolve host.
11116 2c3c 10/12 13:53:16 70402 2020-10-12T13:53:16.043+03:00 warning -[11324] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
11116 32dc 10/12 13:53:16 70402 2020-10-12T13:53:16.043+03:00 warning -[13020] [Originator@6876 sub=vimaccess] cannot get thumbprint: SSL error code '151441516', exception: 'Wrong X.509 Certificate format'
11116 32dc 10/12 13:53:16 70402 VixDiskLib: VixDiskLib_Disconnect: Disconnect.
11116 32dc 10/12 13:53:16 70402 VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
11116 12c 10/12 13:53:16 70402 VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
11116 32dc 10/12 13:53:31 70402 VDDK_PhoneHome: VDDK PhoneHome succeed to exit
11116 32dc 10/12 13:53:31 70402 VixDiskLib: VixDiskLib_Exit: Unmatched Init calls so far: 1.
11116 32dc 10/12 13:53:32 70402 OBJLIB-LIB: ObjLib cleanup done.

vsbkp.log;
11116 12c 10/12 13:53:11 70402 VSBkpWorker::BackupVMDisk() - Extent Size [1048576] bytes [2048] sectors [512] bytes per sector
11116 12c 10/12 13:53:11 70402 VSBkpWorker::BackupVMDisk() - Backing up disk/volume [.vmdk] [64,424,509,440] bytes
11116 2110 10/12 13:53:11 70402 VSBkpWorker::ArchiveExtentProcessor() - Send volume or disk data to MA thread
11116 12c 10/12 13:53:14 70402 CVMDiskInfo::ReadDisk() - VixDiskLib_Read failed Err=1 Unknown error
11116 12c 10/12 13:53:14 70402 VSBkpWorker::BackupVMDisk() - Failed to read from virtual disk [.vmdk] block [1025], [0x80070306:{CArchiveVirtualDiskFile::GetNextBlock(1423)} + {CVmdkVcb4ExtentReader::ReadBlock(690)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-Failed to ReadCurrVmdkFile}]
11116 12c 10/12 13:53:14 70402 VSBkpWorker::BackupVMDisk() - Waiting for request(s) to complete...
11116 2110 10/12 13:53:14 70402 VSBkpWorker::ArchiveExtentProcessor() - Done...

I deleted server names etc. from logs.

I created new subclient with failed virtual machines. I can't backup with auto or san transport mode but I can with nbd.

Anyone can help me with this situation?

BR,

İbrahim

Hi İbrahim,

Can you open a ticket with support?

I beleive this is an issue with VDDK version 7 on the proxies and they can help you downgrade to 673.

are you on Vcenter Version 7 ?

Hi Gary,

Thank you for your reply.

I didn't want to open a call because maybe it's a simple problem.

Our vCenter is 6.7

I opened a case.

I will update when it is done.

Hello,

I'm sharing Commvault Engineer's reply;

"SP20 is using VDDK 7.0 and we have several other cases in development escalation for the same issue. From what I can see, there's an issue with VMware's VDDK 7.0 with SAN transport and our developers are working with VMware on it."

I think they will publish hotfix for this issue.

Hi,

we have get the following info from support:

Our engineering team is still working on fix for this issue.

In the meantime you can add the following 2 additional settings to your VSA proxies to leverage VDDK 673

Name: nVDDKVersionPath
Category: VirtualServer
Type: Integer
Value: 673

Name: bLockVDDKVersionPath
Category: VirtualServer
Type: Boolean
Value: True

Adding Additional Settings:
https://documentation.commvault.com/commvault/v11/article?p=8681.htm

in our case it helps

We've escalated the issue direct to VMware and our Development Team is working with thiers to resolve the issue. For now please use the workaround to downgrade the VDDK.

Name: nVDDKVersionPath
Category: VirtualServer
Type: Integer
Value: 673

Name: bLockVDDKVersionPath
Category: VirtualServer
Type: Boolean
Value: True

Adding Additional Settings:
https://documentation.commvault.com/commvault/v11/article?p=8681.htm

Hello,

Yes, i opened a case at 15 October and they said me we are working with vmware for solution. I changed transport mode to nbd on subclients. We can backup without an issue. Customer know this situation and they don't bother me for this. We can wait hotfix etc. for that issue. But if you want me to try thies additional settings, i can do them for you.

BR,

İbrahim

Cant I get this  update too? It drives me crazy.

I don't believe its in the available releases yet. Open a ticket and reference this forum post and you should be able to get it. 

The hotfix for SP20 is 2986 for reference. 

THX.

I had  today also updated to  v11.20.27. I had send a mail to  TIM AG, because I can't open a case directly. 

This error destroys our DR backups for weeks now. And I  had always searched at the wrong side (we had upgraded Vsphere to 7.01 at the same time).

BTW: The workaround seems to work so far.

I had  tried  to update  the VDDK7  to 7.01 manually, but no luck.

VMware changed how it processes certain requests in VDDK 7.0. The workaround drops to the 6.7.3 VDDK and the hotfix will take into account the changed in 7.0. 

Hm, Thx. But also for developers: RTFM. I'am sure, Vmware has an "Whats new and changed" section in the documentation from VDDK 7.