Connection errors

Last post 03-23-2018, 2:09 AM by Wwong. 9 replies.
Sort Posts: Previous Next
  • Connection errors
    Posted: 03-12-2018, 4:28 AM

    Hello, I have permanent this connection errors to laptops.

    "Error Code: [19:1327] Description: Attempt start error: [[CCvNetwork::AsyncAttach_RemoteCommand]:Remote system [laptopclient.domain.local]. Communications Service returned error when executing remote command [clBackup.exe] arguments [-j 1003533 -a 2:1706 -t 2 -i 3 -d cvmediaagent.domain.local*cvmediaagent*8400*8402 -io 1 -jt 1003533:7:6:3 -idxma cvmediaagent.domain.local*cvmediaagent*8400*8402 -OSInfo -h -w -ot 1 -numstreams 1 -ab 0 -r 1520582583 -c 0 -appType 33 -CFI -fffound].] Source: at0scvcs01, Process: JobManager"

    When I make a cvping to the laptop (port 8400 and 8402) it is successful.

    What does this error exactly mean?

    Commvault V11 SP10 latest Hotfixes.

  • Re: Connection errors
    Posted: 03-12-2018, 8:59 AM

    Hello,

    We need complete Client CVD logs and CS Jobmanager logs to analyze it further.

    Thanks

    Hemant

  • Re: Connection errors
    Posted: 03-12-2018, 7:58 PM

    Hi rba76

    From the review of the error, the issue could be related to communication between the CS/MA to the Laptop client.

    Although CVping work, all that indicates is that we are able to communicate on 8400 and 8402 from the CS/MA to the Laptop client. What we won't know until we have further logs (cvd or cvfwd (assuming you are also setup with CommVault Firewall)) is whether there is disconnection during the course of the backup. 

    If you can provide log snippets at the time of the issue on the CS/MA and Laptop client we will have more of a holistic view of the issue. 

    Thank you 

    Winston

  • Re: Connection errors
    Posted: 03-13-2018, 9:58 AM

    Hi,

    We were got the same kind of error for windows clients. Check the following things to address the issue.

    -Create a 1 or 2 way fireewall communication between them..

    -check all required ports are open beween CV/MA/Clients. for example we use 8400-8410 for communication.

    -Check DNS related issue.

     

    Adhav

  • Re: Connection errors
    Posted: 03-16-2018, 3:05 AM

    I created now a 2-way-firewall communication between client and CS/MA. The communication is better now.

    But I get these errors in cvfwd.log on the client. Why does the client always reset the tunnel to the CS/MA and proxy in the dmz.

    4692 1ee0 03/16 07:51:01 TN:00004 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "cvclient01" to "commserv"via (ANY, 172.20.20.39)
    4692 1ee0 03/16 07:51:01 TN:00005 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "cvclient01" to "mediaagent" via (ANY, 172.20.20.82)
    4692 1ee0 03/16 07:51:01 TN:00006 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "cvclient01" to "proxyindmz" via (ANY, 172.20.10.90)

     

    Here is the whole cvfwd.log from the client:

    cvfwd.log:

    4692 1ee0 03/16 07:50:31 ######## ######## Merging configs in one...
    4692 1ee0 03/16 07:50:31 ######## ######## Current CONNECT_TIMEOUT is 120 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current RESOLVE_TIMEOUT is 120 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current TUNNEL_STARTUP_TIMEOUT is 120 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current TUNNEL_INIT_INTERVAL is 30 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current MAX_TUNNEL_INIT_ATTEMPTS is 15
    4692 1ee0 03/16 07:50:31 ######## ######## Current IFACE_CHECK_INTERVAL is 30 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current CONFIG_CHECK_INTERVAL is 60 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current MAX_CLIENT_BUFFER_SIZE is 2097152
    4692 1ee0 03/16 07:50:31 ######## ######## Current INIT_TUNNEL_BUFFER_SIZE is 32768
    4692 1ee0 03/16 07:50:31 ######## ######## Current INIT_CLIENT_BUFFER_SIZE is 32768
    4692 1ee0 03/16 07:50:31 ######## ######## Current MAX_ON_DEMAND_IDLE_TIME is 300 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current MAX_CLIENT_PATCH_TIME is 120 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current SHUTDOWN_SOCKET_TIMEOUT is 30 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current MAX_LOG_FILE_SIZE is 5 MB
    4692 1ee0 03/16 07:50:31 ######## ######## Current DISABLE_CVFWD is 0
    4692 1ee0 03/16 07:50:31 ######## ######## Detected 4 CPUs
    4692 1ee0 03/16 07:50:31 ######## ######## Current MAX_SSL_WORKER_THREADS is 6
    4692 1ee0 03/16 07:50:31 ######## ######## Current FD_SETSIZE is 65536
    4692 1ee0 03/16 07:50:31 ######## ######## Current LOG_SOCKET_THRESH is 1000
    4692 1ee0 03/16 07:50:31 ######## ######## Current DYNAMIC_TPPM_TIMEOUT is 300 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current VPN_ROUTER_TOKEN_IDLE_TIME is 3600 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current VPN_CS_TOKEN_IDLE_TIME is 7200 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Current nChatterFlag is 0
    4692 1ee0 03/16 07:50:31 ######## ######## Current nIgnoreChatterFlag is 0
    4692 1ee0 03/16 07:50:31 ######## ######## Current nVpnRcaEnabled is 1
    4692 1ee0 03/16 07:50:31 ######## ######## Current nVpnRouterEnabled is 0
    4692 1ee0 03/16 07:50:31 ######## ######## Current TRACK_CS is 1
    4692 1ee0 03/16 07:50:31 ######## ######## Current DEBUG is *all-tun-clnt-sock-iot-ssl-sslv-throt-throtv-ccmd-tcmd-tcmdv-vpn*
    4692 1ee0 03/16 07:50:31 ######## ######## Remove '*' and replace '-' with '+' to selectively enable logging
    4692 1ee0 03/16 07:50:31 ######## ########    +tun    to see additional per-tunnel logs
    4692 1ee0 03/16 07:50:31 ######## ########    +clnt   to see additional per-client connection logs
    4692 1ee0 03/16 07:50:31 ######## ########    +sock   to see extra socket-related logs
    4692 1ee0 03/16 07:50:31 ######## ########    +iot    to see I/O tracker logs (very verbose)
    4692 1ee0 03/16 07:50:31 ######## ########    +ssl    to see SSL-related logs
    4692 1ee0 03/16 07:50:31 ######## ########    +sslv   to see lots more SSL-related logs (very verbose)
    4692 1ee0 03/16 07:50:31 ######## ########    +throt  to see throttling-related logs
    4692 1ee0 03/16 07:50:31 ######## ########    +throtv to see lots more throttling-related logs
    4692 1ee0 03/16 07:50:31 ######## ########    +ccmd   to see every command sent or received from clients
    4692 1ee0 03/16 07:50:31 ######## ########    +tcmd   to see most of commands sent through tunnels
    4692 1ee0 03/16 07:50:31 ######## ########    +tcmdv  to see all commands sent through tunnels
    4692 1ee0 03/16 07:50:31 ######## ########    +vpn    to see VPN-related messages
    4692 1ee0 03/16 07:50:31 ######## ########    +all    all of the above
    4692 1ee0 03/16 07:50:31 ######## ######## Current configuration:
    4692 1ee0 03/16 07:50:31 ######## ######## [general]
    4692 1ee0 03/16 07:50:31 ######## ######## keepalive_interval=300
    4692 1ee0 03/16 07:50:31 ######## ######## tunnel_init_interval=30
    4692 1ee0 03/16 07:50:31 ######## ######## force_incoming_ssl=0
    4692 1ee0 03/16 07:50:31 ######## ######## lockdown=1
    4692 1ee0 03/16 07:50:31 ######## ######## proxy=0
    4692 1ee0 03/16 07:50:31 ######## ######## bind_open_ports_only=0
    4692 1ee0 03/16 07:50:31 ######## ######## dynamic_tppm_iface=localhost
    4692 1ee0 03/16 07:50:31 ######## ######## dynamic_tppm_ports=any
    4692 1ee0 03/16 07:50:31 ######## ######## [http-proxy]=NO
    4692 1ee0 03/16 07:50:31 ######## ######## [incoming]
    4692 1ee0 03/16 07:50:31 ######## ######## tunnel_ports=8403
    4692 1ee0 03/16 07:50:31 ######## ######## [outgoing]
    4692 1ee0 03/16 07:50:31 ######## ######## cvclient01 commserv remote_guid=E7F92EB7-AE34-4140-A33B-8DEA38D8A345 bypassable=1 proxy=proxyindmz
    4692 1ee0 03/16 07:50:31 ######## ######## cvclient01 commserv remote_guid=E7F92EB7-AE34-4140-A33B-8DEA38D8A345 bypassable=1 type=persistent proto=http cvfwd=commserv.domain.local:8403 encryption=med
    4692 1ee0 03/16 07:50:31 ######## ######## cvclient01 mediaagent remote_guid=7B120021-60AE-407E-AACE-108BF2D4C9F7 bypassable=1 proxy=proxyindmz
    4692 1ee0 03/16 07:50:31 ######## ######## cvclient01 mediaagent remote_guid=7B120021-60AE-407E-AACE-108BF2D4C9F7 bypassable=1 type=persistent proto=http cvfwd=mediaagent.domain.local:8403 encryption=med
    4692 1ee0 03/16 07:50:31 ######## ######## cvclient01 proxyindmz remote_guid=5038A52A-D5EF-94AA-E52C-96A8D5C33FD1 bypassable=1 type=persistent proto=https cvfwd=gateway.publicdomain.com:8411 encryption=med
    4692 1ee0 03/16 07:50:31 ######## ######## [throttling]
    4692 1ee0 03/16 07:50:31 ######## ######## group1_remote_clients=proxyindmz
    4692 1ee0 03/16 07:50:31 ######## ######## group1_sun_00:00=0,0/1024/40%
    4692 1ee0 03/16 07:50:31 ######## ######## group1_mon_00:00=0,0/1024/40%
    4692 1ee0 03/16 07:50:31 ######## ######## group1_tue_00:00=0,0/1024/40%
    4692 1ee0 03/16 07:50:31 ######## ######## group1_wed_00:00=0,0/1024/40%
    4692 1ee0 03/16 07:50:31 ######## ######## group1_thu_00:00=0,0/1024/40%
    4692 1ee0 03/16 07:50:31 ######## ######## group1_fri_00:00=0,0/1024/40%
    4692 1ee0 03/16 07:50:31 ######## ######## group1_sat_00:00=0,0/1024/40%
    4692 1ee0 03/16 07:50:31 ######## ######## Creating IPv4 socket for listening to client connections.
    4692 1ee0 03/16 07:50:31 ######## ######## Client fd=1040 bound to 0.0.0.0:51453.
    4692 1ee0 03/16 07:50:31 ######## ######## Creating IPv6 socket for listening to client connections.
    4692 1ee0 03/16 07:50:31 ######## ######## Client fd=1080 bound to :::51456.
    4692 1ee0 03/16 07:50:31 ######## ######## Found throttling group "group1"
    4692 1ee0 03/16 07:50:31 ######## ######## ERROR: insert_timeout(): Timeout is too far in future, reducing to 7200 secs
    4692 1ee0 03/16 07:50:31 ######## ######## Tunnel fd=1016 bound to IPv4 port 8403.
    4692 1ee0 03/16 07:50:31 ######## ######## Start listening to incoming messages.
    4692 1ee0 03/16 07:50:31 ######## ######## Initializing SSL subsystem
    4692 1ee0 03/16 07:50:32 TN:00002 ######## Initialized new PERSISTENT tunnel from "cvclient01" to "mediaagent" via (172.20.121.165, 172.20.20.82)
    4692 1ee0 03/16 07:50:32 TN:00001 ######## Initialized new PERSISTENT tunnel from "cvclient01" to "commserv"via (172.20.121.165, 172.20.20.39)
    4692 1ee0 03/16 07:50:33 ######## ######## CS is ONLINE
    4692 1ee0 03/16 07:50:33 TN:00003 ######## Initialized new PERSISTENT tunnel from "cvclient01" to "proxyindmz" via (172.20.121.165, 172.20.10.90)
    4692 1ee0 03/16 07:51:01 ######## ######## Detected a change in IP configuration. Sending KEEP_ALIVEs through all tunnels.
    4692 1ee0 03/16 07:51:01 ######## ######## Detected a change in IP configuration. Re-reading config files.
    4692 1ee0 03/16 07:51:01 TN:00004 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "cvclient01" to "commserv"via (ANY, 172.20.20.39)
    4692 1ee0 03/16 07:51:01 TN:00005 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "cvclient01" to "mediaagent" via (ANY, 172.20.20.82)
    4692 1ee0 03/16 07:51:01 TN:00006 ######## ERROR: cvfwd_reset_tunnel(): Resetting PERSISTENT tunnel from "cvclient01" to "proxyindmz" via (ANY, 172.20.10.90)

  • Re: Connection errors
    Posted: 03-16-2018, 4:14 AM

    so proxy setup is used.

    This means there must be a persistent tunnel between client -> proxy, CS -> proxy and MA -> proxy

    can you pls check the fwconfig.txt / fw rules for all three servers ?
    the proxy has to be passive, while all other are forced to create a persitent tunnel towards the proxy

    what I don't understand is this combination of proxy and direct communication in the fw logs.
    never saw this before. I only expect direct communication towards the proxy and proxied communication toward CS and MA

    and the port used is 8403/tcp 

    <...>
    4692 1ee0 03/16 07:50:31 ######## ######## cvclient01 commserv remote_guid=E7F92EB7-AE34-4140-A33B-8DEA38D8A345 bypassable=1 proxy=proxyindmz 

    4692 1ee0 03/16 07:50:31 ######## ######## cvclient01 commserv remote_guid=E7F92EB7-AE34-4140-A33B-8DEA38D8A345 bypassable=1 type=persistent proto=http cvfwd=commserv.domain.local:8403 encryption=med 
    <...>

    when using proxies, all traffic between client and CS/MA will be tunneled through the proxy. so no direct connection between client am CS/MA is required

    regards
    Klaus

  • Re: Connection errors
    Posted: 03-16-2018, 4:40 AM

    Hello Klaus,

    yes I use a proxy in the DMZ. But this proxy should only be used if the client (laptop) is outside from the company network. If the client is within the company network the laptop can use a direct connection to the CS/MA. Because I don't want to route all the backuptraffic throught the firewall, if the client is within the companynetwork.

    Should I not use a setup like this?

    This is the fwconfig.txt from the client:

    [general]
    keepalive_interval=300
    tunnel_init_interval=30
    force_incoming_ssl=0
    lockdown=1
    proxy=0
    bind_open_ports_only=0

    # This section describes tunnel server port and the list of additional
    # incoming ports that may be open and used to set up efficient data transfer
    [incoming]
    tunnel_ports=8403


    # This section describes outgoing routes
    [outgoing]
    cvclient01 commserv remote_guid=E7F92EB7-AE34-4140-A33B-8DEA38D8A345 type=persistent proto=http cvfwd=commserv.domain.local:8403 bypassable=1
    cvclient01 mediaagent remote_guid=7B120021-60AE-407E-AACE-108BF2D4C9F7 type=persistent proto=http cvfwd=mediaagent.domain.local:8403 bypassable=1
    cvclient01 proxyindmz remote_guid=5038A52A-D5EF-94AA-E52C-96A8D5C33FD1 type=persistent proto=https cvfwd=proxy.publicdomain.com:8411 bypassable=1
    cvclient01 commserv proxy=proxyindmz remote_guid=E7F92EB7-AE34-4140-A33B-8DEA38D8A345 bypassable=1
    cvclient01 mediaagent proxy=proxyindmz remote_guid=7B120021-60AE-407E-AACE-108BF2D4C9F7 bypassable=1

  • Re: Connection errors
    Posted: 03-21-2018, 8:04 PM

    Hi rba76

    The above configuration where you use "proxy should only be used if the client (laptop) is outside from the company network. If the client is within the company network the laptop can use a direct connection to the CS/MA." -> this is correct

    https://documentation.commvault.com/commvault/v11/article?p=7380.htm

    The additional "bypassable=1" entry is to dictate when a Laptop client is within the environment to use internal network routes rather then the proxy.

    Thank you 

    Winston 

  • Re: Connection errors
    Posted: 03-22-2018, 3:23 AM

    Hi Winston,

    I already selected the checkbox for roaming client. In the meanwhile I found out, if the client is connected per WLAN and after a while it change to a LAN connection, the edge client have problems to establish a connection again to the CS/MA. After I restart the CV services on the client, the connection will be established immediately.

    In the console I see in "Reason for jobdelay" on the client "Error Code: [19:2166] Description: This job was initiated by an Automatic schedule. It is ready to be resumed by the client. Source: at0scvcs01, Process: JobManager".

    I really don't know what the problem is?

  • Re: Connection errors
    Posted: 03-23-2018, 2:09 AM

    Hi rba76

    From this point onwards I would recommend opening a Ticket with Support, so they can review and investigate the cause of the alert.

    Thank you

    Winston

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2018 Commvault | All Rights Reserved. | Legal | Privacy Policy