Windows FS iDA - Lengthy pauses when backing up?

Last post 03-07-2011, 4:05 PM by chadtandy. 29 replies.
Sort Posts: Previous Next
  • Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-26-2011, 10:52 AM

    Running 8.0 SP5.  I’ve posted about this before but I’m still no further forward with why it’s happening really.

    See attached screen grab (if it gets through).  I’m not clear why I should be seeing pauses of a minute or so whilst doing backups to tape?

    The chunk size for the 32bit and x64 FS iDA’s has been set to 32gb, 256kb block size on the tape, 8 readers defined on the default sub client (with multiple readers in drives/mount points) going into a single stream SP with multiplexing of 5 set on the copy.

    It probably goes without saying but gig ethernet end to end with no bottlenecks etc.

    Any pointers?

    Thanks,

    Paul


    Attachment: Capture.PNG
  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 9:40 AM

    Hard to say, but i suspect this is when the backup transitions from file backup to the system state.  When the VSS writers need to be checked and engaged for system state backup, very little data is being passed over the network, so this may equate to that gap.

    Remove system state from that subclient would be a sure fire way to verify this.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 2:43 PM

    Vincenzo_Basolino:

    Hard to say, but i suspect this is when the backup transitions from file backup to the system state.  When the VSS writers need to be checked and engaged for system state backup, very little data is being passed over the network, so this may equate to that gap.

    Remove system state from that subclient would be a sure fire way to verify this.

    Thanks Vincenzo, however it doesn't tie in with System State, seems to happen for periods throughout the backup.

    I guess I could run Process Monitor or something on the client to see if anything shows up, but it's something that appears peculiar to Commvault i.e. I can copy gig after gig of other files using, say, Robocopy in multi-threaded mode and the data flow is pretty consistent even if there are variances in throughput, yet with the FS iDA I seem to have these bursts of "nothing".

    We're not yet licensed for D2D but it would be interesting to see if it continues if we go down the capacity based route.

    Actually, that brings me on to another question but that may be best answered in another thread.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 2:54 PM

    When using robocopy ensure you use the /B switch as this would best simulate the APIs leveraged by CV backup. 

    Things that may expose the issue:

    1.  Turn off client side compression and see if the issue continues

    2.  Is the media agent heavy utilized in this case while this job is running?  Ive seen gaps in backup when the src side is sending data to the MA faster then it can be written to media. 

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 2:59 PM

    Vincenzo_Basolino:

    When using robocopy ensure you use the /B switch as this would best simulate the APIs leveraged by CV backup. 

    Things that may expose the issue:

    1.  Turn off client side compression and see if the issue continues

    2.  Is the media agent heavy utilized in this case while this job is running?  Ive seen gaps in backup when the src side is sending data to the MA faster then it can be written to media. 

    Pretty sure client side compression is disabled anyway, but will check when I'm in the office.

    Regarding utilization, the short answer is no, though quite how you can work out if there is a bottleneck I don't know?  The MA is a quad-core Xeon with 4gb of RAM and a SAS attached LTO4 HH tape library (Dell TL2000) and we see around 75mb/sec average transfer speeds.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 3:33 PM

    Based on the screenshot and the size of the gaps, i would say one side or the other is waiting for something to complete or be acknowledged.  During the span of the job no phase would cause such a delay unless it was waiting for media to be mounted (or scan phase).  From that I conclude there is some type of bottle neck causing this behavior, as backup should barrel through sending data over the requested path, as your copy and paste does. 

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 3:44 PM

    Vincenzo_Basolino:

    Based on the screenshot and the size of the gaps, i would say one side or the other is waiting for something to complete or be acknowledged.  During the span of the job no phase would cause such a delay unless it was waiting for media to be mounted (or scan phase).  From that I conclude there is some type of bottle neck causing this behavior, as backup should barrel through sending data over the requested path, as your copy and paste does. 

    Any ideas where to begin trying to narrow it down, seeing as it appears to be a Commvault specific thing?

    I just checked and client side compression is disabled.

    Probably worth clarifying, environment is 8.0 SP5, planning on moving to 9.0 as soon as I'm sure it's not going to brick everything :)

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 3:59 PM

    Start with the robocopy /B as this would isolate if the API being leveraged is a problem. 

    If you could isolate the delay to a block of time within the log files that would be the best way to determine whats happened. 

     

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 4:03 PM

    Vincenzo_Basolino:

    Start with the robocopy /B as this would isolate if the API being leveraged is a problem. 

    If you could isolate the delay to a block of time within the log files that would be the best way to determine whats happened. 

     

    I'll do some testing.  The client is a newly built server so I've robocopied around 7tb to it over the last few weeks with no sign of these dips, and that was using /ZB so it's not likely to be that IMO.

    I will try some robocopy's between the client and the media agent though and see if I get the same thing.

    One thing that I did wonder is that the server, being a file server, does have some project folders that can contain tens of thousands of tiny files for a particular area - wondered if that may have anything to do with it in the same way that you sometimes see robocopy take a while to think/parse the content of such folders?

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 4:03 PM

    Have you checked the CPU and Memory utilization on the client? 8 Readers is A LOT for a default subclient.

    I would also suggest if this is a large amount of data to break it up into smaller subclients of 2 Readers each instead. The question when using the Windows Filesystem agent is how many files. If you are over 2mil files, I would definitely split up into multiple subclients. That iDataAgent isn't really intended to backup billions of small files on a fileserver, but it can be done.

     

     

     


    DR Donnie
    --------------
    There's an evil monkey in my CommCell!
  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 4:12 PM

    drdonniec_v:

    Have you checked the CPU and Memory utilization on the client? 8 Readers is A LOT for a default subclient.

    It doesn't appear to be an issue in that the CPU isn't maxed out and there's plenty of free RAM.

    drdonniec_v:

    I would also suggest if this is a large amount of data to break it up into smaller subclients of 2 Readers each instead. The question when using the Windows Filesystem agent is how many files. If you are over 2mil files, I would definitely split up into multiple subclients. That iDataAgent isn't really intended to backup billions of small files on a fileserver, but it can be done.

    Interesting, I'm not asking this to try and be smart so take it at face value - what do you mean by "That iDataAgent isn't really intended to backup billions of small files on a fileserver, but it can be done."?

    Any Commvault input would be grand.

    Oh and don't get me wrong, the backups seem to work just fine, I'm just a bit OCD and when I spotted the pauses...

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 4:19 PM

    Actually I've just noticed something.

    Look at the screenshot below, this is the Media Agent.

    "Team 1" is the LAN NIC so it's accepting data coming in from my main file server.

    The two iSCSI NICs are just that, so they're in use doing a backup because the MA also has some files on it.

    Notice how the dips seem to correspond on both the data coming in over the LAN, and the data being read from the servers local iSCSI drives?

    This would suggest it's something on the MA that is the bottleneck wouldn't it?


    Attachment: Capture.PNG
  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 4:19 PM

    What I mean is that, and this is my opinion, it's meant and targeted at normal server backup. I've not seen great throughput traditionally when trying to place this on a fileserver. So I would prefer to have a NAS and NAS agent, NDMP, etc.

    It's obviously a pretty beefy server if it can handle the eight data readers, so it would probably speed up if you split up subclients.


    DR Donnie
    --------------
    There's an evil monkey in my CommCell!
  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 4:24 PM

    Paul Hutchings:

    This would suggest it's something on the MA that is the bottleneck wouldn't it?

    Correct, so either the MA isnt getting the data fast enough or we cannot write to the ISCSI target fast enough so it throttles.

    run robopy from the src to the media agent using a map drive to the ISCSI target to simulate this flow of transfer.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 4:29 PM

    This backup is direct to LTO4 HH tape in a TL2000 SAS library, the iSCSI drives are also a source (for the FS iDA on the MA), so if the iSCSI was slow I'd still like to think I'm able to saturate the gig ethernet consistently enough to get the tape throughput.

    Does seem like some robocopy's are in order tomorrow though.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 02-28-2011, 4:49 PM

    drdonniec_v:

    Have you checked the CPU and Memory utilization on the client? 8 Readers is A LOT for a default subclient.

    This is all dependant on how many disks make up the default subclient.  If there are 8 seperate disks associated with the default subclient, then 8 readers is ok (as you noted assuming we have CPU and memory to accomodate)  This allows 8 reads to be spread across 8 seperate disks, utilizing a single process per phase (ifind, clbackup,ect...).

    Now if you were to break up the backupset into 8 subclients 1 data reader each, you would have 1 read per disk (same as before), however now you have 8 ifinds, 8 clbackups, and 8 indexing processes running.  This changes the dynamics of whats happening on the machine and may or may not produce better results. 

    At this point if CPU and memory are abundant are 8 ifinds better than 1, even though we maximized the reads across the disks.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-01-2011, 9:19 AM

    OK so the results of a robocopy of a folder with 18gb or so in it.

    You'll see towards the ends the throughput tails off a little, that's because I also tried to copy a folder which has more, smaller files in it.

    I think the key thing is that the throughput is pretty consistent which to me doesn't suggest network or any other issue affecting throughput?


    Attachment: Capture.PNG
  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-01-2011, 9:45 AM

    Looks good, and agreed not the network in this case.

    In 8.0 (ma) and above tapetoolGUI.exe there is an option for disk read (just found this).  You can create a UNC path to the data (from the client) to the media agent and specify the tape target so this would simulate the data flow through backup to tape.  Maybe adding tape back into the equation would display similar results.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-01-2011, 9:49 AM

    Ah that could be useful to try, not sure how tapetoolgui deals with things like readers and multiplexing to get throughput though?  I always thought it was a "dumb garbage blocks to simulate chunk/block size" tool?

    I think the key point here is that we're seeing this total drops in activity when the only Commvault job running is backup of the LAN iDA (so only using "Team1") directly to tape.

    So at that point we're not reading/writing anything to/from the iSCSI drives on the MA either from Commvault or from within Windows (other than "background noise" activity).

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-02-2011, 8:43 AM

    I'd be interested if anyone from Commvault can add anything to the suggestions already made?

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-02-2011, 12:30 PM

    Paul,

    The steps that Vincenzo provided are solid and tape tool is a great place to start. You can see those types of throughput dips when writing to tape and the MA not being able to keep up with the drive.

    You can also try increasing readers/multiplexing to increase the amount of data being tossed at that drive.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-02-2011, 12:42 PM

    SConnington:

    Paul,

    The steps that Vincenzo provided are solid and tape tool is a great place to start. You can see those types of throughput dips when writing to tape and the MA not being able to keep up with the drive.

    You can also try increasing readers/multiplexing to increase the amount of data being tossed at that drive.

     

    Thanks Stephen, and please don't misunderstand me, Vincenzo has been most helpful on nearly all of my threads, I was just angling if anyone from Commvault might have a "Ah, seen that before, do this!" moment when reading the thread.

    Soon as I'm able to toss some more tapes in the library (it's not physically here) I'll try the tape tool steps.

    The library is a Dell TL2000 and the drive in the library is an LTO4 HH unit, not sure of the exact model and not sure if it actually helps as I think tape tool is the key here.  I do know when I've ran it before I've seen excellent throughput, though I've never tried with the "read from UNC" option.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-03-2011, 6:40 AM

    OK so I just did some stuff with tapetool.  Not sure how to compensate for the lack of readers when trying to pull data from a UNC with tape tool but see attached screenshot which doesn't exhibit any of the lengthy pauses I've been experiencing.

    I also did a normal write to tape, results below with and without compression:

    The specified directory path 'test' could not be found

    Using SCSI calls to test the tape.

    Start Time = Thu Mar 03 11:23:51 2011

    Opening the tape device '\\.\\\.\Tape0'...Done.

    Data Size = 20000 MB

    Block size = 256 KB

    File mark = 1

    H/W compression = 1

    Loading media...Done

    Getting the tape parameters...Done.

    Setting the tape parameters...Done

    Setting the hardware compression...Done.

    Seeking to file mark 1...

    Seek Start Time = Thu Mar 03 11:25:35 2011

    Seek End Time = Thu Mar 03 11:25:35 2011

    Writing to tape. Please wait...

    Current Total number of file Marks = 2

    Writing the last file mark...Done.

    There are a total of 2 file marks on the tape.

    Wrote 20000 MB of data on the tape

    Total time  taken to write 20000 MB of data is 118 Seconds

    Throughput of data write = 169.491531 MB/s   OR    595.868652 GB/hr

    End Time = Thu Mar 03 11:27:33 2011

     

    The specified directory path 'test' could not be found

    Using Windows API calls to test the tape.

    Start Time = Thu Mar 03 11:39:23 2011

    Opening the tape device '\\.\\\.\Tape0'...Done.

    Data Size = 20000 MB

    Block size = 256 KB

    File mark = 0

    H/W compression = 0

    Loading media...Done

    Getting the tape parameters...Done.

    Setting the tape parameters...Done

    Setting the hardware compression...Done.

    Seeking to file mark 0...

    Seek Start Time = Thu Mar 03 11:39:31 2011

    Seek End Time = Thu Mar 03 11:39:31 2011

    Writing to tape. Please wait...

    Current Total number of file Marks = 1

    Writing the last file mark...Done.

    There are a total of 1 file marks on the tape.

    Wrote 20000 MB of data on the tape

    Total time  taken to write 20000 MB of data is 199 Seconds

    Throughput of data write = 100.502510 MB/s   OR    353.329132 GB/hr

    End Time = Thu Mar 03 11:42:50 2011


    Attachment: Capture.PNG
  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-03-2011, 11:24 AM

    Nice numbers... but now we're back to square one, with whats happening during the backup.  I think at this point only the logs are going to show whats happening.  If this happens in clbackup supply that log (you can use "ScrubLogFiles" from the resource pack to remove any IPs and names which are sensitive) with the corresponding task manager output which has some timeframe linked to it so we can match up the dips to the events in the logs. 

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-03-2011, 1:18 PM

    Vincenzo_Basolino:

    Nice numbers... but now we're back to square one, with whats happening during the backup.  I think at this point only the logs are going to show whats happening.  If this happens in clbackup supply that log (you can use "ScrubLogFiles" from the resource pack to remove any IPs and names which are sensitive) with the corresponding task manager output which has some timeframe linked to it so we can match up the dips to the events in the logs. 

    From the logs I supplied support with, they've pointed to some potential errors in the clbackup logs:

    Version:1.0 StartHTML:0000000167 EndHTML:0000001556 StartFragment:0000000454 EndFragment:0000001540

    2684 bac 03/01 20:35:03 ##### [ NULL] Error Decrementing Allocator semaphore rc =183 FreeBuffersinPool 0

    2684 a90 03/01 20:35:03 ##### [ NULL] WTMOUT PCM timeout = 5000

    2684 a90 03/01 20:35:03 ##### [ NULL] Error Decrementing Allocator semaphore rc =183 FreeBuffersinPool 0

    2684 588 03/01 20:35:03 ##### [ NULL] WTMOUT PCM timeout = 5000

    2684 588 03/01 20:35:03 ##### [ NULL] Error Decrementing Allocator semaphore rc =183 FreeBuffersinPool 0

    2684 dd0 03/01 20:35:03 ##### [ NULL] WTMOUT PCM timeout = 5000

    2684 dd0 03/01 20:35:03 ##### [ NULL] Error Decrementing Allocator semaphore rc =183 FreeBuffersinPool 0

    2684 8ec 03/01 20:35:03 ##### [ NULL] WTMOUT PCM timeout = 5000

    2684 8ec 03/01 20:35:03 ##### [ NULL] Error Decrementing Allocator semaphore rc =183 FreeBuffersinPool

    and

    Version:1.0 StartHTML:0000000167 EndHTML:0000008959 StartFragment:0000000454 EndFragment:0000008943

    1376 1540 03/01 20:49:31 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 1540 03/01 20:49:31 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 2068 len = 0.. BAILING

     

    1376 1540 03/01 20:49:31 ##### [CVD ] STOP_PL Control Message Received 5 (DSBackup exiting gracefully)

    1376 1540 03/01 20:49:31 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 1540 03/01 20:49:31 54145 [NET_READER ] exiting NetReaderThread: tid = 5440

     

    1376 15b8 03/01 20:49:32 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 15b8 03/01 20:49:32 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 3280 len = 0.. BAILING

     

    1376 15b8 03/01 20:49:32 ##### [CVD ] STOP_PL Control Message Received 3 (DSBackup exiting gracefully)

    1376 15b8 03/01 20:49:32 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 15b8 03/01 20:49:32 54145 [NET_READER ] exiting NetReaderThread: tid = 5560

     

    1376 d9c 03/01 20:49:34 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 d9c 03/01 20:49:34 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 2612 len = 0.. BAILING

     

    1376 d9c 03/01 20:49:34 ##### [CVD ] STOP_PL Control Message Received 4 (DSBackup exiting gracefully)

    1376 d9c 03/01 20:49:34 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 d9c 03/01 20:49:34 54145 [NET_READER ] exiting NetReaderThread: tid = 3484

     

    1376 12e4 03/01 20:49:34 ##### [CVD ] Thread 3484(handle 938) at index 0 has exited

    1376 12e4 03/01 20:49:34 ##### [CVD ] Waiting 1780000 ms for remaining 1 PipeLine threads to Exit

    1376 cac 03/01 20:49:34 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 cac 03/01 20:49:34 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 2060 len = 0.. BAILING

     

    1376 cac 03/01 20:49:34 ##### [CVD ] STOP_PL Control Message Received 6 (DSBackup exiting gracefully)

    1376 cac 03/01 20:49:34 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 cac 03/01 20:49:34 54145 [NET_READER ] exiting NetReaderThread: tid = 3244

     

    1376 f4c 03/01 20:53:28 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 f4c 03/01 20:53:28 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 3068 len = 0.. BAILING

     

    1376 f4c 03/01 20:53:28 ##### [CVD ] STOP_PL Control Message Received 6 (Pipeline Thread self-terminated)

    1376 f4c 03/01 20:53:28 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 f4c 03/01 20:53:28 54145 [NET_READER ] exiting NetReaderThread: tid = 3916

     

    1376 145c 03/01 20:53:28 ##### [CVD ] Thread 3916(handle 768) at index 3 has exited

    1376 145c 03/01 20:53:28 ##### [CVD ] Waiting 1784000 ms for remaining 3 PipeLine threads to Exit

    1376 176c 03/01 20:53:28 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 176c 03/01 20:53:28 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 2956 len = 0.. BAILING

     

    1376 176c 03/01 20:53:28 ##### [CVD ] STOP_PL Control Message Received 3 (Pipeline Thread self-terminated)

    1376 176c 03/01 20:53:28 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 176c 03/01 20:53:28 54145 [NET_READER ] exiting NetReaderThread: tid = 5996

     

    1376 b40 03/01 20:53:31 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 b40 03/01 20:53:31 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 1660 len = 0.. BAILING

     

    1376 b40 03/01 20:53:31 ##### [CVD ] STOP_PL Control Message Received 4 (Pipeline Thread self-terminated)

    1376 b40 03/01 20:53:31 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 b40 03/01 20:53:31 54145 [NET_READER ] exiting NetReaderThread: tid = 2880

     

    1376 145c 03/01 20:53:31 ##### [CVD ] Thread 2880(handle 4a0) at index 0 has exited

    1376 145c 03/01 20:53:31 ##### [CVD ] Waiting 1781000 ms for remaining 1 PipeLine threads to Exit

    1376 830 03/01 20:53:31 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 830 03/01 20:53:31 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 3052 len = 0.. BAILING

     

    1376 830 03/01 20:53:31 ##### [CVD ] STOP_PL Control Message Received 5 (Pipeline Thread self-terminated)

    1376 830 03/01 20:53:31 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 830 03/01 20:53:31 54145 [NET_READER ] exiting NetReaderThread: tid = 2096

     

    1376 eb8 03/01 20:58:28 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 eb8 03/01 20:58:28 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 2460 len = 0.. BAILING

     

    1376 eb8 03/01 20:58:28 ##### [CVD ] STOP_PL Control Message Received 4 (DSBackup exiting gracefully)

    1376 eb8 03/01 20:58:28 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 eb8 03/01 20:58:28 54145 [NET_READER ] exiting NetReaderThread: tid = 3768

     

    1376 ad4 03/01 20:58:28 ##### [CVD ] Thread 3768(handle 414) at index 1 has exited

    1376 ad4 03/01 20:58:28 ##### [CVD ] Waiting 1797000 ms for remaining 3 PipeLine threads to Exit

    1376 1688 03/01 20:58:28 ##### [CVD ] Recv failed to read 4 bytes, Error <Socket shutdown detected>

    1376 1688 03/01 20:58:28 54145 [NET_READER ] Network Reader slRecvMsg() failed.socket = 1952 len = 0.. BAILING

     

    1376 1688 03/01 20:58:28 ##### [CVD ] STOP_PL Control Message Received 5 (DSBackup exiting gracefully)

    1376 1688 03/01 20:58:28 54145 [NET_READER ] Network Reader Received STOP_PL After network error: bail out

    1376 1688 03/01 20:58:28 54145 [NET_READER  

    These seem to be present in quite a few of the client logs though.

    Odd as our network is pretty good in so much as clients go direct into a Procurve 5406zl over CAT6, which has a 10gbps fibre link to another 5406zl, which has the MA hooked into it and we don't encounter any day to day issues which point at "iffy" networking.

    Now I have some idea of what and where to look I'll try and check in during a backup and see if anything shows in the logs but any input welcome based on the above.

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-03-2011, 6:07 PM

    I get these errors also:

    "

    2684 dd0 03/01 20:35:03 ##### [ NULL] WTMOUT PCM timeout = 5000

    2684 dd0 03/01 20:35:03 ##### [ NULL] Error Decrementing Allocator semaphore rc =183 FreeBuffersinPool 0

    2684 8ec 03/01 20:35:03 ##### [ NULL] WTMOUT PCM timeout = 5000

    2684 8ec 03/01 20:35:03 ##### [ NULL] Error Decrementing Allocator semaphore rc =183 FreeBuffersinPool

    ""

    But when I test transferring anything else or even different iDA backup job from same server I have no problems.  My problem is the opposite, my filesystem iDA flies and my SQL iDA throws errors like above.  I have a ticket open with support.

    -Chad

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-03-2011, 7:32 PM

    Chad,

    What is the type of NIC you are using?

    JW

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-04-2011, 3:11 AM

    JW I know you aimed your question at Chad but FWIW we're using various vmware adaptors on most of our clients, and our MA has a pair of Broadcom 5708's in a SLB team.

    This is a capture from last nights incremental backup.  Ignore the iSCSI NICs the only Commvault trafficic that coming in over the team into the MA and to the LTO4, so nothing wrong there surely?


    Attachment: Capture.PNG
  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-07-2011, 10:20 AM

    Paul, didn't want to hi-jack your post at all, just chiming in that I'm seeing similar errors. 

    And I'm also using VMWare NIC adapters on the affected server.

    -Chad

  • Re: Windows FS iDA - Lengthy pauses when backing up?
    Posted: 03-07-2011, 4:05 PM

    Just got off the phone with CV support a little bit ago, we checked the "Optimize for concurrent LAN backups" in Storage Resources -> MediaAgents -> right-click MA -> and Control tab.  So far this has sped up the backup on our problem SQL iDA agent drastically and gotten rid of the wait messages in the log.

    -Chad

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy