Outside of actual VSS errors, it's generally one of two things.
1) When you run the command vssadmin list writers you will notice there is no "System Writer" listed. The KB below includes a MSFT "fix" that will correct the issue as long as the writer remains available. I have heard from other customers having the same issue that the writer may disappear again after a reboot. Once the System Writer is available again the system state should complete.
Answer Title: The System State Backup Job on Windows 2008 Servers Fail to Backup the System Protected Files.
Answer Link: https://ma.commvault.com/KB/Details/2306
2) Check the space being allocated for shadow copy in the OS? System State can be quite large on Windows 2008 clients and for this reason the System Protected Files and/or Registry Component(s) can fail during the System State due to lack of space. To see if this is the issue you can remove the limitation in the OS by setting VSS to no limit and run a test backup.
If the failure is intermittent, space is more likely the cause as the shadow copy grows or reduces in size depending on the amount of data being backed up.