Issue

Backups regularly fail with some type of snapshot error

Cause

Snapshot issues are probably the most common cause of backup failures, some common causes of snapshot issues are:

1. Issues with the VSS (volume snapshot services) subsystem: this can be determined by checking the output of vssadmin list writers.
2. Overloaded disk/storage performance on the machine: windows events that log VSS timeouts are often due to an overloaded disk subsystem. Machines with a lack of free RAM can experience significant swapping, which results in overloaded disks.
3. An underlying disk/storage fault: failing storage hardware can often cascade problems into upper levels such as the VSS subsystem. Incorrectly configured VSS hardware providers can also cause problems with backup software that is VSS aware.

Resolution

1. Check the storage system for any faults: predictive failures/bad blocks/failed drives etc.
2. Check the storage system has current firmware and drivers
3. Check the storage system is not overloaded and has enough IOPS and bandwidth to support workloads
4. Check that the VSS writers are in a stable state, restarting those VSS services that are showing to be in a failed state

Issue

Large incremental backups are being seen

Cause

Large incrementals can be caused by numerous reasons, some common reasons are:

1. Backups of backups are being captured ie. SQL database dumps or ZIP backups to a location that is then captured by CyberSecure can cause a doubling-up of data
2. Maintenance tasks such as database reindexing or defrags: these cause sectors on the disk to move around, which some backup technologies will see as changes
3. A dirty shutdown can cause some technologies (such as ShadowProtect) to create a differential, which can be significantly larger than a standard incremental
4. An application is creating large logs or temporary files: we have seen this with some Trend Micro and Symantec software
5. Ransomware infection: ransomware deletes and encrypts files, which causes large changes
6. or simply alot of data was added, which may or may not be expected, we advise checking with your users, IT admins and/or audit logs to investigate further

Resolution

1. If the source of the large data changes is not immediately obvious, CyberSecure has developed numerous strategies to identify how to monitor changes on volumes to detect what programs are causing the changes
2. Processes such as SQL dumps can be redirected to separate storage or to volumes that are not included as part of the CyberSecure backup
3. Database maintenance tasks should be reviewed to ascertain if they are needed, or if moving those workloads to solid state/flash storage would negate the requirement of performing such frequent indexing.
4. If a system is frequently experiencing dirty shutdowns, the cause of these shutdowns should be investigated and resolved.
5. Applications creating large logs should be moved onto separate storage that is not being captured by the backups.

 Prerequisites

1. Setup Genie to dump its database (turn off compression and encryption) to a location, henceforth called GENIE_BACKUPS
2. Create a blank folder somewhere (preferably on the SAME volume as GENIE_BACKUPS), this will be called CYBER_BACKUPS and you will point the CS Advanced to backup this location
3. Download the genie-macos.sh script and save to a stable location (eg. /Users/<username>/.obm)

Process

1. Setup a backup set called “GenieDB”
2. Backup source = CYBER_BACKUPS
3. Add a pre-command with the following parameters:
   1. Name: prepareDB
   2. Command: genie-macos.sh
   3. Working dir: /Users/<username>/.obm
4. Modify the genie-macos.sh script variables (eg):
   1. GENIE_BACKUPS=”/Applications/Genie/Backups”
   2. CYBER_BACKUPS=”/Users/<username>/cs-genie-dest”
5. The Genie backup dump will need to be scheduled so a complete backup is done well before the CS agent attempts its backup.

We occasionally see systems with incorrect system time. The system time can either be out by a few minutes or sometimes several days or more.

Accurate timekeeping is a fundamental assumption in an IT system and adverse negative effects can happen if:

1. the system clock is wrong or;
2. (even worse) the clock is changed in big jumps backwards or forwards.

Inaccurate timekeeping generally can have the following negative impacts:

1. Applications that rely on system time can become corrupted (eg. the backup scheduler that sets when it’s meant to run from the current time. If the system clock is in the past and then jumps into the future, it will result in a stale backup as the system thinks it’s meant to run in the past and therefore never runs)
2. System updates or software versions may rely on timestamps to see when it last ran or installed updates, these can get out of sequence and have unknown consequences.
3. File access and modification times can become corrupted and have incorrect modification times. This can cascade into other applications that perform actions based on file modification times (eg. backup all files that have changed since the last backup, if those dates are in the future the files will not be backed up).
4. Data  loss or corruption may occur if a process that makes data pruning decisions based on time are affected. eg. Many (probably all) backup software that performs retention is solely based on the system clock time. If the time jumps into the future or the past, then backups could be pruned prematurely. This can have disastrous consequences if a backup chain has dependent files pruned, thus rendering the backup chain corrupted and unrestorable.