The failback operation (restoring a VM from replica) allows you to restore your original VM by using the replica VM. This operation can also be used if you decide to stop the replica VM after the Failover operation is started and save the changes to the original or new location. 

There are two options to recover - to an original or to a new location. If the original VM is missing or a new location is chosen then the new virtual machine will be created using the options specified during setting up the Failback VM from Replica task. 

If the original VM is available and is chosen to be recovered to then this virtual machine will be stopped to perform the recovery from replica VM. Before starting the recovery the quiescied snapshot of the target VM using VMware API is created. This technical snapshot is required to be able to rollback in case the failback operation fails. If the new virtual machine is created then the snapshot is not required.

In case replica VM is running, the Failback VM from Replica  task is restoring the original VM without stopping the replica VM. Only when the failback  is finished, the replica VM is stopped. Finally, the Failback VM from Replica task recovers the changes from replica VM to the original (new) VM that have been done during the time of the  failback operation. It allows to minimize the downtime and to restore the VM state as much as close to its replica state.

Follow step-by-step instructions below to troubleshoot the issue.

Failures on initial deduplication configuration: these types of failures and errors happen when setting up a new backup to a deduplicated vault or very shortly after.

• Run the same type of backup to a managed vault without Deduplication. If this backup also fails, the issue is not related to deduplication;
• Make sure that all backup Agents showing the failure have a proper deduplication license assigned;
• Make sure that the vault and database locations are fully accessible by the Storage Node service account.
  To check it, perform the next steps:

  
  

  1. Log on to Windows under Administrator account;
  2. Right click the folders which are the vault and database locations and select “Properties”;
  3. Open “Security” tab and highlight the ASN User (if there is no such user in the list, see below );
  4. Look at the permissions specified:

  If the user has no full control, do the following:

  
  

  1. On “Security” tab of the folder properties, click “Edit” button;
  2. Highlight the ASN User (if there is no such user in the list, see below );
  3. Tick the “Full control” check box and then click “OK” -> “OK” buttons.

  If there is no such user in the list at all, perform the following steps:

  
  

  1. On “Security” tab of the folder properties, click “Edit” button, then click the “Add…” button;
  2. Enter the name of ASN user or find the ASN User by clicking “Advanced” -> “Find now” buttons:
     
  3. Accept the changes and specify permissions for the ASN user by selecting “Full control” check box and then click “OK” button:
     

• Once all of the steps above are checked and the failure is still present, go to Collect information part.

Deduplication failures after a large amount of data has been deduplicated after a number of backups have been successfully created and indexed.

• Make sure that the ASN is not showing any crashes and that the vault is still fully available;
• Make sure that all Agents showing failures can access the vault;
• Make sure that indexing task is finishing correctly before the next backup task is started. If this is not the case, see this article;
• If the issue is not tied to performance, we are probably facing a problem with the existing deduplication data on this vault. Go to Collect information part below.

Tape devices use magnetic tape media to store information and can be divided into 3 type:

- Standalone tape drives. Here one tape cartridge can be inserted into the device and used. To write or read data from other tape cartridge, the previous tape has to be ejected and taken out from the drive first.

- Tape library is a device that apart from a tape drive (one or more) has also additional slots to store tape cartridges. Via the device or third-party software interface tapes can be moved between slots and drives. A slot that allows inserting new tapes and ejecting tapes is called mail slot. Tapes in tape libraries usually have unique barcodes which can be optically read by the reader during inventory. The barcode duplicates the in-formation stored in tags placed in the beginning of the tape data.

- Virtual tape library (VTL) is a storage virtualization technology, that allows the representation of a storage device as a tape library or a tape drive. Usually is it used to represent a hard disk storage device as a tape library, in a special case an RDX device.

When speaking of tape types, LTO is a major standard and the version of the LTO tape cartridge defines its characteristics including capacity. Modern generation LTO-3 and LTO-4 come with HDD competitive read/write speed of 80 and 120 MB/s and capacity of 400 GB and 800 GB respectively.
Tapes are storage media that can only be read sequentially as opposed to HDDs, and per ABR tape vault format we can only add something to the free end of the tape, but we cannot re-write a part of the tape and place new information keeping some piece of previous information. Tape will always be overwritten completely.

Follow step-by-step instructions below to troubleshoot the issue. Complete prerequisites step before proceeding to error troubleshooting step.

The technology to back up virtual machines is described in the Acronis Backup & Recovery 11.5 and Acronis vmProtect: Backup Algorithm on ESX Environment article.

The File Recovery option uses the following algorithm to restore data:

1. Software extracts selected files/folders from the slice that is selected to recover from;
2. Then a .zip archive is created with the content selected for restore.
3. Download of the .zip archive via HTTP starts immediately. Creating the .zip archive and downloading it to the client machine runs simultaneously. 

Follow step-by-step instructions below to troubleshoot the issue. Complete prerequisites step before proceeding to error troubleshooting step.

The detailed information about the backup technology can be found in the Acronis Backup & Recovery 11.5 and Acronis vmProtect: Backup Algorithm on ESX environment article.

During the recovery operation the VM is recovered to the state when the snapshot was created. The different complex application including Active Directory, Microsoft SQL Servers, Microsoft Exchange servers depends on how the snapshot was created and if there were any errors creating the snapshot inside the virtual machine then the data that was backed up can be not consistent that can lead to the problem with application after recovering. 

Follow step-by-step instructions below to troubleshoot the issue. Complete prerequisites step before proceeding to error troubleshooting step.

Acronis vmProtect has two ways of recovering virtual machines: to the original VM or to a new one. If recovery is performed to the original location, Acronis vmProtect tries to locate the source VM using its GUID and if the VM is missing a new VM will be created.

Recovering to the original VM

In case the original VM exists the software uses the following algorithm:

1. When the recovery operation is started Acronis vmProtect powers off and creates a quiescied snapshot of the target VM using VMware API. This technical snapshot is required to be able to rollback in case the recovery operation fails.
2. After the snapshot is created Acronis vmProtect mounts virtual disks to the Virtual Appliance and recovers data to the existing disks. If Windows Agent is used or the Hot-Add functionality is not available, recovery is performed via network directly to the target VM.
3. After data is recovered the software disconnects virtual disks from the Virtual Appliance and sends command to remove the technical snapshot created in the first step. 
4. And only after all these steps are performed the target VM is started.

If restore fails the software will rollback using to the snapshot created in the first step.

Recovering to a new VM

If the original VM is missing or the option to recover to a new VM is chosen the following algorithm is used:

1.  Acronis vmProtect sends the command to create a virtual machine on the VMware ESXi host where the recovery is performed
2.  After the VM is created Acronis vmProtect mounts the virtual disks to the VA and recovers the data to existing disks. If WA is used or the Hot-Add functionality is not available then recovery is performed via the network directly to target VM.
3.  After recovery is finished, the disks are disconnected and the VM is started. 

The difference between these algorithms is that the snapshot creation is not required in case if new VM is created. Besides that if recovery is performed to existing VM then the vmFlashBack feature can be used.

Follow step-by-step instructions below to troubleshoot the issue. 

Product may hang under specific circumstances. Each hang should be investigaed separately. In order for our development team to troubleshoot the problem with lockups they need process dumps and other logs listed in this article.

There are 2 types of hanging:

• Actual hanging. That is a state similar to a crash. That means that software halts and is doing nothing inside.
• Infinite loop. This means that on the outside the software looks hanging but inside it is doing some actions over and over again.

Usually when a software hangs its process uses either 100% of CPU (especially when it is infinite loop) or does not use CPU and does not change RAM usage at all (Actual hanging). Still that is difficult to make sure if the software hangs or not. The above means that usually 1 dump is not enough because if that is Infinite Loop – the developers will see that software is working and there is no problem inside. So in order to troubleshoot application lockups you need about 5 dumps taken one after another. A Process Monitor Log will also be useful especially if it actually turns out to be an Infinite Loop.

Lists of Acronis Software processes:

Acronis True Image 2014: Processes and Services

Acronis Backup & Recovery 11.5: Processes and Services

Acronis vmProtect 9: Processes and Services

Follow these steps to troubleshoot application lockups depending on your operating system:

The technology to back up the virtual machines is described in the Acronis Backup & Recovery 11.5 and Acronis vmProtect: Backup Algorithm on ESX Environment article.

Acronis vmProtect creates backup using Hot-Plug VMware feature which means that virtual disks of the virtual machine to back up will be attached to the virtual appliance. This option will work only when the virtual appliance is deployed to the same ESXi host where the datastore with the virtual disks of the virtual machine are located. In case the Hot-add functionality cannot be used (the virtual appliance is running on another VMware ESXi host or Acronis vmProtect Windows Agent is used) data from virtual disks will be backed up via the network.

Aronis vmProtect supports the following backup locations:

• Acronis Online Storage
• Local disks (USB, internal)
• Network shares
• FTP/SFTP shares

Acronis software performs read/write operations to the data stored in backups using Windows or Linux drivers depends on the environment where the software is running.

Acronis vmProtect: Slow Backup/Replication Speed in Virtual Appliance-Based Installation

Acronis vmProtect: Slow Incremental Backup Caused by CBT

Follow step-by-step instructions below to troubleshoot the issue.

The Replication feature allows to create clones of critical machines and run them in case of original VMs failure. The software uses the following algorithm to replicate virtual machines:

1. Acronis vmProtect sends a request via VMware API to create a quiesced snapshot of the source virtual machine;
2. When the snapshot is created virtual disks of this snapshot will be mounted to the virtual appliance if Hot-Plug feature is available. In case the Hot-add functionality cannot be used (the virtual appliance is running on another VMware ESXi host or Acronis vmProtect Windows Agent is used) data from the source virtual disks will read via network (NBD mode).
3. On the destination host a replica VM is created using VMware API. If a replica VM exists, Acronis vmProtect reverts it to a special technical snapshot which is always created upon each replication task completion. This snapshot is used to revert replica to a known good state in case any replication-related activity fails.
4. The data is transferred from the original VM to the replica VM. In case the replica VM exists when the replication task is stated the changed block tracking (CBT) VMware feature can be used to get data from the original VM and to minimize traffic.
5. After all the data is transferred the new technical snapshot is created on the replica. 

Acronis vmProtect: Slow Backup/Replication Speed in Virtual Appliance-Based Installation

Acronis vmProtect: Slow Incremental Backup Caused by CBT

Follow step-by-step instructions below to troubleshoot the issue. 

Validating backups is an operation that checks the possibility of data recovery from a backup. The algorithm is the following:

1. When the backup is created Acronis vmProtect reads blocks of the source virtual machine's drives.
2. Before backing up a block of data read from the source disk the MD5 checksum of this data is captured and saved in the archive;
3. When the validation operation starts Acronis vmProtect reads data blocks in the backup, creates MD5 checksums from these blocks and checks if these checksums are equal to the MD5 checksums saved in the 2nd step (during backup).
4. If a MD5 checksum differs from the original one, it means the backup is corrupted.

Validation task checks the following items:

• Recovery points - the software will check the integrity of the selected recovery points only. When validate backup option is selected, the software validates the freshly created recovery point only.
• Archives - the software will check the integritycheck the integrity of all recovery points in the selected archive.
• Backup locations - the software will check the integrity of all the archives and recovery points inside this location.

Follow these step-by-step instructions below to troubleshoot the issue.

Error code: 44
Module: 39
LineInfo: d2f435cbb76149ad
Fields: $module : storage_server_vsa64_37976
Message: Failed to create vault 'test'.
--------------------
Error code: 133
Module: 39
LineInfo: d2f435cbb761496d
Fields: $module : storage_server_vsa64_37976
Message: Failed to create data file 'FEE4161E-F504-4328-B95F-754C0422A628_l.fdb'.
--------------------
Error code: 133
Module: 39
LineInfo: 5a99bb5a00cb1ad4
Fields: $module : storage_server_vsa64_37976
Message: Failed to create data file 'C:\ProgramData\Acronis\BackupAndRecovery\ASN\VaultMetadataDatabases\FEE4161E-F504-4328-B95F-754C0422A628_l.fdb'.
--------------------
Error code: 1
Module: 42
LineInfo: 884f438198c8e35d
Fields: $module : storage_server_vsa64_37976
Message: operating system directive CreateFileMapping failed; There is not enough space on the disk. ; (1, 335544373); (2, 'CreateFileMapping'); (17, 112)

The replication feature provides you with the ability to clone your critical VMs and to be able to start your critical service fast in case of failure. To run the New Replication Task, click Actions -> Replication.

1. In the first step of the New Replication Task wizard you should select the virtual machine(s) you want to replicate.

2. In the second step of the New Replication Task wizard, you should define the schedule of replicating your virtual machines. There are two options available – creating a single time replication task (“Do not schedule, run on demand”) and weekly scheduling.

3. In the third step of the New Replication Task wizard you should define the location and Datastore for VM replicas. Here, you have to first define the ESX(i) Host by selecting one from the drop-down list. Then, select the Resource Pool on the destination ESX(i) Host and destination Datastore.

4. In the fourth step of the New Replication Task wizard you should define your replication task name. For replication task preferences click More options...
   

   The following options are available: 

   - E-mail notifications;

   - SNMP notifications;

   - CBT replication;

   - Provisioning mode.

Follow step-by-step instructions below to troubleshoot the issue.

 1. Make sure you do not observe any Active Directory malfuctions within the original Virtual Machine;

 2. Check Windows Event Logs on the original VM for related errors;

 3. Collect the following information and contact Acronis Customer Central:

• vmProtect logs: go to VMProtect Web Console>> View>> Show Logs>> Save All To File
• Collect information about source and target ESXi hosts (version & build);
• Detailed description of the problem itself - what exactly does not work within Active Directory (with screenshots if possible);
• Screenshot of the error message that appears (if any);
• System report from the original VM & from the Replica VM: See AcronisInfo Utility

The replication feature provides you with the ability to clone your critical VMs and to be able to start your critical service fast in case of failure. To run the New Replication Task, click Actions -> Replication.

1. In the first step of the New Replication Task wizard you should select the virtual machine(s) you want to replicate.

2. In the second step of the New Replication Task wizard, you should define the schedule of replicating your virtual machines. There are two options available – creating a single time replication task (“Do not schedule, run on demand”) and weekly scheduling.

3. In the third step of the New Replication Task wizard you should define the location and Datastore for VM replicas. Here, you have to first define the ESX(i) Host by selecting one from the drop-down list. Then, select the Resource Pool on the destination ESX(i) Host and destination Datastore.

4. In the fourth step of the New Replication Task wizard you should define your replication task name. For replication task preferences click More options...
   

   The following options are available: 

   - E-mail notifications;

   - SNMP notifications;

   - CBT replication;

   - Provisioning mode.

Follow step-by-step instructions below to troubleshoot the issue.

1. Check that the application works fine within the original virtual machine;

2. Check partitions inside the replica VM where the application resides for errors using chkdsk utility: go to command prompt (Start -> run -> cmd) and enter the following command: "chkdsk VOLUME /f" where VOLUME is the volume letter you need to check.  Please note, that checking the C:\ drive may require you to restart the machine;

3. Check that the environment where the application should be used after the VM replication is suitable for that;

4. If all steps above have been executed and issue still persists, collect the following information and contact Acronis Customer Central:

• VmProtect logs: go to VMProtect Web Console>> View>> Show Logs>> Save All To File
• Information about source and target ESXi hosts (version & build);
• Detailed description of the problem itself - what exactly does not work within the application (with screenshots if possible);
• Screenshot of the error message that appears (if any) during the attempt to run the application;
• Acronis System Reports from original VM and from the Replica VM that can be captured using AcronisInfo utility.