This is the second part of a three-part blog series on Veeam and Pure Storage FlashBlade. In the previous blog post, we configured a Network File System (NFS) share on a Pure Storage FlashBlade as a Veeam backup repository. In this blog, we’ll focus on configuring SafeMode snapshots to harden the backup files that are residing on the FlashBlade.
Ransomware attacks continue to rise with constantly evolving sophistication and complexity. A key part of a ransomware resilience strategy is backing up data on a regular basis and implementing a strong line of defence against threats targeting the backup data. Adopting industry standards for data protection such as a 3-2-1 rule, offline backups and immutable backup storage are effective techniques to protect backup data sets against malicious attacks. Now let’s discuss how to make your FlashBlade system an immutable backup storage target with SafeMode snapshots.
A storage snapshot is a point-in-time, image-level view of data that is impervious to ransomware. This immutability makes a storage snapshot an ideal layer of defence against ransomware. The problem with storage snapshots is that they can still be removed by rogue admins or attackers if they gain access to the storage array management. In the case of a Pure Storage system, the deleted snapshots are temporarily stored in a ‘destroyed state’ that is similar to a recycle bin. If these snapshots are not recovered in a timely manner, they will be auto-eradicated and can even be manually destroyed prior to the auto-eradication.
The SafeMode snapshots on the other hand, cannot be deleted, modified, or encrypted- either accidentally or intentionally. This prevents the manual and complete eradication (permanent deletion) of data backups that are stored within the FlashBlade. Due to their immutability, the SafeMode snapshots serve as an additional mitigation mechanism against ransomware attacks or rogue administrators.
The SafeMode snapshots are created and managed automatically by Purity, the FlashBlade operating software, independent of administrator control. You can schedule SafeMode snapshot generation of any data residing on the FlashBlade file systems. Primary and backup data can be directly recovered from these snapshots. If original copies are corrupted or destroyed, data can still be recovered from the SafeMode snapshots.
Once the SafeMode snapshots are activated, snapshots for all the file systems get created that incur additional capacity utilisation on FlashBlade. Therefore, prior to SafeMode activation, it is recommended to consult your local Pure Storage Systems Engineer to validate the FlashBlade capacity sizing required to support SafeMode snapshots.
Pure Storage developed and practicing a highly secured, well defined process on how FlashBlade customers collaborate with Pure Storage Support to enable SafeMode or to make changes to an existing SafeMode policy. Only authorised users from the customer organization can work directly with Pure Technical Support to configure the feature, modify policy, or manually eradicate snapshots. We decided not to discuss the full details here as we do not want potential ransomware perpetrators reading this post to be benefited from the information. You are always welcomed to engage your Pure Storage Sales Team if you’d like a more detailed walkthrough of how SafeMode works, how it is enabled, and protections against it being disabled.
Upon agreement with the customer, Pure Support performs a pre-activation health check that involves checking the FlashBlades growth trend to ensure there is sufficient capacity to enable SafeMode.
Once Pure Support has enabled SafeMode, validation is required.
The screenshot above displays a list of snapshots of the file system . There are 2 distinct snapshot groups presented here:
When attempting to eradicate a SafeMode snapshot, the file system will be moved to a destroyed state.
The destroyed snapshot will be placed in a destroyed snapshot folder as highlighted below. In our test, eradication timeout for the destroyed snapshots was changed from the default value of 14 days to 7 days.
When attempting to eradicate (permanently deleting) a SafeMode snapshot, a warning message will be displayed. as shown below.
It is evident that under SafeMode, the administrators cannot manually remove a filesystem or snapshots from FlashBlade.
A sophisticated ransomware attack does not just encrypt production data. Perpetrators often target the backups to ensure organisations cannot recover. There are cases where the backup servers were compromised by ransomware attacks and became inaccessible. Other possible scenarios are when perpetrators and/or rogue admins gain access to the storage systems and attempt to delete the file systems and snapshots hosting backup data.
To demonstrate the effectiveness of SafeMode, we have assumed that access to both the Veeam console and FlashBlade interface has been gained. From the Veeam console, a ‘Delete from disk’ operation was performed on all backups that are stored on the FlashBlade NFS repository. This results in the deletion of the following:
The following screenshot depicts the status of after all of its content is removed from the corresponding Veeam backup repository. Note that the used space displays 0TB as the file system is storing nothing.
The NFS file system that is used for the Veeam repository was then destroyed from the FlashBlade console.
If other protection mechanisms are compromised, the FlashBlade storage becomes a solid final line of defence. As SafeMode snapshots are immutable and cannot be eradicated manually, the recovery process can be started, as below.
To recover a destroyed file system:
Review the file systems in the destroyed container and click Restore. Note that the destroyed file system moves back to the file system folder.
Based on our simulated attack, restoring the file system is just the first step to recovery, as both the Veeam backup data and the File System (NFS) were destroyed. The next step is to revert the File System to a previous snapshot before the data was deleted/compromised.
Reverting the file system to a previous SafeMode snapshot requires the intervention of Pure Support. The FlashBlade administrators are not allowed to perform a filesystem rollback using GUI or CLI. The company designee needs to contact Pure Support to complete the restoration process.
Note: Purity does not allow snapshot restore by an administrator with SafeMode enabled. An error message is shown whenever Administrators attempt to revert a snapshot.
Let’s consider a few scenarios.
Scenario 1: The backup records in the Veeam configuration database were not destroyed.
Scenario 2: The backup records in the Veeam configuration database were destroyed.
Scenario 3: Veeam Backup & Replication server was destroyed
Adding a FlashBlade NFS Share to Veeam as a Backup Repository is covered in part one.
The simulated attack described above involved deleting the backup records from the Veeam configuration database, but not from the NFS backup repository. Therefore, the following steps are performed.
After the rescan is complete, the restore points will reappear under Backups > Disk (Imported)
Veeam can now perform an Instant Recovery to immediately restore VMs by running them directly from the backup files stored on the reverted NFS backup repository.
The following screenshot shows the restored VMs that are available in the vSphere Client.
Following are a few caveats surrounding SafeMode that customers should be aware of:
It is important to consider the security of the device as well. If the storage device can be easily taken over, then the software layer providing the storage immutability may not be of much help. Fortunately, in the case of Pure Store SafeMode, even if the management GUI was accessed, the rogue administrator cannot factory reset/wipe the storage and manipulate the storage snapshots.
To further increase security, Multi-Factor Authentication (MFA) will be introduced in an upcoming Purity release (v6).
While not required, it is recommended to configure the Veeam Backup & Replication (VBR) Configuration Backup in the same NFS Backup Repository. This simplifies the VBR server recovery when the server is corrupted or unavailable. A new VBR can be deployed and the Veeam configuration backup can be recovered onto the server.
The Pure Storage FlashBlade is an on-premises backup storage that provides immutable storage for data such as Veeam backups. SafeMode configuration provides an additional layer of protection against ransomware/cyber-attack.
In the next blog post, we will discuss how Object Storage presented from the FlashBlade will be used as a backup repository target for Veeam Backup for Office 365.
Disclaimer: For advice regarding retention periods and ransomware security, please speak to your local Pure Storage SE.
Standard Snapshot restoration of Veeam backups (GUI)
To restore a non-SafeMode snapshot:
Following are commands that are used to restore the file system with snapshot via Purity FB CLI.
The FlashBlade administrators can use either CLI or GUI to perform file system restoration as if the array is not operating under SafeMode. Customers will need to contact Pure Support to perform the following steps in FlashBlade under SafeMode.
After a successful file system restore, one can observe that the used capacity on file system is increased to 114.51GB. This indicates that the previously removed objects are restored to the file system.
The next step is to proceed to the Veeam console to complete the recovery of affected virtual machines.
