Beyond SnapDiff: Scalable backup for NetApp NAS with IBM Spectrum Protect

The recent announcement from NetApp to deprecate the SnapDiff v1 and v2 APIs, used by software vendors such as IBM to support incremental forever backups, presents another opportunity to showcase an alternative scalable means of backing up NAS with millions of files using IBM Spectrum Protect.

Over the last couple of decades, anyone who has experienced the need to backup NAS filers, from any vendor, will have found the need to overcome the limitations of the ubiquitous NDMP dump.

As a champion of IBM Spectrum Protect and its predecessor Tivoli Storage Manager (TSM), we consider incremental forever backup a default approach to backing up unstructured data, so the idea of having to perform periodic fulls with differentials rather grates.

Simply backing up file shares across a network via a mapped drive is an option, however, the ability to perform an incremental forever backup does rely on a backup client being able to learn what has changed from one job to the next, and spending hours doing a file inventory across millions of files just to discover that a few hundred GB needs to be backed up is not scalable.

IBM Spectrum Protect for backing up NetApp files

So, back in 2009 when IBM announced TSM 6.1 with support for the NetApp SnapDiff API, we then set about replacing those clunky Network Data Management Protocol (NDMP) jobs with incremental forever file-level backups, which took advantage of the NetApp snapshot to learn what had changed and what fraction of the data needed to be backed up, without having to do that tree-walk.

So for over a decade, we’ve been able to provide the benefits of IBM Spectrum Protect efficiently backing up large NetApp files to disk, cloud, and tape and storing the data in a format that can be recovered to any file platform (which is essential when backups are being retained for years beyond the lifespan of the original NAS device!).

The big news announced by NetApp in April means that we have until December 2023 at the latest to find alternative means of protecting NetApp filers where the backup software relies on the soon to be deprecated SnapDiff v1 and v2 APIs.

The NetApp ONTAP 9.8 code will be the final release to support those v1 and v2 SnapDiff APIs, and it is due to remain in full support until December 2023.

For any NetApp users who do intend to upgrade beyond ONTAP 9.8, or are due to implement new systems on later code, planning for backup without SnapDiff is essential.

How CSI is dealing with the NetApp announcement to deprecate the SnapDiff v1 and v2 APIs

Fortunately, CSI is already prepared for the change. Our wide range of clients using large-scale NAS devices from various vendors means that we have already had a need to provide scalable solutions to efficiently backup millions of files on an incremental-forever basis via IBM Spectrum Protect.

Whether those millions of files are on NetApp, Dell Isilon, VNX or simply very large Windows file servers, we have a proven solution, using dsmISI MAGS from General Storage.

MAGS is software that gives the standard IBM Spectrum Protect client a boost and allows the native incremental-forever backup to overcome the latency of the inventory scan across the file system. Rather than relying on a NAS device’s proprietary snapshot or file list function, MAGS allows the Spectrum Protect client to automatically run many incremental backups in parallel, splitting the file system into hundreds or thousands of chunks for the client to scan – like a “Massive Attack” (that’s the MA in MAGS).

Accelerate NAS Backup Processing with Mass Parallelism

It might sound obvious, that splitting a file system into multiple jobs would reduce backup processing, but to do that manually, at scale, is impractical. That is the value of MAGS – it automatically works out how to break down the job into many smaller chunks, and tunes itself, without interfering with how the data is stored.

For example, in one case a Spectrum Protect backup of 6TB of 65 million files to disk reduced from 5 days to just 2.5 hours when MAGS was introduced to the process.

As with the SnapDiff backup, the data can be restored independently of the NAS filer type, and it is even possible to implement MAGS to backup data which has already been backed up via SnapDiff, without having to incur a new full backup.

Furthermore, you can even choose to backup or restore the same data without MAGS, however you’d be missing out on its benefits when restoring large volumes of files as it will accelerate restores using the same methodology. A real-world example has 250 million files totalling half a petabyte restoring from NetApp to Isilon at a rate of over 3TB/hr.

Video: dsmISI MAGS

If you’ve not heard of dsmISI MAGS before, and wonder what IBM thinks of it, then this video of IBM Spectrum Protect evangelists discussing it will help:

Managed Backup and Recovery Service

IBM Spectrum Protect plays a major role in CSI’s Helix Protect managed backup and recovery service.

As both a user and reseller of dsmISI MAGS, we are well-positioned to talk about how we can help protect your unstructured data, whether it is on NetApp, Dell, IBM or any other NAS.

Contact us today for support, or to discuss anything in this blog.

Further reading

The SnapDiff announcement can be found here. You can find more about dsmISI MAGS here.

Scalable backup for NetApp NAS with IBM Spectrum Protect