Data Archiving and Cloud Replication Using Igneous

By: Lily Bowdler, Chris Hoffman, and Andy Ferris; Igneous Systems

Igneous integrates directly with Google Cloud to replicate data and to archive cold data for long-term storage. Igneous provides policy-based data movement and deletion, reducing management overhead and keeping total storage costs down, while helping to ensure regulatory compliance.

You can set up a policy in Igneous that automatically creates a second copy of any on-premises backup data to store in Cloud Storage. You can use Igneous to search through files stored in Cloud Storage and quickly restore them to Igneous.

Features

Igneous is a scale-out secondary storage platform installed in your data center that is designed for unstructured file data. Igneous does the following:

  • Integrates directly with your primary-tier network-attached storage (NAS) device, providing round-the-clock replication, backup, and archiving.
  • Manages unstructured data across your primary-tier NAS, Igneous, and any tier of Cloud Storage.
  • Provides visibility into files that are stored across all storage sources through an intuitive web-based user interface (UI).
  • Searches across your primary-tier NAS, Igneous, and any tier of Cloud Storage and restores data with a single click.
  • Efficiently moves both small and large files using transfer compression and object grouping.

Benefits

Igneous is designed to move enterprise-scale unstructured data to Cloud Storage. Igneous does the following:

  • Reduces the number of Cloud Storage operations through transfer compression and object grouping.
  • Saves IT time by running backups automatically in the background without affecting performance.
  • Boosts the visibility of your data wherever it lives.
  • Maintains continuous control whether you store data in native format or in an open, self-describing format.

Value

Cloud Storage Nearline and Cloud Storage Coldline are valuable for long-term archiving and replication of file data, but it can be difficult, costly, and time-consuming to identify and move the data from your data center to Cloud Storage. These issues are exacerbated when dealing with billions of files across petabytes of NAS. Igneous features a UI that provides visibility, search, and data movement functionality across all connected storage sources. Cross-tier keyword search and a consolidated secondary tier help you find and move your data.

Architecture

Igneous is an on-premises secondary storage tier that provides backup for primary NAS storage and simplifies moving data to Cloud Storage, as illustrated in the following figure.

Architectural diagram of Igneous as an on-premises secondary storage that moves data to Cloud Storage

Igneous offers the benefits of cloud services along with hardware that is installed in your data center. With its on-premises infrastructure, Igneous is designed to be your secondary storage tier, In addition, the cloud-native architecture means you can use Cloud Storage as a tertiary tier.

Igneous provides native API integration for Dell EMC Isilon, NetApp, Pure Storage FlashBlade, and Qumulo QF2. It can also back up generic NAS file systems by using parallel, latency-aware data movement that is capable of handling environments with billions of files and petabytes of data.

Data movement engine

Igneous moves data from primary storage in parallel streams. By accessing NAS systems through frontend protocols, such as Network File System (NFS) and Server Message Block (SMB), Igneous opens many parallel streams. In addition, the scanning and data movement algorithms are engineered to maximize throughput and minimize impact on primary NAS systems.

The data movement engine behind Igneous is a policy-driven system of data-replication microservices that provides integrated backup and archiving for unstructured file data. It is highly scalable, resilient, and efficient.

Cloud integration

To constrain your data center footprint and provide long-term retention, Igneous automates tiering and data replication to Cloud Storage. You can set policies to replicate or group data into tiers for archiving. Backups run automatically in the background without affecting performance. Igneous integrates with and manages data across all Cloud Storage tiers (Standard, Nearline, or Coldline), allowing data to be where it needs to be when it needs to be there.

Performance

Igneous quickly scans and copies data from NAS systems at scale. The exact speed of data movement depends on the size and density of your primary NAS. In addition, the round-the-clock backups are latency aware: when the primary tier is experiencing a lot of other traffic, the backup job slows down to avoid affecting the performance of the primary tier. Whether or not Igneous is currently moving data to and from Cloud Storage, the performance of the primary tier isn't affected.

Igneous is meant to be secondary storage. Its performance level falls between modern primary NAS that is sold by vendors like NetApp, Dell EMC Isilon, Pure Storage, and Qumulo, and modern public cloud archive tiers like Cloud Storage Nearline or Coldline storage classes. Regardless of location, data that is stored with Igneous is readily available to restore. For most people, cold data copy speed isn't paramount. Igneous compresses and groups data to minimize time and costs.

Costs

Igneous is on-premises secondary storage that integrates directly with on-premises primary NAS systems and Cloud Storage. Igneous charges an annual subscription that is based on the capacity of data under management. Your overall cost consists of two parts: your Igneous subscription fee and your Cloud Storage fees. The automatic policy-based tiering of Igneous means that you keep cool data on-premises for a given period of time. You can store cold data in Cloud Storage Nearline and Cloud Storage Coldline to reduce the need to pay fees for egress from Cloud Storage. In addition, Igneous throttles data movement to and from Cloud Storage to reduce total costs and manage network capacity.

Use cases

You can integrate Igneous with Cloud Storage to replicate backups for disaster recovery and to store cold data for archiving.

  • For extra protection of on-premises primary data, replicate each completed backup to the public cloud. Then, as short-term backups age out of Igneous, the replicated copy remains in Cloud Storage.
  • For long-term storage of unstructured data, move your cold data to Cloud Storage. Using this approach, you can control growth of primary NAS, retain data economically, maintain access, and comply with industry-specific regulations.

Try it for yourself

This section describes how to set up a data retention policy and replicate to Cloud Storage.

  1. In Igneous, create a policy and enable replication.

  2. In the Add an account drop-down list, select Google Cloud Platform.

  3. To create a service account for Igneous, click Go to the GCP Service Accounts page.

    GCP services accounts page in Igneous

  4. To create a new service account, follow these steps:

    1. Create a unique name for the account in the Service account name box.

    2. To provide the Igneous service account with project owner privileges, from the Role drop-down list, select Project > Owner.

      Drop-down list of project roles

    3. To download a file with a private key, click JSON.

    4. Click Create and download the JSON key.

  5. To upload the JSON key for the service account to Igneous, click Select JSON key. Locate the downloaded JSON key, and click Open.

  6. Select the storage tier that you want to use in Cloud Storage, and then click Create policy.

    Storage tier selection when you create a policy

  7. Apply the policy to the systems and exports that you want replicated to Cloud Storage. Select your policy from the drop-down list.

  8. To replicate your data to your Cloud Storage bucket immediately, click **Run now **in the Actions drop-down list to the right of your export.

    Data replication selection

Igneous creates the buckets, which appear in your Cloud Storage bucket.

Cloud Storage buckets created by Igneous

What's next