ONTAP

Inline Adaptive Compression & Inline Data Compaction

ONTAP OS contains a number of storage efficiencies, which are based on WAFL functionalities. Supported by all protocols, does not require licenses. In February 2018[9] NetApp claims AFF systems for its clients gain average 4.72:1 Storage Efficiency from deduplication, compression, compaction and clones savings. Starting with ONTAP 9.3 offline deduplication and compression scanners start automatically by default and based on percentage of new data written instead of scheduling.

• Data Reduction efficiency is summary of Volume and Aggregate Efficiencies and Zero-block deduplication:
  • Volume Efficiencies could be enabled/disabled individually and on volume-by-volume basis:
    1. Offline Volume Deduplication, which works on 4KB block level
    2. Additional efficiency mechanism were introduced later, such as Offline Volume Compression also known as Post-process (or Background) Compression, there are two types: Post-process secondary compression and Post-process adaptive compression
    3. Inline Volume Deduplication and Inline Volume Compression are compress some of the data on the fly before it reaches the disks and designed to leave some of the data in uncompressed form if it considered by ONTAP to take a long time to process them on the fly, and to leverage other storage efficiency mechanisms for this uncompressed data later. There are two types of Inline Volume Compression: Inline adaptive compression and Inline secondary compression
  • Aggregate Level Storage Efficiencies includes:
    1. Data Compaction is another mechanism used to compress many data blocks smaller than 4KB to a single 4KB block
    2. Inline Aggregate-wide data deduplication (IAD) and Post-process aggregate deduplication also known as Cross-Volume Deduplication [10] shares common blocks between volumes on an aggregate. IAD can throttle itself when storage system crosses a certain threshold. The current limit of physical space of a single SSD aggregate is 800TiB
  • Inline Zero-block Deduplication[11] deduplicate zeroes on the fly before they reach disks
• Snapshots and FlexClones are also considered efficiency mechanisms. Starting with 9.4 ONTAP by default deduplicate data across active file system and all the snapshots on the volume, saving from snapshot sharing is a magnitude of number of snapshots, the more snapshots the more savings will be, therefore snapshot sharing gives more savings on SnapMirror destination systems.
• Thin Provisioning

Cross-Volume Deduplication storage efficiency features work only for SSD media. Inline and Offline Deduplication mechanisms that leverage databases consist of links of data blocks and checksums, for those data blocks that been handled by the deduplication process. Each deduplication database is located on each volume and aggregates where deduplication is enabled. All Flash FAS systems does not support Post-process Compression.

Order of Storage Efficiencies execution is as follows:

1. Inline Zero-Block deduplication
2. Inline Compression: for files that could be compressed to the 8KB adaptive compression used, for files more than 32KB secondary compression used
3. Inline Deduplication: Volume first, then Aggregate
4. Inline Adaptive Data Compaction
5. Post-process Compression
6. Post-process Deduplication: Volume first, then Aggregate

WAFL FlexVol Layout on an Aggregate

Internal organization of an Aggregate with two plexes

One or multiple RAID groups form an "aggregate", and within aggregates ONTAP operating system sets up "flexible volumes" (FlexVol) to actually store data that users can access. Similarly to RAID-0, each aggregate consolidates space from underlying protected RAID groups to provide one logical piece of storage for flexible volumes. Alongside with aggregates consists of NetApp's own disks and RAID groups aggregates could consist of LUNs already protected with third-party storage systems with FlexArray, ONTAP Select or Cloud Volumes ONTAP. Each aggregate could consist of either LUNs or with NetApp's own RAID groups. An alternative is "Traditional volumes" where one or more RAID groups form a single static volume. Flexible volumes offer the advantage that many of them can be created on a single aggregate and resized at any time. Smaller volumes can then share all of the spindles available to the underlying aggregate and with a combination of storage QoS allows to change the performance of flexible volumes on the fly while Traditional volumes don't. However, Traditional volumes can (theoretically) handle slightly higher I/O throughput than flexible volumes (with the same number of spindles), as they do not have to go through an additional virtualization layer to talk to the underlying disk. Aggregates and traditional volumes can only be expanded, never contracted. Current maximum aggregate physical useful space size is 800 TiB for All-Flash FAS Systems.[12]

ONTAP is considered to be a unified storage system, meaning that it supports both block-level (FC,FCoE, NVMeoF and iSCSI) & file-level (NFS, pNFS, CIFS/SMB) protocols for its clients. SDS versions of ONTAP (ONTAP Select & Cloud Volumes ONTAP) do not support FC, FCoE or NVMeoF protocols due to their software-defined nature.

NFS was the first protocol available in ONTAP. The latest versions of ONTAP 9 support NFSv2, NFSv3, NFSv4 (4.0 and 4.1) and pNFS. Starting with ONTAP 9.5, 4-byte UTF-8 sequences, for characters outside the Basic Multilingual Plane, are supported in names for files and directories.[21]

ONTAP supports CIFS 2.0 and higher up to SMB 3.1. Starting with ONTAP 9.4 SMB Multichannel, which provides functionality similar to multipathing in SAN protocols, is supported. Starting with ONTAP 8.2 CIFS protocol supports Continuous Availability (CA) with SMB 3.0 for Microsoft Hyper-V over SMB and SQL Server over SMB. ONTAP supports SMB encryption, which is also known as sealing. Accelerated AES instructions (Intel AES NI) encryption is supported in SMB 3.0 and later.

ONTAP on physical appliances supports FCoE as well as FC protocol, depending on HBA port speed.

iSCSI Data Center Bridging (DCB) protocol supported with A220/FAS2700 systems.

NVMe over Fabrics (NVMeoF) refers to the ability to utilize NVMe protocol over existed network infrastructure like Ethernet (Converged or traditional), TCP, Fiber Channel or InfiniBand for transport (as opposite to run NVMe over PCI). NVMe is SAN block level data storage protocol. NVMeoF supported only on All-Flash A-Systems and not supported for low-end A200 and A220 systems. Starting with ONTAP 9.5 ANA protocol supported which provide, similarly to ALUA multi-pathing functionality to NVMe. ANA for NVMe currently supported only with SUSE Enterprise Linux 15. FC-NVMe without ANA supported with SUSE Enterprise Linux 12 SP3 and RedHat Enterprise Linux 7.6.

FC-NVMe Supported on systems with 32Gbps FC ports or higher speeds. The supported Operation Systems with FC-NVMe are: Oracle Linux, VMware, Windows Server, SUSE Linux, RedHat Linux.

High Availability (HA) is clustered configuration of a storage system with two nodes or HA pairs, which aims to ensure an agreed level of operational during expected and unexpected events like reboots, software or firmware updates.

Even though a single HA pair consists of two nodes (or controllers), NetApp has designed it in such a way that it behaves as a single storage system. HA configurations in ONTAP employ a number of techniques to present the two nodes of the pair as a single system. This allows the storage system to provide its clients with nearly-uninterruptable access to their data should a node either fail unexpectedly or need to be rebooted in an operation known as a "takeover."

For example: on the network level, ONTAP will temporarily migrate the IP address of the downed node to the surviving node, and where applicable it will also temporarily switch ownership of FC WWPNs from the downed node to the surviving node. On the data level, the contents of the disks that are assigned to the downed node will automatically be available for use via the surviving node.

FAS and AFF storage systems use enterprise level HDD and SSD drives that are housed within disk shelves that have two bus ports, with one port connected to each controller. All of ONTAP's disks have an ownership marker written to them to reflect which controller in the HA pair owns and serves each individual disk. An Aggregate can include only disks owned by a single node, therefore each aggregate owned by a node and any upper objects such as FlexVol volumes, LUNs, File Shares are served with a single controller. Each controller can have its own disks and aggregates and serve them, therefore such HA pair configurations are called Active/Active where both nodes are utilized simultaneously even though they are not serving the same data.

Once the downed node of the HA pair has been repaired, or whatever maintenance window that necessitated a takeover has been completed, and the downed node is up and running without issue, a "giveback" command can be issued to bring the HA pair back to "Active/Active" status.

High-availability clusters (HA clusters) are the first type of clusterization introduced in ONTAP systems. It aimed to ensure an agreed level of operation. It is often confused with the horizontal scaling ONTAP clusterization that came from the Spinnaker acquisition; therefore, NetApp, in its documentation, refers to an HA configuration as an HA pair rather than as an HA cluster.

An HA pair uses some form of network connectivity (often direct connectivity) for communication between the servers in the pair; this is called an HA interconnect (HA-IC). The HA interconnect can use Ethernet or InfiniBand as the communication medium. The HA interconnect is used for non-volatile memory log (NVLOG) replication using RDMA technology and for some other purposes only to ensure an agreed level of operational during events like reboots always between two nodes in a HA pair configuration. ONTAP assigns dedicated, non-sharable HA ports for HA interconnect which could be external or build in chassis (and not visible from the outside). The HA-IC should not be confused with the intercluster or intracluster interconnect that is used for SnapMirror and that can coexist with data protocols on data ports or with Cluster Interconnect ports used for horizontal scaling & online data migration across the multi-node cluster. HA-IC interfaces are visible only on the node shell level. Starting with A320 HA-IC and Cluster interconnect traffic start to use the same ports.

MetroCluster local and DR pare memory replication in NetApp FAS/AFF systems configured as MCC

MetroCluster (MC) is an additional level of data availability to HA configurations and supported only with FAS and AFF storage systems, later SDS version of MetroCluster was introduced with ONTAP Select & Cloud Volumes ONTAP products. In MC configuration two storage systems (each system can be single node or HA pair) form MetroCluster, often two systems located on two sites with the distance between them up to 300 km therefore called geo-distributed system. Plex is the key underlying technology which synchronizes data between two sites in MetroCluster. In MC configurations NVLOG also replicated between storage systems between sites but uses dedicated ports for that purpose, in addition to HA interconnect. Starting with ONTAP 9.5 SVM-DR supported in MetroCluster configurations.

Is a feature of ONTAP Select software, similarly to MetroCluster on FAS/AFF systems MetroCluster SDS (MC SDS) allows to synchronously replicate data between two sites using SyncMirror and automatically switch to survived node transparently to its users and applications. MetroCluster SDS work as ordinary HA pare so data volumes, LUNs and LIFs could be moved online between aggregates and controllers on both sites, which is slightly different than traditional MetroCluster on FAS/AFF systems where data cloud be moved across storage cluster only within site where data originally located. In traditional MetroCluster the only way for applications to access data locally on remote site is to disable one entire site, this process called switchover where in MC SDS ordinary HA process occurs. MetroCluster SDS uses ONTAP Deploy as the mediator (in FAS and AFF world this functionality known as MetroCluster tiebreaker) which came with ONTAP Select as a bundle and generally used for deploying clusters, installing licenses and monitoring them.

Horizontal scaling ONTAP clusterization came from Spinnaker acquisitions and often referred by NetApp as "Single Namespace", "Horizontal Scaling Cluster" or "ONTAP Storage System Cluster" or just "ONTAP Cluster" and therefore often confused with HA pair or even with MetroCluster functionality. While MetroCluster and HA are Data Protection technologies, single namespace clusterization does not provide data protection. ONTAP Cluster is formed out of one or few HA pairs and adds to ONTAP system Non-Disruptive Operations (NDO) functionality such as non-disruptive online data migration across nodes in the cluster and non-disruptive hardware upgrade. Data migration for NDO operations in ONTAP Cluster require dedicated Ethernet ports for such operations called as cluster interconnect and does not use HA interconnect for this purposes. Cluster interconnect and HA interconnect could not share same ports. Cluster interconnect with a single HA pair could have directly connected cluster interconnect ports while systems with 4 or more nodes require two dedicated Ethernet cluster interconnect switches. ONTAP Cluster could consist only with even number of nodes (they must be configured as HA pairs) except for Single-node cluster. Single-node cluster ONTAP system also called non-HA (stand-alone). ONTAP Cluster managed with a single pain of glass built-in management with Web-based GUI, CLI (SSH and PowerShell) and API. ONTAP Cluster provides Single Name Space for NDO operations through SVM. Single Namespace in ONTAP system is a name for collection of techniques used by Cluster to separate data from front-end network connectivity with data protocols like FC, FCoE, FC-NVMe, iSCSI, NFS and CIFS and therefore provide kind of data virtualization for online data mobility across cluster nodes. On network layer Single Namespace provide a number of techniques for non-disruptive IP address migration, like CIFS Continuous Availability (Transparent Failover), NetApp's Network Failover for NFS and SAN ALUA and path election for online front-end traffic re-balancing with data protocols. NetApp AFF and FAS storage systems can consists of different HA pairs: AFF and FAS, different models and generations and can include up to 24 nodes with NAS protocols or 12 nodes with SAN protocols. SDS systems can't intermix with physical AFF or FAS storage systems.

Storage Virtual Machine

Also known as Vserver or sometimes SVM. Storage Virtual Machine (SVM) is a layer of abstraction, and alongside with other functions, it virtualizes and separates physical front-end data network from data located on FlexVol volumes. It is used for Non-Disruptive Operations and Multi-Tenancy. It also forms as the highest form of logical construct available with NetApp. A SVM can not be mounted under another SVM, therefore it can be referred to a Global Namespace.

SVM divides storage system into slices so few divisions or even organizations can share a storage system without knowing and interfering with each other while using same ports, data aggregates and nodes in the cluster and using separate FlexVol volumes and LUNs. One SVM cannot create, delete, change or even see objects of another SVM so for SVM owners such an environment looks like they are only users on the entire storage system cluster.

SAN ALUA in ONTAP: preferred path with direct data link

There few Non Disruptive Operations (NDO) operations with (Clustered) ONTAP system. NDO data operations include: aggregate relocation within an HA pair between nodes, FlexVol volume online migration (known as Volume Move operation) across aggregates and nodes within Cluster, LUNs migration (known as LUN Move operation) between FlexVol volumes within Cluster. LUN move and Volume Move operations use Cluster Interconnect ports for data transfer (HA-CI is not in use for such operations). SVM behave differently with network NDO operations, depending on front-end data protocol. To decrease latency to its original level FlexVol volumes and LUNs have to be located on the same node with network address through which the clients access storage system, so network address could be created for SAN or moved for NAS protocols. NDO operations are free functionality.

For NAS front-end data protocols there are NFSv2, NFSv3, NFSv4 and CIFSv1, SMBv2 and SMB v3 which do not provide network redundancy with the protocol itself, so they rely on storage and switch functionalities for this matter. For this reason ONTAP support Ethernet Port Channel and LACP with its Ethernet network ports on L2 layer (known in ONTAP as interface group, ifgrp), within a single node and also non-disruptive Network Fail Over between nodes in cluster on L3 layer with migrating Logical Interfaces (LIF) and associated IP addresses (similar to VRRP) to survived node and back home when failed node restored.

For front-end data SAN protocols. ALUA feature used for network load balancing and redundancy in SAN protocols so all the ports on node where data located are reported to clients as active preferred path with load balancing between them while all other network ports on all other nodes in the cluster are active non-preferred path so in case of one port or entire node goes down, client will have access to its data using non-preferred path. Starting with ONTAP 8.3 Selective LUN Mapping (SLM) was introduced to reduce the number of paths to the LUN and removes non-optimized paths to the LUN through all other cluster nodes except for HA partner of the node owning the LUN so cluster will report to the host paths only from the HA pare where LUN is located. Because ONTAP provides ALUA functionality for SAN protocols, SAN network LIFs do not migrate like with NAS protocols. When data or network interfaces migration is finished it is transparent to storage system's clients due to ONTAP Architecture and can cause temporary or permanent data indirect access through ONTAP Cluster interconnect (HA-CI is not in use for such situations) which will slightly increase latency for the clients. SAN LIFs used for FC, FCoE, iSCSi & FC-NVMe protocols.

VIP (Virtual IP) LIFs require Top-of-the-Rack BGP Router used. BGP data LIFs alongside with NAS LIFs also can be used with Ethernet for NAS environment but in case of BGP LIFs, automatically load-balance traffic based on routing metrics and avoid inactive, unused links. BGP LIFs provide distribution across all the NAS LIFs in a cluster, not limited to a single node as in NAS LIFs. BGP LIFs provide smarter load balancing than it was realized with hash algorithms in Ethernet Port Channel & LACP with interface groups. VIP LIF interfaces are tested and can be used with MCC and SVM-DR.

Node management LIF interface can migrate with associated IP address across Ethernet ports of a single node and available only while ONTAP running on the node, usually located on e0M port of the node; Node management IP sometimes used by cluster admin to communicate with a node to cluster shell in rare cases where commands have to be issued from a particular node. Cluster Management LIF interface with associated IP address available only while the entire cluster is up & running and by default can migrate across Ethernet ports, often located on one of the e0M ports on one of the cluster nodes and used for cluster administrator for management purposes; it used for API communications & HTML GUI & SSH console management, by default ssh connect administrator with cluster shell. Service Processor (SP) interfaces available only at hardware appliances like FAS & AFF and allows ssh out-of-band console communications with an embedded small computer installed on controller mainboard and similarly to IPMI allows to connect, monitor & manage controller even if ONTAP OS is not booted, with SP it is possible to forcibly reboot or halt a controller and monitor coolers & temperature, etc.; connection to SP by ssh brings administrator to SP console but when connected to SP it is possible to switch to cluster shell through it; each controller has one SP which does not migrate like some other management interfaces. Usually, e0M and SP both lives on a single management (wrench) physical Ethernet port but each has its own dedicated MAC address. Node LIFs, Cluster LIF & SP often using the same IP subnet. SVM management LIF, similarly to cluster management LIF can migrate across all the Ethernet ports on the nodes of the cluster but dedicated for a single SVM management; SVM LIF does not have GUI capability and can facilitate only for API Communications & SSH console management; SVM management LIF can live on e0M port but often located on a data port of a cluster node on a dedicated management VLAN and can be different from IP subnets that node & cluster LIFs.

The cluster interconnect LIF interfaces using dedicated Ethernet ports and cannot share ports with management and data interfaces and for horizontal scaling functionality at times when like a LUN or a Volume migrates from one node of the cluster to another; cluster interconnect LIF similarly to node management LIFs can migrate between ports of a single node. Intercluster interface LIFs can live and share the same Ethernet ports with data LIFs and used for SnapMirror replication; intercluster interface LIFs, similarly to node management & LIFs cluster interconnect can migrate between ports of a single node.

Multi Tenancy

ONTAP provide two techniques for Multi Tenancy functionality like Storage Virtual Machines and IP Spaces. On one hand SVMs are similar to Virtual Machines like KVM, they provide visualization abstraction from physical storage but on another hand quite different because unlike ordinary virtual machines SVMs does not allow to run third party binary code like in Pure storage systems; they just provide virtualized environment and storage resources instead. Also SVMs unlike ordinary virtual machines do not run on a single node but for the end user it looks like an SVM runs as a single entity on each node of the whole cluster. SVM divides storage system into slices, so a few divisions or even organizations can share a storage system without knowing and interfering with each other while utilizing same ports, data aggregates and nodes in the cluster and using separate FlexVol volumes and LUNs. Each SVM can run its own frontend data protocols, set of users, use its own network addresses and management IP. With use of IP Spaces users can have the same IP addresses and networks on the same storage system without interfering. Each ONTAP system must run at least one Data SVM in order to function but may run more. There are a few levels of ONTAP management and Cluster Admin level has all of the available privileges. Each Data SVM provides to its owner vsadmin which has nearly full functionality of Cluster Admin level but lacks physical level management capabilities like RAID group configuration, Aggregate configuration, physical network port configuration. However, vsadmin can manage logical objects inside an SVM like create, delete and configure LUNs, FlexVol volumes and network addresses, so two SVMs in a cluster can't interfere with each other. One SVM cannot create, delete, modify or even see objects of another SVM, so for SVM owners such an environment looks like they are the only users in the entire storage system cluster. Multi Tenancy is free functionality in ONTAP.

NetApp FlexClone works exactly as NetApp RoW Snapshots but allows to write to FlexClones

FlexClone is a licensed feature, used for creating writable copies of volumes, files or LUNs. In case of volumes, FlexClone acts as a snapshot but allows to write into it, while an ordinary snapshot allows only to read data from it. Because WAFL architecture FlexClone technology copies only metadata inodes and provides nearly instantaneous data copying of a file, LUN or volume regardless of its size.

SnapRestore is a licensed feature, used for reverting active file system of a FlexVol to a previously created snapshot for that FlexVol with restoring metadata inodes in to active file system. SnapRestore is used also for a single file restore or LUN restore from a previously created snapshot for the FlexVol where that object located. Without SnapRestore license in NAS environment it is possible to see snapshots in network file share and be able to copy directories and files for restore purposes. In SAN environment there is no way of doing restore operations similar to NAS environment. It is possible to copy in both SAN and NAS environments files, directories, LUNs and entire FlexVol content with ONTAP command ndmpcopy which is free. Process of copying data depend on the size of the object and could be time consuming, while SnapRestore mechanism with restoring metadata inodes in to active file system almost instant regardless of the size of the object been restored to its previous state.

FlexGroup is a free feature introduced in version 9, which utilizes the clustered architecture of the ONTAP operating system. FlexGroup provides cluster-wide scalable NAS access with NFS and CIFS protocols.[22] A FlexGroup Volume is a collection of constituent FlexVol volumes distributed across nodes in the cluster called just "Constituents", which are transparently aggregated in a single space. Therefore, FlexGroup Volume aggregates performance and capacity from all the Constituents and thus from all nodes of the cluster where they located. For the end user, each FlexGroup Volume is represented by a single, ordinary file-share.[23] The full potential of FlexGroup will be revealed with technologies like pNFS (currently not supported with FlexGroup), NFS Multipathing (session trunking, also not available in ONTAP) SMB multichannel (currently not supported with FlexGroup), SMB Continuous Availability (FlexGroup with SMB CA Supported with ONTAP 9.6), and VIP (BGP). The FlexGroup feature in ONTAP 9 allows to massively scale in a single namespace to over 20PB with over 400 billion files, while evenly spreading the performance across the cluster.[24] Starting with ONTAP 9.5 FabricPool supported with: FlexGroup, it is recommended to have all the constituent volumes to backup to a single S3 bucket; supports SMB features for native file auditing, FPolicy, Storage Level Access Guard (SLA), copy offload (ODX) and inherited watches of changes notifications; Quotas and Qtree. SMB Contiguous Availability (CA) supported on FlexGroup allows running MS SQL & Hyper-V on FlexGroup, and FlexGroup supported on MetroCluster.

Unified Replication

Snapshots form the basis for NetApp's asynchronous disk-to-disk replication (D2D) technology, SnapMirror, which effectively replicates Flexible Volume snapshots between any two ONTAP systems. SnapMirror is also supported from ONTAP to Cloud Backup and from SolidFire to ONTAP systems as part of NetApp's Data Fabric vision. NetApp also offers a D2D backup and archive feature named SnapVault, which is based on replicating and storing snapshots. Open Systems SnapVault allows Windows and UNIX hosts to back up data to an ONTAP, and store any filesystem changes in snapshots (not supported in ONTAP 8.3 and onwards). SnapMirror is designed to be part of a Disaster recovery plan: it stores an exact copy of data on time when snapshot was created on the disaster recovery site and could keep the same snapshots on both systems. SnapVault, on the other hand, is designed to store less snapshots on the source storage system and more Snapshots on a secondary site for a long period of time.
Data captured in SnapVault snapshots on destination system could not be modified nor accessible on destination for read-write, data can be restored back to primary storage system or SnapVault snapshot could be deleted. Data captured in snapshots on both sites with both SnapMirror and SnapVault can be cloned and modified with the FlexClone feature for data cataloging, backup consistency and validation, test and development purposes etc.
Later versions of ONTAP introduced cascading replication, where one volume could replicate to another, and then another, and so on. Configuration called fan-out is a deployment where one volume replicated to multiple storage systems. Both fan-out and cascade replication deployments support any combination of SnapMirror DR, SnapVault, or unified replication. It is possible to use fan-in deployment to create data protection relationships between multiple primary systems and a single secondary system: each relationship must use a different volume on the secondary system. Starting with ONTAP 9.4 destination SnapMirror & SnapVault systems enable automatic inline & offline deduplication by default.
Intercluster is a relationship between two clusters for SnapMirror, while Intracluster is opposite to it and used for SnapMirror relationship between storage virtual machines (SVM) in a single cluster.
SnapMirror can operate in version-dependent mode, where two storage systems must run on the same version of ONTAP or in version-flexible mode. Types of SnapMirror replication:

• Data Protection (DP): Also known as SnapMirror DR. Version-dependent replication type originally developed by NetApp for Volume SnapMirror, destination system must be same or higher version of ONTAP. Not used by default in ONTAP 9.3 and higher. Volume-level replication, block-based, metadata independent, uses Block-Level Engine (BLE).
• Extended Data Protection (XDP): Used by SnapMirror Unified replication and SnapVault. XDP uses the Logical Replication Engine (LRE) or if volume efficiency different on the destination volume the Logical Replication Engine with Storage Efficiency (LRSE). Used as Volume-level replication but technologically could be used for directory-based replication, inode-based, metadata dependent (therefore not recommended for NAS with millions of files).
• Load Sharing (LS): Mostly used for internal purposes like keeping copies of root volume for an SVM.
• SnapMirror to Tape (SMTape): is Snapshot copy-based incremental or differential backup from volumes to tapes; SMTape feature performing a block-level tape backup using NDMP-compliant backup applications such as CommVault Simpana.

SnapMirror-based technologies:

• Unified replication: A volume with Unified replication can get both SnapMirror and SnapVault snapshots. Unified replication is combination of SnapMirror Unified replication and SnapVault which using a single replication connection. Both SnapMirror Unified replication and SnapVault are using same XDP replication type. SnapMirror Unified Replication is also known as Version-flexible SnapMirror. Version-flexible SnapMirror/SnapMirror Unified Replication introduced in ONTAP 8.3 and removes the restriction to have the destination storage use the same, or higher, version of ONTAP.
• SVM-DR (SnapMirror SVM): replicates all volumes (exceptions allowed) in a selected SVM and some of the SVM settings, replicated settings depend on protocol used (SAN or NAS)
• Volume Move: Also known as DataMotion for Volumes. SnapMirror replicates volume from one aggregate to another within a cluster, then I/O operations stops for acceptable timeout for end clients, final replica transferred to destination, source deleted and destination becomes read-write accessible to its clients

SnapMirror is a licensed feature, a SnapVault license is not required if a SnapMirror license is already installed.

SVM DR based on SnapMirror technology which transferring all the volumes (exceptions allowed) and data in them from a protected SVM to a DR site. There are two modes for SVM DR: identity preserve and identity discard. With Identity discard mode, on the one hand, data in volumes copied to the secondary system and DR SVM does not preserve information like SVM configuration, IP addresses, CIFS AD integration from original SVM. On another hand in identity discard mode, data on the secondary system can be brought online in read-write mode while primary system online too, which might be helpful for DR testing, Test/Dev and other purposes. Therefore, identity discard requires additional configuration on the secondary site in the case of disaster occurs on the primary site.

In the identity preserve mode, SVM-DR copying volumes and data in them and also information like SVM configuration, IP addresses, CIFS AD integration which requires less configuration on DR site in case of disaster event on primary site but in this mode, the primary system must be offline to ensure there will be no conflict.

SnapMirror Sync (SM-S) for short is zero RPO data replication technology previously available in 7-mode systems and was not available in (clustered) ONTAP until version 9.5. SnapMirror Sync replicates data on Volume level and has requirements for RTT less than 10ms which gives distance approximately of 150 km. SnapMirror Sync can work in two modes: Full Synchronous mode (set by default) which guarantees zero application data loss between two sites by disallowing writes if the SnapMirror Sync replication fails for any reason. Relaxed Synchronous mode allows an application to write to continue on primary site if the SnapMirror Sync fails and once the relationship resumed, automatic re-sync will occur. SM-S supports FC, iSCSI, NFSv3, NFSv4, SMB v2 & SMB v3 protocols and have the limit of 100 volumes for AFF, 40 volumes for FAS, 20 for ONTAP Select and work on any controllers which have 16GB memory or more. SM-S is useful for replicating transactional logs from: Oracle DB, MS SQL, MS Exchange etc. Source and destination FlexVolumes can be in a FabricPool aggregate but must use backup policy, FlexGroup volumes and quotas are not currently supported with SM-S. SM-S is not free feature, the license is included in the premium bundle. Unlike SyncMirror, SM-S not uses RAID & Plex technologies, therefore, can be configured between two different NetApp ONTAP storage systems with different disk type & media.

FlexCache technology previously available in 7-mode systems and was not available in (clustered) ONTAP until version 9.5. FlexCache allows serving NAS data across multiple global sites with file locking mechanisms. FlexCache volumes can cache reads, writes, and metadata. Writes on the edge generating push operation of the modified data to all the edge ONTAP systems requested data from the origin, while in 7-mode all the writes go to the origin and it was an edge ONTAP system's job to check the file haven't been updated. Also in FlexCache volumes can be less size that original volume, which is also an improvement compare to 7-mode. Initially, only NFS v3 supported with ONTAP 9.5. FlexCache volumes are sparsely-populated within an ONTAP cluster (intracluster) or across multiple ONTAP clusters (inter-cluster). FlexCache communicates over Intercluster Interface LIFs with other nodes. Licenses for FlexCache based on total cluster cache capacity and not included in the premium bundle. FAS, AFF & ONTAP Select can be combined to use FlexCache technology. Allowed to create 10 FlexCache volumes per origin FlexVol volume, and up to 10 FlexCache volumes per ONTAP node. The original volume must be stored in a FlexVol while all the FlexCache Volumes will have FlexGroup volume format.

SyncMirror replication using plexes

Data ONTAP also implements an option named RAID SyncMirror (RSM), using the plex technique, where all the RAID groups within an aggregate or traditional volume can be synchronously duplicated to another set of hard disks. This is typically done at another site via a Fibre Channel or IP link, or within a single controller with local SyncMirror for a single disk-shelf resiliency. NetApp's MetroCluster configuration uses SyncMirror to provide a geo-cluster or an active/active cluster between two sites up to 300 km apart or 700 km with ONTAP 9.5 and MCC-IP. SyncMirror can be used either in software-defined storage platforms, on Cloud Volumes ONTAP, or on ONTAP Select. It provides high availability in environments with directly attached (non-shared) disks on top of commodity servers, or at FAS and AFF platforms in Local SyncMirror or MetroCluster configurations. SyncMirror is a free feature.

SnapLock implements Write Once Read Many (WORM) functionality on magnetic and SSD disks instead of to optical media so that data cannot be deleted until its retention period has been reached. SnapLock exists in two modes: compliance and enterprise. Compliance mode was designed to assist organizations in implementing a comprehensive archival solution that meets strict regulatory retention requirements, such as regulations dictated by the SEC 17a-4(f) rule, FINRA, HIPAA, CFTC Rule 1.31(b), DACH, Sarbanes-Oxley, GDPR, Check 21, EU Data Protection Directive 95/46/EC, NF Z 42-013/NF Z 42-020, Basel III, MiFID, Patriot Act, Graham-Leach-Bliley Act etc. Records and files committed to WORM storage on a SnapLock Compliance volume cannot be altered or deleted before the expiration of their retention period. Moreover, a SnapLock Compliance volume cannot be destroyed until all data has reached the end of its retention period. SnapLock is a licensed feature.

SnapLock Enterprise is geared toward assisting organizations that are more self-regulated and want more flexibility in protecting digital assets with WORM-type data storage. Data stored as WORM on a SnapLock Enterprise volume is protected from alteration or modification. There is one main difference from SnapLock Compliance: as the files being stored are not for strict regulatory compliance, a SnapLock Enterprise volume can be destroyed by an administrator with root privileges on the ONTAP system containing the SnapLock Enterprise volume, even if the designed retention period has not yet passed. In both modes, the retention period can be extended, but not shortened, as this is incongruous with the concept of immutability. Also, NetApp's SnapLock data volumes are equipped with a tamper-proof compliance clock, which is used as a time reference to block forbidden operations on files, even if the system time tampered.

Starting with ONTAP 9.5 SnapLock supports Unified SnapMirror (XDP) engine, re-synchronization after fail-over without data loss, 1023 snapshots, efficiency mechanisms and clock synchronization in SDS ONTAP.

FabricPool tiering to S3

Available for SSD-only aggregates in FAS/AFF systems or Cloud Volumes ONTAP on SSD media. Starting with ONTAP 9.4 FabricPool supported on ONTAP Select platform. Cloud Volumes ONTAP also supports HDD + S3 FabricPool configuration. Fabric Pool provides automatic storage tiering capability for cold data blocks from fast media (usually SSD) on ONTAP storage to cold media via object protocol to object storage such as S3 and back. Fabric Pool can be configured in two modes: One mode is used to migrate cold data blocks captured in snapshots, while the other mode is used to migrate cold data blocks in an active file system. FabricPool preserves offline deduplication & offline compression savings. Starting with ONTAP 9.4 introduced FabricPool 2.0 with the ability to tier-off active file system data (by default 31-day data not been accessed) & support data compaction savings. The recommended ratio is 1:10 for inodes to data files. For clients connected to the ONTAP storage system, all the Fabric Pool data-tiering operations are completely transparent, and in case data blocks become hot again, they are copied back to fast media in the ONTAP storage system. Fabric Pool is currently compatible with the NetApp StorageGRID, Amazon S3, Google Cloud, and Alibaba object storage services. Starting with ONTAP 9.4 Azure Blob supported, starting with 9.5 IBM Cloud Object Storage (ICOS) and Amazon Commercial Cloud Services (C2S) supported, other object-based SW & services could be used if requested by the user and that service will be validated by NetApp. FlexGroup volumes supported with FabricPool starting with ONTAP 9.5. The Fabric Pool feature in FAS/AFF systems is free for use with NetApp StorageGRID external object storage. For other object storage systems such as Amazon S3 & Azure Blob, Fabric Pool must be licensed per TB to function (alongside costs for Fabric Pool licensing, the customer needs to also pay for consumed object space). While with the Cloud Volumes ONTAP storage system, Fabric Pool does not require licensing, costs will apply only for consumed space on the object storage. Starting with ONTAP 9.5 capacity utilization triggering tiering from hot tier can be adjusted. SVM-DR also supported by FlexGroups.

FabricPool, first available in ONTAP 9.2, is a NetApp Data Fabric technology that enables automated tiering of data to low-cost object storage tiers either on or off-premises. Unlike manual tiering solutions, FabricPool reduces the total cost of ownership by automating the tiering of data to lower the cost of storage. It delivers the benefits of cloud economics by tiering to public clouds such as Alibaba Cloud Object Storage Service, Amazon S3, Google Cloud Storage, IBM Cloud Object Storage, and Microsoft Azure Blob Storage as well as to private clouds such as NetApp StorageGRID®. FabricPool is transparent to applications and allows enterprises to take advantage of cloud economics without sacrificing performance or having to re-architect solutions to leverage storage efficiency.

NetApp storage systems running ONTAP can Flash Cache (formally Performance Accelerate Module or PAM) custom purpose-built proprietary PCIe card for hybrid NetApp FAS systems. Flash Cache can reduce read latencies and allows the storage systems to process more read-intensive work without adding any further spinning disk to the underlying RAID since read operations do not require redundancy in case of Flash Cache failure. Flash Cache works on controller level and accelerates only read operations. Each separate volume on the controller can have a different caching policy or read cache could be disabled for a volume. FlashCache caching policies applied on FlexVol level. FlashCache technology is compatible with the FlexArray feature. Starting with 9.1, a single FlexVol volume can benefit from both FlashPool & FlashCache caches simultaneously. Beginning with ONTAP 9.5 Flash Cache read-cache technology available in Cloud Volumes ONTAP with the use of ephemeral SSD drives.

NDAS proxy is a service introduced in ONTAP 9.5; it works in conjunction with NDAS service in a cloud provider. Similarly to FabricPool, NDAS stores data in object format, but unlike FabricPool, it stores WAFL metadata in object storage as well. The information which been transferred from the ONTAP system is snapshot deltas, not the entire set of data, and already deduplicated & compressed (on volume level). NDAS proxy is HTTP-based with an S3 object protocol and few additional API calls to the cloud. NDAS in ONTAP 9.5 works only in a schema with primary ONTAP 9 storage replicating data via Snapmirror to secondary ONTAP 9.5 storage, where secondary storage is also NDAS proxy.

Storage QoS is a free feature in ONTAP systems. There are few types of storage QoS in ONTAP systems: Adaptive QoS (A-QoS), which includes Absolute minimum QoS; Ordinary static QoS or Minimum QoS (QoS min); and Maximum QoS (QoS max). Maximum QoS can be configured as a static upper limit in IOPS, MB/s, or both. It can be applied to an object such as Volume, LUN or a file, to prevent from such an object from consuming more storage performance resources than defined by the administrator (thus isolating performance-intensive bullies and protecting other workloads). Minimum QoS is contrary to maximum set on volumes to ensure that the volume will get no less than configured by the administrator static number of IOPS when there is contention for storage performance resources and could be applied to volumes. A-QoS is a mechanism of automatically changing QoS, based on consumed space by a flexible volume, because consumed space in it could grow or decrease, and the size of FlexVol can be changed. On FAS systems, A-QoS reconfigures only Peak performance (QoS max), while on AFF systems, it reconfigure both Expected performance (QoS min) and Peak performance (QoS max) on a volume. A-QoS allows ONTAP to automatically adjust the number of IOPS for a volume based on A-QoS policies. There are three basic A-QoS policies: Extreme, Performance and Value. Each A-QoS policy has a predefined fixed ratio IO per TB for Peak performance and Expected performance (or Absolute minimum QoS). Absolute minimum QoS is used instead of Expected performance (QoS min) only when volume size and ratio IO per TB is too small for example 10GB.

ONTAP OS has a number of features to increase security on the storage system like Onboard Key Manager, the passphrase for controller boot with NSE & NVE encryption and USB key manager (available starting with 9.4). Auditing for NAS events is another security measure in ONTAP that enables the customer to track and log certain CIFS and NFS events on the storage system. This helps to track potential security problems and provides evidence of any security breaches. ONTAP accessed over SSH has an ability to Authenticate with a Common Access Card. ONTAP supports RBAC: Role-based access control allows administrative accounts to be restricted and/or limited in what actions they can take on the system. RBAC prevents a single account from being allowed to perform all potential actions available on the system. Beginning with ONTAP 9, Kerberos 5 authentication with privacy service (krb5p) is supported for NAS. The krbp5 authentication mode protects against data tampering and snooping by using checksums to encrypt all traffic between client and server. The ONTAP solution supports 128-bit and 256-bit AES encryption for Kerberos.

Onboard Key Manager is a free feature introduced in 9.1 and can store keys from NVE encrypted volumes & NSE disks. NSE Disks are available only on AFF/FAS platforms. ONTAP systems also allow storing encryption keys on a USB drive connected to the appliance. ONTAP also can use an external key manager like Gemalto Trusted Key Manager.

NetApp Volume Encryption (NVE) is FlexVol volume-level software-based encryption, which uses storage CPU for data encryption purposes; thus, some performance degradation is expected though it is less noticeable on high-end storage systems with more CPU cores. NVE is licensed, but free features compatible nearly with all NetApp ONTAP features and protocols. Similarly to NetApp Storage Encryption (NSE), NVE can store encryption keys locally or on a dedicated key manager like IBM Security Key Lifecycle Manager, SafeNet KeySecure or cloud key managers. NVE, like NSE, is also data at rest encryption, which means it protects only from physical disks theft and does not give an additional level of data security protection in a healthy operational and running system. NVE with a combination of FabricPool technology also protects data from unauthorized access in external S3 storage systems like Amazon and since data already encrypted it transferring over the wire in encrypted form.

Starting with ONTAP 9.4 new feature introduced called Secure Purge which provides ability to securely delete a file to comply with GDPR requirements.

ONTAP Vscan and FPolicy are aimed at malware prevention in ONTAP systems with NAS. Vscan provides a way for NetApp antivirus scanner partners to verify that files are virus-free. FPolicy integrates with NetApp partners to monitor file access behaviors. FPolicy file-access notification system monitor activity on NAS storage and prevent unwanted access or change to files based on policy settings. Both help in preventing ransomware from getting a foothold in the first place.

MTU black-hole detection and path MTU discovery (PMTUD) is the processes by which the ONTAP system connected via an Ethernet network detects maximum MTU size. In ONTAP 9.2: Online Certificate Status Protocol (OCSP) for LDAP over TLS; iSCSI Endpoint Isolation to specify a range of IP addresses that can log in to the storage; limit the number of failed login attempts over SSH. NTP symmetric authentication supported starting with ONTAP 9.5.

NetApp Workflow Automation (WFA) is a free, server-based product used for NetApp storage orchestration. It includes a self-service portal with a web-based GUI, where nearly all routine storage operations or sequences of operations can be configured as workflows and published as a service, so end users can order and consume NetApp storage as a service.

SnapCenter, previously known as SnapManager Suite, is a server-based product. NetApp also offers products for taking application-consistent snapshots by coordinating the application and the NetApp Storage Array. These products support Microsoft Exchange, Microsoft SQL Server, Microsoft Sharepoint, Oracle, SAP and VMware ESX Server data. These products form part of the SnapManager suite. SnapCenter also includes third-party plugins for MongoDB, IBM DB2, MySQL, and allows the end user to create their own plugins for integration with the ONTAP storage system. SnapManager and SnapCenter are enterprise-level licensed products. A similar, free, and less capable NetApp product exists, named SnapCreator. It is intended for customers who wish to integrate ONTAP application-consistent snapshots with their applications, but do not have a license for SnapCenter. NetApp claims that SnapCenter capabilities will expand to include SolidFire storage endpoints. SnapCenter has controller based licensing for AFF/FAS systems and by Terabyte for SDS ONTAP. SnapCenter Plug-in for VMware vSphere called NetApp Data Broker is a separate linux-based applience which can be used without SnapCenter itself.

NetApp Services Level Manager or NSLM for short is software for provisioning ONTAP storage that delivers predictable performance, capacity and data protection for a workload which exposes RESTful APIs and has built-in Swagger documentation with the list of the available APIs, and also can be integrated with other NetApp storage products like ActiveIQ Unified Manager. NSLM exposes three standard service levels (SSL) based on service level objectives (SLO) and creates custom service levels. NSLM created to provide predicted ServiceProvider-like storage consumption. NSLM is a space-based licensed product.

ONTAP systems have the ability to integrate with Hadoop TeraGen, TeraValidate and TeraSort, Apache Hive, Apache MapReduce, Tez execution engine, Apache Spark, Apache HBase, Azure HDInsight and Hortonworks Data Platform Products, Cloudera CDH, through NetApp In-Place Analytics Module (also known as NetApp NFS Connector for Hadoop) to provide access and analyze data by using external shared NAS storage as primary or secondary Hadoop storage.

A qtree[25] is a logically defined file system with no restrictions on how much disk space can be used or how many files can exist. In general, qtrees are similar to volumes. However, they have the following key restrictions:

• Snapshot copies can be enabled or disabled for individual volumes but not for individual qtrees.
• Qtrees do not support space reservations or space guarantees.

FAS[26] and All Flash FAS (AFF)[27] systems are proprietary, custom-built hardware by NetApp for ONTAP software. AFF systems can contain only SSD drives, because ONTAP on AFF is optimized and tuned only for Flash memory, while FAS systems may contain HDD (HDD-only systems) or HDD and SSD (Hybrid systems). ONTAP on FAS and AFF platforms can create RAID arrays, such as RAID-4, RAID-DP and RAID-TEC arrays, from disks or disk partitions for data protection reasons, while ONTAP Select and Cloud Volumes ONTAP leverage RAID data protection provided by the environment they run on. FAS and AFF systems support Metro Cluster functionality, while ONTAP Select and Cloud Volumes ONTAP platforms do not.

Both ONTAP Select and Cloud Volumes ONTAP are virtual storage appliances (VSA) which are based on previous product ONTAP Edge also known as ONTAP-v and considered as a Software-defined storage.[28] ONTAP Select as Cloud Volumes ONTAP includes plex and aggregate abstractions, but didn't have a lower level RAID module included in the OS; therefore RAID-4, RAID-DP and RAID-TEC were not supported so ONTAP storage system similarly to FlexArray functionality leverages RAID data protection on SSD and HDD drive level with underlying storage systems. Starting with ONTAP Select 9.4 & ONTAP Deploy 2.8 software RAID supported with no requirements for 3rd party HW RAID equipment. Because ONTAP Select and Cloud Volumes ONTAP are virtual machines, they don't support Fibre Channel and Fibre Channel over Ethernet as front-end data protocols and consume space from underlying storage in hypervisor added to VSA as virtual disks represented and treated inside ONTAP as disks. ONTAP Select and Cloud Volumes ONTAP provide high availability, deduplication, resiliency, data recovery, robust snapshots which can be integrated with application backup (application consistent snapshots) and nearly all ONTAP functionality but with few exceptions. Software-defined versions of ONTAP have nearly all the functionality except for Hardware-centric features like ifgroups, service processor, physical disk drives with encryption, MetroCluster over FCP, Fiber Channel protocol.

NetApp ONTAP Select

NetApp ONTAP Select

ONTAP Select can run on VMware ESXi and Linux KVM hypervisors. ONTAP Select leveraged RAID data protection on SSD and HDD drive level with underlying DAS, SAN, or vSAN storage systems. Starting with ONTAP Select 9.4 & ONTAP Deploy 2.8 software RAID supported with no requirements for 3rd party HW RAID equipment for KVM and starting with ONTAP 9.5 with ESXi. ONTAP Deploy is a virtual machine that provides a mediator function in MetroCluster or 2-node configurations, keeps track of licensing, and used to initial cluster deployment. Starting with ONTAP Deploy 2.11.2 vCenter plugin was introduced, which allows performing all the ONTAP Deploy functionality from vCenter. In contrast, previously management performed from either command line or with vSphere VM OVA setup master. Like on the FAS platform, ONTAP Select supports high availability and clustering. As a FAS platform, ONTAP Select is offered in two versions: HDD-only or All-Flash optimized. Previously ONTAP Select known as Data ONTAP Edge. Data ONTAP Edge product has Data ONTAP OS with version 8 and was able to run only atop of VMware ESXi. Starting with ONTAP 9.5 SW-MetroCluster over NSX overlay network supported. Starting with ONTAP 9.5 licensing changed from capacity tier-based, where licenses are linked with a node and perpetual to Capacity Pool Licensing with a time-limited subscription. ONTAP Select 9.5 get MQTT protocol supported for data transferring from the edge to a data center or a cloud. In April 2019, Octavian Tanase SVP ONTAP, posted a preview photo in his twitter of ONTAP running in Kubernetes as a container for a demonstration.

Cloud Volumes ONTAP (formally ONTAP Cloud[29]) includes nearly the same functionality as ONTAP Select, because it is also a virtual storage appliance (VSA) and can be ordered in hyper-scale providers (cloud computing) such as Amazon AWS, Microsoft Azure and Google Cloud Platform. IBM Cloud uses ONTAP Select for the same reasons, instead of Cloud Volumes ONTAP. Cloud Volumes ONTAP can provide high availability of data across different regions in the cloud. Cloud Volumes ONTAP leverages RAID data protection on SSD and HDD drive level with underlying IP SAN storage system in Cloud Provider.