Network Working Group                                    Harald Skardal,
INTERNET DRAFT                                        Network Appliance.
Category: Applications
Document: draft-skardal-ndmp4-Snapmirror-01.txt

NDMP Version 4 Snapmirror Management Extension Interface

Status of this Memo
This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. 
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." 
The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt 
The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 

Abstract
The Network Data Management Protocol (NDMP) defines a mechanism and protocol for controlling backup, recovery, and other transfers of data between source and destination storage. The XDR and TCP/IP protocols are foundations for NDMP.
The NDMP Snapmirror Management Extension Interfaces leverage the extensibility of NDMP v4 to define a mechanism and protocol for controlling the configuration and management of Snapmirror ™. Snapmirror is a Network Appliance proprietary system for data movement between two disk based storage systems.
Copyright
Copyright (C) The Internet Society (2002).  All Rights Reserved.

Table of Contents

1. Overview
1.1 Definition of Terms
1.4. Key Words
2. Snapmirror
2.1. Snapmirror Operational Overview
2.1.1. Sources and Destinations
2.1.2. The Snapmirror Data Set
2.1.3. The Snapmirror Relationship
2.1.3.1. The Snapmirror Configuration file
2.1.3.2. Asynchronous Snapmirror
2.1.3.3. Synchronous Snapmirror
2.1.4 Snapmirror User Modes
2.1.4.1. Scripted Snapmirror Operation
2.1.4.2. Scheduled Snapmirror Operation
2.1.5. Snapmirror Configuration
2.1.6. The Snapmirror Schedule
2.1.7. The Snapmirror Synchronous Configuration
2.1.8. Snapmirror Configuration: Default Values
2.1.9. Snapmirror Configuration Examples
2.1.10. Snapmirror and Snapshots
2.2. Operations on Snapmirror Relationships
2.2.1. Quiescing the Mirror
2.2.2. Breaking the Mirror
2.2.3. Restarting a Broken Mirror
2.2.4. Migrating a Data Set
2.2.5. Reissuing Transfers
2.2.6. Snapmirror and Snapshots
2.2.7. Snapmirror Lag Time
2.2.8. Snapmirror and Tape
2.3. Managing System Name and Addresses
2.4. Retrying Data Transfers
2.5. Data Sets and their Granularity
2.5.1. Data Set Location and Transfer Granularity
2.5.1.1. Location Granularities:
2.5.2.2. Recovery location granularities:
2.5.2.3. Transfer granularities:
2.6. Configuring Snapmirror
2.6.1. Snapmirror system configuration
2.6.1.1. Access control
2.6.1.2. Snapmirror Relationship configuration
2.6.1.3. Snapmirror Relationship Initiation
2.6.2. Snapmirror Address and Connection Management
2.7. Data ONTAP Issues
2.7.1. Transfer Limits
2.7.2. Blocking vs. non-blocking commands:
2.7.3. ONTAP Naming Rules
2.7.4. Aborting Transfers
2.7.5. NDMP and Snapmirror Legacy Configuration
2.7.6. Snapmirror and NDMP
2.7.7. Sequencing of Non-Blocking messages
2.8. Snapmirror Extension Interface Messages
2.8.1. Messages Numbers
2.8.2. Error Codes
2.8.3. Snapmirror Replies and notifications
2.8.4. Snapmirror Enums
3. Data Server Interfaces
3.1. The Snapmirror Interface
3.1.1. NDMP_SM_DEFINE_RELATIONSHIP
3.1.2. NDMP_SM_DELETE_RELATIONSHIP
3.1.3. NDMP_SM_START_RELATIONSHIP
3.1.4. NDMP_SM_TRANSFER_UPDATE
3.1.5. NDMP_SM_ABORT_TRANSFER
3.1.6. NDMP_SM_QUIESCE_RELATIONSHIP
3.1.7. NDMP_SM_RESUME_RELATIONSHIP
3.1.8. NDMP_SM_BREAK_RELATIONSHIP
3.1.9. NDMP_SM_RESTART_RELATIONSHIP
3.1.10. NDMP_SM_RELEASE_RELATIONSHIP
3.1.11. NDMP_SM_MIGRATE_RELATIONSHIP
3.1.12. NDMP_SM_DEFINE_CONNECTION
3.1.13. NDMP_SM_DELETE_CONNECTION
3.1.14. NDMP_SM_DESTINATIONS
3.1.15. NDMP_SM_GET_STATUS
3.1.16. NDMP_SM_GET_RELATIONSHIP_STATUS
3.1.17. NDMP_SM_GET_ACCESS_INFO
3.1.18. NDMP_SM_SET_ACCESS_INFO
3.1.19. NDMP_SM_GET_INFO
4. Data Management Application Interfaces
4.1. Destination Notify Interface.
4.1.1. NDMP_SM_NOTIFY_START_RELATIONSHIP
4.1.2. NDMP_SM_NOTIFY_TRANSFER_UPDATE
4.1.3. NDMP_SM_NOTIFY_ABORT_TRANSFER
4.1.4. NDMP_SM_NOTIFY_QUIESCE_RELATIONSHIP
4.1.5. NDMP_SM_NOTIFY_RESTART_RELATIONSHIP
4.1.6. NDMP_SM_NOTIFY_MIGRATE_RELATIONSHIP
5. Security
6. Recognition of Prior Work
7. References
8. Authors and Contributors
8.1. Document Editor
8.2. Authors' Addresses
8.3. Contributors
Appendix A: Specific Data ONTAP Properties and Issues
Appendix B: XDR definitions:
Appendix C: Snapmirror: NDMP API vs. ONTAP CLI mapping:

1. Overview
The purpose of the NDMP v4 Snapmirror management extension interfaces are to define a mechanism and protocol for creating and managing “Snapmirror relationships” between source and destination disk based storage systems using Network Appliance’s Snapmirror ™.
1.1 Definition of Terms
The following terms are specific to NTAP technologies and products, and key to understanding the operation of Snapmirror:

Data ONTAP
The operating system and system software controlling a Network Appliance storage appliance.

Aggregate
In ONTAP release 7.0 the aggregate is the physical storage container in Data ONTAP. It contains one or more volumes.
An aggregate is composed of one or more RAID groups, each consisting of two or more disks. An aggregate is specified by “aggr:aggr_name”.
Notice that there is no physical connection between an aggregate and a volume or qtree. They are orthogonal. A volume does not “belong” to an aggregate, but may be moved around between aggregates for various reasons, for instance capacity.
Volume
A volume is the logical storage container in Data ONTAP, similar to a “LUN” or a Unix partition. A volume can contain a file system with zero or more qtrees, or a set of zero or more LUNs.
In NDMP extensions in Data ONTAP volume names starts with “vol:”. A volume named “my_volume” is specified as “vol:my_volume”.
Qtree
We recommend that volumes are used as the primary data set granularity for data management. Support for qtree operations will be reduced over time. 
A qtree is a top level directory tree in a Data ONTAP volume with a set of unique DATA ONTAP system attributes. Two of these attributes are a unique qtree identifier (qid) within the volume which enables qtree specific snapmirror operations, and a quota which limits the amount of data to be stored in the qtree.
The qtree directory name itself can be up to 64 bytes long, and must NOT include the characters “.”, “*”, or “->”.
In NDMP extensions in Data ONTAP qtree names starts with “qtree:”. A qtree directory “my_qtree”, in a volume named “my_volume” is specified as “qtree:my_volume/my_qtree”.
Snapshot
A snapshot is a Data ONTAP system function that allows users to capture “point-in-time” images of volumes and qtrees particularly quickly and efficiently. The Data ONTAP snapshot function creates a point in time copy which is embedded in the production file system.
A “snapshot” is also frequently used as the name for a “point-in-time” image in Network Appliance’s appliances.
Snapmirror
Snapmirror (SM) is a Data ONTAP system primitive for moving data between appliances. Snapmirror is based on snapshots. It is typically used to maintain asynchronous replicas of volumes (volume snapmirror, VSM) or qtrees (qtree snapmirror, QSM) on two Data ONTAP storage appliances.
Relationship
A persistent configuration which defines the source and destination, or source and destination parties of a snapmirror based data transfer, including the schedule for transfers, and optional bandwidth throttle and retry counts in case of failures.
Source
A Snapmirror source is the data set to be protected, or mirrored, by using snapmirror to maintain replica.
Destination
The snapmirror replica data set. Typically the replica data set is located on a different system from the source, but snapmirror can also be used to maintain replicas within the same system.
Initialize
“Initialize” is the first initializing data movement across a relationship. All data in the source aggregate, volume or qtree is moved to the destination.
Update transfer
A “post initialize” data transfer from source to destination which utilizes snapshots and snapmirror to only move the disk blocks that has been modified since the last transfer, restart, or relationship start operation.
CLI
Command Line Interface. The native Unix style interface for a Network Appliance system.
Resync
The “resync” operation reestablishes a relationship that has been broken, for instance after a source had failed and the destination had been accessed by applications.
1.4. Key Words
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

2. Snapmirror
This section gives an introduction to the Snapmirror capabilities in Data ONTAP. The objective is to describe the basic operation of Snapmirror, and to be able to use the NDMP extensions documented in this document to manage Snapmirror.
2.1. Snapmirror Operational Overview
The following sections give an overview of the operation of a snapmirror backup system in operation.
Snapmirror is data replication service between a source and a destination system. The source and destination can be the same system. The source system is typically the production system, holding one or more data sets for one or more business applications. Snapmirror is used to create and maintain a replica of the production data set(s) on a second storage system.
One may use Snapmirror to maintain more than one replica, in a parallel (star) or cascaded configuration.
Snapmirror is destination (“pull”) driven. This means that the relationship, incl. the schedule, is configured on the destination system, and that it is the destination system which connects to the source to request an update.
Snapmirror goes through two phases of operation. First a baseline replica is created by performing an initializing data transfer, using the NDMP_SM_START_RELATIONSHIP request. From this point the replica can be maintained synchronously or asynchronously. Asynchronous transfers are started by the scheduler, or with the NDMP_SM_TRANSFER_UPDATE request.
2.1.1. Sources and Destinations
Snapmirror is used to maintain a replica of a “source” data set. We call the replica a snapmirror destination. 
A source or a destination is specified by three components: the system name, the type of data set to be replicated, and the name or path of the data set. The first component is the source or destination system name. The remaining two components are concatenated into a single string, separated by colon (“:”).
The filer name is specified either with the symbolic system name with or without the local DNS domain (“toaster.acme.com”, “toaster”), or as the IP address of the filer (“10.56.19.148”). For management purposes it is recommended that one use symbolic names.
 The data set type can be one of the three values “aggr” (aggregate), “vol” (volume), “qtree”. These are supported in ONTAP 7.0. In addition we define and reserve “dir” (directory), “tree” (directory with recursion), and “file” for future use.
Notice that aggregate is NOT supported in ONTAP 6.5.
The data set name is ‘aggr_name’ or ‘vol_name’ for aggregates or volumes, or ‘vol_name’ + ”/” + ’qtree_name’ for qtrees.
Example source or destination names:
“aggr:aggr_01” : The aggregate with name “aggr_01”.
“vol:message_store” : The volume named “message_store”.
“qtree:vol10/home” : The qtree “home” in the volume “vol10”.
2.1.2. The Snapmirror Data Set
For ONTAP release 6.5, the first version supporting this extension, Snapmirror supports replication of volumes and qtrees. In ONTAP release 7.0 the aggregate and volume will be separated, and Snapmirror will support three different data sets: aggregates, volumes or qtrees.
For more information regarding the legacy issues between old “aggregate/physical volumes” in 6.5 and new separated aggregates and volumes in 7.0, consult separate ONTAP documentation regarding snapmirror between releases 6.4, 6.5 and 7.0.
In the following we will describe the protocol for all three data types, however, keep in mind that aggregates are only supported for 7.0 and beyond. If one attempts to specify an operation on and aggregate in 6.5, one will receive the error NDMP_ILLEGAL_ARGS. 
In 6.5 and before, “volumes” was both a physical container, i.e. a number of RAID groups banded together to form a physical container of contiguous disk blocks, and the logical container inside which we create file systems, allocate LUN’s etc. 
In ONTAP 7.0 the old volume is split into two independent entities: the “aggregate”, a group of RAID groups forming the physical block storage container, and the volume, a arbitrary size contiguous sequence of blocks to be used as a file system, a LUN, etc.
Aggregates and volumes have separate name spaces, there may therefore be an aggregate and a volume with the same name in a filer. 
A qtree is a top level directory in a volume.
When replicating an aggregate, all volumes in that aggregate are updated.
When replicating a volume, only the data inside the volume is updated. This includes all qtrees in the volumes.
When replicating a qtree only the data in the qtree is updated. 
Starting with ONTAP 7.0 we encourage users to use many smaller volumes and volume level replicas instead of large volumes with qtrees that require separate qtree replicas.
2.1.3. The Snapmirror Relationship
A snapmirror relationship describes a replication relationship between a source data set and a replica. The source data set, an aggregate, a volume or a qtree, is typically a data set hosting an application, or a shared file system. The replica is maintained for disaster recovery, data protection or performance or load balancing purposes, for instance for bringing a replica closer to a set of users in a remote location. 
Example here !!!!!
Snapmirror relationships are defined implicitly. That is, the relationship does NOT depend upon information stored any other place than simply the meta information that is part of the aggregate, volume or the qtree. 
When the START_RELATIONSHIP command is issued and completes, the result is that the destination has also acquired data set specific meta data. This meta data is used on successive update, migrate or relationship restart (resync) transfers to validate that the data from the source is from the data set that the destination is a replica of. Therefore relationships can be updated on demand without the relationship existing in the conf file or system registry.
The only requirements that must exist on order to perform an update is i) that at least one initializing transfer has been performed, and ii) that the source and the destination must share a snapshot. The most recent snapshot that is shared between source and destination will be used as the base for the requested update. 
One example illustrates the implicit relationship definition clearly. Assume a filer FA with a volume Va. Assume two other filers FB and FC. We go through the following steps:
First we create a snapmirror relationship from FA to FA, mirroring Va to Vab.
After a while we quiesce and break the mirror. Shared snapshots between Va and Vab are S0, S1 and S2.
Then we create a snapmirror relationship from FA to FC, mirroring Va to Vac.
After a while we quiesce and break this mirror. Shared snapshots between Va and Vac are S0, S1, S2, S3 and S4.
The we restart (!!) an “implicit” snapmirror relationship between Vac on FC to Vab on FB.
This last step is a legal operation since Vac and Vab have the same origin, and since there is a shared snapshot S2 in both volumes. The restart operation will now update Vab with the changes to Vac since S2, this implies up to S4. However, as a result of this update the changes that was done to Vab after the relationship with Va was broken, will be lost.
2.1.3.1. The Snapmirror Configuration file
The relationship entries that are added to or deleted from the snapmirror.conf file using DEFINE_RELATIONSHIP or DELETE_RELATIONSHIP contains all the relationship information. However, the relationship does NOT depend upon the conf file. See the previous section.
The primary objective for the conf file is twofold. First is provides the location for the snapmirror schedule for a relationship. Only relationships specified in the conf file can perform scheduled or synchronous updates.
Secondly, the conf file provides convenience. The START_RELATIONSHIP or TRANSFER_UPDATE requests may be issued with only the destination path specified. The destination path is the primary identifier of the relationship. The rest of the relationship information, including transfer options, are taken from the conf file. 
The snapmirror service will poll the snapmirror.conf file every two minutes, therefore changes in the conf file will take effect within two minutes.
Notice that with the introduction of aggregates in ONTAP 7.0, there will be a new item in the relationship entry in snapmirror.conf. This is the “type” identifier which will be “aggr” for aggregates, or “vol” for volumes and qtrees.
2.1.3.2. Asynchronous Snapmirror
In asynchronous mode the snapmirror replica is updated at scheduled times, or on demand by a remote party. When a snapmirror relationship is defined, the provided schedule specifies when updates should be performed. These updates are then activated by the scheduler system in the destination system.
2.1.3.3. Synchronous Snapmirror
An asynchronous mode relationship can be configured into a synchronous mode relationship by setting the “outstanding=’sync_val’”  and “visibility_interval=’snap_freq’” parameters in the options section in the relationship entry in the conf file, AND by changing the schedule field to “sync”. 
The relationship can operate in two modes:
Synchronous: 
Every write to the file system is replicated to the destination before the write is acknowledged to the application.
Semisync: 
The destination may ONLY be up to N file system write operations or N milliseconds behind the source.
2.1.4 Snapmirror User Modes
Snapmirror is typically operated in two mutually exclusive “user modes”. These modes are “scripted mode” and “scheduled mode”.
In scripted mode the control of snapmirror, including the schedule for when transfers should be performed, are directed from a data management application. This data management application, or a script on a remote system allows a user to configure or implement specific replication, protection or recovery strategies, this configuration is used to determine which relationships to start and when to perform updates, when to break or restart the relationship, etc.
In scheduled mode the DMA or the user directly takes advantage of the built in scheduler in ONTAP. Relationships are configured in the snapmirror.conf file. Each relationship entry contains a schedule that is used by snapmirror to configure scheduled updates. The schedule can be configured to update on a granularity of weeks, days, hours or minutes, or to perform synchronous or semi synchronous updates.
2.1.4.1. Scripted Snapmirror Operation
In scripted mode the typical scenario is that a third process, a data management agent (DMA), is coordinating the application, its production storage which is a snapmirror source, and the snapmirror destination agent for the purpose of creating and transferring an application consistent snapshot. Typically the steps are:
1: The DMA puts the application in “hot backup mode”, meaning that the application flushes all its internal buffers.
2: The DMA flushes file system buffers, driver buffers etc.
3: The DMA takes an application specific snapshot Sa on the source filer.
4: The DMA initiates a snapmirror transfer from the source to the destination, using Sa as the source snapshot, and creates a snapshot Da on the destination filer.
In this class of application there is typically no scheduled relationship, and often no entry for this source and destination in snapmirror.conf. Every transfer operation is a “one-off” request, the application typically issues TRANSFER_UPDATE requests to maintain replicas. The control of the replication process resides outside of the snapmirror destination system.
Notice that there is still an implicit relationship between the source and the destination, in that the source and destination agents have the necessary information to validate the connecting party, and the incremental data that is transferred.
2.1.4.2. Scheduled Snapmirror Operation
In scheduled mode the regular operation is controlled by the content of the snapmirror.conf file which resides in the /etc directory in the root volume on the destination filer.
Each line in this file represents a snapmirror relationship. The relationship is specified with:
Destination: The filer name/address and the aggregate, volume or qtree name on the destination.
Source: The filer name/address and the aggregate, volume or qtree on that filer.
Type: The type of data set: aggregate, volume or qtree.
Throttle: The upper bound on the data transfer rate from source to destination. Measured in kilobytes/s.
Restart-ability: A Boolean signaling whether a restart point should be maintained in case of aborted transfers.
Schedule: For asynchronous operation this includes a set of strings specifying the times of monthly/weekly/daily/hourly/minutely updates between source and destination. For (semi-) synchronous operation this includes the allowed outstanding write requests between source and destination, and the frequency of snapshots on the destination. See separate section on Snapmirror scheduling.
2.1.5. Snapmirror Configuration
In order to configure scheduled or (semi-) synchronous updates of a relationship one MUST use the NDMP_SM_CREATE_RELATIONSHIP command to create an entry in the snapmirror.conf file. The last two items in this command, and in the NDMP_SM_MODIFY_RELATIONSHIP command, is a schedule structure and a synchronization structure. These elements define automatic behavior of snapmirror.
When the schedule activates a transfer, snapmirror will first take a snapshot on the source. This snapshot name will be generated by the system, and will have “snapmirror” as part of its name.
Then the transfer will start, using the most recent shared snapshot between source and destination as the base. Updates between this base snapshot and the new snapshot on the source will be transferred. Once the transfer is complete a new snapshot will be taken on the destination. For this relationship the new source and destination snapshot becomes the base for the next update transfer.
The automatic operation of Snapmirror can be disabled, the transfers must then be carried out manually. For more information on this type of configuration and use, see section 2.1.2.1., “Scripted Mode”.
2.1.6. The Snapmirror Schedule
The Snapmirror destination schedule is a data structure that governs all asynchronous automatic snapmirror operations for a volume on a destination. Asynchronous or synchronous operation is controlled by the <syncmode> parameter in the DEFINE/MODIFY_RELATIONSHIP requests. The value “false” signifies asynchronous operation.
The schedule specifies when data transfers should occur and backups should be captured. The schedule is processed by Data ONTAP every minute. When a scheduled event becomes current, the scheduler will start up the corresponding transfers according to the parameters specified in the schedule item.
The XDR structure <ndmp_sm_snap_sched_item> is used in the protocol extension to set and get Snapmirror relationship schedule items on the destination:
struct ndmp_sm_sched_item
{
      string    monthdays<>;
      string    weekdays<>;
      string    hours<>;
      string    minutes<>;
};

The meaning and syntax for the structure elements:
monthdays
The day of the months when the snapmirror transfers are carried out. The days are specified by the numbers, 1 – 31. “*” means “not applicable”. The schedule will be controlled by the weekly schedule.
“1,3,5,. . .,27,29,31”: Every other day.
weekdays
The weekdays when the snapmirror transfers are carried out. The days are specified by three letter abbreviations or by week day numbers: “Sun” (=”0”), “Mon” (=”1”), “Tue” (=”2”), “Wed”, “Thu”, “Fri”, “Sat” (=”6”). The options for syntax are:
“Mon-Fri”: Starting on Monday, ending on Friday, including both.
“1,3,5”: Monday, Wednesday and Friday only.
“*” (asterisk): Every day.
hours
The hours when the destination will contact the source for a Snapmirror transfer. The time is specified in local time, 24 hour format:
“9,21”: 9 AM and 9 PM local time. 
“7-17”: Every hour between 7AM and 5PM, inclusive. 
“” (empty string): Used when the minute value is “sync”, for a synchronous relationship.
minutes
The minute points when the destination will contact the source for a Snapmirror transfer. The time is specified in minutes:
“15,45”: every 15 and 45 minutes past the hour on the hours and days defined on the other schedule items. 
“sync”: This mirror is synchronous. The other time point values MUST be “”, i.e. an empty string.
The value “-“ means “not applicable”. If all schedule strings are “-“, not transfers will be scheduled. Transfers will have to be started but commands only.
For weekdays, hours and minutes, the value “*” means “start transfers as frequently as possible”. On every minute when the relationship is not already transferring, a new transfer will be started.
2.1.7. The Snapmirror Synchronous Configuration
The Snapmirror synchronous operation is configured using the structure ndmp_sm_sync_item. It is used in the DEFINE/MODIFY_RELATIONSHIP requests. 
Asynchronous or synchronous operation is controlled by the <syncmode> parameter in the DEFINE/MODIFY_RELATIONSHIP requests. The value “true” signifies synchronous operation. In this case the structure ndmp_sm_sync_item defines the synchronous or semi-synchronous behaviour of snapmirror.

struct ndmp_sm_sync_item
{
      long          outstanding;
      sm_sync_type  type;
      long          vis_interval;
};

outstanding
The value controlling the semi-synchronous behavior of snapmirror. The value “0” implies fully synchronous operation. 
type
The unit of measure used for controlling semi synchronous operation of snapmirror. Allowed measurements are:
File system write operations,(IOPS),
Milliseconds,
Notice that if the type is milliseconds, and the value is greater than 10 seconds, i.e. 10,000 milliseconds, we will only transfer consistency points. This will be the first consistency point following after the specified interval. In this case the specified semi-synchronous delay is an average over time, and may occasionally be a longer or shorter than the specified interval.
vis_interval
Specifies how frequently snapmirror will take snapshots on the destination in order to make the mirrored data visible on the destination. The value is specified in milliseconds. The default value is 5 minutes, i.e. 18,000 milliseconds.
2.1.8. Snapmirror Configuration: Default Values
A snapmirror relationship may drop back into asynchronous mode, depending upon various error conditions. Therefore the schedule SHOULD always be configured, and it should be configured for scheduled updates at a relatively high frequency.
For relationships that will only be operating in asynchronous mode, the following values should be used in the sync item:
Outstanding:  0.
Sm_sync_type: SM_SYNC_TYPE_NONE.
Viz_interval: 0.

Notice also that when reading relationship status for a relationship that was configured via the CLI, for asynchronous operation only, the returned information from a GET_RELATIONSHIP_STATUS will contain “sync mode” = false, and the above sync item.
2.1.9. Snapmirror Configuration Examples 
The following give some examples of entries, including the schedule, in the snapmirror conf files. Notice that there is a difference between how data sets are specified in the conf file and in the protocol.
toaster:/vol/vol1/home fridge:/vol/vol2/home kbs=2000 * * * *
This entry indicates that filer fridge's qtree home, in volume vol2 will mirror qtree home, in volume vol1 from the filer toaster. Transfer speed is set at a maximum rate of 2,000 kilobytes per second. The four asterisks mean transfers to the mirror are initiated every minute, if possible. (If a previous transfer is in progress at the minute edge, it will continue; a new transfer will be initiated at the first minute edge after the transfer has completed.)
netapp0-gig:home netapp1:home_mirror - 30 9,13,19 * 1,2,3,4,5
This entry indicates that filer netapp1's volume home_mirror will mirror volume home via the netapp0-gig interface. (The netapp0-gig interface is whatever IP address netapp1 can resolve that name to. In this case, it might be a gigabit Ethernet link on filer netapp0.) The mirror is updated at 9:30 a.m., 1:30 p.m., and 7:30 p.m., Monday through Friday. The asterisk means that the data replication schedule is not affected by the day of month; it is the same as entering numbers 1 through 31 (comma-separated) in that space. The dash in the arguments field indicates that both the kbs and restart arguments are set to default.
filer1:build filer2:backup - 15,45 8,9,10,11,12,13,14,15,16,17,18 * *
This entry makes transfers every half hour, with the first at 8:15 a.m., and the last at 6:45 p.m. The asterisks mean that the data replication schedule is not affected by the day of month or week; in other words, this series of transfers is initiated every day.
dev:/vol/dept/docs icebox:/vol/backup/docs_bak - 0 0 * 0
This entry, between the docs qtree on dev and docs_bak on icebox, is kicked off on every Sunday, at 12:00 midnight.
dev:/vol/dept/docs icebox:/vol/backup/docs_bak - - - - -
This entry, between the docs qtree on dev and docs_bak on icebox, is never kicked off, transfers will only happen when issued by update commands.
icebox:home icebox:backup - 30 0,7-21 * *
This entry, between the home and backup volumes on icebox, is kicked off once every half past the hour between 7:30 a.m. and 9:30 p.m., and once at midnight.
icebox:home snowcan:sync – outstanding=3ops visibility_interval=60 sync * *
This entry, between the home volume on icebox and the sync volume on snowcan, is kept synchronous with a delay of up to 3 file system write operations. Destination snapshots are taken every 60 seconds.
fridge-gig:db icebox:db - 0-55/5 * * *
This entry, between the db volumes on fridge-gig dev and icebox, is kicked off on every five minutes, starting at 0. (Note that fridge-gig is just a network interface name. In this case, it could be a gigabit Ethernet link on fridge.)
2.1.10. Snapmirror and Snapshots
Snapmirror uses ONTAP snapshots internally to synchronize between a source and a destination. For instance, when a relationship is STARTED, snapmirror creates a snapshot Ss(N) on the source, transfers the data, and creates a snapshot Sd(N) on the destination. These snapshots are named specifically by snapmirror and includes the filer name, the string “snapmirror”, a time stamp etc.
Likewise when doing an UPDATE, snapmirror creates a new snapshot Ss(N+1) on the source, replicates the delta between Ss(N) and Ss(N+1) to the destination, and then creates snapshot Sd(N+1) on the destination.
One may also specify in the UPDATE command the name of the snapshot on the source to be used for the transfer, or also the name of the snapshot to be created on the destination.
Notice that two different syntaxes exist for snapshots in NDMP extensions. The Snapshot Management interface (SnapCmd) currently uses a syntax of “/vol/’volume_name’/.’snapshot_name’”. An example is “/vol/v_exchange/.nightly.0”. 
In this extension, and also in the SnapVault extension, the syntax is to only use the snapshot name itself. For the above example, specifying the same snapshot in this or the SnapVault extension, one must specify “nightly.0”.
2.2. Operations on Snapmirror Relationships
Once a relationship has been started, and the initializing transfer has completed, several operations can be performed on the relationship. 
2.2.1. Quiescing the Mirror
If no more updates are desired, either because one wants to break the mirror and make the replica writeable, or because the source ahas failed and needs to be repaired or replaced, the relationship should be quiesced.
This operation prevents new transfer operations for the relationship to be started.
If a snapmirror transfer is already in progress this operation will not complete until the transfer has completed. 
2.2.2. Breaking the Mirror
If a source system fails, or if a snapmirror relationship is used to migrate data over to a new production system, the destination is made writable by “breaking the mirror”. After the “break” operation the application that used to store its data on the source can now use the (previous) destination as a temporary or permanent production storage system.
The break command must be preceded by the quiesce command.
If one wants to do a transparent and permanent migration of a data set, one should consider the migrate command.
2.2.3. Restarting a Broken Mirror
If a relationship was broken due to a failed source system, or simply because one wanted to pause the updates, the mirror can be restarted using the NDMP_SM_RESTART_RELATIONSHIP command. 
2.2.4. Migrating a Data Set
One transfer command has a particular application. It is the NDMP_SM_MIGRATE_RELATIONSHIP command. It is used to migrate a data set permanently from one storage system to another. The reason may be that the current system does no have sufficient performance or capacity, or a new storage system has been installed and IT wants to distribute the load more evenly.
Note that the migrate command is only one step in a complete and potentially transparent migration process.
Migrate assumes that a relationship exists between source and destination, and that at least one transfer has been performed from source to destination.
The migrate command will turn off the source file system services and take the source off-line, it will perform a last update transfer from source to destination, and then make the destination writeable. The update includes moving over the same file handles as was provided by the previous source. At the data access level storage clients see a transparent move.
The migrate command should be used in a very controlled setting. When migrate starts, the file system service is stopped. Applications will therefore NOT be able to get to the data until the operation is complete.
The migrate operation, and the corresponding data outage can be shortened by running 1-2 (successive) update transfers right before migrate is run. This will assure that only a minimum of data has changed between the migrate transfer and the previous transfer, thus the transfer done as part of migrate will be short.
One can use the restart command to restart a relationship that was “broken” with migrate. However, notice that in that case the changes to the destination after migrate will be lost!
2.2.5. Reissuing Transfers
Snapmirror will typically take checkpoints at regular intervals during a transfer. Currently the time interval is approx. 5 minutes. These checkpoints are used for restarting transfers if a transfer was aborted.
One can determine if there is a restart checkpoint present as follows. If the status for a relationship is SM_STATUS_IDLE or SM_STATUS_PENDING, AND the value of <bytes_transferred> in NDMP_SM_GET_STATUS is != 0, this means that the previous transfer was aborted, and a restart checkpoint remains.
If a transfer was aborted and a restart checkpoint exisits, one can restart the transfer from this checkpoint simply by reissuing the transfer command that was aborted. The transfer will restart from the most recent restart point.
If one wants to skip the partial restart, and rather start a new transfer from the very beginning, incl. from a new snapshot, one must issue an ABORT_TRANSFER request with <no_restart> = true. This will clear the restart point and assure that the next transfer for the relationship starts from a new snapshot.
2.2.6. Snapmirror and Snapshots
One can only specify snapshots when the transfer data set is qtree. Snapshots can be specified for the START_RELATIONSHIP and TRANSFER_UPDATE requests.
For qtree transfers one can specify the snapshot name on the source system that defines the baseline after the transfer, or the name of the snapshot to be created on the destination after the transfer completes. The source snapshot must have been created before the transfer is requested.
2.2.7. Snapmirror Lag Time
It is important to be aware that the lag time reported in a snapmirror GET_STATUS response is the time since the last transferred snapshot was taken on the source. It is NOT the time since the last transfer started or ended.
For instance, if the snapshot was taken on the source 24 hours ago, but the transfer happened 8 hours ago, the lag will say “24 hours”.
2.2.8. Snapmirror and Tape
[Not supported in version 1 of this protocol.]
Data ONTAP has support for backing up volumes to tape, and restoring tape images to a volume, using the snapmirror data format. There is also support in core NDMPv4 for using snapmirror as a backup method, instead of dump or tar. Therefore we will [possibly???] omit support for snapmirror to tape in this extension.
Snapmirror includes commands for storing a volume image to tape, and for retrieving the image back. See the NDMP_SM_{STORE|RETRIEVE} commands for the specific functions.
One particular optimization feature should be noted. Because the writing of a volume to disk can be optimized significantly if one generates the tape image using the geometry of the destination volume, the NDMP_SM_GET_VOLUME_METADATA provides a way of getting this information for a volume, and then using this metadata to generate the tape image during the NDMP_SM_STORE operation. This tape image can be written to disk during NDMP_SM_RETRIEVE with much higher throughput.
2.3. Managing System Name and Addresses
[This is incomplete, is only used for synchronous snapmirror relationships, and will most likely not be available in Ancor Steam.]
When defining or starting snapmirror relationships, DMA’s MUST specify one source system name, and zero or more source addresses. The system name is for user interface purposes, and allows users do use a symbolic name, such as “toaster”, when managing snapmirror relationships. Note that the system name can also be specified as an IP address, such as “168.192.45.56”.
If no separate address(es) is specified the system name is used as the address. If the system name is specified as an IP address it is used directly. Otherwise the system name os combined with the domain suffix, such as system name “toaster” plus domain name “acme.com” used to form “toaster.acme.com”, which is looked up via DNS, NIS etc. to get the IP address.
The maximum number of addresses in the address list is implementation dependent. A suggested minimum is 2. The addresses can be symbolic addresses (“toaster-e0”), or hard IP addresses (“192.168.45.56”). 
The addresses can be used in multiple modes. If only one address is specified, this address is used as default. If a (re-) connection can not be made using this address, the address generated from the system name is used.
If two or more addresses are specified, these addresses can be used in multi-path or fail-over mode: 
In multi-path mode the data transfer is attempted using two or more physical connections generated from two or more addresses.
In fail-over mode the first address is the default, if the connection cannot be made the second address will be tried. And so on.
2.4. Retrying Data Transfers
Transfers can be prevented or aborted for many reasons, including source or destination system failure, network partitioning, user error, etc. Prevented or aborted transfers are treated differently for scripted (i.e. direct) or scheduled transfers.
When the DMA requests a transfer, i.e. the user mode is “scripted”, emphasis is put on responding quickly to the DMA, and allowing it to decide what retry strategy to pursue. In this case the transfer is tried once for each specified address in the specified address list. If the destination can’t perform the transfer with the specified addresses, it will report an error back to the DMA. The DMA can then decide to retry, or take a different action.
When a transfer is requested from a scheduled relationship, as specified in snapmirror.conf, and the destination agent cannot make a connection to a source using all the specified addresses, the connection will be retried one minute later. Failed connection/transfer attempts will be logged to the snapmirror log file.
2.5. Data Sets and their Granularity
Snapmirror currently supports mirroring of volumes and qtrees, and will in ONTAP release 7.0 also support aggregates.
2.5.1. Data Set Location and Transfer Granularity
The following actions define the two aspects of data set granularities that are supported in snapmirror: location granularity specifying the data sets to be defined in a relationship, and transfer granularity which defines if and what subset of the relationship can be updated.
Which granularities are supported by a storage system is implementation dependent.
2.5.1.1. Location Granularities:
Depending upon the underlying operating system or file system, implementations of source agents for this protocol may choose to allow for replicating data sets of different size, corresponding to different levels in the file/storage system. For instance, a source agent may not allow for replication of a full drive or (logical) volume, but only for a directory, incl. a top level directory.
Likewise, a destination agent implementation may require that all backup data sets are stored under top level directories of a LUN or (logical) volume. Other agents may allow for these data sets to be stored as general directories, or as separate volumes.
The location granularities that specifies the locations of source data sets or destination data sets that are replicas of the source data sets are:
Aggregate: 
An ONTAP aggregate, i.e. a collection of RAID groups used as a continuous storage pool for volumes. This is the highest level storage container on the system.
Volume: 
An ONTAP volume. This is the highest level storage object in ONTAP to be used by applications to store data.
Top-directory: 
A top level directory in a container defined above. For instance /tmp in a Unix partition, or the \Temp directory on the D: drive in a Windows based system.
Directory: 
An arbitrary directory in a source or destination’s file system.
File: 
An arbitrary file in a source’s file system. One example is a composite file with several streams.
Notice that separate location granularity parameters are needed for the source and destination system. 
Notice also that the location granularities on the source and destination does not need to match. For instance, a source data set may be a 3rd level directory, but it’s replica data set may be stored as a top level directory on the destination.
2.5.2.2. Recovery location granularities:
Separate from the granularity of the source and replica data set is the set of locations that a recovered data set can be written into. A source agent may support data recoveries to one or more of the following locations:
Aggregate: 
An ONTAP aggregate, i.e. a collection of RAID groups used as a continuous storage pool for volumes. This is the highest level storage container on the system.
Volume: 
An ONTAP volume. This is the highest level storage object in ONTAP to be used by applications to store data.
Top-directory: 
A top level directory in the volume. For instance D:\Temp on a Windows system.
Directory: 
An arbitrary directory in a source’s file system.
File: 
An arbitrary file in a source’s file system. One example is a composite file with several streams.
Notice that this location granularity ONLY refers to the allowed locations of the recovered data set. It is independent of whether this data set is the complete data set according to the relationship, or only a subset thereof, and also independent of the original location of this data set on the source or destination. 
2.5.2.3. Transfer granularities:
Depending upon the software platform and on design decisions, agents may allow for multiple levels of backup or recovery within one relationship. For instance, a source agent may allow for file and arbitrary sub-directory incremental backup transfers in addition to the incremental transfer of the full data set specified in the relationship. Additionally, it may only allow recovery of the full data set and individual files.
A destination may support a different subset of (incoming) backup or (outgoing) recovery transfer granularities.
The granularities that are defined for incremental and recovery transfers are:
Full: 
The agent can handle updates or recoveries of the full data set, i.e. all data under the root paths defined in the relationships.
Volume: 
An ONTAP volume. This is the highest level storage object in ONTAP to be used by applications to store data.
Top-directory: 
A top level directory in the volume. For instance D:\Temp on a Windows system.
Directory tree: 
The agent can handle updates or recoveries of an arbitrary directory tree under the relationship paths.
Directory: 
The agent can handle updates or recoveries of an arbitrary directory under the relationship paths. Notice that subdirectories under the specified directory are not included in the transfer.
File: 
The agent can handle updates or recoveries of single files in the data set.
Notice that in order for an incremental transfer to succeed, the requested data set must be one that is supported by both the source and destination agent.
2.6. Configuring Snapmirror
There are three steps to configuring Snapmirror: system configuration, relationship configuration, and schedule configuration.
2.6.1. Snapmirror system configuration
The first configuration step is basic system configuration: licensing, enabling, and setting access permissions. Licensing and enabling is carried out by a system admin on the source or destination system using native interfaces. How this is done is not in the scope of this specification. However, a Snapmirror source or destination must be correctly licensed and enabled in order for a DMA to use the Snapmirror capability.
Notice that the Snapmirror destination and source are separately licensed and require separate snapmirror destination and snapmirror source licenses.
2.6.1.1. Access control
Access control to Data ONTAP appliances is typically managed on a per protocol basis via the “options” command in the Data ONTAP CLI.
Two aspects of access control is important for the purpose of using the Snapmirror extensions. These are the access control mechanism for NDMP and Snapmirror respectively.
In order for a DMA to connect to a Data ONTAP appliance, the ndmpd.access option must be set correctly. This can only be accomplished via the CLI, by a system admin. This option controls which remote systems may connect to the appliance using the NDMP protocol only. See below for how to configure the option.
The option snapmirror.access controls which remote systems are allowed to request data from a Snapmirror based source or destination storage system.
To give the Snapmirror destination permission to transfer data from a set of ONTAP primaries, one must set the snapmirror.access option on the primaries. To give primaries permission to restore data from the destination, one must set the snapmirror.access option on the destination.
The option is specified by the snapmirror.options string. The syntax of the string is:
access_string = ‘access_spec’ [ [ “AND” | “OR” ]
                               ‘access_string’ ]

access_spec   = “host” [ “=” |”!=” ] host spec  OR 
              “if” [ “=” | ”!=” ] network interface spec OR
              “all” OR
              “none” OR
              “*” OR
              “-“

Host spec is a comma separated list consisting of either a host name, an IP address, or an IP address with a netmask. Valid host name is a string and cannot contain the following characters: "=", "(", ")", "!", "*", and ",". An IP address is of the format aa.bb.cc.dd . If the IP address contains a netmask, then the format is: aa.bb.cc.dd/mm where mm represents the number of bits from the left.
Network interface spec is a comma separated list of one or more network interface names. Valid network interface names can be obtained from the ONTAP ifconfig -a command. An example is “e0”. 
The access specs may be and'ed and or'ed by the keywords AND and OR respectively. The keywords AND and OR are not case-sensitive.
Operational precedence is from left to right. Parentheses may be used to force operational order.
The keyword “all” is used to allow access to all. The keyword “none” is used to allow access to none.
Sample access specification strings are:
"host = gnesha.zo" – allow only host gensha.zo.
"host=abc,xyz AND if=e0" – allow only hosts “abc” and “xyz”, and ONLY over interface “e0”.
2.6.1.2. Snapmirror Relationship configuration
The second step is to configure which data sets on the primaries are backed up to which locations on the destination. This configuration is done on the Snapmirror destination.
The NDMP_SM_CREATE_RELATIONSHIP request configures a snapmirror source-destination relationship entry in the snapmirror.conf file. If the relationship is for scheduled or synchronous or scheduled operation, the relationship MUST be configured in the conf file. If the relationship is for asynchronous transfers only, the configuration entry in the conf file is optional.
2.6.1.3. Snapmirror Relationship Initiation
The first (“initializing”) transfer of data from the specified source to the destination is performed by the NDMP_SM_START_RELATIONSHIP request.
2.6.2. Snapmirror Address and Connection Management
Large enterprises increasingly implement separate (private) networks for separating user or application traffic from data management traffic. The snapmirror extension enables DMA’s or users to configure one or more separate network connections in a multi-path or fail-over configuration. 
Connection configuration is done using the NDMP_SM_DEFINE_CONNECTION message. The request takes a system name, the name of the system with which this system has a snapmirror relationship, and either one connection, or a pair of connections. Each connection is specified by two addresses, the address to use when connecting to the remote system, and the address of the interface to use on this system. The addresses are specified in a list of sm_tcp_addr structures. 
When a single connection entry with connections for the specified system is defined, the first connection is used. Otherwise the system name is used in a DNS lookup to generate the address.
In synchronous mode, if a two connection entry is defined, the connection can be used in one of two modes, as specified in the configuration file. 
In a fail-over mode the first connection in the list will be used as default. If a transfer from the remote system can not be made using this connection, snapmirror will try the other connection.
In a multi-path configuration snapmirror will attempt to use the two connections in parallel to increase bandwidth.
One connection is specified by the two interface addresses and port numbers on the source and destination filers. Within one sm_tcp_addr structure the binary address takes priority over the symbolic address. However, if one wants the addresses to always be current with the enterprise DNS configuration, one should ONLY specify symbolic addresses. This is recommended.
The structure is defined as:
struct sm_tcp_addr
{
      u_long ip_addr;
      u_long port;
      string symaddr<>;
      ndmp_pval sm_pval<>;
};

sm_tcp_addr
A structure containing addresses to be used when connecting to the remote system. The elements are:
ip_addr
A binary encoded IP address.
port
The tcp port number.
symaddr
A string containing the symbolic name of the network interface to connect to. It is recommended to use symbolic name which are resolved through dns at the time of making the connection.
sm_pval
A list of ndmp_pval structures containing application specific information.
2.7. Data ONTAP Issues
The following subsections list some specific issues, limitations and other useful information on the implementation of this protocol in ONTAP.
2.7.1. Transfer Limits
Data ONTAP has limits as to how many snapmirror based data transfers it allows in parallel. The limit depends upon the particular hardware platform. A scheduled transfer to the destination will therefore start on or after the time specified in the schedule.
In Data ONTAP, in NTAP proprietary NDMP extensions all volumes are named “vol_name”. Qtree names and path names, starts with “vol:”. A qtree directory with name “my_qtree” in vol0 is specified as “vol:vol0/my_qtree”.
Currently the only supported transfer granularity on an ONTAP based source is aggregates, volumes and qtrees.
2.7.2. Blocking vs. non-blocking commands:
In ONTAP, each NDMP session will only allow for one single non-blocking command at any time. This command will have to complete, and the notify post sent back to the DMA before the system will allow the start of a second non-blocking command.  If a previous non-blocking command had not finished, the system will respond with an error.
The only exception to this rule is ABORT_TRANSFER. An ABORT_TRANSFER request can be issued on a session where there is currently an active non-blocking command, such as TRANSFER_UPDATE.
Notice that when a transfer operation is aborted using ABORT_TRANSFER, both the non-blocking transfer request, and ABORT_TRANSFER will return notifications.
The success of the abort is returned in the notification from ABORT_TRANSFER. The notification from the aborted command will return some form of failure, often related to a network connection problem. 
2.7.3. ONTAP Naming Rules
In Data ONTAP the following naming rules apply:
In ONTAP path names MUST NOT include a “*”, a “.”, nor a “->”. 
2.7.4. Aborting Transfers
If the NDMP session is aborted while an ABORT_TRANSFER or DELETE_RELATIONSHIP operation issued via that NDMP session is in process, the operation will complete.
If the NDMP session is aborted while a transfer is in progress, the transfer will be aborted, but with a stored checkpoint. If a session is re-established and the transfer command reissued, the transfer will continue from the checkpoint.
In ONTAP, the underlying data replication engine records checkpoints for all transfers. If a START_RELATIONSHIP or RESTART_RELATIONSHIP operation is aborted for external reasons, or by ABORT_TRANSFER with “no_restart” set to FALSE, the operation is restarted from the checkpoint by simply reissuing the original operation.
2.7.5. NDMP and Snapmirror Legacy Configuration
ONTAP release 6.1 and before used a configuration file (/etc/snapmirror.allow) to configure the source side of relationships, i.e. to define which destinations might request data. In ONTAP 6.2 the new protocol access mechanism was implemented.
In this protocol we assume that access control is ONLY managed using the protocol access mechanism controlled via the option “option.snapmirror.access”. See sections 2.6.1.1 and the GET/SET_ACCESS_INFO requests for more information.
Notice that there may be configuration issues if the source system uses the legacy allow file mechanism. However, if this is the case, the access string for the source system will be “legacy”.
2.7.6. Snapmirror and NDMP
The Snapmirror interfaces for management of source and destination storage systems are implemented as part of the NTAP proprietary data service extension, class 0x2050, version 3. This class is an extension to the NTAP data service, for version 4 of the core NDMP protocol.
In order to use this extension, the DMA must connect to the regular NDMP port, typically port 10,000, negotiate version 4 of the core NDMP protocol, and then negotiate the use of version 3 of the 0x2050 extension class.
Snapmirror differs from the normal NDMP behavior in a few very important areas. This section explains these differences. 
First, despite the fact that the Snapmirror destination represents the backup system in the NDMP session, the Snapmirror destination as well as the Snapmirror source is an NDMP data server, and not an NDMP “tape server”. This is due to the fact that the Snapmirror destination “understands” the format of the received data, and the format of the file system repository where these data transfers go.
Secondly, in a typical NDMP session using tar, dump, etc. to produce the backup image, connection management is explicit, and carried out by the DMA in the typical “listen – connect” sequence. In particular, the DMA commands the data or tape services to listen for the data connection, or to make the data connection.
With Snapmirror relationships, aside from the DMA making the control connections to the two data services, management of the data connection is implicit and handled by the Snapmirror destination or source. Both the destination and the source are always in a listen state. When a transfer is requested by the Snapmirror schedule or by the DMA, the destination or source will automatically create a connection to the other party and start the transfer. No explicit data connection management by the DMA is required.
2.7.7. Sequencing of Non-Blocking messages
One NDMP session will only allow for ONE non-blocking command at any given time. This means that until the final notification has come back from a non-blocking request, no further non-blocking command can be issued. 
There is one exception. A transfer started by a non-blocking command can be aborted with an ABORT_TRANSFER command using the same session. However, no more than one non-blocking command aside from ABORT_TRANSFER is allowed.
For instance, if the DMA has issued an NDMP_SM_START_RELATIONSHIP, and the DMA issues an NDMP_SM_TRANSFER_UPDATE request before the DMA has received notification that the START_RELATIONSHIP command completed, the UPDATE_RELATIONSHIP command will return an error: NDMP_SM_BUSY_ERR. This tells the DMA to wait until the notification for the previous non-blocking command (START_RELATIONSHIP) has been received before the next non-blocking command (UPDATE_RELATIONSHIP) is issued. 
2.8. Snapmirror Extension Interface Messages
The NDMP Snapmirror management interfaces are defined in extension class 0x2050 version 2. The source and destination interfaces, and the corresponding Notify interfaces, are assigned message codes in non-overlapping ranges as indicated in the following sections.
Notice that the Network Appliance data service extension class 0x2050, version 1 and 2 exists already and has interfaces for managing snapshots and for testing the extension mechanism. Version 3 of this class will have the two new Snapmirror interfaces, in addition to the existing 0x2050/V2 interfaces.
2.8.1. Messages Numbers
The following messages are defined:

enum ndmp_Snapmirror_message
{
    NDMP_SM_DEFINE_RELATIONSHIP      = 0x20500200,
    NDMP_SM_START_RELATIONSHIP       = 0x20500201,
    NDMP_SM_TRANSFER_UPDATE          = 0x20500203,
    NDMP_SM_ABORT_TRANSFER           = 0x20500204,
    NDMP_SM_QUIESCE_RELATIONSHIP     = 0x20500205,
    NDMP_SM_RESUME_RELATIONSHIP      = 0x20500206,
    NDMP_SM_BREAK_RELATIONSHIP       = 0x20500207,
    NDMP_SM_DELETE_RELATIONSHIP      = 0x20500208,
    NDMP_SM_RESTART_RELATIONSHIP     = 0x20500209,
    NDMP_SM_RELEASE_RELATIONSHIP     = 0x2050020A,
    NDMP_SM_MIGRATE_RELATIONSHIP     = 0x2050020B,
    NDMP_SM_DESTINATIONS             = 0x2050020C,
    NDMP_SM_GET_STATUS               = 0x20500213,
    NDMP_SM_GET_RELATIONSHIP_STATUS  = 0x20500214,
    NDMP_SM_GET_ACCESS_INFO          = 0x20500215,
    NDMP_SM_SET_ACCESS_INFO          = 0x20500216,
    NDMP_SM_GET_INFO                 = 0x20500217,
    NDMP_SM_DEFINE_CONNECTION        = 0x20500219,
    NDMP_SM_DELETE_CONNECTION        = 0x20500221,
};

enum ndmp_snapmirror_notify_message
{
    NDMP_SM_NOTIFY_START_RELATIONSHIP     = 0x20500280,
    NDMP_SM_NOTIFY_TRANSFER_UPDATE        = 0x20500281,
    NDMP_SM_NOTIFY_ABORT_TRANSFER         = 0x20500282,
    NDMP_SM_NOTIFY_QUIESCE_RELATIONSHIP   = 0x20500283,
    NDMP_SM_NOTIFY_RESTART_RELATIONSHIP   = 0x20500284,
    NDMP_SM_NOTIFY_MIGRATE_RELATIONSHIP   = 0x20500287,
};

2.8.2. Error Codes
[Not complete yet.]
All error codes for NDMP Snapmirror management extension interfaces are listed below.  This list is a combination of core NDMP error values and new errors defined specifically for Snapmirror management.  The DMA MUST be implemented to accept any core NDMP error values.
The following new error codes are referenced by the Snapmirror management interface:
enum ndmp_sm_error
{
NDMP_NO_ERR                      0x00000000,
NDMP_PERMISSION_ERR              0x00000005,
NDMP_IO_ERR                      0x00000007,
NDMP_ILLEGAL_ARGS_ERR            0x00000009,
NDMP_ILLEGAL_STATE_ERR           0x00000013,
NDMP_NO_MEM_ERR                  0x00000016,
NDMP_SM_RPL_SUCCESS              0x20500000,
NDMP_SM_RPL_GENERIC              0x20500001,
NDMP_SM_RPL_PATHNAME             0x20500002,
NDMP_SM_RPL_NETWORK              0x20500003,
NDMP_SM_RPL_NEGOTIATION          0x20500004,
NDMP_SM_RPL_PRE_XFER             0x20500005,
NDMP_SM_RPL_XFER                 0x20500006,
NDMP_SM_RPL_POST_XFER            0x20500007,
NDMP_SM_RPL_CLEANUP              0x20500008,
NDMP_SM_RPL_VOL_STATE            0x20500009,
NDMP_SM_RPL_REGISTRY             0x2050000a,
NDMP_SM_RPL_CMDS                 0x2050000b,
NDMP_SM_RPL_SNAPSHOT             0x2050000c,
NDMP_SM_RPL_CONFIG               0x2050000d,
NDMP_SM_RPL_LRD_GENERIC          0x2050000e,
NDMP_SM_RPL_LRD_SETUP            0x2050000f,
NDMP_SM_RPL_LRD_ACL              0x20500010,
NDMP_SM_RPL_LRD_DIRENTS          0x20500011,
NDMP_SM_RPL_LRD_RENAME           0x20500012,
NDMP_SM_RPL_LRD_DIROPS           0x20500013,
NDMP_SM_RPL_LRD_FILE             0x20500014,
NDMP_SM_RPL_LRS                  0x20500015,
NDMP_SM_RPL_LRD_EXTATTR          0x20500016,
NDMP_SM_RPL_SVE_CONFIG           0x20500017,

/* Misc errors.  These are NDMP specific snapmirror errors */

NDMP_SM_MISC_T                   0x20500200,
NDMP_SM_NOT_LICENSED             0x20500201,
NDMP_SM_OFF                      0x20500202,
NDMP_SM_INTERNAL_ERR             0x20500203,
NDMP_SM_ABORT_FAILED             0x20500204,
NDMP_SM_NOT_FOUND_ERR            0x20500205,
NDMP_SM_NO_REL                   0x20500206,
NDMP_SM_NO_SNAPS                 0x20500207,
NDMP_SM_NOT_DONE_YET             0x20500208,
};

The following list describes each error code. Included are the core NDMPerror codes that are used by the extension. The errors that are returned in reply to specific requests, and their specific meaning, are described in detail under the relevant message descriptions in section 3.
NDMP_NO_ERR
No error. 
NDMP_PERMISSION_ERR
The file system entity (drive, volume, etc.) is in a state that disallows access to the snapshot.
NDMP_IO_ERR
This general error is used if none of the more specific error codes apply. 
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_ILLEGAL_STATE_ERR
Message cannot be processed in the current state.
NDMP_NO_MEM_ERR
A memory allocation failure was detected. 
 [HS: This section will be expanded.]
2.8.3. Snapmirror Replies and notifications 
For the Snapmirror interface, ALL replies and ALL notify posts include the sequence number of the initiating message. For instance, if the blocking message NDMP_SM_GET STATUS carries the sequence number 1001, the corresponding ndmp_sm_get_state_reply includes a “cookie” with this sequence number. Likewise, if the non-blocking command NDMP_SM_CREATE_RELATIONSHIP carries sequence number 1002, the corresponding notification includes a cookie with this sequence number. This is done to assist DMA’s to match responses with commands, in particular when commands take some time to finish, and replies or notifications arrive out of order.
2.8.4. Snapmirror Enums
The following enums are defined for use in the Snapmirror extensions.
Enums for the level of synchronicity between source and destination. Returned by GET_INFO:
enum sm_sync_type
{
     SM_SYNC_TYPE_NONE       = 0,
     SM_SYNC_TYPE_IOPS       = 1,
     SM_SYNC_TYPE_MS         = 2
};

SM_SYNC_TYPE_IOPS: Synchronization delay measured in number of file system or block level writes.
SM_SYNC_TYPE_MS: Synchronization delay measured in milliseconds.
Enums for the state of a relationship. Returned by GET_STATUS:
enum sm_rel_state
{
     SM_STATE_UNINITIALIZED = 0, 
     SM_STATE_SNAPMIRRORED  = 1,
     SM_STATE_BROKENOFF     = 2,
     SM_STATE_QUIESCED      = 3,
     SM_STATE_SOURCE        = 4,
     SM_STATE_UNKNOWN       = 5
};

Enums for the status of a relationship. Returned by GET_STATUS:
enum sm_rel_status
{
     SM_STATUS_IDLE         = 0,
     SM_STATUS_TRANSFERRING = 1,
     SM_STATUS_PENDING      = 2,
     SM_STATUS_ABORTING     = 3,
     SM_STATUS_MIGRATING    = 4,
     SM_STATUS_QUIESCING    = 5,
     SM_STATUS_RESYNCING    = 6,
     SM_STATUS_WAITING      = 7
};

Enums for the state of the contents of a relationship. Returned by GET_STATUS:
enum sm_contents
{
     SM_CNT_NONE            = 0,
     SM_CNT_STABLE          = 1,
     SM_CNT_IN_TRANSITION   = 2,
     SM_CNT_ORIGINAL        = 3,
     SM_CNT_ROD             = 4
};

Enums for describing supported recovery methods. Returned by GET_INFO:
enum sm_recover_method
{
     SM_RECOVER_METHOD_ATOMIC      = 0,
     SM_RECOVER_METHOD_ONDEMAND    = 1
};
	
Enums for the connection mode when using multiple paths for data transfers. Used in NDMP_SM_DEFINE/MODIFY_CONNECTION, NDMP_SM_GET_INFO:
enum sm_conn_mode
{
     SM_MODE_FAILOVER   = 0,
     SM_MODE_MULTI      = 1
};

[The following set of location and transfer granularities is a proposal for new implementations. See section 2.1.2.x]
Enums for the granularity of backup or restore supported by a source or destination. Returned by GET_INFO:
enum sm_granularity
{
     SM_BACKUP_LOCATION_AGGR        = 0x0010,
     SM_BACKUP_LOCATION_VOL         = 0x0011,
     SM_BACKUP_LOCATION_TOPDIR      = 0x0012,
     SM_BACKUP_LOCATION_DIR         = 0x0013,
     SM_BACKUP_LOCATION_FILE        = 0x0014,

     SM_RECOVER_LOCATION_AGGR       = 0x0018,
     SM_RECOVER_LOCATION_VOL        = 0x0019,
     SM_RECOVER_LOCATION_TOPDIR     = 0x001A,
     SM_RECOVER_LOCATION_DIR        = 0x001B,
     SM_RECOVER_LOCATION_FILE       = 0x001C,

     SM_BACKUP_GRANULARITY_FULL     = 0x0020,
     SM_BACKUP_GRANULARITY_SUBTREE  = 0x0021,
     SM_BACKUP_GRANULARITY_DIR      = 0x0022,
     SM_BACKUP_GRANULARITY_FILE     = 0x0023,

     SM_RECOVER_GRANULARITY_FULL    = 0x0028,
     SM_RECOVER_GRANULARITY_SUBTREE = 0x0029,
     SM_RECOVER_GRANULARITY_DIR     = 0x002A,
     SM_RECOVER_GRANULARITY_FILE    = 0x002B
};

Enums for the transfer restart mode for a relationship, i.e. whether restart check-points are being generated and kept.

enum sm_restart_mode
{

     SM_RESTART_DEFAULT     = 0x0000,
     SM_RESTART_ALWAYS      = 0x0001,
     SM_RESTART_NEVER       = 0x0002
};

3. Data Server Interfaces
The Snapmirror interface is an extension to an NDMP Data server. Typically, on a single system, an NDMP data server will activate the interface.
The following sections define the protocol interface implemented by the NDMP data server for Snapmirror. The interface is used by the DMA to initiate Snapmirror management operations.
3.1. The Snapmirror Interface
This interface provides complete control of the snapmirror function on a storage system on the NDMP server. The interface is used to manage a Snapmirror agent on a disk based storage system such as a Network Appliance filer.
The following sections specify the protocol messages that are used to manage Snapmirror: 

3.1.1. NDMP_SM_DEFINE_RELATIONSHIP
This request is used by the DMA to instruct the Snapmirror mirror destination agent to create a new snapmirror relationship entry in the snapmirror.conf file. The request will cause a relationship entry to be created and configured with the specified bandwidth throttle values, try count, and source and destination path.
In order to actually create the relationship, and create the baseline replica data set on the destination, the DMA must issue an NDMP_SM_START_RELATIONSHIP message with the same destination path as is specified in this request.
There are two ways of maintaining a replica over time. Either the relationship is configured with a schedule for regular updates. See below. Or the DMA will issue NDMP_SM_UPDATE_RELATIONSHIP requests for the destination specified below.
This is a blocking command.
Message XDR definition 
/* NDMP_SM_DEFINE_RELATIONSHIP */
struct ndmp_sm_define_relationship_request 
{
      string                    dstsys<>;
      string                    dstpath<>;
      string                    srcsys<>;
      string                    srcpath<>;
      u_long                    throttle;
      sm_restart_mode           restart;
      bool                      syncmode;
      ndmp_sm_sched_item        schedspec;
      ndmp_sm_sync_item         syncspec;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
dstsys
The name of the destination system, i.e. this system.
This parameter is required since the snapmirror.conf file includes complete source and end specification: system and data set.
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
This parameter is the main identifier of the relationship.
srcsys
The system (filer) name for the snapmirror source. The source filer can be specified as a DNS hostname (toaster, toaster-e10, ..), or as a direct IP address (n.n.n.n).
Notice that this is the system name that identifies the source system from the destination.
srcpath
The source path as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
throttle
Kilobytes per second: the upper limit of data transfer bandwith for the relationship. Allowed range is >= 0. “0” means “unlimited”.
restart
An enum which signifies if restart points should be maintained for transfers. A restart point allows a transfer to be restarted from the restart point instead of having to restart from the very beginning.
syncmode
A Boolean which specifies whether the relationship is specified to operate in synchronous or semi synchronous mode (TRUE), or in asynchronous mode (FALSE).
schedspec
A structure containing the schedule to for the relationship. See section 2.1.??? for specifics on how to configure a snapmirror schedule.
syncspec
A structure containing the configuration of the synchronous or semi synchronous snapmirror behavior. See section 2.1.7.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_PERMISSION_ERR
The file system entity (drive, volume, etc.) is in a state that disallows access to the source or destination data set.
----- More to come.

3.1.2. NDMP_SM_DELETE_RELATIONSHIP
This request is used by the DMA to instruct the destination agent to delete the relationship entry in the conf file. The effects are twofold. First this action will terminate a schedule driven Snapmirror relationship. No more schedule driven updates will be performed. Secondly this action will remove the convenience of specifiying transfers only by the destination path.
This is a blocking command. 
Message XDR definition 
/* NDMP_SM_DELETE_RELATIONSHIP */
struct ndmp_sm_delete_relationship_request 
{
      string         dstpath<>;
};

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
This identifies the Snapmirror relationship to be stopped and removed.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
----- More to come.

3.1.3. NDMP_SM_START_RELATIONSHIP
[XXX: How to use with defined relationships!!]
This request is used by the DMA to instruct the Snapmirror mirror destination agent to initialize a new relationship between two storage systems. 
If the specified relationship has already been defined with NDMP_SM_DEFINE_RELATIONSHIP, both the destination AND the source MUST be identical to the values in the snapmirror.conf file. In this case it is recommended to only specify the destination.
The initializing transfer will be started. The schedule specified in NDMP_SM_DEFINE_RELATIONSHIP will be activated, and updates will be performed according to this schedule.
If the specified relationship was NOT defined previously, a new relationship will be created and the initializing transfer started. However, this relationship will NOT be visible in the snapmirror.conf file, and there will be no scheduled updates. This relationship is ONLY updated with explicit NDMP_SM_UPDATE_RELATIONSHIP requests.
This is a non-blocking command. The server will send an NDMP_SM_NOTIFY_START_RELATIONSHIP message when the transfer has completed.
Message XDR definition 
/* NDMP_SM_START_RELATIONSHIP */
struct ndmp_sm_start_relationship_request 
{
      string           dstpath<>;
      string           srcsys<>;
      string           srcpath<>;
      u_long           throttle;
      string           srcsnapname<>;
      string           dstsnapname<>;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error    error;
      u_long           ndmp_rseq;
};

Request Arguments
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
This parameter is the main identifier of the relationship.
srcsys
The system (filer) name for the snapmirror source. The source filer can be specified as a DNS hostname (toaster, toaster-e10, ..), or as a direct IP address (n.n.n.n).
Notice that this is the system name that identifies the source system from the destination.
srcpath
The source path “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
throttle
Kilobytes per second: the upper limit of data transfer bandwith for the relationship. Allowed range is >= 0. “0” means unlimited.
srcsnapname
This parameter ONLY applies to transfers of qtrees.
This parameter only applies to source systems that have snapshot capability.
If no snapshot name is provided, the snapmirror source agent will create a new snapshot and use this in the transfer.
If specified, the string specifies the snapshot on the source that defines the point-in-time copy to be transferred. This is the new baseline for the next data transfer.
See section 2.1.10. for the correct syntax for snapshots, and for compatibility issues with other extension interfaces.
dstsnapname
This parameter ONLY applies to transfers of qtrees.
If specified, the string specifies the name of the snapshot that the snapmirror destination agent will create on the destination after the transfer has completed. This snapshot will NOT be deleted by subsequent transfers, and need to be deleted separately by the DMA or a user.
If no snapshot name is provided, the snapmirror destination agent will NOT create a new snapshot.
See section 2.1.10. for the correct syntax for snapshots, and for compatibility issues with other extension interfaces.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The destination was able to start the command correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_PERMISSION_ERR
The file system entity (drive, volume, etc.) is in a state that disallows access to the source or destination data set.
----- More to come.

3.1.4. NDMP_SM_TRANSFER_UPDATE
[XXX: How to use with defined relationships!!]
This request is used by the DMA to instruct the snapmirror destination agent to start a non-scheduled incremental transfer from the Snapmirror source to the destination for an existing relationship.
This is a non-blocking command. The server will send an NDMP_SM_NOTIFY_TRANSFER_UPDATE message when the operation is complete. 
Message XDR definition 
/* NDMP_SM_TRANSFER_UPDATE */
struct ndmp_sm_transfer_update_request 
{
      string           dstpath<>;
      string           srcsys<>;
      string           srcpath<>;
      u_long           throttle;
      string           srcsnapname<>;
      string           dstsnapname<>;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error   error;
      u_long          ndmp_rseq;
};

Request Arguments
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
This parameter is the main identifier of the relationship.
srcsys
The system (filer) name for the snapmirror source. The source filer can be specified as a DNS hostname (toaster, toaster-e10, ..), or as a direct IP address (n.n.n.n).
Notice that this is the system name that identifies the source system from the destination.
srcpath
The source path as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’. 
throttle
The upper limit of data transfer bandwith for the relationship. Measured in kilobytes per second. The parameter is optional. If the value is “0” the throttle value is unlimited.
srcsnapname
This parameter ONLY applies to transfers of qtrees.
This parameter only applies to source systems that have snapshot capability.
If no snapshot name is provided, the snapmirror source agent will create a new snapshot and use this in the transfer.
If specified, the string specifies the snapshot on the source that defines the point-in-time copy to be transferred. This is the new baseline for the next data transfer.
See section 2.1.10. for the correct syntax for snapshots, and for compatibility issues with other extension interfaces.
dstsnapname
This parameter ONLY applies to transfers of qtrees.
If specified, the string specifies the name of the snapshot that the snapmirror destination agent will create on the destination after the transfer has completed. This snapshot will NOT be deleted by subsequent transfers, and need to be deleted separately by the DMA or a user.
If no snapshot name is provided, the snapmirror destination agent will NOT create a new snapshot.
See section 2.1.10. for the correct syntax for snapshots, and for compatibility issues with other extension interfaces.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The destination was able to start the command correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
----- More to come.

3.1.5. NDMP_SM_ABORT_TRANSFER
This request is used by the DMA to instruct the snapmirror agent to abort an ongoing Snapmirror transfer. The transfer is identified by the destination for the transfer. The transfer has been started by NDMP_SM_START_RELATIONSHIP, NDMP_SM_TRANSFER_UPDATE, NDMP_SM_RESTART_RELATIONSHIP, NDMP_SM_MIGRATE_RELATIONSHIP, or by the scheduler from a relationship defined by NDMP_SM_DEFINE_RELATIONSHIP.
The transfer can be aborted from either the source or the destination end. The abort operation itself is final. It can NOT be aborted.
Notice that if the transfer was started from an NDMP transfer command (START_RELATIONSHIP or TRANSFER_UPDATE), and not by the scheduler item, both the transfer command itself, and this ABORT command will issue notification messages.
The success signal from an ABORT is a “no_error” error code in the notification from this command. The error code in the notification from aborted transfer is implementation dependent.
This is a non-blocking command. The server will send an NDMP_SM_NOTIFY_ABORT_TRANSFER message when the operation is complete.
Message XDR definition 
/* NDMP_SM_ABORT_TRANSFER */
struct ndmp_sm_abort_transfer_request 
{
      string        dstsys<>;
      string        dstpath<>;
      bool          no_restart;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
dstsys
The system that is the destination for the transfer. Used to when aborting a transfer from the source end.
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
no_restart
If this field is TRUE, snapmirror will NOT keep the restart point. This implies that a following transfer will start from the beginning.
If this field is FALSE, the restart point for the transfer will be kept, and a following transfer request for the relationship will start from the restart point.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The destination was able to start the command correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
----- More to come.

3.1.6. NDMP_SM_QUIESCE_RELATIONSHIP
This request is used by the DMA to instruct the destination to quiesce a relationship. This causes successive transfer requests to fail until the relationship has been resumed with the NDMP_SM_RESUME_RELATIONSHIP request.
This command has special meaning to qtree destinations. A qtree destination which is being modified by SnapMirror during a transfer will have changes present in it. These changes will not be exported to NFS or CIFS clients. However, if a snapshot is taken during this time, the snapshot will contain the transitioning contents of the qtree. Quiesce will bring that qtree out of a transitioning state, by either finishing or undoing any changes a transfer has made. 
Snapmirror status reports whether a destination is quiesced or not. The quiesce process can take some time to complete while SnapMirror makes changes to the destination’s contents. Any snapshot taken while a destination is quiesced will contain an image of that destination which matches the contents exported to NFS and CIFS clients.
This is a non-blocking command. The server will send a NDMP_SM_NOTIFY_QUIESCE_RELATIONSHIP message when the operation is complete.
Message XDR definition 
/* NDMP_SM_QUIESCE_RELATIONSHIP */
struct ndmp_sm_quiesce_relationship_request 
{
      string       dstpath<>;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command was started correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_MEM_ERR
The system does not have sufficient resources to complete the command.
----- More to come.

3.1.7. NDMP_SM_RESUME_RELATIONSHIP
This request is used by the DMA to instruct the destination agent to resume a relationship that was quiesced. The relationship is identified by the destination.
This is a blocking command. 
Message XDR definition 
/* NDMP_SM_RESUME_RELATIONSHIP */
struct ndmp_sm_resume_relationship_request 
{
      string          dstpath<>;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
The request contains one or more invalid arguments.
NDMP_??_ERR
There is no quiesced relationship with the specified name.
----- More to come.

3.1.8. NDMP_SM_BREAK_RELATIONSHIP
This request is used by the DMA to break an active Snapmirror relationship, and signal that there will be no more transfers for the destination. In addition the destination is made read and writeable.
This command must be preceded by the NDMP_SM_QUIESCE_RELATIONSHIP command.
This is a blocking command. 
This command can NOT be aborted.
Message XDR definition 
/* NDMP_SM_BREAK_RELATIONSHIP */
struct ndmp_sm_break_relationship_request 
{
      string         dstpath<>;
};

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_??_ERR
There is no quiesced relationship with the specified name.
----- More to come.

3.1.9. NDMP_SM_RESTART_RELATIONSHIP
This request is used by the DMA to instruct the Snapmirror destination to restart an existing relationship between a Snapmirror source and destination. The relationship was broken with the NDMP_SM_BREAK_RELATIONSHIP command. 
This includes a re-initializing transfer from source to destination. 
Notice that this command is NOT for a reissue of an aborted transfer!
If the relationship was created with NDMP_SM_DEFINE_RELATIONSHIP, so that there is an entry in the conf file, the command will also re-activate the configured snapmirror schedule. Additional parameters such as bandwidth throttle values, try count, and source path already exist in the existing relationship configuration and should NOT be provided. If the previous source data set is on a different system or path, these parameters MUST be altered with the NDMP_SM_DELETE_RELATIONSHIP command followed by the NDMP_SM_DEFINE_RELATIONSHIP command.
If the relationship was NOT created with NDMP_SM_DEFINE_RELATIONSHIP, so that there is NOT an entry in the conf file, all parameters must be provided.
The typical use of this command is for reactivating a relationship after the source has performed a recovery.
This is a non-blocking command. The server will send a NDMP_SM_NOTIFY_RESTART_RELATIONSHIP message when the operation is complete.
Message XDR definition 
/* NDMP_SM_RESTART_RELATIONSHIP */
struct ndmp_sm_restart_relationship_request 
{
      string       dstpath<>;
      string       srcsys<>;
      string       srcpath<>;
      int          throttle<>;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
This parameter is the main identifier of the relationship.
srcsys
The source as “filer_name”. If the value is “”, the current value remains unchanged.
srcpath
The source path as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’. If the value is “”, the current value remains unchanged.
throttle
The upper limit of data transfer bandwith for the relationship. If the specified value is “-1” the already configured value remains unchanged.
Reply Arguments
error
Error code.
ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command was started correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_PERMISSION_ERR
The file system entity (drive, volume, etc.) is in a state that disallows access to the source or destination data sets.
----- More to come.

3.1.10. NDMP_SM_RELEASE_RELATIONSHIP
This request is used by the DMA to instruct the Snapmirror source end of a relationship to release all source end Snapmirror related resources for a Snapmirror relationship. This includes source end snapshots created for this relationship.
This command should follow the NDMP_SM_BREAK_RELATIOSNHIP request for the corresponding relationship on the destination end.
The relationship is identified by the destination system and source and destination path.
This is a blocking command. 
Message XDR definition 
/* NDMP_SM_RELEASE_RELATIONSHIP */
struct ndmp_sm_release_relationship_request 
{
      string       srcpath<>;
      string       dstsys<>;
      string       dstpath<>;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
srcpath
The path of the source data set as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
dstsys
The system name of the snapmirror destination.
dstpath
The path of the destination replica data set as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
Reply Arguments
error
Error code.
ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_NO_CONFIG_PERMISSION_ERR
The DMA is not authorized to use configuration commands.
----- More to come.

3.1.11. NDMP_SM_MIGRATE_RELATIONSHIP
This request is used by the DMA to migrate a data set transparently from the source to the destination. The command is issued to the snapmirror source agent.
The purpose of this command is to transfer the data as well as the file system handles such that NFS or CIFS clients will be able to maintain the file handles.
[XXX which step??]
This command is only one step of a complete migration process. With the “check flag” set the command can also be used to check that a migrate operation can be performed successfully before the actual migrate operation is started.
The relationship must exist and must have had at least had one data transfer (NDMP_SM_START_RELATIONSHIP) from source to destination. The individual steps of migrate are:
First the file system services on the source will be stopped and the volume is taken off-line.
Then a final update transfer is performed from source to destinaton. 
This transfer includes the transfer of the specific file handles.
Then the destination volume is made writeable.
This is a non-blocking command. The server will send an NDMP_SM_NOTIFY_MIGRATE_RELATIONSHIP message when the operation is complete. 
Message XDR definition 
/* NDMP_SM_MIGRATE */
struct ndmp_sm_migrate_request 
{
      string           srcsys<>;
      string           srcpath<>;
      string           dstsys<>;
      string           dstpath<>;
      u_long           throttle;
      bool             migr_check;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error   error;
      u_long          ndmp_rseq;
};

Request Arguments
srcsys
The system (filer) name for the snapmirror source. The source filer can be specified as a DNS hostname (toaster, toaster-e10, ..), or as a direct IP address (n.n.n.n).
Notice that this is the system name that identifies the source system from the destination.
srcpath
The source path as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
dstsys
The system (filer) name for the snapmirror destination. The destination system filer can be specified as a DNS hostname (toaster, toaster-e10, ..), or as a direct IP address (n.n.n.n).
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
This parameter is the main identifier of the relationship.
throttle
Kilobytes per second: the upper limit of data transfer bandwith for the relationship. Allowed range is >= 0. “0” means unlimited.
migr_check
A Boolean which signifies that the request should not actually carry out the migrate, but instead verify that the source and destination are both in a proper state for a migrate operation.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command started correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
----- More to come.

3.1.12. NDMP_SM_DEFINE_CONNECTION
This request is used by the DMA to instruct the Snapmirror mirror destination agent to create a new snapmirror connection entry in the snapmirror.conf file. The request will cause an entry to be created and configured with the specified connection mode and source-destination network paths. The configuration will take effect within two minutes, which is the frequency by which the system read the snapmirror.conf file.
The name is the identifier of the connection. It must be the name of a source filer that this filer has a snapmirror relationship with.
The information in this entry is used in successive data transfers from the source filer.
This is a blocking command.
Message XDR definition 
/* NDMP_SM_DEFINE_CONNECTION */
struct ndmp_sm_define_connection_request 
{
      string                    srcsys<>;
      sm_conn_mode              cmode;
      sm_tcp_addr               addrlist<>;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
srcsys
The system (filer) name for the snapmirror source. The source filer can be specified as a DNS hostname (toaster, toaster-e10, ..), or as a direct IP address (n.n.n.n).
This is the identifier of the connection.
cmode
An enum specifying the connection mode when two or more network paths a re specified. If the mode is S_CONN_FAILOVER snapmirror will use the first path as default, and the second path is the first path fails. If the mode is S_CONN_MULTI the two paths will be used in parallel, in a load balancing configuration.
addrlist
A list of pairs of sm_tcp_addr structures to be used for the connection. 
Each address pair specify a network path, the first address being the address of the remote (source) filer, the second address being the address of the interface on this filer to be used when contacting the remote filer with the first address.
Connection configuration is ONLY supported for synchronous snapmirror. We support only two address pairs. 
If the address list is empty, the address is derived by dns resolution using the srcsys name together with the domain.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_PERMISSION_ERR
The file system entity (drive, volume, etc.) is in a state that disallows access to the source or destination data sets.
----- More to come.

3.1.13. NDMP_SM_DELETE_CONNECTION
This request is used by the DMA to instruct the Snapmirror mirror destination agent to delete a snapmirror connection entry in the snapmirror.conf file.
See NDMP_SM_DEFINE_CONNECTION for more information.
This is a blocking command.
Message XDR definition 
/* NDMP_SM_DELETE_CONNECTION */
struct ndmp_sm_delete_connection_request 
{
      string                    srcsys<>;
}; 

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
srcsys
The system (filer) name for the snapmirror source. This is the identifier of the connection.
Reply Arguments
error
Error code.
Ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.

3.1.14. NDMP_SM_DESTINATIONS
The DMA uses this command to request from a snapmirror source agent a complete snapmirror destinations map for one or more source data sets on this system. If the request is issued to a system that is both a source and a destination, only the downstream destinations are returned.
For qtrees the reply includes only all direct destinations. For volumes and aggregates (in ONTAP 7.0) the reply include cascaded destinations, i.e. all filers and volumes that contain a replica of the specified source volume, as defined by existing snapmirror relationships.
When snapmirror is cascaded, such that a volume Va on filer A is mirrored to a replica Vab on file B, which again is mirrored to Vac on file C, this command will return the complete set of cascaded replicas for each specified relationship. 
This is blocking command. 
Message XDR definition 
/* NDMP_SM_DESTINATIONS */
struct ndmp_sm_destinations_request 
{
      string        path<>;
}; 

struct ndmp_sm_status
{
      string         srcsys<>;
      string         srcpath<>;
      string         dstsys<>;
      string         dstpath<>;
      sm_rel_state   state;	
      int            lag;
      sm_rel_status  status;
      ndmp_u_quad    bytes_transferred;
      string         base_snap<>;
      opaque         snapid[64];
};

struct ndmp_sm_destinations_reply 
{
      ndmp_snap_error error;
      u_long          ndmp_rseq;
      ndmp_sm_status  sm_status_list<>;
};

Request Arguments
path
A string specifying the full or first part of the path for one or more source data sets. Specified as empty, or as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
If the path is empty, the destinations for all relationships on this system are returned.
Reply Arguments
error
Error code.
ndmp_rseq
The sequence number of the request.
sm_status_list
A list of status items returning the status of the cascaded relationships for each source specified.
The status item structures will be in a number of status item sequences. Within one sequence the destination system and path of one item MUST be identical to the source system and path for the next item. A new sequence starts between two items for which this equality is NOT true.
The complete status for each relationship may not be filled in.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
----- More to come.

3.1.15. NDMP_SM_GET_STATUS
This request is used by the DMA to request operational status from the Snapmirror agent.
The agent will return the state of a set of Snapmirror relationships managed by a Snapmirror agent. If the path specifies a single relationship, the status for that relationship is returned, otherwise the status for all relationships under the specified path is returned. Otherwise all relationships in all volumes managed by the agent are returned.
If a transfer is in progress, the reply will include number of bytes transferred. The reply will also show the lag time since the snapshot was taken that was used in the last completed transfer.
This is a blocking command.
Message XDR definition 
/* NDMP_SM_GET_STATUS */
struct ndmp_sm_get_status_request 
{
      string     path<>;
}; 

struct ndmp_sm_status_long
{
      string         srcsys<>;
      string         srcpath<>;
      string         dstsys<>;
      string         dstpath<>;
      sm_rel_state   state;	
      int            lag;
      sm_rel_status  status;
      ndmp_u_quad    bytes_transferred;
      string         base_snap<>;
      opaque         snapid[64];
      int            mirror_timestamp;
      string         curr_trans_type<>;
      string         curr_trans_error<>;
      sm_contents    contents;
      string         last_trans_type<>;
      ndmp_u_quad    last_trans_size;
      int            last_trans_duration;
      string         last_trans_srcsys<>;
      string         last_trans_srcpath<>;
};

struct ndmp_sm_get_status_reply 
{
      ndmp_sm_error         error;
      u_long                ndmp_rseq;
      ndmp_sm_status_long   sm_status_list<>;
};

Request Arguments
path
The path or front end sub-path for one or more relationships: “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
Reply Arguments
error
Error code.
ndmp_rseq
The sequence number of the request.
sm_status_list
An array of ndmp_sm_status structure containing the status information requested. 
Note that this information is equivalent to the short format of the corresponding snapmirror status command.
The elements of the ndmp_sm_status structure are:
srcsys
The source system of the snapmirror relationship, i.e. the system running the Snapmirror source agent. The format is “filer”.
srcpath
The source path of the relationship. The format is “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
dstsys
The destination system of the snapmirror relationship, i.e. the system running the Snapmirror destination agent. The format is “filer”.
dstpath
The destination identifier as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
state
An enum defining the state of the relationship. See section 2.8.4. for enum definitions.
 
lag
The time since the snapshot used in the last transfer was created. Measured in seconds. Note that lag can be negative if there is time skew between a destination and some of its primaries.
status
An enum defining the status of the relationship. See section 2.8.4. for enum definitions.
Notice that if the status for a relationship is SM_STATUS_IDLE or SM_STATUS_PENDING, AND the value of <bytes_transferred> is != 0, this means that the previous transfer was aborted, and a restart checkpoint remains.
bytes_transferred
If status=SM_STATUS_TRANSFERRING, this 64 bit number holds the number of bytes transferred. The number is approximate. 
If status = SM_STATUS_IDLE, or status = SM_STATUS_PENDING, AND the value of bytes_transferred is != 0, this number holds the position of the restart checkpoint.
Otherwise the value is zero. 
base_snap
The symbolic (ASCII character) name of the base snapshot for the snapmirror relationship. 
snapid
An opaque 64 byte globally unique identifier of the base snapshot. This identifier is guaranteed to be unique forever, across all Network Appliance filers.
The snapid is compatible with the snapid defined and used in the snapshot management and Snapvault extensions. 
mirror_timestamp
The time when the last transfer occurred. This is the traditional “Unix time”, measured in seconds since January 1st, 1970.
curr_trans_type
A string stating the type of transfer for the currently ongoing or last completed transfer.
curr_trans_error
The error message that was displayed with the above transfer if there was a problem.
contents
An enum describing the state of the contents of the relationship.
last_trans_type
A string stating the type of transfer for the pf the transfer previous to the one displayed on curr_trans_type.
last_trans_size
A 64 bit integer presenting the size in bytes of the last (previous) transfer.
last_trans_duration
The duration of the last transfer. Measured in seconds.
last_trans_srcsys
The name of the source system from which the last transfer was performed.
last_trans_srcpath
The name of the source data set from which the last transfer was performed.

Reply Errors
NDMP_NO_ERR
The command completed correctly.
----- More to come.

3.1.16. NDMP_SM_GET_RELATIONSHIP_STATUS
This request is used by the DMA to request the status of active snapmirror relationships on the agent. The relationship is defined by the path name of the destination data set. 
If no destination is specified, all relationships are included.
A relationship state is described by its destination system and path name, followed by the source system and path name, followed by the configuration settings for bandwidth throttle.
This is a blocking command.
Message XDR definition 
/* NDMP_SM_GET_RELATIONSHIP_STATUS */
struct ndmp_sm_get_relationship_status_request 
{
      string        dstpath<>;
}; 

struct ndmp_sm_relationship_item
{
      string                    dstsys<>;
      string                    dstpath<>;
      string                    srcsys<>;
      string                    srcpath<>;
      u_long                    throttle;
      sm_restart_mode           restart;
      bool                      syncmode;
      ndmp_sm_sched_item        schedspec;
      ndmp_sm_sync_item         syncspec;
}; 

struct ndmp_sm_get_relationship_status_reply 
{
      ndmp_sm_error                 error;
      u_long                        ndmp_rseq;
      ndmp_sm_relationship_item     rel_status<>;
};

Request Arguments
dstpath
The destination as “vol:” + ‘vol_name’, “aggr:” + ‘agr_name’ or “qtree:” + ‘vol_name/myqtree_name’.
Reply Arguments
error
Error code.
ndmp_rseq
The sequence number of the request.
rel_status
An array of Snapmirror relationships and their current configuration. See NDMP_SM_DEFINE_RELATIONSHIP for definition of the individual parameters.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
----- More to come.

3.1.17. NDMP_SM_GET_ACCESS_INFO
This request is used by the DMA to inquire the access settings for the Snapmirror system. The access string defines which source storage systems are allowed to connect to this system using the Snapmirror protocol. The access string can specify both which remote hosts may or may not connect, as well as over which network interface on this system they may connect.
Notice that since Snapmirror is “pull based”, where it is the destination end that contacts the source end, the access string on the source system determines which destinations may request initial or update transfers.
This is a blocking command.
Message XDR definition 
/* NDMP_SM_GET_ACCESS_INFO */
struct ndmp_sm_get_access_info_reply 
{
      ndmp_sm_error   error;
      u_long          ndmp_rseq;
      string          sm_access<>;
};

Request Arguments
The request has no message body.

Reply Arguments
error
Error code.
ndmp_rseq
The sequence number of the request.
sm_access
A string indicating which remote systems are allowed to make data connections to and request data from the destination. The string can also specify over which local network interface the connection should be made. See section 2.1.1 for the syntax and examples of the access string.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
----- More to come.

3.1.18. NDMP_SM_SET_ACCESS_INFO
This request is used by the DMA to set the snapmirror access string on the system.
This is a blocking command.
Message XDR definition 
/* NDMP_SM_SET_ACCESS_INFO */
struct ndmp_sm_set_access_info_request 
{
      string          sm_access<>;
};

struct ndmp_sm_reply 
{
      ndmp_sm_error error;
      u_long        ndmp_rseq;
};

Request Arguments
sm_access
The access string. See section 2.1.1 for the correct syntax.
Reply Arguments
error
Error code.
ndmp_rseq
The sequence number of the request.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.

3.1.19. NDMP_SM_GET_INFO
This request is used by the DMA to inquire the capabilities of the NDMP and Snapmirror agent. This includes information stating whether the Snapmirror destination is licenced, is enabled, and the major and minor version number.
This is a blocking command.
Message XDR definition 
/* NDMP_SM_GET_INFO */
struct ndmp_sm_get_info_reply 
{
      ndmp_sm_error        error;
      u_long               ndmp_rseq;
      bool                 sm_licensed;
      bool                 sm_enabled;
      u_long               major_version;	
      u_long               minor_version;
      sm_granularity       granularities<>;
      sm_tcp_addr          data_addr<>;
};

Request Arguments
The request has no message body.

Reply Arguments
error
Error code.
ndmp_rseq
The sequence number of the request.
sm_licensed
A boolean indicating whether the Snapmirror destination function is licensed. The value is TRUE or FALSE.
sm_enabled
A boolean indicating whether the Snapmirror destination function is enabled. The value is TRUE or FALSE.
major_version
An integer containing the major version number for the snapmirror destination.
minor_version
An integer containing the minor version number for the snapmirror destination.
granularities
A list of enums; each enum defining the data set granularity supported by snapmirror transfers. Notice that for a DMA to use a capability, the capability MUST be supported BOTH by the source and the destination.
See section 2.5.1. and subsections for more information on granularities.
addlist
A list of sm_tcp_addr structure containing the set of addresses to be used by a remote party when defining connections in the snapmirror.conf file. If the list is empty, the address is derived by dns resolution using the host name.
Reply Errors
NDMP_NO_ERR
The command completed correctly.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
----- More to come.

4. Data Management Application Interfaces
This section defines the protocol interfaces implemented by the Data Management Application (DMA). Note that these unsolicited data server generated messages do not have a reply associated with them. Rather they allow one-way communication of diagnostic, error, event and state information from the data server to the DMA.
Snapmirror source and destination notifications required by protocol definition are conveyed using the set of NDMP_SM_NOTIFY_* interface messages defined in this section. Implementation specific diagnostic, error, and informational updates a conveyed using the standard NDMP_LOG_MESSAGE defined in Ref 1.
4.1. Destination Notify Interface.
This interface is used to asynchronously inform the DMA that a non-blocking Snapmirror destination operation has ended and to provide a final status of the operation.
The interface includes one response message for each of the non-blocking Snapmirror destination requests described in 3.1.x A Snapmirror non-blocking operation is potentially a time intensive operation. The data destination issues a reply message once the request is validated and the operation is initiated. The following post messages are is issued by the data server when the non-blocking operations complete.
Implementation guideline: It is the recommended DMA behavior that all of this information SHOULD be placed in a file for purposes of logging. If there is textual information to be communicated to the user, an NDMP_LOG_MESSAGE MAY be sent following the NDMP_SM_NOTIFY message.

4.1.1. NDMP_SM_NOTIFY_START_RELATIONSHIP
This asynchronous post message is used to notify a DMA that the current NDMP_SM_START_RELATIONSHIP operation on the Snapmirror agent has ended.  It indicates whether the create operation completed successfully, completed with exceptions, failed, or was aborted due to a data server internal error.  It also provides an error indication for non-successful NDMP_SM_START_RELATIONSHIP operations.
When this post message indicates an error condition, the data server MAY generate one or more NDMP_LOG_MESSAGE POST providing further status information.  The data server SHOULD include the sequence number of the NDMP_SM_NOTIFY_START_RELATIONSHIP message in the associated_message_sequence of the NDMP_LOG_MESSAGE.  
If the transfer was aborted by an ABORT_TRANSFER command, the error code is implementation dependent. A successful ABORT is signaled through a “no_error” error code in the notification for the ABORT command.
Post Message
struct ndmp_sm_notify_post
{
     ndmp_sm_error         error;
     u_long                ndmp_rseq;
};

Post Message Arguments 
error
Indicates the final status and the specific ndmp_error or ndmp_sm_error detected by the data server that prevented the NDMP_SM_START_RELATIONSHIP operation from completing successfully. If the operation succeeds, the returned error code MUST contain NDMP_NO_ERR. 
Examples of create errors that MAY be reported include the following:
NDMP_NO_ERROR
The operation completed successfully. The resultant snapshot image contains all specified data. 
NDMP_INTERNAL_ERROR
The operation failed due to unrecoverable error incurred by the Snapmirror destination during processing. The specified data set state is undetermined. 

-- MORE TO COME --
ndmp_rseq
The sequence number of the originating request.

4.1.2. NDMP_SM_NOTIFY_TRANSFER_UPDATE
This asynchronous post message is used to notify a DMA that the current NDMP_SM_TRANSFER_UPDATE operation has ended. It indicates whether the transfer operation completed successfully, failed, or was aborted due to a data server or Snapmirror internal error. It also provides an error indication for non-successful NDMP_SM_TRANSFER_UPDATE operations.
When this post message indicates an error condition, the data server MAY generate one or more NDMP_LOG_MESSAGE POST providing further status information.  The data server SHOULD include the sequence number of the NDMP_SM_NOTIFY_TRANSFER_UPDATE message in the associated_message_sequence of the NDMP_LOG_MESSAGE.
If the TRANSFER_UPDATE command was aborted by an ABORT_TRANSFER command, the error code is implementation dependent. A successful ABORT is signaled through a “no_error” error code in the notification for the ABORT command.
Post Message
struct ndmp_sm_notify_post
{
     ndmp_sm_error         error;
     u_long                ndmp_rseq;
};

Post Message Arguments 
error
Indicates success or the specific ndmp_error or ndmp_sm_error detected by the data server that prevented the NDMP_SM_TRANSFER_UPDATE operation from completing successfully. If the operation succeeds, the returned error code MUST contain NDMP_NO_ERR.
Examples of delete errors that MAY be reported include the following:
In all case except SUCCESS the data server MAY generate one or more NDMP_SM_LOG messages providing further exception information.
NDMP_NO_ERROR
The operation completed successfully. The resultant snapshot image contains all specified data. 
NDMP_INTERNAL_ERROR
The operation failed due to unrecoverable error incurred by the Snapmirror destination during processing. The specified data set state is undetermined.
-- MORE TO COME --
ndmp_rseq
The sequence number of the originating request.

4.1.3. NDMP_SM_NOTIFY_ABORT_TRANSFER
This asynchronous post message is used to notify a DMA that the current NDMP_SM_ABORT_TRANSFER operation has ended. It indicates whether the operation completed successfully, failed, or was aborted due to a data server or Snapmirror internal error. It also provides an error indication for non-successful NDMP_SM_ABORT_TRANSFER operations.
When this post message indicates an error condition, the data server MAY generate one or more NDMP_LOG_MESSAGE POST providing further status information. The data server SHOULD include the sequence number of the NDMP_SM_NOTIFY_ABORT_TRANSFER message in the associated_message_sequence of the NDMP_LOG_MESSAGE.
Post Message
struct ndmp_sm_notify_post
{
     ndmp_sm_error         error;
     u_long                ndmp_rseq;
};

Post Message Arguments 
Error
Indicates the specific ndmp_error or ndmp_sm_error detected by the data server that prevented the NDMP_SM_ABORT_TRANSFER operation from completing successfully. If the operation succeeds, the returned error code MUST contain NDMP_NO_ERR. 
In all case except SUCCESS the data server MAY generate one or more NDMP_SM_LOG messages providing further exception information.
NDMP_NO_ERROR
The operation completed successfully. The resultant snapshot image contains all specified data. 
NDMP_INTERNAL_ERROR
The operation failed due to unrecoverable error incurred by the Snapmirror destination during processing. The specified data set state is undetermined.
-- MORE TO COME --

ndmp_rseq
The sequence number of the originating request.

4.1.4. NDMP_SM_NOTIFY_QUIESCE_RELATIONSHIP
This asynchronous post message is used to notify a DMA that the current NDMP_SM_QUIESCE_RELATIONSHIP operation has ended. It indicates whether the operation completed successfully, failed, or was aborted due to a data server or Snapmirror internal error. It also provides an error indication for non-successful NDMP_SM_QUIESCE_RELATIONSHIP operations.
When this post message indicates an error condition, the data server MAY generate one or more NDMP_LOG_MESSAGE POST providing further status information.  The data server SHOULD include the sequence number of the NDMP_SM_NOTIFY_QUIESCE_RELATIONSHIP message in the associated_message_sequence of the NDMP_LOG_MESSAGE.
This asynchronous post message is used to notify a DMA that the 
Post Message
struct ndmp_sm_notify_post
{
     ndmp_sm_error             error;
     u_long                    ndmp_rseq;
};

Post Message Arguments 
error
Indicates the specific ndmp_error or ndmp_sm_error detected by the data server that prevented the NDMP_SM_QUIESCE_RELATIONSHIP operation from completing successfully. If the operation succeeds, the returned error code MUST contain NDMP_NO_ERR.
In all case except SUCCESS the data server MAY generate one or more NDMP_SM_LOG messages providing further exception information.
NDMP_NO_ERROR
The operation completed successfully. The resultant snapshot image contains all specified data. 
NDMP_INTERNAL_ERROR
The operation failed due to unrecoverable error incurred by the Snapmirror destination during processing. The specified data set state is undetermined. 
-- MORE TO COME --

ndmp_rseq
The sequence number of the originating request.

4.1.5. NDMP_SM_NOTIFY_RESTART_RELATIONSHIP
This asynchronous post message is used to notify a DMA that the current NDMP_SM_RESTART_RELATIONSHIP operation has ended. It indicates whether the operation completed successfully, failed, or was aborted due to a data server or Snapmirror internal error. It also provides an error indication for non-successful NDMP_SM_RESTART_RELATIONSHIP operations.
When this post message indicates an error condition, the data server MAY generate one or more NDMP_LOG_MESSAGE POST providing further status information.  The data server SHOULD include the sequence number of the NDMP_SM_NOTIFY_RESTART_RELATIONSHIP message in the associated_message_sequence of the NDMP_LOG_MESSAGE.
This asynchronous post message is used to notify a DMA that the 
Post Message
struct ndmp_sm_notify_post
{
     ndmp_sm_error           error;
     u_long                  ndmp_rseq;
};

Post Message Arguments 
Error
Indicates the specific ndmp_error or ndmp_sm_error detected by the data server that prevented the NDMP_SM_RESTART_RELATIONSHIP operation from completing successfully. If the operation succeeds, the returned error code MUST contain NDMP_NO_ERR.
In all case except SUCCESS the data server MAY generate one or more NDMP_SM_LOG messages providing further exception information.
NDMP_NO_ERROR
The operation completed successfully. The resultant snapshot image contains all specified data. 
NDMP_INTERNAL_ERROR
The operation failed due to unrecoverable error incurred by the Snapmirror destination during processing. The specified data set state is undetermined. 
-- MORE TO COME --

ndmp_rseq
The sequence number of the originating request.
4.1.6. NDMP_SM_NOTIFY_MIGRATE_RELATIONSHIP
This asynchronous post message is used to notify a DMA that the current NDMP_SM_MIGRATE_RELATIONSHIP operation has ended. It indicates whether the operation completed successfully, failed, or was aborted due to a data server or Snapmirror internal error. It also provides an error indication for non-successful NDMP_SM_MIGRATE_RELATIONSHIP operations.
When this post message indicates an error condition, the data server MAY generate one or more NDMP_LOG_MESSAGE POST messages to provide further status information.  The data server SHOULD include the sequence number of the NDMP_SM_NOTIFY_MIGRATE_RELATIONSHIP message in the associated_message_sequence of the NDMP_LOG_MESSAGE.
Post Message
struct ndmp_sm_notify_post
{
     ndmp_sm_error           error;
     u_long                  ndmp_rseq;
};

Post Message Arguments 
Error
Indicates the specific ndmp_error or ndmp_sm_error detected by the data server that prevented the NDMP_SM_MIGRATE_RELATIONSHIP operation from completing successfully. If the operation succeeds, the returned error code MUST contain NDMP_NO_ERR.
In all case except SUCCESS the data server MAY generate one or more NDMP_SM_LOG messages providing further exception information.
NDMP_NO_ERROR
The operation completed successfully. The resultant snapshot image contains all specified data. 
NDMP_INTERNAL_ERROR
The operation failed due to unrecoverable error incurred by the Snapmirror destination during processing. The specified data set state is undetermined. 
-- MORE TO COME --
ndmp_rseq
The sequence number of the originating request.

5. Security
TBD
6. Recognition of Prior Work
TBD
7. References
[1] draft-skardal-ndmpv4-02.txt, "Network Data Management Protocol Version 4", H. Skardal, Network Appliance, April 2001
8. Authors and Contributors
8.1. Document Editor

Harald Skardal
Network Appliance Inc.
Telephone: 603 882 3881
Email: Harald.Skardal@netapp.com 

8.2. Authors' Addresses

Harald Skardal
Network Appliance Inc.
Telephone: 603 882 3881
Email: Harald.Skardal@netapp.com 

8.3. Contributors
In addition to the authors, the following people have helped the effort by participating in the reviews:
Kiyoshi Komatsu,
Network Appliance Inc.
Telephone: 408 822 6000
Email: Kiyoshi.Komatsu@netapp.com

Greg Linn
Network Appliance Inc.
Telephone: 408 822 6000
Email: Greg.Linn@netapp.com

Sandeep Mukherjee
Network Appliance Inc.
Telephone: 408 822 6000
Email: Sandeep.Mukherjee@netapp.com

Collin Park
Network Appliance Inc.
Telephone: 408 822 6000
Email: Collin.Park@netapp.com

Appendix A: Specific Data ONTAP Properties and Issues
This appendix lists the known set of issues and capabilities of Data ONTAP that are relevant to this specification.

General Comments:

Volumes:

Qtrees:

Recoveries:

Comments on specific commands:
NDMP_SV{S|P}_GET_DATA_ADDRESS:
The resulting addresses are based on filtering the specified remote address against the snapmirror.access string of the system receiving the message. The system will do a DNS request to determine the host name of the given IP address, and filter this host name against the access string to determine if the host is allowed access, and over which network interfaces, i.e. IP addresses.
The returned list of IP addresses contain the subset of the network interfaces that will accept data connections for snapmirror data transfer from the given IP address. The list is given in prioritized order based on internal criteria or configuration data such as bandwidth.
This list should be filtered against knowledge about the existing network topology to pick the most efficient or reliable data path between a source and a destination.

Appendix B: XDR definitions:
The appendix lists all the xdr’s defined in this specification:

/*
 * XDR definitions for the NDMPv4 Snapmirror extension.
 *
 * .h file is generated with
 *       rpcgen -C -h -o ndmpv4-snapmirror.h ndmpv4-snapmirror.x
 *
 * .c file is generated with
 *       rpcgen -c -o ndmpv4-snapmirror-xdr.c ndmpv4-snapmirror.x
 *
 */

/*
 * Include files:
 */

/*
 * #include "ndmpv4.h"
 */

/*
 * NDMPv4 Snapmirror Message Numbers.
 */

Appendix C: Snapmirror: NDMP API vs. ONTAP CLI mapping:

The appendix list the mapping from the NDMP protocol requests to the ONTAP snapmirror CLI commands:

NDMP_SM_DEFINE_RELATIONSHIP     – (no mapping) - Edit the
                                   snapmirror.conf file.
NDMP_SM_DELETE_RELATIONSHIP     – (no mapping) - Edit the
                                  snapmirror.conf file.
NDMP_SM_START_RELATIONSHIP      – snapmirror initialize
NDMP_SM_TRANSFER_UPDATE         – snapmirror update
NDMP_SM_ABORT_TRANSFER          – snapmirror abort
NDMP_SM_QUIESCE_RELATIONSHIP    – snapmirror quiesce
NDMP_SM_RESUME_RELATIONSHIP     – snapmirror resume
NDMP_SM_BREAK_RELATIONSHIP      – snapmirror break
NDMP_SM_RESTART_RELATIONSHIP    – snapmirror restart
NDMP_SM_RELEASE_RELATIONSHIP    – snapmirror release
NDMP_SM_MIGRATE_RELATIONSHIP    – snapmirror migrate 
NDMP_SM_DESTINATIONS            – snapmirror destinations
NDMP_SM_GET_STATUS              – snapmirror status 
NDMP_SM_GET_RELATIONSHIP_STATUS – (no mapping) – View entries
                                  in the snapmirror.conf file.
NDMP_SM_GET_ACCESS_INFO         – options snapmirror.access
NDMP_SM_SET_ACCESS_INFO         – options snapmirror.access
                                  ‘new_value’
NDMP_SM_GET_INFO – (no mapping) – get capabilities
NDMP_SM_DEFINE_CONNECTION       – (no mapping) – 
                                  edit snapmirror.conf
                                  to create new
                                  connection entry.
NDMP_SM_DELETE_CONNECTION       – (no mapping) – 
                                  edit snapmirror.conf
                                  to delete connection
                                  entry.