Network Working Group                                         Greg Linn,  
INTERNET DRAFT                                        Network Appliance,    
Category: Applications                                      Bill Taylor,                                      
Document: draft-xxxx-ndmp4-restart-009.txt            Network Appliance,
                                                               Jim Ward,
                                                           Reliaty, Inc.

NDMP Version 4 Backup Restart Extension Interface

Status of this Memo
This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. 
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." 
The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt 
The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 

Abstract
The Network Data Management Protocol (NDMP) defines a mechanism and protocol for controlling backup, recovery, and other transfers of data between primary and secondary storage. The XDR and TCP/IP protocols are foundations for NDMP.
The NDMP Backup Restart Extension Interface leverages the extensibility of NDMP v4 to define a mechanism and protocol allowing failed backup operations to be restarted from a known good point (checkpoint) in the data stream prior to the failure.

Copyright
Copyright (C) The Internet Society (2001).  All Rights Reserved.

Table of Contents

Revision Log	
1. Overview	
1.1. Restart Granularity	
1.2. Selection of Restart Checkpoints	
1.3. Backup of Live File Systems	
2. Extension Definitions	
2.1. Extension Class	
2.2. Messages Numbers	
2.3. Error Codes	
3. Data Server Interfaces	
3.1. BRE Interface	
3.1.1. NDMP_BRE_CREATE_CONTEXT	
3.1.2. NDMP_BRE_DELETE_CONTEXT	
3.1.3. NDMP_BRE_FLUSH_FILE_HISTORY	
3.1.4. NDMP_BRE_PRESET_MOVER	
3.1.5. NDMP_BRE_RESTART_BACKUP	
3.1.6. NDMP_BRE_GET_SERVER_INFO	
3.1.7. NDMP_BRE_GET_CONTEXT_LIST	
3.1.8. NDMP_BRE_GET_PATH_INFO	
4. Data Management Application Interfaces	
5. References	
6. Security	
7. Recognition of Prior Work	
8. Authors and Contributors	
8.1. Document Editor	
8.2. Authors' Addresses	
8.3. Contributors	
Appendix A: XDR Definitions	

	

Revision Log
[This revision log will be removed from the final draft. For now it is used to keep track of what has been added.]
Doc Version                  Update Date and Change Log
---------------------------------------------------------------------
0.0.1	05/20/02 Added NDMP_BRE_GET_SERVER_INFO definition. Added context_handle to ndmp_bre_context_state.
 
0.0.2	05/23/02 Corrected typos identified by Bill Taylor. Corrected ndmp_bre_fh_data and ndmp_bre_context_state structure definitions. Added the restart_block_size field to the NDMP_BRE_GET_SERVER_INFO definition.

0.0.3	05/30/02 Corrected definition inconsistencies identified by Bill Taylor. Modified the context_handle definitions to specify an explicit size of 16 bytes. Modified the union ndmp_bre_fh_data definition to reference NDMP core definitions for NDMP_BUTYPE_BACKUP_FH_DIR and NDMP_BUTYPE_BACKUP_FH_FILE.

0.0.4	06/12/02 Integrated changes agreed to during the 06/06/02 functional review with Jim Ward as follows:

 
1) The definition of the NDMP_BRE_GET_SERVER_INFO restart_block_size now specifies that the DMA MUST set the mover record_size to an integer multiple of the restart_block_size.

2) Added the NDMP_BRE_FLUSH_FILE_HISTORY request/reply message pair to force transmission of any queued file history entries. 

3) Added the NDMP_BRE_PRESET_MOVER request/reply message pair to allow the DMA to modify the bytes_moved and record_number state values prior to restarting a backup operation.

4) Modified the context_handle definition to specify a variable length opaque to allow more implementation flexibility.

5) Defined new constants for file history type as the existing core definitions were misleading (file & dir vs. file and node).

6) Removed definition of the NDMP_BRE_GET_CONTEXT_LIST DELETEABLE context_status bit. Also changed the BUSY status to indicate a context should not be deleted.
0.0.5	06/24/02 Incorporated changes recommended by Bill Taylor and Jim Ward.

1)	corrected miscellaneous typos.

2)	Modified ndmp_bre_fh_data structure definition to to specify a single node or file entry vs. a list.

3)	Modified ndmp_bre_fh_data structure to be consistent with the previous enum definition.

4)	Modified the ndmp_bre_restart_backup_request definition to specify the restart_offset as a ndmp_u_quad vs. a u_long.

5)	Modified ndmp_bre_context_state definition to reference backup_results in a consistent manner.

6)	Modified ndmp_bre_context_state to explicitly specify the backup_results as an ndmp_data_halt_reason enum value rather than an ndmp_data_halted_post message. 
0.0.6	07/01/02 Incorporated additional changes recommended by Jim Ward.

1) 
Changed Jim’s affiliation from WS to "Reliaty, Inc."

2) Updated the NDMP_BRE_RESTART_BACKUP request fh_data description to reference the ndmp_node & ndmp_flle structures rather than ndmp_fh_add_file_post and ndmp_fh_add_node_post structures.  Also deleted the now unnecessary mandate that the fh_data only contain a single entry.

08/20/02 removed expectation that a NDMP_MOVER_SET_WINDOW must be issued prior to a NDMP_BRE_PRESET_MOVER request.

08/30/02 updated ndmp_bre_message enum definitions to reflect correct message numbers.

08/30/02 updated ndmp_bre_error enum definitions to reflect correct message numbers. Also updated XDR definitions in Appendix A.
0.0.7	10/01/02 
0.0.11	Incorporated yet more changes as recommended by Jim Ward.

1)	Added ndmp_bre_get_path_info request/reply.

2) Updated Appendix A to remove a duplicate and incorrect XDR definition for ndmp_bre_restart_backup
0.0.8    01/31/03 
Incorporated Jim Ward’s live file system text.
0.0.9	05/20/03 Final revisions as follows:

1)	Incorporated Jim Ward’s new checkpoint selection text.

2)	Added NDMP_PRECONDITION_ERR to ndmp_bre_error enum and NDMP_BRE_RESTART_BACKUP description.

3)	Expanded the NDMP_BRE_CONTEXT_STATUS_BACKUP_RESTARTABLE definition contained in NDMP_BRE_GET_CONTEXT_LIST description.

Issues Log
[This issues log will be removed from the final draft. For now it is used to keep track of open and unresolved issues.]

1. Overview
The purpose of the NDMP v4 backup restart extension interface is to define a mechanism and protocol allowing NDMP backup applications (DMAs) to restart a failed backup operation at a known good point prior to the failure. A known good point is defined as a point in the backup data stream where the data has been successfully written to tape and that data can be located and accessed by subsequent NDMP recovery operations. The backup restart extension interface mechanism supports both local and 3-way NDMP topologies in an identical manner.
Backup failures can occur for a variety of reasons.  The most common reasons include secondary storage tape media or hardware failures, loss of network connectivity between NDMP peers and primary storage software failures. 
Without the NDMP v4 backup restart extension, any failure encountered during a backup operation renders the entire backup tape set unusable even though a significant portion of the backup data may have been successfully transferred to tape. For example consider an 800 GB NDMP backup that requires approximately 32 hours to complete and spans 8 tapes. If a failure occurs while writing to the last tape, the previous 7 tapes representing ~28 hours of backup time will be unusable and the entire 32 hour backup will need to be re-executed.  As data sets continue to increase in size, so does the impact of unrecoverable backup failures.
The backup restart interface addresses the restart problem by allowing the backup application to create and manage ‘contexts’ representing information associated with specific backup operations, and to specify a backup restart point to a Data Server in terms of a context handle, a file system entity (file or inode) and an absolute byte offset within backup stream. 
The backup application uses the standard NDMP file history mechanism to obtain file system entity information and NDMP_MOVER_GET_STATE requests to obtain backup stream offset information at appropriate ‘checkpoints’ during the backup. 
The backup restart interface is dependent on an assumption that the Data Server specifies the fh_info field of file history posts (ndmp_fh_add_file_post or ndmp_fh_add_node_post) as an absolute byte offset to the start of the file or inode within the backup data stream. 
Note: Although NDMP v4 does not mandate the contents of fh_info; it does indicate fh_info will typically contain the needed absolute byte offset.

1.1. Restart Granularity
The protocol design permits checkpoints or restart granularity at any byte offset within the backup stream. The actual restart granularity is determined by the Data Server implementation and may be at file or inode boundaries, sub-file or sub-inode boundaries (blocks), or at any backup stream byte offset. The NDMP_BRE_GET_SERVER_INFO request provides a mechanism for the backup application to determine the backup restart granularity of a specific Data Server implementation. 
At a minimum restarts SHOULD be possible at either tape boundaries or file boundaries. Usability of partial tapes is dependent on the backup applications ability to terminate a tape prior to the end of media and maintain an indication of which tapes were prematurely terminated. 
The backup restart interface can take advantage of a Data Server’s ability to generate a snapshot or point in time image of the file system as part of the backup context, but is not dependent upon snapshot support.  
A backup context that includes a snapshot may offer a higher degree of restart granularity based on the Data Server’s ability to deterministically reproduce the exact backup stream.  In this case a backup restart point within a file or inode should be possible.
A Data Server whose backup method operates on live file systems without snapshots can still use the backup restart interface.  In this case backup restart points will be limited to file boundaries, since the consistency of a partial file can not be guaranteed between two points in time. 
Note: when a Data Server is limited to file boundary restart points, the backup application MUST be capable of supporting premature tape termination since it is unlikely that file boundaries will correspond to tape boundaries.
1.2. Selection of Restart Checkpoints
A DMA may take a checkpoint from which it may later restart the backup only when:
1.	The Mover is paused.
2.	The DMA has received at least one file history "add node" or "add file" message from the Data Service.
3.	The Data Service is otherwise at point from which a restart may occur, the definition of which is implementation-specific.
In the case of Network Appliance Data Ontap, a DMA may restart any backup from a checkpoint taken when the Data Service is in Pass 4, dumping file content data. The earliest point is following the DMA's receipt of the first file history "add node" message, which may be prompted by a DMA-issued NDMP_BRE_FLUSH_FILE_HISTORY request; the latest is 1024 bytes before the backup image offset defined by the ACL_START environment variable. Should ACL_START not yet be included in the response to the NDMP_DATA_GET_ENV request, this "latest" bound does not yet apply.
As the DMA may take a checkpoint only when the Mover is paused, the implementer may choose to use the Mover window to force the Mover to pause periodically throughout each backup. Without using a Mover window, the DMA would be constrained to taking checkpoints only at end of medium. Mover window size is the implementer's choice, providing that it's an even multiple of the Mover record size.
Once the Mover is paused and the DMA has determined to take a checkpoint, it must: 
1.	Take any required action to prepare the medium for a restart from this point, such as flushing a drive's on-board buffer to tape.
2.	Capture to stable storage all DMA-specific state that it must later restore in order to restart the backup from this point.
3.	Mark (such as for later consumption by a scheduler) that the backup operation is restartable from this mid-point, rather than from its beginning.

1.3. Backup of Live File Systems
Support of this extension requires special considerations by a data service that backs up file systems whose content may change during a backup. 
First, such a data service must be able to determine whether a file system object at which a restart occurs changed between time that object and any associated meta-data was written to the data stream (as a result of a foregoing NDMP_DATA_START_BACKUP or NDMP_BRE_RESTART_BACKUP) and the time at which the restart took place. "Change," in this case, refers to one or more of the object's existence, contents, size, or meta-data.
As a data service has no direct control over which file system object the DMA chooses to restart at, this requirement applies to each object written to the data stream after the backup becomes restartable.
If the file system object at which a DMA chooses to restart has not changed, the restart operation proceeds identically as if backing up a static file system. Changes to objects saved earlier in the data stream do not affect the restart logic. 
On the other hand, if the DMA-selected file system object has changed, the data service must take additional actions not required for data services supporting only static file system backup:
It must preserve the integrity of the data stream. (As the data service itself defines the format of the data stream, the means by which it satisfies this requirement is implementation-specific.)
At its option, it may mark the object at which the restart occurred as "invalid" in the backup image.  (Doing so guarantees that, when the object is recovered, it can be identified as self-inconsistent and hence, discarded.) 
It may rewrite the object to the data stream and republish the object's file history data to the DMA. A DMA supporting this extension is obligated to consider such updated file history as authoritative. The DMA should, however, take care to preserve any context associated with restart point, which may include the original file history, so that another restart from the same point is possible should it become necessary. 
Data services that produce unstructured "tar"-like backup images will have substantial difficulty with this model. This arises from the declaration of the length of each file system object in the tar header, followed by file content. Should a backup restart occur in the middle of a file whose length has changed, the data service has no means of learning the required length of the "file content" portion of the image.

2. Extension Definitions

2.1. Extension Class
The NDMP Backup Restart Extension (BRE) interface is a proprietary extension defined in class 0x2050 version 2, interface 02, as messages 0x00 thru 0x04.
2.2. Messages Numbers
The following messages are defined:

enum ndmp_bre_message
{
    NDMP_BRE_CREATE_CONTEXT      = 0x20500500,
    NDMP_BRE_DELETE_CONTEXT      = 0x20500501,
    NDMP_BRE_FLUSH_FILE_HISTORY  = 0x20500502,
    NDMP_BRE_PRESET_MOVER        = 0x20500503,
    NDMP_BRE_RESTART_BACKUP      = 0x20500504,
    NDMP_BRE_GET_SERVER_INFO     = 0x20500505,
    NDMP_BRE_GET_CONTEXT_LIST    = 0x20500506,
    NDMP_BRE_GET_PATH_INFO       = 0x20500507
};

2.3. Error Codes
All possible error codes for NDMP Backup Restart Extension interface are listed below.  This list is a combination of core NDMP error values and new errors defined specifically for backup restart reply messages.  The DMA MUST be implemented to accept any core NDMP error values.
The error codes are referenced by the backup restart interface:
enum ndmp_bre_error
{
    NDMP_NO_ERR                    = 0,
    NDMP_PERMISSION_ERR            = 5,
    NDMP_IO_ERR                    = 7,
    NDMP_ILLEGAL_ARGS_ERR          = 9,
    NDMP_ILLEGAL_STATE_ERR         = 19,
    NDMP_NO_MEM_ERR                = 22,
    NDMP_PRECONDITION_ERR          = 26,
    NDMP_BRE_CONTEXT_NOT_FOUND_ERR = 0x20500500,
    NDMP_BRE_CONTEXT_BUSY_ERR      = 0x20500501,
    NDMP_BRE_INTERNAL_ERR          = 0x20500502,
    NDMP_BRE_NO_SPACE_ERR          = 0x20500503,  
    NDMP_BRE_NO_DISK_SPACE_ERR     = 0x20500504
};

The following list describes each error code.  The errors that are returned in reply to specific requests are described in detail under the relevant message descriptions in section 3.
NDMP_NO_ERR
No error. 
NDMP_PERMISSION_ERR
The file system entity (drive, volume, etc.) is in a state that disallows access to the snapshot.
NDMP_IO_ERR
This general error is used if none of the more specific error codes apply. 
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_ILLEGAL_STATE_ERR
Message cannot be processed in the current state.
NDMP_NO_MEM_ERR
A memory allocation failure was detected. 
NDMP_PRECONDITION_ERR
Message received before a required prerequisite message. 
NDMP_BRE_CONTEXT_NOT_FOUND_ERR
The specified backup context does not exist.
NDMP_BRE_CONTEXT_BUSY_ERR
The backup context on which the operation was attempted was in a busy state.
NDMP_BRE_INTERNAL_ERR
The Data Server encountered an internal error that prevented successful completion of the specified backup restart operation. The error encountered does not correspond to one of the defined core NDMP or NDMP BRE error codes. In this case the Data Server SHOULD issue a NDMP_LOG_MESSAGE containing additional information about the error.
NDMP_BRE_NO_SPACE_ERR
The requested operation failed due to lack of an available snapshot.
NDMP_BRE_NO_DISK_SPACE_ERR
The requested operation failed due to lack of disc space.

3. Data Server Interfaces
This section defines the protocol interfaces implemented by the NDMP server. These interfaces are used by the DMA to manage backup contexts and backup restart operations on the Data Server.
3.1. BRE Interface
The Backup Restart Extension (BRE) interface provides complete control of NDMP Data Server backup contexts and allows resumption of a Data Server backup operation from a known good point in the backup data stream based on a previously created Data Server backup context and backup application identified checkpoint.

3.1.1. NDMP_BRE_CREATE_CONTEXT
This request, which has no arguments, is used by the backup application to instruct the server to create a new backup restart context that will eventually be initialized with Data Server specific information about the subsequent backup operation. 
This request enables the next backup operation in the session (NDMP control connection) for subsequent restarts by returning an opaque context handle. Implicitly this request changes NDMP Data Server behavior by forcing the Data Server to preserve the backup arguments, snapshot and metadata until the backup application explicitly releases the context by issuing a NDMP_BRE_DELETE_CONTEXT.  It is the responsibility of the backup application to ensure contexts are deleted in a timely manner.  Failure to do so will result in unreleased Data Server resources (snapshots, disk space, etc) that may prevent further backups.
Only one new backup context SHOULD be created by the NDMP application prior to issuing a NDMP_DATA_START_BACKUP request since the initial association between the context and the backup operation is implicit. The Data Server MUST prevent the creation of multiple contexts within a given session prior to the invocation of a backup operation.  
This is a blocking operation. As such a reply message will not be issued until the create context operation successfully completes or fails.  
Message XDR definition 
/* NDMP_BRE_CREATE_CONTEXT */
/* this request has no arguments */ 

struct ndmp_bre_create_context_reply 
{
      ndmp_bre_error error;
      opaque         context_handle<>;
};

Reply Arguments
error
Error code.
context_handle
Specifies the opaque context handle identifying this backup restart context.  
Reply Errors
NDMP_NO_ERR
The context create request was successfully processed. An uninitialized context was created and the context_handle contained in this reply message is valid.
NDMP_ILLEGAL_STATE_ERR
The context create request was received while a previous initialized or uninitialized context was outstanding. The context_handle contained in this reply message is not valid.
NDMP_BRE_INTERNAL_ERR
The Data Server encountered an internal error that prevented creation of the context. The Data Server SHOULD issue a NDMP_LOG_MESSAGE containing additional information describing the specific error condition.

3.1.2. NDMP_BRE_DELETE_CONTEXT
This request is used by the DMA to instruct the server to delete the specified backup restart context. The single request argument specifies the context handle returned by a NDMP_BRE_CREATE_CONTEXT or NDMP_BRE_GET_CONTEXT_LIST request.
Upon receipt of a NDMP_BRE_DELETE_CONTEXT request, the Data Server SHOULD validate the request arguments and release all server resources associated with the specified context.  
If the specified context is locked or otherwise in use, the Data Server SHOULD fail the delete request. Note: although not exposed through this extension interface, Data Server implementations SHOULD provide a local mechanism to force the release of locked contexts.
This is a blocking operation. As such a reply message will not be issued until the delete context operation successfully completes or fails.  
Message XDR definition 
/* NDMP_BRE_DELETE_CONTEXT */
struct ndmp_bre_delete_context_request 
{
      opaque         context_handle<>;
}; 

struct ndmp_bre_delete_context_reply 
{
      ndmp_bre_error error;
};

Request Arguments
context_handle
Specifies the opaque context handle identifying the backup restart context to delete.  
Reply Arguments
error
Error code.
Reply Errors
NDMP_NO_ERR
The backup context delete operation was successful and all associated Data Server resources have been released. The backup operation described by the context is no longer restartable.
NDMP_BRE_CONTEXT_NOT_FOUND_ERR
The specified backup context does not exist.
NDMP_BRE_CONTEXT_BUSY_ERR
The backup context on which the operation was attempted was in a busy state.
NDMP_BRE_INTERNAL_ERR
The Data Server encountered an internal error that prevented deletion of the specified context. The Data Server SHOULD issue a NDMP_LOG_MESSAGE containing additional information describing the specific error condition.

3.1.3. NDMP_BRE_FLUSH_FILE_HISTORY
This request is used by the DMA to instruct the data server to send any remaining file history information for the current backup operation. The reply is used by the DMA to determine when all file history has been received.
Some data server implementations queue file history information during the backup operations.  This is typically done to improve transport efficiency by sending more entries per message.
A DMA identifies restart checkpoints by associating byte stream offsets in file history (specified by fh_info) with confirmed tape writes (specified by bytes_moved when in mover paused state). Therefore it is necessary for the DMA to periodically ensure it has received all file history entries. 
This is a blocking operation. As such a reply message will not be issued until all file history information has been returned to the DMA. Since the same NDMP control connection is used to convey both file history and the reply, receipt of the flush reply provides an  explicit indication that the DMA has received all file history.
Message XDR definition 
/* NDMP_BRE_FLUSH_FILE_HISTORY */
/* this request has no arguments */

struct ndmp_bre_flush_file_history_reply 
{
      ndmp_bre_error error;
};

Reply Arguments
error
Error code.
Reply Errors
NDMP_NO_ERR
All file history information has been sent.

3.1.4. NDMP_BRE_PRESET_MOVER
This request is used by the DMA to adjust the mover state to match the selected backup stream restart checkpoint prior to initiating a backup restart operation.  It allows the DMA to preset the mover bytes_moved and record_number state values to reflect the absolute offset of the restart checkpoint. 
A backup failure (typically or always?) results in a data server or mover transition to the halted state and subsequent termination of the backup data connection between the data server and the mover. (In this case) restarting a failed backup operation requires the DMA to establish a new data connection and reconfigure the mover window before issuing a NDMP_BRE_RESTART_BACKUP request. 
Since all state information for the previous mover session is lost when the mover enters the IDLE state, the DMA MUST adjust subsequent NDMP_MOVER_GET_STATE bytes_moved and record_number values to include the valid backup stream data committed to tape prior to the restart checkpoint. 
The NDMP_BRE_PRESET_MOVER request provides an optional mechanism allowing the DMA to update the mover bytes_moved and record_number values before the backup operation is restarted. The request is valid only in the mover IDLE or LISTEN states and only after a valid NDMP_MOVER_SET_WINDOW request. The bytes_moved value SHOULD be set to absolute offset of the restart checkpoint and the record number value to the new bytes_moved value divided by the previous mover record size.
Note: DMA backup restart processing MUST NOT depend on this request since not all NDMP servers, especially standalone tape servers, will support the backup restart extension. In the absence of tape server BRE support, the DMA MUST be capable of dynamically adjusting the bytes_moved and record_number NDMP_MOVER_GET_STATE reply values.
Message XDR definition 
/* NDMP_BRE_PRESET_MOVER */
struct ndmp_bre_preset_mover_request 
{
    ndmp_u_quad  bytes_moved;
    u_long       record_num;
};

struct ndmp_bre_preset_mover_reply 
{
      ndmp_bre_error error;
};

Request Arguments
bytes_moved
Specifies the cumulative number of data stream bytes read from the data connection and written to the tape subsystem at the time of the restart checkpoint.  
record_num
Specifies the last tape record processed by the mover at the time of the restart checkpoint.  
Reply Arguments
error
Error code.
Reply Errors
NDMP_NO_ERR
The preset mover request was successfully processed. The specified mover bytes_moved and record_number values have been updated in the current mover state.
NDMP_ILLEGAL_STATE_ERR
The preset mover request was received while the mover state machine was in a state that prevented processing this request. Set preset mover requests are only valid in the IDLE and LISTEN states.
NDMP_PRECONDITION_ERR
The preset mover request was received prior to establishing a valid mover record_size. 

3.1.5. NDMP_BRE_RESTART_BACKUP
This request is used by the backup application (DMA) to instruct the Data Server to restart a previous backup operation at a specific point in the backup stream.  Request arguments include a handle to a previously created backup context, the restart point specified as an absolute offset within the backup stream, and a file history entry corresponding to the restart point. 
The backup restart mechanism requires that the Data Server generate file history information and that the Data Server implementation of file history generate fh_info values as absolute byte offsets within the backup stream.
The backup application determines restart points by accessing the Tape Server NDMP_MOVER_GET_STATE bytes_moved value at appropriate ‘checkpoints’ during the original backup operation. A checkpoint represents a known good point in the backup operation. A valid backup restart checkpoint requires the following conditions to be met:
1) A file history entry has been received by the backup application for the file or inode corresponding to the bytes_moved data offset.
2) All data up to the bytes_moved offset have actually been written to tape and are not buffered within the tape subsystem. 
3) Sufficient information has been recorded by the backup application to allow discard of any post-checkpoint backup stream data written to tape during the failed backup operation.  
Note: This is especially important if the backup application supports checkpoints at other than full tape boundaries. In this case the backup application MUST be capable of adjusting byte offsets to skip post-checkpoint data during recovery operations.
The Data Server locates the backup stream restart point by first finding the file system entity (file or inode) specified by the file history entry, then calculating the relative offset within the file system entity based on the absolute offset of the entity (file history fh_info) and the absolute offset of the specified restart point. The Data Server invokes the backup method starting at the specified file system entity, discarding data until the relative offset within the entity is reached. 
This is a blocking operation. As such a reply message will not be issued until the restart backup operation successfully completes or fails.  
Message XDR definition 
enum ndmp_bre_fh_type 
{
    NDMP_BRE_FH_NODE_TYPE = 0,
    NDMP_BRE_FH_FILE_TYPE = 1
};

union ndmp_bre_fh_data 
      switch (ndmp_bre_fh_type type) 
{
      case NDMP_BRE_FH_NODE_TYPE:
           ndmp_node  fh_node;
      case NDMP_BRE_FH_FILE_TYPE:
           ndmp_file fh_file;
};

/* NDMP_BRE_RESTART_BACKUP */
struct ndmp_bre_restart_backup_request 
{
      opaque            context_handle<>;
      ndmp_u_quad       restart_offset;
      ndmp_bre_fh_data  fh_data;
}; 

struct ndmp_bre_restart_backup_reply
{
      ndmp_bre_error error;
};

Request Arguments
context_handle
Specifies the backup context to be used by the Data Server when restarting the backup operation. To enable a backup operation for restart support, the backup application (DMA) MUST obtain a context handle from the Data Server prior to sending the original NDMP_DATA_START_BACKUP_REQUEST.
restart_offset
Specifies the backup restart point as an absolute byte offset within the backup stream. More specifically the restart offset identifies the first byte of backup stream data to be generated by the Data Server when restarting the backup.
fh_data
Specifies the Data Server generated file history entry for the file or inode represented by the restart_offset. The fh_data field consists of a union of the same data structure definitions (ndmp_file and ndmp_node) used by the Data Server for file history generation. However there is one important difference. 
Reply Arguments
error
Error code.
Reply Errors
NDMP_NO_ERR
The restart backup request was successfully processed. The Data Server has transitioned to the ACTIVE state and the backup operation represented by the specified context was successfully resumed at the checkpoint identified by the restart_offset. The Data Server will issue a subsequent NDMP_NOTIFY_DATA_HALTED post to indicate the final result of the restarted backup.
NDMP_ILLEGAL_STATE_ERR
The restart backup request was received while the Data Server state machine was in a state that prevented processing this request. Restart backup requests are only valid in the CONNECTED state.
NDMP_ILLEGAL_ARGS_ERR
The restart backup request specified an invalid restart context or fh_data specification.
NDMP_NOT_AUTHORIZED_ERR
The NDMP Server requires DMA authentication, but has not received a valid NDMP_CONNECT_CLIENT_AUTH request from the DMA.
NDMP_PRECONDITION_ERR
The restart backup request was received prior to a successful NDMP_BRE_PRESET_MOVER request.
NDMP_BRE_CONTEXT_NOT_FOUND_ERR
The specified backup restart context does not exist.
NDMP_BRE_CONTEXT_BUSY_ERR
The specified backup restart context in a busy state indicating it is being used by another operation.
NDMP_BRE_INTERNAL_ERR
The Data Server encountered an internal error that prevented successful invocation of the specified backup restart operation. The error encountered does not correspond to one of the defined core NDMP or NDMP BRE error codes. In this case the Data Server SHOULD issue a NDMP_LOG_MESSAGE containing additional information about the error.

3.1.6. NDMP_BRE_GET_SERVER_INFO
This request, which has no arguments, is used by the backup application to obtain information about the Data Server’s specific backup restart capabilities and limitations.  This information should remain static for the duration of an NDMP control connection.
Message XDR definition 
/* NDMP_BRE_GET_SERVER_INFO */
/* ndmp_bre_get_server_info_reply context attributes */
const NDMP_BRE_CONTEXT_IS_PERSISTENT       =0x00000001;
const NDMP_BRE_CONTEXT_INCLUDES_SNAPSHOT   =0x00000002;
const NDMP_BRE_CONTEXT_IS_SERVER_DELETEABLE =0x00000004;

/* ndmp_bre_get_server_info_reply restart attributes */
const NDMP_BRE_RESTART_AT_FILE_ALLOWED            =0x00000001;
const NDMP_BRE_RESTART_AT_NODE_ALLOWED            =0x00000002;
const NDMP_BRE_RESTART_AT_SUBFILE_ALLOWED         =0x00000004;
const NDMP_BRE_RESTART_AT_SUBNODE_ALLOWED         =0x00000008;

/* NDMP_BRE_GET_SERVER_INFO */
/* this request has no arguments */ 

struct ndmp_bre_get_server_info_reply 
{
      ndmp_bre_error  error;
      u_long          context_attributes;
      u_long          restart_attributes;
      u_long          maximum_contexts;
      u_long          restart_block_size;
};

Reply Arguments
error
Error code.
context_attributes
A bit field describing Data Server specific backup restart context capabilities and limitations.
NDMP_BRE_CONTEXT_IS_PERSISTENT
Asserting this bit indicates that the Data Server backup context is persistent across system reboots.
NDMP_BRE_CONTEXT_INCLUDES_SNAPSHOT
Asserting this bit indicates Data Server backup context includes a snapshot or other point in time image of the file system being backed up.  Implementations that backup from the live file system MUST NOT assert this bit.

NDMP_BRE_CONTEXT_IS_SERVER_DELETEABLE
Asserting this bit indicates Data Server is capable of independently deleting a backup context.  Context deletion may be invoked via a Data Server specific UI or as part of an automated Data Server resource recovery mechanism.
restart_attributes
A bit field indicating the Data Server specific granularity at which backup restarts can be specified. This implies the boundaries at which the backup application can identify backup checkpoints.
NDMP_BRE_RESTART_AT_FILE_ALLOWED
Asserting this bit indicates the Data Server is capable of restarting a backup operation at any file boundary.
NDMP_BRE_RESTART_AT_NODE_ALLOWED
Asserting this bit indicates the Data Server is capable of restarting a backup operation at any node boundary.
NDMP_BRE_RESTART_AT_SUBFILE_ALLOWED
Asserting this bit indicates the Data Server is capable of restarting a backup operation at any block boundary within a file. Block 
NDMP_BRE_RESTART_AT_SUBNODE_ALLOWED
Asserting this bit indicates the Data Server is capable of restarting a backup operation at any block boundary within a node.
maximum_contexts
Specifies the maximum number of concurrent contexts the Data Server can support. If the Data Server does not impose a specific limit on the number of concurrent backup contexts, this value MUST be set to all ones (binary).
restart_block_size
Specifies the minimum granularity of backup restart operations supported by the Data Server in terms of file or node block size. The restart_block_size field is valid only if the restart attribute NDMP_BRE_RESTART_AT_SUBFILE_ALLOWED or NDMP_BRE_RESTART_AT_SUBNODE_ALLOWED bits are asserted.
For example, a restart_block_size value of 1024 indicates the data server is capable of restarts at any 1k boundary within the backup stream. A value of 1 indicates restart operations are allowed at any byte boundary. A value of zero is invalid. 
The DMA MUST set the mover record_size to an integer multiple of the restart_block_size.
Reply Errors
NDMP_NO_ERR
The get server_info request was successfully processed. The reply message accurately represents the Data Server’s backup restart capabilities.

3.1.7. NDMP_BRE_GET_CONTEXT_LIST
This request, which has no arguments, is used by the backup application to obtain a list of all backup restart contexts and associated context state currently maintained by the Data Server.
Each backup context is identified by an opaque context handle. Context state consists of the context creation date, the backup operation arguments previously specified by the backup application via a NDMP_DATA_START_BACKUP_REQUEST, the results of the backup operation, and a context status value.
This request provides a mechanism allowing backup applications to verify their implementation specific context management, and to cleanup Data Server backup context information following backup application failures.
This is a blocking operation. As such a reply message will not be issued until the get context list operation successfully completes or fails.  
Note: Data servers MAY delete backup contexts to recover system resources as needed to maintain overall system integrity and performance. Whenever possible, data servers SHOULD preserve DMA created backup contexts and delegate responsibility for context deletion to the DMA.
Message XDR definition 
/* ndmp_bre_context_state status bits */
const NDMP_BRE_CONTEXT_STATUS_BACKUP_ARGS_VALID     =0x00000001;
const NDMP_BRE_CONTEXT_STATUS_BACKUP_RESULTS_VALID  =0x00000002;
const NDMP_BRE_CONTEXT_STATUS_BACKUP_RESTARTABLE    =0x00000004;
const NDMP_BRE_CONTEXT_STATUS_CONTEXT_BUSY          =0x00000008;

struct ndmp_bre_context_state 
{
      opaque                          context_handle<>;
      u_long                          context_timestamp;
      u_long                          context_status;
      ndmp_data_start_backup_request  backup_args;
      ndmp_data_halt_reason           backup_results;
};

/* NDMP_BRE_GET_CONTEXT_LIST */
/* this request has no arguments */ 

struct ndmp_bre_get_context_list_reply 
{
      ndmp_bre_error          error;
      ndmp_bre_context_state  context_list<>;
};

Reply Arguments
error
Error code.
context_list
An array of zero or more ndmp_bre_context_state structures containing one structure member for each backup restart context maintained by the Data Server.
The ndmp_bre_context_state structure contains the following fields:
context_handle
Represents the opaque context handle identifying this backup restart context.  This is the same value returned by NDMP_BRE_CREATE_CONTEXT.
context_timestamp
represents the date and time the context was created as seconds since 00:00:00 GMT, Jan 1, 1970.
context_status
A bit field indicating the status of the backup context.
NDMP_BRE_CONTEXT_STATUS_BACKUP_ARGS_VALID
Asserting this bit indicates a backup operation represented by this context was successfully invoked. The backup_args field contains the arguments specified by the backup application in the NDMP_DATA_START_BACKUP_REQUEST.
NDMP_BRE_CONTEXT_STATUS_BACKUP_RESULTS_VALID
Asserting this bit indicates the backup operation represented by this context has completed. The backup_results field contains the success or failure indication previously sent by the Data Server to the backup application via a NDMP_NOTIFY_DATA_HALTED post message. 
NDMP_BRE_CONTEXT_STATUS_BACKUP_RESTARTABLE
Asserting this bit indicates the backup operation represented by this context has progressed to the point that it SHOULD be restartable, however it does not guarantee a restart backup operation will be successful.
Note: The point at which a backup becomes restartable is Data Server implementation specific.  Since the backup restart interface depends on the file history mechanism to establish restart points, a backup will not be restartable from points within the data stream which are not described by file history entries.  Implementation specific methods may be required to communicate to the backup application whether or not a backup may be restarted from a particular point in the data stream.  
For example, for NetApp servers, additional data is written to the data stream after the data which is described by file history entries. When the data server reaches the point in the data stream which is no longer described by file history, it sets an environment variable, ACL_START.  ACL_START has as its value the offset in the data stream at which the non-restartable portion of the data stream begins.  The backup application can determine whether or not the ACL_START environment variable has been set, and its value, by sending an NDMP_DATA_GET_ENV.
NDMP_BRE_CONTEXT_STATUS_CONTEXT_BUSY
Asserting this bit indicates this backup context is currently being referenced by one or more Data Server operations. Deleting a busy context may result in a Data Server failure. A Data Server MAY prevent backup application deletion of busy backup contexts. 
backup_args
An ndmp_data_start_backup_request structure containing the backup arguments specified by the backup application (DMA) when the NDMP_DATA_START_BACKUP_REQUEST was issued. Refer to section 3.5.3 of the NDMP v4 specification for a detailed definition of the backup arguments represented by the ndmp_data_start_backup_request structure.
It is possible for a backup context to exist without associated backup request arguments.  This occurs when a context is created, yet a valid NDMP_DATA_START_BACKUP request has not been received by the Data Server. The context_status NDMP_BRE_CONTEXT_STATUS_BACKUP_ARGS_VALID bit indicates whether the backup_args field contains valid data.
backup_results
A ndmp_data_halt_reason enum value representing the results of the previous backup operation.  This is the same value reported by the ndmp_notify_data_halted_post message. Refer to section 4.1.1 of the NDMP v4 specification for a detailed definition of the backup result represented by the ndmp_notify_data_halted_post message.
It is possible for a backup context to exist without associated backup result.  This occurs when a context is created, but the backup operation has either not been invoked or is still in progress. The context_status NDMP_BRE_CONTEXT_STATUS_BACKUP_RESULTS_VALID bit indicates whether the backup_results field contains is valid data.
Reply Errors
NDMP_NO_ERR
The get context list operation was successful. The returned context_list is complete and accurate.
NDMP_BRE_INTERNAL_ERR
The Data Server encountered an internal error that prevented the return of the context_list. The Data Server SHOULD issue a NDMP_LOG_MESSAGE containing additional information describing the specific error condition.

3.1.8. NDMP_BRE_GET_PATH_INFO
This request is used by the backup application to obtain information about a path name. Specifically this request will return information about the restartability of a backup of this path. 
This request provides a mechanism allowing backup applications to determine, before they initiate a Data Start Backup, whether or not a restartable backup of this path would be allowed by the Data Server.  For some Data Servers, restartable backups will not be supported on volumes/paths with certain characteristics that would not otherwise be visible to backup applications.
This is a blocking operation.  As such, a reply message will not be issued until the get path info operation successfully completes or fails.
Message XDR definition 
/* ndmp_bre_get_path_info_reply path_attributes */
const NDMP_BRE_PATH_RESTARTABLE            =0x00000001;

/* NDMP_BRE_GET_PATH_INFO */

struct ndmp_bre_get_path_info_request 
{
      string          path<>;
};

struct ndmp_bre_get_path_info_reply 
{
      ndmp_bre_error  error;
      u_long          path_attributes;
};

Request Arguments
        path
Specifies the path to be checked for attributes.  It is of the same form as the path for the value of the FILESYSTEM environment variable.
Reply Arguments
error
Error code.
path_attributes
A bit field describing path specific attributes.
NDMP_BRE_PATH_RESTARTABLE
Asserting this bit indicates that a backup specified by this path name would be restartable at this time.  There is no guarantee that it will be restartable at a future time.

Reply Errors
NDMP_NO_ERR
The get_path_info request was successfully processed. The reply message accurately represents the input path’s characteristics.
NDMP_ILLEGAL_ARGS_ERR
Message received containing one or more invalid arguments.
NDMP_BRE_INTERNAL_ERR
The Data Server encountered an internal error that prevented successful completion of the request.  The error encountered does not correspond to one of the defined core NDMP or NDMP BRE error codes.  In this case the Data Server SHOULD issue an NDMP_LOG_MESSAGE containing additional information about the error.

4. Data Management Application Interfaces
This section defines the protocol interfaces implemented by the Data Management Application (DMA). Note that these unsolicited Data Server generated messages do not have a reply associated with them. Rather they allow one-way communication of diagnostic, error, event and state information from the Data Server to the DMA.
Currently the backup restart extension does not include any DMA interface definitions.
5. References
[1] draft-skardal-ndmpv4-02.txt, "Network Data Management Protocol Version 4", H. Skardal, Network Appliance, April 2001
6. Security
TBD
7. Recognition of Prior Work
TBD
8. Authors and Contributors
8.1. Document Editor

Greg Linn
Network Appliance Inc.
Telephone: 408 822 3752
Email: Greg.Linn@netapp.com

8.2. Authors' Addresses

Greg Linn
Network Appliance Inc.
Telephone: 408 822 3752
Email: Greg.Linn@netapp.com

Bill Taylor
Network Appliance Inc.
Telephone: 408 822 6383
Email: Bill.Taylor@netapp.com

Jim Ward
Reliaty Inc.
Telephone: 603 672-8600
Email: jimw@reliaty.com

8.3. Contributors
In addition to the authors, the following people have helped the effort by participating in the reviews:
Tbd

Appendix A: XDR Definitions
Although there are certainly many very different XDR files that could define an NDMP specification, one is included below for purposes of reference.
enum ndmp_bre_message
{
    NDMP_BRE_CREATE_CONTEXT      = 0x20500500,
    NDMP_BRE_DELETE_CONTEXT      = 0x20500501,
    NDMP_BRE_FLUSH_FILE_HISTORY  = 0x20500502,
    NDMP_BRE_PRESET_MOVER        = 0x20500503,
    NDMP_BRE_RESTART_BACKUP      = 0x20500504,
    NDMP_BRE_GET_SERVER_INFO     = 0x20500505,
    NDMP_BRE_GET_CONTEXT_LIST    = 0x20500506,
    NDMP_BRE_GET_PATH_INFO       = 0x20500507
};

enum ndmp_bre_error
{
    NDMP_NO_ERR                    = 0,
    NDMP_PERMISSION_ERR            = 5,
    NDMP_IO_ERR                    = 7,
    NDMP_ILLEGAL_ARGS_ERR          = 9,
    NDMP_ILLEGAL_STATE_ERR         = 19,
    NDMP_NO_MEM_ERR                = 22,
    NDMP_PRECONDITION_ERR          = 26,
    NDMP_BRE_CONTEXT_NOT_FOUND_ERR = 0x20500500,
    NDMP_BRE_CONTEXT_BUSY_ERR      = 0x20500501,
    NDMP_BRE_INTERNAL_ERR          = 0x20500502,
    NDMP_BRE_NO_SPACE_ERR          = 0x20500503,  
    NDMP_BRE_NO_DISK_SPACE_ERR     = 0x20500504
};

/* NDMP_BRE_CREATE_CONTEXT */
/* this request has no arguments */ 

struct ndmp_bre_create_context_reply 
{
      ndmp_bre_error error;
      opaque         context_handle<>;
};

/* NDMP_BRE_DELETE_CONTEXT */
struct ndmp_bre_delete_context_request 
{
      opaque         context_handle<>;
}; 

struct ndmp_bre_delete_context_reply 
{
      ndmp_bre_error error;
};

enum ndmp_bre_fh_type 
{
    NDMP_BRE_FH_NODE_TYPE = 0,
    NDMP_BRE_FH_FILE_TYPE = 1
};

union ndmp_bre_fh_data 
      switch (ndmp_bre_fh_type type) 
{
      case NDMP_BRE_FH_NODE_TYPE:
           ndmp_node  fh_node;
      case NDMP_BRE_FH_FILE_TYPE:
           ndmp_file fh_file;
};

/* NDMP_BRE_FLUSH_FILE_HISTORY */
/* this request has no arguments */

struct ndmp_bre_flush_file_history_reply 
{
      ndmp_bre_error error;
};

/* NDMP_BRE_PRESET_MOVER */
struct ndmp_bre_preset_mover_request 
{
    ndmp_u_quad  bytes_moved;
    u_long       record_num;
};

struct ndmp_bre_preset_mover_reply 
{
      ndmp_bre_error error;
};

/* NDMP_BRE_RESTART_BACKUP */
struct ndmp_bre_restart_backup_request 
{
      opaque            context_handle<>;
      ndmp_u_quad       restart_offset;
      ndmp_bre_fh_data  fh_data;
}; 

struct ndmp_bre_restart_backup_reply
{
      ndmp_bre_error error;
};

/* NDMP_BRE_GET_SERVER_INFO */
/* ndmp_bre_get_server_info_reply context attributes */
const NDMP_BRE_CONTEXT_IS_PERSISTENT        =0x00000001;
const NDMP_BRE_CONTEXT_INCLUDES_SNAPSHOT    =0x00000002;
const NDMP_BRE_CONTEXT_IS_SERVER_DELETEABLE =0x00000004;

/* ndmp_bre_get_server_info_reply restart attributes */
const NDMP_BRE_RESTART_AT_FILE_ALLOWED            =0x00000001;
const NDMP_BRE_RESTART_AT_NODE_ALLOWED            =0x00000002;
const NDMP_BRE_RESTART_AT_SUBFILE_ALLOWED         =0x00000004;
const NDMP_BRE_RESTART_AT_SUBNODE_ALLOWED         =0x00000008;

/* NDMP_BRE_GET_SERVER_INFO */
/* this request has no arguments */ 

struct ndmp_bre_get_server_info_reply 
{
      ndmp_bre_error  error;
      u_long          context_attributes;
      u_long          restart_attributes;
      u_long          maximum_contexts;
      u_long          restart_block_size;

};

/* ndmp_bre_context_state status bits */
const NDMP_BRE_CONTEXT_STATUS_BACKUP_ARGS_VALID     =0x00000001;
const NDMP_BRE_CONTEXT_STATUS_BACKUP_RESULTS_VALID  =0x00000002;
const NDMP_BRE_CONTEXT_STATUS_BACKUP_RESTARTABLE    =0x00000004;
const NDMP_BRE_CONTEXT_STATUS_CONTEXT_BUSY          =0x00000008;

struct ndmp_bre_context_state 
{
      opaque                          context_handle<>;
      u_long                          context_timestamp;
      u_long                          context_status;
      ndmp_data_start_backup_request  backup_args;
      ndmp_data_halt_reason           backup_results;
};

/* NDMP_BRE_GET_CONTEXT_LIST */
/* this request has no arguments */ 

struct ndmp_bre_get_context_list_reply 
{
      ndmp_bre_error          error;
      ndmp_bre_context_state  context_list<>;
};

/* ndmp_bre_get_path_info_reply path_attributes */
const NDMP_BRE_PATH_RESTARTABLE            =0x00000001;

/* NDMP_BRE_GET_PATH_INFO */

struct ndmp_bre_get_path_info_request 
{
      string          path<>;
};

struct ndmp_bre_get_path_info_reply 
{
      ndmp_bre_error  error;
      u_long          path_attributes;
};

/* END */