N/A Matt Benjamin Internet-Draft Expires: March 23, 2010 September 23, 2009 AFS Callback Extensions draft-benjamin-extendedcallbackinfo-00 Status of this Memo This document specifies a standards track protocol extension for the OpenAFS community, and requests discussion and suggestions for improvements. Thanks to Jeffrey Altman, Tom Keiser, Jeffrey Hutzelman, Derrick Brashear, and Steven Jenkins for their feedback and suggested improvements from previous drafts. Thanks to participants at the 2009 Hackathon in Edinburgh for their feedback and assistance. This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on March 23, 2010. Copyright Notice Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Key Words The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in Internet Engineering Task Force RFC 2119. Abstract AFS cache-control strategy is callback (invalidate) based. The AFS callback design allows a client to know when an object it has cached is no longer consistent, but the callback notification message itself provides no specific information about the triggering event. This is a protocol inefficiency, as in several scenarios it results in unnecessary round-trips to file servers to verify file status information, file access information, or to fetch file data which has not changed. We propose an extension of the callback mechanism to provide information about the event(s) triggering a callback, in the payload of the callback notification message itself. The proposed mechanism eliminates most or all unnecessary round-trips imposed by the current callback mechanism, and simultaneously allows AFS implementations to (efficiently) provide correct semantics in several scenarios involving multiple writers (ie, where AFS currently provides incorrect semantics). Benjamin Expires March 23, 2009 [Page 1] Internet-Draft Extended Callback Information September 2009 Editorial Note To provide feedback on this Internet-Draft, join the afs3-standardisation mailing list (afs3-standardization@openafs.org). Table of Contents Status of this Memo Key Words Abstract 1 Introduction 2 The AFS Callback Mechanism 2.1 Description 2.2 Analysis 3 Extended Callback Interface 3.1 Backward Compatibility 3.2 Interface Changes 3.2.1 Procedures 3.2.2 Constants All Sequences AFSXCBInvocation AFSExtendedCallBack AFSExtendedCallBackResult Sequence Types 3.2.3 Data Types HostIdentifier AFSXCBInvocation Fid Flags lowDV highDV ExpirationTime CallBacks_Array AFSExtendedCallBack Flags and ExtraFlags Origin NCoalesced DataVersion Data AFSCBFileStatus AFSCBDirStatus AFSCB_NotificationData 3.3 Semantic Changes Edinburgh Consensus 3.3.1 DataVersion Rule 3.4 Callback Invocations 3.4.1 AFSXCBInvocation Origin ExpirationTime 3.4.2 AFSExtendedCallBack Flags ExtraFlags DataVersion Data 3.4.3 Reasons for Cancellation AFSCB_Cancel_Shutdown AFSCB_Cancel_CallbackGC AFSCB_Cancel_VolumeOffline AFSCB_Cancel_VolumeMoved AFSCB_Cancel_LostMyMind AFSCB_Cancel_IHateYou 3.4.4 ExtendedCallBack Procedure 3.4.5 Callback Coalescing Call Consolidation (Sequences of Notifications) Coalescing of Equivalent Notifications Implementation Note 3.4.6 Asynchronous Delivery Edinburgh Consensus 3.4.7 AFSCB_Event_StoreData 3.4.8 AFSCB_Event_StoreACL 3.4.9 AFSCB_Event_StoreStatus 3.4.10 AFSCB_Event_CreateFile 3.4.11 AFSCB_Event_MakeDir 3.4.12 AFSCB_Data_Symlink 3.4.13 AFSCB_Event_Link 3.4.14 AFSCB_Event_RemoveFile 3.4.15 AFSCB_Event_RemoveDir 3.4.16 AFSCB_Event_Rename 3.4.17 AFSCB_Event_Deleted 3.4.18 AFSCB_Event_ReleaseLock 3.5 Callbacks And Read-Only Volume Replicas 3.5.1 Constants AFSCB_Flag_Release AFSCB_IFlag_Release AFSCB_Release_WholeVolumeCancel 3.5.2 Semantic Changes 3.6 Security Implications Edinburgh Consensus 4 Appendix A: XDR Grammar 1 Introduction The AFS protocol provides a comprehensive framework for scalable, secure, wide-area file sharing over IP networks. The AFS system has historically distinguished itself through its emphasis on scalability, a key source of which is client-side caching[1, 4]. File data, file and directory metadata, and access control information may all be cached. Cache consistency is maintained through client registration and an associated asynchronous notification mechanism known as the callback. The current AFS consistency model (which is of larger scope than the callback mechanism, eg, it includes AFS sync-on-close semantics) has allowed AFS to scale to large numbers of clients (tens of thousands today), and to perform well under the workloads for which AFS was originally designed. However, AFS does not perform efficiently under other conditions, such as when more than one client is interested in a file which is changing--even if the file has only one writer, and many readers[footnote: NFSv4.1 in particular efficiently supports this scenario with byte-range delegation, see[9]. ]. In general, the AFS protocol arguably (still, considering improvements made between AFS-2 and AFS-3) places too little emphasis on efficient caching of mutable data. The current AFS consistency model is insufficient to correctly support single-file, multiple-writer scenarios, including those required for POSIX semantics, and therefore is insufficient to support many applications which may be run correctly on competing distributed file systems (e.g., CIFS, Novell Netware, or NFSv4). The efficiency of the current AFS cache management algorithm could be substantially improved if specific triggering event information and current status were included in the payload of the callback notifications sent to clients. In particular, inclusion of the current DataVersion number and affected byte ranges in response to StoreData operations would significantly reduce the need for cache revalidation and reconstruction traffic in response to callbacks--in many cases, altogether. These changes would allow efficient support for single-writer updates on a file with multiple readers. More importantly, they would permit AFS to correctly and efficiently support multiple writers updating disjoint ranges on a single file, a prerequisite for supporting granular file locking (and applications which require it) in future. 2 The AFS Callback Mechanism 2.1 Description When an AFS-3 client contacts a file server to perform any of several operations on a file, or explicitly to fetch its status, the file server includes in its RPC response an AFSCallBack structure, representing the server's promise to call back the client "if any modifications are made to the data in the file".[footnote: A key paper on AFS-2 has "before allowing a modification by any other workstation"[1]. The wording of this statement appears calculated to imply that the file server's promise to execute callbacks synchronously with the triggering operations (e.g., StoreData) specifically constitutes part of the AFS cache consistency guarantee. In our analysis, it does not, though it does contribute strongly to the simplification of the file server design and to reduction of file server workload. ]The AFSCallBack structure contains the callback expiration time, and two integer values treated as invariants. When any client executes an operation which would change a file (e.g., StoreData), and in a variety of other situations, the file server invalidates the client's cached copy by executing a call to the CallBack[footnote: formerly BreakCallback ] procedure in the client's RPC interface. (The call includes in its arguments an AFSCallBack structure for each file being invalidated. However, the value of the passed AFSCallBack is unused [e.g., afs/afs_callback.c:643 ff., openafs-1.5.54]). Between the time of issue and either expiry or receipt of a callback, the client may consider any information it has cached on a file to be consistent with the file server's on-disk copy. Conversely, on receipt of a callback, the client must consider that it knows nothing about the file. Thus the client must re-establish a relationship with the file at the file server before executing any further operations on it. The AFS callback mechanism obviates the need for clients to send frequent cache validation requests before performing operations on their locally cached copies of objects, reducing network traffic as well as file server workload[4]. The callback innovation has been since taken up, with variations, by other distributed file system protocols[2, 3, 5]. 2.2 Analysis The AFS callback mechanism reliably notifies clients when information they may have cached becomes invalidated, but omits to send information it trivially knows, ie, the triggering event, that could certainly be used by the client to more efficiently manage cache state. For example, consider the case where 2 clients A and B are interested in a file F, each having read chunks 1-15 into cache. Now another client C initiates a change in the file, writing a new state to chunk 45. This event increments the dataversion of the file, and triggers a callback to A and B. (C, because it initiated the change, is not called back.) On receipt of the callback, A and B must issue FetchStatus requests on F to acquire its current status information, including its current data version. Since the data version of F has increased, any chunks of F which A or B has cached are invalidated, including 30 chunks correctly cached. Should A or B remain interested, it must refetch these chunks (up to 2 megabytes of data, in this case). This scenario will occur reasonably often in environments where mutable data is common, and a related scenario involving directory entries (omitted for brevity) is much more common. In these cases, an AFS callback mechanism capable of sending triggering event information with the callback would have facilitated a more efficient result, at small marginal cost. In another set of scenarios where a client A has changed data in a file invalidated by non-overlapping stores by B, a revised mechanism would be capable of delivering a correct result, whereas a correct result would be impossible with the mechanism in AFS today. (In the AFS-3 callback model, either As or Bs changes must be rejected. In the extended callback model, the range-based invalidate mechanism means that As and Bs changes will be merged, as they are disjoint.) The justification for sending minimal information with the callback is presumably to minimize the execution cost of the callback procedure. The increased cost of sending a limited but informative callback notification to clients, relative to sending an uninformative one, is small. Analysis of the OpenAFS file server code reveals that the file server always has the information that would logically be sent as extended callback information in response to file operations (e.g., file ranges affected by StoreData operations, or changed entries for various directory modification operations). For these reasons, enhancement of the AFS callback interface to supply triggering event information seems likely to improve both correctness and performance of AFS implementations, and experimental implementation and profiling appear justified. 3 Extended Callback Interface 3.1 Backward Compatibility AFS clients will indicate their preference to receive extended callback notifications through a new client capability flag: const CLIENT_CAPABILITY_EXT_CALLBACK = 0x0002; 3.2 Interface Changes 3.2.1 Procedures We propose a new procedure ExtendedCallBack in the client's RPC interface. The ExtendedCallBack procedure arguments consist of a HostIdentifier containing the UUIDs of the sending fileserver and of its cell, and a (variable-length) sequence of AFSXCBInvocation structures. And AFSXCBInvocation represents a (variable-length) sequence of AFSExtendedCallBack events on one AFSFid at Server. One invocation of the ExtendedCallBack procedure can thus deliver up to AFSXCBMAX event notifications on each of up to AFSXCBMAX fids. An OUT-direction sequence of variant AFSExtendedCallBackResult structures is added for future callback notification styles (e.g., locks, delegations) which may return structured data on receipt of notifications: proc ExtendedCallBack( IN HostIdentifier *Server, IN AFSXCBInvocationSeq *Invocations_Array, OUT AFSExtendedCallBackRSeq *CallBack_Result_Array ) multi = 65540; 3.2.2 Constants All Sequences The AFSXCBMAX constant is the maximum allowed length for AFSXCBInvocation and AFSExtendedCallBack sequences: const AFSXCBMAX = 512; AFSXCBInvocation The following constants are flag values are used as flag values on AFSXCBInvocation instances: const AFSCB_IFlag_SOrigin = 1; /* callbacks on this invocation have a single origin host */ const AFSCB_IFlag_Release = 2; /* this invocation was triggered by a volume release */ AFSExtendedCallBack As detailed in section [sub:Data-Types], an AFSExtendedCallBack is an XDR union, discriminated on the callback event type. The following callback event types are defined: const AFSCB_Event_Cancel = 1; /* extended */ const AFSCB_Event_StoreData = 2; /* data in file changed */ const AFSCB_Event_StoreACL = 3; /* ACL changed on vnode */ const AFSCB_Event_StoreStatus = 4; /* status stored on vnode */ const AFSCB_Event_CreateFile = 5; /* file created in directory vnode */ const AFSCB_Event_MakeDir = 6; /* dir created in directory vnode */ const AFSCB_Event_Symlink = 7; /* symlink created in directory vnode */ const AFSCB_Event_Link = 8; /* hard link created in directory vnode */ const AFSCB_Event_RemoveFile = 9; /* file removd from directory vnode */ const AFSCB_Event_RemoveDir = 10; /* dir removed from directory vnode */ const AFSCB_Event_Rename = 11; /* object renamed (moved) */ const AFSCB_Event_Deleted = 12; /* object no longer exists, ex object */ const AFSCB_Event_ReleaseLock = 13; /* traditional AFS lock released */ A flag constant is provided to indicate callback cancellation along with an extended notification message of any of the above types: const AFSCB_Flag_Cancel = 1; /* Callback promise is cancelled */ The following constants indicate reasons for cancellation, when (Flags & AFSCB_Flag_Cancel) const AFSCB_Cancel_Shutdown = 1; const AFSCB_Cancel_CallbackGC = 2; const AFSCB_Cancel_VolumeOffline = 3; const AFSCB_Cancel_VolumeMoved = 4; const AFSCB_Cancel_LostMyMind = 5; const AFSCB_Cancel_IHateYou = 6; The following constants indicate direction (from or to called back FID) in the atomic AFSCB_Event_Rename notification: const AFSCB_Rename_From = 1; const AFSCB_Rename_To = 2; AFSExtendedCallBackResult The following constants a provided as descriminator for the AFSCB_ResultData union: const AFSCB_Result_NoResult = 1; /* void result type */ const AFSCB_Result_Diag = 2; /* diagnostic result type (string) */ Sequence Types The following sequences are defined, and are used to construct the arguments for the ExtendedCallBack procedure: typedef AFSExtendedCallBack AFSExtendedCallBackSeq; typedef AFSExtendedCallBackResult AFSExtendedCallBackRSeq; 3.2.3 Data Types HostIdentifier A HostIdentifier structure contains the unique server and cell UUIDs of a specific host in some AFS cell. struct HostIdentifier { afsUUID ServerUuid; afsUUID CellUuid; }; AFSXCBInvocation The AFSXCBInvocation data type represents a sequence of 0 or more callback events on one fid. The enclosed AFSExtendedCallBack objects MUST be in DataVersion order. Fid Fid is the fid object of the callback sequence. Flags Flags provide specializing information about the invocation. lowDV The lowest data version of Fid at all events in the sequence. highDV The highest data version of Fid at all events in the sequence. ExpirationTime ExpirationTime indicates a new expiration time for the receiving client's callback on fid. And ExpirationTime of 0 indicates no change in ExpirationTime. CallBacks_Array A sequence of 0 or more AFSExtendedCallBack notifications on FID. struct AFSXCBInvocation { AFSFid Fid; afs_uint32 Flags; afs_uint64 lowDV; /* lowest DV at invocation */ afs_uint64 highDV; /* highest */ afs_uint64 ExpirationTime; /* new expiration, or 0 if unchanged */ AFSExtendedCallBackSeq CallBacks_Array; }; AFSExtendedCallBack The AFSExtendedCallBack data type represents a single callback event on some fid, that of its containing AFSXCBInvocation when sent with an ExtendedCallBack RPC. Flags and ExtraFlags Flags and ExtraFlags (added for future expansion) provide possibly event-specific information. Origin Origin is the UUID of the host or server which originated the event, ie, the client whose operation on fid triggered some event, in the typical case. If the origin is unknown to the server or would not be meaningful, it MAY send the null UUID. NCoalesced As specified later in this document, certain operations (ie, StoreData, StoreStatus) MAY be regarded by the file server as idempotent and sent as one callback. NCoalesced indicates the number of equivalent or combined operations coalesced on the event, or 0 if the event is singular. DataVersion DataVersion is the (possibly updated) data version of fid at the completion of the operation which triggered the event. Considering coalescing, DataVersion is the data version at the completion of the first event in the coalesced sequence. Data Data is an object of the discriminated union type AFSCB_NotificationData: struct AFSExtendedCallBack { afs_uint32 Flags; afs_uint32 ExtraFlags; afsUUID Origin; /* originator of changes */ afs_uint32 NCoalesced; /* calls combined on this */ afs_uint64 DataVersion; AFSCB_NotificationData Data; }; A non-zero value in Flags for the AFSCB_Flag_Cancel bit indicates cancellation of the callback upon receipt of the message. In that event, a non-zero value of ExtraFlags indicates the reason for the cancellation. AFSCBFileStatus The AFSCBFileStatus structure is a reduced-footprint AFSFetchStatus replacement intended to communicate changed vnode information in response to StoreData operations: struct AFSCBFileStatus { afs_uint32 LinkCount; afs_uint64 ClientModTime; }; AFSCBDirStatus The AFSCBDirStatus structure is a reduced-footprint AFSFetchStatus replacement intended to communicate changed vnode information in response to directory change operations: struct AFSCBDirStatus { afs_uint32 LinkCount; afs_uint64 ClientModTime; }; AFSCB_NotificationData AFSCB_NotificationData is a union discriminated by callback event type, ie, its value may be any of the constants defined in section [sub:Constants]. union AFSCB_NotificationData switch (afs_uint32 Event_Type) { case AFSCB_Event_StoreData: AFSCB_Data_StoreData u_store_data; case AFSCB_Event_StoreACL: void; case AFSCB_Event_StoreStatus: AFSCB_Data_StoreStatus u_store_status; case AFSCB_Event_CreateFile: AFSCB_Data_CreateFile u_create_file; case AFSCB_Event_MakeDir: AFSCB_Data_MakeDir u_make_dir; case AFSCB_Event_Symlink: AFSCB_Data_Symlink u_symlink; case AFSCB_Event_Link: AFSCB_Data_Link u_link; case AFSCB_Event_RemoveFile: AFSCB_Data_RemoveFile u_remove_file; case AFSCB_Event_RemoveDir: AFSCB_Data_RemoveDir u_remove_dir; case AFSCB_Event_Rename: AFSCB_Data_Rename u_rename; case AFSCB_Event_Deleted: void; case AFSCB_Event_ReleaseLock: AFSCB_Data_Lock u_lock; case AFSCB_Event_Cancel: void; }; The types for the variant member u_data are enumerated and discussed in detail in section [sub:Callback-Invocations]. 3.3 Semantic Changes A file server MAY send traditional callback messages, with traditional semantics, to any AFS client in response to any event. A file server MAY send extended callback notifications to any client which has announced the capability to use the extended interface, with the following semantics: * extended callback notification messages, in general, preserve the file server's callback promise to send further notifications for the called-back FID * the file server MAY revoke the callback promise with any extended callback notification message, by setting the AFSCB_Flag_Cancel bit in the Flags member of the AFSExtendedCallBack structure * the AFSCB_Event_Cancel message is similar to a traditional AFS callback, breaking the callback promise, and requesting the client not request further status on the FID Edinburgh Consensus The preference to preserve the fileserver's callback promises to clients across file operations is a significant behavioral change. In particular, file servers configured with traditional small callback databases may be vulnerable to callback exhaustion, which in turn may lead to thrashing or other undesirable behavior. Sites are recommended to explicitly increase provisioned callbacks, likely to at least 1 million. The OpenAFS viced implementation will increase the number of callbacks in the "large" configuration (-L) to a new, large value in a future release. 3.3.1 DataVersion Rule The various extended callback notification messages include information a client may use to selectively invalidate or reconstruct its cache. In interpreting each message, the client MUST observe the dataversion rule, which states: If the client's cached DataVersion is DataVersion or (DataVersion-1), the client MAY invalidate or update its cache using the type-dependent information contained in the message. In all other cases, the client MUST regard the message as equivalent to a traditional AFS callback. The semantics of specific callback events are enumerated in section [sub:Callback-Invocations]. 3.4 Callback Invocations The various extended callback notification types generally respond to specific events at the file server, but present a view of it relevant to a specific callback promise at one client. In one case (ie, AFSCB_Event_Rename), the file server is sending notification of an event which effects two FIDs, either or both of which may be cached by the receiving client. A structure of type AFSExtendedCallBack is sent with each extended callback notification message, as noted above. Unless otherwise noted, FID is the FID of the object that is the subject of the callback. 3.4.1 AFSXCBInvocation Origin A file server MAY omit to send extended callback notifications triggered by a file operation to the client host which originated the change. (Omission to send such callbacks has been the general behavior of AFS file servers.) A client MUST be prepared to appropriately process (or ignore) callbacks for which its own UUID is the Origin. ExpirationTime The new expiration time asserted for the server's callback promise, not necessarily different from the existing expiration cached by the client. 3.4.2 AFSExtendedCallBack The members of the AFSExtendedCallBack structures are to be interpreted as follows: Flags If the 1-bit (AFSCB_Flag_Cancel) is set, the notification effects a callback break. The client may make use of the information sent with the message. ExtraFlags If (Flags & AFSCB_Flag_Cancel), a non-zero value for ExtraFlags indicates the reason for cancellation. DataVersion The value of DataVersion at completion of the event of which the client is being notified. Considering coalescing, the new data version after completion of all events summarized at this callback is (DataVersion+NCoalesced). Data The message-specific data for this notification. 3.4.3 Reasons for Cancellation The following reasons for cancellation are defined: AFSCB_Cancel_Shutdown The server or service is shutting down. AFSCB_Cancel_CallbackGC Callback has been disposed during periodic garbage collection. AFSCB_Cancel_VolumeOffline The volume associated with FID is now offline. AFSCB_Cancel_VolumeMoved The volume associated with FID has moved. AFSCB_Cancel_LostMyMind The server may be having problems related to provisioning an insufficient number of callback structures. AFSCB_Cancel_IHateYou Callback has been administratively revoked. 3.4.4 ExtendedCallBack Procedure Extended callbacks are delivered through a new ExtendedCallBack procedure. proc ExtendedCallBack( IN HostIdentifier *Server, IN AFSCBFids *Fids_Array, IN AFSExtendedCallBackSeq *CallBacks_Array, OUT AFSExtendedCallBackRSeq *CallBack_Result_Array ) multi = 65540; ExtendedCallBack is modelled on the traditional CallBack procedure, but adds UUIDs uniquely identifying the file server host. 3.4.5 Callback Coalescing A server implementation MAY, in addition, coalesce sequences of effectively-simultaneous notifications to a single client. This provision avoids performance regression in scenarios where a single logical event or operation would otherwise trigger potentially many notification messages per client (for example, many near-simultaneous CreateFile operations in a single directory as might occur on expansion of a tar archive, or many near-simultaneous stores appending to a single file).[footnote: The callback coalescing concept is re-introduced following discussions at the 2008 AFS and Kerberos Best Practices Workshop. ] Call Consolidation (Sequences of Notifications) A server implementation MAY coalesce any sequence of effectively simultaneous notifications into sequences of AFSExtendedCallBack objects enclosed in one AFSXCBInvocation object. Any number of such callbacks may be combined, up to the limit of AFSXCBMAX. Coalescing of Equivalent Notifications A server implementation MAY coalesce a sequence of effectively simultaneous and equivalent notifications to the same client into a single callback in a notification message. The following combinations of operations are explicitly permitted: * sequences of AFSCB_EventStoreAcl notifications on FID from a single Origin MAY be delivered as a single notification * sequences of AFSCB_EventStoreStatus notifications on FID from a single Origin MAY be delivered as the single notification of the most recently stored status * sequences of AFSCB_Event_StoreData notifications on FID from a single Origin at adjacent or overlapping byte ranges MAY deliver a single notification at the consolidated range Implementation Note "Effectively simultaneous" is left intentionally unspecified. An adaptive window expanding from 100ms to a small number of seconds appears to work well with commonly available switched networks and multi-core fileservers, in 2008. The current OpenAFS implementation uses an adaptive per-Fid window, which extends on repeated events, and closes when client contention is detected. Peer RTT or other considerations may be added to the windowing algorithm in future. 3.4.6 Asynchronous Delivery Edinburgh Consensus Prior drafts of this document defined asynchronous deliver semantics for extended callbacks, under specific circumstances. There is consensus that this and other consistency changes may be supported in future protocol revisions, but asynchronous delivery is removed from the current draft. 3.4.7 AFSCB_Event_StoreData The notification is sent in response to a successful StoreData RPC on FID. A structure of type AFSCB_Data_StoreData is sent with the message. struct AFSCB_Data_StoreData { afs_uint64 StoreOffset; afs_uint64 StoreLength; afs_uint64 Length; AFSCBFileStatus FileStatus; }; StoreLength bytes were stored starting at position StoreOffset in FID. Length is the current file length and FileStatus contains the modification time of FID following the operation. The client must regard cached file data in the range [StoreOffset, StoreOffset+StoreLength) as invalidated, and may regard data outside that range as up-to-date. The client MUST discard undirtied cached data in the invalidated range. The client MAY send dirtied data in the invalidated range to the file server prior to discarding (as allowed in current AFS semantics). 3.4.8 AFSCB_Event_StoreACL ACL and/or access information cached by the client for FID, if any, is invalidated. 3.4.9 AFSCB_Event_StoreStatus A StoreStatus RPC was successfully executed on FID. A structure of type AFSFetchStatus is sent with the message. struct AFSCB_Data_StoreStatus { struct AFSFetchStatus Status; }; Status is the new AFSFetchStatus of FID, ie, the message communicates the current status information of FID.[footnote: This is changed from earlier drafts. ] 3.4.10 AFSCB_Event_CreateFile A file has been created in the vnode corresponding to FID. A structure of type AFSCB_Data_CreateFile is sent with the message. struct AFSCB_Data_CreateFile { string Name; AFSFid Fid; AFSFetchStatus FidStatus; AFSCBDirStatus DirStatus; }; Name and Fid are, respectively, the name and FID of the created file. FidStatus is the AFSFetchStatus of the created file, and DirStatus the current modification time and link count of FID, at the completion of the call. 3.4.11 AFSCB_Event_MakeDir A directory has been created in the vnode corresponding to FID. A structure of type AFSCB_Data_MakeDir is sent with the message. struct AFSCB_Data_MakeDir { string Name; AFSFid Fid; AFSFetchStatus FidStatus; AFSCBDirStatus DirStatus; };; Name and Fid are, respectively, the name and FID of the created directory. FidStatus is the AFSFetchStatus of the created directory, and DirStatus the current modification time and link count of FID, at the completion of the call. 3.4.12 AFSCB_Data_Symlink A symbolic link has been created in the vnode corresponding to FID. A structure of type AFSCB_Data_Symlink is sent with the message. struct AFSCB_Data_Symlink { string Name; AFSFid Fid; string LinkContents; AFSFetchStatus FidStatus; AFSCBDirStatus DirStatus; }; Name is the name of the symbolic link. Fid is its AFSFid. The link points to LinkContents. FidStatus is the AFSFetchStatus of the created symbolic link, and DirStatus the current modification time and link count of FID, at the completion of the call. 3.4.13 AFSCB_Event_Link A hard link has been created in the vnode corresponding to FID. A structure of type AFSCB_Data_Link is sent with the message. struct AFSCB_Data_Link { string Name; AFSFid LinkTarget; AFSFetchStatus FidStatus; AFSCBDirStatus DirStatus; }; Name is the name of the link. The link is a synonym for LinkTarget. FidStatus is the AFSFetchStatus of the created link, and DirStatus the current modification time and link count of FID, at the completion of the call. 3.4.14 AFSCB_Event_RemoveFile A file has been removed from the vnode corresponding to FID. A structure of type AFSCB_Data_RemoveFile is sent with the message. struct AFSCB_Data_RemoveFile { string Name; AFSCBDirStatus DirStatus; }; Name indicates the removed entry. DirStatus the current modification time and link count of FID, at the completion of the call. 3.4.15 AFSCB_Event_RemoveDir A directory has been removed from the vnode corresponding to FID. A structure of type AFSCB_Data_RemoveDir is sent with the message. struct AFSCB_Data_RemoveDir { string Name; AFSCBDirStatus DirStatus; }; Name indicates the removed entry. DirStatus the current modification time and link count of FID, at the completion of the call. 3.4.16 AFSCB_Event_Rename A file or directory has been renamed, ie moved, from or to the vnode corresponding to FID. A structure of type AFSCB_Data_RemoveDir is sent with the message. const AFSCB_Rename_From = 1; const AFSCB_Rename_To = 2; struct AFSCB_Data_Rename { afs_uint32 Direction; string OldName; string NewName; AFSCBDirStatus FromStatus; AFSCBDirStatus ToStatus; }; Direction indicates whether FID is the source or the destination directory of the move. OldName is the name of the object in its old location, NewName the name of the object in its new location. FromStatus is the current modification time and link count of the source directory vnode, and ToStatus is the current modification time and link count of the destination directory vnode, and FidStatus the at the completion of the call. To preserve atomicity, the AFSCB_Data_Rename message is constructed so that changes to cached copies of both the source and directory vnodes may be recovered from a single notification. If a client owns callbacks for both the source and destination FIDs, a file server MAY elect to send only one notification, for either the source or the destination FID. 3.4.17 AFSCB_Event_Deleted The object corresponding to FID not longer exists, and so may no longer be cached. It is an ex-object. (I.e., the client MUST discard any information it has cached about FID.) 3.4.18 AFSCB_Event_ReleaseLock A traditional AFS whole-file lock has been released on FID. A structure of type AFSCB_Data_Lock is sent with the message. struct AFSCB_Data_Lock { afs_uint32 LockType; }; LockType is the type of the lock released. Receipt of an AFSCB_Event_ReleaseLock notification in no way implies an intention on the part of a file server to grant a lock on FID to client. Non-receipt of a notification of this type in no way implies non-release of locks that may be held on FID. The file server SHOULD send notifications of this type only to clients which have indicated probable interest in the event, e.g., by having recently requested a lock on FID. 3.5 Callbacks And Read-Only Volume Replicas Callbacks associated with read-only volume replicas have traditionally been handled specially in AFS. When any file in an RO volume is accessed the AFS file server establishes a single callback promise considered to be on the entire volume. Any event which updates the replica (e.g., vos release) triggers a whole-volume callback break. The whole-volume callback optimization significantly reduced file server memory utilization, which was at a premium in 1988. However, the whole-volume callback is less of an optimization in OpenAFS in 2008: * modern AFS file servers have sufficient memory to track millions of callbacks (and do track up to 1 million callbacks at one site we know of, with up to 3 million callback structures available)[8] * whole-volume callback semantics require clients (and the file server) to potentially expend considerable effort re-establishing cache consistency, and so whole-volume callbacks are necessarily a considerable protocol inefficiency for sites relying heavily on AFS replication (in particular, incremental replication now possible in OpenAFS) For these reasons, we propose that the scope of extended callback information include notifications concerning changes that originate in the release of a volume. We provide the option for the file server to provide whole-volume or per file notifications, at its discretion. We provide the option for the file server to track client interest in specific files (ie, issue per-file callbacks on files in RO volumes), and speculate that this implementation would be preferred, but do not mandate it. 3.5.1 Constants The following flag constants are added: const AFSCB_Flag_Release = 2; const AFSCB_IFlag_Release = 2; const AFSCB_Release_WholeVolumeCancel = 1; AFSCB_Flag_Release In an AFSExtendedCallBack instance, (Flags & AFSCB_Flag_Release) indicates a notification in response to the (possibly incremental) release of a read-only replica. AFSCB_IFlag_Release In an AFSXCBInvocation instance, (Flags & AFSCB_IFlag_Release) indicates a notification in response to the (possibly incremental) release of a read-only replica. AFSCB_Release_WholeVolumeCancel If additionally (ExtraFlags & AFSCB_Release_WholeVolumeCancel), then the callback invalidates the entire volume, otherwise it is a selective invalidation of just the FIDs in Fids_Array. 3.5.2 Semantic Changes An AFS file server MAY send selective or whole-volume extended callback notifications. The file server MAY choose to regard files in RO volumes equivalently to files in RW volumes, ie, effectively maintain callback state on them. Alternatively it MAY send selective notifications on any FIDs changed, removed, or added in the volume without regard to client cache state. The AFS client must handle such notifications gracefully. 3.6 Security Implications Extended callback information messages that only invalidate information that may be cached at clients have equivalent security implications to AFS-3 callback messages. This class of messages includes AFSCB_Event_Cancel and probably AFSCB_Event_StoreData. The remaining extended callback information messages (most of them) contain explicit metadata information which could potentially be used by an attacker impersonating a file server to introduce malicious information into a client cache. Rx security extensions in development (eg, rxk5) include provisions for secure transmission of callback messages. Edinburgh Consensus Implementations should ensure that extended callbacks which send explicit metadata use a secure communication channel. Cancellation and StoreData messages may be sent over any channel. 4 Appendix A: XDR Grammar /* Cache Manager Capability Flags */ const CLIENT_CAPABILITY_EXT_CALLBACK = 0x0002; /* Host Tracking/Extended Information */ struct HostIdentifier { afsUUID ServerUuid; afsUUID CellUuid; }; /* Extended Callback Information */ /* callback event types, predominantly events on the vnode for * which the callback is being made, but also (e.g., Deleted) side * effects of operations on related vnodes */ const AFSCB_Event_Cancel = 1; const AFSCB_Event_StoreData = 2; const AFSCB_Event_StoreACL = 3; const AFSCB_Event_StoreStatus = 4; const AFSCB_Event_CreateFile = 5; const AFSCB_Event_MakeDir = 6; const AFSCB_Event_Symlink = 7; const AFSCB_Event_Link = 8; const AFSCB_Event_RemoveFile = 9; const AFSCB_Event_RemoveDir = 10; const AFSCB_Event_Rename = 11; const AFSCB_Event_Deleted = 12; const AFSCB_Event_ReleaseLock = 13; /* for use in AFSExtendedCallBack Flags */ const AFSCB_Flag_Cancel = 1; const AFSCB_Flag_Release = 2; /* intended for use in AFSExtendedCallBack ExtraFlags, * when (flags & AFSCB_Flag_Cancel), to indicate reason for * cancellation */ const AFSCB_Cancel_Shutdown = 1; const AFSCB_Cancel_CallbackGC = 2; const AFSCB_Cancel_VolumeOffline = 3; const AFSCB_Cancel_VolumeMoved = 4; const AFSCB_Cancel_LostMyMind = 5; /* ran out of callbacks? */ const AFSCB_Cancel_IHateYou = 6; /* callback administratively revoked */ /* for use in AFSXCBInvocation Flags */ const AFSCB_IFlag_SOrigin = 1; const AFSCB_IFlag_Release = 2; /* flags intended for use in AFSExtendedCallBack ExtraFlags * to indicate RO volume callback events */ const AFSCB_Release_WholeVolumeCancel = 1; /* callback result types */ const AFSCB_Result_NoResult = 1; const AFSCB_Result_Diag = 2; /* differential status to be sent with StoreData msgs */ struct AFSCBFileStatus { afs_uint32 LinkCount; afs_uint64 ClientModTime; }; /* differential status to be sent with directory change msgs */ struct AFSCBDirStatus { afs_uint32 LinkCount; afs_uint64 ClientModTime; }; /* variant data types for AFSCB_Notification_Data */ struct AFSCB_Data_StoreData { afs_uint64 StoreOffset; afs_uint64 StoreLength; afs_uint64 Length; AFSCBFileStatus FileStatus; }; struct AFSCB_Data_StoreStatus { struct AFSFetchStatus Status; }; struct AFSCB_Data_CreateFile { string Name; AFSFid Fid; AFSFetchStatus FidStatus; AFSCBDirStatus DirStatus; }; struct AFSCB_Data_MakeDir { string Name; AFSFid Fid; AFSFetchStatus FidStatus; AFSCBDirStatus DirStatus; }; struct AFSCB_Data_Symlink { string Name; AFSFid Fid; string LinkContents; AFSFetchStatus FidStatus; AFSCBDirStatus DirStatus; }; struct AFSCB_Data_Link { string Name; AFSFid LinkTarget; AFSFetchStatus FidStatus; AFSCBDirStatus DirStatus; }; struct AFSCB_Data_RemoveFile { string Name; AFSCBDirStatus DirStatus; }; struct AFSCB_Data_RemoveDir { string Name; AFSCBDirStatus DirStatus; }; const AFSCB_Rename_From = 1; const AFSCB_Rename_To = 2; struct AFSCB_Data_Rename { afs_uint32 Direction; string OldName; string NewName; AFSCBDirStatus FromStatus; AFSCBDirStatus ToStatus; }; struct AFSCB_Data_Lock { afs_uint32 LockType; }; union AFSCB_NotificationData switch (afs_uint32 Event_Type) { case AFSCB_Event_StoreData: AFSCB_Data_StoreData u_store_data; case AFSCB_Event_StoreACL: void; case AFSCB_Event_StoreStatus: AFSCB_Data_StoreStatus u_store_status; case AFSCB_Event_CreateFile: AFSCB_Data_CreateFile u_create_file; case AFSCB_Event_MakeDir: AFSCB_Data_MakeDir u_make_dir; case AFSCB_Event_Symlink: AFSCB_Data_Symlink u_symlink; case AFSCB_Event_Link: AFSCB_Data_Link u_link; case AFSCB_Event_RemoveFile: AFSCB_Data_RemoveFile u_remove_file; case AFSCB_Event_RemoveDir: AFSCB_Data_RemoveDir u_remove_dir; case AFSCB_Event_Rename: AFSCB_Data_Rename u_rename; case AFSCB_Event_Deleted: void; case AFSCB_Event_ReleaseLock: AFSCB_Data_Lock u_lock; case AFSCB_Event_Cancel: void; }; struct AFSExtendedCallBack { afs_uint32 Flags; afs_uint32 ExtraFlags; afsUUID Origin; /* originator of changes */ afs_uint32 NCoalesced; /* calls [StoreData] combined on this */ afs_uint64 DataVersion; AFSCB_NotificationData Data; }; const AFSXCBMAX = 512; typedef AFSExtendedCallBack AFSExtendedCallBackSeq; struct AFSXCBInvocation { AFSFid Fid; afs_uint32 Flags; afs_uint64 lowDV; /* lowest DV at invocation */ afs_uint64 highDV; /* highest */ afs_uint64 ExpirationTime; /* new expiration, or 0 if unchanged */ AFSExtendedCallBackSeq CallBacks_Array; }; typedef AFSXCBInvocation AFSXCBInvocationSeq; /* Forward-looking union for callback results */ union AFSCB_ResultData switch (afs_uint32 Result_Type) { case AFSCB_Result_NoResult: void; case AFSCB_Result_Diag: string msg<30>; }; /* extended callback result structure */ struct AFSExtendedCallBackResult { afs_uint32 Flags; afs_uint32 ExtraFlags; AFSCB_ResultData Data; }; typedef AFSExtendedCallBackResult AFSExtendedCallBackRSeq; proc ExtendedCallBack( IN HostIdentifier *Server, IN AFSXCBInvocationSeq *Invocations_Array, OUT AFSExtendedCallBackRSeq *CallBack_Result_Array ) multi = 65540; References [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [3] Howard, J.H., Kazar, M.L., Menees, S.G., Nichols, D.A., Satyanarayanan, M., Sidebotham, R.N. and West, M. "Scale and Performance in a Distributed File System" ACM Transactions on Computer Systems, February 1988 [4] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003. [5] Edward R Zayas, "AFS-3 Programmer's Reference: File Server/Cache Manager Interface", Transarc Corporation, FS-00-D162, 20th August 1991 [6] Paul J. Leach, Dilip C. Naik. A Common Internet File System (CIFS/1.0) Protocol [http://www.tools.ietf.org/html/draft-leach-cifs-v1-spec-01], 1997. [7] Kazar, Michael Leon, "Synchronization and Caching Issues in the Andrew File System," USENIX Conference Proceedings, USENIX Association, Berkeley, CA, Dallas Winter 1988, pages 27-36. [7] Lily B. Mummert, Mahadev Satyanarayanan: Large Granularity Cache Coherence for Intermittent Connectivity. USENIX Summer 1994: 279-289 [8] Alistair Ferguson. OpenAFS and the Dawn of a New Era. AFS and Kerberos Best Practices Workshop, 2008. [9] Trond Myklebust. Byte Range Delegations. [https://www3.ietf.org/proceedings/05nov/slides/nfsv4-3.pdf ], November 2006. Authors' Addresses Matt Benjamin (editor) Email: matt@linuxbox.com URI: http://linuxbox.com