--- 1/draft-ietf-extra-imap-objectid-07.txt 2018-08-02 07:14:21.700699262 -0700 +++ 2/draft-ietf-extra-imap-objectid-08.txt 2018-08-02 07:14:21.732700029 -0700 @@ -1,19 +1,19 @@ EXTRA B. Gondwana, Ed. Internet-Draft FastMail -Updates: 3501 (if approved) July 31, 2018 +Updates: 3501 (if approved) August 2, 2018 Intended status: Standards Track -Expires: February 1, 2019 +Expires: February 3, 2019 IMAP Extension for object identifiers - draft-ietf-extra-imap-objectid-07 + draft-ietf-extra-imap-objectid-08 Abstract This document updates RFC3501 (IMAP4rev1) with persistent identifiers on mailboxes and messages to allow clients to more efficiently re-use cached data when resources have changed location on the server. Status of This Memo This Internet-Draft is submitted in full conformance with the @@ -22,21 +22,21 @@ Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on February 1, 2019. + This Internet-Draft will expire on February 3, 2019. Copyright Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents @@ -52,47 +52,48 @@ 2. Conventions Used In This Document . . . . . . . . . . . . . . 3 3. CAPABILITY Identification . . . . . . . . . . . . . . . . . . 3 4. MAILBOXID object identifier . . . . . . . . . . . . . . . . . 3 4.1. New response code for CREATE . . . . . . . . . . . . . . 4 4.2. New OK Untagged Response for SELECT and EXAMINE . . . . . 4 4.3. New attribute for STATUS . . . . . . . . . . . . . . . . 5 5. EMAILID object identifier and THREADID correlator . . . . . . 6 5.1. EMAILID identifier for identical messages . . . . . . . . 6 5.2. THREADID identifer for related messages . . . . . . . . . 6 5.3. New Message Data Items in FETCH and UID FETCH Commands . 7 - 6. New Filters on SEARCH command . . . . . . . . . . . . . . . . 9 - 7. Formal syntax . . . . . . . . . . . . . . . . . . . . . . . . 9 - 8. Implementation considerations . . . . . . . . . . . . . . . . 10 - 8.1. Assigning object identifiers . . . . . . . . . . . . . . 10 - 8.2. Interaction with special cases . . . . . . . . . . . . . 11 - 8.3. Client usage . . . . . . . . . . . . . . . . . . . . . . 11 - 9. Future considerations . . . . . . . . . . . . . . . . . . . . 12 - 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 - 11. Security Considerations . . . . . . . . . . . . . . . . . . . 12 - 12. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 - 12.1. draft-ietf-extra-imap-objectid-07 . . . . . . . . . . . 13 - 12.2. draft-ietf-extra-imap-objectid-06 . . . . . . . . . . . 13 - 12.3. draft-ietf-extra-imap-objectid-05 . . . . . . . . . . . 13 - 12.4. draft-ietf-extra-imap-objectid-04 . . . . . . . . . . . 13 - 12.5. draft-ietf-extra-imap-objectid-03 . . . . . . . . . . . 14 - 12.6. draft-ietf-extra-imap-objectid-02 . . . . . . . . . . . 14 - 12.7. draft-ietf-extra-imap-objectid-01 . . . . . . . . . . . 14 - 12.8. draft-ietf-extra-imap-objectid-00 . . . . . . . . . . . 15 - 12.9. draft-ietf-extra-imap-uniqueid-00 . . . . . . . . . . . 15 - 12.10. draft-gondwana-imap-uniqueid-01 . . . . . . . . . . . . 15 - 12.11. draft-gondwana-imap-uniqueid-00 . . . . . . . . . . . . 15 - 13. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 15 - 13.1. Appendix 1: ideas for implementing object identifiers . 16 - 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 16 - 14.1. Normative References . . . . . . . . . . . . . . . . . . 16 - 14.2. Informative References . . . . . . . . . . . . . . . . . 17 - Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 18 + 6. New Filters on SEARCH command . . . . . . . . . . . . . . . . 10 + 7. Formal syntax . . . . . . . . . . . . . . . . . . . . . . . . 10 + 8. Implementation considerations . . . . . . . . . . . . . . . . 11 + 8.1. Assigning object identifiers . . . . . . . . . . . . . . 11 + 8.2. Interaction with special cases . . . . . . . . . . . . . 12 + 8.3. Client usage . . . . . . . . . . . . . . . . . . . . . . 12 + 9. Future considerations . . . . . . . . . . . . . . . . . . . . 13 + 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 + 11. Security Considerations . . . . . . . . . . . . . . . . . . . 13 + 12. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 + 12.1. draft-ietf-extra-imap-objectid-08 . . . . . . . . . . . 14 + 12.2. draft-ietf-extra-imap-objectid-07 . . . . . . . . . . . 14 + 12.3. draft-ietf-extra-imap-objectid-06 . . . . . . . . . . . 14 + 12.4. draft-ietf-extra-imap-objectid-05 . . . . . . . . . . . 15 + 12.5. draft-ietf-extra-imap-objectid-04 . . . . . . . . . . . 15 + 12.6. draft-ietf-extra-imap-objectid-03 . . . . . . . . . . . 15 + 12.7. draft-ietf-extra-imap-objectid-02 . . . . . . . . . . . 16 + 12.8. draft-ietf-extra-imap-objectid-01 . . . . . . . . . . . 16 + 12.9. draft-ietf-extra-imap-objectid-00 . . . . . . . . . . . 16 + 12.10. draft-ietf-extra-imap-uniqueid-00 . . . . . . . . . . . 16 + 12.11. draft-gondwana-imap-uniqueid-01 . . . . . . . . . . . . 17 + 12.12. draft-gondwana-imap-uniqueid-00 . . . . . . . . . . . . 17 + 13. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 17 + 13.1. Appendix 1: ideas for implementing object identifiers . 17 + 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 + 14.1. Normative References . . . . . . . . . . . . . . . . . . 18 + 14.2. Informative References . . . . . . . . . . . . . . . . . 19 + Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 19 1. Introduction IMAP stores are often used by many clients. Each client may cache data from the server so that they don't need to re-download information. [RFC3501] defines that a mailbox can be uniquely referenced by its name and UIDVALIDITY, and a message within that mailbox can be uniquely referenced by its mailbox (name + UIDVALIDITY) and UID. The triple of mailbox name, UIDVALIDITY and UID is guaranteed to be immutable. @@ -158,21 +159,21 @@ 4.1. New response code for CREATE This document extends the CREATE command to have the response code MAILBOXID on successful mailbox creation. A server advertising the OBJECTID capability MUST include the MAILBOXID response code in the tagged OK response to all successful CREATE commands. - Syntax: "MAILBOXID" SP "(" ")" + Syntax: "MAILBOXID" SP "(" objectid ")" Response code in tagged OK for successful CREATE command. Example: C: 3 create foo S: 3 OK [MAILBOXID (F2212ea87-6097-4256-9d51-71338625)] Completed C: 4 create bar S: 4 OK [MAILBOXID (F6352ae03-b7f5-463c-896f-d8b48ee3)] Completed C: 5 create foo @@ -180,22 +181,21 @@ 4.2. New OK Untagged Response for SELECT and EXAMINE This document adds a new untagged response code to the SELECT and EXAMINE commands. A server advertising the OBJECTID capability MUST return an untagged OK response with the MAILBOXID response code on all successful SELECT and EXAMINE commands. - Syntax: "OK" SP "[" "MAILBOXID" SP "(" ")" "]" text - + Syntax: "OK" SP "[" "MAILBOXID" SP "(" objectid ")" "]" SP text Untagged OK response to SELECT or EXAMINE. Example: C: 27 select "foo" [...] S: * OK [MAILBOXID (F2212ea87-6097-4256-9d51-71338625)] Ok [...] S: 27 OK [READ-WRITE] Completed @@ -204,21 +204,21 @@ This document adds the MAILBOXID attribute to the STATUS command using the extended syntax defined in [RFC4466]. A server that advertises the OBJECTID capability MUST support the MAILBOXID status attribute. Syntax: "MAILBOXID" The attribute in the STATUS command. - Syntax: "MAILBOXID" SP "(" ")" + Syntax: "MAILBOXID" SP "(" objectid ")" The response item in the STATUS response contains the objectid assigned by the server for this mailbox. Example: C: 6 status foo (mailboxid) S: * STATUS foo (MAILBOXID (F2212ea87-6097-4256-9d51-71338625)) S: 6 OK Completed C: 7 status bar (mailboxid) @@ -282,62 +282,63 @@ THREADID calculation is generally based on some combination of References, In-Reply-To and Subject, but the exact logic is left up to the server implementation. [RFC5256] describes some algorithms that could be used, however this specfication does not mandate any particular strategy. The server MUST return the same THREADID for all messages with the same EMAILID. The server SHOULD return the same THREADID for related messages even - if they are in different mailboxes. + if they are in different mailboxes, e.g. messages which would appear + in the same thread if they were in the same mailbox SHOULD have the + same THREADID even if they are in different mailboxes. The server MUST NOT change the THREADID of a message once reported. - THREADID is optional, if the server doesn't support THREADID or is + THREADID is OPTIONAL; if the server doesn't support THREADID or is unable to calculate relationships between messages, it MUST return NIL to all FETCH responses for the THREADID data item, and a SEARCH for THREADID MUST NOT match any messages. The server MUST NOT use the same objectid value for both EMAILIDs and THREADIDs. If they are stored with the same value internally, the server can generate prefixed values (as shown in the examples below with M and T prefixes) to avoid clashes. 5.3. New Message Data Items in FETCH and UID FETCH Commands This document defines two FETCH items: - Syntax: EMAILID + Syntax: "EMAILID" The EMAILID message data item causes the server to return EMAILID FETCH response data items. - Syntax: THREADID + Syntax: "THREADID" The THREADID message data item causes the server to return THREADID FETCH response data items. And the following responses: - Syntax: EMAILID ( ) + Syntax: "EMAILID" SP "(" objectid ")" The EMAILID response data item contains the server-assigned objectid for each message. - Syntax: THREADID ( ) + Syntax: "THREADID" SP "(" objectid ")" The THREADID response data item contains the server-assigned objectid for the set of related messages to which this message belongs. - Syntax: THREADID NIL - + Syntax: "THREADID" SP nil The NIL value to the THREADID response data item is returned when the server mailbox does not support THREADID calculation. Example: C: 5 append inbox "20-Mar-2018 03:07:37 +1100" {733} [...] Subject: Message A Message-ID: [...] @@ -386,25 +387,25 @@ C: 26 fetch 1:* (emailid threadid) S: * 1 FETCH (EMAILID (M00000001) THREADID NIL) S: * 2 FETCH (EMAILID (M00000002) THREADID NIL) S: 26 OK Completed (0.000 sec) 6. New Filters on SEARCH command This document defines filters EMAILID and THREADID on the SEARCH command. - EMAILID + Syntax: "EMAILID" SP objectid Messages whose EMAILID is exactly the specified objectid. - THREADID + Syntax: "THREADID" SP objectid Messages whose THREADID is exactly the specified objectid. Example: (as if run before the MOVE above when the mailbox had 3 messages) C: 27 search emailid M6d99ac3275bb4e S: * SEARCH 1 S: 27 OK Completed (1 msgs in 0.000 secs) C: 28 search threadid T64b478a75b7ea9 @@ -416,39 +417,41 @@ The following syntax specification uses the Augmented Backus-Naur Form (ABNF) [RFC5234] notation. Elements not defined here can be found in the formal syntax of the ABNF [RFC5234], IMAP [RFC3501], and IMAP ABNF extensions [RFC4466] specifications. Except as noted otherwise, all alphabetic characters are case- insensitive. The use of upper- or lowercase characters to define token strings is for editorial clarity only. Implementations MUST accept these strings in a case-insensitive fashion. + Please note specifically that objectid values are case sensitive. + capability =/ "OBJECTID" fetch-att =/ "EMAILID" / "THREADID" fetch-emailid-resp = "EMAILID" SP "(" objectid ")" ; follows tagged-ext production from [@!RFC4466] fetch-threadid-resp = "THREADID" SP ( "(" objectid ")" / nil ) ; follows tagged-ext production from [@!RFC4466] msg-att-static =/ fetch-emailid-resp / fetch-threadid-resp objectid = 1*255(ALPHA / DIGIT / "_" / "-") ; characters in object identifiers are case ; significant resp-text-code =/ "MAILBOXID" SP "(" objectid ")" ; incorporated before the expansion rule of - ; atom [SP 1*<any TEXT-CHAR except "]">] + ; atom [SP 1*] ; that appears in [@!RFC3501] search-key =/ "EMAILID" SP objectid / "THREADID" SP objectid status-att =/ "MAILBOXID" status-att-value =/ "MAILBOXID" SP "(" objectid ")" ; follows tagged-ext production from [@!RFC4466] 8. Implementation considerations @@ -456,53 +459,56 @@ 8.1. Assigning object identifiers All objectid values are allocated by the server. In the interests of reducing the possibilities of encoding mistakes, objectids are restricted to a safe subset of possible byte values, and in order to allow clients to allocate storage, they are restricted in length. An objectid is a string of 1 to 255 characters from the following set - of 64 codepoints. a-z, A-Z, 0-9, '_', '-'. These characters are safe + of 64 codepoints: a-z, A-Z, 0-9, '_', '-'. These characters are safe to use in almost any context (e.g. filesystems, URIs, IMAP atoms). + These are the same characters defined as base64url in [RFC4648]. For maximum safety, servers should also follow defensive allocation strategies to avoid creating risks where glob completion or data type detection may be present (e.g. on filesystems or in spreadsheets). In particular it is wise to avoid: o ids starting with - o ids starting with digits o ids which contain only digits o ids which differ only by ASCII case (A vs a) - o the specific sequence of 3 characters NIL + o the specific sequence of 3 characters NIL in any case (as this + sequence can be confused with the IMAP protocol expression of the + null value) A good solution to these issues is to prefix every ID with a single alphabetical character. 8.2. Interaction with special cases The case of RENAME INBOX may need special handling, as it has special behaviour as defined in [RFC3501] section 6.3.5. It is advisable (though not required) to have MAILBOXID be globally unique, but it is only required to be unique within messages offered to a single client login to a single server hostname. For example, a proxy which aggregates multiple independent servers MUST NOT advertise the OBJECTID capability unless it can guarantee that different objects will never use the same identifiers, even if - backend object collide. + backend object identifiers collide. 8.3. Client usage Servers that implement both RFC 6154 and this specification should optimize their execution of command like UID SEARCH OR EMAILID 1234 EMAILID 4321. Clients can assume that searching the all-mail mailbox using OR/ EMAILID or OR/THREADID is a fast way to find messages again if some other client has moved them out of the mailbox where they were @@ -551,72 +557,92 @@ current security research and choose secure digests. As the IDs are generated by the server, it will be possible to migrate to a new hash by just using the new algorith when creating new IDs. This is particularly true if a prefix is used on each ID, which can be changed when the algorithm changes. The use of a digest for ID generation may be used as proof that a particular sequence of bytes was seen by the server, however this is only a risk if IDs are leaked to clients who don't have permission to fetch the data directly. Servers that are expected to handle highly - sensitive data should consider using a ID generation mechanism which - doesn't derive from a digest. + sensitive data should consider this when choosing how to create IDs. See also the security considerations in [RFC3501] section 11. 12. Changes To be removed by the editor before publication -12.1. draft-ietf-extra-imap-objectid-07 +12.1. draft-ietf-extra-imap-objectid-08 + + o added reference to RFC4648 for base64url (Adam Roach review) + + o less prescriptive instruction for digests (Adam Roach review) + + o minor punctuation fix (Alissa Cooper review) + + o clarified SHOULD around THREADID in different mailboxes (Alissa + Cooper review) + + o clarified that objectids are case sensitive (Eric Rescorla review) + + o clarified why NIL is a bad objectid (Eric Rescorla review) + + o upper case OPTIONAL for threadid (Pete Resnick genart review) + + o missing word in object identifiers clash (Pete Resnick genart + review) + +12.2. draft-ietf-extra-imap-objectid-07 o updated boilerplate to RFC8174 (Benjamin Kaduk review) o fixed spelling of invariants (Benjamin Kaduk review) o block quoted ABNF for better text formatting (Benjamin Kaduk review) o clarified that servers can just switch to a new digest without changing old IDs (Benjamin Kaduk review) o changed use of folder to mailbox to avoid confusion (Warren Kumari review) o made both IANA requests say "reference of this RFC" (Warren Kumari review) -12.2. draft-ietf-extra-imap-objectid-06 +12.3. draft-ietf-extra-imap-objectid-06 o fixed one more missing space in ABNF (ad review) o made one more MUST for mailbox being retained on rename (genart review) o updated ABNF to also extend msg-att-static (validator review) o lowercased NIL => nil in ABNF (validator review) -12.3. draft-ietf-extra-imap-objectid-05 +12.4. draft-ietf-extra-imap-objectid-05 o changed some SHOULD to lower case in advice sections (genart review) o clarified that THREADID MUST NOT change -12.4. draft-ietf-extra-imap-objectid-04 +12.5. draft-ietf-extra-imap-objectid-04 o described NIL THREADID in more detail (ad review) o made RFC5256 a normative reference (ad review) o fixed ABNF missing quote (ad review) + o documented hash upgrade process (ad review) o referenced RFC3501 for INBOX rename (ad review) o referenced RFC3501 security considerations (secdir review) o turned mealy-mouthed "SHOULDs" in to "MUSTs" on immutability (genart review) o remove suggested algorithms which are no longer legitimate (genart @@ -622,35 +648,35 @@ o remove suggested algorithms which are no longer legitimate (genart review) o updated proxy advice to suggest rewriting ids (genart review) o fixed minor gramatical issues (genart review) o required that EMAILID and THREADID are not identical (own decision) -12.5. draft-ietf-extra-imap-objectid-03 +12.6. draft-ietf-extra-imap-objectid-03 o added RFC3501 to Abstract o updated [[THIS RFC]] to not fail idnits o changed jmap-mail to be informative rather than normative o shortened IDs to stop wrapping and outdents in IMAP examples -12.6. draft-ietf-extra-imap-objectid-02 +12.7. draft-ietf-extra-imap-objectid-02 o added "Client usage" section -12.7. draft-ietf-extra-imap-objectid-01 +12.8. draft-ietf-extra-imap-objectid-01 o added "updates" for RFC3501 o fixed domains in thread example o described threading in more detail o added IANA request for Response Code o clarified RFC2119 references @@ -651,59 +677,60 @@ o described threading in more detail o added IANA request for Response Code o clarified RFC2119 references o simplified some waffle in wording o added security consideration to choose good digest + o added MAILBOXID-UID suggestion for EMAILID generation o updated ABNF normative reference to RFC5234 -12.8. draft-ietf-extra-imap-objectid-00 +12.9. draft-ietf-extra-imap-objectid-00 o renamed draft to be objectid rather than uniqueid o renamed UNIQUEID (capability) to OBJECTID o restricted objectid to 64 safe characters o added security considerations and advice about choosing objectid o wrapped all responses in () for RFC4466 compatibility o signifiant rewrite of all sections -12.9. draft-ietf-extra-imap-uniqueid-00 +12.10. draft-ietf-extra-imap-uniqueid-00 o renamed draft to be an EXTRA document o added example for LIST RETURN STATUS o started work on ABNF o attempted to add response codes for EMAILID and THREADID -12.10. draft-gondwana-imap-uniqueid-01 +12.11. draft-gondwana-imap-uniqueid-01 o renamed UNIQUEID (status item) to MAILBOXID o renamed MSGID to EMAILID o renamed THRID to THREADID o added TODO section -12.11. draft-gondwana-imap-uniqueid-00 +12.12. draft-gondwana-imap-uniqueid-00 o initial upload with names UNIQUEID/MSGID/THRID 13. Acknowledgments The EXTRA working group at IETF. In particular feedback from Arnt Gulbrandsen, Brandon Long, Chris Newman and Josef Sipek. The Gmail X-GM-THRID and X-GM-MSGID implementation as currently defined at . + [RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data + Encodings", RFC 4648, DOI 10.17487/RFC4648, October 2006, + . + Author's Address Bron Gondwana (editor) FastMail Level 2, 114 William St Melbourne VIC 3000 Australia Email: brong@fastmailteam.com URI: https://www.fastmail.com