[http] Custom HTTP headers : naming conventions

Several of our users have asked us to include data relative to their account in the HTTP headers of requests we send them, or even responses they get from our API. What is the general convention to add custom HTTP headers, in terms of naming, format... etc.

Also, feel free to post any smart usage of these that you stumbled upon on the web; We're trying to implement this using what's best out there as a target :)

The format for HTTP headers is defined in the HTTP specification. I'm going to talk about HTTP 1.1, for which the specification is RFC 2616. In section 4.2, 'Message Headers', the general structure of a header is defined:

   message-header = field-name ":" [ field-value ]
   field-name     = token
   field-value    = *( field-content | LWS )
   field-content  = <the OCTETs making up the field-value
                    and consisting of either *TEXT or combinations
                    of token, separators, and quoted-string>

This definition rests on two main pillars, token and TEXT. Both are defined in section 2.2, 'Basic Rules'. Token is:

   token          = 1*<any CHAR except CTLs or separators>

In turn resting on CHAR, CTL and separators:

   CHAR           = <any US-ASCII character (octets 0 - 127)>

   CTL            = <any US-ASCII control character
                    (octets 0 - 31) and DEL (127)>

   separators     = "(" | ")" | "<" | ">" | "@"
                  | "," | ";" | ":" | "\" | <">
                  | "/" | "[" | "]" | "?" | "="
                  | "{" | "}" | SP | HT

TEXT is:

   TEXT           = <any OCTET except CTLs,
                    but including LWS>

Where LWS is linear white space, whose definition i won't reproduce, and OCTET is:

   OCTET          = <any 8-bit sequence of data>

There is a note accompanying the definition:

The TEXT rule is only used for descriptive field contents and values
that are not intended to be interpreted by the message parser. Words
of *TEXT MAY contain characters from character sets other than ISO-
8859-1 [22] only when encoded according to the rules of RFC 2047
[14].

So, two conclusions. Firstly, it's clear that the header name must be composed from a subset of ASCII characters - alphanumerics, some punctuation, not a lot else. Secondly, there is nothing in the definition of a header value that restricts it to ASCII or excludes 8-bit characters: it's explicitly composed of octets, with only control characters barred (note that CR and LF are considered controls). Furthermore, the comment on the TEXT production implies that the octets are to be interpreted as being in ISO-8859-1, and that there is an encoding mechanism (which is horrible, incidentally) for representing characters outside that encoding.

So, to respond to @BalusC in particular, it's quite clear that according to the specification, header values are in ISO-8859-1. I've sent high-8859-1 characters (specifically, some accented vowels as used in French) in a header out of Tomcat, and had them interpreted correctly by Firefox, so to some extent, this works in practice as well as in theory (although this was a Location header, which contains a URL, and these characters are not legal in URLs, so this was actually illegal, but under a different rule!).

That said, i wouldn't rely on ISO-8859-1 working across all servers, proxies, and clients, so i would stick to ASCII as a matter of defensive programming.

Modifying, or more correctly, adding additional HTTP headers is a great code debugging tool if nothing else.

When a URL request returns a redirect or an image there is no html "page" to temporarily write the results of debug code to - at least not one that is visible in a browser.

One approach is to write the data to a local log file and view that file later. Another is to temporarily add HTTP headers reflecting the data and variables being debugged.

I regularly add extra HTTP headers like X-fubar-somevar: or X-testing-someresult: to test things out - and have found a lot of bugs that would have otherwise been very difficult to trace.

The header field name registry is defined in RFC3864, and there's nothing special with "X-".

As far as I can tell, there are no guidelines for private headers; in doubt, avoid them. Or have a look at the HTTP Extension Framework (RFC 2774).

It would be interesting to understand more of the use case; why can't the information be added to the message body?

The question bears re-reading. The actual question asked is not similar to vendor prefixes in CSS properties, where future-proofing and thinking about vendor support and official standards is appropriate. The actual question asked is more akin to choosing URL query parameter names. Nobody should care what they are. But name-spacing the custom ones is a perfectly valid -- and common, and correct -- thing to do.

Rationale:
It is about conventions among developers for custom, application-specific headers -- "data relevant to their account" -- which have nothing to do with vendors, standards bodies, or protocols to be implemented by third parties, except that the developer in question simply needs to avoid header names that may have other intended use by servers, proxies or clients. For this reason, the "X-Gzip/Gzip" and "X-Forwarded-For/Forwarded-For" examples given are moot. The question posed is about conventions in the context of a private API, akin to URL query parameter naming conventions. It's a matter of preference and name-spacing; concerns about "X-ClientDataFoo" being supported by any proxy or vendor without the "X" are clearly misplaced.

There's nothing special or magical about the "X-" prefix, but it helps to make it clear that it is a custom header. In fact, RFC-6648 et al help bolster the case for use of an "X-" prefix, because -- as vendors of HTTP clients and servers abandon the prefix -- your app-specific, private-API, personal-data-passing-mechanism is becoming even better-insulated against name-space collisions with the small number of official reserved header names. That said, my personal preference and recommendation is to go a step further and do e.g. "X-ACME-ClientDataFoo" (if your widget company is "ACME").

IMHO the IETF spec is insufficiently specific to answer the OP's question, because it fails to distinguish between completely different use cases: (A) vendors introducing new globally-applicable features like "Forwarded-For" on the one hand, vs. (B) app developers passing app-specific strings to/from client and server. The spec only concerns itself with the former, (A). The question here is whether there are conventions for (B). There are. They involve grouping the parameters together alphabetically, and separating them from the many standards-relevant headers of type (A). Using the "X-" or "X-ACME-" prefix is convenient and legitimate for (B), and does not conflict with (A). The more vendors stop using "X-" for (A), the more cleanly-distinct the (B) ones will become.

Example:
Google (who carry a bit of weight in the various standards bodies) are -- as of today, 20141102 in this slight edit to my answer -- currently using "X-Mod-Pagespeed" to indicate the version of their Apache module involved in transforming a given response. Is anyone really suggesting that Google should use "Mod-Pagespeed", without the "X-", and/or ask the IETF to bless its use?

Summary:
If you're using custom HTTP Headers (as a sometimes-appropriate alternative to cookies) within your app to pass data to/from your server, and these headers are, explicitly, NOT intended ever to be used outside the context of your application, name-spacing them with an "X-" or "X-FOO-" prefix is a reasonable, and common, convention.

RFC6648 recommends that you assume that your custom header "might become standardized, public, commonly deployed, or usable across multiple implementations." Therefore, it recommends not to prefix it with "X-" or similar constructs.

However, there is an exception "when it is extremely unlikely that [your header] will ever be standardized." For such "implementation-specific and private-use" headers, the RFC says a namespace such as a vendor prefix is justified.