API links vs keys: Why you should use links, not keys, to represent relationships in APIs

When it comes to information modeling, how to denote the relation or relationship between two entities is a key question. Describing the patterns that we see in the real world in terms of entities and their relationships is a fundamental idea that goes back at least as far as the ancient Greeks, and is also the foundation of how we think about information in IT systems today.

For example, relational database technology represents relationships using a foreign key, a value stored in one row in a database table that identifies another row, either in a different table or the same table.

 Expressing relationships is very important in APIs too. For example, in a retailer's API, the entities of the information model might be customers, orders, catalog items, carts, and so on. The API expresses which customer an order is for, or which catalog items are in a cart. A banking API expresses which customer an account belongs to or which account each credit or debit applies to.

The most common way that API developers express relationships is to expose database keys, or proxies for them, in the fields of the entities they expose. However, at least for web APIs, that approach has several disadvantages over the alternative: the web link.

Standardized by the Internet Engineering Task Force (IETF), you can think of a web link as a way of representing relationships on the web. The best-known web links are of course those that appear in HTML web pages expressed using the link or anchor elements, or in HTTP headers. But links can appear in API resources too, and using them instead of foreign keys significantly reduces the amount of information that has to be separately documented by the API provider and learned by the user.

A link is an element in one web resource that includes a reference to another resource along with the name of the relationship between the two resources. The reference to the other entity is written using a special format called Uniform Resource Identifier (URI), for which there is an IETF standard. The standard uses the word ‘resource’ to mean any entity that is referenced by a URI. The relationship name in a link can be thought of as being analogous to the column name of a relational database foreign key column, and the URI in the link is analogous to the foreign key value. By far the most useful URIs are the ones that can be used to get information about the referenced resource using a standard web protocol—such URIs are called Uniform Resource Locators (URLs)—and by far the most important kind of URL for APIs is the HTTP URL.

While links aren’t widely used in APIs, some very prominent web APIs use links based on HTTP URLs to represent relationships, for example the Google Drive API and the GitHub API. Why is that? In this post, I'll show what using API foreign keys looks like in practice, explain its disadvantages compared to the use of links, and show you how to convert that design to one that uses links.

Representing relationships using foreign keys
Consider the popular pedagogic "pet store" application. The application stores electronic records to track pets and their owners. Pets have attributes like name, species and breed. Owners have names and addresses. Each pet has a relationship to its owner—the inverse of the relationship defines the pets owned by a particular owner.

In a typical key-based design the pet store application’s API makes two resources available that look like this:

The relationship between Lassie and Joe is expressed in the representation of Lassie using the "owner" name/value pair. The inverse of the relationship is not expressed. The "owner" value, "98765," is a foreign key. It is likely that it really is a database foreign key—that is, it is the value of the primary key of some row in some database table—but even if the API implementation has transformed the key values a bit, it still has the general characteristics of a foreign key.

The value "98765" is of limited direct use to a client. For the most common uses, the client needs to compose a URL using this value, and the API documentation needs to describe a formula for performing this transformation. This is most commonly done by defining a URI template, like this:

/people/{person_id}

The inverse of the relationship—the pets belonging to an owner—can also be exposed in the API by implementing and documenting one of the following URI templates (the difference between the two is a question of style, not substance):

APIs that are designed in this way usually require many URI templates to be defined and documented. The most popular language for documenting these templates for APIs isn’t the one defined in the IETF specification—it’s OpenAPI (formerly known as Swagger). Until version 3.0, OpenAPI did not provide a way to specify which field values can be plugged into which templates, so some amount of natural language documentation from the provider or guesswork by the client was also required. Version 3.0 of OpenAPI introduced a new syntax called "links" to address this need, but using this feature consistently takes work.

In summary, although this style is common, it requires the provider to document, and the client to learn and use, a significant number of URI templates whose usage is not perfectly described by current API specification languages. Fortunately there’s a better option.

Representing relationships using links
Imagine the resources above were modified to look like this:

The primary difference is that the relationships are expressed using links, rather than foreign key values. In these examples, the links are expressed using simple JSON name/value pairs (see the section below for a discussion of other approaches to writing links in JSON).

Note also that the inverse relationship of the pet to its owner has been made explicit by adding the "pets" field to the representation of Joe.

Changing "id" to "self" isn't really necessary or significant, but it’s a common convention to use "self" to identify the resource whose attributes and relationships are specified by the other name/value pairs in the same JSON object. "self" is the name registered at IANA for this purpose.

Viewed from an implementation point of view, replacing all the database keys with links is a fairly simple change—the server converted the database foreign keys into URLs so the client didn't have to—but it significantly simplifies the API and reduces the coupling of the client and the server. Many URI templates that were essential for the first design are no longer required and can be removed from the API specification and documentation.

The server is now free to change the format of new URLs at any time without affecting clients (of course, the server must continue to honor all previously-issued URLs). The URL passed out to the client by the server will have to include the primary key of the entity in a database plus some routing information, but because the client just echoes the URL back to the server and the client is never required to parse the URL, clients do not have to know the format of the URL. This reduces coupling between the client and server. Servers can even obfuscate their URLs with base64 or similar encoding if they want to emphasize to clients that they should not make assumptions about URL formats or infer meaning from them.

In the example above, I used a relative form of the URIs in the links, for example /people/98765. It might have been slightly more convenient for the client (although less convenient for the formatting of this blog post), if I had expressed the URIs in absolute form, e.g., https://pets.org/people/98765. Clients only need to know the standard rules of URIs defined in the IETF specifications to convert between these two URI forms, so which form you choose to use is not as important as you might at first assume. Contrast this with the conversion from foreign key to URL described previously, which requires knowledge that is specific to the pet store API. Relative URLs have some advantages for server implementers, as described below, but absolute URLs are probably more convenient for most clients, which is perhaps why Google Drive and GitHub APIs use absolute URLs.

In short, using links instead of foreign keys to express relationships in APIs reduces the amount of information a client needs to know to use an API, and reduces the ways in which clients and servers are coupled to each other.

Caveats
Here are some things you should think about before using links.

Many API implementations have reverse proxies in front of them for security, load-balancing, and other reasons. Some proxies like to rewrite URLs. When an API uses foreign keys to represent relationships, the only URL that has to be rewritten by a proxy is the main URL of the request. In HTTP, that URL is split between the address line (the first header line) and the host header.

In an API that uses links to express relationships, there will be other URLs in the headers and bodies of both the request and the response that would also need to be rewritten. There are a few different ways of dealing with this:

1. Don't rewrite URLs in proxies. I try to avoid URL rewriting, but this may not be possible in your environment.

2. In the proxy, be careful to find and map all URLs wherever they appear in the header or body of the request and response. I have never done this, because it seems to me to be difficult, error-prone, and inefficient, but others may have done it.

3. Write all links relatively. In addition to allowing proxies some ability to rewrite URLs, relative URLs may make it easier to use the same code in test and production, because the code does not have to be configured with knowledge of its own host name. Writing links using relative URLs with a single leading slash, as I showed in the example above, has few downsides for the server or the client, but it only allows the proxy to change the host name (more precisely, the parts of the URL called the scheme and authority), not the path. Depending on the design of your URLs, you could allow proxies some ability to rewrite paths if you are willing to write links using relative URLs with no leading slashes, but I have never done this because I think it would be complicated for servers to write those URLs reliably. Relative URLs without leading slashes are also more difficult for clients to use—they need to use a standards-compliant library rather than simple string concatenation to handle those URLs, and they need to be careful to understand and preserve the base URL. Using a standards-compliant library to handle URLs is good practice for clients anyway, but many don't.

Using links may also cause you to re-examine how you do API versioning. Many APIs like to put version numbers in URLs, like this:

This is the kind of versioning where the data for a single resource can be viewed in more than one "format" at the same time—these are not the sort of versions that replace each other in time sequence as edits are made.

This is closely analogous to being able to see the same web document in different natural languages, for which there is a web standard; it is a pity there isn't a similar one for versions. By giving each version its own URL, you raise each version to the status of a full web resource. There is nothing wrong with "version URLs" like these, but they are not suitable for expressing links. If a client asks for Lassie in the version 2 format, it does not mean that they also want the version 2 format of Lassie's owner, Joe, so the server can't pick which version number to put in a link. There may not even be a version 2 format for owners. It also doesn't make conceptual sense to use the URL of a particular version in links—Lassie is not owned by a specific version of Joe, she is owned by Joe himself. So, even if you expose a URL of the form /v1/people/98765 to identify a specific version of Joe, you should also expose the URL /people/98765 to identify Joe himself and use the latter in links. Another option is to define only the URL /people/98765 and allow clients to request a specific version by including a request header. There is no standard for this header, but calling it Accept-Version would fit well with the naming of the standard headers. I personally prefer the approach of using a header for versioning and avoiding URLs with version numbers, but URLs with version numbers are popular, and I often implement both a header and "version URLs" because it's easier to do both than argue about it. For more on API versioning, check out this blog post.

You might still need to document some URL templates

In most web APIs, the URL of a new resource is allocated by the server when the resource is created using POST. If you use this method for creation and you are using links for relationships, you do not need to publish a URI template for the URIs of these resources. However, some APIs allow the client to control the URL of a new resource. Letting clients control the URL of new resources makes many patterns of API scripting much easier for client programmers, and it also supports scenarios where an API is used to synchronize an information model with an external information source. HTTP has a special method for this purpose: PUT. PUT means "create the resource at this URL if it does not already exist, otherwise update it"¹. If your API allows clients to create new entities using PUT, you have to document the rules for composing new URLs, probably by including a URI template in the API specification. You can also allow clients partial control of URLs by including a primary key-like value in the body or headers of a POST. This doesn't require a URI template for the POST itself, but the client will still need to learn a URI template to take advantage of the resulting predictability of URIs.

The other place where it makes sense to document URL templates is when the API allows clients to encode queries in URLs. Not every API lets you query its resources, but this can be a very useful feature for clients, and it is natural to let clients encode queries in URLs and use GET to retrieve the result. The following example shows why.

In the example above we included the following name/value pair in the representation of Joe:

The client doesn’t have to know anything about the structure of this URL, beyond what’s written in standard specifications, to use it. This means that a client can get the list of Joe's pets from this link without learning any query language, and without the API having to document its URL formats—but only if the client first does a GET on /people/98765. If, in addition, the pet store API documents a query capability, the client can compose the same or equivalent query URL to retrieve the pets for an owner without first retrieving the owner—it is sufficient to know the owner's URI. Perhaps more importantly, the client can also form queries like the following ones that would otherwise not be possible:

The URI specification describes a portion of the HTTP URL for this purpose called the query component—the portion of the URL after the first “?” and before the first “#”. The style of query URI that I prefer always puts client-specified queries in the query component of the URI, but it’s also permissible to express client queries in the path portion of a URL. In either case, you need to describe to clients how to compose these URLs—you are effectively designing and documenting a query language specific to your API. Of course, you can also allow clients to put queries in the request body rather than the URL and use the POST method instead of GET. Since there are practical limits on the size of a URL—anything over 4k bytes is tempting fate—it is a good practice to support POST for queries even if you also support GET.

Because query is such a useful feature in APIs, and because designing and implementing query languages is not easy, technologies like GraphQL have emerged. I have never used GraphQL, so I can't endorse it, but you may want to evaluate it as an alternative to designing and implementing your own API query capability. API query capabilities, including GraphQL, are best used as a complement to a standard HTTP API for reading and writing resources, not an alternative.

And another thing… What’s the best way to write links in JSON?
Unlike HTML, JSON has no built-in mechanism for expressing links. Many people have opinions on how links should be expressed in JSON and some have published their opinions in more or less official-looking documents, but there is no standard ratified by a recognized standards organization at the time of writing. In the examples above, I used simple JSON name/value pairs to express links—this is my preferred style, and is also the style used by Google Drive and GitHub. Another style that you will likely encounter looks like this:

I personally don't see the merits of this style, but several variants of it have achieved some level of popularity.

There is another style for links in JSON that I do like, which looks like this:

The benefit of this style is that it makes it explicit that "/people/98765" is a URL and not just a string. I learned this pattern from RDF/JSON. One reason to adopt this pattern is that you will probably have to use it anyway whenever you have to show information about one resource nested inside another, as in the following example, and using it everywhere gives a nice uniformity:

For further ideas on how best to use JSON to represent data, see Terrifically Simple JSON.

Finally, what’s the difference between an attribute and a relationship?
I think most people would agree with the statement that JSON does not have a built-in mechanism for expressing links, but there is a way of thinking about JSON that says otherwise. Consider this JSON:

A common view is that shoeSize is an attribute, not a relationship, and 10 is a value, not an entity. However, it is also reasonable to say that the string '10” is in fact a reference, written in a special notation for writing references to numbers, to the eleventh whole number, which itself is an entity. If the eleventh whole number is a perfectly good entity, and the string '10' is just a reference to it, then the name/value pair '"shoeSize": 10' is conceptually a link, even though it doesn't use URIs.

The same argument can be made for booleans and strings, so all JSON name/value pairs can be viewed as links. If you think this way of looking at JSON makes sense, then it’s natural to use simple JSON name/value pairs for links to entities that are referenced using URLs in addition to those that are referenced using JSON's built-in reference notations for numbers, strings, booleans and null.

This argument says more generally that there is no fundamental difference between attributes and relationships; attributes are just relationships between an entity and an abstract or conceptual entity like a number or color that has historically been treated specially. Admittedly, this is a rather abstract way of looking at the world—if you show most people a black cat, and ask them how many objects they see, they will say one. Not many would say they see two objects—a cat, and the color black—and a relationship between them.

Links are simply better
Web APIs that pass out database keys rather than links are harder to learn and harder to use for clients. They also couple clients and servers more tightly together by requiring more shared knowledge and so they require more documentation to be written and read. Their only advantage is that because they are so common, programmers have become familiar with them and know how to produce them and consume them. If you strive to offer your clients high-quality APIs that don’t require a ton of documentation and maximize independence of clients from servers, think about exposing URLs rather than database keys in your web APIs.

For more on API design, read the eBook “Web API Design: The Missing Link.”

_{1. This meaning can be refined using the if-match and if-not-match headers}