[rest] API pagination best practices

Option A: Keyset Pagination with a Timestamp

In order to avoid the drawbacks of offset pagination you have mentioned, you can use keyset based pagination. Usually, the entities have a timestamp that states their creation or modification time. This timestamp can be used for pagination: Just pass the timestamp of the last element as the query parameter for the next request. The server, in turn, uses the timestamp as a filter criterion (e.g. WHERE modificationDate >= receivedTimestampParameter)

{
    "elements": [
        {"data": "data", "modificationDate": 1512757070}
        {"data": "data", "modificationDate": 1512757071}
        {"data": "data", "modificationDate": 1512757072}
    ],
    "pagination": {
        "lastModificationDate": 1512757072,
        "nextPage": "https://domain.de/api/elements?modifiedSince=1512757072"
    }
}

This way, you won't miss any element. This approach should be good enough for many use cases. However, keep the following in mind:

  • You may run into endless loops when all elements of a single page have the same timestamp.
  • You may deliver many elements multiple times to the client when elements with the same timestamp are overlapping two pages.

You can make those drawbacks less likely by increasing the page size and using timestamps with millisecond precision.

Option B: Extended Keyset Pagination with a Continuation Token

To handle the mentioned drawbacks of the normal keyset pagination, you can add an offset to the timestamp and use a so-called "Continuation Token" or "Cursor". The offset is the position of the element relative to the first element with the same timestamp. Usually, the token has a format like Timestamp_Offset. It's passed to the client in the response and can be submitted back to the server in order to retrieve the next page.

{
    "elements": [
        {"data": "data", "modificationDate": 1512757070}
        {"data": "data", "modificationDate": 1512757072}
        {"data": "data", "modificationDate": 1512757072}
    ],
    "pagination": {
        "continuationToken": "1512757072_2",
        "nextPage": "https://domain.de/api/elements?continuationToken=1512757072_2"
    }
}

The token "1512757072_2" points to the last element of the page and states "the client already got the second element with the timestamp 1512757072". This way, the server knows where to continue.

Please mind that you have to handle cases where the elements got changed between two requests. This is usually done by adding a checksum to the token. This checksum is calculated over the IDs of all elements with this timestamp. So we end up with a token format like this: Timestamp_Offset_Checksum.

For more information about this approach check out the blog post "Web API Pagination with Continuation Tokens". A drawback of this approach is the tricky implementation as there are many corner cases that have to be taken into account. That's why libraries like continuation-token can be handy (if you are using Java/a JVM language). Disclaimer: I'm the author of the post and a co-author of the library.

Examples related to rest

Access blocked by CORS policy: Response to preflight request doesn't pass access control check Returning data from Axios API Access Control Origin Header error using Axios in React Web throwing error in Chrome JSON parse error: Can not construct instance of java.time.LocalDate: no String-argument constructor/factory method to deserialize from String value How to send json data in POST request using C# How to enable CORS in ASP.net Core WebAPI RestClientException: Could not extract response. no suitable HttpMessageConverter found REST API - Use the "Accept: application/json" HTTP Header 'Field required a bean of type that could not be found.' error spring restful API using mongodb MultipartException: Current request is not a multipart request

Examples related to pagination

How to use paginator from material angular? how to implement Pagination in reactJs In Flask, What is request.args and how is it used? Custom pagination view in Laravel 5 Simple pagination in javascript UITableView load more when scrolling to bottom like Facebook application How to use pagination on HTML tables? how to remove pagination in datatable Laravel Pagination links not including other GET parameters API pagination best practices

Examples related to api-design

When do I use path params vs. query params in a RESTful API? What are best practices for REST nested resources? API pagination best practices What is the proper REST response code for a valid request but an empty data?