Mastering Pagination in Express:A Step-by-Step Guide for Efficient API

Introduction

When building applications, we interact with a lot of data. As backend developers, we need to learn how to manage our data and ensure that clients can fetch the data they want, no matter how large. Pagination is one of the ways we can achieve this.
Pagination refers to dividing content into smaller/discrete pages. Pagination can also be referred to as paging.

How does pagination improve our websites?

It gives our pages direction and structure.
When we don't paginate, we overwhelm the user with scrolling.
Continuous scrolling might result in text being hastily skimmed rather than thoroughly read. Pagination encourages users to make tiny commitments at regular intervals (clicking to the next page), which can improve retention and engagement.
Pagination helps users remember where they left off. If a user is on the fifth page of a blog, they may quickly return to where they left off, which is not as simple with endless scrolling.

In this article, we will be implementing pagination in our express.js application using a relational database (Postgres); in the next series, we will use a non-relational database (MongoDB), and the next we’ll see if we can do this same operation in Nestjs.

How Pagination works

Pagination is a technique for dividing huge datasets into smaller, more manageable parts. This is critical for optimizing the speed and user experience of programs that must show or process massive volumes of data. Here is an in-depth look at how pagination works:

The pagination process

The pagination process undergoes the following steps:

User Request: The client requests a specific page by sending a page number via a query or parameter. The client can also request the limit, which means the number of items that should be displayed per page. Some other parameters that can be passed by the client are the filter parameters, this would filter our responses based on the passed parameters. The client can also attach the date range to the request. The date range ensures that the queried data is between the specified dates.
Data fetching: With these in place, the data is fetched from the database and the response is constructed to return the requested data

Parameters in pagination

Pagination: This is the breaking down of large data into small chunks. For example, our user database has a lot of users—over 5000 users. But we know that to fetch and display this data, it would be a large task. So when fetching the data, we decided to break it into smaller chunks. To achieve this, we need to know the following to make our work easier:
Page Size/Limit: This gives us the number of items displayed per page (or chunk of data fetched).
Page Number: This information gives the current page.
Total Page: This gives the number of pages that the fetched data has been split into.
Total Count: This gives the total count of items items that exist in the database
Offset: This is an internal calculation used to determine the starting point for the data to be fetched. It is calculated as follows:(page - 1) * limit

In this article we will be exploring pagination in express app, using Sequelize as our ORM to communicate with our relational database. Next series we will be try NoSQL databases.

How to paginate in an express application using `sequelize` for a relational database

To start this simple project, let's start by cloning our boilerplate code from this repo

# Create a folder for our project
mkdir pagination

# navigate into it and clone the repo into a relational-db folder
git clone https://github.com/nwokporochukwuebuka/express-sequlize.git relational-db

# navigate into the relational-db directory
cd relational-db

# install all the necessary dependencies
npm install

After installing, you should have a folder structure of this sort

└── relational-db
    └── .env
    └── .gitignore
    └── Makefile
    └── package-lock.json
    └── package.json
    └── readme.md
    └── src
        └── app.js
        └── config
            └── config.js
        └── controllers
            └── user.controller.js
        └── index.js
        └── middlewares
            └── error.js
            └── validate.js
        └── models
            └── index.js
            └── user.model.js
        └── routers
            └── v1
                └── index.js
                └── user.route.js
        └── services
            └── user.service.js
        └── utils
            └── ApiError.js
            └── catchAsync.js
            └── pick.js
        └── validations

Please don't forget to add your .env file to run your project and also update your Makefile with these variables. The purpose was to help us run Postgres using Docker.

When fetching paginated data, some fields are

Now, to the task at hand: Now in our utils folder, create a file called query-helper.js

touch ./src/utils/query-helper.js

Inside the query-helper.js file, let's paste the following line of code

const { Op } = require("sequelize");

const getPaginatedData = async (
  model,
  {
    page = 1,
    limit = 10,
    sortField = "createdAt",
    sortOrder = "DESC",
    filters = {},
    dateField = "createdAt",
    startDate,
    endDate,
    include = [],
  }
) => {
  // This tells us where the data should be fetched from
  const offset = (page - 1) * limit;

  // incase we have filters, 
  const where = {};

  // This caters for filters that have multiple values e.g status = active or inactive
  Object.keys(filters).forEach((key) => {
    if (Array.isArray(filters[key])) {
      where[key] = {
        [Op.or]: filters[key],
      };
    } else {
      where[key] = filters[key];
    }
  });

  // Caters for date range
  if (startDate && endDate) {
    where[dateField] = {
      [Op.between]: [new Date(startDate), new Date(endDate)],
    };
  } else if (startDate) {
    where[dateField] = {
      [Op.gte]: new Date(startDate),
    };
  } else if (endDate) {
    where[dateField] = {
      [Op.lte]: new Date(endDate),
    };
  }

  if (include.length > 0) {
    const { count, rows } = await model.findAndCountAll({
      where,
      order: [[sortField, sortOrder]],
      limit,
      offset,
      include,
    });
    return {
      totalItems: count,
      totalPages: Math.ceil(count / limit),
      currentPage: page,
      limit,
      data: rows,
    };
  }
  // Fetch data with pagination, sorting, and filtering
  const { count, rows } = await model.findAndCountAll({
    where,
    order: [[sortField, sortOrder]],
    limit,
    offset,
  });
  return {
    totalItems: count,
    totalPages: Math.ceil(count / limit),
    currentPage: page,
    limit,
    data: rows,
  };
};

module.exports = {
  getPaginatedData,
};

Now let's take the function bit by bit to understand properly,

Import Function and Definition

const { Op } = require("sequelize");

The Op function is a function that contains most sequelize comparison operators. Here are some comparison operators: $gt,$gte,$lt,$lte,$or,$eq,$like,$not,$is,$ne, etc.

const getPaginatedData = async (
  model,
  {
    page = 1,
    limit = 10,
    sortField = "createdAt",
    sortOrder = "DESC",
    filters = {},
    dateField = "createdAt",
    startDate,
    endDate,
    include = [],
  }
) => {
  // body
};

modelThis parameter represents the sequelize parameter to query.
pageThis supplies the page number that should be queried; it defaults to 1 if not supplied by the client.
limitThis gives the number of records per page; if not provided, it defaults to.10
sortFieldThis is the field to sort the results by; for databases created with Sequelize, we love to use the createdAt field.
sortOrderThis parameter specifies the order of sorting this result in descending or ascending order. default is,DESC which represents descending order. To use ascending order, we pass.ASC
filtersThis is an object containing the filter criteria. We could decide to filter based on different fields in our table; e.g., in the status field, a user can either beactiveinactive, or.suspended If we want to filter based on this status, then we pass it into our filter, so our filter will be
```
  filter = {
      status: "active"
  }
```
In this example, our filter is just a single value, i.e., we want to fetch users with an active status. If we also want to fetch users that are either active or,inactive here is how we go about it:
```
  filter = {
      status: ["active", "inactive"]
  }
```
dateFieldThis parameter tells us which field we will use for the date range filter. The default field is usually createdAt
startDateThisendDate is the passed start and end date for the date range.
includeThis represents the array of models associated with the query. This is useful when there is an association between models and we want to fetch the related model.

Unto the next,

Calculateoffset

offsetdetermines the starting point for fetching the records. Here is how we calculate the offset:

const offset = (page -1) * limit

Handlefilters

const where = {};

Object.keys(filters).forEach((key) => {
  if (Array.isArray(filters[key])) {
    where[key] = {
      [Op.or]: filters[key],
    };
  } else {
    where[key] = filters[key];
  }
});

This handles the filter based on a certain field. The where variable holds the condition for the query.

When a filtered value has more than one possibility, e.g when the query should fetch both active or inactive users, our filter would be

filters = {
    status: ['active', 'inactive'],
}

This would check if the supplied filter contains an array, and if yes, the Op.or operator is used to query the database.

For a single filter condition, e.g for only active users, our filter would be

filter = {
    status: 'active',
}

PS: You can have multiple fields in your filter object. e.g

filter = {
    status: 'active',
    firstName: 'Eyilademi',
    lastName: 'Chukwuebuka',
}

Handle Date Range

if (startDate && endDate) {
    where[dateField] = {
      [Op.between]: [new Date(startDate), new Date(endDate)],
    };
  } else if (startDate) {
    where[dateField] = {
      [Op.gte]: new Date(startDate),
    };
  } else if (endDate) {
    where[dateField] = {
      [Op.lte]: new Date(endDate),
    };
  }

This handles the date range, so when the startDate and endDate the field is submitted to the,dateField which is createdAt by default given a value to search between using the sequelize Op.between operator, meaning that the data provided will be greater than the startDate but less than the.endDate If only the startDate is provided, then the queried data must have dates that are greater than the.startDate Likewise, if onlyendDate the queried data should have data provided dateField is less than theendDate

Querying withinclude

if (include.length > 0) {
  const { count, rows } = await model.findAndCountAll({
    where,
    order: [[sortField, sortOrder]],
    limit,
    offset,
    include,
  });
  return {
    totalItems: count,
    totalPages: Math.ceil(count / limit),
    currentPage: page,
    limit,
    data: rows,
  };
}

This option allows for eager loading. This addresses fetching associated models at once; you can read more on the Sequelize website.

The data is queried with the specified parameter if the include function is not specified.

Next, we update theuser.service.js,user.controller.jsfile:

// ....
const queryUsers = async (
  page,
  limit,
  sortField,
  sortOrder,
  startDate,
  endDate,
  filters
) =>
  await getPaginatedData(db.users, {
    page: parseInt(page === undefined ? page : 1, 10),
    limit: parseInt(limit === undefined ? limit : 10, 10),
    sortField: sortField || "createdAt",
    sortOrder: sortOrder || "DESC",
    dateField: "createdAt",
    filters,
    startDate,
    endDate,
  });

module.exports = {
    // ...,
    queryUsers,
}

user.controller.js

// ....
const getUsers = catchAsync(async (req, res) => {
  const { page, limit, startDate, endDate, status, firstName, lastName, sortField, sortBy } =
    req.query;

  const filters = {};

  if (status) {
    filters.status = status;
  }

  if (firstName) {
    filters.firstName = firstName;
  }

  if (lastName) {
    filters.lastName = lastName;
  }
  const users = await userService.queryUsers(
    page,
    limit,
    sortField || "createdAt",
    sortBy || "DESC",
    startDate,
    endDate,
    filters
  );
  return res.status(httpStatus.OK).json(users);
});

module.exports = {
    // ...
    getUsers,
}

💡

Note: When you expose it to your route, ensure that you guard it using a validator. Examples of some validators are Joi, Express Validator etc.

Now, when you want to use this on POSTMAN, this is how your query should look like.

Conclusion

Pagination is an essential skill for every backend developer and it is a powerful tool that enhances the usability and performance of our applications by managing large data and breaking it into smaller chunks which will improve both user experience and backend performance.
In this article, we explored the key concepts of pagination and demonstrated its implementation in Express. So, whether you are designing a social media feed, e-commerce platform, or any data-intensive application.

As you integrate pagination into your project, remember to tailor it to your specific use cases, and consider advanced techniques like cursor-based pagination for high-performance applications.

To receive an update when the article drops, do well to subscribe to my newsletter or follow me on any of the social media platforms.

Twitter 🐦: @nwokporo_ebuka

LinkedIn ⚡: @chukwuebuka_nwokporo

GitHub 🚀: @ebukvick

Hashnode 📗: Nwokporo Chukwuebuka

Pagination in Express

Efficiently Managing Large Datasets in Express Applications for a Seamless User Experience

Table of contents

Introduction

How to paginate in an express application using `sequelize` for a relational database

Conclusion

Pagination in Express

Efficiently Managing Large Datasets in Express Applications for a Seamless User Experience

Table of contents

Introduction

How does pagination improve our websites?

How Pagination works

The pagination process

Parameters in pagination

How to paginate in an express application using sequelize for a relational database

Conclusion

How to paginate in an express application using `sequelize` for a relational database