Create a dynamic sitemap with Next.js

dreamecho100

dreamecho100

Mazen Mohamed
created at: tags: sitemap, next.js
Create a dynamic sitemap with Next.js

If you're building a Next.js site or app that needs to be visible to search engines like google, bing, and other search engines, a sitemap is essential especially if it has many pages that you want to be indexed.

What is a sitemap

A sitemap is a map of the URLs on your site and makes it easier for search engine crawler bots to index your content, and increase your likelihood for ranking in search results.

browsers and search engines anticipate our sitemap being returned as an XML file

What will we use?

In Next.js we can rely on:

  • The built-in router.

  • [ "getServerSideProps" ] function provided by Next.js.

  • basic data fetching (for dynamic content).

  • installing [ "sitemap" ] more about it here package by npm/yarn, and use it with [ "stream" ] package provided by a node to easily build the XML Content.

Start

Let's start the Work!

in your [ "/pages" ] directory at the root folder create a file named [ "sitemap.xml.js" ] and add the following to it.

1const Sitemap = () => {};
2
3export default Sitemap;

Next.js will automatically create a route in our app at /sitemap.xml which is the location where browsers and search engine crawlers expect our sitemap to live.

Why is it an empty returned function you ask?

because we don't want to render a component at the URL.

Instead we will use [ "getServerSideProps" ] and build the XML content and return it as [ "text/xml" ]

let's continue.

First, in your terminal navigate to the root component of your project and type [ "npm I sitemap" ]

in the [ "/pages/sitemap.xml.js" ] do as the following:

1import { SitemapStream, streamToPromise } from 'sitemap';
2import { Readable } from 'stream';
3
4const Sitemap = () => {};
5
6export const getServerSideProps = ({ res }) => {
7	// An array with your links
8	const links = [];
9
10	// Add static pages
11	const pages = ['/auth', '/about'];
12
13	pages.map((url) => {
14		links.push({
15			url,
16			lastmod: new Date().toISOString(),
17			changefreq: 'monthly',
18			priority: 0.9,
19		});
20	});
21
22	const stream = new SitemapStream({
23		hostname: `https://${req.headers.host}`,
24	});
25
26	res.writeHead(200, {
27		'Content-Type': 'text/xml',
28	});
29
30	const xmlString = await streamToPromise(
31		Readable.from(links).pipe(stream)
32	).then((data) => data.toString());
33
34	res.write(xmlString);
35	res.end();
36
37	return {
38		props: {},
39	};
40};
41
42export default Sitemap;

Now, What happened?

  1. Instead of returning an object of props from getServerSideProps We will override it.

  2. We built the [ "links" ] array that we will push an object with [ "url" ], [ "changefreq" ], [ "priority" ] to it

  3. Create a stream to write to and add the [ "hostname" ] to it.

  4. Set the Content-Type header of the response to `text/XML.

  5. Write the response body and end the request.

if we remove the [ "return { props: {}, };" ] Next.js will throw an error at us

Congratulations now if you start your development server by typing [ "npm run dev" ] and in your browser and go to [ "localhost:3000/sitemap.xml" ] you will get the following:

1<urlset
2	xmlns='http://www.sitemaps.org/schemas/sitemap/0.9'
3	xmlns:news='http://www.google.com/schemas/sitemap-news/0.9'
4	xmlns:xhtml='http://www.w3.org/1999/xhtml'
5	xmlns:image='http://www.google.com/schemas/sitemap-image/1.1'
6	xmlns:video='http://www.google.com/schemas/sitemap-video/1.1'
7>
8	<url>
9		<loc>https://localhost:3000/auth</loc>
10		<lastmod>2021-09-09T20:19:53.534Z</lastmod>
11		<changefreq>monthly</changefreq>
12		<priority>0.9</priority>
13	</url>
14	<url>
15		<loc>https://localhost:3000/about</loc>
16		<lastmod>2021-09-09T20:19:53.534Z</lastmod>
17		<changefreq>monthly</changefreq>
18		<priority>0.9</priority>
19	</url>
20</urlset>

In terms of what we're returning here we return the XML content expected by a web browser (or search engine crawler) when reading a sitemap.

  • For each URL in our site that we want to add to our map, we add the [ "<url></url>" ] tag.

  • Placing a [ "<loc></loc>" ] tag inside that specifies the location of our URL.

  • The [ "<lastmod></lastmod>" ] tag specifies when the content at the URL was last updated.

Time is in ISO-8601 string If you have a date available for when these pages were last updated, it's best to be as accurate as possible with this date and pass that specific date here

  • The [ "<changefreq></changefreq>" ] tag specifies how frequently the content at the URL is updated.

We're setting a sensible default of monthly, but this can be any of the following:

  • never
  • yearly,
  • monthly
  • weekly
  • daily
  • hourly
  • always
  • And a [ "<priority></priority>" ] tag to specify the importance of the URL (which translates to how frequently a crawler should crawl that page).

we set a base of 1.0 (the maximum level of importance). If you wish to change this to be more specific, this number can be anything between 0.0 and 1.0 with 0.0 being unimportant, 1.0 being most important.

Adding paths dynamically to the sitemap

Let's say we have a dynamic path like [ "Profile/[user_name_id]" ] or [ "article/[slug]" ] how can we add them to the sitemap without adding them manually every single time.

Super easy barely an unconvinced

I have built a function to return the [ "user_name_id" ] and another one to return the [ "slug" ] needed from the database and put it in the [ "/lib" ] directory in the root project.

Create the functions and put them were ever you like and use the path to it to export the functions

example for the [ "user_name_id" ] function:

1// I am using pg package to connect to the my postgresql database
2const { Pool } = require('pg');
3
4export const connectionString = process.env.PG_CONNECTION_STRING;
5
6export const pool = new Pool({
7	connectionString
8});
9// I will return the data using the new Promise so I can easily access it through then
10
11export const getAllUsersNameId = () => {
12	return new Promise(async (resolve, reject) => {
13		try {
14			await pool
15				.query('SELECT user_name_id FROM user_profile')
16				.then(async (response) => response.rows)
17				.then((data) => resolve(data));
18		} catch (error) {
19			console.error(`Error, ${error.message}`);
20			resolve([]);
21		}
22	});
23};
24
25export const getAllArticlesSlugs = () => {
26	return new Promise(async (resolve, reject) => {
27		try {
28			await pool
29				.query('SELECT slug FROM news_article')
30				.then(async (response) => response.rows)
31				.then((data) => resolve(data));
32		} catch (error) {
33			console.error(`Error, ${error.message}`);
34			resolve([]);
35		}
36	});
37};

and now in [ "page/sitemap.xml.js" ] we will import them:

1import { SitemapStream, streamToPromise } from 'sitemap';
2import { Readable } from 'stream';
3
4import { getAllArticlesSlugs, getAllUsersNameId } from '../lib/pg';
5
6const Sitemap = () => {};
7
8// Don't forget to add async when using await
9export const getServerSideProps = async ({ res }) => {
10	const links = [];
11
12	await getAllArticlesSlugs().then((articles = []) => {
13		articles.map((article) => {
14			links.push({
15				url: `/article/${article.slug}`,
16				lastmod: new Date().toISOString(),
17				changefreq: 'weekly',
18				priority: 0.9
19			});
20		});
21	});
22
23	await getAllUsersNameId().then((profiles = []) => {
24		profiles.map((Profile) => {
25			links.push({
26				url: `/Profile/${Profile.user_name_id}`,
27				lastmod: new Date().toISOString(),
28				changefreq: 'weekly',
29				priority: 0.9
30			});
31		});
32	});
33
34	const pages = ['/auth', '/about'];
35
36	pages.map((url) => {
37		links.push({
38			url,
39			lastmod: new Date().toISOString(),
40			changefreq: 'monthly',
41			priority: 0.9
42		});
43	});
44
45	const stream = new SitemapStream({
46		hostname: `https://${req.headers.host}`
47	});
48
49	res.writeHead(200, {
50		'Content-Type': 'text/xml'
51	});
52
53	const xmlString = await streamToPromise(
54		Readable.from(links).pipe(stream)
55	).then((data) => data.toString());
56
57	res.write(xmlString);
58	res.end();
59
60	return {
61		props: {}
62	};
63};
64
65export default Sitemap;

And now when we visit [ "localhost:3000/sitemap.xml" ] in the browser we get:

1<urlset
2	xmlns='http://www.sitemaps.org/schemas/sitemap/0.9'
3	xmlns:news='http://www.google.com/schemas/sitemap-news/0.9'
4	xmlns:xhtml='http://www.w3.org/1999/xhtml'
5	xmlns:image='http://www.google.com/schemas/sitemap-image/1.1'
6	xmlns:video='http://www.google.com/schemas/sitemap-video/1.1'
7>
8	<url>
9		<loc>
10			https://localhost:3000/article/full-guide-to-cookies-and-javascript-clint-side
11		</loc>
12		<lastmod>2021-09-09T20:21:47.464Z</lastmod>
13		<changefreq>weekly</changefreq>
14		<priority>0.9</priority>
15	</url>
16	<url>
17		<loc>
18			https://localhost:3000/article/basic-guide-to-json-in-postgresql-jsonb
19		</loc>
20		<lastmod>2021-09-09T20:21:47.464Z</lastmod>
21		<changefreq>weekly</changefreq>
22		<priority>0.9</priority>
23	</url>
24	<url>
25		<loc>
26			https://localhost:3000/article/vocabulary-workshop-level-f-grade-11-unit-1
27		</loc>
28		<lastmod>2021-09-09T20:21:47.464Z</lastmod>
29		<changefreq>weekly</changefreq>
30		<priority>0.9</priority>
31	</url>
32	<url>
33		<loc>https://localhost:3000/Profile/mazen-mohamed</loc>
34		<lastmod>2021-09-09T20:21:47.688Z</lastmod>
35		<changefreq>weekly</changefreq>
36		<priority>0.9</priority>
37	</url>
38	<url>
39		<loc>https://localhost:3000/Profile/mohamed-bek</loc>
40		<lastmod>2021-09-09T20:21:47.688Z</lastmod>
41		<changefreq>weekly</changefreq>
42		<priority>0.9</priority>
43	</url>
44	<url>
45		<loc>https://localhost:3000/about</loc>
46		<lastmod>2021-09-09T20:21:47.688Z</lastmod>
47		<changefreq>monthly</changefreq>
48		<priority>0.9</priority>
49	</url>
50	<url>
51		<loc>https://localhost:3000/auth</loc>
52		<lastmod>2021-09-09T20:21:47.689Z</lastmod>
53		<changefreq>monthly</changefreq>
54		<priority>0.9</priority>
55	</url>
56</urlset>

Resources