Salesforce, Python, SQL, & other ways to put your data where you need it

Need event music? 🎸

Live and recorded jazz, pop, and meditative music for your virtual conference / Zoom wedding / yoga class / private party with quality sound and a smooth technical experience

Next.js Markdown routing

24 Aug 2021 🔖 architecture web development jamstack minimum viable build
💬 EN

Table of Contents

I discovered that a directory only needs two small files in it for the Next.js static website generator to turn it into a functioning index.html with a body of <div>Hello world!</div>.

I’ve already covered some fundamentals of React componentization in my Gatsby series. I’m going to jump pretty far ahead from the minimum viable build” as I do my Next.js part 2, because when I was trying to write a series that mirrored the Gatsby one, what I found was that I couldn’t structure things the same way, so I couldn’t teach this series the same way.

In the end, what ended up being the hardest thing to do was build a complex routing engine that “felt right” to me with respect to the way I like to organize my collections of .md file (far, far away from the templates that render them, and able to end up all kinds of mixed up with each other as far as URL output is concerned).

You can see all the work I did on GitHub – the commit itself shows all changes from the Next.js minimum viable build

What I’ve built is a hybrid of my “Hello World” Gatsby Markdown project (as a “basic page” template) and my “WYSIWYG” Gatsby Markdown project (as an “advanced page” template).

End goal HTML

Files

This looks like a lot of files, but it’s not so bad:

  1. The /src/css/global.css file is copied & pasted from my “WYSIWYG” Gatsby Markdown project with no changes. That said, to get Next.js to recognize it, instead of putting code into a /gatsby-browser.js file, I had to put code into a /src/pages/_app.js file.
  2. There are 3 .md Markdown files instead of one, and a few more folders surrounding them, so the file tree looks more verbose.
  3. Instead of using system-wide files like gatsby-config.js and gatsby-node.js and a querying engine built into the static site generator, I had to write a few files under /src/util/ to find and parse Markdown files by hand.
  4. Instead of having a single /src/templates/xyzzy.js page-type template, the top-level template is now called /src/pages/[[...rooturlslug]].js (yes, that’s its filename!).
    • This template in turn delegates to /src/templates/advancedPage.js and /src/templates/basicPage.js.
    • In this Next.js project, advancedPage.js is almost an exact copy of the Gatsby project’s xyzzy.js template: it delegates content rendering to all of the /src/components/ files.
  5. The /src/templates/layoutHello.js file is copied & pasted from my “WYSIWYG” Gatsby Markdown project with only two changes:
    • I used semantic HTML header and footer tags because I’ve learned more about accessibility since I made the Gatsby theme
    • I broke out the Header Placeholder content into its own /src/components/header.js file that gives me a hand-coded rudimentary navbar to easily visit my site’s /, /about/, and /contact/ pages.
  6. I played with Next.js’s component-level CSS by adding /src/templates/advancedPage.module.css and /src/templates/basicPage.module.css files and importing them into their respective themes.
    • Both CSS files have a single class definition in them: testingcss.
    • It’s pretty cool – if I look at an “advanced” page, it’s surrounded by a pink border, and if I look at a “basic” page, it’s surrounded by a green border.
    • Either way, if I look at the HTML, I don’t see class="testingcss" – it’s more like class="advancedPage_testingcss__AGaVN" or class="basicPage_testingcss__1IGOp".
    • I’m sure this party trick would be even more impressive if I scoped redundant CSS class names to components meant to display in the same page, like sectionPink and sectionBlue, but I think I get the point as-is, and for teaching purposes, it’s nice not to change too much at once.
  7. Everything under /src/components/ is copied & pasted from my “WYSIWYG” Gatsby Markdown project verbatim, so there’s no new code to cover there.
.
├── src
│   ├── components
│   │   ├── header.js
│   │   ├── indexSectionComponents.js
│   │   ├── sectionBlue.js
│   │   ├── sectionPink.js
│   │   ├── sectionTaskList.js
│   │   └── task.js
│   ├── content
│   │   ├── advanced
│   │   │   └── contact.md
│   │   └── basic
│   │       ├── about.md
│   │       └── index.md
│   ├── pages
│   │   └── [[...rooturlslug]].js
│   ├── templates
│   │   ├── advancedPage.js
│   │   ├── advancedPage.module.css
│   │   ├── basicPage.js
│   │   ├── basicPage.module.css
│   │   ├── indexMdRootContentTemplates.js
│   │   └── layoutHello.js
│   └── util
│       ├── getRootContentMdFiles.js
│       ├── readSingleMdFile.js
│       └── tokenFile.js
└── package.json

/src/content/advanced/contact.md

As with index.md in the Gatsby WYSIWYG project, YAML-formatted front matter only, no “main matter.”

---
template: AdvancedPage
sections:
  - type: SectionPink
    say: I did it!
  - type: SectionTaskList
    accomplishments:
      - task: eat
        done: true
        how: well
      - task: sleep
        done: false
        how: soundly
      - task: jump
        done: true
        how: high
      - task: write
        done: true
      - task: hydrate
        done: true
        how: regularly
  - type: SectionBlue
    mention: Hello World
---

/src/content/basic/about.md and /src/content/basic/index.md

As with index.md in the Gatsby Markdown hello-world project, YAML-formatted front matter only, no “main matter.”

---
template: BasicPage
message: About me
---

/package.json

Now with NPM packages glob (for filtering files that exist in a folder), gray-matter (for parsing the “front matter” out of Markdown-formatted files), and I threw in remark and remark-html (for parsing Markdown-formatted data) for good measure.

  • Q: How did I pick these new packages?
  • A: I found them in sample code that smart people wrote, and they worked.

In this particular theme, I don’t do any transformation of Markdown-formatted data into HTML, so remark and remark-html aren’t strictly necessary. However, since I’m storing data in the front matter of files named with .md extensions, it seemed reasonable to assume that one day I’d extend the theme to handle Markdown-formatted data.

{
  "name": "netlify-nextjs-test-02",
  "version": "0.0.2",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "develop": "next dev",
    "start": "next start",
    "build": "next build"
  },
  "dependencies": {
    "glob": "latest",
    "gray-matter": "latest",
    "next": "latest",
    "react": "latest",
    "react-dom": "latest",
    "remark": "latest",
    "remark-html": "latest"
  }
}

/src/templates/advancedPage.module.css and /src/templates/advancedPage.js

.testingcss {
  border-style: dashed;
  border-color: pink;
  border-width: 20px;
}

Note that for “advanced pages,” the primary area of content below the page’s title will be surrounded by a pink dashed border.

import React from "react";
import sectionComponentTypeList from "../components/indexSectionComponents.js";
import styles from './advancedPage.module.css'

export default function AdvancedPage(props) {
  const sections = props.meta.sections;
  const SectionComponents = sections.map((section) => {
    let sectionType = section.type;
    let Component = sectionComponentTypeList[sectionType];
    return <Component section={section} />;
  });
  return (
    <>
      <h1>An advanced page called {props.hello}</h1>
      <div className={styles.testingcss}>{SectionComponents}</div>
    </>
  );
}

To better understand what’s going on in this template’s use of SectionComponents, see my breakdown in the Gatsby WYSIWYG theme.

/src/templates/basicPage.module.css and /src/templates/basicPage.js

.testingcss {
  border-style: dashed;
  border-color: green;
  border-width: 20px;
}

Note that for “basic pages,” the primary area of content below the page’s title will be surrounded by a green dashed border.

import React from "react";
import styles from "./basicPage.module.css";

export default function BasicPage(props) {
  return (
    <>
      <h1>A basic page called {props.hello}</h1>
      <div className={styles.testingcss}>I am basic, but I can tell you:  {props.meta.message}</div>
    </>
  );
}

This page template is effectively variation #1 of xyzzy.js in the Gatsby Markdown minimum viable build theme.

/src/templates/indexMdRootContentTemplates.js

The only goal in life of indexMdRootContentTemplates.js is to make it easier for [[...rooturlslug]].js to “look up” an appropriate React template from among AdvancedPage and BasicPage based on the data it finds in a Markdown-formatted file’s front matter.

https://katiekodes.com/gatsby-multi-level/#srccomponentsindexsectioncomponentsjs

import AdvancedPage from "./advancedPage.js";
import BasicPage from "./basicPage.js";

export default {
  AdvancedPage,
  BasicPage,
};

/src/templates/layoutHello.js

import React from "react"
import HeaderComponent from "../components/header"

export default function LayoutHello({ children }) {
  return (
    <div className="hello-layout-wrapper">
      <HeaderComponent />
      <main className="hello-layout-main">{ children }</main>
	  <footer className="hello-layout-footer">Footer placeholder</footer>
	</div>
  )
}

/src/components/header.js and other components

import React, { Component } from "react";
import Link from 'next/link';

class HeaderComponent extends Component {
  render() {
    return (
      <header className="hello-layout-header">
        <nav>
          <Link href="/"><a>Home</a></Link>
          <Link href="/about/"><a>About</a></Link>
          <Link href="/contact/"><a>Contact</a></Link>
        </nav>
      </header>
    );
  }
}
export default HeaderComponent;

For all other files under /src/components/, see the source code on GitHub and my breakdown in the Gatsby WYSIWYG project.

/src/pages/[[...rooturlslug]].js

import React from "react";
import { readTokenFromFile, saveTokensToFile } from "../util/tokenFile";
import { getMdRootStaticPaths } from "../util/getRootContentMdFiles";
import { getDocByFilePath } from "../util/readSingleMdFile";
import mdRootTemplateTypeList from "../templates/indexMdRootContentTemplates";
import LayoutHello from "../templates/layoutHello";

export default function RootUrlPage(props) {
  let TemplateComponent = mdRootTemplateTypeList[props.meta.template];
  return (
    <LayoutHello>
      <TemplateComponent {...props} />
    </LayoutHello>
  );
}

export async function getStaticProps(context) {
  // Props in the return value is a magic keyword -- I can't rename it to myStaticProps.
  // https://nextjs.org/docs/basic-features/data-fetching#getstaticprops-static-generation
  const { params } = context;
  if (!params) { return; }
  const tokentopass = !params.rooturlslug ? "" : params.rooturlslug;
  const extraparams = await readTokenFromFile(tokentopass);
  if (!extraparams) {
    return { props: { meta: { template: "ERROR" } } };
  }
  const doc = await getDocByFilePath(extraparams.params.originalfilepath);
  return {
    props: {
      hello: params ? JSON.stringify(params) : "world",
      templatetype: extraparams.params.templatetype,
      ...doc,
    },
  };
}

// Credit to Alex Christie at https://www.inadequatefutures.com/garden/04-dynamic-routes-nextjs
const contentSource = ["src/content/advanced", "src/content/basic"];
export async function getStaticPaths() {
  const paths = await getMdRootStaticPaths();
  await saveTokensToFile(paths);
  return {
    paths,
    fallback: false,
  };
}

Routing

There’s one way in which [[...rooturlslug]].js acts like my Gatsby themes’ xyzzy.js: it’s a top-level template that is actually responsible for turning Markdown files into URLs with HTML in them.

Gatsby’s xyzzy.js files didn’t really impact the choice of URLs that Markdown files were transformed into. That was, arguably, more handled by gatsby-config.js, gatsby-node.js, etc.

In Next.js, the very existence of [[...rooturlslug]].js within /src/pages/ has a direct impact on URLs available in the web site I’m generating.

Tip: I think it helps if you avoid thinking of Next.js as an HTML->URLs builder like 11ty, and instead think of it as a URL->fetch some HTML server like Wordpress.

  • Once you do that, it sort of makes sense that /pages/ or /src/pages/ in Next.js is architected to answer the question of a server asking, “wait, what do I do about this URL someone just tried to visit?”
  • Eleventy is far more comfortable allowing you to use the permalink property of data files’ “front matter” to cause a free-for-all of templates writing output to the same URL patterns as each other. But that makes sense, because it’s only doing “one-and-done” writing of files for upload to a CDN.

I can granularly control URL generation logic in a “catch-all route” like [[...rooturlslug]].js by making sure that my code has getStaticPaths() and getStaticProps() functions exported from it.

  • I let getStaticPaths() delegate to /src/util/getRootContentMdFiles.js and /src/util/tokenFile.js.
  • I let getStaticProps() delegate to /src/util/readSingleMdFile.js and /src/util/tokenFile.js.

/src/util/getRootContentMdFiles.js

import path from "path";
import glob from "glob";
import { promises as fs } from "fs";

const contentSource = ["src/content/advanced", "src/content/basic"];
const files = contentSource.reduce((acc, src) => {
  // using process.cwd() based on next.js documentation
  // https://nextjs.org/docs/basic-features/data-fetching#reading-files-use-processcwd
  const contentGlob = `${path.join(process.cwd(), src)}/**/*.md`;
  const files = glob.sync(contentGlob);
  return [...acc, ...files];
}, []);

export async function getMdRootStaticPaths() {
  if (!files.length) return [];
  const paths = await Promise.all(
    files.map(async (filepath) => {
      // get file name and use as slug
      const slug = filepath
        .replace(/^.*[\\\/]/, "")
        .replace(new RegExp(`${path.extname(filepath)}$`), "");
      const slugArray = slug === "index" ? [] : [slug];
      // get parent folder and use as template datapoint
      const myTemplate = path.dirname(filepath).replace(/^.*[\\\/]/, "");
      return {
        params: {
          rooturlslug: slugArray,
          templatetype: myTemplate,
          originalfilepath: filepath,
        },
      };
    })
  );
  return paths;
}

I can’t take credit for this. Alex Christie is the smartypants.

It URL-“slugifies” the filename of every .md file found in /src/content/advanced/ or /src/content/basic/ and returns the results to getStaticPaths() from [[...rooturlslug]].js.

/src/util/readSingleMdFile.js

import matter from "gray-matter";
import { promises as fs } from "fs";

export async function getDocByFilePath(filepath) {
  const fileContents = await fs.readFile(filepath, "utf8");
  const { data, content } = matter(fileContents);
  return { meta: data, content };
}

This, given a .md file’s original filepath, re-opens it and parses it for its front matter. It returns the results to getStaticProps() from [[...rooturlslug]].js.

/src/util/tokenFile.js

import path from "path";
import { promises as fs } from "fs";

const TokensFilePath = path.join(process.cwd(), ".ignoreme/tokens.json");

export async function readTokenFromFile(tokenId) {
  let tokensFile = await fs.readFile(TokensFilePath);
  let tokenObj = JSON.parse(tokensFile.toString());
  return tokenObj[tokenId];
}

export async function saveTokensToFile(tokens) {
  try {
    await fs.truncate(TokensFilePath);
  } catch {}
  let tokenObj = {};
  for (let token of tokens) {
    if (!token || !token.params) {
      return;
    }
    let tokenidtowrite = !!token.params.rooturlslug
      ? token.params.rooturlslug
      : "";
    tokenObj[tokenidtowrite] = token;
  }
  return fs.writeFile(TokensFilePath, JSON.stringify(tokenObj));
}

The purpose of this code, tweaked from Yonatan Bendahan’s example, is to make sure that the original file path makes it through from getStaticPages() to getStaticProps() (by writing the data to a temporary file), since Next.js natively refuses to let me pass anything but the part of the filename that’s destined for the outpage page’s URL between the two.


Routing – a deeper dive into getStaticPages()

Baseline

Let’s go back and look at /src/pages/[[...rooturlslug]].js in my Next.js minimum viable build:

import React from "react"

export default function Home() {
  return <div>Hello world!</div>
}

Minimum viable dynamic routing

Now, take a look at an intermediate version of it I made, before I built the final version you see above:

import React from "react"

export default function RootUrlPage(props) {
  return <div>Hello world!  If there are any props in this page, they are:  {props.hello}</div>
}

export async function getStaticProps(context) {
  // Props in the return value is a magic keyword -- I can't rename it to myStaticProps.
  // https://nextjs.org/docs/basic-features/data-fetching#getstaticprops-static-generation
  const { params } = context;
  return {
    props: { hello: !!params ? JSON.stringify(params) : "world" },
  };
}

export async function getStaticPaths() {
  // Credit to Ashutosh at https://dev.to/akuks/what-is-getstaticpaths-in-nextjs-5ee5
  return {
    paths: [
      { params: { rooturlslug: [] } },
      { params: { rooturlslug: ["1"] } },
      { params: { rooturlslug: ["2"] } },
      { params: { rooturlslug: ["3"] } },
    ],
    fallback: false,
  };
}

Whereas my “minimum viable build” generates a single /.next/server/pages/index.html file with Hello World! in it, this “intermediate” version of /src/pages/[[...rooturlslug]].js generates four HTML pages: 1.html, 2.html, 3.html, and index.html, with content like Hello world! If there are any props in this page, they are: <!-- -->{&quot;rooturlslug&quot;:[&quot;3&quot;]} or Hello world! If there are any props in this page, they are: <!-- -->{} in the case of the home page.

  1. Hopefully you can see that the most important thing to do to make a “catch-all route” template generate multiple output files instead of a single output file is to add a getStaticPaths() to it that returns an object with a paths property containing an array of objects…
  2. Each of which have a params property…
  3. Those values, in turn, should be objects with a key named after whatever is between the [[... and ]].js of your “catch-all route” template filename. (If you used a really complex filename with multiple single or square bracket pairs in its filename, you might have to give it more than one key.) The value of that key – for me, rooturlslug – should be an array of parts that will concatenate with slashes into a URL path if you’re using double brackets (catch-all routing), and a string if you’re using single brackets (ordinary dynamic routing).

Obviously, this is a comically simple getStaticPaths() function. A real one would be far more complex. For example, it could comb through a bunch of Markdown files like we did above. Or it could fetch data from a headless API-based CMS.

I deployed this to Netlify and confirmed that I could visit example.com/1/, example.com/2, etc.

So I guess maybe I should’ve broken out “Next.js minimum viable dynamic routing” into is own post! Oh well.

Failure: redundant files

Next.js refused to let me create a second file /src/pages/[[...zzdifferentslug]].js with the same codebase but without the first “slug” of [], with [1]-[3] changed to [4]-[4], and with the name of the export changed RootUrlPageTwo.

Error: You cannot use different slug names for the same dynamic path ('rooturlslug' !== 'zzdifferentslug').

Remember – this isn’t Eleventy. You can’t make different templates at the same template-file-and-folder-system hierarchy fight each other for URL routing.

Again, I think it helps if you avoid thinking of Next.js as an HTML->URLs builder like 11ty, and instead think of it as a URL->fetch some HTML server like Wordpress.

Success: hierarchichal routing competitions without overlap

Next.js is not, however, mad at me if I move this new file to /src/pages/another/[[...zzdifferentslug]].js

Next.js isn’t even mad at me if I do { params: { rooturlslug: ["another", "2"] } }, in the original /src/pages/[[...zzdifferentslug]].js file. It just generates /.next/server/pages/another/2.html alongside /.next/server/pages/another/4.html, /.next/server/pages/another/5.html, and /.next/server/pages/another/6.html as output.

This might be a nifty trick for situations where I might want to simply use an advancedPage “site builder” template to render list-link pages like /blog/ or /events/ that have “child URLs” for individual items.

I could leave it up to the owner of the website to remember that it’s their common-sense duty to drag-and-drop a “blog posts list” component if they’ve decided to create an “advanced page” called /blog/ with my Wix-style UI. But Next.js could use a more “fixed” blog-article-specific template to render the standalone blog post pages.

Failure: hierarchichal routing competitions with overlap

Next.js will, however, error out after “collecting page data” if I try to do that trick to sneak a second example.com/another/4/ in from the original /src/pages/[[...zzdifferentslug]].js file like { params: { rooturlslug: ["another", "4"] } }, which I figured it would:

error - Conflicting paths returned from getStaticPaths, paths must unique per page.
--- ---