Zach’s ugly mug (his face)

Zach Leatherman

Which Generator builds Markdown the fastest?

July 29, 2022 #1 Popular

Blogging. The quintessential starter project for most—if not all—site generators. Here’s a few samples:

  • Next.js guides new developers to Learn Next.js “by creating a very simple blog app.”
  • Gatsby’s tutorial guides folks to create their “first Gatsby site: a blog site…”
  • Remix’s home page has a prominent Get Started call to action button that links to their Blog Tutorial.
Some of these adamantly SPA-first *web app* frameworks have really focused their documentation on content-based *web site* use cases—but let’s not read into that too much…

Given that Markdown is a very popular document format for blogging, this gives us the opportunity to compare the performance of different site generators for this pervasive use case (highlighted on each generator’s documentation, as noted above).

Let’s see how each generator stacks up when consuming markdown files from the local project’s file system and building into a production-ready project.

(Order is alphabetical. Disclosure: I am the maintainer of Eleventy)

Benchmark Results

Times shown are in seconds. Lower is better.

Markdown Files: 250 500 1000 2000 4000
Astro 1.0.1 2.270 3.172 5.098 9.791 22.907
Eleventy 1.0.1 0.584 0.683 0.914 1.250 1.938
Gatsby 4.19.0-cli 14.462 15.722 17.967 22.356 29.059
Hugo v0.101.0 0.071 0.110 0.171 0.352 0.684
Next.js 12.2.3 (JS routing) 6.552 6.932 8.034 9.582 13.409
Next.js 12.2.3 (File routing) 7.958 9.551 14.304 25.038 70.653
Remix 1.6.5 (File routing) 2.876 8.258 46.918 349.125 1800
The last Remix (File routing) test was force-quit at 1800 seconds (30 minutes)—it had not completed.
Updated August 10, 2022: Astro results were updated to 1.0.1.
Markdown Files: 250 500 1000 2000 4000
Remix 1.6.5 (JS routing) 0.868 0.834 0.891 0.887 0.901
Updated August 3, 2022: Perhaps controversially, the Remix (JS routing) method is excluded from the chart because it does not perform any processing of markdown. One could argue that this is the fastest way but also simultaneously irrelevant to this benchmark! Make sure you read the Remix addendum below.

Each run was repeated 3 times and the lowest/fastest time was selected. This result set was generated on a MacBook Air (M1, 2020), macOS Monterey 12.5, 16 GB memory.

All of the code for this benchmark is fully open source and welcomes review.

Test Notes

For each generator sample I attempted to create a reduced project with the sole use case of processing markdown files. I opt-ed out of TypeScript when options were presented in various cli tools to do so. Output folders and framework specific cache folders were deleted before each run.

  • For Astro I used the Blog example which uses a pre-release of Astro 1.0. Updated August 10, 2022 Results were updated to Astro 1.0.1 after stable release!
  • For Eleventy I used eleventy-base-blog, even though it has a few extra plugins and templates in play on top of the barebones core experience.
  • For Gatsby, I used npm init gatsby with Markdown support (not MDX).
  • For Hugo, I went through the Quickstart (skipping the theme)
  • For Next.js I deleted a bunch of things out of the blog-starter example linked from the docs on Static Generation.
  • For Remix I went through the Blog tutorial but did not use a database. (Read the Remix addendum below).

I put out a Twitter poll to gauge how folks felt about project sizes. 1000 files was considered a Large project by 58.8% of voters, Medium for 36.8% of voters. Markdown samples borrowed from Sean C Davis’s SSG Build Performance Comparison repository. I have not yet added tests for Jekyll or Nuxt but I’m open to it!

Summary

Updated August 3, 2022 with notes about file-based routing of markdown in Remix and Next.js, as well as Astro MDX.

  1. Hugo remains the undisputed speed champ—no question about that.
  2. Eleventy was the fastest JavaScript-based generator.
  3. Generators that create per-page JavaScript bundles (for single page apps, primarily) are usually slower to build, unsurprisingly. Heavier pages aren’t exclusively slower for end users—they’re slower for developers too.
  4. File-based routing of markdown files in both Next.js and Remix is a bit slow! Take care to add the additional boilerplate routing code needed to load markdown files more efficiently.
  5. Astro was on-par with Next.js at mid-range (1k) and on-par with Gatsby at the upper-range (4k) of this benchmark.
    • Jon Neal submitted a PR to do Astro markdown processing via the MDX plugin. I re-ran the tests but the data for Astro didn’t change in a statistically significant way (yet?). Likely more to come there, this feature is brand new!
    • Updated August 10, 2022 Astro results were updated to use newly released stable version 1.0.1. This resulted in a ~25% improvement for Astro across the board 🏆
  6. I would welcome a code review on the Remix site—it scaled so poorly that I suspect that I may have misconfigured something? I would be happy to update with corrections.
    • As noted in point 4 above, it is recommended for Remix projects by the Remix team to not use file-system based routing for .mdx or .md files. (Read the Remix addendum below).

Bonus: npm install Benchmarks

Times shown are in seconds. Lower is better.

Show table of results
Framework npm install Time
Astro 1.0.1 13.214
Eleventy 1.0.1 15.168
Eleventy 2.0.0-canary.14 7.195
Gatsby 4.19.0-cli 68.516
Next.js 12.2.3 15.589
Remix 1.6.5 28.619

Each run was repeated 5 times and the lowest/fastest time was selected. npm cache clean --force was run before each to ensure a cold install. This result set was generated on a MacBook Air (M1, 2020), macOS Monterey 12.5, 16 GB memory.

Remix Addendum

Updated August 3, 2022 (this entire section is new)

I understand why Remix folks think this benchmark is unfair to Remix. I’ve heard lots of feedback and some of it has been decidedly unfriendly—but I do get it. I don’t really have any desire to wade into some of the larger debates about Remix or Jamstack on this post.

From my perspective, the benchmark encountered a build performance-related bug in how Remix—a request-time architected tool—performs a build-time precompilation of Markdown routes. It doesn’t have to be any more than a build-performance related bug. Once the bug is fixed (or if the approach is deprecated), I’ll make updates to this post.

For completeness and a bit more technical detail, the Remix team has asked me to highlight that it is not recommended to use MDX file-system based routing for .mdx or .md files. Specifically: do not put .md or .mdx files into your app/routes/ folder. Make sure you read/heed the warnings on the Remix MDX plugin documentation.

I maintain that markdown file-based routing is a useful feature and it seems likely that the Remix folks will be able to (at some point in the future) allow a peaceful coexistence of markdown routes without a heavy build-time compilation step. The demand clearly exists as many other frameworks do support it!

But until then the Remix team recommends to opt out of file-based routing for Markdown to bypass the costly build-time compilation step. You can use a scripted route file to load Markdown files from the file system in a different source directory (not app/routes/). You can see an implementation example from @ebey_jacob in the bench-framework-markdown repository.

One last note about Build Performance

“While you were waiting for your static site to build, distributed web infra­structure got really good. Break through the static.”—https://remix.run/

The only other thing I’d like to say here is that whether a tool spits out bundled JavaScript for runtime server deployment on the Edge or HTML in a static build for the CDN, whether it’s categorized as a Static Site Generator or a Site Generator or none of the above: all of the tools tested in this benchmark are performing a build step to generate code for deployment and can encounter build-related performance issues.

All that to say: I think we can agree that build performance is not a concern exclusive to static site generators! There will always be a healthy tension between request-time and build-time and the tools that make use of both to complement each other will likely end up being the most useful to developers.

Keep measuring and keep building—appreciate y’all!

Zach’s ugly mug (his face)

Zach is a builder for the web with IndieWeb Avatar for https://www.netlify.comNetlify. He created the IndieWeb Avatar for https://www.11ty.devEleventy site generator and is still fixated on web fonts. His public speaking résumé includes talks in eight different countries at events like Jamstack Conf, Beyond Tellerrand, Smashing Conference, CSSConf, and The White House. He is an emeritus of Filament Group, NEJS CONF, and still helps out with nebraskajs’s AvatarNebraskaJS. Read more about Zach »

Previous
Ryan Carniato: Exploring 11ty with Zach Leatherman
Next
Dear Paul

34 Retweets

Eleventy 🎈 2.0.0-canary.14Khalid 🫧Bryce WrayAndrew KepsonJBEstela Franco (she/her)Justin HachemeisterPhillNicolas HoizeyEmma needs ☕️priya josephMatt BiilmannHarits SyahDavid EastDave 🧱Zack HoherchakIndieWeb Avatar for https://flmarket.ruFrontend DogmaIndieWeb Avatar for https://www.alvinashcraft.comDave GlickTyler Sticka, ☁️4IndieWeb Avatar for https://thedevnews.commiyaokaBraxMaciek Palmowski 🌴IndieWeb Avatar for https://dev.toNate MoorefredMiguel 💬Chris Short 🇺🇸🇺🇦IndieWeb Avatar for https://prodsens.livePablo Lara HKOO서현빈 | Hyunbin

107 Likes

TSGandalfMatt Rossman 🍌Svante BengtsonMarc Littlemore 👋🏻bLaCkwEwhigbyикар кисилифMatt BiilmannCharlie GOwen Buckley ♻️Elliott MarquezRyan MulliganRodneyDaniel CeciliumMayankkitajchukEduardo UribeBryan RobinsonJason GormanHugo NogueiraLive on Twitch! 🔴 – Nick TaylorMike ApariciofredJosef Polodna⚡️ Salma | whitep4nth3rElian ☕ErikaBrian RinaldiMiguel 💬𝕕𝔾𝕣𝕒𝕞𝕞𝕒𝕥𝕚𝕜𝕠Amelia Bellamy-RoydsChris Bongers 🤘Jabran RafiqueMehdi AchourDavid EastJace Benson 👨‍💻⚙️Okiki OjoCody SwannNate MooreTobias FedderJesse HeadyOwen Buckley ♻️Alistair ShepherdBrian RinaldiMehdi AchourRhian van EschEd StephinsonThomas Michael SemmlerBrian SinclairthespartaNikitaTyler Sticka, ☁️4Barthelemy RochatAnilSean van ZuidamAndrea LeardiniSpinalMr AndersonMarc FriederichMeet DaveZack UrlockerMatthias WestonModern Frontends Live! November 17-18, 2022 LondonZack HoherchakGiancarlo SoveriniOgunChris LaRocque𝕕𝔾𝕣𝕒𝕞𝕞𝕒𝕥𝕚𝕜𝕠David EastSia KaramalegosJabran RafiqueBradley BurgessTorben HaackMarc Filleul 🇫🇷Joshua YoesIain SimmonsMatthias OttMartin BerglundFynn BeckerMatt Biilmannkatharina udemaduDavid MortonhigbyJohn MeyerhoferJesse HeadyTristanMayankBrian RinaldiJonathan Neal🐬 I'M YourOnly.One ❄️ 🔑Kyle Mitofsky #BLM🌸🌕⭐🌕🌸Héctor AguilarColin FahrionKhalid 🫧Andrew KepsonJoan León 🏞⚡️Justin HachemeisterPeter BinkowskiEleventy 🎈 2.0.0-canary.14JBAlistair ShepherdPhillBryce Wray✧ Ajit ✦Tom VanAntwerpEric Wallace
41 Comments
  1. Khalid 🫧

    @buhakmeh

    The first time I used Hugo, I thought it was broken. I couldn't believe it was that fast.

  2. Tom VanAntwerp

    @tvanantwerp

    I think SSG goes against what Remix is trying to be. That said, still, wow...

  3. JB

    @SCRWD

    I would be interested to see where the ever-popular Jekyll sits in this test. I'm just itching for more reasons to convince management to let me switch everything to Eleventy haha. I still need some method for sharing and updating layouts and includes across multiple sites.

  4. JB

    @SCRWD

    Also, can you think of a way to compare the speed of templating between generators?

  5. Zach Leatherman

    @zachleat

    yeah, for sure—specifically this wasn’t attempting to use SSG, just sourcing markdown from the file system for a remix production build

  6. Zach Leatherman

    @zachleat

    so fast!

  7. Zach Leatherman

    @zachleat

    ah right on! maybe I can have a look at this on Monday 🙌🏻

  8. Zach Leatherman

    @zachleat

    The inputs vary a bit more across frameworks there—hmm

  9. Zach Leatherman

    @zachleat

    subscribe to this one github.com/zachleat/bench…

  10. Bryce Wray

    @BryceWrayTX

    These numbers could be a big part of the reason why Astro essentially is dropping (or at least discouraging) pure Markdown in favor of MDX — which, at least in my own testing in dev mode, it handles much more quickly. 🤔

  11. Zach Leatherman

    @zachleat

    oh, maybe! @jon_neal was so quick to PR that and left some comments there: github.com/zachleat/bench…

  12. Bryce Wray

    @BryceWrayTX

    Interesting! I should note that my tests involved only about 350 MDX files, albeit most of them very short (just quickies for the test), so his findings are probably much more meaningful than mine.

  13. Nicolas Hoizey

    @nhoizey

    I wonder if trends would be the same for 10k or 100k Markdown files… 😅

  14. Zach Leatherman

    @zachleat

    that might be my next test, then maybe on different hosting providers too 😅

  15. Nicolas Hoizey

    @nhoizey

    Hosting providers are interesting indeed! 👍

  16. Mayank

    @m_yxnk

    the other day i saw someone use hugo and i thought "why?". but now i'm ok with it after seeing the performance of remix (a "modern" metaframework) 😅

  17. Zach Leatherman

    @zachleat

    Hugo is good!

  18. Mayank

    @m_yxnk

    but 11ty is just better!

  19. Zach Leatherman

    @zachleat

    I MeaN, I aGREE bUT I Am BiASEd 😇

  20. Barry Pollard

    @tunetheweb

    FYI the charts don’t work properly on mobile (iOS 15.5 using Safari in case it matters)

  21. Bradley Burgess

    @bburgess_keys

    2 things: 1. Woah, look at that Eleventy 2.0.0 install time!! 2. Lol Gatsby 😂

  22. And this one, because of the interesting generators comparison (even *if* @ZachLeat or I, as an @eleven_ty user, were biased—I think!): Which Generator Builds Markdown the Fastest?, by @zachleat, featured on @frontenddogma: zachleat.com/web/build-benc…

  23. 不郑

    @jianjian086152

    Remix: Thank you.

  24. Amelia Bellamy-Royds

    @AmeliasBrain

    Enjoying how you handled the y-axis in the first figure, there. 😝

  25. Zach Leatherman

    @zachleat

    the noise I made when I learned that SVG could overflow: visible 😂

  26. Amelia Bellamy-Royds

    @AmeliasBrain

    Next step in benchmarking would be the "add one new file & regenerate", I guess. Which of the generators have smarts to detect which content has changed, vs re-writing everything?

  27. Paul Applegate

    @papplegate

    I don't know much, but I know I want the one that shoots up the whole page 😂

  28. Zach Leatherman

    @zachleat

    are you sure 😅

  29. Paul Applegate

    @papplegate

    nOpe. Two tHinGS I AM SuRE OF, dEAth &Amp; payINg My tAxEs .😅

  30. Zach Leatherman

    @zachleat

    yep it’s open source, linked up here I welcome the competition and will update it if a framework requests it!

  31. Swamp Kitten Michael

    @_furstenberg

    And it’s pretty fast for Markdown as well. zachleat.com/web/build-benc… I currently use Hugo, but have created a version in Eleventy earlier.

  32. Zach Leatherman

    @zachleat

    Pushed a bunch of additions to this post after community feedback! 1️⃣ Added data for Next.js (file based routing) 2️⃣ A note about Astro and MDX (probably more to come there!) 3️⃣ A whole bunch of words on Remix! Deep link: zachleat.com/web/build-benc…

  33. Bryan Robinson

    @brob

    Who's pink here?

  34. Zach Leatherman

    @zachleat

    whew, I really don’t care to reopen this can-o-worms but you can look at the data table below the chart if it’s unclear (a bunch more conversation at )

  35. Bryan Robinson

    @brob

    Mostly just love that that line continues 😂

  36. 🍩 Pablopang.svelte

    @PaoloRicciuti

    Someone should definitely copy @remix_run code so that we can have more vertical lines.

  37. Emilia Zapata

    @synecdokey

    That probably came too late to change the initial perception. It's also arguably its Achille heel still zachleat.com/web/build-benc…

  38. Nicolas Hoizey

    @nhoizey

    Ça mérite un ajout dans le benchmark de @zachleat ! zachleat.com/web/build-benc…

  39. Magentix

    @_magentix

    J'ai répondu trop vite sans lire ^^

  40. Arnaud Ligny 👨‍💻 💡 🚀

    @ArnaudLigny

    Excellente idée, ce sera plus qualifiant que faire des tests dans mon coin. Merci !

  41. Nicolas Hoizey

    @nhoizey

    🙏

Shamelessly plug your related post:

Sharing on social media?

This is what will show up when you share this post on Social Media:

How did you do this? I automated my Open Graph images. (Peer behind the curtain at the test page)