Zach’s ugly mug (his face) Zach Leatherman

Netlify’s Disingenuous Survey-based Attack on Next.js (and Eleventy, too)

December 19, 2023 #7 Popular

As Netlify distances itself from Jamstack in an attempt to survive a VC-funded battle with Vercel, they have also rebranded the Jamstack Community Survey to a more generic The State of Web Development report (behind an email signup form). It should be called the Netlify Community Survey (as the results only represent Netlify customers) but I digress.

A few disclaimers before we get started:

  1. I am the creator and maintainer of Eleventy.
  2. From 2020–2023 I was employed by Netlify.
  3. In 2021 I worked on the Jamstack Community Survey results website (but was not involved in the survey data collection nor the results content).

Problems

This year’s report has a few changes to the presentation of results that I think have traversed the report into territory that I called “intellectually dishonest” on Mastodon. My default in these situations is to let these things go but enough people have tagged me on this—while also drawing some damaging conclusions from the disingenuity in this report—that I feel the need to weigh in (though frankly I’m a little annoyed that I need to spend time on it at all).

My primary beef with this year’s report is on page 10 (of 13) and features a section on rendering frameworks and site generators, with one and only one chart documenting changes in satisfaction score versus change in usage.

This chart shows Astro’s stellar growth this year, the runaway winner for increased satisfaction and usage. Next and 11ty are shown in the bottom left corner, losing usage and satisfaction

This chart looks bad for Next. It also looks bad for Eleventy. But the bigger problem here is that this chart shows changes without any context of absolute scores. Next might have gotten some caveats in the text written in the section but 11ty did not.

In previous years, the report discussed absolute scores prior to deltas because this context is important.

(I also took special note that the label for Gatsby—a framework purchased by Netlify and under very recent criticism for languishing investment—was obscured behind Gridsome)

On page 32 (of 39) of the report buried deep into the Appendices section the report finally includes the real scores (conveniently unordered):

Astro has the best positive sentiment too at 87%, Next is 78% and Eleventy is 74%

Now we learn that Next.js is ranked #3 (tied with SolidStart and Remix) for satisfaction and 11ty is ranked #6 (tied with Nuxt).

Here’s what the chart looks like using absolute values for satisfaction percentages and usage, painting a story that Netlify probably doesn’t want to share:

Next.js is the clear winner here. A cluster of Nuxt, Astro, 11ty, and Sveltekit all sit together with high scores and similar usage. Gatsby is an outlier with large usage but low satisfaction.

Without ascribing malice

I am aware that Netlify has historically employed a Data Scientist with statistics and data analysis background to conduct this survey and prepare the report. This person departed the company months ago. The Data Scientist in question was very good at documenting and publishing the survey and report’s methodology and you can see one such (2022) document.

The 2023 report did not include a methodology document.

The 2022 methodology document reported how the data was collected (sources) and margins of error and included the following notable section at the end:

Conclusions we cannot draw:

We do not believe these results are indicative of preferences amongst web developers as a whole. We are also not making claims as to:

  • Popularity (or not) of Netlify products and services, given bias in respondents

The 2023 report name change to The State of Web Development seemingly goes against the first point.

The 2023 report also goes against the second point by eschewing any mention of bias. It (surprisingly) includes a section on the popularity of Netlify itself:

88% of Netlify customers want to use Netlify more. 79% of Netlify customers want to use Vercel more.

I was taken-aback that Netlify would publish statistics to show what percentage of Netlify customers want to use Vercel more—but apparently it’s 79%.

Conclusions

Importantly: After Eleventy’s Netlify-induced setbacks this year, we are back on our feet and we will continue to punch far above our node_modules weight-class. Many of the alternative frameworks discussed here are VC-funded with full-time developers and have millions in funding. We don’t.

Eleventy’s project goals are different: stay focused, keep our core feature set small, and ruthlessly minimize our external dependencies for a long-term future.

On Astro (since folks have asked), I will say that Astro and Eleventy share the same zero JavaScript footprint vision for the web and in that regard we are allies in the web development framework melee.

Reading the report, it’s clear that Netlify has a vested interest in elevating Astro because Astro is best poised to dethrone Next.js. If Astro wins, Netlify wins. Perhaps surprisingly I also believe that if Astro wins, Eleventy wins! It is not a zero-sum game and I know that 2022 Netlify believed this (even if 2023 Netlify does not).

Appreciate y’all—and keep building for the web!


IndieWeb Avatar for https://unsplash.com/Poster image by Volodymyr Hryshchenko

< Newer
Building a multi-language Taylor Swift fan site (10 Minute Version) (Zach's Version)
Older >
Eleventy v3 with ESM support now on the canary channel

Zach Leatherman IndieWeb Avatar for https://zachleat.com/is a builder for the web and the creator/maintainer of IndieWeb Avatar for https://www.11ty.devEleventy (11ty), an award-winning open source site generator. At one point he became entirely too fixated on web fonts. He has given 81 talks in nine different countries at events like Beyond Tellerrand, Smashing Conference, Jamstack Conf, CSSConf, and The White House. Formerly part of CloudCannon, Netlify, Filament Group, NEJS CONF, and NebraskaJS. Learn more about Zach »

40 Reposts

IndieWeb Avatar for https://show-hn.comIndieWeb Avatar for https://banyak.idIndieWeb Avatar for https://okeyua.comIndieWeb Avatar for https://reganthapa.com.npIndieWeb Avatar for https://animehubinsider.comIndieWeb Avatar for https://www.thaparegan.com.npIndieWeb Avatar for https://derp.fooStefan BaumgartnerMartin FennerBaldur BjarnasonGerrit van Aakengcv@hachyderm.ioJean Pierre KolbManuel MatuzovićDylan </closingtags.com>૮༼⚆︿⚆༽つMike NeumegenBen EverardRyan MulliganChristian AlderSimon Cox :SEO:MayankOld Man VandeheyKoos LooijesteijnThord D. HedengrenTomBryce WrayAdrian CochraneDan Ryan :dryan:Alistair ShepherdApple Annie :prami_santa:Holger HellingerRandom GeekwesruvJosh “Yoshi” VickersonSnugugsurprise, tenebrous overlord!Eleventy ???? v2.0.1Ryuno-KiIndieWeb Avatar for https://zerobytes.monster

47 Likes

Luke HansfordMark ReevesBaldur BjarnasonFernando MateusHawk TicehurstManuel MatuzovićStezKevin ColehigbyHeather BuchelnateDaniel Roe :nuxt:Connor BärMike NeumegenTyler StickaRezzedUpThomas Maria HelzleChristian AlderSimon Cox :SEO:Cem K.phaeLennartRob AndersonMayankHasan AliElly Loel ✨????Egor KloosOld Man VandeheyAlex TrostJohn Kemp-CruzTomjurajkScott BenishBryce WrayEric McCormickLynn FisherHolger HellingerKieran McGuireLene Saile'Tired of people's shit', aka EmberwesruvAshur CabreraAaron In IowaMarcelle RusuEleventy ???? v2.0.1Fynn BeckerBoris Mann
21 Comments
  1. Random Geek

    Random Geek

    @zachleat thanks for sharing the insight!This is disappointingly consistent with my recently downgraded expectations for Netlify.

  2. Zach Leatherman

    Zach Leatherman

    @randomgeek Appreciate you reading and sharing ????

  3. Ryuno-Ki

    Ryuno-Ki

    @randomgeek @zachleat When I finally discovered a value-aligned VPS hosting company and moved my way out of Netlify I was disappointed how hard it felt (especially changing DNS records back).Part of why I can't recommend it really.I like the idea of branch previews. But I thi… Truncated

  4. Ryuno-Ki

    Ryuno-Ki

    @zachleat Once I read Storytelling With Data I can't unsee how bad many charts are made ????

  5. Random Geek

    Random Geek

    @zachleat @RyunoKi I am pretty far removed from Netlify's target market — I could probably host my blog from an arduino board lost under the couch, and I pay my rent with backend dev. I can't really judge their services. All I can judge is their community efforts and annu… Truncated

  6. Eric McCormick

    Eric McCormick

    @zachleat @eleventy well said and with just enough nuance.

  7. Zach Leatherman

    Zach Leatherman

    @edm00se @eleventy I consider this GIF extremely high praise, thank you!

  8. Eric McCormick

    Eric McCormick

    @zachleat @eleventy I see you’re a man of good taste. Cheers.

  9. Hidde

    Hidde

    @zachleat appreciate you speaking out, mustn't have been easy. Thanks for sharing your independent take <3

  10. Zach Leatherman

    Zach Leatherman

    @hdv appreciate you Hidde!

  11. Hidde

    Hidde

    @matuzo @zachleat oh shit manuel didn't realise you would see this ????

  12. Hidde

    Hidde

    @matuzo @zachleat (or I would have cc't you muwhaha)

  13. Zach Leatherman

    Zach Leatherman

    @matuzo @hdv for the record I had a beer with Hidde in 2023 ????you are both welcome in Omaha, Nebraska any time

  14. akahn

    akahn

    @zachleat thanks for calling it like you see it. small language note: I think you meant "milieu" when you said "melee"

  15. Zach Leatherman

    Zach Leatherman

    @akahn Appreciate it! I intended to say melee https://www.merriam-webster.com/dictionary/melee but maybe I’m missing some diacritics Definition of MELEE

  16. Zach Leatherman

    Zach Leatherman

    @biilmann appreciate the response!I hope it’s understood that my issue is not with the data, it’s in how the data was presented—the solo chart was misleading.

  17. Wolfr

    Wolfr

    @zachleat good for calling this out and hope you don't get into trouble.

  18. akahn

    akahn

    @zachleat I understand what you mean now ????

  19. Thord D. Hedengren

    Thord D. Hedengren

    @zachleat Glad you're not pulling any punches. There's something rotten going on here, not just with Netlify (who're probably scrambling) but with all these hyped-up frameworks. Stay the course.

  20. Tim Severien - open to work

    Tim Severien - open to work

    @zachleat I’m not angry, I’m disappointed

  21. Tyler Sticka

    Tyler Sticka

    @zachleat Love the positive relationship you have with “competing” tools. (I know that’s not the main point of this post, it just stood out to me while reading)

Shamelessly plug your related post

These are webmentions via the IndieWeb and webmention.io.

Sharing on social media?

This is what will show up when you share this post on Social Media:

How did you do this? I automated my Open Graph images. (Peer behind the curtain at the test page)