Follow along with case studies and examples showing how I build programmatic niche sites.
This page contains all of the details from my thread on programmatic SEO with a few additions, case studies, and examples of how you can get started with programmatic SEO for your own niche sites.
What is programmatic SEO?
My little network of sites generate ~2m pageviews a month, and 99% of that traffic is from programmatic SEO.
Put simply, it's a way to generate a lot of pages that target different keywords using code.
These keywords tend to be less competitive long-tail terms, and because of the volume of pages generated, it amounts to quite a lot of traffic.
I caught some shade on the thread because a lot of people think programmatic SEO == spam, but I always try to surface data in a way that's helpful to people searching for a specific term.
How programmatic SEO can benefit you
When it comes to niche sites, content production is a costly and ongoing investment.
Creating pages programmatically is a quicker way to scale your site to a lot of pages targeted at niche longtail keywords.
I can write one post of ~1k words and insert all of the variables into it, and I have 1000s of articles that exactly answer the specific query a user is asking for.
It can also complement existing sites by adding a new section, and may even become a decent link magnet if you're able to surface data and information in an interesting way.
I never build links but add a link/citation tool after all pages and attract them naturally by providing data.
An example
One example of a programmatic SEO site that is very simple to build, and is generating 920k visitors a month according to Ahrefs, is PercentageCal.com.
It answers simple questions like:
- 5 is what percent of 100?
- what is the increase/decrease in percentage from 5 to 10?
- what is 5% of 10?
The numbers are pulled from the URL, for example:
/answer/5-is-what-percent-of-100
/answer/what-is-percentage-difference-from-5-to-10
/answer/what-is-5-percent-of-10
In PHP, the answers to those questions can be calculated using very simple formulas:
If you have a site which can extract the values from the URLs above, it's very straightforward to then calculate the answer and output it on the page.
Finding a data source
Percentages are a bad example. Anyone can do it, there's not a lot of content or value to add beyond the answer, and Google shows the answer in the search results, so you don't get a lot of clicks.
So where do you find data for programmatic sites?
There are lots of places to find datasets, APIs or ideas for a programmatic site. Here are a few:
Once you find a data source, you can start building your site. I like to start with a CSV file with all of the data, and anything I want to use as part of a URL, I'll add an extra column to the CSV for a slugified version.
So if my site is about car manufacturers, and the make of the car is an Aston Martin, I'll make sure I have a slugified version of the name, like aston-martin.
I use Python to create slugs, but you can just as easily do it with an online tool like this.
Creating the database
Again, I use Python for this but if you have a CSV that is ready to go with your data source, you can create a database from it.
I tend to use TablePlus as my tool of choice and upload the CSV, creating a new table, and setting the right data types for the columns.
Don't forget to add indexes (or ask your dev to) any columns that you'll need to look up data by (slugs, years, etc) as that will speed up the queries.
Building the backend of the site
I've done this a bunch of times now, so the way I build the sites has evolved from something very janky to something only slightly less janky.
The basics of my framework are:
- All traffic routes to an index.php file
- Check the request URI and route to a controller (or 404 if it doesn't exist)
- My controller grabs the data from the database
- The controller then passes the data to a template to render the frontend
If I was a better dev I could maybe use Laravel, but I found it way too much for my needs. The best I've seen for a half-coder like me would be Slim or Fat-Free Framework but if it aint broke, don't fix it!
What I love about building these sites is there are so many long-tail phrases you can target from the same data source.
So let's say I'm building a site about the fuel efficiency of cars. If I have a database which has the year of manufacture, the make, the model, and the average miles per gallon, I can target all kinds of phrases like:
- {{make}} fuel efficiency
- {{make}} {{model}} fuel efficiency
- {{year}} {{make}} {{model}} fuel efficiency
- Most fuel efficent cars of all time
- Most fuel efficient cars in {{year}}
- Most fuel efficient {{make}} car
- Most fuel efficient {{make}} car in {{year}}
At this point you could have thousands of URLs on your site, and to add more pages, all you need to do is add new rows to your data source.
Building the frontend
Each new project I start, I just copy/paste the most recently built one and start from scratch.
I use Bulma CSS, plain JS, and Twig templates to pass the data to. I do try and change the design and colours ever so slightly on each site, but it's 95% the same.
Twig is a template engine that lets me add the variables into the content like this:
Looking for information on the fuel efficiency of the {{year}} {{make}} {{model}}?
You can also do advanced stuff with it like in the below snippet where I'm creating a table of cars with data about each of them:
I can loop through all of the cars and create a table row for each of them, with links to the car's page.
A few people tweeted me saying that programmatic SEO is spammy, or call these doorway pages. I don't agree!
All I'm trying to do is build a page with content and data on it that is useful for somebody searching for a specific term.
The goal for me isn't just 100,000 pages. It's to provide value on those 100k pages and answer the very specific search query the user is looking for.
To provide the most value possible, I try to include:
- At least 700-1000 words of content (with headings, lists etc)
- Conditional content based on the specific page
- HTML tables, maps or interactive data charts
- Internal links to related pages
I always try to make my sites better than the competition. More detailed, more data, easier to use, the ability to download or export the data.
Programmatic SEO is a tool, but you can and should still build top quality sites with it.
The SEO basics
You may think that because I've got a network of sites with 2m+ pageviews a month that I'm some kind of SEO expert. I'm not!
I just do the basics really well:
- Keyword research (can I rank? if so, how quickly? what is the competition?)
- Decent H1 title with the keyword phrase at the beginning
- Same for the page title
- Breadcrumbs to internally link to related pages
- Quality content written around the data
- Images, tables, and charts to highlight the data
- Using bold text to highlight key pieces of information
That's pretty much it. I did add some basic schema for the breadcrumbs but on most sites I haven't added anything more.
How do I get all these URLs indexed?
Getting your pages indexed in Google is kinda important if you want people to find them. Recently it's been a bit of a problem as the indexing rate has slowed down, but generally here's what I do.
I've got all my data in the database, so I create a route for /sitemaps/index.xml or whatever name I choose.
In my controller, I pull the slugs and any other values I need from the db to create the URLs and then pass them to a template which renders valid XML.
Most of my early sites have sitemaps with 50k URLs in them, and I've since been advised to keep them to ~10k per sitemap at a maximum.
If you have more than 10k URLs, then you need to create a sitemap index, which is a sitemap that points to other sitemaps.
I do that by creating a route like /sitemaps/index-([0-9]).xml and then grabbing the number from the URL and passing it to the controller.
With that number, I can offset my database lookup so I only get the URLs for the sitemap I want.
So if it's the first sitemap, I'll get the first 10k URLs. If it's the second sitemap, I'll get the next 10k URLs etc.
Any decent dev can do this for you, and probably a lot better than I've done it!
I add sitemaps to robots.txt file, Google Search Console, and submit some pages to IndexMeNow. Rather than do all 10k (which would cost a fortune) I try to index pages which have a lot of links to other pages - like car manufacturer and model pages but not the specific years.
On top of the sitemaps, I internal link to as many pages as possible using breadcrumbs, links to other pages in tables or sidebars etc.
How can you get started?
I realise that my process is only really useful right now if you have some dev skills, the willingness to learn some, or are able to hire a dev to build something for you.
The most important thing to get started is to do some good keyword research and find a topic or niche that has the potential for a lot of modifier terms.
With the car example, if I have a database of cars, I can have pages like:
/cars/aston-martin
/cars/aston-martin/db9
/cars/aston-martin/db9/2016
So if my site is about the miles per gallon of cars, the head term would be "Aston Martin MPG", and I might not be able to rank easily for that.
But I could definitely have a shot at ranking for "Aston Martin DB9 MPG" or "2016 Aston Martin DB9 MPG".
Finding the right data source and preparing the basic SEO is the most important part.
At this point you can either build the site yourself (using code or no code) or pass the database as a CSV to a developer to build it for you.
Start simple, and you can always build it out further and change it later.
How can I learn more about programmatic SEO?
The best way to learn is by doing it and figuring things out for yourself.
This site is a work in progress but over time I will add case studies and examples here for you to follow along with (using code and no-code).
Questions
If you have any questions, feedback, or topics you want to suggest for the blog, hit me up on Twitter @iannuttall.