Gatsby Robots file
by John Vincent
Posted on September 20, 2019
Building a robot.txt
file with Gatsby is straightforward.
Gatsby Robots Plugin
Using the plugin gatsby-plugin-robots-txt
, add to gatsby-config.js
{
resolve: `gatsby-plugin-robots-txt`,
options: {
host: siteUrl,
sitemap: `${baseUrl}sitemap.xml`,
policy: [{ userAgent: `*`, allow: `/` }],
},
},
The robots.txt
file will be build only for production builds.
Gatsby
- Gatsby and Client Only Components
- Gatsby Getting Started
- Gatsby Google Analytics
- Gatsby Google Maps
- Gatsby Helmet
- Gatsby Manifest
- Gatsby Markdown
- Gatsby Overview of johnvincent.io
- Gatsby React Icons
- Gatsby Robots file
- Gatsby RSS Feed
- Gatsby Sass
- Gatsby Sitemap
- Gatsby Webpack Bundle Analyzer
- Migration of johnvincent.io to Next.js from Gatsby
- Overview of port to Next.js from Gatsby
- Configuring Google Domains
- Configuring Google Domains
- Configuring Meta Tags
- Feediator Google Authentication
- Gatsby Robots file
- Gatsby Sitemap
- Google Authentication
- Google Coverage Issues
- Google Custom Search and Jekyll
- Google Fonts and Webfontloader
- Google Gmail Configuration
- Google Mobile Usability Issues
- Google Youtube APIs
- Website Review Reference