Robots

Robots v1.2.5

This component for GoHugo adds a customisable robots.txt to your website.

Add this module 
[[module.imports]]
	path = "github.com/davidsneighbour/hugo-robots"
	disable = false
	ignoreConfig = false
	ignoreImports = false
Latest Version v1.2.5 (2022-08-08)
Fix to this version 
hugo mod get github.com/davidsneighbour/[email protected]

Usage

This component can be used as drop-in without much configuration. However, robots.txt generation needs to be enabled in your configuration, eg. config.toml:

1enableRobotsTXT = true

1enableRobotsTXT: true

1{
2   "enableRobotsTXT": true
3}

You can add configuration parameters per page in your frontmatter:

1robotsdisallow = true

1robotsdisallow: true

1{
2   "robotsdisallow": true
3}

This will add a Disallow line for the current URL. Note, that with clean URLs this will disallow bot-access for all sub-folders and sub-urls of the current item.

Adding global dis/allows

You can add global additions to your robots.txt via config/_default/params.toml configuration:

1[dnb]
2  [dnb.robots]
3[[dnb.robots.useragents]]
4    disallow = ['/nogooglebot/', '/anotherdirectory/']
5    name = 'Googlebot'
6[[dnb.robots.useragents]]
7    allow = ['/yesgooglebot/', '/anotherdirectory/']
8    name = 'Googlebot2'

 1dnb:
 2  robots:
 3    useragents:
 4    - disallow:
 5      - /nogooglebot/
 6      - /anotherdirectory/
 7      name: Googlebot
 8    - allow:
 9      - /yesgooglebot/
10      - /anotherdirectory/
11      name: Googlebot2

 1{
 2   "dnb": {
 3      "robots": {
 4         "useragents": [
 5            {
 6               "disallow": [
 7                  "/nogooglebot/",
 8                  "/anotherdirectory/"
 9               ],
10               "name": "Googlebot"
11            },
12            {
13               "allow": [
14                  "/yesgooglebot/",
15                  "/anotherdirectory/"
16               ],
17               "name": "Googlebot2"
18            }
19         ]
20      }
21   }
22}

Configure meta-robots tags

Configure the robots tag with the following individual configuration parameters in your frontmatter:

1[config]
2  follow = true
3  index = false

1config:
2  follow: true
3  index: false

1{
2   "config": {
3      "follow": true,
4      "index": false
5   }
6}

Add/edit global defaults in config.toml or config/_defaults/params.toml:

1[dnb]
2  [dnb.robots]
3    follow = true
4    index = true

1dnb:
2  robots:
3    follow: true
4    index: true

1{
2   "dnb": {
3      "robots": {
4         "follow": true,
5         "index": true
6      }
7   }
8}

Default without any configuration is true in both parameters.

If you are using davidsneighbour/hugo-head then the meta tag is automatically added to your head-tag. If not you will need to add a call to the meta tag:

1{{- partial "head/robots.html" . -}}

Are you into ASCII-art?

If you like to do your robots.txt proud — you know what I mean — then you can use the following configuration parameters in config/_default/params.toml to add some flourish to your robots.txt:

1[dnb]
2  [dnb.robots]
3    concludingComment = "# comment at the end of robots.txt\n"
4    initialComment = "# comment at the beginning of robots.txt\n"

1dnb:
2  robots:
3    concludingComment: |
4      # comment at the end of robots.txt      
5    initialComment: |
6      # comment at the beginning of robots.txt      

1{
2   "dnb": {
3      "robots": {
4         "concludingComment": "# comment at the end of robots.txt\n",
5         "initialComment": "# comment at the beginning of robots.txt\n"
6      }
7   }
8}

Be careful to properly comment out these parts.

Remove dnb-org notices

The plugin adds some branding notes to your robots.txt. It’s a free plugin. If you wish to remove these mentions feel free to set the disableBranding option in your config/_default/params.toml to true:

1[dnb]
2  [dnb.robots]
3    disableBranding = true

1dnb:
2  robots:
3    disableBranding: true

1{
2   "dnb": {
3      "robots": {
4         "disableBranding": true
5      }
6   }
7}

See this module in action