Robots

This component for GoHugo adds a customizable robots.txt to your website. This module overrides the internal robots.txt generation of Hugo and lets you configure what robots.txt in your public folder will contain. It also offers a meta-robots tag for your head section.

Usage

This component can be used as drop-in without much configuration. However, robots.txt generation must be enabled in your main configuration, for instance config.toml or hugo.toml:

1enableRobotsTXT = true

1enableRobotsTXT: true

1{
2   "enableRobotsTXT": true
3}

You can add configuration parameters per content page in its frontmatter:

1robotsdisallow = true

1robotsdisallow: true

1{
2   "robotsdisallow": true
3}

This will add a Disallow line for the current URL. Note, that with clean URLs this will disallow bot-access for all sub-folders and sub-urls of the current item.

Adding global (dis)allows

You can add global additions to your robots.txt via config/_default/params.toml configuration:

1[dnb]
2  [dnb.robots]
3[[dnb.robots.useragents]]
4      disallow = ['/nogooglebot/', '/anotherdirectory/']
5      name = 'Googlebot'
6[[dnb.robots.useragents]]
7      allow = ['/yesgooglebot/', '/anotherdirectory/']
8      name = 'Googlebot2'

 1dnb:
 2  robots:
 3    useragents:
 4    - disallow:
 5      - /nogooglebot/
 6      - /anotherdirectory/
 7      name: Googlebot
 8    - allow:
 9      - /yesgooglebot/
10      - /anotherdirectory/
11      name: Googlebot2

 1{
 2   "dnb": {
 3      "robots": {
 4         "useragents": [
 5            {
 6               "disallow": [
 7                  "/nogooglebot/",
 8                  "/anotherdirectory/"
 9               ],
10               "name": "Googlebot"
11            },
12            {
13               "allow": [
14                  "/yesgooglebot/",
15                  "/anotherdirectory/"
16               ],
17               "name": "Googlebot2"
18            }
19         ]
20      }
21   }
22}

Configure meta-robots tags

Configure the robots tag with the following individual configuration parameters in your frontmatter:

1[config]
2  [config.robots]
3    follow = true
4    index = false

1config:
2  robots:
3    follow: true
4    index: false

1{
2   "config": {
3      "robots": {
4         "follow": true,
5         "index": false
6      }
7   }
8}

Add or edit global defaults in the [params] section of config.toml or in config/_defaults/params.toml:

1[dnb]
2  [dnb.robots]
3    follow = true
4    index = true

1dnb:
2  robots:
3    follow: true
4    index: true

1{
2   "dnb": {
3      "robots": {
4         "follow": true,
5         "index": true
6      }
7   }
8}

The default without any configuration is true for both parameters.

If you are using davidsneighbour/hugo-head then the robots meta tag is automatically added to your head section. If not, you need to add a call to the meta tag:

1{{- partial "head/robots.html" . -}}

You can cache this partial, but based on a per-page level:

1{{- partialCached "head/robots.html" . page -}}

Are you into ASCII-art?

If you like to do your robots.txt proud — if you catch my drift — then you can use the following configuration parameters in config/_default/params.toml to add some flourish to your robots.txt:

1[dnb]
2  [dnb.robots]
3    concludingComment = "# comment at the end of robots.txt\n"
4    initialComment = "# comment at the beginning of robots.txt\n"

1dnb:
2  robots:
3    concludingComment: |
4      # comment at the end of robots.txt      
5    initialComment: |
6      # comment at the beginning of robots.txt      

1{
2   "dnb": {
3      "robots": {
4         "concludingComment": "# comment at the end of robots.txt\n",
5         "initialComment": "# comment at the beginning of robots.txt\n"
6      }
7   }
8}

Be careful to properly comment out these parts.

Remove dnb-org notices

The plugin adds some branding notes to your robots.txt. It’s a free plugin. If you need to remove these mentions feel free to set the disableBranding option in your config/_default/params.toml to true:

1[dnb]
2  [dnb.robots]
3    disableBranding = true

1dnb:
2  robots:
3    disableBranding: true

1{
2   "dnb": {
3      "robots": {
4         "disableBranding": true
5      }
6   }
7}

See this module in action

• kollitsch.dev/robots.txt