Robots

This component for GoHugo adds a customizable robots.txt to your website.

Add this module
[[module.imports]]
path = "github.com/davidsneighbour/hugo-robots"
disable = false
ignoreConfig = false
ignoreImports = false
Latest Version v1.2.5 (2022-08-08)
Fix to this version
hugo mod get github.com/davidsneighbour/[email protected]

Usage

This component can be used as drop-in without much configuration. However, robots.txt generation needs to be enabled in your configuration, eg. config.toml:

1enableRobotsTXT = true

You can add configuration parameters per page in your frontmatter:

1robotsdisallow: true

This will add a Disallow line for the current URL. Note, that with clean URLs this will disallow bot-access for all sub-folders and sub-urls of the current item.

Adding global dis/allows

You can add global additions to your robots.txt via config/_default/params.toml configuration:

1[[dnb.robots.useragents]]
2name = "Googlebot"
3disallow = ["/nogooglebot/", "/anotherdirectory/"]
4
5[[dnb.robots.useragents]]
6name = "Googlebot2"
7allow = ["/yesgooglebot/", "/anotherdirectory/"]

Configure meta-robots tags

Configure the robots tag with the following individual configuration parameters in your frontmatter:

1config:
2  follow: true
3  index: false

Add/edit global defaults in config.toml or config/_defaults/params.toml:

1[dnb.robots]
2index = true
3follow = true

Default without any configuration is true in both parameters.

If you are using davidsneighbour/hugo-head then the meta tag is automatically added to your head-tag. If not you will need to add a call to the meta tag:

1{{- partial "head/robots.html" . -}}

Are you into ASCII-art?

If you like to do your robots.txt proud — you know what I mean — then you can use the following configuration parameters in config/_default/params.toml to add some flourish to your robots.txt:

1[dnb.robots]
2initialComment = '''
3# comment at the beginning of robots.txt
4'''
5concludingComment = '''
6# comment at the end of robots.txt
7'''

Be careful to properly comment out these parts.

Remove dnb-org notices

The plugin adds some branding notes to your robots.txt. It’s a free plugin. If you wish to remove these mentions feel free to set the disableBranding option in your config/_default/params.toml to true:

1[dnb.robots]
2disableBranding = true

See this module in action