How to edit Shopify robots.txt?

184

Shopify is a popular e-commerce platform that allows you to create and manage your own online store. Sell online and in person. Sell locally and globally. Sell direct and wholesale. Sell on desktop and mobile.

What is robots.txt?​

The Robots.txt file, which consists of generally accepted code content, whose name part of “The Robots Exclusion Protocol”, it is located in the root directory of your website, provides guidance on which parts of your website can and cannot be crawled by robots (spiders, crawlers, bots) visiting your site.

Search engines crawl sites with robot computer software and index them to show in search results. With these directions, which are also important for search engine optimization, you can ensure that your website is listed in the most accurate and effective way. And adjustments to this small file can significantly impact your store’s search engine visibility and Shopify SEO performance.

It should not be forgotten that the main purpose of robots.txt file commands is to restrict the access of search robots and specify the areas of your site that you do not want to be indexed in search results. So, even if you do not have a robots.txt file, your site will be indexed somehow and even you have an existing file, still some of the unsupported fields (such as “crawl-delay”) will be ignored by Google or other search engines. According to the updated documentation, Google officially supports the following fields in robots.txt files:

• user-agent
• allow
• disallow
• sitemap

On the other hand, updating that robots.txt on Shopify store, is a little different than Wordpress and other platforms. As almost every other simple solutions -ridiculously- there is a “paid” app in Shopify app store, but it’s quite easy to learn it, so there is no need use any paid option.

You can follow the steps below to prepare robots.txt on this blog post. However, if you are not confident to make these changes, it’s better to consult to a professional, as incorrect usage of the feature may result in the loss of traffic.

Why and when should you update Shopify robots.txt ?​

As default, Shopify automatically publish a robots.txt file with using optimum rules for your website, you can review by visiting “www.yourwebsite.com/robots.txt” URL. In most cases, that default file might be enough to keep as it is and there won’t be any need to amend it.

However, you need to observe your Google Search Console to have a clear understanding the current situation of your website. In addition to default pages created by Shopify, your theme or the apps could create dozens of new URLs which might have negative impact.

Crawl Budget/Limit: It’s a term describing the amount of resources search engines allocate to crawling each website. If your website requires more resources than is allocated to it, then pages will be skipped from crawling regularly. You should make sure the important pages, collection or product pages are not affected due to that. As best practice, examine your site, find out unnecessary URLs and use “Disallow” rule to remove or block them to prevent low quality pages from getting crawled by Googlebot.

Thin and needless content: There are several pages are created automatically and submit to search engines which mostly has insignificant value for your ranking and better to review to optimize as part of SEO practices.

• The pagination pages (/?page=1,2,3 ),
• blog tag pages (/blogs/*/tagged/),
• brand or vendor pages (/collections/vendors?q=XXX)
• faceted navigations & filter pages (/collections/types?q=) or (/collections/?color=) or (/collections/all?sort_by=)
• etc.

Update on search engine algorithms: The supported rules might change in time, better to follow Google’s robots.txt official documents to be keep updated.

 

How to customize robots.txt in Shopify?

If you are using Wordpress website, you can use dozens of free plugin or simply access to your FTP to download the file or edit it on cpanel options, if any. But the process is a little different for Shopify websites, first you need to create a robots.txt.liquid file, here are the steps to follow;

1. Open your Shopify Dashboard
2. In the Sales channels section, go to “Online Store > Themes“
3. In the Live theme section, click “…” > “Edit code“
4. In the left sidebar, under the Templates heading, click “Add a new template“
5. Change “Template type” to Robots.txt (If you don’t see that type, it means robots.txt was created previously, close here and go find it)
6. Click “Create template” or “Done“

The robots.txt.liquid template is located in the templates directory of the theme, it means if you will change the theme and already made changes on robots.txt file, then you should repeat it with the new theme. So, I recommend to make a copy of this file for future implementation.

└── theme
├── layout
├── templates
| …
| ├── robots.txt.liquid
| …

The file should contains the following codes, and it’s recommended not to remove these default rules as these are updated regularly to ensure that SEO best practices are always applied.

# we use Shopify as our ecommerce platform
{%- comment -%}
# Caution! Please read https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
{% endcomment %}
{% for group in robots.default_groups %}
  {{- group.user_agent -}}

  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Customize Robots.txt.liquid

At this file you can make the following customizations:

• Add a new rule to an existing group
• Remove a rule from an existing group
• Add custom rules
  • Block certain crawlers
  • Allow certain crawlers
  • Add extra sitemap URLs

In this example, we are using disallow rule to block search, vendor and blog pagination URLs. The * applies a wildcard to mean “anything here”

{% for group in robots.default_groups %}
  {{- group.user_agent -}}

  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}

    {%- if group.user_agent.value == '*' -%}
{{ 'Disallow: /*?q=*' }}
{{ 'Disallow: /collections/vendors*?*q=*' }}
{{ 'Disallow: /blogs/?page=*' }}
  {%- endif -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

The default Shopify robots.txt file contains a sitemap URL, if you wanted to add an additional sitemap, you could add this:

Sitemap: [sitemap-url]

How to reset Shopify robots.txt to default?

In order to reset your Shopify robots.txt file to default, simply find and delete the created robots.txt.liquid template. And as you cannot recover it, it’s better to save the copy of the customizations.

1. Open your Shopify Dashboard
2. In the Sales channels section, go to “Online Store > Themes“
3. In the Live theme section, click “…” > “Edit code“
4. In the left sidebar, search for robots.txt.liquid in “Filter files” field
5. Locate the template and click to trash bin icon to delete it.