Robots file is necessary for your website or blog because it will help you to boost your SEO. With the help of Robots.txt file, you can give the robots an order which web page should crawl or which should not. So you should know about How to Create Robots.txt file in WordPress.
Before create or submit a robots.txt file, you should know about What is Robots.txt file?
What is Robots.txt File?
The robots.txt file is used to instruct the search engines about how to index or crawl your web pages present on the website. With the help of robots.txt, you can decide which web page should index or crawl for your website.
If you do not want to index or crawl any web page then you can prevent search engine form indexing and crawling.
Robots.txt file is present in the root directory of your Website or blog.
Basic Format of the robots.txt file.
User-agent: ( User-Agent Name )
Disallow: ( URL String not to crawl and index)
User-agent: ( User-Agent Name )
Allow: ( URL String to crawl and Index )
Sitemap: ( Your website Sitemap URL ) (Know how to Create a Sitemap in WordPress)
Example: This is the ideal robots.txt file for your WordPress website or blog.
Why You Should Use Robots.txt file?
- You can give permission to search engines for crawling and indexing of any web page.
- If you do not want to crawl or index any page then you can do with the help of robots file.
- You can block unnecessary pages like admin, plugins, themes pages.
- Search bots have quota then you have to need a robots.txt file.
How to Create Robots.Txt File in WordPress for SEO Purpose?
So you can add robots.txt file in WordPress with the help of two methods.
- Create Robots.txt file with the help of Plugin.
- You can add Robots.Txt file with the help of cPanel.
How to Create Robots.Txt File in WordPress using Yoast SEO Plugin?
Step 1: Go to the WordPress admin dashboard and Install the Yoast SEO plugin.
Step 2: Go to Yoast SEO and click on Tools.
Step 3: In tools, there are three tools available in Tools.
- Import and Export
- File Editor
- Bulk Editor
- Text Link Counter
So go to File Editor. You can edit robots.txt file and .htaccess file in File Editor.
Step 4: Click on Create Robots.Txt File.
By default, you will see this robots.txt file in Yoast SEO.
So delete this file and paste your robots.txt file in robots.txt section.
Step 5: Put your robots.txt in Yoast SEO. Click on Save Changes to robots.txt.
So here your robots.txt file is created and added to your WordPress website or blog.
This is the first method for How to Create Robots.Txt file in WordPress using a Yoast SEO plugin.
How to Add or Edit Robots.Txt File in WordPress using cPanel?
Step 1: Open a Notepad and Put the robots file instructions in Notepad. Save this notepad as robots.txt in your computer folder.
Step 2: Go to your WordPress website cPanel. Click on File Manager and open public_html file.
Step 3: Upload the robots.txt file in public_html. Click on Upload. Your robots.txt file will upload by choosing Select File.
Step 4: So your file will upload in WordPress and robots.txt file successfully added to your WordPress website or blog successfully. You can see robots.txt file in public_html.
If you want to edit your robots file then select your robots.txt file and click on Edit. So you can do any changes after edit your robots.txt file.
This is the complete guide about How to Add Robots.Txt file in WordPress using cPanel.
How to Test Your Robots.Txt File?
There are many robots.txt testing tools but we recommend to use the Google Search Console tool for robots.txt. So you should test your robots.txt with the help of robots.txt tester tool.
Step 1: Open your Google Search Console by login through Gmail.
Step 2: Click on Go to Old Version. It will redirect you to the old version of the Google Search Console tool.
Step 3: In Crawl, you will see robots.txt tester. Go to robots.txt tester and your robots.txt file will show in this section.
So with the help of this tool, you can test or validate your robots.txt file. It will give you errors and warnings if exists.
Step 4: If you want to test the archives then put archives in the testing tool.
- If it has blocked then it will show in red.
- Otherwise status has shown in the green format if it has not blocked.
So this is the complete guide about how to test your Robots.Txt file with the help of robots tester tool.
Note: If you want to see your website robots.txt file then put robots.txt behind your website URL.
Otherwise, You can see your robots.txt file in robots.txt tester tool which is present in the Crawl section. Click on See Live robots.txt in the robots.txt tester and you can see your robots.txt file.
So this is the complete guide about How to Create Robots.Txt in WordPress and how to test the robots.txt file through the testing tool.
With the help of this guide, you can easily create robot file for your WordPress website or blog. You can easily test your website or blog robots.txt file.
If you have not created robots.txt file for your website then you should create robots.txt file for unindexing unnecessary pages of your WordPress website or blog. If you want to know about more plugins to help you do so here is the list : 6 Best WordPress Robots.Txt Editor Plugins for Your Website or Blog
It will help you to index your desired web pages and you can get more help through robots file for SEO purposes.
If you have liked this article then share it on your social media profiles.