The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix?

The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix? - If a page has internal and external outgoing links to redirecting URLs, it’s returning 3xx (301, 302, etc.) HTTP status codes standing for redirection. This issue means that the page does not exist on a permanent or temporary basis. It appears on most of the popular web browsers, usually caused by a misconfigured website. However, there are some steps you can take to ensure the issue isn’t on your side. You can find more details about redirecting URLs by reading the Google Search Central overview. In this article, we’ll go over how you can fix the The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix? error on your web browser. Problem :


We have a website:
www.example.com which redirects to https://www.example.com.



Therefore when you type www.example.com/robots.txt you are redirected to https://www.example.com/robots.txt.



Another point to note:
The main mirror in Yandex will be decided by the crawler, it won't allow us to set www.example.com or set https://www.example.com as the main mirror.



Please note: This error that I am getting



The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used is from this page :
https://webmaster.yandex.com/robots.xml





Update:



We updated our Robots.txt to the following after the answers.



    User-agent: Yandex
    Sitemap: https://www.example.com/sitemap.xml
    Disallow: /some_pages/
    Host: https://www.example.com

    User-agent: *
    Sitemap: https://www.example.com/sitemap.xml
    Disallow: /some_pages/

Solution :

You can set a main mirror in Yandex.
I think you need something like this:



Host: myhost.ru # uses

User-agent: *
Disallow: /cgi-bin

User-agent: Yandex
Disallow: /cgi-bin
Host: www.myhost.ru # is not used


or even:



# domen.myhost.com is the main mirror
# www.domen.myhost.com, is the correct use of 
# the host Host directive:User-Agent: *
Disallow:Host: domen.myhost.com


The Host directive should contain:




An HTTPS protocol indication if the mirror is only available via a protected channel (Host: https://myhost.com).

One concrete domain name that corresponds to RFC 952 and is not an IP-address.

The port number, if necessary (Host: myhost.com:8080).



More information : https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml


We hope that this article has helped you resolve the redirects, https, sitemap error in your web browsers. Enjoy browsing the internet uninterrupted!

Comments

Post a Comment

Popular posts from this blog

How to redirect to any domain [duplicate]

How to redirect to any domain [duplicate] - If a page has internal and external outgoing links to redirecting URLs, it’s returning 3xx (301, 302, etc.) HTTP status codes standing for redirection. This issue means that the page does not exist on a permanent or temporary basis. It appears on most of the popular web browsers, usually caused by a misconfigured website. However, there are some steps you can take to ensure the issue isn’t on your side. You can find more details about redirecting URLs by reading the Google Search Central overview. In this article, we’ll go over how you can fix the How to redirect to any domain [duplicate] error on your web browser.  Problem : This question already has an answer here : What are the most commonly used and basic Apac...
Read more

"302 found" for index page bad for SEO?

"302 found" for index page bad for SEO? - If a page has internal and external outgoing links to redirecting URLs, it’s returning 3xx (301, 302, etc.) HTTP status codes standing for redirection. This issue means that the page does not exist on a permanent or temporary basis. It appears on most of the popular web browsers, usually caused by a misconfigured website. However, there are some steps you can take to ensure the issue isn’t on your side. You can find more details about redirecting URLs by reading the Google Search Central overview. In this article, we’ll go over how you can fix the "302 found" for index page bad for SEO? error on your web browser.  Problem : I'm having trouble finding a clear answer on this. I'm optimizing someone else's website, and a tool pointed out that the index page returns a "302 found" (it redirects to a long URL with cookie info). Is this bad for the index page SEO to have a 302 redirect? S...
Read more

Create redirect from url like www.example.us/?p=100&option=

Create redirect from url like www.example.us/?p=100&option= - If a page has internal and external outgoing links to redirecting URLs, it’s returning 3xx (301, 302, etc.) HTTP status codes standing for redirection. This issue means that the page does not exist on a permanent or temporary basis. It appears on most of the popular web browsers, usually caused by a misconfigured website. However, there are some steps you can take to ensure the issue isn’t on your side. You can find more details about redirecting URLs by reading the Google Search Central overview. In this article, we’ll go over how you can fix the Create redirect from url like www.example.us/?p=100&option= error on your web browser.  Problem : I am switching CMS and have about 600 URLs that need to get redirected to the new scheme. The URL on the old site looks like the following on my test server: www.example.us/?p=100&option=com_wordpress&Itemid=619 The new URL scheme will exist as: www.exa...
Read more