Why can't GoogleBot forget a 404 File Does Not Exist?

Why can't GoogleBot forget a 404 File Does Not Exist? - If a page has internal and external outgoing links to redirecting URLs, it’s returning 3xx (301, 302, etc.) HTTP status codes standing for redirection. This issue means that the page does not exist on a permanent or temporary basis. It appears on most of the popular web browsers, usually caused by a misconfigured website. However, there are some steps you can take to ensure the issue isn’t on your side. You can find more details about redirecting URLs by reading the Google Search Central overview. In this article, we’ll go over how you can fix the Why can't GoogleBot forget a 404 File Does Not Exist? error on your web browser. Problem :

For two months now, googlebot is trying to get a file which does not exist anymore. This is just one example out of many: I had renamed the file to a better name and removed the old file. Now, why does Google insist on getting a file which it already has seen does not exist for months? doesn't it just give up and get on being a happy bot?

My Error log file is filled with these repeating lines all wanting to get that one file, although they know for already long time that its not there:

1. What to do in these situations?


2. Are there automatic redirection rules that handle via 301 to the home page?

My error log:

[Sat Mar 05 01:55:41 2011] [error] [client 66.249.66.177] File does not exist: 
/var/www/vhosts/website.org/httpdocs/extraNeus.php 

[Sat Mar 05 01:58:20 2011] [error] [client 66.249.66.177] File does not exist: 
/var/www/vhosts/website.org/httpdocs/extraNeus.php

[Sat Mar 05 02:03:57 2011] [error] [client 66.249.66.177] File does not exist: 
/var/www/vhosts/website.org/httpdocs/extraNeus.php

on and on ... and on...

Solution :

put this in .htaccess file (change the path/to/files to your file locations)

Redirect 301 /extraNeus.php http://www.yoursite.com/new_page.php

you can leave it in there forever or next time google indexes the page you should be good to remove it.

You should use a permanent redirect to tell GoogleBot and others where the page has moved to.

If the page no longer exists at all then you can add entries in robots.txt to tell bots not to access the missing page:

User-agent: *     
Disallow: /extraNeus.php

The reason that Googlebot is still trying to hit the page is that there is probably a link to it somewhere. When Google finds a link, it follows it without checking whether it has previous seen a 404 for that page or not. So if this page keeps getting hit, it's probably because of a link.

I agree with Crayon Violet and suggest that you include the 301 redirect from the old page to the new one.

We hope that this article has helped you resolve the google, redirects, 404 error in your web browsers. Enjoy browsing the internet uninterrupted!