Google will not index my 1and1 site!

[john_a77]

I have been trying to get Googlebot to index my site for about 3 months now. I have been monitoring it via Google Sitemaps.

http://www.google.com/webmasters/sitemaps/

I don't use an actual sitemap, but this site shows the results of Googlebot's attempted crawls. It always receives errors when it tries to index my site, such as "Network unreachable" or "robots.txt unreachable." However, I know my robots.txt file is ok. In fact, I can test my robots.txt file from within the Google Sitemaps site, and it loads it and tells me everything is ok.

There is also a related discussion which leads me to believe this is a problem with 1and1. I believe 1and1 is either blocking Googlebot or giving it low priority. See here:

http://groups.google.com/group/google-sitemaps/browse_thread/thread/16acdb1f39e42f64/627238340c64cde3?lnk=st&q=%22robots.txt+unreachable%22&rnum=1&hl=en#627238340c64cde3

Has anyone else had this problem? Are there any suggestions about what I can do?

Thank you!

[john_a77]

I just spotted another good post on this topic:

http://groups.google.com/group/google-sitemaps/browse_thread/thread/3b34eefc3bbbbcb7/6e696b848eb084c6?lnk=st&q=1and1+%22network+unreachable%22&rnum=2&hl=en#6e696b848eb084c6

It seems that many people are experiencing this problem when using shared hosting. I am using shared Linux hosting, so that makes sense. Unless 1and1 is planning to respond to this issue, I think I'll have to jump ship.

This is bad news because I've put websites for two small companies online using 1and1's shared hosting. Google results are important to them, so I may need to move them also. This sucks.

[skunkboy]

Google is constantly changing the way they do things. The more relevant sites you have linking into yours, the better. I know I see googlebot in my logs almost daily on all of the site we run on our server. Needless to say, the main one I spend my own personal time on averages two-million pageviews per month and when searching for any of the names of people within my site, their page on my site is often in the top ten if not the very first result. As well, other key search terms that I've targeted also place me in very high ranks.

Granted I'm on a level 3 dedicated server but still, I don't see why 1and1 would have a different policy for other sites it hosts. Check that you have key terms in your page and that you get high-hitters to link into your site. The more times google is told to check you out (by other similar sites), the better.

[eWebtricity]

We have no problems with Googlebot accessing our dedicated Root III servers on 1and1's network at all.

[Highland]

How old is your domain? If it's 3 months old then this isn't unusual.

[john_a77]

How old is your domain? If it's 3 months old then this isn't unusual.

My domain is 2+ years old. And I'm not really that concerned about how high I show up on Google's ranking, I just want to show up period. When Google is reporting that it can't load my robots.txt, something funky is going on.

I tried submitting a sitemap a few hours ago, and it seems to have caused Google to reprocess some things. Now when I search for site:mydomain.com, I get one result, which is the current version of my homepage. I used to get several results from how my site looked 5 months ago, but those results have all disappeared. So, hopefully this is first step to Google recrawling my site properly. Time will tell.

[Highland]

Something that Google doesn't make clear is that they never show you everything behind the curtain. A site: search will seldom return all the pages in Google's index. It's also highly inaccurate as to what had been indexed. In fact, Google errs on the side of keeping things too long as sometimes an update will have site: return pages that are years old and don't even exist anymore. So if site: returns any pages you're indexed, even if Google says you only have one page.

I don't know what to tell you on the robots.txt problem. If you can access it by doing a yourdomain.com/robots.txt then Google should be able to see it. If not, you might want to contact Google and see if they can help. I'm willing to bet Google would like to know why as well if that's the case.

[nullensc]

Is this problem also affecting MSN, Yahoo and others?

Do you use .htaccess? If so is it free of any directives that may prevent a SE from accessing the site?

[john_a77]

I just received this email from 1and1. I'm impressed that they're at least acknowledging that a problem exists. Hopefully they'll work it out soon.

"Thank you for contacting us. We do not block the Google bots from indexing the site however we do due to technical reasons block some traffic from Google, this we are looking to resolve in the near future, however we do not have an exact time line. Therefore, your google site maps should still work as usual provided google doesn't verify them every few minutes. And, google should still be able to index most of your pages if their caching is setup correctly with the site map."

I wonder what those "technical reasons" are? Perhaps that they've got too many shared site crammed onto each server?

I think I get a similar problem from Yahoo, but not from MSN.

[C-4 Hosting]

I wonder what those "technical reasons" are? Perhaps that they've got too many shared site crammed onto each server?



If your domain is being hosted on a shared IP, then you can goto http://www.WhoIs.sc and enter your domain name for lookup. It will then tell you how many domains that particular server is hosting.