cancel
Showing results for 
Search instead for 
Did you mean: 

Two Robots.txt, One Domain and One Hosting Account

Just spent the morning working on an issues that was revealed using GoogleBot viewer, and SEMRush's seobot. Both are showing issues with robots.txt

 

Here's the issue:

1.) Different Robots.txt files based on WWW

On every Wordpress managed hosting account we have, there's a different version of Robots.txt depending on whether you're viewing the domain with "www" or without. For example:

http://www.example.com/robots.txt   * vs *

http://example.com/robots.txt

 

2.) Content Proves They Are Different

The content in the files is different, and only the version with the URL that matches the primary domain in the Managed WordPress domain settings shows the correct version. However, multiple crawlers/bots are viewing both, or the wrong one.

 

http://www.example.com/robots.txt - has this content:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: wp-config.php

 

http://example.com/robots.txt has this content:

User-Agent: *
Disallow:

 

This shows after htaccess has been disabled, GoDaddy Cache plugin clears, and default themes and no plugins active. It appears to not be an issue with WordPress/Themes/Plugins.

 

3.) Server Settings and DNS Zones Are Correct

This issue occurs regardless of .htaccess settings, and SSL. DNS zones are set correctly on all sites tested.

 

At this point, a brand new WordPress site, with a brand new theme, and no plugins that edit htaccess or robots.txt is not being crawled by Google. I was going to remove the hosting account, and reset it, but once I found this was occurring on all instances of Managed WordPress Hosting we have, which is 10 accounts within our reseller account alone.

 

Ultimately, because we are seeing crawler issues due to this, my concern is about the SEO performance of my client's sites. So, any ideas on how to fix this issue would be greatly appreciated.

 

If examples on actual GoDaddy Managed Wordpress accounts are necessary, please let me know where to send them to GoDaddy reps only. Rather not share those urls publicly. Thanks!

 

Looking forward to any feedback or ideas.

3 REPLIES 3
Moderator
Moderator

Re: Two Robots.txt, One Domain and One Hosting Account

Hi @sbmedvec80, thanks for posting.

I just checked this on my personal Managed WordPress account and didn't see this behavior. I also haven't heard of this happening before.

The issue you are describing does seem to be specific to your account. I suggest contacting Support so they can investigate your account in detail.

^Gary

Gary - GoDaddy | Community Moderator
24/7 Support | Check System Status

Re: Two Robots.txt, One Domain and One Hosting Account

@GaryA - Thanks for checking.

 

I posted here because Support suggested I use the Product Feedback form so it could be investigated further, but I couldn't find that form. This issue stumped support from Domains, SEO, and Hosting.  

 

Any thoughts if this could be part of a reseller account issue? I am a reseller, but can't think of any reason why this would be occurring across all our Managed WordPress accounts.

Moderator
Moderator

Re: Two Robots.txt, One Domain and One Hosting Account

Hi @sbmedvec80, sorry for the delay in getting back to you.

Since I'm not able to duplicate this on my account, I don't think it's related to the Managed WordPress service. Each website/domain on the service is separate from other sites so there shouldn't be any cross-contamination of issues, so to speak.

Without knowing your specific sites, it's hard to be able to offer more suggestions. I'd only be able to suggest what you've already stated you've done. I would look at any custom caching or security plugins you might be using and try temporarily disabling them to see if that alleviates the issue.

^Gary

Gary - GoDaddy | Community Moderator
24/7 Support | Check System Status