By therobyouknow


2012-01-09 15:59:44 8 Comments

I'm not sure if this is a problem specific to how my hosting is structured or not.

Let's say the main domain for my account is maindomain.com (for purpose of discussion).

I am building other sites, e.g. anothersite.com and website.com

The folder/directory structure on my hosting would therefore be:

maindomain.com (root)
+
+---> anothersite.com (subfolder)
+
+---> website.com (subfolder)

The domains are mapped such that they map to their respective folders. So for example, anothersite.com takes the visitor to the index.php in the folder anothersite.com.

However, in Google, as well as just search results for these sites and their content - anothersite.com and website.com (what I want), I also see results that include:

  • maindomain.com/anothersite.com
  • maindomain.com/website.com

Obviously this is not the way I want search data for these sites to occur. I would like them to be omitted from the search results.

How can the hosting be configured so that the results for each of these respective domains are separate, i.e. not concatenated, so if I search for items relating to these respective sites, I only get results for anothersite.com relevant to it, same for website.com and same for maindomain.com - also used itself.

Such a solution to this problem should not damage the results that I do want to see.

I'm aware of robots.txt but I'm not sure what I need to put in these to stop the mixed-up/concatenated searches without damaging the real results. Or even if all search engines obey them. Is there something I can also do with .htaccess?

As it is shared-hosting, I don't have complete control over how the folder/directories are structured for this hosting so I would have to stick with the structure but ideas as to how to get desirable search results are welcome.

1 comments

@Kris 2012-01-09 16:29:59

I'd advise you to add a rewrite rule to your webserver configuration so that URLs are rewritten thusly:

maindomain.com/anothersite.com --> anothersite.com
maindomain.com/anothersite.com/page.html --> anothersite.com/page.html
etc....

With a 301 Moved Permanently redirection.

This should cause Google to correct this as pages are recrawled.

You could also add these folders to your main domain's robots.txt file and forbid Google from crawling them (and rely on it finding that material again or promoting the 'direct' version). You can speed this up by using Google Webmaster to remove these folders from the main domain (although it is anyones guess as to how long it will take for the 'direct' sites to show up on their own in Google).

@therobyouknow 2012-01-09 16:34:09

+1 @Kris thanks - this looks good! If I do this, how long before Google etc update their listing/indexing? I hope to accept your answer after trying it out, I've edited Apache directives in .htaccess before.

@therobyouknow 2012-01-09 16:48:42

Specific .htaccess directives needed might be these: webmasters.stackexchange.com/a/16816/3738

@Kris 2012-01-09 16:55:18

@therobyouknow How long it takes depends on how often Google crawls your site. Could be a few days, could be a few weeks. No good way to tell.

@therobyouknow 2012-01-09 17:00:02

I think your suggestion is better than that shown in webmasters.stackexchange.com/questions/16805/… because I would think yours stops the unwanted paths appearing in results in the first place, whereas theirs doesn't - rather it presents the visitor with a 404 page, it seems.

@therobyouknow 2012-01-10 00:44:23

I found out that my provider lets me move the sites' content up a level to avoid this scenario. But nonetheless this answer is good to know, to learn a bit more about Search Engine Optimisation and crawling to give certain results. Thanks again.

@MrWhite 2015-12-18 19:17:49

@therobyouknow Serving a 404 (as in that other answer) will also stop "the unwanted paths appearing in results in the first place" - but the problem here is that the unwanted paths are already indexed. Since they are already indexed it is preferable to redirect them.

Related Questions

Sponsored Content

2 Answered Questions

[SOLVED] Block third domain from being indexed

1 Answered Questions

1 Answered Questions

2 Answered Questions

1 Answered Questions

1 Answered Questions

[SOLVED] Merging multiple top-level domains into a single domain

1 Answered Questions

1 Answered Questions

Sponsored Content