+ Start a Discussion

CNAME, Force.com Sites, and Robots

I am setting up a CNAME to map a custom domain I purchased with my Force.com Site (free Force.com).  Free Force.com doesn't let you set your Force.com domain so I am using a CNAME solution instead. I am able to get the CNAME and custom web address configuration working. So I can go to my domain: http://www.customdomain.com and it automatically masks and loads my free force.com site. 


My question is this: With CNAME enabled, should the robots.txt file be on the Force.com Site configuration? Or should it be stored within my custom domain folder structure and not within Force.com? I believe it should on Force.com since my custom domain will not have any actual HTML files - they are all Visualforce pages on Force.com. I just wanted to confirm my understanding is correct (or not).




There is an option in Sites setup to pick a visualforce page to use as your robots.txt file.


It will render on both the .force.com and custom web address version of your site.


Your robots.txt file would look something like this:



<apex:page contentType="text/plain" showHeader="false">
User-agent: msnbot
Disallow: /



The robots.txt file is cached for 24 hours, so I suggest developing in a DE org, testing it, and then migrating to your force.com org.



Thanks for the reply. I have the Visualforce page ready to go as you illustrated. I am new to SEO, CNAME, and robots.txt so I want to make sure I understand this correctly. If I put the robots.txt file within Force.com Site (as a VF page), you mention search engines (google, bing, yahoo, so on) will see the robots.txt file as both my force.com domain (customdomain.force.com) and as my godaddy.com purchased domain, mydomain.com.


I will create a CNAME on godaddy to map to my force.com site landing page. And create a custom web address in force.com to map back to my godaddy domain. 


All that said, my question is this: Assuming my website is highly ranked via Google (example), will it show up as godaddy.com domain, force.com site domain, or both? Ideally I only want the search engines to display my godaddy.com custom domain but index my Force.com Site visualforce pages (and never show my force.com site domain within search engine results). 




If you only give out the link to your custom web address and link to your custom web address, google will (probably) never know about your force.com address.


If it becomes a problem down the road, there are tricks you can use, but for getting started you can just always reference the custom web address.