Most sites do not have a robots.txt file created. However it can certainly have benefits in terms of controlling which web page content will be indexed by search engines. Likewise if you do not want to allow specific crawlers or folders to be indexed you can also add restrictions.
All you need to do is add a new file to the root of your website. If you are using a Sitemap file you can also link to it in your Robots.txt file so that it will be easier to find. Not having a robots file will also increase your 404 page not found errors because search engines will request this file first.
Here are some examples:
1. Standard Robots.txt template to allow everything, allow all robots and all files.
User-agent: * Disallow:
2. This example keeps all robots out (you may want to use on a web application login landing page):
User-agent: * Disallow: /
3. More often than not you will want to restrict specific folders from being indexed as well:
User-agent: * Disallow: /images/ Disallow: /private/
4. Linking to your sitemap file.
User-agent: * Sitemap: http://www.example.com/sitemap.xml # tell the bots where your sitemap is located/
For more information or to generate your own Robots.txt file visit Google’s support section.
Image Credit: davedehetre (via Flickr)
Possibly Related Posts:
- WordPress plugins to check for security threats
- Security Tips for Securing WordPress
- Reasons Customers Don’t Trust Websites
- Google Analytics Event Tracking Explained
- New Study: Small Businesses Not Updating their Websites
FavorHosting will have an upcoming planned data center migration currently scheduled for
Jan 14th, 2012 starting at 10PM CT
Since this is a planned data center migration it is a bit more involved then our standard maintenance windows and there will be an expected down time for your website. We will contact each of our customers that are impacted by this data center migration move and will provide you with information to keep you up to date and informed.
During this time the expected impact includes the following:
- During this time your website will likely experience some down time in the evening hours
- The website control panel you use to make changes including will be inaccessible
As part of this change we will be moving over to new IP addresses. If your domain name is currently using the following two new name servers your IP address for your website will be updated automatically. Name severs affected: dns1.stabletransit.com and dns2.stabletransit.com
Important: If you are using another domain name server (DNS) you will need to update your IP address manually. We will be contacting all of our impacted customers to provide the details and your new IP address.
Note: FavorHosting customers with a domain name only will not be affected by this change.
We realize that downtime is unavoidable but we want to make sure all of our customers are well informed and notified in advance of this change. If you have any questions or concerns please feel free to reach out to our Support Team at support @ favorhosting.com
We value all of our customers and want to make this transition as seamless as possible.
Sincerely,
The FavorHosting Team
UPDATE: As of Jan 15 at 4:47AM ET the web hosting migration is still under way. Some websites are coming up again, and some of the Databases are still not yet available. We will continue to provide updates.
UPDATE: As of Jan 15 at 6AM CT the migration has been complete. Please contact support if you see anything out of the ordinary. Thank you for your patience!
Possibly Related Posts: