Differences

This shows you the differences between two versions of the page.

--- administration:robots.txt [2015/12/14 19:15] – ↷ Page moved from robots.txt to administration:robots.txt demiankatz
+++ administration:robots.txt [2020/06/04 15:06] (current) – [More Information] demiankatz
@@ Line 12: / Line 12: @@
 ===== File Location =====
-The most important thing to know about robots.txt is that it must exist at the root of your server. If VuFind is running in the root of your server, this means you can simply create a robots.txt file in VuFind's web folder (either web/ in VuFind 1.x, or public/ in VuFind 2.x). If VuFind is running in a directory (the most common use case), you will need to place the robots.txt file in your Apache web root or manage it through your site's Content Management System (if applicable).
+The most important thing to know about robots.txt is that it must exist at the root of your server. If VuFind is running in the root of your server, this means you can simply create a robots.txt file in VuFind's web folder (public/ in VuFind 2.x or later). If VuFind is running in a directory (the most common use case), you will need to place the robots.txt file in your Apache web root or manage it through your site's Content Management System (if applicable).
 To summarize: the URL //must// be <nowiki>http://your-server/robots.txt</nowiki>. The file will not be found if it is <nowiki>http://your-server/vufind/robots.txt</nowiki>.
@@ Line 26: / Line 26: @@
 Disallow: /vufind/Browse
 Disallow: /vufind/Combined
+Disallow: /vufind/EDS
+Disallow: /vufind/EdsRecord
 Disallow: /vufind/Search/Results
 Disallow: /vufind/Summon
@@ Line 33: / Line 35: @@
 Disallow: /vufind/Browse/
 Disallow: /vufind/Combined/
+Disallow: /vufind/EDS/
+Disallow: /vufind/EdsRecord/
 Disallow: /vufind/Search/Results/
 Disallow: /vufind/Summon/
@@ Line 45: / Line 49: @@
 We've recently added the Browse module to avoid redundancy. We also disabled access to the Alphabrowse and Results pages in compliance with the Google crawling guidelines and to reduce server strain. We recommend providing a sitemap of all records to the bot to make sure each of your records is crawled. See here for more information: [[administration:search_engine_optimization|Search Engine Optimization]]
+===== More Information =====
+Google offers some [[https://support.google.com/webmasters/answer/7424835?hl=en|answers to frequently asked questions]]. This explains some of Google's crawling behavior in more detail, including some important information such as the fact that a robots.txt "Disallow" directive may be ignored based on other criteria, and the "[[https://support.google.com/webmasters/answer/93710?hl=en|noindex]]" robots meta tag may be a stronger way to hide content.
 ---- struct data ----
+properties.Page Owner :
 ----