About Features Downloads Getting Started Documentation Events Support GitHub

Love VuFind®? Consider becoming a financial supporter. Your support helps build a better VuFind®!

Site Tools


Warning: This page has not been updated in over over a year and may be outdated or deprecated.
administration:robots.txt

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
administration:robots.txt [2020/06/04 15:01] – [Example] demiankatzadministration:robots.txt [2020/06/04 15:06] (current) – [More Information] demiankatz
Line 49: Line 49:
  
 We've recently added the Browse module to avoid redundancy. We also disabled access to the Alphabrowse and Results pages in compliance with the Google crawling guidelines and to reduce server strain. We recommend providing a sitemap of all records to the bot to make sure each of your records is crawled. See here for more information: [[administration:search_engine_optimization|Search Engine Optimization]] We've recently added the Browse module to avoid redundancy. We also disabled access to the Alphabrowse and Results pages in compliance with the Google crawling guidelines and to reduce server strain. We recommend providing a sitemap of all records to the bot to make sure each of your records is crawled. See here for more information: [[administration:search_engine_optimization|Search Engine Optimization]]
 +
 +===== More Information =====
 +
 +Google offers some [[https://support.google.com/webmasters/answer/7424835?hl=en|answers to frequently asked questions]]. This explains some of Google's crawling behavior in more detail, including some important information such as the fact that a robots.txt "Disallow" directive may be ignored based on other criteria, and the "[[https://support.google.com/webmasters/answer/93710?hl=en|noindex]]" robots meta tag may be a stronger way to hide content.
 ---- struct data ---- ---- struct data ----
 +properties.Page Owner : 
 ---- ----
  
administration/robots.txt.txt · Last modified: 2020/06/04 15:06 by demiankatz