Differences

This shows you the differences between two versions of the page.

--- administration:search_engine_optimization [2020/06/04 15:17] – [Best Practices] demiankatz
+++ administration:search_engine_optimization [2021/04/16 10:51] – [Sitemaps] emaijala
@@ Line 7: / Line 7: @@
 // This feature was added after the release of VuFind 1.0.1. //
-Several search engines look for sitemap files which list all of the pages of your site and ensure that no content is missed during indexing.  For example, sitemaps are useful in combination with [[http://www.google.com/webmasters/tools/|Google Webmaster Tools]].
+Several search engines look for sitemap files which list all of the pages of your site and ensure that no content is missed during indexing.  For example, sitemaps are useful in combination with [[https://search.google.com/search-console|Google Search Console]].
-VuFind comes with a simple tool for generating sitemaps.  To use it, follow these steps:
+VuFind comes with a tool for generating sitemaps.  To use it, follow these steps:
-. Edit the [[configuration:files:sitemap.ini]] file under your VuFind installation to specify the location of the generated sitemaps and some other settings.  All options are explained by comments within the file.
+. Edit the [[configuration:files:sitemap.ini]] file under your VuFind local directory to specify the location of the generated sitemaps, what to include and some other settings.  All options are explained by comments within the file.
-. Switch to the util folder under your VuFind installation, and run:
+. Run the sitemap generator:
+:!: This is an example command. See also [[administration:command_line_utilities|Command Line Utilities]] for more information on local modules etc.
 <code>
-php sitemap.php
+cd $VUFIND_HOME
+VUFIND_LOCAL_DIR=`pwd`/local php public/index.php util/sitemap -v [additional parameters]
 </code>
-This may take some time.  When it is complete, your sitemap file(s) will be generated.
+This may take some time.  When it is complete, your sitemap file(s) have been generated.
 You may wish to automate this so that sitemaps are built on a regular basis; see the [[administration:automation|Automation]] page for tips on automating VuFind-related tasks.
+Since the sitemap generation takes some time, it is recommended to generate the sitemap files in a temporary directory and move them in place only when completed. Otherwise the robots may try to crawl incomplete sitemap information while the generation is still in progress. Make sure to clean up any old sitemap files from the temporary directory before generating new ones, and from the public directory when copying them over.
+. Include a reference to the sitemap in [[administration:robots.txt|robots.txt]]:
+:!: This is an incomplete excerpt to illustrate the Sitemap option. The Sitemap URL must be fully-qualified.
+<code>
+User-agent: *
+Disallow: /AJAX
+Sitemap: http://www.example.com/sitemap.xml
+</code>
 ==== Best Practices ====
@@ Line 28: / Line 44: @@
 If you have important static content pages (landing pages, texts about services/institutions, etc.), you may wish to include these in a base sitemap as a complement to the record list build by VuFind's sitemap generator. If your VuFind instance serves content in multiple languages, you may also wish to take advantage of the ?lng= GET parameter to provide multiple language-specific versions of the link in the sitemap. See [[https://relbib.de/sitemaps/relbib/baseSitemap.xml|RelBib's baseSitemap.xml]] for an example.
+=== Understand Crawling Budgets ===
+A VuFind site can easily have hundreds of thousands, or even millions, of pages in its sitemaps -- it's all dependent on the number of records in your index. Search engines will take a significant amount of time to crawl all of these pages, and even more time to detect changes. Be aware that publishing a sitemap will not instantly lead to full visibility of all of your content.
 === Using Search Engine Tools ===