Differences

This shows you the differences between two versions of the page.

--- videos:administering_a_vufind_server [2021/11/16 15:19] – [Transcript] akilsdonk
+++ videos:administering_a_vufind_server [2023/04/26 13:34] (current) – [Transcript] crhallberg
@@ Line 1: / Line 1: @@
-====== Video 5: Administering a VuFind Server ======
+====== Video 5: Administering a VuFind® Server ======
-The fifth VuFind instructional video shows you some important server administration tasks to keep your VuFind site happy and healthy: making sure that Solr auto-starts on boot-up (using systemd), and configuring cleanup of old search and session data.
+The fifth VuFind® instructional video shows you some important server administration tasks to keep your VuFind site happy and healthy: making sure that Solr auto-starts on boot-up (using systemd), and configuring cleanup of old search and session data.
 Video is available as an [[https://vufind.org/video/Administering_VuFind.mp4|mp4 download]] or through [[https://www.youtube.com/watch?v=KsSomqmB19g&feature=youtu.be|YouTube]].
@@ Line 18: / Line 18: @@
 ===== Transcript =====
-// This is a raw machine-generated transcript; it will be cleaned up as time permits. //
+Welcome to the fifth VuFind video. This time around, we are going to talk about administering a VuFind server after you've built and configured it, since there are a few common tasks that it's useful to know about. We're going to cover both getting Solr set up in a secure way and making sure that it starts automatically when your server boots up. And we're going to talk about some cleanup you'll want to do to make sure that you don't accidentally fill up your disk without realizing it.
-Welcome to the fifth VuFind video
+So let's start with Solr setup. For the purposes of this video, I am going to show you how to set up VuFind autostart using systemd. Systemd is a set of tools shared across Linux distributions which are focused on system and service management. This is a comparatively new development in Linux land, which is to say it's been around for several years, but if you've been in Linux a long time, you may be more familiar with the earlier system that used symbolic links to a directory called init.d. Systemd gets rid of obscure bash scripts and replaces them with configuration files, and while it took a little getting used to, I've come to really like it, and so I'm going to go through it in some detail in this video to help you understand how it works and what it's doing for you.
-this time around we are going to talk
-about administering a VuFind server
+But first, let's talk about Solr security quickly. So when you install Solr, you're creating a web service that can have data sent to it and that VuFind communicates with to do searching. I hope it goes without saying, but it might not, that you do not want to expose your Solr index to the whole world because people can do malicious things to it. So you should always have Solr behind a firewall so that VuFind can talk to it, but nobody who doesn't need to has access. Additionally, it's a really good idea to create a user account dedicated to running Solr and give ownership of the Solr directories to that account, so that if somebody does somehow get to your Solr web interface and exploit a bug that allows them to do something malicious, their ability to do harm is somewhat constrained by file ownerships and so forth.
-after you've built and configured it
-since there are a few common tasks that
+So what we are going to do in this video is create a Solr user, give ownership of the Solr directories to that Solr user, and then set up systemd so that the Solr user starts up Solr when the server boots. So first of all, we'll just do the bare minimum to create a user. We will say ''sudo adduser solr''. So this creates a user named Solr, and we're going to set the disabled password switch (''--disabled-password'') because we don't need to have a password for this account. We're not going to be logging in as it.
-it's useful to know about we're going to
-cover both getting Solr setup in a
+And we're just going to accept all the defaults because this is a demo. So now that we have a user created called solr, we can give it ownership of our Solr directories, "-R" for recursive, "solr:solr" to give both user and group ownership of the directory in question, and we're going to say "$VUFIND_HOME/solr" (''sudo chown -R solr:solr $VUFIND_HOME/solr''). And now if I do an ''ls -l'' of the VuFind home Solr directory, I see that it is owned by Solr and in the Solr group.
-secure way and making sure that it
-starts automatically when your server
+Now we are all set to create a systemd service to boot up Solr web. There is a directory called ''/etc/systemd/system'', which is where service definitions live so I'm going to use my nano editor. So, ''sudo nano /etc/systemd/system/vufind.service''. Every service definition needs to end in dot service to tell systemd that it is a service. So I'm creating a blank file and I'm going to type in a whole bunch of parameters here to explain how the services supposed to work.
-boots up and we're going to talk about
-some cleanup you'll want to do to make
+I'm going to start with ''After=network.target''. The After command allows us to essentially define dependencies, so you can make services that wait for other services to start before they start and so forth, but in this instance we're just going to use network dot target which is a predefined setting in systemd, which means wait for there to be network online before you do anything Solr doesn't do much good without network access.
-sure that you don't accidentally fill up
-your disk without realizing it. So let's
+And I'm going to create a ''[Service]'' section where most of the main settings will live I'm going to say ''Type=forking''. This is used when you run a script that exits quickly but spawns a long lived child process, which is an accurate description of the Solr dot sh script that you find uses to start up Solr script returns, but it forks a process that lives until we stop it.
-start with Solr setup. For the purposes
-of this video I am going to show you how
+Next I'm going to say ''ExecStart=/bin/sh -l -c '/usr/local/vufind/solr.sh start' -x''. So, as you can probably guess from the setting name, ExecStart is where you specify the systemd what command you use to start the service. You'll notice that I use full paths to everything because we don't want to make any assumptions about the environment that's set up when systemd is running things. Just saying use the standard shell. The ''-l'' switch makes the shell act as if a user has logged into it when running a command which sets up the environment correctly so this gives us access to VUFIND_HOME, VUFIND_LOCAL_DIR, etc. Then the ''-c'' is just telling the shell what command to run: that quoted string we're running the Solr scripts to start Solr. Finally the ''-x'' just provides extra detailed outpuit from the shell, which can be useful for error logging and troubleshooting.
-to set up if you find auto start using
-systemd. Systemd is a set of tools
+Next I'm going to say ''PIDFile=/usr/local/vufind/solr/vendor/bin/solr-8080.pid''. This tells systemd where the file containing the process ID of the running Solr process lives. It's something that's set up by the ''solr.sh'' script when we start things up and it's useful for knowing whether or not the service is running and also understanding how to stop the process.
-shared across Linux distributions which
-are focused on system and service
+Next I'm going to say ''User=solr''. This is where we specify that the Solr user we created earlier will be used to run the Solr process.
-management this is a comparatively new
-development in Linux land which is to
+Next, ''ExecStop=/bin/sh -l -c '/usr/local/vufind/solr.sh stop' -x''. So as you can see this exactly mirrors the ExecStart command except this is the command used to stop Solr instead of to start it up.
-say it's been around for several years
-but if you've been in Linux a long time
+We also add ''SuccessExitStatus=0''. This tells systemd that the ExecStart command will return a return value of zero when it succeeds so if ''solr.sh'' comes back with some other exit status something is wrong and the process will throw an error.
-you'd be you may be more familiar with
-the earlier system that used symbolic
+Finally we add ''LimitNOFILE=65000'' and ''LimitNPROC=65000''. You may have noticed recently when you start up ''solr.sh'' from the command line it throws warnings about file and process limit settings if you don't allow Solr to have lots of files open and potentially cause performance issues and so it's recommended that you set these limits to these values when running solar and production and using systemd provides a really convenient way to set those settings and then not have to think about them anymore.
-links to a directory called init.d
-systemd gets rid of obscure bash
+Finally, we create an [Install] section containing ''WantedBy=multi-user.target''. This section tells systemd what circumstances it should start this process under when the process is enabled. ''multi-user.target'' means that the server is running and accepting logins but isn't necessarily presenting a graphical interface, so it's kind of a low threshold for system is up and running in a normal mode.
-scripts and replaces them with
-configuration files and well it took a
+If you use the older Linux startup system, within init.d, there were things called run levels. Those don't exist anymore. Instead there are these targets, but ''multi-user.target'' is a safe and appropriate option for this use case.
-little getting used to I've come to
-really like it and so I'm going to go
+So I've now saved this file, I'm going to exit out of here and hope I didn't make any typos.
-through it in some detail in this video
-to help you understand how it works and
+So just to show you I'm going to open up VuFind in a web browser, and try to do a search and it fails because Solr is not running. But now that I've defined a service, I can start Solr up using the standard system CTL command which systemd uses to start and stop services.
-what it's doing for you but first let's
-talk about Solr security quickly so
+I just do ''sudo systemctl start vufind'' because I named my file ''vufind.service''.
-when you install Solr you're creating a
-web service that can have data sent to
+I wait a moment while it spins up. Now I'm back at the prompt so it appears to have succeeded let's refresh our browser. And now we have search results it worked. So if I wanted to stop or restart the service, I could do ''sudo systemctl stop vufind'' which stops it. And if it were running I could just substitute restart for stop to stop and then start it.
-it and that you find communicates with
-to do searching I hope it goes without
+What we are really concerned about here is ensuring that Solr starts every time our server boots up so that we don't have to remember to start it by hand and if something happens in the middle of the night, it just recovers on its own. So that's easily done. We just say ''sudo systemctl enable vufind''. And now the system has enabled the service based on that wanted by setting we put in the service file it knows that when it's enabled it needs to start up whenever the system is running in multi user mode and accepting connections.
-saying but it might not that you do not
-want to expose your Solr index to the
+So let's prove that this works and reboot the server. All right. So now I'm going to log in, and now if everything worked, I should be able to open up a web browser and access my VuFind instance and do searches without having to manually start anything. So here's VuFind. Do a blank search. Results. We are successful. Now that we've covered getting Solr to start automatically, let's also talk a little bit about cleaning up, because VuFind potentially has a lot of users accessing it, and some of the activity that users perform creates traces that can over time accumulate into quite a bit of data. So first of all, search history. Every time anybody does a search, it creates a row in a database in MySQL or whatever database platform you're using called search. The reason for the search table is that it allows us to maintain a search history. So if I go here and look, I can see that I did a blank search. Every user has a search history maintained in the search table. Also, there's the save button here, so users can potentially save their searches so that they can refer back to them in future. Also, recent releases of VuFind have a notification feature that when enabled lets people subscribe to searches and get e-mails indicating what new results have showed up in those sets. So it's useful to have this database, but of course, the vast majority of searches that get entered in the database are just forgotten about and never referred to again, and if you have people doing thousands or millions of searches in your system, this database table can get really big. Fortunately, VuFind has a command line utility called expire searches, which will clean out the table.
-whole world because people can do
-malicious things to it so you should
+So just to demonstrate, if I cd into my VuFind home directory and run php public index.php util expire searches, in this example, it deleted 70 old searches from all of the times I've done searching in past videos, and you can see that if I manage to create 70 searches just in the process of recording these videos, you can end up with a lot of these things if you have search engines crawling you and or a large user base. So I strongly encourage you to set up a cron job that regularly runs this expire searches task. Otherwise, you can find that your MySQL database gets incredibly enormous. You'll also learn if you find yourself in that situation that while MySQL can grow, it can never shrink. Once a MySQL database gets really, really big, even if you clear data out of it, it doesn't get smaller again. It just reuses the already claimed space. If you really need to reclaim disk space from an out-of-control MySQL database, the best thing to do is to dump the whole database with MySQL dump, then drop the database and reimport it, which will clean up all the disk space and make a nice, new, small, optimized file for you. So just a heads-up. This might also be a good time to point out that VuFind has a whole bunch of command line tools for you, and if you just run the public index.php script from the command line, you will get a list of all of them. So there are a few different kinds of expiration tasks and all sorts of odds and ends, some of which we will go into more detail in on future videos, but just be aware these exist. They might come in handy. So getting back to the subject of cleaning up after ourselves, there's one other thing that can potentially take up a lot of space, and that is user sessions. So the way that PHP and really any web-based system allows users to have a persistent state within the system, such as being logged in or tracking a partially completed workflow, is to store some data on the server called a session. So PHP sends a session cookie to the user, which gives them a unique identifier that's tied to a session file on the server, and every time the user comes in with that cookie, PHP loads that session data and then can use it to see who is being interacted with and what they're currently in the process of doing. VuFind doesn't use the session too heavily most of the time, but there are certainly places where it's important, such as enabling you to log in and stay logged in, or tracking what page to redirect you to after you've completed a login process. VuFind has a configuration setting that controls how user session data is stored, because there are actually several options.
-always have Solr behind a firewall so
-that VuFind can talk to it but nobody
+The default is to use PHP's built-in disk-based session handling, where it just sticks files in a directory, but you can also set it up to use a database table or to use different kinds of memory-based stores like Redis or Memcached. Depending on what option you choose, you may have different maintenance issues to deal with.
-who doesn't need to has access.
-Additionally it's a really good idea to
+With the default disk-based sessions, normally PHP should clean up after itself, and you shouldn't have to worry about it, but I have experienced situations where things have not quite gone as planned, and session files have accumulated faster than desired. For example, if you have such a heavy load that there are too many files in the session directory for PHP to handle, it might stop cleaning up after itself. So this is something you may want to monitor on your server, perhaps with a cron job that cleans out files past a certain age in the directory used for holding sessions.
-create a user account dedicated to
-running Solr and give ownership of the
+If you use the database-based session storage, there's an expire sessions command line utility, which you can see listed right here, which cleans up the session table in the database. And just to show you where these settings live, if you look in your config.ini file, here in local slash config slash view find slash config.ini, there is a section in the file called session, which I'm going to search for, and as you can see, you can set the type, which here defaults to file, but other options include memcache and database. You can set the lifetime of the session, which defaults to an hour, so in theory, these things should be cleaned up after an hour if a user stops being active. You can encrypt the session data if you're worried about anything sensitive in there, and then there are a number of settings that are specific to different session handlers, so for example, if you're using files, you can specify a non-default save path for the directory where the sessions live. If you're using memcache, you can specify how to connect to the memcache server, etc.
-Solr directories
-to that account so that if somebody does
+So that's all I have for today. I hope that's helpful. There are certainly other issues to think about when administering a server, and there are some wiki pages that talk about this in more detail, but if you can get Solr to start and you can avoid filling up your disk, you are well on your way to having a happy and healthy VuFind server. More next month.
-somehow get to your Solr web interface
-and exploit a bug that allows them to do
+//This is an edited version of an automated transcript. Apologies for any errors.//
-something malicious of their ability to
-do harm is somewhat constrained by file
-ownerships and so forth. So what we are
-going to do in this video is create a
-Solr user give ownership of the Solr
-directories to that Solr user and then
-set up systemd so that the Solr user
-starts up Solr when the server boots so
-first of all we'll just do the bare
-minimum to create a user we will say
-sudo add user solr so this creates a
-user named solr and we're going to set
-the disabled password switch because we
-don't need to have a password for this
-account we're not going to be logging in
-as it and we're just going to accept all
-the defaults because this is a demo so
-now that we have a user created called
-solr we can give it ownership of our
-solr directories sudo CH own minus
-capital R for recursive Solr kolja :
-Solr to give both user and group
-ownership of the directory in question
-and we're going to say you find home
-slash solar and now if I do an LS - L of
-the VuFind home Solr directory I see
-that it is owned by solar and in the
-solar group now we are all set to create
-a systemd service to boot up Solr with
-there is a directory called Etsy slash
-system D slash system which is where
-service definitions live so I'm going to
-use my Nano editor so sudo nano slash
-Etsy system D slash system slash you
-find that service every service
-definition needs to end in that service
-to tell system view that
-is a service so I'm creating a blank
-file and I'm going to type in a whole
-bunch of parameters here to explain how
-the service is supposed to work. I'm
-going to start with after equals network
-dot target the after command allows us
-to essentially define dependencies so
-you can make services that wait for
-other services to start before they
-start and so forth
-but in this instance we're just going to
-use network target which is a predefined
-setting in systemd which means wait
-further to be networking online before
-you do anything Solr doesn't do much
-good without network access then I'm
-going to create a service section where
-most of the main settings will live I'm
-going to say type equals forking this is
-used when you run a script that exits
-quickly but spawns a long-lived child
-process which is a an accurate
-description of the solr SH script that
-you find uses to start up solr script
-returns but it forks a process that
-lives until we stop it next I'm going to
-say exact start equal slash bin slash s
-H minus L minus C and then in single
-quotes a slash user slash local slash Q
-fine slash solar dot s H starch and
-single quote minus X so as you can
-probably guess from the setting name
-exact start is where you specify the
-systemd what command you use to start
-the service you'll notice that I use
-full paths to everything because we
-don't want to make any assumptions about
-the environment that's set up when
-systemd is running things so we're just
-saying use the standard shell the minus
-else which makes the shell act as if a
-user has logged into it when running a
-command which sets up the environment
-correctly so this gives us access to if
-you find home if you find local dir etc
-then the minus C is just telling the
-shell what command to run so that quoted
-string we're running the Solar scripts
-to start solar and finally the minus X
-just provides extra detailed output from
-the shell which can be useful for error
-logging and troubleshooting next I'm
-going to say Pig file equals user local
-few fine solar vendor bin solar - 88 e
-dot Pig this tells system D where the
-file containing the process ID of the
-running solar process lives and this is
-something that's set up by the solar dot
-s H script when we start things up and
-it's useful for knowing whether or not
-the service is running and also
-understanding how to stop the process
-next I'm going to say user equals solar
-this is where we specify that the solar
-user we created earlier will be used to
-run the solar process next exact stop
-equals slash bin slash s H minus L minus
-C user local view fine solar SH stop all
-in single quotes and minus X so as you
-can see this exactly mirrors the exact
-start command except this is the command
-used to stop solar instead of to start
-it up we add success exit status equals
- this tells system D that the exact
-start command will return a return value
-of zero when it succeeds
-so if solar dive SH comes back with some
-other exit status something is wrong and
-the process will throw an error
-finally limit n Oh file equals 65,000
-and limit in proc equals 65,000 you may
-have noticed recently when you start up
-solar data SH from the command line
-it throws warnings about file
-and process limit settings if you don't
-allow a solar to have lots of files open
-it can potentially cause performance
-issues and so it's recommended that you
-set these limits to these values when
-running solar in production and using
-system D provides a really convenient
-way to set those settings and then not
-have to think about them anymore finally
-we create an install section containing
-wanted by equals multi - user dot target
-this section tells system D what
-circumstances it should start this
-process under when the process is
-enabled and multi-user target means that
-the server is running and accepting
-logins but isn't necessarily presenting
-a graphical interface so it's kind of a
-low threshold for system is up and
-running in a normal mode if you use the
-older Linux startup system with the nib
-D there were things called run levels
-those don't exist anymore
-instead there are these targets but
-multi-user target is a safe and
-appropriate option for this use case so
-I've now saved this file I'm going to
-exit out of here and hope I didn't make
-any typos so just to show you I'm going
-to open up if you find in a web browser
-and try to do a search and it fails
-because solar is not running but now
-that I've defined a service I can start
-solar up using the standard systemctl
-command which systemd uses to start and
-stop services so I just say sudo
-systemctl start view fine because I
-named my file view find dot service and
-I wait a moment well it spins up
-and now I'm back
-prompt so it appears to have succeeded
-let's refresh our browser and now we
-have search results it worked so if I
-wanted to stop or restart the service I
-could do sudo systemctl stop view find
-which stops it and if it were running I
-could just substitute restart for stop
-to stop and then start it but what we
-are really concerned about here is
-ensuring that solar starts every time
-our server boots up so that we don't
-have to remember to start it by hand and
-if something happens in the middle of
-the night it just recovers on its own so
-that's easily done we just say sudo
-systemctl enable view find and now the
-system has enabled the service based on
-that wanted by setting we put in the
-service file it knows that when it's
-enabled it needs to start up whenever
-the system is running in multi-user mode
-and accepting connections so let's prove
-that this works and reboot the server
-all right so now I'm going to login and
-now if everything worked I should be
-able to open up a web browser and access
-my view find instance and do searches
-without having to manually start
-anything
-so here's you find do a blank search
-results we are successful now that we've
-covered getting solar to start
-automatically let's also talk a little
-bit about cleaning up because view find
-potentially has a lot of users accessing
-it and some of the activity that users
-perform creates traces that can over
-time accumulate into quite a bit of data
-so first of all search history every
-time anybody does a search it creates a
-row in a database in my sequel or
-whatever database platform you're using
-called search the reason for the search
-table is that it allows us to maintain a
-search history so if I go here and look
-I can see that I did a blank search
-every user has a search history
-maintained in the search table also
-there's the Save button here so users
-can potentially save their searches so
-that they can refer back to them in
-future also recent releases of you find
-have a notification feature that when
-enabled lets people subscribe to
-searches and get emails indicating what
-new results have showed up in those sets
-so it's useful to have this database but
-of course the vast majority of searches
-that get entered in the database are
-just forgotten about and never referred
-to again and if you have people doing
-thousands or millions of searches in
-your system this database table can get
-really big fortunately view find has a
-command-line utility called expire
-searches which will clean out the table
-so just to demonstrate if I CD into my
-view find home directory and run PHP
-public index dot PHP util expire
-searches in this example it deleted 70
-old searches from all of the time
-I've done searching in past videos and
-you can see that if I manage to create
-searches just in the process of
-recording these videos you can end up
-with a lot of these things if you have
-search engines crawling you and/or a
-large user base so I strongly encourage
-you to set up a cron job that regularly
-runs this expire searches tasks
-otherwise you can find that your my
-sequel database gets incredibly enormous
-you'll also learn if you find yourself
-in that situation that well my sequel
-can grow it can never shrink once a my
-sequel database gets really really big
-even if you clear data out of it it
-doesn't get smaller again it just reuses
-the already claimed space if you really
-need to reclaim disk space from an
-out-of-control my sequel database the
-best thing to do is to dump the whole
-database with my sequel dump then drop
-the database and re-import it which will
-clean up all the disk space and make a
-nice new small optimized file for you so
-just a heads up this might also be a
-good time to point out that if you find
-has a whole bunch of command line tools
-for you and if you just run the public
-index dot PHP script from the command
-line you will get a list of all of them
-so there are a few different kinds of
-exploration tasks and all sorts of odds
-and ends some of which we will go into
-more detail in on future videos but just
-be aware these exist they might come in
-handy
-so getting back to the subject of
-cleaning up after ourselves there's one
-other thing that can potentially take up
-a lot of space and that is user sessions
-so the way that PHP and really any
-web-based system allows users to have a
-persistent state within the system such
-as being logged in or tracking a
-partially completed workflow is to store
-some
-data on the server called a session so
-PHP sends a session cookie to the user
-which gives them a unique identifier
-that's tied to a session file on the
-server and every time the user comes in
-with that cookie PHP loads that session
-data and then can use it to see who is
-being interacted with and what they're
-currently in the process of doing whew
-fine doesn't use the session too heavily
-most of the time but there are certainly
-places where it's important such as
-enabling you to log in and stay logged
-in or tracking what page to redirect you
-to after you've completed a login
-process view find has a configuration
-setting that controls how user session
-data is stored because there are
-actually several options the default is
-to use PHP s built-in disk based session
-handling where it just sticks files in a
-directory but you can also set it up to
-use a database table or to use different
-kinds of memory based stores like Redis
-or memcache D depending on what option
-you choose you may have different
-maintenance issues to deal with
-with the default disk based sessions
-normally PHP should clean up after
-itself and you shouldn't have to worry
-about it
-but I have experienced situations where
-things have not quite gone as planned
-and session files have accumulated
-faster than desired for example if you
-have such a heavy load that there are
-too many files in the session directory
-for PHP to handle it might stop cleaning
-up after itself so this is something you
-may want to monitor on your server
-perhaps with a cron job that cleans out
-files past a certain age in the
-directory used for holding sessions if
-you use the database based session
-storage there's an expire sessions
-command-line utility which you can see
-listed right here which cleans up the
-table in the database and just to show
-you where these settings live if you
-look in your config dot ini file here in
-local slash config slash you find slash
-config dot ini there is a section in the
-file called session which I'm going to
-search for and as you can see you can
-set the type which here defaults to file
-that other options include memcache and
-database you can set the lifetime of the
-session which defaults to an hour so in
-theory these things should be cleaned up
-after an hour if a user stops being
-active you can encrypt the session data
-if you're worried about anything
-sensitive in there and then there are a
-number of settings that are specific to
-different session handlers so for
-example if you're using files you can
-specify a non-default save path from the
-directory where the sessions live if
-you're using memcache you can specify
-how to connect the memcache server etc
-so that's all I have for today I hope
-that's helpful
-there are certainly other issues to
-think about when administering a server
-and there are some wiki pages that talk
-about this in more detail but if you can
-get solar to start and you can avoid
-filling up your disk you are well on
-your way to having a happy and healthy
-view fine server more next month
 ---- struct data ----
+properties.Page Owner :
 ----