About Features Downloads Getting Started Documentation Events Support

Site Tools


administration:automation

Automation

Once VuFind is running, you still need to do some work to keep it up to date and stable. The exact details of VuFind automation will vary significantly based on your ILS and operating system. However, these are the general common goals:

  • Update VuFind's index with the latest changed and added records from your ILS. See the MARC Export Notes for help with this.
  • Remove deleted and suppressed records from VuFind's index. VuFind is packaged with tools in the util directory that can help with this, but functionality may be limited depending on the capabilities of your ILS.
  • You may wish to make sure the Solr index is regularly optimized for minimal space usage and improved performance, though this is not as important for newer versions of Solr as it used to be. However, optimization can be a good way to ensure that all uncommitted changes become visible. There is a stand-alone optimize tool in the util directory.
  • Dump the MySQL database for backup purposes.
  • Periodically restart Solr – if it runs for extremely long periods of time, like any complex piece of software, it may become unstable.

The remainder of this page shows specific solutions to these goals. Feel free to add your own ideas to this page if you wish. You may not be able to use these solutions exactly as-is, but hopefully the ideas included will help you get things running the way you want.

General Advice

Using cron

If you need to automate tasks under Linux, you will want to become familiar with the cron process. This allows you to schedule programs to run at specific intervals, either as root or as a particular user. The details of using cron are beyond the scope of this document, but there are many helpful resources available elsewhere. The Wikipedia page on the subject is probably a good starting point.

cron-friendly VuFind script

If you try to run the standard vufind.sh script to restart VuFind from within cron, you may run into problems. The context in which VuFind runs may be wrong, and the attempted output to the TTY may cause unexpected failures. If you have trouble, try adding this script (you can call it vufind_cron.sh) to your VuFind home directory:

#!/bin/sh
 
# Disable JETTY_CONSOLE output -- it causes problems when run by cron:
export JETTY_CONSOLE=/dev/null
 
# Pass parameters along to vufind.sh:
CURRENTPATH=`dirname $0`
cd $CURRENTPATH
$CURRENTPATH/vufind.sh $*

Important: Note that because of the way CURRENTPATH is determined, this script only works if it is in the same directory as vufind.sh!

Why restart VuFind?

The main reason to regularly restart the VuFind is to ensure system stability. Due to the way Java garbage collection works, Solr can eventually run out of memory and stop responding; periodic restarts are an easy way to avoid this problem. It is also a good idea to investigate Java tuning to address the root cause – but even with a well-tuned system, occasional restarts (such as, on a weekly basis) can help ensure long-term stability.

Using mysqldump

If you want to back up VuFind's MySQL database, the mysqldump tool that comes with MySQL itself makes this pretty simple. You can see the tool's man page for all the details. Here is a sample script for nightly backups via cron:

#!/bin/sh
WORKDIR=/usr/local/vufind/mysqldump
DATE=`date '+%y%m%d'`
MYSQLUSER=root
MYSQLPASS=password
VUFIND_DB=vufind
 
# Dump MySQL database to disk
/usr/bin/mysqldump -l --default-character-set=utf8 --password=$MYSQLPASS -u$MYSQLUSER $VUFIND_DB >$WORKDIR/vufind_mysql_dump.$DATE
 
# Compress the dump to save space
gzip -f $WORKDIR/vufind_mysql_dump.$DATE
exit

Important: The backup will cause your database to lock while it is running, so you should schedule it for a time of low activity.

Automation with Voyager and Linux

The following steps list a simplified version of the process used by Villanova to maintain its VuFind installation. These are intended as guidelines only, and you will want to customize the provided scripts for your own needs. In particular, better error checking is important so you can figure out when things go wrong. You may wish to add code to send emails when conditions are not met. You may wish to add “echo” statements at various points to output status messages and then redirect the output of the scripts to a log file for later analysis. None of this logic is included here since your needs may vary, and the point is to show the general processes needed by VuFind.

Also note that some of these cron jobs will put a heavy load on your VuFind server. You should schedule them to run at times of low usage if at all possible!

Updating VuFind's Index with the Latest MARC Records

Getting MARC records from Voyager into VuFind requires configuration on two servers – your Voyager server AND your VuFind server. The Voyager processes dump out the changed MARC records, while the VuFind processes read them in and update VuFind's index.

Voyager Configuration

You will want to set up this script to run regularly by cron:

#!/bin/sh
 
# Note: The path below will vary from system to system:
exporter=/m1/voyager/yourdb/sbin/Pmarcexport
file=catalog_INCR
PROG="$exporter -o $file -rB -mB -t"
YEST=`/usr/local/bin/gdate -d yesterday +%F`
TODAY=`/usr/local/bin/gdate +%F`
 
# Dump the latest MARC records
exec $PROG$YEST:$TODAY
 
# Wait a second for things to settle down
/usr/bin/sleep 1
 
# Move the file into position to be retrieved by the VuFind server
chmod 777 $file
mv $file /scratch/$file-`date +%Y%m%d`.mrc
 
# now exit gracefully...
exit 0

VuFind Configuration

The following script needs to run by cron on the VuFind server, obviously at a later time than the Voyager script runs so that there is sufficient time for the records to be exported. It should be run as the user that controls VuFind.

Note that you will need to customize some variables in the script to match your environment. You may also wish to add some error checks and e-mail alerts to administrators to detect unexpected failures in the process.

In order for this script to work, it is also necessary to set up SSH keys so that the VuFind server can complete the SFTP operation without being prompted for a password. (This article may help).

#!/bin/bash
 
# This is the directory on the Voyager server where MARC records may be found:
DIR="/scratch/"
 
# This is the filename of the MARC records on the Voyager server:
VOYRECS=catalog_INCR-`date +%Y%m%d`.mrc
 
# This is the directory on the VuFind server where MARC records will be downloaded:
LDIR="/tmp/marc_data"
 
# This is the user and server name for the Voyager server:
VOYUSER=vufind
VOYSERVER=voyager.myinstitution.edu
 
# Open sftp session and start the transfer
/usr/bin/sftp $VOYUSER@$VOYSERVER <<EOF>/dev/null
		cd $DIR
		lcd $LDIR
		mget $VOYRECS
		quit
 
EOF
# done
 
# Import the records to VuFind:
IMPORT_SCRIPT=import-marc.sh
$VUFIND_HOME/$IMPORT_SCRIPT $LDIR/$VOYRECS
 
# Remove suppressed records and optimize index:
cd $VUFIND_HOME/util
/usr/bin/php suppressed.php
 
exit

Windows Versions

The following VBScripts, courtesy of Amos Chung (achung AT molloy DOT edu), offer Windows-based functionality roughly equivalent to the shell scripts above.

Voyager Configuration (VuFind_Export.vbs)
Dim fso, VOYRECS
Set fso = WScript.CreateObject("Scripting.Filesystemobject")
Dim WshShell, oExec, ProgramFile,outFile,objFile
Dim VuFindHome, VuFindLocal, VuFindImporter
Dim WshProcEnv
Dim system_architecture
Dim process_architecture
Dim sDate, sDate2
sDate = CStr(Year(Date)) & "-" &CStr(Month(Date)) & "-" &CStr(Day(Date)) 
sDate2 = CStr(Year(DateAdd("D",-1,Date))) & "-" &CStr(Month(DateAdd("D",-1,Date))) & "-" &CStr(Day(DateAdd("D",-1,Date))) 
Set WshShell = CreateObject("WScript.Shell")
Set WshProcEnv = WshShell.Environment("Process")

VuFindHome = WshProcEnv("VUFIND_HOME")
VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR")
VuFindImporter = "import-marc.bat"

WshShell.CurrentDirectory = "C:\Voyager_Interface"

outFile = WshShell.CurrentDirectory + "\VuFindExp.txt"

Set objFile = fso.CreateTextFile(outFile,True)
VOYRECS = "catalog_INCR-" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".mrc"
objFile.Write "option batch on" & vbCrLf
objFile.Write "option confirm off" & vbCrLf
objFile.Write "open voyager:Hy6};(@YourCat.hosted.exlibrisgroup.com -hostkey=""YourSSHKey""" & vbCrLf
objFile.Write "cd /m1/voyager/YourDB/local" & vbCrLf
objFile.Write "rm catalog_INCR.mrc" & vbCrLf
objFile.Write "cd /m1/voyager/YourDB/sbin" & vbCrLf 
objFile.Write "option transfer binary" & vbCrLf
objFile.Write "call /m1/voyager/YourDB/sbin/Pmarcexport -o /m1/voyager/YourDB/local/catalog_INCR.mrc -rB -mB -t" & sDate2 & ":" & sDate & vbCrLf
'objFile.Write "lcd C:\Voyager_Interface" & vbCrLf
'objFile.Write "mget marc.exp.20131023.*" & vbCrLf
objFile.Write "close" & vbCrLf
objFile.Write "exit" & vbCrLf
objfile.Close
 
'Wscript.Echo WshShell.CurrentDirectory

process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE") 

If process_architecture = "x86" Then    
    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432")

    If system_architecture = ""  Then    
        system_architecture = "x86"
        ProgramFile = WshProcEnv("ProgramFiles")
    End if    
Else    
    system_architecture = process_architecture
    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindExp.txt /log=C:\Voyager_Interface\myvoyager_get.log"   
End If


Set oExec = WshShell.Exec(ProgramFile)

Do While oExec.Status = 0
     WScript.Sleep 100
Loop

'ProgramFile = "CMD /K " +  VuFindHome + "/" + VuFindImporter + " " +  WshShell.CurrentDirectory + "\" + VOYRECS + " > import-marc.log"
'Wscript.Echo ProgramFile > "bbb.txt"
'Set oExec = WshShell.Run(ProgramFile,0,true)

'Do While oExec.Status = 0
'     WScript.Sleep 100
'Loop
'Wscript.Echo VOYRECS, ProgramFile
'WScript.Echo "Running as a " & process_architecture & " process on a " _ 
'    & system_architecture & " system."
'WScript.Echo oExec.Status
WScript.Quit
VuFind Configuration (vufind-import.vbs)
Dim fso, VOYRECS
Set fso = WScript.CreateObject("Scripting.Filesystemobject")
Dim WshShell, oExec, ProgramFile,outFile,objFile
Dim VuFindHome, VuFindLocal, VuFindImporter
Dim WshProcEnv
Dim system_architecture
Dim process_architecture

Set WshShell = CreateObject("WScript.Shell")
Set WshProcEnv = WshShell.Environment("Process")

VuFindHome = WshProcEnv("VUFIND_HOME")
VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR")
VuFindImporter = "import-marc.bat"

WshShell.CurrentDirectory = "C:\Voyager_Interface"

outFile = WshShell.CurrentDirectory + "\VuFindGet.txt"

Set objFile = fso.CreateTextFile(outFile,True)
VOYRECS = "catalog_INCR-" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".mrc"
objFile.Write "option batch on" & vbCrLf
objFile.Write "option confirm off" & vbCrLf
objFile.Write "open voyager:Hy6};(@YourCat.hosted.exlibrisgroup.com -hostkey=""YourSSHKey""" & vbCrLf 
objFile.Write "option transfer binary" & vbCrLf
objFile.Write "cd /m1/voyager/YourDB/local" & vbCrLf
objFile.Write "lcd C:\Voyager_Interface" & vbCrLf
objFile.Write "get catalog_INCR.mrc " & VOYRECS  & vbCrLf
objFile.Write "close" & vbCrLf
objFile.Write "exit" & vbCrLf
objfile.Close

'Wscript.Echo WshShell.CurrentDirectory

process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE") 

If process_architecture = "x86" Then    
    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432")

    If system_architecture = ""  Then    
        system_architecture = "x86"
        ProgramFile = WshProcEnv("ProgramFiles")
    End if    
Else    
    system_architecture = process_architecture
    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindGet.txt /log=C:\Voyager_Interface\myvoyager_get.log"   
End If


Set oExec = WshShell.Exec(ProgramFile)

Do While oExec.Status = 0
     WScript.Sleep 100
Loop

ProgramFile = "cmd /C " + VuFindHome + "/" + VuFindImporter + " " +  WshShell.CurrentDirectory + "\" + VOYRECS + " > import-marc" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".log"
'Wscript.Echo ProgramFile > "bbb.txt"
Dim sStatus
sStatus = WshShell.Run(ProgramFile,1,true)

WScript.Quit
Script to Email Logs (sendLog.vbs)
strSMTPFrom = "no-reply@YourInstitution.edu"
strSMTPTo = "recipients@YourInstitution.edu,recipient2@YourInstitution.edu"
'strSMTPTo = "You@YourInstitution.edu"
strSMTPRelay = "mail.YourInstitution.edu"
strTextBody = "Please review VuFind MARC Import log."
strSubject = "VuFind MARC Import"
strAttachment = "C:\Voyager_Interface\import-marc" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".log"
dim filesys

set filesys=CreateObject("Scripting.FileSystemObject")

If filesys.FileExists(strAttachment) Then

filesys.CopyFile strAttachment , "C:\Voyager_Interface\import_mrc_log.txt"

strAttachment = "C:\Voyager_Interface\import_mrc_log.txt"
Set oMessage = CreateObject("CDO.Message")
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2 
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = strSMTPRelay
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25 
oMessage.Configuration.Fields.Update

oMessage.Subject = strSubject
oMessage.From = strSMTPFrom
oMessage.To = strSMTPTo
oMessage.TextBody = strTextBody
oMessage.AddAttachment strAttachment


oMessage.Send
End If
WScript.Quit

Removing Deleted and Suppressed Records

Suppressed records are actually already taken care of by the scripts above. Deleted records require a bit more scripting. Voyager keeps a MARC file of all deleted records. VuFind comes with a utility to read a MARC file and delete all listed records within it. We just need to automate downloading of the deleted record list and use of the appropriate VuFind utility. A sample script follows that can be run by cron on your VuFind server.

Notes:

  • As with the previous script, you need to have SSH keys set up so that SFTP can transfer files automatically.
  • You'll need to customize some variables to match your local setup.
  • When using VuFind 1.x, be sure you run this script a significant amount of time AFTER the update script listed above. In 1.x, both the update's suppressed cleanup and this script will optimize your Solr index. You don't want to have two optimize actions going on at the same time! If you are using VuFind 2.x, the delete script does not automatically optimize your index, giving you more control over the timing.
#!/bin/bash
 
# Directory and filename on the Voyager server containing your deleted records file:
DIR="/m1/voyager/yourdb/rpt"
FILE="deleted.bib.marc"
 
# Directory on your VuFind server to download the file:
LDIR="/tmp/marc_data"
 
# This is the user and server name for the Voyager server:
VOYUSER=vufind
VOYSERVER=voyager.myinstitution.edu
 
# Download the deleted records:
cd $LDIR
/usr/bin/sftp $VOYUSER@$VOYSERVER <<EOF>/dev/null
                cd $DIR
                get $FILE
                quit
EOF
 
# Remove deleted records from VuFind's index:
cd $VUFIND_HOME/util
/usr/bin/php deletes.php $LDIR/$FILE
 
exit

Windows version

The following VBScript, courtesy of Amos Chung (achung AT molloy DOT edu), offers Windows-based functionality roughly equivalent to the shell script above.

Vufind_Delete.vbs

Dim fso, VOYRECS
Set fso = WScript.CreateObject("Scripting.Filesystemobject")
Dim WshShell, oExec, ProgramFile,outFile,objFile
Dim VuFindHome, VuFindLocal, VuFindImporter
Dim WshProcEnv, MarcFile
Dim system_architecture
Dim process_architecture

Set WshShell = CreateObject("WScript.Shell")
Set WshProcEnv = WshShell.Environment("Process")

VuFindHome = WshProcEnv("VUFIND_HOME")
VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR")
VuFindImporter = "php.exe deletes.php"

WshShell.CurrentDirectory = VuFindHome + "\util"
VoyagerDir = "C:\Voyager_Interface"
outFile = VoyagerDir + "\VuFindGetBiB.txt"
logfile = "C:\Voyager_Interface\myvoyager_bib.log"

Set objFile = fso.CreateTextFile(outFile,True)
objFile.Write "option batch on" & vbCrLf
objFile.Write "option confirm off" & vbCrLf
objFile.Write "open voyager:Hy6};(@YOURSERVERHOSTNAME.exlibrisgroup.com -hostkey=""YOUR SSH HOST KEY""" & vbCrLf 
objFile.Write "option transfer binary" & vbCrLf
objFile.Write "cd /m1/voyager/YOURdb/rpt" & vbCrLf
objFile.Write "lcd C:\Voyager_Interface" & vbCrLf
DIM sMonth, sDay
Dim myDate, i
myDate = Dateadd("d",0,Date)
For i = 1 to 7
myDate = Dateadd("d",-1,myDate)

sMonth = Mid("00" + CStr(Month(myDate)), Len(CStr(Month(myDate))) + 1,2)
sDay = Mid("00" + CStr(Day(myDate)), Len(CStr(Day(myDate))) + 1,2)

VOYRECS = "deleted.bib-" + CStr(Year(Date)) + sMonth + sDay + ".mrc"

objFile.Write "mget deleted.bib.marc." & CStr(Year(Date)) & "-" & sMonth & "-" & sDay & "-* " & VOYRECS  & vbCrLf
'objFile.Write "get deleted.bib.marc " & VOYRECS  & vbCrLf
Next
objFile.Write "close" & vbCrLf
objFile.Write "exit" & vbCrLf
objfile.Close

process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE") 

If process_architecture = "x86" Then    
    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432")

    If system_architecture = ""  Then    
        system_architecture = "x86"
        ProgramFile = WshProcEnv("ProgramFiles")
    End if    
Else    
    system_architecture = process_architecture
    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindGetBib.txt /log=" + logfile   
End If

if (fso.FileExists(logfile)) Then
	fso.DeleteFile(logfile)
End If

Set oExec = WshShell.Exec(ProgramFile)

Do While oExec.Status = 0
     WScript.Sleep 500
Loop
Dim bOptimize
myDate = Dateadd("d",0,Date)
For i = 1 to 7
myDate = Dateadd("d",-1,myDate)
sMonth = Mid("00" + CStr(Month(myDate)), Len(CStr(Month(myDate))) + 1,2)
sDay = Mid("00" + CStr(Day(myDate)), Len(CStr(Day(myDate))) + 1,2)
VOYRECS = "deleted.bib-" + CStr(Year(Date)) + sMonth + sDay + ".mrc"

if (fso.FileExists(VoyagerDir + "\" + VOYRECS)) Then	

Set MarcFile = fso.GetFile(VoyagerDir + "\" + VOYRECS)

If MarcFile.Size > 0 Then
	bOptimize = 1
	ProgramFile = "CMD /c " + VuFindImporter + " " +  VoyagerDir + "\" + VOYRECS 
	Dim sStatus
	sStatus = WshShell.Run(ProgramFile,1,true)
	WScript.Sleep 500
End If
End If
Next
If bOptimize = 1 Then
	ProgramFile = "CMD /c php.exe optimize.php"
	sStatus = WshShell.Run(ProgramFile,1,true)
	WScript.Sleep 500

End If

WScript.Quit
administration/automation.txt · Last modified: 2016/05/13 15:39 by demiankatz