VuFind® / Voyager Automation

Villanova migrated from Voyager to FOLIO in 2023, and this documentation will no longer be maintained (unless another institution chooses to adopt it).

The following steps list a simplified version of the process used by Villanova to maintain its VuFind® installation. These are intended as guidelines only, and you will want to customize the provided scripts for your own needs. In particular, better error checking is important so you can figure out when things go wrong. You may wish to add code to send emails when conditions are not met. You may wish to add “echo” statements at various points to output status messages and then redirect the output of the scripts to a log file for later analysis. None of this logic is included here since your needs may vary, and the point is to show the general processes needed by VuFind®.

Also note that some of these cron jobs will put a heavy load on your VuFind® server. You should schedule them to run at times of low usage if at all possible!

Updating VuFind®'s Index with the Latest MARC Records

Getting MARC records from Voyager into VuFind® requires configuration on two servers – your Voyager server AND your VuFind® server. The Voyager processes dump out the changed MARC records, while the VuFind® processes read them in and update VuFind®'s index.

Voyager Configuration

You will want to set up this script to run regularly by cron:

#!/bin/sh
 
# Note: The path below will vary from system to system:
exporter=/m1/voyager/yourdb/sbin/Pmarcexport
file=catalog_INCR
PROG="$exporter -o $file -rB -mB -t"
YEST=`/usr/local/bin/gdate -d yesterday +%F`
TODAY=`/usr/local/bin/gdate +%F`
 
# Dump the latest MARC records
exec $PROG$YEST:$TODAY
 
# Wait a second for things to settle down
/usr/bin/sleep 1
 
# Move the file into position to be retrieved by the VuFind® server
chmod 777 $file
mv $file /scratch/$file-`date +%Y%m%d`.mrc
 
# now exit gracefully...
exit 0

VuFind® Configuration

The following script needs to run by cron on the VuFind® server, obviously at a later time than the Voyager script runs so that there is sufficient time for the records to be exported. It should be run as the user that controls VuFind®.

Note that you will need to customize some variables in the script to match your environment. You may also wish to add some error checks and e-mail alerts to administrators to detect unexpected failures in the process.

In order for this script to work, it is also necessary to set up SSH keys so that the VuFind® server can complete the SFTP operation without being prompted for a password. (This article may help).

#!/bin/bash
 
# This is the directory on the Voyager server where MARC records may be found:
DIR="/scratch/"
 
# This is the filename of the MARC records on the Voyager server:
VOYRECS=catalog_INCR-`date +%Y%m%d`.mrc
 
# This is the directory on the VuFind® server where MARC records will be downloaded:
LDIR="/tmp/marc_data"
 
# This is the user and server name for the Voyager server:
VOYUSER=vufind
VOYSERVER=voyager.myinstitution.edu
 
# Open sftp session and start the transfer
/usr/bin/sftp $VOYUSER@$VOYSERVER <<EOF>/dev/null
		cd $DIR
		lcd $LDIR
		mget $VOYRECS
		quit
 
EOF
# done
 
# Import the records to VuFind®:
IMPORT_SCRIPT=import-marc.sh
$VUFIND_HOME/$IMPORT_SCRIPT $LDIR/$VOYRECS
 
# Remove suppressed records and optimize index:
cd $VUFIND_HOME/util
/usr/bin/php suppressed.php
 
exit

Windows Versions

The following VBScripts, courtesy of Amos Chung (achung AT molloy DOT edu), offer Windows-based functionality roughly equivalent to the shell scripts above.

Voyager Configuration (VuFind_Export.vbs)

Dim fso, VOYRECS
Set fso = WScript.CreateObject("Scripting.Filesystemobject")
Dim WshShell, oExec, ProgramFile,outFile,objFile
Dim VuFindHome, VuFindLocal, VuFindImporter
Dim WshProcEnv
Dim system_architecture
Dim process_architecture
Dim sDate, sDate2
sDate = CStr(Year(Date)) & "-" &CStr(Month(Date)) & "-" &CStr(Day(Date)) 
sDate2 = CStr(Year(DateAdd("D",-1,Date))) & "-" &CStr(Month(DateAdd("D",-1,Date))) & "-" &CStr(Day(DateAdd("D",-1,Date))) 
Set WshShell = CreateObject("WScript.Shell")
Set WshProcEnv = WshShell.Environment("Process")

VuFindHome = WshProcEnv("VUFIND_HOME")
VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR")
VuFindImporter = "import-marc.bat"

WshShell.CurrentDirectory = "C:\Voyager_Interface"

outFile = WshShell.CurrentDirectory + "\VuFindExp.txt"

Set objFile = fso.CreateTextFile(outFile,True)
VOYRECS = "catalog_INCR-" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".mrc"
objFile.Write "option batch on" & vbCrLf
objFile.Write "option confirm off" & vbCrLf
objFile.Write "open voyager:Hy6};(@YourCat.hosted.exlibrisgroup.com -hostkey=""YourSSHKey""" & vbCrLf
objFile.Write "cd /m1/voyager/YourDB/local" & vbCrLf
objFile.Write "rm catalog_INCR.mrc" & vbCrLf
objFile.Write "cd /m1/voyager/YourDB/sbin" & vbCrLf 
objFile.Write "option transfer binary" & vbCrLf
objFile.Write "call /m1/voyager/YourDB/sbin/Pmarcexport -o /m1/voyager/YourDB/local/catalog_INCR.mrc -rB -mB -t" & sDate2 & ":" & sDate & vbCrLf
'objFile.Write "lcd C:\Voyager_Interface" & vbCrLf
'objFile.Write "mget marc.exp.20131023.*" & vbCrLf
objFile.Write "close" & vbCrLf
objFile.Write "exit" & vbCrLf
objfile.Close
 
'Wscript.Echo WshShell.CurrentDirectory

process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE") 

If process_architecture = "x86" Then    
    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432")

    If system_architecture = ""  Then    
        system_architecture = "x86"
        ProgramFile = WshProcEnv("ProgramFiles")
    End if    
Else    
    system_architecture = process_architecture
    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindExp.txt /log=C:\Voyager_Interface\myvoyager_get.log"   
End If


Set oExec = WshShell.Exec(ProgramFile)

Do While oExec.Status = 0
     WScript.Sleep 100
Loop

'ProgramFile = "CMD /K " +  VuFindHome + "/" + VuFindImporter + " " +  WshShell.CurrentDirectory + "\" + VOYRECS + " > import-marc.log"
'Wscript.Echo ProgramFile > "bbb.txt"
'Set oExec = WshShell.Run(ProgramFile,0,true)

'Do While oExec.Status = 0
'     WScript.Sleep 100
'Loop
'Wscript.Echo VOYRECS, ProgramFile
'WScript.Echo "Running as a " & process_architecture & " process on a " _ 
'    & system_architecture & " system."
'WScript.Echo oExec.Status
WScript.Quit

VuFind® Configuration (vufind-import.vbs)

Dim fso, VOYRECS
Set fso = WScript.CreateObject("Scripting.Filesystemobject")
Dim WshShell, oExec, ProgramFile,outFile,objFile
Dim VuFindHome, VuFindLocal, VuFindImporter
Dim WshProcEnv
Dim system_architecture
Dim process_architecture

Set WshShell = CreateObject("WScript.Shell")
Set WshProcEnv = WshShell.Environment("Process")

VuFindHome = WshProcEnv("VUFIND_HOME")
VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR")
VuFindImporter = "import-marc.bat"

WshShell.CurrentDirectory = "C:\Voyager_Interface"

outFile = WshShell.CurrentDirectory + "\VuFindGet.txt"

Set objFile = fso.CreateTextFile(outFile,True)
VOYRECS = "catalog_INCR-" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".mrc"
objFile.Write "option batch on" & vbCrLf
objFile.Write "option confirm off" & vbCrLf
objFile.Write "open voyager:Hy6};(@YourCat.hosted.exlibrisgroup.com -hostkey=""YourSSHKey""" & vbCrLf 
objFile.Write "option transfer binary" & vbCrLf
objFile.Write "cd /m1/voyager/YourDB/local" & vbCrLf
objFile.Write "lcd C:\Voyager_Interface" & vbCrLf
objFile.Write "get catalog_INCR.mrc " & VOYRECS  & vbCrLf
objFile.Write "close" & vbCrLf
objFile.Write "exit" & vbCrLf
objfile.Close

'Wscript.Echo WshShell.CurrentDirectory

process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE") 

If process_architecture = "x86" Then    
    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432")

    If system_architecture = ""  Then    
        system_architecture = "x86"
        ProgramFile = WshProcEnv("ProgramFiles")
    End if    
Else    
    system_architecture = process_architecture
    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindGet.txt /log=C:\Voyager_Interface\myvoyager_get.log"   
End If


Set oExec = WshShell.Exec(ProgramFile)

Do While oExec.Status = 0
     WScript.Sleep 100
Loop

ProgramFile = "cmd /C " + VuFindHome + "/" + VuFindImporter + " " +  WshShell.CurrentDirectory + "\" + VOYRECS + " > import-marc" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".log"
'Wscript.Echo ProgramFile > "bbb.txt"
Dim sStatus
sStatus = WshShell.Run(ProgramFile,1,true)

WScript.Quit

Script to Email Logs (sendLog.vbs)

strSMTPFrom = "no-reply@YourInstitution.edu"
strSMTPTo = "recipients@YourInstitution.edu,recipient2@YourInstitution.edu"
'strSMTPTo = "You@YourInstitution.edu"
strSMTPRelay = "mail.YourInstitution.edu"
strTextBody = "Please review VuFind MARC Import log."
strSubject = "VuFind MARC Import"
strAttachment = "C:\Voyager_Interface\import-marc" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".log"
dim filesys

set filesys=CreateObject("Scripting.FileSystemObject")

If filesys.FileExists(strAttachment) Then

filesys.CopyFile strAttachment , "C:\Voyager_Interface\import_mrc_log.txt"

strAttachment = "C:\Voyager_Interface\import_mrc_log.txt"
Set oMessage = CreateObject("CDO.Message")
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2 
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = strSMTPRelay
oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25 
oMessage.Configuration.Fields.Update

oMessage.Subject = strSubject
oMessage.From = strSMTPFrom
oMessage.To = strSMTPTo
oMessage.TextBody = strTextBody
oMessage.AddAttachment strAttachment


oMessage.Send
End If
WScript.Quit

Removing Deleted and Suppressed Records

Suppressed records are actually already taken care of by the scripts above. Deleted records require a bit more scripting. Voyager keeps a MARC file of all deleted records. VuFind® comes with a utility to read a MARC file and delete all listed records within it. We just need to automate downloading of the deleted record list and use of the appropriate VuFind® utility. A sample script follows that can be run by cron on your VuFind® server.

Notes:

As with the previous script, you need to have SSH keys set up so that SFTP can transfer files automatically.
You'll need to customize some variables to match your local setup.

#!/bin/bash
 
# Directory and filename on the Voyager server containing your deleted records file:
DIR="/m1/voyager/yourdb/rpt"
FILE="deleted.bib.marc"
 
# Directory on your VuFind® server to download the file:
LDIR="/tmp/marc_data"
 
# This is the user and server name for the Voyager server:
VOYUSER=vufind
VOYSERVER=voyager.myinstitution.edu
 
# Download the deleted records:
cd $LDIR
/usr/bin/sftp $VOYUSER@$VOYSERVER <<EOF>/dev/null
                cd $DIR
                get $FILE
                quit
EOF
 
# Remove deleted records from VuFind®'s index:
cd $VUFIND_HOME/util
/usr/bin/php deletes.php $LDIR/$FILE
 
exit

Windows version

The following VBScript, courtesy of Amos Chung (achung AT molloy DOT edu), offers Windows-based functionality roughly equivalent to the shell script above.

Vufind_Delete.vbs

Dim fso, VOYRECS
Set fso = WScript.CreateObject("Scripting.Filesystemobject")
Dim WshShell, oExec, ProgramFile,outFile,objFile
Dim VuFindHome, VuFindLocal, VuFindImporter
Dim WshProcEnv, MarcFile
Dim system_architecture
Dim process_architecture

Set WshShell = CreateObject("WScript.Shell")
Set WshProcEnv = WshShell.Environment("Process")

VuFindHome = WshProcEnv("VUFIND_HOME")
VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR")
VuFindImporter = "php.exe deletes.php"

WshShell.CurrentDirectory = VuFindHome + "\util"
VoyagerDir = "C:\Voyager_Interface"
outFile = VoyagerDir + "\VuFindGetBiB.txt"
logfile = "C:\Voyager_Interface\myvoyager_bib.log"

Set objFile = fso.CreateTextFile(outFile,True)
objFile.Write "option batch on" & vbCrLf
objFile.Write "option confirm off" & vbCrLf
objFile.Write "open voyager:Hy6};(@YOURSERVERHOSTNAME.exlibrisgroup.com -hostkey=""YOUR SSH HOST KEY""" & vbCrLf 
objFile.Write "option transfer binary" & vbCrLf
objFile.Write "cd /m1/voyager/YOURdb/rpt" & vbCrLf
objFile.Write "lcd C:\Voyager_Interface" & vbCrLf
DIM sMonth, sDay
Dim myDate, i
myDate = Dateadd("d",0,Date)
For i = 1 to 7
myDate = Dateadd("d",-1,myDate)

sMonth = Mid("00" + CStr(Month(myDate)), Len(CStr(Month(myDate))) + 1,2)
sDay = Mid("00" + CStr(Day(myDate)), Len(CStr(Day(myDate))) + 1,2)

VOYRECS = "deleted.bib-" + CStr(Year(Date)) + sMonth + sDay + ".mrc"

objFile.Write "mget deleted.bib.marc." & CStr(Year(Date)) & "-" & sMonth & "-" & sDay & "-* " & VOYRECS  & vbCrLf
'objFile.Write "get deleted.bib.marc " & VOYRECS  & vbCrLf
Next
objFile.Write "close" & vbCrLf
objFile.Write "exit" & vbCrLf
objfile.Close

process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE") 

If process_architecture = "x86" Then    
    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432")

    If system_architecture = ""  Then    
        system_architecture = "x86"
        ProgramFile = WshProcEnv("ProgramFiles")
    End if    
Else    
    system_architecture = process_architecture
    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindGetBib.txt /log=" + logfile   
End If

if (fso.FileExists(logfile)) Then
	fso.DeleteFile(logfile)
End If

Set oExec = WshShell.Exec(ProgramFile)

Do While oExec.Status = 0
     WScript.Sleep 500
Loop
Dim bOptimize
myDate = Dateadd("d",0,Date)
For i = 1 to 7
myDate = Dateadd("d",-1,myDate)
sMonth = Mid("00" + CStr(Month(myDate)), Len(CStr(Month(myDate))) + 1,2)
sDay = Mid("00" + CStr(Day(myDate)), Len(CStr(Day(myDate))) + 1,2)
VOYRECS = "deleted.bib-" + CStr(Year(Date)) + sMonth + sDay + ".mrc"

if (fso.FileExists(VoyagerDir + "\" + VOYRECS)) Then	

Set MarcFile = fso.GetFile(VoyagerDir + "\" + VOYRECS)

If MarcFile.Size > 0 Then
	bOptimize = 1
	ProgramFile = "CMD /c " + VuFindImporter + " " +  VoyagerDir + "\" + VOYRECS 
	Dim sStatus
	sStatus = WshShell.Run(ProgramFile,1,true)
	WScript.Sleep 500
End If
End If
Next
If bOptimize = 1 Then
	ProgramFile = "CMD /c php.exe optimize.php"
	sStatus = WshShell.Run(ProgramFile,1,true)
	WScript.Sleep 500

End If

WScript.Quit

VuFind Documentation

Table of Contents