About Features Downloads Getting Started Documentation Events Support GitHub

Love VuFind®? Consider becoming a financial supporter. Your support helps build a better VuFind®!

VuFind Documentation

Site Tools

You are here: start » administration » automation
Trace:
Warning: This page has not been updated in over over a year and may be outdated or deprecated.
administration:automation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
administration:automation [2022/10/13 14:18] – demiankatzadministration:automation [2023/03/06 20:48] (current) – [Automation Examples] demiankatz
Line 50: Line 50:
 Important:  The backup will cause your database to lock while it is running, so you should schedule it for a time of low activity. Important:  The backup will cause your database to lock while it is running, so you should schedule it for a time of low activity.
  
-===== Automation with Voyager and Linux =====+===== Automation Examples =====
  
-The following steps list a simplified version of the process used by Villanova to maintain its VuFind® installation.  These are intended as guidelines only, and you will want to customize the provided scripts for your own needs.  In particular, better error checking is important so you can figure out when things go wrong.  You may wish to add code to send emails when conditions are not met.  You may wish to add "echo" statements at various points to output status messages and then redirect the output of the scripts to a log file for later analysis.  None of this logic is included here since your needs may vary, and the point is to show the general processes needed by VuFind®.+This section provides examples of automating interactions between VuFind® and other systems:
  
-Also note that some of these cron jobs will put a heavy load on your VuFind® server.  You should schedule them to run at times of low usage if at all possible! +  * [[administration:automation:koha|Koha]] 
- +  * [[administration:automation:voyager|Voyager]]
-==== Updating VuFind®'s Index with the Latest MARC Records ==== +
- +
-Getting MARC records from Voyager into VuFind® requires configuration on two servers -- your Voyager server AND your VuFind® server.  The Voyager processes dump out the changed MARC records, while the VuFind® processes read them in and update VuFind®'s index. +
- +
-=== Voyager Configuration === +
- +
-You will want to set up this script to run regularly by cron: +
- +
-<code bash> +
-#!/bin/sh +
- +
-# Note: The path below will vary from system to system: +
-exporter=/m1/voyager/yourdb/sbin/Pmarcexport +
-file=catalog_INCR +
-PROG="$exporter -o $file -rB -mB -t" +
-YEST=`/usr/local/bin/gdate -d yesterday +%F` +
-TODAY=`/usr/local/bin/gdate +%F` +
- +
-# Dump the latest MARC records +
-exec $PROG$YEST:$TODAY +
- +
-# Wait a second for things to settle down +
-/usr/bin/sleep 1 +
- +
-# Move the file into position to be retrieved by the VuFind® server +
-chmod 777 $file +
-mv $file /scratch/$file-`date +%Y%m%d`.mrc +
- +
-# now exit gracefully... +
-exit 0 +
-</code> +
- +
-=== VuFind® Configuration === +
- +
-The following script needs to run by cron on the VuFind® server, obviously at a later time than the Voyager script runs so that there is sufficient time for the records to be exported.  It should be run as the user that controls VuFind®. +
- +
-Note that you will need to customize some variables in the script to match your environment.  You may also wish to add some error checks and e-mail alerts to administrators to detect unexpected failures in the process. +
- +
-In order for this script to work, it is also necessary to set up SSH keys so that the VuFind® server can complete the SFTP operation without being prompted for a password.  ([[http://ask-leo.com/how_can_i_automate_an_sftp_transfer_between_two_servers.html|This article]] may help). +
- +
-<code bash> +
-#!/bin/bash +
- +
-# This is the directory on the Voyager server where MARC records may be found: +
-DIR="/scratch/" +
- +
-# This is the filename of the MARC records on the Voyager server: +
-VOYRECS=catalog_INCR-`date +%Y%m%d`.mrc +
- +
-# This is the directory on the VuFind® server where MARC records will be downloaded: +
-LDIR="/tmp/marc_data" +
- +
-# This is the user and server name for the Voyager server: +
-VOYUSER=vufind +
-VOYSERVER=voyager.myinstitution.edu +
- +
-# Open sftp session and start the transfer +
-/usr/bin/sftp $VOYUSER@$VOYSERVER <<EOF>/dev/null +
- cd $DIR +
- lcd $LDIR +
- mget $VOYRECS +
- quit +
- +
-EOF +
-# done +
- +
-# Import the records to VuFind®: +
-IMPORT_SCRIPT=import-marc.sh +
-$VUFIND_HOME/$IMPORT_SCRIPT $LDIR/$VOYRECS +
- +
-# Remove suppressed records and optimize index: +
-cd $VUFIND_HOME/util +
-/usr/bin/php suppressed.php +
- +
-exit +
-</code> +
- +
- +
-=== Windows Versions === +
- +
-The following VBScripts, courtesy of Amos Chung (achung AT molloy DOT edu), offer Windows-based functionality roughly equivalent to the shell scripts above. +
- +
-== Voyager Configuration (VuFind_Export.vbs) == +
-<code> +
-Dim fso, VOYRECS +
-Set fso = WScript.CreateObject("Scripting.Filesystemobject") +
-Dim WshShell, oExec, ProgramFile,outFile,objFile +
-Dim VuFindHome, VuFindLocal, VuFindImporter +
-Dim WshProcEnv +
-Dim system_architecture +
-Dim process_architecture +
-Dim sDate, sDate2 +
-sDate = CStr(Year(Date)) & "-" &CStr(Month(Date)) & "-" &CStr(Day(Date))  +
-sDate2 = CStr(Year(DateAdd("D",-1,Date))) & "-" &CStr(Month(DateAdd("D",-1,Date))) & "-" &CStr(Day(DateAdd("D",-1,Date)))  +
-Set WshShell = CreateObject("WScript.Shell") +
-Set WshProcEnv = WshShell.Environment("Process") +
- +
-VuFindHome = WshProcEnv("VUFIND_HOME") +
-VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR") +
-VuFindImporter = "import-marc.bat" +
- +
-WshShell.CurrentDirectory = "C:\Voyager_Interface" +
- +
-outFile = WshShell.CurrentDirectory + "\VuFindExp.txt" +
- +
-Set objFile = fso.CreateTextFile(outFile,True) +
-VOYRECS = "catalog_INCR-" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".mrc" +
-objFile.Write "option batch on" & vbCrLf +
-objFile.Write "option confirm off" & vbCrLf +
-objFile.Write "open voyager:Hy6};(@YourCat.hosted.exlibrisgroup.com -hostkey=""YourSSHKey""" & vbCrLf +
-objFile.Write "cd /m1/voyager/YourDB/local" & vbCrLf +
-objFile.Write "rm catalog_INCR.mrc" & vbCrLf +
-objFile.Write "cd /m1/voyager/YourDB/sbin" & vbCrLf  +
-objFile.Write "option transfer binary" & vbCrLf +
-objFile.Write "call /m1/voyager/YourDB/sbin/Pmarcexport -o /m1/voyager/YourDB/local/catalog_INCR.mrc -rB -mB -t" & sDate2 & ":" & sDate & vbCrLf +
-'objFile.Write "lcd C:\Voyager_Interface" & vbCrLf +
-'objFile.Write "mget marc.exp.20131023.*" & vbCrLf +
-objFile.Write "close" & vbCrLf +
-objFile.Write "exit" & vbCrLf +
-objfile.Close +
-  +
-'Wscript.Echo WshShell.CurrentDirectory +
- +
-process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE")  +
- +
-If process_architecture = "x86" Then     +
-    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432") +
- +
-    If system_architecture = ""  Then     +
-        system_architecture = "x86" +
-        ProgramFile = WshProcEnv("ProgramFiles") +
-    End if     +
-Else     +
-    system_architecture = process_architecture +
-    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindExp.txt /log=C:\Voyager_Interface\myvoyager_get.log"    +
-End If +
- +
- +
-Set oExec = WshShell.Exec(ProgramFile) +
- +
-Do While oExec.Status = 0 +
-     WScript.Sleep 100 +
-Loop +
- +
-'ProgramFile = "CMD /K " +  VuFindHome + "/" + VuFindImporter + " " +  WshShell.CurrentDirectory + "\" + VOYRECS + " > import-marc.log" +
-'Wscript.Echo ProgramFile > "bbb.txt" +
-'Set oExec = WshShell.Run(ProgramFile,0,true) +
- +
-'Do While oExec.Status = 0 +
-'     WScript.Sleep 100 +
-'Loop +
-'Wscript.Echo VOYRECS, ProgramFile +
-'WScript.Echo "Running as a " & process_architecture & " process on a " _  +
-'    & system_architecture & " system." +
-'WScript.Echo oExec.Status +
-WScript.Quit +
-</code> +
-== VuFind® Configuration (vufind-import.vbs) == +
-<code> +
-Dim fso, VOYRECS +
-Set fso = WScript.CreateObject("Scripting.Filesystemobject") +
-Dim WshShell, oExec, ProgramFile,outFile,objFile +
-Dim VuFindHome, VuFindLocal, VuFindImporter +
-Dim WshProcEnv +
-Dim system_architecture +
-Dim process_architecture +
- +
-Set WshShell = CreateObject("WScript.Shell") +
-Set WshProcEnv = WshShell.Environment("Process") +
- +
-VuFindHome = WshProcEnv("VUFIND_HOME") +
-VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR") +
-VuFindImporter = "import-marc.bat" +
- +
-WshShell.CurrentDirectory = "C:\Voyager_Interface" +
- +
-outFile = WshShell.CurrentDirectory + "\VuFindGet.txt" +
- +
-Set objFile = fso.CreateTextFile(outFile,True) +
-VOYRECS = "catalog_INCR-" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".mrc" +
-objFile.Write "option batch on" & vbCrLf +
-objFile.Write "option confirm off" & vbCrLf +
-objFile.Write "open voyager:Hy6};(@YourCat.hosted.exlibrisgroup.com -hostkey=""YourSSHKey""" & vbCrLf  +
-objFile.Write "option transfer binary" & vbCrLf +
-objFile.Write "cd /m1/voyager/YourDB/local" & vbCrLf +
-objFile.Write "lcd C:\Voyager_Interface" & vbCrLf +
-objFile.Write "get catalog_INCR.mrc " & VOYRECS  & vbCrLf +
-objFile.Write "close" & vbCrLf +
-objFile.Write "exit" & vbCrLf +
-objfile.Close +
- +
-'Wscript.Echo WshShell.CurrentDirectory +
- +
-process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE")  +
- +
-If process_architecture = "x86" Then     +
-    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432") +
- +
-    If system_architecture = ""  Then     +
-        system_architecture = "x86" +
-        ProgramFile = WshProcEnv("ProgramFiles") +
-    End if     +
-Else     +
-    system_architecture = process_architecture +
-    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindGet.txt /log=C:\Voyager_Interface\myvoyager_get.log"    +
-End If +
- +
- +
-Set oExec = WshShell.Exec(ProgramFile) +
- +
-Do While oExec.Status = 0 +
-     WScript.Sleep 100 +
-Loop +
- +
-ProgramFile = "cmd /C " + VuFindHome + "/" + VuFindImporter + " " +  WshShell.CurrentDirectory + "\" + VOYRECS + " > import-marc" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".log" +
-'Wscript.Echo ProgramFile > "bbb.txt" +
-Dim sStatus +
-sStatus = WshShell.Run(ProgramFile,1,true) +
- +
-WScript.Quit +
-</code> +
- +
-== Script to Email Logs (sendLog.vbs) == +
- +
-<code> +
-strSMTPFrom = "no-reply@YourInstitution.edu" +
-strSMTPTo = "recipients@YourInstitution.edu,recipient2@YourInstitution.edu" +
-'strSMTPTo = "You@YourInstitution.edu" +
-strSMTPRelay = "mail.YourInstitution.edu" +
-strTextBody = "Please review VuFind MARC Import log." +
-strSubject = "VuFind MARC Import" +
-strAttachment = "C:\Voyager_Interface\import-marc" + CStr(Year(Date)) + CStr(Month(Date)) + CStr(Day(Date)) + ".log" +
-dim filesys +
- +
-set filesys=CreateObject("Scripting.FileSystemObject") +
- +
-If filesys.FileExists(strAttachment) Then +
- +
-filesys.CopyFile strAttachment , "C:\Voyager_Interface\import_mrc_log.txt" +
- +
-strAttachment = "C:\Voyager_Interface\import_mrc_log.txt" +
-Set oMessage = CreateObject("CDO.Message") +
-oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2  +
-oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = strSMTPRelay +
-oMessage.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25  +
-oMessage.Configuration.Fields.Update +
- +
-oMessage.Subject = strSubject +
-oMessage.From = strSMTPFrom +
-oMessage.To = strSMTPTo +
-oMessage.TextBody = strTextBody +
-oMessage.AddAttachment strAttachment +
- +
- +
-oMessage.Send +
-End If +
-WScript.Quit +
-</code> +
-==== Removing Deleted and Suppressed Records ==== +
- +
-Suppressed records are actually already taken care of by the scripts above.  Deleted records require a bit more scripting.  Voyager keeps a MARC file of all deleted records.  VuFind® comes with a utility to read a MARC file and delete all listed records within it.  We just need to automate downloading of the deleted record list and use of the appropriate VuFind® utility.  A sample script follows that can be run by cron on your VuFind® server. +
- +
-Notes: +
-  * As with the previous script, you need to have SSH keys set up so that SFTP can transfer files automatically. +
-  * You'll need to customize some variables to match your local setup. +
- +
-<code bash> +
-#!/bin/bash +
- +
-# Directory and filename on the Voyager server containing your deleted records file: +
-DIR="/m1/voyager/yourdb/rpt" +
-FILE="deleted.bib.marc" +
- +
-# Directory on your VuFind® server to download the file: +
-LDIR="/tmp/marc_data" +
- +
-# This is the user and server name for the Voyager server: +
-VOYUSER=vufind +
-VOYSERVER=voyager.myinstitution.edu +
- +
-# Download the deleted records: +
-cd $LDIR +
-/usr/bin/sftp $VOYUSER@$VOYSERVER <<EOF>/dev/null +
-                cd $DIR +
-                get $FILE +
-                quit +
-EOF +
- +
-# Remove deleted records from VuFind®'s index: +
-cd $VUFIND_HOME/util +
-/usr/bin/php deletes.php $LDIR/$FILE +
- +
-exit +
-</code> +
- +
-=== Windows version === +
- +
-The following VBScript, courtesy of Amos Chung (achung AT molloy DOT edu), offers Windows-based functionality roughly equivalent to the shell script above.  +
- +
-**Vufind_Delete.vbs** +
- +
-<code> +
-Dim fso, VOYRECS +
-Set fso = WScript.CreateObject("Scripting.Filesystemobject") +
-Dim WshShell, oExec, ProgramFile,outFile,objFile +
-Dim VuFindHome, VuFindLocal, VuFindImporter +
-Dim WshProcEnv, MarcFile +
-Dim system_architecture +
-Dim process_architecture +
- +
-Set WshShell = CreateObject("WScript.Shell") +
-Set WshProcEnv = WshShell.Environment("Process") +
- +
-VuFindHome = WshProcEnv("VUFIND_HOME") +
-VuFindLocal = WshProcEnv("VUFIND_LOCAL_DIR") +
-VuFindImporter = "php.exe deletes.php" +
- +
-WshShell.CurrentDirectory = VuFindHome + "\util" +
-VoyagerDir = "C:\Voyager_Interface" +
-outFile = VoyagerDir + "\VuFindGetBiB.txt" +
-logfile = "C:\Voyager_Interface\myvoyager_bib.log" +
- +
-Set objFile = fso.CreateTextFile(outFile,True) +
-objFile.Write "option batch on" & vbCrLf +
-objFile.Write "option confirm off" & vbCrLf +
-objFile.Write "open voyager:Hy6};(@YOURSERVERHOSTNAME.exlibrisgroup.com -hostkey=""YOUR SSH HOST KEY""" & vbCrLf  +
-objFile.Write "option transfer binary" & vbCrLf +
-objFile.Write "cd /m1/voyager/YOURdb/rpt" & vbCrLf +
-objFile.Write "lcd C:\Voyager_Interface" & vbCrLf +
-DIM sMonth, sDay +
-Dim myDate, i +
-myDate = Dateadd("d",0,Date) +
-For i = 1 to 7 +
-myDate = Dateadd("d",-1,myDate) +
- +
-sMonth = Mid("00" + CStr(Month(myDate)), Len(CStr(Month(myDate))) + 1,2) +
-sDay = Mid("00" + CStr(Day(myDate)), Len(CStr(Day(myDate))) + 1,2) +
- +
-VOYRECS = "deleted.bib-" + CStr(Year(Date)) + sMonth + sDay + ".mrc" +
- +
-objFile.Write "mget deleted.bib.marc." & CStr(Year(Date)) & "-" & sMonth & "-" & sDay & "-* " & VOYRECS  & vbCrLf +
-'objFile.Write "get deleted.bib.marc " & VOYRECS  & vbCrLf +
-Next +
-objFile.Write "close" & vbCrLf +
-objFile.Write "exit" & vbCrLf +
-objfile.Close +
- +
-process_architecture= WshProcEnv("PROCESSOR_ARCHITECTURE")  +
- +
-If process_architecture = "x86" Then     +
-    system_architecture= WshProcEnv("PROCESSOR_ARCHITEW6432") +
- +
-    If system_architecture = ""  Then     +
-        system_architecture = "x86" +
-        ProgramFile = WshProcEnv("ProgramFiles") +
-    End if     +
-Else     +
-    system_architecture = process_architecture +
-    ProgramFile = WshProcEnv("ProgramFiles(X86)") + "\WinSCP\WinSCP.com /console /script=C:\Voyager_Interface\VuFindGetBib.txt /log=" + logfile    +
-End If +
- +
-if (fso.FileExists(logfile)) Then +
- fso.DeleteFile(logfile) +
-End If +
- +
-Set oExec = WshShell.Exec(ProgramFile) +
- +
-Do While oExec.Status = 0 +
-     WScript.Sleep 500 +
-Loop +
-Dim bOptimize +
-myDate = Dateadd("d",0,Date) +
-For i = 1 to 7 +
-myDate = Dateadd("d",-1,myDate) +
-sMonth = Mid("00" + CStr(Month(myDate)), Len(CStr(Month(myDate))) + 1,2) +
-sDay = Mid("00" + CStr(Day(myDate)), Len(CStr(Day(myDate))) + 1,2) +
-VOYRECS = "deleted.bib-" + CStr(Year(Date)) + sMonth + sDay + ".mrc" +
- +
-if (fso.FileExists(VoyagerDir + "\" + VOYRECS)) Then  +
- +
-Set MarcFile = fso.GetFile(VoyagerDir + "\" + VOYRECS) +
- +
-If MarcFile.Size > 0 Then +
- bOptimize = 1 +
- ProgramFile = "CMD /c " + VuFindImporter + " " +  VoyagerDir + "\" + VOYRECS  +
- Dim sStatus +
- sStatus = WshShell.Run(ProgramFile,1,true) +
- WScript.Sleep 500 +
-End If +
-End If +
-Next +
-If bOptimize = 1 Then +
- ProgramFile = "CMD /c php.exe optimize.php" +
- sStatus = WshShell.Run(ProgramFile,1,true) +
- WScript.Sleep 500 +
- +
-End If +
- +
-WScript.Quit +
-</code>+
 ---- struct data ---- ---- struct data ----
 properties.Page Owner : demiankatz properties.Page Owner : demiankatz
 ---- ----
  
administration/automation.1665670686.txt.gz · Last modified: 2022/10/13 14:18 by demiankatz