Warning: This page has not been updated in over over a year and may be outdated or deprecated.
administration:automation
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
administration:automation [2016/05/13 19:39] – [Using cron] demiankatz | administration:automation [2023/03/06 20:48] (current) – [Automation Examples] demiankatz | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Automation ====== | ====== Automation ====== | ||
- | Once VuFind | + | Once VuFind® |
- | * Update | + | * Update |
- | * Remove deleted and suppressed records from VuFind's index. | + | * Remove deleted and suppressed records from VuFind®'s index. |
+ | * Remove expired data from VuFind®' | ||
* You may wish to make sure the Solr index is regularly optimized for minimal space usage and improved performance, | * You may wish to make sure the Solr index is regularly optimized for minimal space usage and improved performance, | ||
* Dump the MySQL database for backup purposes. | * Dump the MySQL database for backup purposes. | ||
Line 20: | Line 21: | ||
If you need to automate tasks under Linux, you will want to become familiar with the cron process. | If you need to automate tasks under Linux, you will want to become familiar with the cron process. | ||
- | === cron-friendly | + | === cron-friendly |
- | If you try to run the standard vufind.sh script | + | If you want to restart |
- | <code bash> | + | === Why restart |
- | #!/bin/sh | + | |
- | + | ||
- | # Disable JETTY_CONSOLE output -- it causes problems when run by cron: | + | |
- | export JETTY_CONSOLE=/ | + | |
- | + | ||
- | # Pass parameters along to vufind.sh: | + | |
- | CURRENTPATH=`dirname $0` | + | |
- | cd $CURRENTPATH | + | |
- | $CURRENTPATH/ | + | |
- | </ | + | |
- | + | ||
- | Important: | + | |
- | + | ||
- | === Why restart | + | |
- | The main reason to regularly restart the VuFind | + | The main reason to regularly restart the VuFind® Solr instance |
==== Using mysqldump ==== | ==== Using mysqldump ==== | ||
- | If you want to back up VuFind's MySQL database, the mysqldump tool that comes with MySQL itself makes this pretty simple. | + | If you want to back up VuFind®'s MySQL database, the mysqldump tool that comes with MySQL itself makes this pretty simple. |
<code bash> | <code bash> | ||
Line 63: | Line 50: | ||
Important: | Important: | ||
- | ===== Automation | + | ===== Automation |
- | The following steps list a simplified version | + | This section provides examples |
- | Also note that some of these cron jobs will put a heavy load on your VuFind server. | + | * [[administration:automation: |
- | + | * [[administration:automation:voyager|Voyager]] | |
- | ==== Updating VuFind' | + | |
- | + | ||
- | Getting MARC records from Voyager into VuFind requires configuration on two servers -- your Voyager server AND your VuFind server. | + | |
- | + | ||
- | === Voyager Configuration === | + | |
- | + | ||
- | You will want to set up this script to run regularly by cron: | + | |
- | + | ||
- | <code bash> | + | |
- | #!/bin/sh | + | |
- | + | ||
- | # Note: The path below will vary from system to system: | + | |
- | exporter=/ | + | |
- | file=catalog_INCR | + | |
- | PROG=" | + | |
- | YEST=`/ | + | |
- | TODAY=`/ | + | |
- | + | ||
- | # Dump the latest MARC records | + | |
- | exec $PROG$YEST: | + | |
- | + | ||
- | # Wait a second for things to settle down | + | |
- | / | + | |
- | + | ||
- | # Move the file into position to be retrieved by the VuFind server | + | |
- | chmod 777 $file | + | |
- | mv $file / | + | |
- | + | ||
- | # now exit gracefully... | + | |
- | exit 0 | + | |
- | </ | + | |
- | + | ||
- | === VuFind Configuration === | + | |
- | + | ||
- | The following script needs to run by cron on the VuFind server, obviously at a later time than the Voyager script runs so that there is sufficient time for the records to be exported. | + | |
- | + | ||
- | Note that you will need to customize some variables in the script to match your environment. | + | |
- | + | ||
- | In order for this script to work, it is also necessary to set up SSH keys so that the VuFind server can complete the SFTP operation without being prompted for a password. | + | |
- | + | ||
- | <code bash> | + | |
- | # | + | |
- | + | ||
- | # This is the directory on the Voyager server where MARC records may be found: | + | |
- | DIR="/ | + | |
- | + | ||
- | # This is the filename of the MARC records on the Voyager server: | + | |
- | VOYRECS=catalog_INCR-`date +%Y%m%d`.mrc | + | |
- | + | ||
- | # This is the directory on the VuFind server where MARC records will be downloaded: | + | |
- | LDIR="/ | + | |
- | + | ||
- | # This is the user and server name for the Voyager server: | + | |
- | VOYUSER=vufind | + | |
- | VOYSERVER=voyager.myinstitution.edu | + | |
- | + | ||
- | # Open sftp session and start the transfer | + | |
- | / | + | |
- | cd $DIR | + | |
- | lcd $LDIR | + | |
- | mget $VOYRECS | + | |
- | quit | + | |
- | + | ||
- | EOF | + | |
- | # done | + | |
- | + | ||
- | # Import the records to VuFind: | + | |
- | IMPORT_SCRIPT=import-marc.sh | + | |
- | $VUFIND_HOME/ | + | |
- | + | ||
- | # Remove suppressed records and optimize index: | + | |
- | cd $VUFIND_HOME/ | + | |
- | / | + | |
- | + | ||
- | exit | + | |
- | </ | + | |
- | + | ||
- | + | ||
- | === Windows Versions === | + | |
- | + | ||
- | The following VBScripts, courtesy of Amos Chung (achung AT molloy DOT edu), offer Windows-based functionality roughly equivalent to the shell scripts above. | + | |
- | + | ||
- | == Voyager Configuration (VuFind_Export.vbs) == | + | |
- | < | + | |
- | Dim fso, VOYRECS | + | |
- | Set fso = WScript.CreateObject(" | + | |
- | Dim WshShell, oExec, ProgramFile, | + | |
- | Dim VuFindHome, VuFindLocal, | + | |
- | Dim WshProcEnv | + | |
- | Dim system_architecture | + | |
- | Dim process_architecture | + | |
- | Dim sDate, sDate2 | + | |
- | sDate = CStr(Year(Date)) & " | + | |
- | sDate2 = CStr(Year(DateAdd(" | + | |
- | Set WshShell = CreateObject(" | + | |
- | Set WshProcEnv = WshShell.Environment(" | + | |
- | + | ||
- | VuFindHome = WshProcEnv(" | + | |
- | VuFindLocal = WshProcEnv(" | + | |
- | VuFindImporter = " | + | |
- | + | ||
- | WshShell.CurrentDirectory = " | + | |
- | + | ||
- | outFile = WshShell.CurrentDirectory + " | + | |
- | + | ||
- | Set objFile = fso.CreateTextFile(outFile, | + | |
- | VOYRECS = " | + | |
- | objFile.Write " | + | |
- | objFile.Write " | + | |
- | objFile.Write "open voyager: | + | |
- | objFile.Write "cd / | + | |
- | objFile.Write "rm catalog_INCR.mrc" | + | |
- | objFile.Write "cd / | + | |
- | objFile.Write " | + | |
- | objFile.Write "call / | + | |
- | ' | + | |
- | ' | + | |
- | objFile.Write " | + | |
- | objFile.Write " | + | |
- | objfile.Close | + | |
- | + | ||
- | ' | + | |
- | + | ||
- | process_architecture= WshProcEnv(" | + | |
- | + | ||
- | If process_architecture = " | + | |
- | system_architecture= WshProcEnv(" | + | |
- | + | ||
- | If system_architecture = "" | + | |
- | system_architecture = " | + | |
- | ProgramFile = WshProcEnv(" | + | |
- | End if | + | |
- | Else | + | |
- | system_architecture = process_architecture | + | |
- | ProgramFile = WshProcEnv(" | + | |
- | End If | + | |
- | + | ||
- | + | ||
- | Set oExec = WshShell.Exec(ProgramFile) | + | |
- | + | ||
- | Do While oExec.Status = 0 | + | |
- | | + | |
- | Loop | + | |
- | + | ||
- | ' | + | |
- | ' | + | |
- | 'Set oExec = WshShell.Run(ProgramFile, | + | |
- | + | ||
- | 'Do While oExec.Status = 0 | + | |
- | ' | + | |
- | ' | + | |
- | ' | + | |
- | ' | + | |
- | ' | + | |
- | ' | + | |
- | WScript.Quit | + | |
- | </ | + | |
- | == VuFind Configuration (vufind-import.vbs) == | + | |
- | < | + | |
- | Dim fso, VOYRECS | + | |
- | Set fso = WScript.CreateObject(" | + | |
- | Dim WshShell, oExec, ProgramFile, | + | |
- | Dim VuFindHome, VuFindLocal, | + | |
- | Dim WshProcEnv | + | |
- | Dim system_architecture | + | |
- | Dim process_architecture | + | |
- | + | ||
- | Set WshShell = CreateObject(" | + | |
- | Set WshProcEnv = WshShell.Environment(" | + | |
- | + | ||
- | VuFindHome = WshProcEnv(" | + | |
- | VuFindLocal = WshProcEnv(" | + | |
- | VuFindImporter = " | + | |
- | + | ||
- | WshShell.CurrentDirectory = " | + | |
- | + | ||
- | outFile = WshShell.CurrentDirectory + " | + | |
- | + | ||
- | Set objFile = fso.CreateTextFile(outFile, | + | |
- | VOYRECS = " | + | |
- | objFile.Write " | + | |
- | objFile.Write " | + | |
- | objFile.Write " | + | |
- | objFile.Write " | + | |
- | objFile.Write "cd / | + | |
- | objFile.Write "lcd C: | + | |
- | objFile.Write "get catalog_INCR.mrc " & VOYRECS | + | |
- | objFile.Write " | + | |
- | objFile.Write " | + | |
- | objfile.Close | + | |
- | + | ||
- | ' | + | |
- | + | ||
- | process_architecture= WshProcEnv(" | + | |
- | + | ||
- | If process_architecture = " | + | |
- | system_architecture= WshProcEnv(" | + | |
- | + | ||
- | If system_architecture = "" | + | |
- | system_architecture = " | + | |
- | ProgramFile = WshProcEnv(" | + | |
- | End if | + | |
- | Else | + | |
- | system_architecture = process_architecture | + | |
- | ProgramFile = WshProcEnv(" | + | |
- | End If | + | |
- | + | ||
- | + | ||
- | Set oExec = WshShell.Exec(ProgramFile) | + | |
- | + | ||
- | Do While oExec.Status = 0 | + | |
- | | + | |
- | Loop | + | |
- | + | ||
- | ProgramFile = "cmd /C " + VuFindHome + "/" | + | |
- | ' | + | |
- | Dim sStatus | + | |
- | sStatus = WshShell.Run(ProgramFile, | + | |
- | + | ||
- | WScript.Quit | + | |
- | </ | + | |
- | + | ||
- | == Script to Email Logs (sendLog.vbs) == | + | |
- | + | ||
- | < | + | |
- | strSMTPFrom = " | + | |
- | strSMTPTo = " | + | |
- | ' | + | |
- | strSMTPRelay = " | + | |
- | strTextBody = " | + | |
- | strSubject = " | + | |
- | strAttachment = " | + | |
- | dim filesys | + | |
- | + | ||
- | set filesys=CreateObject(" | + | |
- | + | ||
- | If filesys.FileExists(strAttachment) Then | + | |
- | + | ||
- | filesys.CopyFile strAttachment , " | + | |
- | + | ||
- | strAttachment = " | + | |
- | Set oMessage = CreateObject(" | + | |
- | oMessage.Configuration.Fields.Item(" | + | |
- | oMessage.Configuration.Fields.Item(" | + | |
- | oMessage.Configuration.Fields.Item(" | + | |
- | oMessage.Configuration.Fields.Update | + | |
- | + | ||
- | oMessage.Subject = strSubject | + | |
- | oMessage.From = strSMTPFrom | + | |
- | oMessage.To = strSMTPTo | + | |
- | oMessage.TextBody = strTextBody | + | |
- | oMessage.AddAttachment strAttachment | + | |
- | + | ||
- | + | ||
- | oMessage.Send | + | |
- | End If | + | |
- | WScript.Quit | + | |
- | </ | + | |
- | ==== Removing Deleted and Suppressed Records ==== | + | |
- | + | ||
- | Suppressed records are actually already taken care of by the scripts above. | + | |
- | + | ||
- | Notes: | + | |
- | * As with the previous script, you need to have SSH keys set up so that SFTP can transfer files automatically. | + | |
- | * You'll need to customize some variables to match your local setup. | + | |
- | * When using VuFind 1.x, be sure you run this script a significant amount of time AFTER the update script listed above. | + | |
- | + | ||
- | <code bash> | + | |
- | # | + | |
- | + | ||
- | # Directory and filename on the Voyager server containing your deleted records file: | + | |
- | DIR="/ | + | |
- | FILE=" | + | |
- | + | ||
- | # Directory on your VuFind server to download the file: | + | |
- | LDIR="/ | + | |
- | + | ||
- | # This is the user and server name for the Voyager server: | + | |
- | VOYUSER=vufind | + | |
- | VOYSERVER=voyager.myinstitution.edu | + | |
- | + | ||
- | # Download the deleted records: | + | |
- | cd $LDIR | + | |
- | / | + | |
- | cd $DIR | + | |
- | get $FILE | + | |
- | quit | + | |
- | EOF | + | |
- | + | ||
- | # Remove deleted records from VuFind' | + | |
- | cd $VUFIND_HOME/ | + | |
- | / | + | |
- | + | ||
- | exit | + | |
- | </ | + | |
- | + | ||
- | === Windows version === | + | |
- | + | ||
- | The following VBScript, courtesy of Amos Chung (achung AT molloy DOT edu), offers Windows-based functionality roughly equivalent to the shell script above. | + | |
- | + | ||
- | **Vufind_Delete.vbs** | + | |
- | + | ||
- | < | + | |
- | Dim fso, VOYRECS | + | |
- | Set fso = WScript.CreateObject(" | + | |
- | Dim WshShell, oExec, ProgramFile, | + | |
- | Dim VuFindHome, VuFindLocal, | + | |
- | Dim WshProcEnv, MarcFile | + | |
- | Dim system_architecture | + | |
- | Dim process_architecture | + | |
- | + | ||
- | Set WshShell = CreateObject(" | + | |
- | Set WshProcEnv = WshShell.Environment(" | + | |
- | + | ||
- | VuFindHome = WshProcEnv(" | + | |
- | VuFindLocal = WshProcEnv(" | + | |
- | VuFindImporter = " | + | |
- | + | ||
- | WshShell.CurrentDirectory = VuFindHome + " | + | |
- | VoyagerDir = " | + | |
- | outFile = VoyagerDir + " | + | |
- | logfile = " | + | |
- | + | ||
- | Set objFile = fso.CreateTextFile(outFile, | + | |
- | objFile.Write " | + | |
- | objFile.Write " | + | |
- | objFile.Write "open voyager: | + | |
- | objFile.Write " | + | |
- | objFile.Write "cd / | + | |
- | objFile.Write "lcd C: | + | |
- | DIM sMonth, sDay | + | |
- | Dim myDate, i | + | |
- | myDate = Dateadd(" | + | |
- | For i = 1 to 7 | + | |
- | myDate = Dateadd(" | + | |
- | + | ||
- | sMonth = Mid(" | + | |
- | sDay = Mid(" | + | |
- | + | ||
- | VOYRECS = " | + | |
- | + | ||
- | objFile.Write "mget deleted.bib.marc." | + | |
- | ' | + | |
- | Next | + | |
- | objFile.Write " | + | |
- | objFile.Write " | + | |
- | objfile.Close | + | |
- | + | ||
- | process_architecture= WshProcEnv(" | + | |
- | + | ||
- | If process_architecture = " | + | |
- | system_architecture= WshProcEnv(" | + | |
- | + | ||
- | If system_architecture = "" | + | |
- | system_architecture = " | + | |
- | ProgramFile = WshProcEnv(" | + | |
- | End if | + | |
- | Else | + | |
- | system_architecture = process_architecture | + | |
- | ProgramFile = WshProcEnv(" | + | |
- | End If | + | |
- | + | ||
- | if (fso.FileExists(logfile)) Then | + | |
- | fso.DeleteFile(logfile) | + | |
- | End If | + | |
- | + | ||
- | Set oExec = WshShell.Exec(ProgramFile) | + | |
- | + | ||
- | Do While oExec.Status = 0 | + | |
- | | + | |
- | Loop | + | |
- | Dim bOptimize | + | |
- | myDate = Dateadd(" | + | |
- | For i = 1 to 7 | + | |
- | myDate = Dateadd(" | + | |
- | sMonth = Mid(" | + | |
- | sDay = Mid(" | + | |
- | VOYRECS = " | + | |
- | + | ||
- | if (fso.FileExists(VoyagerDir + " | + | |
- | + | ||
- | Set MarcFile = fso.GetFile(VoyagerDir + " | + | |
- | + | ||
- | If MarcFile.Size > 0 Then | + | |
- | bOptimize = 1 | + | |
- | ProgramFile = "CMD /c " + VuFindImporter + " " + VoyagerDir + " | + | |
- | Dim sStatus | + | |
- | sStatus = WshShell.Run(ProgramFile, | + | |
- | WScript.Sleep 500 | + | |
- | End If | + | |
- | End If | + | |
- | Next | + | |
- | If bOptimize = 1 Then | + | |
- | ProgramFile = "CMD /c php.exe optimize.php" | + | |
- | sStatus = WshShell.Run(ProgramFile, | + | |
- | WScript.Sleep 500 | + | |
- | + | ||
- | End If | + | |
- | + | ||
- | WScript.Quit | + | |
- | </ | + | |
---- struct data ---- | ---- struct data ---- | ||
+ | properties.Page Owner : demiankatz | ||
---- | ---- | ||
administration/automation.1463168373.txt.gz · Last modified: 2016/05/13 19:39 by demiankatz