The tenth VuFind instructional video explains how to add Git version control to an existing installation, if that installation is not already version-controlled, and how to use Git and shell scripting to simplify the process of upgrading both the core code and your local configurations/customizations.

Video is available as an mp4 download or through YouTube.

1. VuFind's changelog: a vital resource when planning an upgrade
2. Git wiki page: useful advice and background information about Git
3. Git Branches wiki page: information on how branches and tags are used in VuFind's Git repository
4. Automatically updating locally customized files with Git and diff3: the blog post containing the script referenced in the video

This is a raw machine-generated transcript; it will be cleaned up as time permits.

with the recent release of ufind 7.0 i thought now would be a good time to share my upgrade workflow uh to show how i use git to upgrade viewfind this video is going to assume that you have a basic familiarity with git and if you do not i strongly strongly recommend learning at least the basics doing software development and deployment without version control is like driving without a seat belt and you will find that investing a little time and learning these tools will save you all kinds of trouble down the rib so this video will cover two basic topics the first will be turning an existing installation of viewfind into a git repository so that you can use git to manage it and the second part will show some processes and tools for using git to perform actual upgrades so the virtual machine i've been using for this series of tutorials was initially set up by installing ufi from the debian package which just puts the files on disk but does not include any kind of version control we are going to take advantage of a useful characteristic of git namely that all of the version control data is stored in a dot get subdirectory which can be easily moved around and we're going to use that to turn our existing installation into a git repository we will check out a temporary clone of the repo move the git directory into our viewfind home and then we'll be able to work from there but before we can begin doing all of that uh the most important thing we need to do first is figure out exactly which version of viewfind we are already running because that will enable us to put get into the correct state to start all of the work that we need to do there are a couple of ways to identify which version of viewfinder running a simple one particularly useful if you don't have access to the server running viewfind is to simply view the page source while looking at any new find web page and look for the generator meta tag which should have a version number embedded in it so we can see here 6.1 6.1.1 however the generator meta tag is not 100 reliable because it's actually created based on a config.ini setting and if that configuration file gets out of date or is customized it's possible that what it's reporting is not actually the truth if you want to be more confident that you have the right number you should go to the command line switch to your viewfind home directory and edit the build.xml file this is the control file used by the thing automation tool to do various viewfinder related build tasks most users don't have to worry about it on a day-to-day basis but it does have the useful characteristic of having a version number embedded in it so if you scroll down until you find the property named version you can see that once again this confirms we are working with you find 6.1.1 now that we know what version we're running as i said we can start creating a git repository to turn this bare file collection into a tracked version controlled repository and before i begin i'm just going to confirm if i run a git status from my viewfinder it tells me it's not a git repository so let's go get one if we go to the temp directory we can run git clone https github.com viewfind org viewfind.get and that will get us a copy of the full repository and all of the history of viewfind development and while this takes time to download i wanted to talk a little bit about how we use git to track various versions of viewfind first of all mainline bleeding edge development takes place in a branch called dev so if you want the latest code the most up-to-date the most current you would want to check out dev but of course leading edge code comes with risks because there may be some recently introduced bugs and while we try to never break the dev branch less tested code is always more risky if you want more stable code we have a series of release branches so for every major or minor release we create a branch named release dash and a number so for example release dash 7.0 or release dash 6.1 and these branches are where we do bug fixing on existing releases so we'll never add a new feature to a release branch but we will add fixes as needed and it's these release branches that we use to issue bug fix releases that only include fixes even after development in the main dev branch has moved on to further new exciting territory a final tool that you may find useful is that we also tag every release so if you want the exact point in time for a particular viewfinder release you can check out a tag which is just going to be v and a number so for example v 7.0 or v 6.1.1 if you want to get to a very precise state in the code you can use a tag and that will be useful for example in this situation where we're trying to sync up with a particular known version but if you're trying to get the latest stable code on a particular thread of development use a release branch instead all that being said the git repo is now cloned we need to switch into the directory that the git clone operation created cd viewfinder and then we need to check out the appropriate version tag in this case the 6.1.1 now git is going to complain that it's in a detached head state because we've selected a tag rather than a branch we will clean up that mess in a moment but first since we now have git at the same point in time as the release of ufind we're working with uh we should be able to move that hidden.getdirectory into the viewfind home directory to turn it into a git repo so now i'm going to switch to viewfind home and now if i do a git status instead of telling me this is not a repository it's going to re-index itself and then tell me we're at a detached head at version 6.1.1 and there are a whole bunch of untracked files this is exactly what we want we're not seeing any diffs which means that we picked the right version and we haven't accidentally edited any core files but it is seeing all of the local files that we added in our local directory and our custom theme so what we want to do now is to establish a branch for our local code so that we can begin to track the things that we change and customize over time what we can do is say get check out minus b and make up a name for our branch i'm going to call this local tutorial and now git has created a local branch where we can begin to add files and track changes i should note that in this example i'm going to be committing files containing passwords and local information to this git branch uh so if you do something similar you just have to be careful that you don't accidentally publish this branch somewhere you don't wish to so watch out for that but here there's not too much danger and tracking these files is useful so i'm just going to go ahead and start adding files to git so i'm going to git add the env.that file that the installer set up even though we don't need it since we're not running in windows i'm going to add everything under the local slash config viewfind directory i'm going to add localharvestoai.ini because that configuration is useful to track i'm going to add local import local httpd viewfind.com and themes.tutorial if i do another git status just to check that i did that right we see that i've added a whole bunch of files that are waiting to be committed the only things i've omitted are these two harvest subdirectories which contain harvested metadata and it really doesn't make any sense to track those in version control so in fact what i'm going to do is i'm going to edit my git ignore file which tells get certain directories they can safely ignore and i'm going to add local harvest expositions and local harvest viewfind so that they don't confuse me or get accidentally committed in the future i do a git status again it now tells me that i've modified git ignore but it doesn't show me those directories i want to hide so i'm going to git add git ignore to my list and then i'm going to commit everything with a message of initial local files so now i've got you find 6.1 branched off into my own local branch where i have added all of my local configuration files uh we now have set the stage for performing the upgrade but before i upgrade i want to talk about a few things that are really important to keep in mind first of all before you start to upgrade be sure that you understand what your local customizations are uh what themes have you created do you have a local code module have you customized configuration files what's in there and why obviously if you're working on a team you may not know every detail of everything but that's one of the reasons to use version control because your version history of your local files can tell you who did things when they did them why they did them and so forth but at least at a high level it's really helpful to go into the upgrade process with an idea of what you've customized so that you can test those customizations post upgrade and be sure that they didn't get broken one of the reasons why viewfind separates local files from core files uh so deliberately with the local settings directory a separate module for local code a separate theme directory for local themes is to make it very clear what is yours and what is part of the project so that if for example a new developer needs to take over a viewfinder instance that someone has already customized they can review those local directories and see what's been done and sort of understand the context this is why it's important to be disciplined about separating your localizations from the core and one of the advantages of using git is that it helps you to do this because if you accidentally change a core file that will show up when you do a git status and then you'll know oh i shouldn't have done that i need to move that to the appropriate place in any case once you're comfortable that you understand your local customizations and their scope uh the next thing i highly recommend doing is looking at the change log in the viewfinder wiki and i will link to this from the video recording but for every release of viewfind we include notes not only on new features that might be of interest but also of changes to code and configuration that could potentially cause problems during an upgrade we are very inclusive here because we want to catch every possible issue that might be a problem most of these are unlikely to affect most users but that's why it's helpful to have a broad idea of what you've customized because you can then read through this list and take note of which issues are likely to be a problem and which ones you can very safely ignore in the case of this specific upgrade that we are about to run the one change log note that uh i need to be concerned about is this one uh starting with viewfind seven we changed the default port that solar runs on from 8080 to 8983 this is because 8983 is the standard port number used by solar and using 8080 historically has caused port conflicts with other applications it seemed to make sense to standardize that but now at upgrade time we need to be aware of this change so that we can deal with it appropriately right now our solar is running on port 8080 and after the upgrade viewfinder will be looking for it on port 8983 i'll show you how to deal with that after the rest of the process and of course this particular issue only applies to viewfind seven but this is just an example of the kind of thing you should be aware of when you're reviewing the changelog finally the third thing that you should always do before attempting an upgrade is back everything up and of course it's best to test and upgrade on a non-production server before you dive in uh in the real world uh in this instance i'm not going to show you how i back things up because this is a virtual machine and i've just backed up the whole disk image before i started so i can roll back if i have to but whatever your situation just be sure that you have a rollback plan so if something goes wrong during upgrade you haven't broken your system and gotten into an unrecoverable position with all that background out of the way we're just about ready to begin there's only one other thing to watch out for and this is that when we use get to do updates the user running the git command needs to have permission to write to all of the folders and files in your viewfind home because it's going to be updating core files uh so it's a good idea to check and make sure that you're using an appropriate user so in this instance if i look at the file ownerships of my viewfinder directory i see that most of these are owned by me d-cats except for some reason this batch file is owned by root and the solar directory is owned by solar we changed the solar ownership because the user running the solar process needs to be able to write there but what we can do is work around this with a group ownership i will leave solar owned by solar but i'm going to add it to the d-caps group which will give my user account permission to update files in that directory and i'm going to do that with the 2-ch grp minus capital r for recursive uh d cats solar so that's going to change the group ownership of the solar directory and then i'm going to use ch mod minus r g plus w to add group write permissions to that directory so now because all these files and folders have my group and they have group write permission i will be able to modify them again the way that you actually modify your you find directory permissions in a real-world situation will depend very much on your strategy for deployment but if you're using git to update things you need to make sure that git can write to all the files so uh now that that's done all i need to do is run a git merge operation to pull in all of the changes between release 6.1.1 where we are currently and the target endpoint we want to get to so if i wanted to upgrade to specifically the 7.0 release i could say git merge b 7.0 using the v 7.0 tag and i would get exactly to the code as it was released in 7.0 however in this instance i happen to know that there were some bugs in 7.0 that have been subsequently fixed and that would cause me problems on this test box so i'm instead going to choose to merge the release 7.0 branch which is the stable code from release 7.0 with subsequent bug fixes applied so all i need to do is say git merge origin slash release 7.0 the origin part is because origin is the default remote repository name that you get when you clone something so that refers to the public viewfind repo and of course release 7.0 is the branch i want so when i run that command i am prompted to customize the commit message if i want to but i'm happy to take the default so i'll just exit out of this text editor and now i watched a bunch of uh changes fly by and my core code has been upgraded to viewfind 7.0 but we're not quite done first of all we need to make sure that all of our dependencies are up to date because some parts of viewfinder loaded in with composer and updating the code with git will have changed the composer configuration but it won't have automatically triggered a composer installed so if i just say composer install following my merge that will bring all of our dependencies up to date and there are quite a few changes in play uh with this upgrade because this is where we switched from zen framework to its successor laminas so almost every core dependency of ufi changed its name here even though the functionality is the same so we have to wait for all of that to download and update itself the other thing we want to be sure to clean up is hugh finds cash because it's possible that there is data in the cache that is now out of date that could cause problems post upgrade so i am just going to sudu rm minus rf local slash cache star to clear out all of the cache directories under there uh but then i'm also going to recreate the local slash cache cli directory and change that to be owned by me because we need a command line cache separate from the web-based cache to allow our command line utilities to run correctly um so in in reality i usually have a pre-written script for automating all the steps of the upgrade which includes removing the cache and recreating the command line cache you would probably benefit from uh doing that as well but for now i just wanted to show those important steps so now we have updated code we have updated dependencies we have a clean cache but there's one more very important step uh because all of our local files are separated from the core obviously when we did our git merge and update it updated all the core files but the upstream changes don't apply to our local files at all because they're local files however because all of our local files are just copies of core files with a few changes applied we can do a bit of clever scripting to find changes to the core equivalence to the local files and then merge those changes into the local files i've written a bash script that does all of this work uh and in fact i've written the blog post explaining exactly how that works and the the reasoning behind it that's another link that i will include on the notes in this video but for right now i am just going to skip to the answer and copy and paste my script code out of the blog post and into a file on disk so i'm going to create a file called merge local.sh i'm going to paste all my code into it now the code i copied and pasted from my blog post is an example of how we do things at villanova and it's a little bit out of date because i wrote this blog post a few years ago so it's going to need minor adjustments uh to be useful for us here the idea is that this script defines a function called merge directory that takes two parameters the first parameter is the path of a directory containing locally customized files the second parameter is the core directory that includes equivalence to those files so we've customized things for local harvest local import and local config viewfind so all of those uh defaults can remain as they are in the example however our custom theme here is called tutorial so i'm just going to customize a couple lines of code here so we are going to merge our local tutorial templates and javascript files with changes from the core bootstrap 3 theme on the assumption that any custom templates in tutorial that share names with templates from bootstrap 3 are copies of those that have been customized so with this file all ready to go i just need to make it executable so i can run the script now i'm ready to apply the upgrades from the core to my local files one very important note about this script is that you have to run it immediately after you perform a merge because it works by looking at what changed in the most recent git commit and if the most recent git commit is not the merge to upgrade you find it won't know how to change your local files so it's always important to do this merge process right away uh after doing a merge to update the core so i'm just going to run my script and then it's going to spew out a bunch of messages note that it complained about all of the metadata files under my local harvest directory because of course those have no equivalent in the core code because they're metadata files so the script just safely skips over them it just gives me an alert that it saw them and didn't know what to do with them uh it also reports that it ran into at least one conflict and conflicts are somewhat inevitable in any kind of uh difference merging situation because what my script is essentially doing is saying i know that we started with a particular file in viewfind 6.1.1 and when viewfind 7 came out some changes were made to that file to upgrade to viewfind 7. but you've also created a local copy where you've made some changes of your own the script process will try to reconcile those changes as best it can but sometimes both paths of history will have tried to change the same part of the file and then of course there's no automated way to figure out which thing is right and that's how we end up having to do some manual conflict resolution fortunately uh it's not too difficult in most cases as long as you understand how to read the conflict markers and you understand the history and reasoning for your customizations so first of all let's find out which files have conflicts in them to figure out which files contain conflicts we are going to run a git if command which will output a list of all the changes to files that haven't been committed yet we're going to pipe that into a grep command looking for a series of greater than symbols which is a conflict marker that the merge process will have inserted into the files and that should give us a hint of which files need attention as you can see in this example we have three conflicts in config.ini you'll notice that the file names we're seeing here actually refer to the core file so we also need to be able to figure out uh based on familiarity with our code which local file is the equivalent to that in this case i know that configuration files exist in my local directory and otherwise have an equivalent path so what i need to do is edit my local slash config viewfind config.ini file and i can search through this file for any instances of series of less than or greater than signs to find areas of potential conflict so for example here is one conflict that's presenting me with three different sets of text it's showing me what my original local file looked like then it's showing me what the old viewfind 6.1 file looked like and then it's showing me what the viewfind 7 file looks like and what i essentially need to do is pick one of these versions delete all the other ones and get rid of the surrounding markers and this particular conflict occurred because at some point in the past i upgraded this tutorial box to uh from viewfind 6.0 to viewfind 6.1 and i didn't do a very good job so i introduced an inconsistency so i apologize for causing that confusion but it did create a useful example here uh and in this instance it really doesn't matter too much what we do because these are all just comments and they're not going to have any effect on the code but to be ready for future updates the smart thing to do is to accept the newest version of the comment this one down here that's associated with uh config slash viewfinder config.ini the current core code so i'm going to delete the two unwanted old versions delete the conflict marker at the end and then i can move on and then there's another conflict right below here as well this is telling me that there are some additions that did not exist in either um my locally customized file or um the previous version of viewfind again this is all related to me not having fully updated my config.ini the last time i upgraded but once again the solution is simply to accept the final option here which is the newest version of the configuration so i'm going to delete the old versions and take off the conflict marker and then there is just one more and it's the same kind of thing there are three different versions of the form setting in this example in the comment the one on the bottom is the current default in viewfind 7.0 so i am just going to delete all the other versions and delete the conflict marker and that's it all of my conflicts are now resolved this obviously requires a bit of brain power to get through sometimes it's not entirely obvious how to resolve a conflict but as long as you understand that each section of the conflict block is showing you a different version of that chunk of code or configuration it's usually possible to figure out from the context which is the right one and if nothing else you can make notes and review uh your commit history to see why you had gotten a particular line of code into a particular state anyway now that i have completed my conflict resolution i can do a git diff and this will just summarize everything that got automatically applied by that merge script and it's always a good idea to look through these and just see if anything stands out as a potential problem so as i mentioned at the very top of the video the generator value that viewfind uses is stored in config.ini and we can see here that the upgrade process correctly updated that from 6.1.1 to 7.0 uh then i think most of the remainder of the dips we're going to see here are new settings or changed comments uh that took place during the course of ufi development between the versions uh you can see here the solar port number has changed from 8080 to 8983 as i was expecting to see and then we just have more uh changed comments some things have been removed here because of the removal of support for amazon services in viewfind 7.0 but most of these changes are in comments so it's pretty safe to trust that they're not going to cause any problems i'll just keep scrolling and of course looking through these dips is also a great way to learn about new viewfinder features because any new thing that gets added will show up as a new set of options in the config files so uh it certainly pays to pay attention to these if you're doing this for real but right now i'm just going to move quickly past the rest of config.ini ah now i've reached my import settings where the only change is related to the solar port number change once again and this is so that the mark import tool knows where to find solar so we have two files with that port number changing uh we have some new examples added to our mark local.properties again it's all just comments so it won't hurt anything but it brings our local version up to date in case we want to turn this on in the future and finally a little bit of adjustment to our local custom theme to reflect some style changes that were made to the core and which have been automatically applied correctly here thanks to the merge script so that's it our viewfinder is now fully upgraded we just have one last thing to deal with which is the issue i raised earlier of the changed solar port number because right now we haven't done anything to solar during this upgrade process so it's still running on port 8080 where it used to live so first let's stop it by saying solar port equals 8080 system ctl stop do find this is taking advantage of the system d configuration that was set up in an earlier video so viewfind has now stopped but we are going to need to make one small adjustment to the system d configuration because the port number is embedded in the pid file that viewfind uses to keep track of running processes so we need to tell systemd that this 8080.pid has changed to an 8983 kid and because we made a change to a systemd file we need to say pseudo pseudo system ctl daemon reload to make sure that the latest version of that is all ready and then we can start the viewfind service once again there's just one important finishing touch which is of course that we want to commit all of the files that were customized by the merge scripts to get so that the whole history is tracked and we're ready to move on to our next set of changes when the time comes so let's just do a git status to look again at what has changed not too much so we can say git add local config local import theme tutorial do one more gift status to be sure all the right files were selected they were then we can git commit with a message like merge 7.0 changes to local files and we're all set and now if all has gone smoothly i should be able to refresh my viewfind homepage and if i view the source of it i see that my generator now says 7.0 instead of 6.1.1 that's a good sign and let's perform a search and confirm that yes the whole thing still works we are now upgraded to viewfind 7.0 so just to summarize upgrading viewfind with git is not an easy process it requires an understanding of your local system and configuration it requires some problem solving and resolving conflicts it's always good to have backups before you attempt to tackle it but as you become familiar with the tools of git and the supplemental merge script it does automate the vast majority of the work for you and draws your attention to key areas that might require additional work so i hope this demonstration has been of some help and will help you form useful habits to keep up to date on viewfind in the future and as always if you have problems or questions please reach out to me directly or to the viewfind mailing lists and i'll be happy to help thank you for your time