Warning: This page has not been updated in over over a year and may be outdated or deprecated.
videos:upgrading_vufind_using_git
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
videos:upgrading_vufind_using_git [2020/08/07 11:46] – [Related Resources] demiankatz | videos:upgrading_vufind_using_git [2023/04/26 13:30] (current) – [Transcript] crhallberg | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Video 10: Upgrading | + | ====== Video 10: Upgrading |
- | The tenth VuFind | + | The tenth VuFind® |
Video is available as an [[https:// | Video is available as an [[https:// | ||
Line 14: | Line 14: | ||
===== Transcript ===== | ===== Transcript ===== | ||
- | // Coming soon! // | + | With the recent release of VuFind 7.0, I thought now would be a good time to share my upgrade workflow to show how I use Git to upgrade VuFind. This video is going to assume that you have a basic familiarity with Git, and if you do not, I strongly, strongly recommend learning at least the basics. Doing software development and deployment without version control is like driving without a seatbelt, and you will find that investing a little time in learning these tools will save you all kinds of trouble down the road. |
+ | |||
+ | So this video will cover two basic topics. The first will be turning an existing installation of VuFind into a Git repository so that you can use Git to manage it, and the second part will show some processes and tools for using Git to perform actual upgrades. | ||
+ | |||
+ | So the virtual machine I've been using for this series of tutorials was initially set up by installing VuFind from the Debian package, which just puts the files on disk but does not include any kind of version control. We are going to take advantage of a useful characteristic of Git, namely that all of the version control data is stored in a dot Git subdirectory, | ||
+ | |||
+ | But before we can begin doing all of that, the most important thing we need to do first is figure out exactly which version of VuFind we are already running, because that will enable us to put Git into the correct state to start all of the work that we need to do. There are a couple of ways to identify which version of VuFind you're running. A simple one, particularly useful if you don't have access to the server running VuFind, is to simply view the page source while looking at any VuFind webpage, and look for the generator meta tag, which should have a version number embedded in it, so we can see here 6.1.1. However, the generator meta tag is not 100% reliable, because it's actually created based on a config.ini setting, and if that configuration file gets out of date or is customized, it's possible that what it's reporting is not actually the truth. If you want to be more confident that you have the right number, you should go to the command line, switch to your VuFind home directory, and edit the build.xml file. This is the control file used by the phing automation tool to do various VuFind related build tasks. Most users don't have to worry about it on a day-to-day basis, but it does have the useful characteristic of having a version number embedded in it. So if you scroll down until you find the property named version, you can see that once again this confirms we are working with you find 6.1.1. | ||
+ | |||
+ | Now that we know what version we're running, as I said, we can start creating a Git repository to turn this bare file collection into a tracked version controlled repository. And before I begin, I'm just going to confirm if I run a Git status from my VuFind home, it tells me it's not a Git repository. So let's go get one. | ||
+ | |||
+ | If we go to the temp directory, we can run git clone https github.com/vufind-org/vufind.git, and that will get us a copy of the full repository and all of the history of VuFind development. And while this takes time to download, I wanted to talk a little bit about how we use Git to track various versions of VuFind. First of all, mainline bleeding edge development takes place in a branch called dev. So if you want the latest code, the most up-to-date, the most current you would want to check out dev. But of course, bleeding edge code comes with risks because there may be some recently introduced bugs. And while we try to never break the dev branch, less tested code is always more risky. If you want more stable code, we have a series of release branches. So for every major or minor release, we create a branch named release dash and a number. So for example, release-7.0 or release-6.1. And these branches are where we do bug fixing on existing releases. So we'll never add a new feature to a release branch, but we will add fixes as needed. And it's these release branches that we use to issue bug fix releases that only include fixes even after development in the main dev branch has moved on to further new exciting territory. A final tool that you may find useful is that we also tag every release. So if you want the exact point in time for a particular VuFind release, you can check out a tag which is just going to be V and a number. So for example, V7.0 or V6.1.1. So if you want to get to a very precise state in the code, you can use a tag. And that will be useful, for example, in this situation where we're trying to sync up with a particular known version. But if you're trying to get the latest stable code on a particular thread of development, | ||
+ | |||
+ | What we can do is say '' | ||
+ | |||
+ | This is why it's important to be disciplined about separating your localizations from the core. And one of the advantages of using Git is that it helps you to do this because if you accidentally change a core file that will show up when you do a Git status and then you'll know oh I shouldn' | ||
+ | |||
+ | In any case once you're comfortable that you understand your local customizations in their scope the next thing I highly recommend doing is looking at the change log in the VuFind wiki and I will link to this from the video recording. But for every release of VuFind we include notes not only on new features that might be of interest but also of changes to code and configuration that could potentially cause problems during an upgrade. We are very inclusive here because we want to catch every possible issue that might be a problem. Most of these are unlikely to affect most users but that's why it's helpful to have a broad idea of what you've customized because you can then read through this list and take note of which issues are likely to be a problem and which ones you can very safely ignore. | ||
+ | |||
+ | In the case of this specific upgrade that we are about to run the one change log note that I need to be concerned about is this one. Starting with VuFind 7 we changed the default port that Solr runs on from 8080 to 8983. This is because 8983 is the standard port number used by Solr and using 8080 historically has caused port conflicts with other applications. It seems to make sense to standardize that but now at upgrade time we need to be aware of this change so that we can deal with it appropriately. Right now our Solr is running on port 8080 and after the upgrade VuFind will be looking for it on port 8983. I'll show you how to deal with that after the rest of the process and of course this particular issue only applies to VuFind 7 but this is just an example of the kind of thing you should be aware of when you're reviewing the change log. | ||
+ | |||
+ | Finally the third thing that you should always do before attempting an upgrade is back everything up and of course it's best to test an upgrade on a non-production server before you dive in in the real world. In this instance I'm not going to show you how I backed things up because this is a virtual machine and I've just backed up the whole disk image before I started so I can roll back if I have to but whatever your situation just be sure that you have a rollback plan so if something goes wrong during upgrade you haven' | ||
+ | |||
+ | With all that background out of the way we're just about ready to begin. There' | ||
+ | |||
+ | We changed the Solr ownership because the user running the Solr process needs to be able to write there but what we can do is work around this with a group ownership. I will leave Solr owned by Solr but I'm going to add it to the dkatz group which will give my user account permission to update files in that directory and I'm going to do that with the '' | ||
+ | |||
+ | So I am just going to '' | ||
+ | |||
+ | So in reality I usually have a pre written script for automating all the steps of the upgrade which includes removing the cache and recreating the command line cache. You would probably benefit from doing that as well but for now I just wanted to show those important steps. | ||
+ | |||
+ | So now we have updated code, we have updated dependencies, | ||
+ | |||
+ | I've written a bash script that does all of this work and in fact I've written a blog post explaining exactly how that works and the reasoning behind it. That's another link that I will include on the notes in this video. But for right now I am just going to skip to the answer and copy and paste my script code out of the blog post and into a file on disk. So I'm going to create a file called merge local.sh and I'm going to paste all my code into it. | ||
+ | |||
+ | Now the code I copied and pasted from my blog post is an example of how we do things at Villanova and it's a little bit out of date because I wrote this blog post a few years ago so it's going to need minor adjustments to be useful for us here. The idea is that this script defines a function called merge directory that takes two parameters. The first parameter is the path of a directory containing locally customized files. The second parameter is the core directory that includes equivalence to those files. So we've customized things for local harvest, local import, and local config VuFind so all of those defaults can remain as they are in the example. However our custom theme here is called tutorial so I'm just going to customize a couple lines of code here. So we are going to merge our local tutorial templates and JavaScript files with changes from the core bootstrap 3 theme on the assumption that any custom templates in tutorial that share names with templates from bootstrap 3 are copies of those that have been customized. | ||
+ | |||
+ | So with this file all ready to go I just need to make it executable so I can run the script. Now I'm ready to apply the upgrades from the core to my local files. One very important note about this script is that you have to run it immediately after you perform a merge because it works by looking at what changed in the most recent Git commit and if the most recent Git commit is not the merge to upgrade VuFind it won't know how to change your local files. | ||
+ | |||
+ | So it's always important to do this merge process right away after doing a merge to update the core. So I'm just going to run my script and then it's going to spew out a bunch of messages. Note that it complained about all of the metadata files under my local harvest directory because of course those have no equivalent in the core code because they' | ||
+ | |||
+ | It also reports that it ran into at least one conflict and conflicts are somewhat inevitable in any kind of different merging situation because what my script is essentially doing is saying I know that we started with a particular file in vufind 6.1.1 and when vufind 7 came out some changes were made to that file to upgrade to vufind 7. But you've also created a local copy where you've made some changes of your own. The script process will try to reconcile those changes as best it can but sometimes both paths of history will have tried to change the same part of the file and then of course there' | ||
+ | |||
+ | So first of all let's find out which files have conflicts in them. To figure out which files contain conflicts, we are going to run a get diff command which will output a list of all the changes to files that haven' | ||
+ | |||
+ | As you can see in this example we have three conflicts in config.ini. You'll notice that the file names we're seeing here actually refer to the core file so we also need to be able to figure out based on familiarity with our code which local file is the equivalent to that. In this case, I know that configuration files exist in my local directory and otherwise have an equivalent path so what I need to do is edit my local / | ||
+ | |||
+ | So for example, here is one conflict that's presenting me with three different sets of text. It's showing me what my original local file looked like, then it's showing me what the old vufind 6.1 file looked like, and then it's showing me what the vufind 7 file looks like and what I essentially need to do is pick one of these versions, delete all the other ones and get rid of the surrounding markers. | ||
+ | |||
+ | Sure, here's the text separated into sentences and paragraphs using punctuation: | ||
+ | |||
+ | This particular conflict occurred because at some point in the past, I upgraded this tutorial box to from vufind 6.0 to vufind 6.1, and I didn't do a very good job, so I introduced an inconsistency. So, I apologize for causing that confusion, but it did create an useful example here, and in this instance, it really doesn' | ||
+ | |||
+ | This obviously requires a bit of brainpower to get through. Sometimes it's not entirely obvious how to resolve a conflict, but as long as you understand that each section of the conflict block is showing you a different version of that chunk of code or configuration, | ||
+ | |||
+ | So, as I mentioned at the very top of the video, the generator value that you find uses is stored in config.ini, and we can see here that the upgrade process correctly updated that from 6.1.1 to 7.0. Then, I think most of the remainder of the dips we're going to see here are new settings or changed comments that took place during the course of you find development between the versions. You can see here the solar port number has changed from 8080 to 8983, as I was expecting to see, and then we just have more changed comments. Some things have been removed here because of the removal of support for Amazon services and you find 7.0, but most of these changes are in comments, so it's pretty safe to trust that they' | ||
+ | |||
+ | Right now, I'm just going to move quickly past the rest of config.ini. And now I've reached my import settings where the only change is related to the solar port number change once again. And this is so that the mark import tool knows where to find solar. So we have two files with that port number changing. We have some new examples added to our mark local dot properties. Again, it's all just comments, so it won't hurt anything, but it brings our local version up to date in case we want to turn this on in the future. And finally, a little bit of adjustment to our local custom theme to reflect some style changes that were made to the core and which have been automatically applied correctly here thanks to the merge script. | ||
+ | |||
+ | So that's it, our VuFind is now fully upgraded. We just have one last thing to deal with, which is the issue I raised earlier of the change solar port number. Because right now, we haven' | ||
+ | |||
+ | So VuFind has now stopped, but we are going to need to make one small adjustment to the system deconfiguration because the port number is embedded in the PID file that VuFind uses to keep track of running processes. So we need to tell systemd that this 8080 dot PID has changed to an 8983 PID. And because we made a change to a systemd file, we need to say pseudo systemctl demon reload to make sure that the latest version of that is already. And then we can start the VuFind service once again. | ||
+ | |||
+ | There' | ||
+ | |||
+ | And now, if all has gone smoothly, I should be able to refresh my VuFind homepage. And if I view the source of it, I see that my generator now says 7.0 instead of 6.1.1. That's a good sign. And let's perform a search and confirm that yes, the whole thing still works. We are now upgraded to VuFind 7.0. | ||
+ | |||
+ | So just to summarize, upgrading VuFind with git. It's not an easy process. It requires an understanding of your local system and configuration. It requires some problem solving and resolving conflicts. It's always good to have backups before you attempt to tackle it. But as you become familiar with the tools of git and the supplemental merge script, it does automate the vast majority of the work for you and draws your attention to key areas that might require additional work. So I hope this demonstration has been of some help and will help you form useful habits to keep up to date on VuFind in the future. And as always, if you have problems or questions, please reach out to me directly or to the VuFind mailing lists, and I'll be happy to help. Thank you for your time. | ||
+ | |||
+ | //This is an edited version of an automated transcript. Apologies for any errors.// | ||
---- struct data ---- | ---- struct data ---- | ||
+ | properties.Page Owner : | ||
---- | ---- | ||
videos/upgrading_vufind_using_git.1596800779.txt.gz · Last modified: 2020/08/07 11:46 by demiankatz