This recording, from the April 18, 2023 Community Call, highlights new features and significant changes in the VuFind® 9.0 release.

Video is available as an mp4 download or through YouTube.

Okay. So, welcome, everyone, to the presentation on what's new and what's changed in VuFind 9.

We'll start with the high-level stuff. We'll get into the nitty-gritty details. And hopefully, most of this will not apply to everyone, but I just want to be sure that we all are aware of what's coming. So, first of all, the exciting things about VuFind 9.0 are major new features. Probably the biggest new feature is what's called blended search. And there was a video recorded about this in detail at the last VuFind summit, which you can find on YouTube. So, I'm not going to talk about it in depth now. But the upshot of this is that it allows you to connect multiple search backends and create a single result list from them with merged facets. This is something people have been asking for for a long time. And now we can offer it.

Another significant new feature is a cookie consent dialog box. This is off by default but can be easily enabled through configuration.

Another one is the ability for users to leave star ratings on records. Either in conjunction with comments or independently. Again, this feature is off by default, but it's a new social setting that can be turned on. We also have a few minor new features.

We have something called the interval capture handler. This puts a time-based limitation on certain actions. So, rather than providing a visible control that people need to interact with to indicate that they're not a robot, instead, this just puts time limits. So, for example, you can say a user can only send an email once every five minutes. And then if they try to send an email sooner than that, they just get a message saying, you've got to wait a little longer.

We also now have configurable user name policies so you can control what characters are allowed in user names. And by default, we have a very broad policy that just disables really bizarre special characters. Hopefully, this should not impact any legacy users in any real-life systems. But it also hopefully protects against weird behavior that we don't want to allow. And you can change it if you don't like it. Also some significant improvements to existing functionality.

We've done quite a bit of work on the feedback form. One note is that historically if you turn on feedback forms, they just send emails. But now there's an option in the feedback form configuration to also persist form results to the database. And there's a new module in the admin area where you can browse through the saved results of the feedback form and also set workflow-based statuses on them so you can indicate, yes, we've acted on this one or this one is pending. So, again, don't have to use it. But it's a useful new feature for some people.

Additionally, there's a new configuration setting that allows you to prepopulate certain form fields within feedback forms by using query parameters. So, for example, if you want to send out a URL that prepopulates a particular field with a particular value so that people don't have to completely fill out a form, you now have the ability to do that if you turn it on. And do keep in mind that if you've locally customized feedback form code, some of these changes might have an impact on your local code. So, please review.

There have also been some changes to some search backends. The EDS and Primo backends now support facet value filtering. So, if you want to not display certain specific facet values, you can turn that on. Additionally, those two backends also now support search response caching. So, if you want to speed things up, you can save search responses locally and reduce the number of API calls that VuFind is making. Also, some more minor improvements. The staff view can now be configured so that when you're looking at the staff view on a MARC record, you can also turn on the ability to look at the raw Solr fields, which can be helpful if you want to see how the MARC data is getting mapped to Solr.

There's a new security-related setting that can be used to completely disable the ability for users to provide their ILS credentials through a form. And this is useful in scenarios where you're preloading ILS credentials from your single sign-on attributes. So, like you're loading the user's catalog username directly into the database and you don't want any possibility of somebody somehow getting to that logging screen and entering credentials that don't belong to them. But, of course, you don't want to turn this on if you actually rely on people entering their ILS credentials.

Another small improvement is that when alphabetic browse results are clicked and lead to search results, those results no longer automatically apply default filters because that could have caused some inconsistencies where the browse results say that a certain number of records match the result and then you click on it and you don't get anything because it's filtered out by a default filter. So, this is more consistent. But, again, this behavior can be configured if you have a strong reason for it to act differently.

We've also added an API endpoint that can be used to automate the clearing of caches, which can be useful if you want to do that as part of, say, an overnight process. We've added configuration file inheritance to YAML files. So, you know, for a long time you've been able to inherit settings from one any file to another. Now you can do similar things in YAML files.

We've also added and updated some integrations with third-party systems in VuFind9. One new feature is that you can display cover images that are stored locally in Koha if you're using Koha.

We've updated some Google-related tooling so you can now easily incorporate Google Tag Manager. And we've added support for Google Analytics version 4 since Google Universal Analytics is getting shut down this summer.

We've added support for Marc in JSON records in the Solr index. So, now you can do binary Marc, Marc XML, or Marc in JSON. We've added retraction notice support in the Browsine DOI linker. So, now if you have Browsine turned on and appropriately configured and you have records with DOIs, VuFind can automatically detect articles that have been retracted and display messages in line about that fact. You just have to turn it on in Browsene.ini if you want it.

And we've added support for using VuFind as an OAuth2 or OpenID Connect authorization server, which can be useful for interacting with some third-party systems.

We've renamed the EZB link resolver driver to JOP. This is sort of a more accurate name for this third-party service. I think this is primarily used in Germany.

And finally, we've added an example import configuration for the subjects plus subject guide tool, which is an open-source content manager that supports OAI PMH. So, now you can ingest that content into VuFind very easily. So, that ends all the things that are new.

So, now let's talk about all the things that have changed. So, dependencies, the endless march of change continues. So, first of all, we've raised the minimum PHP version to 7.4.1. I think this is an uncontroversial change because PHP 7 has hit end-of-life and everyone really ought to be using PHP 8 by now. But in the interest of making progress incrementally, we didn't raise the version all the way to 8 yet. But we will do that in release 9.1. So, be aware, next release is going to require PHP 8. I strongly advise being ready for that. But if it's taking a little bit more time to get there, VuFind 9 will still work with PHP 7.4. We've also dropped support for Apache 2.2, which has been end-of-life forever. So, again, I don't expect this is going to inconvenience anyone. It's just that the default VuFind Apache configuration had a whole lot of conditional logic to detect whether you were using Apache 2.2 or 2.4. It really made sense to simplify that at this point since everyone should be using Apache 2.4 anyway. So, I don't expect this to negatively impact anyone. But keep in mind that you can potentially simplify your own Apache configuration now if you want to.

We've also upgraded Solr to Solr 9, specifically Solr 9.1.1. And this is a significant Solr upgrade. You absolutely need to reindex all your records when you make this upgrade because there are some changes to the way highlighting works. And if you just point Solr 9 at an index that was built with VuFind's Solr 8 schema, you're going to get fatal errors if you do anything related to the highlighting of authors. So, be aware of that. Plan for it. Get ahead of it. I think you might be able to work around it by disabling highlighting, but that's obviously not ideal. Another important change is that Solr 9 now only binds to localhost by default. So, if you're running Solr across servers, accessing it through a particular hostname, you need to set the Solr JETI host environment variable to tell Solr what name to bind to, or else it's not going to work. So, that's a couple of major gotchas to be aware of related to the Solr upgrade.

Other dependency changes. Open SSL 3.0, which is being included in more and more distributions. It no longer supports Blowfish encryption by default, but Blowfish encryption is what VuFind has been using for a long time to encrypt ILS credentials unless configured otherwise. So, you're going to need to change that or things will stop working. But the web-based upgrade tool that most people use as part of every upgrade will detect the situation and guide you through what you need to do to fix it. So, shouldn't be a big problem. Just be aware that this is a thing you will have to do either now or probably in the near future.

Also significant is that Laminas has updated itself to use the PSR container library instead of container interop. So, PSRs are PHP community standards for interoperability and the standard for containers like the Laminas service manager has been around for a while, but it was predated by a non-official sort of interim version. And that's what's happening here. We're updating from the interim version to the real standard. This just means that any code that refers to the interop container namespace needs to be changed to PSR container. And one slight exception to that is interop container exception. That class becomes PSR container, container exception. So, the class name gains the word container. Anyway, this change has been made throughout VuFind, but if you have local custom factories, there's a good chance you're going to need to update them just to make these minor namespace changes. So, be prepared for that.

We have made some updates to our dependencies. We have upgraded our common Marc library that we use for rendering Markdown. If you use Marcdown, please check Marcdown.ini and any custom code you have made to ensure that you are taking advantage of new features and that nothing has gotten out of date. Additionally, all of our code dealing with Marc records has been revised. We have moved away from PAIR, the old PHP code-sharing platform, because it was holding us back. As a result, we have developed our own local custom Marc library called VuFind Marc, which does all the same things as the old file Marc library, but in a more modern way. If you have local custom code that uses file Marc to interact with Marc records, you will need to revise it to use the new Marc library instead. We have also updated the API specification to use the OpenAPI 3.0.3 standard. This doesn't change anything about how the API works. All the endpoints and parameters remain the same, but the published description of the API has been modernized.

Moving on to other changes, there are some updates in ILS drivers. The get config method, which reports the capabilities of the driver to the VuFind code, has been aligned across all the drivers to avoid weird problems related to inconsistency. Additionally, the abstract API base class has had its make request methods signature changed, which currently only impacts the Folio and genie plus drivers. If you have customized either of those drivers and you're making requests, make sure that those calls match the new method signature. We have removed the old Sierra driver as it was out of date, and we now use the API-driven one instead.

Lastly, we have made some changes to indexing. We now support Marc in JSON storage of records in the Solr index by default instead of using binary Marc, which has a problem of dropping anything over 100,000 bytes and resulting in an invalid record. We used binary Marc in the past as it takes up less disk space than Marc XML, which is very verbose.

Marc in JSON is a compromise between binary Marc and Marc XML - it's not as concise as binary Marc, but it's more concise than Marc XML, and avoids the length limit. By default, VuFind's Solr configuration will index everything in Marc in JSON, but this can be changed. The format calculator class has been significantly refactored to improve accuracy and add new capabilities, so if you have a custom subclass of it, you may need to update it to ensure compatibility. Some old shell and batch scripts in the import bin subdirectory have been deleted.

Record drivers now differentiate between source identifier and search backend identifier, which is useful for Blender, a meta backend that talks to multiple sources. The set source identifier method has been deprecated and replaced with set source identifiers, and custom record controllers may need to change their property names. The holdings display no longer hides item data for records with no barcode, and if you want to hide them, you can set display items without barcodes to false.

Some changes have been made to authentication classes due to the new username policy, and the VuFind validator CSRF class has been renamed to session CSRF and an interface created for alternate implementations of how CSRF works in the future.

Now, onto search-related changes. VuFind now passes around a new query parameter called SID on many of its URLs, which tracks the ID of the last search the user performed. This allows VuFind to consistently keep track of its state. Previously, the search ID was only stored in the session, which could result in strange behavior, particularly when a user had multiple tabs open and was performing actions in different tabs simultaneously. Although it is still not perfect and there may be some edge cases that cause issues, this update should make the system much more stable. Implementing this required changes to several classes, including the VuFind Record Tab Collection List, the Results Scroller Controller Plugin, and the VuFind Search Memory Class. If you have customized these components, please check that your customizations still work.

There has also been some refactoring of the hierarchical facet processing logic, which was previously built into some of the search results classes. This logic has now been moved out into a separate helper class to make it more reusable and reduce the size of individual classes. The way configuration files are loaded in the factories for Solr search backends has been changed, making things more consistent and simplifying some code. If you have a custom Solr search backend factory extending Abstract Solr Backend Factory, you may need to make some adjustments. The hide facet value listener in the Solr area of the code has been moved to the base area of the code because it does not contain Solr-specific logic and is now being shared between multiple backends. This is a minor change.

Moving onto search backend-specific changes, the internal format that VuFind uses to represent facet data has been standardized across the Solr, EDS, and Primo backends. Previously, each of these backends represented facets in their own unique ways, but standardizing them enables the blender to operate more straightforwardly. If you have custom EDS or Primo code working with facets, be aware that the data structures have changed slightly, so you may need to update them. However, we did not have the opportunity to standardize the facet data format for Summon, so it still has its own unique format and is not yet compatible with the blender. If you want to blend Summon results, some additional work is needed, but the community will be happy to work with you on that.

Lastly, the Solr connector class has been revised, so if you are directly accessing the Solr connector in any custom code, please check that it still works. Additionally, factory and constructor expectations for the EDS search options have changed to support on-demand retrieval of data from the EDS API. Previously, VuFind would make API calls every time it constructed EDS objects, potentially causing overhead and performance issues. Now, the code has been refactored to request data only when necessary, which should improve performance.

Now moving on to internationalization changes. We've done some work to clean up some of the language files. And just a handful of translation strings have either been removed because they're not needed, renamed for clarity, or split into multiple versions so that they can be translated differently in different contexts.

This slide shows a list of all the major changes. I'm not going to read it all to you. I will share this slide deck after the presentation. And all these notes are also in the changelog in the Wiki. So, if you're concerned about the details, feel free to take a look.

But the point is, if you use any of these translation strings in local custom code or templates, you'll need to make adjustments to keep everything up to date and ensure that it still gets translated.

Now on to theme-related changes. So, here's a big one. In VuFind 8, we introduced an icon system that allows us to render icons in a standard way and override them through configuration so you can change what icon is displayed in a given context just through a config change in your theme configuration.

But in VuFind 8, we only use this feature in a small number of places. In VuFind 9, every icon in the themes is using the new icon system. So, it is widespread. There are a few cases where icons have changed just because while reviewing it, we found a better option. So, you may notice some minor visual changes.

Also, the CSS around icons has been significantly revised. Particularly in cases for displaying a link containing both an icon and text. Historically, in this situation, VuFind had some pretty ugly displays where underlining of links, you have a weird sort of floating underline between the icon and the text or things wouldn't line up right or whatever. And we've added some more explicit markup and styling to make everything look more uniform and clean.

But this required a lot of changes to a lot of templates. So, be aware of that when you're updating. And if you have any custom links with icons in them, you might need to style them to match the new markup style of the core.

Also, minor change, the icon view helper as defined in VuFind 8 had a dependency on the head link helper. We got rid of that in VuFind 9. So, it's all a little bit simpler. I doubt anyone has their own custom local icon view helper. But if you do, be aware of that change.

Additionally, the theme info class used in the VuFind theme module has a get merged config method that's used to retrieve theme configuration. And it had a parameter called flatten, which I don't know why it was added. I don't know what it was supposed to do. But it never worked right. It didn't really do anything. And so, we got rid of it in VuFind 9. So, since it didn't work and didn't seem useful, hopefully nobody is using it for anything and won't miss it.

But if somehow you're doing something weird custom with get merged config, be aware of this change.

Another important but minor change is that in the account area, in the sidebar menu, if you have the AJAX status setting turned on, you'll get little notifications about things like overdue books or things ready for pickup. The CSS classes used to display some of those counts have been renamed to be a little bit more descriptive. Previously, it was okay, warn, and overdue. They have been changed to account info, account warning, and account alert, to be more general and specific at the same time. So, if you've customized those, anything related to that, just be aware of these name changes.

On the theme front, currency formatting has been refactored to its own service. Previously, the get user finds AJAX handler was using the safe money format view helper, but that was a weird use of a view helper. So, now the currency formatting has its own class and that class is shared between the things that need to do formatting of currency. That transitions nicely into other changes related to AJAX handlers. One small thing is the get user finds AJAX handler previously returned a key called value. This has been changed to total to make it a little bit more precise and consistent with other user status AJAX handlers.

The DOI lookup AJAX handler and the browsing DOI linker plugin have both had constructor signatures and factories changed. This is related to the new browsing functionality I mentioned earlier. So, that's all of those changes.

Let's talk about things that have been deprecated or removed. One important thing is that the PHP-based less compiler, which was deprecated before, is now fully removed. If you were compiling less into CSS with PHP, you were getting significantly different results than if you were using Node. And that was a problem. So, we just got rid of it to prevent people from getting confused. Use grunt less for now and that's going to be NPM run something in the near future.

We've also gotten rid of the hunt library, which was used to detect when the user scrolled and certain areas needed to have statuses plugged into them. But the native intersection observer does the same thing. We don't need a third-party library to detect when areas come into view anymore. So, hunt is gone. So, if you used hunt for something local, just be aware you can do that in a cleaner way now.

Additionally, the controller plugin abstract request base get valid IDs method has been replaced by a validate IDs method. This affects how we confirm that people are passing in valid IDs when they place holds or other types of requests. Again, not a thing that I think many people are likely to have changed. But if somehow you have, be aware of this. Additionally, all code reported as deprecated in VuFind 8 has been fully removed in VuFind 9.

So, the ILS driver cache trait has been removed. You should use the more generic cache cache trait. The record link view helper is gone. Use record linker instead. And the VuFind search services all use the command pattern for every call. There were a bunch of more specific back-end methods like search and retrieve and so forth. Those are all gone. You have to use the command pattern now. But we had already refactored to use the command pattern in VuFind 8. So, hopefully, this is not a shock for anyone.

Now, some database related changes. First of all, the database schema has grown because we've added some new features like the ratings. So, you'll need to run a database upgrade to just add all the new stuff to your local database. This is part of the web based update process, which as I mentioned, I think everyone should already be doing anyway.

The VuFind DB table save search method has been changed. There was a bug that was causing duplicate searches in search history. And in order to fix it, we had to make some low level changes, including this minor method signature change.

Additionally, there's a VuFind DB table expiration trait, which is shared by a number of command line tools for expiring outdated data in the database like old searches, for example. And that trait had to be revised to fix a problem that would allow it to go into an infinite loop. So, if you built anything custom relying on this trait, just be aware it's changed a little bit.

Finally, some changes to VuFind's styles and tooling that we used during development. These don't really affect the code itself, but they do affect the ecosystem. So, it's good to be aware of them.

First of all, we've modernized our code style standards to use PSR 12 instead of the pair standard. Really not a whole lot has changed. But there are a few minor things, particularly white space related that are a little bit different. So, if you use VuFind's tools to apply automatic style fixes to your local code, just be aware that you might get a lot of changes when you merge in the 9.0 update because we've changed our styles. Additionally, some of our function signatures now have more specific typing on them. This was based on using static analysis through PHP Stan. It had some recommendations for fixes.

So, I strongly recommend running PHP against local code. It should detect if any of your local code has signatures that are out of sync with upstream code. And it's a great tool to find a variety of common problems as well. Also significant is we've added the local directory to our get ignore file just to prevent people from accidentally committing local code and pushing it upstream if they are using the default local directory.

So, if you actually rely on committing local code to Git as at Villanova we do for our local private repository, just be aware that you'll need to override that setting to tell Git that you really do want to commit local files. It's just now opt in instead of opt out. But it's not hard to do. If you need help with it, let me know. I can give you the details.

Finally, we've been making progress on improving the validation of the HTML that VuFind generates. One suggestion was to not include trailing slashes in HTML void elements. For example, a BR tag should just be BR, not BR slash. The slashes in void elements were a big thing in XHTML because they're needed for XML validation. But stylistically, that's not popular in HTML5. So we've gotten rid of it to make things consistent in our templates.

There are a few other miscellaneous things to be aware of. One important change is that previously, VuFind changed the present working directory to VuFind home as part of its bootstrapping process. We've gotten rid of it, but if you rely on VuFind home being the present working directory in any of your custom code, that's no longer something you can assume. You should use more explicit paths. Additionally, there was a relative file-aware command base class that was shared by some of our command-line tools. That became obsolete when we stopped doing the change directory. So if you're extending that for a custom tool, be aware that you need to make some changes.

Non-TAB record action handling in the route generator has been refactored to make it easier to add additional actions as part of local modules. If you've generated some custom record routes, you might want to look at your locally generated route code and you might be able to simplify it a little bit.

I just wanted to share a few resources. I recorded a video a couple of years ago about how you can do a VuFind upgrade using Git to automate some of the processes, like updating your configuration and local templates. You might find that helpful if you're about to dive into a major update to 9.0. Additionally, this blog post talks about the scripting that I used in the video. It provides a way to automatically apply core changes to equivalent local files, and even though it likely will require some manual conflict resolution, it's nice to know exactly which of your local files are impacted by upstream changes.

If you have any questions or comments, we can talk about them now. If you want to reach out to me later, you can always email me or find me on the VuFind Slack. I'm happy to help in any way that I can.

—

This is an edited version of an automated transcript. Apologies for any errors.