Table of Contents
This page is aimed at developers who want to test that their changes have not broken existing VuFind functionality or who are interested in creating standard tests for new VuFind components. If you are interested in testing performance rather than functionality, see the Testing Performance page instead.
The test modules provided with VuFind use the PHPUnit testing framework. The framework is installed as part of VuFind's Composer development dependencies. Once installed, you will have a vendor/bin/phpunit command line tool that you can use to run tests (as well as some convenient Phing tasks to make running the tasks more convenient).
These tests were designed for use with VuFind's continuous integration system. As such, some tests create and destroy data. Although some safety mechanisms exist to reduce the chances of accidental data loss, THE TESTS SHOULD NEVER BE RUN ON A PRODUCTION SYSTEM.
The testing process was developed under Linux; this is the recommended platform for VuFind testing, and different procedures may need to be developed for other platforms like Windows.
The easiest way to run VuFind's tests is with the help of the Phing build tool. VuFind comes with a build.xml file that Phing can use to automate tasks.
Recommended setup procedure
1.) Create a fresh copy of VuFind (i.e. git clone the repository to a fresh directory, then run “composer install”)
2.) Create a phing.sh script to automatically pass important parameters to Phing (see build.xml for other parameters that may be set here with the -D parameter):
#!/bin/sh $VUFIND_HOME/vendor/bin/phing -Dmysqlrootpass=mypasswd $*
If you are managing multiple VuFind test environments, it may make sense to have a different phing.sh in each VuFind directory (e.g. $VUFIND_HOME/phing.sh). In the more common scenario where you have just one test environment, it is usually more convenient to put this in your own home directory (i.e. ~/phing.sh). The examples below will assume that you are using a script in your home directory.
Running tests after setup
Follow these steps to run tests. Keep in mind that testing will create a test Solr index listening on port 8983. This may cause port conflicts if you test on a server that is already running applications on that port – plan accordingly.
1.) ~/phing.sh startup
This command will start up an instance of VuFind containing test data that may be used by some of the tests.
This command may overwrite any existing configuration etc.
2.) ~/phing.sh phpunit
(the phpunit command will run tests and generate report data for use by continuous integration; phpunitfast will run tests more quickly by skipping reports; phpunitfaster is like phpunitfast but stops upon the first test failure instead of going through the full suite)
3.) ~/phing.sh shutdown
This command will turn off the VuFind test instance created by step 1.
This command will reset the installation including configuration and any changes made to files in the git repository. Any uncommitted changes to files contained in the git repository will be lost.
The Faster Version
If you don't want to run integration tests using a running VuFind, you can simply bypass the startup/shutdown steps and only execute:
The integration tests will be skipped, but all unit tests not depending on a running VuFind instance will still run.
Running Just One Test
Sometimes you want to run a specific test without the rest of the suite. You can use the phpunit_extra_params parameter for this:
~/phing.sh phpunitfast -Dphpunit_extra_params=$VUFIND_HOME/path/to/your/test.php
Resetting the Setup
Available from VuFind 9.1
If a test, particularly one of the Mink tests described below, gets interrupted e.g. with ctrl-c, it can leave the configuration and/or database of the CI setup in a state that will cause further test runs to fail. In this case you can use the reset_setup command to reset the configuration and database:
Browser Automation Tests with Mink
Some of VuFind's tests are designed to use Mink to automate a browser. These tests will be skipped unless you have properly configured your environment so the tests have a VuFind instance they can drive.
If you wish to run this portion of the test suite, revise your phing.sh script (described above) to include some additional variables:
- extra_shutdown_cleanup - an extra command to run as part of the shutdown process; it's usually a good idea to change the ownership of the local/cache directory here since running tests will cause some files to be owned by Apache, and that will interfere with subsequent file deletion.
- apacheconfdir - a directory from which Apache will auto-load configurations; we need this to start up a new VuFind instance by injecting a file
- apachectl - the command to restart Apache; you'll probably need to prefix this with sudo to make it work when running tests as a different user
- dbtype (optional) - defaults to mysql, but can be set to “pgsql” to test with PostgreSQL instead
- mysqlrootpass (optional) - the MySQL root password (needed for building VuFind's database, when dbtype = mysql)
- vufindurl (optional) - the URL where VuFind will be accessed (defaults to http://localhost/vufind if omitted)
- mink_driver (optional) - the name of the Mink driver to use (e.g. “selenium” or “chrome”).
Here's an example script from an Ubuntu environment using MySQL:
#!/bin/sh phing -Dextra_shutdown_cleanup="sudo chown -R myusername:mygroup /path/to/vufind/local/cache" -Dapacheconfdir="/etc/apache2/conf-enabled" -Dapachectl="sudo /etc/init.d/apache2" -Dmysqlrootpass=myrootpasswd -Dvufindurl=http://localhost/vufind_test -Dmink_driver=chrome $*
You will also need to install and run a few things, depending on how you want to run your tests.
Selenium testing can be useful for testing a variety of browsers, but headless Chrome browsing is generally easier to set up; see below for details on that.
Before running your tests, you should download the Selenium server .jar file from here. It may also be called Grid. The Selenium Server needs to be started before you run phpunit with “java -jar [downloaded file]”.
If you want to use a browser other than Firefox with Selenium, use the
-Dselenium_browser=[name] setting to switch. For example, you can use
-Dselenium_browser=chrome, though this requires you to have the ChromeDriver installed on your system. Download the version that matches your installed Chrome version and place the unpacked file along your bin path, such as in
When running in a continuous integration environment, you should use Xvfb to create a virtual frame buffer so that the browser driven by Selenium can run without errors. However, for manually-executed tests, you can watch the browser executing the tests, which can be useful for troubleshooting purposes.
If you don't want to monitor test results or watch the Selenium server jar's output, you can have Phing automatically start up/shut down Selenium and Xvfb by passing the
-Dseleniumjar=[downloaded file] option when you startup and shutdown your test instance.
When running tests in a windowed environment, it's a good idea to open a couple of Terminal windows – that way you can run the Selenium server in one window (to watch it receive commands) while running your tests themselves in another (to see failures, etc.).
Headless Chrome Testing
Headless Chrome testing was introduced in VuFind 7.
Versions of Chrome 80+ have a built-in remote debugging feature. You can run Chrome instead of the Selenium Server like so:
google-chrome --disable-gpu --headless --remote-debugging-address=0.0.0.0 --remote-debugging-port=9222 --window-size="1366,768" --disable-extensions
Then, when you run the phpunit Phing task, add the parameter
-Dmink_driver="chrome" (or make sure this is set as a default in your phing.sh script).
This section contains some notes that may help you avoid problems when trying to run VuFind's test suite.
- Some of VuFind's tests use a record ID containing a slash. In order for these tests to work, your Apache installation needs to be configured with “AllowEncodedSlashes on” set in the VirtualHost used for running the VuFind instance being tested.
- When testing with PostgreSQL, it may be necessary to edit the pg_hba.conf file and change the line “local all all peer” to “local all all md5” to allow password-based database logins required my the test environment.
- If you encounter issues with the browser tests, you can try running Chrome without the –headless parameter to make the process visible. It's also possible to step through the tests with a PHP debugger if the XDebug extension is installed while checking the results in the visible window. There are a couple of prerequisites for the tests to work properly when running with a visible Chrome window:
- Disable Chrome's “Warn you if passwords are exposed in a data breach” in Privacy and Security settings while running the tests so that the password checks don't interfere with patron tests.
- Set English (US) as the preferred language.
Mink Tests with Chrome on macOS
Running Chrome Headless
A command to start Chrome before running the tests:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --disable-gpu --remote-debugging-address=0.0.0.0 --remote-debugging-port=9222 --window-size=1600,900 --headless
Performance Issues with Headless Chrome
Note that on macOS, headless Chrome is slow. Running without the –headless parameter is much faster, but since Chrome takes focus every time a new tab is opened, this will interfere with any other work.
You can disable the focus stealing by adding the following lines to /Applications/Google Chrome.app/Contents/Info.plist right after the first <dict> tag:
Note: making the change above will break the application's code signature, which may affect the behavior, automatic updates etc.
Required GNU Utilities
Note: This applies only to VuFind versions before 8.0.
VuFind's Mink tests up to version 7.x rely on options available only in GNU versions of find, sed and basename, so you will need to install those versions and make sure they are used when running Mink tests on macOS. One option is to use Homebrew to install them:
brew install findutils gnu-sed gnu-basename
Then prefix the phing command in phing.sh with a path that contains these utils. Here is an example that uses a static Apache configuration and mysql with passwordless authentication (both installed with Homebrew):
PATH="/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/opt/findutils/libexec/gnubin:$PATH" $VUFIND_HOME/vendor/bin/phing -Dmysqlrootuser=$LOGNAME -Dmysqlrootpass="" -Dvufindurl=http://localhost:8080/vufind-test -Dmink_driver=chrome $*
It's possible to run Mink tests so that HTML of the final page of each test is validated. You will need to run NU Validator locally for this. NU can be downloaded from https://github.com/validator/validator/releases/. Get the vnu.jar package, e.g. vnu.jar_20.6.30.zip. Unzip and start for example with the following command:
java -cp ./vnu.jar -Dnu.validator.servlet.bind-address=127.0.0.1 nu.validator.servlet.Main 8888
Then run the tests with some extra parameters to enable validation:
./phing.sh phpunitfast -Dhtml_validator=http://localhost:8888/ -Dhtml_validator_fail_tests=0 -Dhtml_validator_log_file=html_validation.log
Results will be written into html_validation.log. It is currently (April 2023) not feasible to use html_validator_fail_tests=1 since there are still some known issues.
Starting with VuFind version 9.1 it's possible to collect code coverage data from Mink tests. It is worth noting that gathering coverage data will slow down the tests quite significantly.
- Apache must be running on the same server that runs the tests and be able to write to the tmp build directory
- Remote profiling enabled with
SetEnv VUFIND_CODE_COVERAGE 1in Apache config (see config/vufind/httpd_vufind.conf for more information)
Note that enabled PCOV or XDebug in coverage mode will prevent normal debugging with XDebug.
To run tests with coverage enabled, use the phpunit task with the
-Dphpunit_remote_coverage=1 option, e.g.
./phing.sh phpunit -Dphpunit_remote_coverage=1. As is the case with normal unit tests, the coverage reports are written to $TMP/build/vufind/reports/coverage directory. Coverage reports are not available with
VuFind unit tests (which check the functionality of individual components) can be found in the module/VuFind/tests/unit-tests directory of your VuFind installation.
VuFind integration tests (which check the interaction of components in a real running system) can be found in the module/VuFind/tests/integration-tests directory of your VuFind installation.
VuFind support modules (VuFindHttp, etc.) have their own tests directories, but these are configured to be run automatically as part of VuFind's main test suite.
The layout of the each test directory is designed to mirror the VuFind module's src directory (module/VuFind/src/VuFind). Tests should be placed in a directory that corresponds with the component being tested. For example, the unit tests for the \VuFind\Date\Converter class are found in “module/VuFind/tests/unit-tests/src/VuFindTest/Date/ConverterTest.php”.
The VuFindTest namespace (code found in module/VuFind/src/VuFindTest) contains several classes and traits which may be useful in writing new tests – a fake record driver in VuFindTest\RecordDriver, some helpful traits in VuFindTest\Feature, and some base test classes containing reusable patterns in VuFindTest\Unit.
When working on integration tests, pay special attention to the VuFindTest\Feature\Live* traits (introduced during VuFind 8 development), which provide helpful methods for retrieving services that provide access to the real database and Solr instances for tests where actual data needs to be manipulated. Such tests should be written with caution, and should only be run in a safe test environment, as noted elsewhere.
Mink Browser Automation
If you want to write browser automation tests, you should extend \VuFindTest\Integration\MinkTestCase. This class will automatically set up the Mink environment for you. It also provides a changeConfigs() method which can be used to reconfigure the running VuFind instance by modifying its .ini files. Any changes made by this method will be automatically rolled back at the end of the test, so you can test a variety of configurations in a single test class.
Some example Mink tests can be found in module/VuFind/tests/integration-tests/src/VuFindTest/Mink/.
Guidelines for Writing Mink Tests
Here are some guidelines that help create tests that are robust and as fast as possible. MinkTestCase class has several methods that are recommended for interacting with the page. Many of the methods have retry or timeout logic to make sure they are as robust as possible.
- Use hard-coded snooze only if absolutely necessary.
- Wait for required conditions to be met with
- Wait for something to go away with
clickCss()instead of find + click.
assertEqualsWithTimeout()to wait for a condition to be met and asserted.
- Close lightbox without further action with
- Wait for lightbox to close with
waitForLightboxHidden()after performing an action.
assertLightboxTitle()to make sure a correct lightbox is being displayed.
waitForPageLoad()if you need to make sure that the page has loaded before continuing (including any jQuery “ready” event handlers). Useful also when you look for an element that may exist also on the previous page.
Create a directory in your local module mimicking the current structure of the VuFind tests. The test suite should automatically detect all tests within modules under the VuFind directory.