About Features Downloads Getting Started Documentation Events Support GitHub

Love VuFind®? Consider becoming a financial supporter. Your support helps build a better VuFind®!

VuFind Documentation

Site Tools

You are here: start » development » testing » unit_tests
Trace:
Warning: This page has not been updated in over over a year and may be outdated or deprecated.
development:testing:unit_tests

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
development:testing:unit_tests [2022/02/14 11:50] – [Mink Tests with Chrome on macOS] emaijaladevelopment:testing:unit_tests [2023/11/28 18:38] (current) – demiankatz
Line 1: Line 1:
 ====== Unit Tests ====== ====== Unit Tests ======
  
-//This page is aimed at developers who want to test that their changes have not broken existing VuFind functionality or who are interested in creating standard tests for new VuFind components.  If you are interested in testing performance rather than functionality, see the [[administration:testing_performance|Testing Performance]] page instead.//+//This page is aimed at developers who want to test that their changes have not broken existing VuFind® functionality or who are interested in creating standard tests for new VuFind® components.  If you are interested in testing performance rather than functionality, see the [[administration:testing_performance|Testing Performance]] page instead.//
  
 ===== Background ===== ===== Background =====
  
-The test modules provided with VuFind use the [[http://www.phpunit.de/|PHPUnit]] testing framework.  The framework is installed as part of VuFind's [[development:recommended_tools:composer|Composer]] development dependencies. Once installed, you will have a vendor/bin/phpunit command line tool that you can use to run tests (as well as some convenient [[#using_phing|Phing]] tasks to make running the tasks more convenient).+The test modules provided with VuFind® use the [[http://www.phpunit.de/|PHPUnit]] testing framework.  The framework is installed as part of VuFind®'s [[development:recommended_tools:composer|Composer]] development dependencies. Once installed, you will have a vendor/bin/phpunit command line tool that you can use to run tests (as well as some convenient [[#using_phing|Phing]] tasks to make running the tasks more convenient).
  
 ===== Running Tests ===== ===== Running Tests =====
Line 11: Line 11:
 ==== Important Warnings ==== ==== Important Warnings ====
  
-These tests were designed for use with VuFind's [[development:testing:continuous_integration|continuous integration]] system.  As such, some tests create and destroy data.  Although some safety mechanisms exist to reduce the chances of accidental data loss, **THE TESTS SHOULD NEVER BE RUN ON A PRODUCTION SYSTEM.**+These tests were designed for use with VuFind®'s [[development:testing:continuous_integration|continuous integration]] system.  As such, some tests create and destroy data.  Although some safety mechanisms exist to reduce the chances of accidental data loss, **THE TESTS SHOULD NEVER BE RUN ON A PRODUCTION SYSTEM.**
  
-The testing process was developed under Linux; this is the recommended platform for VuFind testing, and different procedures may need to be developed for other platforms like Windows.+The testing process was developed under Linux; this is the recommended platform for VuFind® testing, and different procedures may need to be developed for other platforms like Windows.
  
 ==== Using Phing ==== ==== Using Phing ====
  
-The easiest way to run VuFind's tests is with the help of the [[http://www.phing.info/|Phing]] build tool.  VuFind comes with a build.xml file that Phing can use to automate tasks.+The easiest way to run VuFind®'s tests is with the help of the [[http://www.phing.info/|Phing]] build tool.  VuFind® comes with a build.xml file that Phing can use to automate tasks.
  
 === Recommended setup procedure === === Recommended setup procedure ===
  
-1.) Create a fresh copy of VuFind (i.e. git clone the repository to a fresh directory, then run "composer install")+1.) Create a fresh copy of VuFind® (i.e. git clone the repository to a fresh directory, then run "composer install")
  
 2.) Create a phing.sh script to automatically pass important parameters to Phing (see build.xml for other parameters that may be set here with the -D parameter): 2.) Create a phing.sh script to automatically pass important parameters to Phing (see build.xml for other parameters that may be set here with the -D parameter):
Line 30: Line 30:
 </code> </code>
  
-If you are managing multiple VuFind test environments, it may make sense to have a different phing.sh in each VuFind directory (e.g. $VUFIND_HOME/phing.sh). In the more common scenario where you have just one test environment, it is usually more convenient to put this in your own home directory (i.e. ~/phing.sh). The examples below will assume that you are using a script in your home directory.+If you are managing multiple VuFind® test environments, it may make sense to have a different phing.sh in each VuFind® directory (e.g. $VUFIND_HOME/phing.sh). In the more common scenario where you have just one test environment, it is usually more convenient to put this in your own home directory (i.e. ~/phing.sh). The examples below will assume that you are using a script in your home directory.
  
 === Running tests after setup === === Running tests after setup ===
Line 38: Line 38:
 1.) ~/phing.sh startup 1.) ~/phing.sh startup
  
-This command will start up an instance of VuFind containing test data that may be used by some of the tests.+This command will start up an instance of VuFind® containing test data that may be used by some of the tests. 
 + 
 +:!: This command may overwrite any existing configuration etc.  
  
 2.) ~/phing.sh phpunit 2.) ~/phing.sh phpunit
Line 54: Line 56:
 3.) ~/phing.sh shutdown 3.) ~/phing.sh shutdown
  
-This command will turn off the VuFind test instance created by step 1.+This command will turn off the VuFind® test instance created by step 1. 
 + 
 +:!: This command will reset the installation including configuration and any changes made to files in the git repository. Be sure you save/commit anything important BEFORE you shut down.
  
 === The Faster Version === === The Faster Version ===
  
-If you don't want to run integration tests using a running VuFind, you can simply bypass the startup/shutdown steps and only execute:+If you don't want to run integration tests using a running VuFind®, you can simply bypass the startup/shutdown steps and only execute:
  
 ~/phing.sh phpunitfast ~/phing.sh phpunitfast
  
-The integration tests will be skipped, but all unit tests not depending on a running VuFind instance will still run.+The integration tests will be skipped, but all unit tests not depending on a running VuFind® instance will still run.
  
 === Running Just One Test === === Running Just One Test ===
Line 69: Line 73:
  
 ~/phing.sh phpunitfast -Dphpunit_extra_params=$VUFIND_HOME/path/to/your/test.php ~/phing.sh phpunitfast -Dphpunit_extra_params=$VUFIND_HOME/path/to/your/test.php
 +
 +=== Resetting the Setup ===
 +
 +:!: Available from VuFind® 9.1
 +
 +If a test, particularly one of the Mink tests described below, gets interrupted e.g. with ctrl-c, it can leave the configuration and/or database of the CI setup in a state that will cause further test runs to fail. In this case you can use the reset_setup command to reset the configuration and database:
 +
 +~/phing.sh reset_setup
 ===== Browser Automation Tests with Mink ===== ===== Browser Automation Tests with Mink =====
  
-Some of VuFind's tests are designed to use [[http://mink.behat.org|Mink]] to automate a browser. These tests will be skipped unless you have properly configured your environment so the tests have a VuFind instance they can drive.+Some of VuFind®'s tests are designed to use [[http://mink.behat.org|Mink]] to automate a browser. These tests will be skipped unless you have properly configured your environment so the tests have a VuFind® instance they can drive.
  
 If you wish to run this portion of the test suite, revise your phing.sh script (described above) to include some additional variables: If you wish to run this portion of the test suite, revise your phing.sh script (described above) to include some additional variables:
  
   * extra_shutdown_cleanup - an extra command to run as part of the shutdown process; it's usually a good idea to change the ownership of the local/cache directory here since running tests will cause some files to be owned by Apache, and that will interfere with subsequent file deletion.   * extra_shutdown_cleanup - an extra command to run as part of the shutdown process; it's usually a good idea to change the ownership of the local/cache directory here since running tests will cause some files to be owned by Apache, and that will interfere with subsequent file deletion.
-  * apacheconfdir - a directory from which Apache will auto-load configurations; we need this to start up a new VuFind instance by injecting a file+  * apacheconfdir - a directory from which Apache will auto-load configurations; we need this to start up a new VuFind® instance by injecting a file
   * apachectl - the command to restart Apache; you'll probably need to prefix this with sudo to make it work when running tests as a different user   * apachectl - the command to restart Apache; you'll probably need to prefix this with sudo to make it work when running tests as a different user
   * dbtype (optional) - defaults to mysql, but can be set to "pgsql" to test with PostgreSQL instead   * dbtype (optional) - defaults to mysql, but can be set to "pgsql" to test with PostgreSQL instead
-  * mysqlrootpass (optional) - the MySQL root password (needed for building VuFind's database, when dbtype = mysql) +  * mysqlrootpass (optional) - the MySQL root password (needed for building VuFind®'s database, when dbtype = mysql) 
-  * vufindurl (optional) - the URL where VuFind will be accessed (defaults to http://localhost/vufind if omitted)+  * vufindurl (optional) - the URL where VuFind® will be accessed (defaults to http://localhost/vufind if omitted)
   * mink_driver (optional) - the name of the Mink driver to use (e.g. "selenium" or "chrome").   * mink_driver (optional) - the name of the Mink driver to use (e.g. "selenium" or "chrome").
  
Line 108: Line 120:
 ==== Headless Chrome Testing ==== ==== Headless Chrome Testing ====
  
-:!: Headless Chrome testing was introduced in VuFind 7.+:!: Headless Chrome testing was introduced in VuFind® 7.
  
 Versions of Chrome 80+ have a built-in remote debugging feature. You can run Chrome instead of the Selenium Server like so: Versions of Chrome 80+ have a built-in remote debugging feature. You can run Chrome instead of the Selenium Server like so:
Line 120: Line 132:
 ==== Troubleshooting ==== ==== Troubleshooting ====
  
-:!: This section contains some notes that may help you avoid problems when trying to run VuFind's test suite.+:!: This section contains some notes that may help you avoid problems when trying to run VuFind®'s test suite.
  
-  * Some of VuFind's tests use a record ID containing a slash. In order for these tests to work, your Apache installation needs to be configured with "AllowEncodedSlashes on" set in the VirtualHost used for running the VuFind instance being tested.+  * Some of VuFind®'s tests use a record ID containing a slash. In order for these tests to work, your Apache installation needs to be configured with "AllowEncodedSlashes on" set in the VirtualHost used for running the VuFind® instance being tested.
   * When testing with PostgreSQL, it may be necessary to edit the pg_hba.conf file and change the line "local all all peer" to "local all all md5" to allow password-based database logins required my the test environment.   * When testing with PostgreSQL, it may be necessary to edit the pg_hba.conf file and change the line "local all all peer" to "local all all md5" to allow password-based database logins required my the test environment.
 +  * If you encounter issues with the browser tests, you can try running Chrome without the ''<nowiki>--headless</nowiki>'' parameter to make the process visible. It's also possible to step through the tests with a PHP debugger if the XDebug extension is installed while checking the results in the visible window. There are a couple of prerequisites for the tests to work properly when running with a visible Chrome window:
 +    * Disable Chrome's "Warn you if passwords are exposed in a data breach" in Privacy and Security settings while running the tests so that the password checks don't interfere with patron tests.
 +    * Set English (US) as the preferred language.
  
 ==== Mink Tests with Chrome on macOS ==== ==== Mink Tests with Chrome on macOS ====
Line 132: Line 147:
  
 <code> <code>
-/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --disable-gpu --headless --remote-debugging-address=0.0.0.0 --remote-debugging-port=9222 --window-size=1600,900+/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --disable-gpu --remote-debugging-address=0.0.0.0 --remote-debugging-port=9222 --window-size=1600,900 --headless 
 </code> </code>
  
Line 150: Line 165:
 === Required GNU Utilities === === Required GNU Utilities ===
  
-:!: Note: This applies only to VuFind versions before 8.0. +:!: Note: This applies only to VuFind® versions before 8.0. 
  
-VuFind's Mink tests up to version 7.x rely on options available only in GNU versions of find, sed and basename, so you will need to install those versions and make sure they are used when running Mink tests on macOS. One option is to use [[https://brew.sh/|Homebrew]] to install them:+VuFind®'s Mink tests up to version 7.x rely on options available only in GNU versions of find, sed and basename, so you will need to install those versions and make sure they are used when running Mink tests on macOS. One option is to use [[https://brew.sh/|Homebrew]] to install them:
  
 <code> <code>
Line 163: Line 178:
 PATH="/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/opt/findutils/libexec/gnubin:$PATH" $VUFIND_HOME/vendor/bin/phing -Dmysqlrootuser=$LOGNAME -Dmysqlrootpass="" -Dvufindurl=http://localhost:8080/vufind-test -Dmink_driver=chrome $* PATH="/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/opt/findutils/libexec/gnubin:$PATH" $VUFIND_HOME/vendor/bin/phing -Dmysqlrootuser=$LOGNAME -Dmysqlrootpass="" -Dvufindurl=http://localhost:8080/vufind-test -Dmink_driver=chrome $*
 </code> </code>
 +
 +==== HTML Validation ====
 +
 +It's possible to run Mink tests so that HTML of the final page of each test is validated. You will need to run NU Validator locally for this. NU can be downloaded from https://github.com/validator/validator/releases/. Get the vnu.jar package, e.g. vnu.jar_20.6.30.zip. Unzip and start for example with the following command:
 +
 +<code>
 +java -cp ./vnu.jar -Dnu.validator.servlet.bind-address=127.0.0.1 nu.validator.servlet.Main 8888
 +</code>
 +
 +Then run the tests with some extra parameters to enable validation:
 +
 +<code>
 +./phing.sh phpunitfast -Dhtml_validator=http://localhost:8888/ -Dhtml_validator_fail_tests=0 -Dhtml_validator_log_file=html_validation.log
 +</code>
 +
 +Results will be written into html_validation.log. It is currently (April 2023) not feasible to use html_validator_fail_tests=1 since there are still some known issues.
 +
 +==== Code Coverage ====
 +
 +Starting with VuFind® version 9.1 it's possible to collect code coverage data from Mink tests. It is worth noting that gathering coverage data will slow down the tests quite significantly.
 +
 +Prerequisites:
 +  * Apache must be running on the same server that runs the tests and be able to write to the tmp build directory
 +  * [[https://github.com/krakjoe/pcov|PCOV]] (installed and enabled) or [[https://xdebug.org/|XDebug]] (installed and enabled in coverage mode)
 +  * Remote profiling enabled with ''SetEnv VUFIND_CODE_COVERAGE 1'' in Apache config (see config/vufind/httpd_vufind.conf for more information)
 +
 +:!: Note that enabled PCOV or XDebug in coverage mode will prevent normal debugging with XDebug.
 +
 +To run tests with coverage enabled, use the phpunit task with the ''-Dphpunit_remote_coverage=1'' option, e.g. ''./phing.sh phpunit -Dphpunit_remote_coverage=1''. As is the case with normal unit tests, the coverage reports are written to $TMP/build/vufind/reports/coverage directory. Coverage reports are not available with ''phpunitfast'' or ''phpunitfaster''.
  
 ===== Writing Tests ===== ===== Writing Tests =====
Line 168: Line 212:
 ==== Location ==== ==== Location ====
  
-VuFind unit tests (which check the functionality of individual components) can be found in the module/VuFind/tests/unit-tests directory of your VuFind installation.+VuFind® unit tests (which check the functionality of individual components) can be found in the module/VuFind/tests/unit-tests directory of your VuFind® installation.
  
-VuFind integration tests (which check the interaction of components in a real running system) can be found in the module/VuFind/tests/integration-tests directory of your VuFind installation.+VuFind® integration tests (which check the interaction of components in a real running system) can be found in the module/VuFind/tests/integration-tests directory of your VuFind® installation.
  
-VuFind support modules (VuFindHttp, etc.) have their own tests directories, but these are configured to be run automatically as part of VuFind's main test suite.+VuFind® support modules (VuFindSearch, etc.) have their own tests directories, but these are configured to be run automatically as part of VuFind®'s main test suite.
  
-The layout of the each test directory is designed to mirror the VuFind module's src directory (module/VuFind/src/VuFind).  Tests should be placed in a directory that corresponds with the component being tested.  For example, the unit tests for the \VuFind\Date\Converter class are found in "module/VuFind/tests/unit-tests/src/VuFindTest/Date/ConverterTest.php".+The layout of the each test directory is designed to mirror the VuFind® module's src directory (module/VuFind/src/VuFind).  Tests should be placed in a directory that corresponds with the component being tested.  For example, the unit tests for the \VuFind\Config\Writer class are found in "module/VuFind/tests/unit-tests/src/VuFindTest/Config/WriterTest.php".
  
 ==== Support Classes/Traits ==== ==== Support Classes/Traits ====
Line 180: Line 224:
 The VuFindTest namespace (code found in module/VuFind/src/VuFindTest) contains several classes and traits which may be useful in writing new tests -- a fake record driver in VuFindTest\RecordDriver, some helpful traits in VuFindTest\Feature, and some base test classes containing reusable patterns in VuFindTest\Unit. The VuFindTest namespace (code found in module/VuFind/src/VuFindTest) contains several classes and traits which may be useful in writing new tests -- a fake record driver in VuFindTest\RecordDriver, some helpful traits in VuFindTest\Feature, and some base test classes containing reusable patterns in VuFindTest\Unit.
  
-:!: When working on integration tests, pay special attention to the VuFindTest\Feature\Live* traits (introduced during VuFind 8 development), which provide helpful methods for retrieving services that provide access to the real database and Solr instances for tests where actual data needs to be manipulated. Such tests should be written with caution, and should only be run in a safe test environment, as noted elsewhere.+:!: When working on integration tests, pay special attention to the VuFindTest\Feature\Live* traits (introduced during VuFind® 8 development), which provide helpful methods for retrieving services that provide access to the real database and Solr instances for tests where actual data needs to be manipulated. Such tests should be written with caution, and should only be run in a safe test environment, as noted elsewhere.
  
 ==== Mink Browser Automation ==== ==== Mink Browser Automation ====
  
-If you want to write browser automation tests, you should extend \VuFindTest\Integration\MinkTestCase. This class will automatically set up the Mink environment for you. It also provides a changeConfigs() method which can be used to reconfigure the running VuFind instance by modifying its .ini files. Any changes made by this method will be automatically rolled back at the end of the test, so you can test a variety of configurations in a single test class.+If you want to write browser automation tests, you should extend \VuFindTest\Integration\MinkTestCase. This class will automatically set up the Mink environment for you. It also provides a changeConfigs() method which can be used to reconfigure the running VuFind® instance by modifying its .ini files. Any changes made by this method will be automatically rolled back at the end of the test, so you can test a variety of configurations in a single test class.
  
 Some example Mink tests can be found in [[https://github.com/vufind-org/vufind/tree/dev/module/VuFind/tests/integration-tests/src/VuFindTest/Mink|module/VuFind/tests/integration-tests/src/VuFindTest/Mink/]]. Some example Mink tests can be found in [[https://github.com/vufind-org/vufind/tree/dev/module/VuFind/tests/integration-tests/src/VuFindTest/Mink|module/VuFind/tests/integration-tests/src/VuFindTest/Mink/]].
  
-==== Local Tests ====+=== Guidelines for Writing Mink Tests ===
  
-Create a directory in your local module mimicking the current structure of the vufind tests. Then all you have to do is require the class you are testing. If you figure out a way to autoload local module classes for testing, please let us know!+Here are some guidelines that help create tests that are robust and as fast as possible. MinkTestCase class has several methods that are recommended for interacting with the page. Many of the methods have retry or timeout logic to make sure they are as robust as possible.
  
-Prior to vufind 2.2 this will require changes to phpunit.xml. See [[https://github.com/vufind-org/vufind/commit/6e2b13852e2c1846cc45ceedfad20b6ae8c36aef|commit 6e2b138]].+  * Use hard-coded snooze only if absolutely necessary. 
 +  * Wait for required conditions to be met with ''findCss()'', ''waitStatement()'' etc. 
 +  * Wait for something to go away with ''unFindCss()''. 
 +  * Prefer ''clickCss()'' instead of find + click. 
 +  * Use ''assertEqualsWithTimeout()'' to wait for a condition to be met and asserted. 
 +  * Close lightbox without further action with ''closeLightbox()''. 
 +  * Wait for lightbox to close with ''waitForLightboxHidden()'' after performing an action. 
 +  * Use ''assertLightboxTitle()'' to make sure a correct lightbox is being displayed. 
 +  * Use ''waitForPageLoad()'' if you need to make sure that the page has loaded before continuing (including any jQuery "ready" event handlers). Useful also when you look for an element that may exist also on the previous page. 
 + 
 +==== Local Tests ====
  
 +Create a directory in your local module mimicking the current structure of the VuFind® tests. The test suite should automatically detect all tests within modules under the VuFind® directory.
 ===== Related Video ===== ===== Related Video =====
  
 See the [[videos:testing_1|Testing, Part 1]] and [[videos:testing_2|Testing, Part 2]] videos for more on this topic. See the [[videos:testing_1|Testing, Part 1]] and [[videos:testing_2|Testing, Part 2]] videos for more on this topic.
 ---- struct data ---- ---- struct data ----
 +properties.Page Owner : 
 ---- ----
  
development/testing/unit_tests.1644839453.txt.gz · Last modified: 2022/02/14 11:50 by emaijala