Differences

This shows you the differences between two versions of the page.

--- videos:search_system [2021/11/08 18:32] – created demiankatz
+++ videos:search_system [2021/11/09 11:35] (current) – demiankatz
@@ Line 1: / Line 1: @@
 ====== Video 21: Search System: An Overview and Update ======
-The twenty-first VuFind® instructional video provides an overview of VuFind's search architecture as well as a discussion of significant changes introduced to the system in release 8.0. This is fairly technical discussion, designed to be most useful and relevant to developers working with VuFind's search code.
+The twenty-first VuFind® instructional video provides an overview of VuFind®'s search architecture as well as a discussion of significant changes introduced to the system in release 8.0. This is fairly technical discussion, designed to be most useful and relevant to developers working with VuFind®'s search code.
 Video is available as an [[https://vufind.org/video/Search_Overview.mp4|mp4 download]] or through [[https://www.youtube.com/watch?v=HNa85xo1eGU|YouTube]].
@@ Line 8: / Line 8: @@
 ===== Related Resources =====
-  - [[https://vufind.org/video/Ils2_MultiBackend.pptx|PowerPoint Slides]]
+  - [[https://vufind.org/video/Search_Overview.pptx|PowerPoint Slides]]
-  - [[https://www.youtube.com/watch?v=8InrSNbt1Sk|2013 VuFind Summit Search System Introduction Video]]
+  - [[https://www.youtube.com/watch?v=8InrSNbt1Sk|2013 VuFind® Summit Search System Introduction Video]]
   - [[development:architecture:search_service|Search Service Architecture Notes]]
@@ Line 16: / Line 16: @@
 // This is a machine-generated transcript and will be corrected as time permits. //
-:!: Coming soon
+ hello and welcome to this month's
+viewfinder video uh this time around we
+are covering a more technical and less
+hands-on topic but uh i thought this was
+important and timely and might come in
+handy so i hope you enjoy it we have
+a few goals
+this month
+first of all to provide a quick overview
+of viewfinder internal search system the
+code that actually
+brings up your search results when you
+use the search function
+also to explain some changes to the
+search system that were introduced in
+the most recent major release viewfinder
+
+and to provide guidance so that if you
+have done local customizations related
+to the search system
+uh you can accommodate our new changes
+and be aware of some things that are
+coming up in the future in viewfinder
+so first a little bit of background
+uh the current search system in viewfind
+uh has been in place more or less
+unchanged since the 2.0 release in 2013.
+uh it was designed to replace the logic
+that was in viewfind one
+which was much harder to understand and
+less extensible
+of course nothing is perfect
+and that's why the most recent release
+.0 addressed some shortcomings of the
+existing system
+so first of all let's talk a little bit
+about
+how the system is designed
+uh there are a few key design principles
+that were used to build this
+first of all separation of concerns as i
+mentioned in the very early versions of
+viewfind the search code was quite
+monolithic and hard to follow and so
+this system was designed in direct
+response to that
+rather than one big class that does lots
+of things we have many smaller classes
+with distinct responsibilities that
+interact with one another so you know
+one class to parse user parameters and
+create a search query another one to
+connect to the actual api that does the
+searching etc etc
+we also make a distinction between
+general purpose code
+which is stored in the viewfind search
+namespace all one word
+and which is sort of the most reusable
+part of the system and the most abstract
+versus application specific integration
+code which is in the viewfind backslash
+search namespace
+which is where the viewfinder
+application actually makes use of
+the searching system
+another main design principle here is
+parameter aggregation
+so
+there's a class called param bag which
+is a container for parameters and when
+you do a solar search for example a
+number of pieces of code add parameters
+to the param bag and these all get sent
+to solar to perform the search the idea
+here is that we build up our search
+query with different components working
+on different aspects of that obviously
+the biggest part is building the actual
+search query language that retrieves
+records
+but there are other things in solar like
+spell checking and recommendations and
+we can have different pieces of code all
+adding parameters to the param bag and
+doing their distinct functions so
+parameter aggregation helps with
+separation of concerns uh if you've
+worked with any middleware based
+frameworks you might be familiar with
+this kind of approach
+uh the final
+key piece of the design is the idea of
+events as extension points so all of the
+search system code is event driven there
+are three major events that get
+triggered
+by the search service at different
+points in time
+and these events can be hooked by code
+which then responds to the events
+and does things
+the three key events are the resolve
+event
+which is how the search system figures
+out uh which search back-end to use for
+searching
+um
+generally you wouldn't be customizing
+this but
+you could if you wanted to override
+things in a particular way
+and the two more important events pre
+and post
+so the pre-event is triggered before we
+interact with the search back end
+and this is an opportunity to do more
+parameter aggregation or change
+parameters you know process data before
+the search occurs
+and of course post is
+the mirror image of that this event
+triggers after the search has occurred
+and is an opportunity to manipulate and
+format the search response
+so
+all of our separate components hook
+these events in order to use parameter
+aggregation
+to build a request and process a
+response and through all of that
+viewfind searching occurs
+and because of the way it's designed
+these individual components can be
+overridden or extended so if you want to
+customize something it's possible to
+focus in fairly narrowly on what you
+want to customize without having to
+touch all the many moving parts of this
+quite complex system
+so uh there are a few key classes and
+interfaces that it's good to know about
+if you're working with the search system
+uh of course the biggest one is the top
+level uh viewfind search service
+this is the class that manages the whole
+search process this is what is
+triggering the events and sort of making
+it all happen
+so when you perform any action with the
+search system
+you're doing it through the search
+service
+also important is the viewfinder search
+back end backend interface i've used the
+word backend several times already uh
+and this just refers to the code that
+interfaces with a particular search
+system
+a viewfinder is designed to
+aggregate search results from a number
+of different platforms and that's all
+made possible by the flexibility of the
+search system and the fact that we have
+back ends implemented for multiple
+services whether that be solar or
+various third-party apis they all work
+through the search service using the
+same mechanisms and the same interface
+another key class is the query builder
+and while
+there is no
+rule that every backend needs to have
+its query builder it's a pattern that
+you'll find in the code that most of
+them do
+the query builder simply being
+a piece of code which takes a viewfind
+search query query interface object and
+turns it into a param bag
+the query interface just defines some
+classes used to represent user searches
+essentially a container for
+the search terms and the field being
+searched or for a more complex
+combination of these if an advanced
+search is being performed
+and so
+it makes sense to have code that takes
+this
+uh abstract representation of a search
+and turns it into concrete parameters
+that can be sent to an api
+to actually perform a search
+uh another detail that you'll find in
+all of the back ends though there's not
+a strict rule about how it needs to be
+designed is a connector
+which is just the class that handles the
+actual communication
+uh in theory you could put everything
+directly in the back end and that would
+be fine uh but again for separation of
+concerns it's often helpful to design
+the back end as blue code between the
+search service and some more abstract
+code that handles actual over-the-wire
+communication
+in most cases the connector is something
+that's part of viewfind itself but in a
+couple of cases we use third-party
+libraries as connectors
+for example in the case of the summon
+api where a php connector
+is provided as a third party project
+that is of course a very quick summary
+of the search system
+if you want a deeper dive including kind
+of a step-by-step
+uh walkthrough of how a search uh takes
+place
+you might want to watch uh my video from
+the 2013 viewfinder summit where the
+search system was introduced that goes
+into a bit more depth than i've been
+able to hear and it is still very
+relevant even with the changes i am
+about to discuss
+and i'll include that link in the video
+notes
+so as i say nothing is perfect so what
+are the problems with this search system
+uh the first is that its scope is fairly
+rigid uh this was designed really with
+two specific actions in mind
+searching for a set of records and
+retrieving specific records
+but there are lots of things that our
+search backends can potentially do
+so for example there's a service called
+browzine which is able to look up
+information about particular dois this
+is a very narrowly focused specific
+service
+that perhaps doesn't even have an
+equivalent in any other back end but if
+i want to access that through the search
+service
+where do i put it
+um
+so that's that's one
+major issue with the current or or the
+pre 8.0 design
+uh bottom line is that adding new
+features to viewfind that involved the
+search service required us to do one of
+two things
+either we need to add a new method to
+the viewfind search service itself so we
+have a way to call the new functionality
+and this results in a bloated viewfinder
+search service uh especially if we're
+adding methods for really specific
+things that are only used
+narrowly in particular back ends we
+don't really want the viewfind search
+service to have to know about the
+internal details of every possible
+backend that could ever exist
+alternatively
+we can just bypass the search service
+entirely and directly access the backend
+code that the search service calls
+and
+in in past code this is often what we
+did it was too complicated to use the
+search service so we just worked around
+it by directly accessing objects in the
+service manager
+but this isn't good either
+because this is inconsistent
+as i mentioned a lot of the search
+services benefits have to do with event
+driven functionality
+if we bypass the search service and
+access back ends directly then the
+search service isn't able to fire events
+we can't use any hooks to extend those
+events we lose a lot of flexibility
+so
+what's the solution to this problem the
+solution is the command pattern
+so credit where it is due um
+david mouse was a developer who worked
+on the viewfind project for a few years
+around the time that version 2.0 was
+being developed and
+he did the vast majority of design work
+on our current
+search system
+we clearly still owe him a debt of
+gratitude for that because it has worked
+out quite well
+but during the original design phase he
+actually proposed using the command
+pattern because i think he recognized
+the problems that we eventually
+encountered
+but there wasn't time to implement it
+and time flew by uh and it wasn't until
+release 8.0 that we actually implemented
+this original idea
+thanks in large part uh to alexi peebles
+from the national library of finland who
+raised some concerns and uh noticed some
+of these issues
+and took the time to help address them
+so
+thank you to both of those people for uh
+all of their work supporting the project
+but anyway
+to to explain what the command pattern
+actually is
+the idea here is that instead of having
+a method on the search service for every
+action we could possibly
+perform
+instead we just have one method on the
+search service the method is called
+invoke
+and it accepts a class representing a
+command as its parameter
+thus
+the search service doesn't have to know
+about every possible command you could
+ever issue it just needs to be able to
+deal with these command objects
+so
+command classes
+are generally constructed using the
+parameters of the command
+and they contain code that actually
+interacts with the back end to execute
+the command
+the command class also acts as a
+container for the results of the command
+so sort of the workflow here is you
+construct the command with all the
+parameters for what you need it to to do
+so for example a search command you'll
+pass it the search query and other
+parameters like page number and page
+size and so forth
+then you pass that constructed object to
+the search services invoke method
+and the search service does all of the
+event stuff
+uh which ultimately results in the back
+end object getting passed to the command
+object so that the command object can
+execute the command on the back end
+then the command object stores the
+results of that command internally
+and is done
+so
+the
+search service then returns the same
+command object that was passed into it
+which now contains the results of the
+command and the calling code can act on
+them so it's all pretty seamless
+but it has a lot of flexibility
+so what are the benefits of this
+well obviously this moves more logic
+into the back end and out of the search
+service
+the search service is now
+more of a separated concern it does
+event processing and command routing it
+doesn't really care about what the
+commands are doing that's the job of the
+commands
+it makes us a lot more focused
+it also reduces the amount of plumbing
+needed to add new search features you
+know previously
+you might potentially have to add a new
+method to a connector then add a new
+method to the back end to call that
+connector then add a new method to the
+search service
+and then call the search service method
+i wouldn't say that using commands
+actually reduces the amount of code that
+you're creating because you still have
+to deal with back ends and connectors
+and you have to write a command class
+to process the command
+but it just feels cleaner to me that the
+command objects are somewhat standalone
+and we're not wiring everything
+explicitly through the search service
+it also streamlines and centralizes
+parameter processing uh in event
+handling code so if you're
+trying to act on an event you create
+listener code that responds to the event
+and previously depending on what action
+you were listening for
+you might have any number of different
+parameters in the event object that you
+would have to extract and work with
+with the command pattern
+all we really need to put into the event
+is the command object and then your
+listener code can interact directly with
+the command
+you can look at the class of the command
+to figure out what command is uh being
+passed so you can specifically target
+specific events
+uh and since the command object is a
+class it can have any kind of methods
+that you need
+so you can create functionality there
+for changing and adding and removing
+parameters
+specific to the needs of the command
+that you're executing
+it just makes things a little bit more
+targeted and a little more contained
+of course a theme here is that nothing
+is perfect and the command pattern
+itself is not perfect there are some
+challenges here that we need to think
+about
+uh one of the biggest ones is that it is
+easy and tempting to put too much logic
+into commands
+you if you're writing a new command you
+really need to think about the intent of
+the command pattern
+the command class is just meant to be
+sort of a carrier of parameters and
+responses
+it's not intended to do a lot of logic
+but you could because
+when your command executes it's given
+access to the back end so you can
+theoretically write a command that has a
+whole lot of processing logic and does a
+whole lot of work in it
+and that would function but i think
+logically that gets you into some
+troublesome territory uh i think the the
+goal in designing commands should be to
+put most of the logic in your search
+back end
+and ha and limit your command to simply
+passing parameters and executing that
+uh and this
+again is just about separation of
+concerns the back end is where the
+back-end processing logic is intended to
+live
+the command is just a carrier
+um
+of course another challenge here is that
+commands make this already complex and
+abstract system a little more complex
+and abstract and perhaps a little bit
+harder to learn
+which is why i'm recording a video
+talking about it in an effort to help
+with some of that
+uh we'll see if that's actually helpful
+but i hope it is
+uh finally
+we we need to make decisions about where
+the command classes we create live
+within viewfinds modules and namespaces
+as i alluded to earlier there are some
+commands that are very back-end specific
+and probably will only ever be used with
+one particular system whereas there are
+others that are quite global and could
+apply in many places
+so
+uh you sort of have to think about if
+you're introducing a new command is this
+something universal that belongs in sort
+of a high level namespace or does it
+belong inside a particular back end
+because that is
+the only place where it will ever be
+used
+i anticipate over time some commands may
+begin life as back-end specific and then
+end up getting moved up as we discover
+that there are in fact commonalities
+between multiple systems
+so let's get into just a few more
+specifics about command classes
+there are a few interfaces and abstract
+classes that you should be aware of
+if you're looking to add a new command
+to the system
+first of all all commands need to
+conform to viewfind search command
+command interface
+and this is just the interface for
+the obvious basic functionality of a
+command it needs to know
+whether it has executed yet
+it needs to have a way of storing the
+response
+that is the result of the command
+etc it's pretty lightweight
+there's also a viewfind search command
+abstract based class which implements
+useful defaults for
+some of the methods defined in the
+command interface
+you can save yourself a little bit of
+time by using this
+and then one more useful abstract class
+is viewfind search command call method
+command this extends abstract base uh
+with some logic
+that's built around the assumption that
+your command just needs to call a method
+on the back end
+so uh most of the commands that ship
+with viewfinder actually subclasses of
+call method command
+and so
+the idea here is that
+you specify in your constructor what
+method is going to be called and some
+rules around how it's going to get
+called
+then your subclass of call method
+command is mainly interested in managing
+the parameters to that command
+and can rely on the parent class for
+actually doing the work of calling the
+backend storing the results etc
+this results in some very simple
+lightweight classes if you want to add a
+method to the back end and call it just
+extend call method command and you have
+a quick way to get your command object
+uh created
+there are also a couple of useful traits
+there's the
+viewfind search command feature query
+offset limit trait and in the same
+namespace the record identifier trait
+these are both traits for dealing with
+commonly used parameters so if you have
+something within offset and limit or if
+you have something that relies on a
+record id you can use these traits and
+they just give you getters and setters
+so you don't have to define them over
+and over again
+uh and if you're interested in example i
+would suggest uh take a look at the
+viewfind search command search command
+this is the command that is used for
+performing searches the most common
+function of the search backend
+it's a subclass of the call method
+command
+and if you look at it i think you'll get
+some idea of how all of these things
+work
+so
+of course i've talked in the abstract
+about uh what has changed in viewfinder
+uh but let's go into
+a little bit more specific uh detail
+since this may impact your local custom
+code
+so first of all most of the public
+methods on the viewfind search service
+have been deprecated in favor of using
+command objects with the invoke method
+so you know previously the search
+service had lots of methods like search
+and retrieve and so forth
+all of those are deprecated and have
+corresponding commands
+in viewfind 8
+these methods are still in use
+because we're we're looking to uh sort
+of smooth the transition and not break
+too much custom code all at once but
+they are going to go away in viewfind
+nine
+so
+be prepared uh to stop using them
+in the future
+additionally event parameters and
+targets have changed
+again
+we've introduced some legacy
+compatibility logic so that if you call
+the old deprecated
+individual search service methods
+um
+they will set up the events very
+similarly to viewfind seven and earlier
+which should allow most
+custom listeners to continue to function
+without any change
+but
+as i say in viewfind nine things are
+going to change more dramatically so
+this legacy compatibility logic is here
+uh just to smooth the transition
+you know old code will continue to work
+but new style code will also work so
+this is an opportunity to start
+making the shift if you have custom
+listeners
+another important change is that all
+code in viewfind which bypassed the
+search service and directly accessed the
+backend objects through the backend
+manager has been refactored to use
+commands so you can now more reliably
+use events to hook actions
+and all of our sort of cheating has been
+uh removed
+so what do you need to do
+a couple things
+first of all revise your custom
+listeners to interact with the command
+object which is as i mentioned in the
+events command parameter instead of
+using legacy event parameters so right
+now in viewfind 8 all of the events are
+going to have the legacy parameters and
+the command parameter but all of the
+information in the legacy event
+parameters is also available through the
+command object so you should be able to
+refactor your code to just replace all
+the legacy stuff with direct interaction
+with the command and then you're good to
+go and when we upgrade to viewfind 9 and
+take away the legacy parameters your
+code won't need to be touched again
+we're just giving you a window of time
+where this upgrade is not absolutely
+essential it's just highly recommended
+uh the other thing to do is avoid that
+anti-pattern of directly accessing
+search back-ends instead of sending
+commands
+just use a command
+to
+access a method on the back end
+if you're not sure whether you're doing
+this or not i recommend searching your
+custom code for the back end manager
+because the easiest way to cheat and get
+directly to a back end is to pull the
+back end manager out of
+the service manager and then pull back
+ends out of it that is how
+the core code used to do this cheating
+so if you're not referencing the back
+end manager anywhere
+you should be in good shape you don't
+need to change anything but if you are
+look at that code find a better way
+hopefully there's not a whole lot of
+custom search code out there and so i
+don't think a lot of people are going to
+be affected by this but if you are
+please be aware
+and that is really everything i wanted
+to share
+but if you have any questions about this
+please feel free to reach out to me
+my email is damiencats villanova.edu
+or you can go directly to vufine's
+various community support channels
+mailing lists slack etc
+send your questions we will be happy to
+help uh as i say i know that the search
+system is complex and has a lot of
+abstraction in it it may not be easy to
+get your head around it first
+but we're happy to talk to you and help
+you along and once you understand it it
+is really powerful and useful
+so it's worth the investment
+and that's all i have so thank you again
+for your time and i will talk to you
+again soon
 ---- struct data ----
+properties.Page Owner :
 ----