Warning: This page has not been updated in over over a year and may be outdated or deprecated.
videos:xml_change_tracking_and_email_alerts
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
videos:xml_change_tracking_and_email_alerts [2020/12/24 14:36] – [Transcript] demiankatz | videos:xml_change_tracking_and_email_alerts [2023/05/09 18:16] (current) – [Transcript] crhallberg | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Video 13: XML Change Tracking and Email Alerts ====== | ====== Video 13: XML Change Tracking and Email Alerts ====== | ||
- | The thirteenth | + | The thirteenth |
Video is available as an [[https:// | Video is available as an [[https:// | ||
Line 10: | Line 10: | ||
- [[configuration: | - [[configuration: | ||
+ | ===== Update Notes ===== | ||
+ | |||
+ | This video was recorded using VuFind 7. In VuFind 8.0, a change was made which impacts the content of this video: | ||
+ | |||
+ | * All XML import configuration examples now include an easy mechanism for turning on change tracking; it is no longer necessary to customize the XSLT, but simply to uncomment the relevant line in the matching properties file. | ||
===== Transcript ===== | ===== Transcript ===== | ||
- | // This is a raw machine-generated | + | Hello. In this month' |
+ | |||
+ | Fortunately, | ||
+ | |||
+ | This is pretty simple. I'm just going to edit my local harvest oai.ini file which we set up in that earlier video and go down to the Expositions section and I'm just going to add the line inject date equals date stamp. So in every Dublin core record that we harvest from Expositions, | ||
+ | |||
+ | So now that I've changed my harvesting rules, I need to actually re-harvest all of the records. And if I were to simply run the harvester now, I wouldn' | ||
+ | |||
+ | So I'm going to RM minus RF local harvest Expositions to recursively force the delete of the Expositions harvest directory. And now I can repeat my harvest by running PHP harvest OAIP dot PHP and tell it to harvest the Expositions repository. And now in just a few seconds, I will have my 293 records downloaded again. And just to show you that this really worked, let me open up one of these files that we just harvested. And if I scroll to the end of the first line, I will find that it has added a date stamp tag with a modification date for this record. | ||
+ | |||
+ | So now that I have harvested data with dates in it, I just need to index it and take advantage of those dates so that you can try and contract changes. Unfortunately, | ||
+ | |||
+ | I want to insert a dynamic value here. And for the select value, I'm going to use the PHP function call, which is how we run custom PHP code from inside an XSLT. The first parameter to this is the name of the PHP function, which in this case is VuFind get first indexed. And then we follow this with all of the parameters that the actual PHP function requires. In this case, it takes three. First, the name of the Solr core that we are indexing into, which in this case is Biblio. Next, it takes the ID of the record that we want to look up. And this we can get from the identifier field of our harvested records. So I'm just going to say normalized space string identifier, which will take the contents of the identifier tag presented as a string, which is the data type that VuFind expects and normalize all the space. Finally, the third parameter is the last change date of the record that we're looking up. This is the reason we added that gate stamp tag to our harvested records. So again, I'm going to normalize space and convert the string, the date stamp. And that is my function call that I can just close my value of tag. And that should do the job. Of course, I realized that if you're not particularly familiar with XSLT or PHP, this could be a bit intimidating. But even if you don't fully understand it, you can copy and paste this out of the Jira ticket. And in a future, if you find release, this can all be configured through your property file instead of having to do this work. Now, as I mentioned, there are actually two fields we need, not just first index, but also last indexed. But the code is extremely similar for both of them. So I'm just going to copy and paste what I wrote for first indexed and change every instance of the word first to last. And that should do the job. Now, it all comes down to whether I typed that correctly. Let's do a quick test to see if I did. I'm just going to run the XML import tool import slash import XSL.php in test only mode to look at some of my recently harvested records. | ||
+ | |||
+ | I'm using this new set of changes. And if I did it correctly, after a moment, I see the generated | ||
+ | |||
+ | I'm going to turn that on for today' | ||
+ | |||
+ | The final setting here in config.ini is scheduled search frequencies, | ||
+ | |||
+ | Also, while I'm in config.ini anyway, I should mention email configuration, | ||
+ | |||
+ | It's also worth noting that while VuFind currently assumes you're using SMTP, the Laminas mail module that it relies on, for sending messages, is quite flexible and can support other methods | ||
+ | |||
+ | The other possibility here is that you can set your mail to test only, which will cause it to pretend to send emails, but not actually send them. Obviously, in a real world situation, you never want to do that. | ||
+ | |||
+ | But for this demo, since I don't have an SMTP server to show you right now, I'm just going to put this in test mode. So it will appear to be working, even though nothing will really be happening. So that's it. We've configured email alerts. So now, let me show you what this looks like in the user interface. So I'm going to go to my catalog. I'm going to search for test. And having performed this search, it's now in my search history. So if I scroll down to search options and go to the search history screen, you'll see that there' | ||
+ | |||
+ | And that's all there is to email alerts. Thanks for watching. And as always, feel free to reach out with questions or suggestions. See you in the next video. | ||
- | hello in this month' | + | // This is an edited |
- | i'm going to tell you about | + | |
- | VuFind' | + | |
- | along the way i'm going to build on some | + | |
- | concepts that were previously discussed | + | |
- | in the videos about oai pmh and | + | |
- | indexing xml records if you haven' | + | |
- | already watched those videos you might | + | |
- | want to review them before proceeding | + | |
- | if you're ready to go forward i'll start | + | |
- | by pointing out that email alerts are a | + | |
- | comparatively recent feature | + | |
- | in VuFind having been introduced in | + | |
- | release 6.1 | + | |
- | also known as selective dissemination of | + | |
- | information or | + | |
- | sdi if you want to be technical about it | + | |
- | the email alerts | + | |
- | allow a user to subscribe to a search | + | |
- | query and receive emails notifying them | + | |
- | of any new results matching that query | + | |
- | as they' | + | |
- | obviously if we want to be able to | + | |
- | notify users about new records | + | |
- | we need to be able to tell which records | + | |
- | are new | + | |
- | and this requires us to use VuFinds | + | |
- | change tracking system | + | |
- | which we first talked about in the oai | + | |
- | pmh video where we showed you how to set | + | |
- | up change tracking for marc records | + | |
- | since that time uh we've also gone | + | |
- | through how to index | + | |
- | xml records and so in this video i'm | + | |
- | going to show you how to set up change | + | |
- | tracking for xml records so that you can | + | |
- | set up email alerts | + | |
- | regardless of what type of record you' | + | |
- | working with | + | |
- | just to very quickly recap the | + | |
- | point of view finds change tracking | + | |
- | system | + | |
- | is to identify at index time uh which | + | |
- | records are new to the index | + | |
- | and which records have changed since the | + | |
- | last time they are indexed | + | |
- | this uses a couple of solar fields | + | |
- | called first | + | |
- | index and last indexed to keep track of | + | |
- | dates | + | |
- | and in order to function properly it | + | |
- | needs to know | + | |
- | the last change date of each record | + | |
- | that's being indexed | + | |
- | fortunately using our oai pmh harvesting | + | |
- | tools | + | |
- | it's not too difficult to do this | + | |
- | so i'm going to go to the terminal | + | |
- | and switch into my VuFind home | + | |
- | directory | + | |
- | so in the xml indexing video as an | + | |
- | example | + | |
- | we indexed the expositions open access | + | |
- | journal hosted at villanova | + | |
- | and so today i'm going to proceed with | + | |
- | that example but just make some | + | |
- | modifications | + | |
- | to enable change tracking the first | + | |
- | thing that we need to do | + | |
- | is as i mentioned we need to know the | + | |
- | modification dates of records | + | |
- | so that the change tracking system can | + | |
- | properly judge what has changed | + | |
- | and what hasn't when we re-index things | + | |
- | uh fortunately oai pmh provides date | + | |
- | information with every record | + | |
- | it just uh doesn' | + | |
- | record metadata itself it's part of the | + | |
- | uh oai pmh package that's used to | + | |
- | deliver | + | |
- | the metadata but VuFind' | + | |
- | oai harvester includes a setting called | + | |
- | inject date which we can use to take | + | |
- | the modification date out of the oai pmh | + | |
- | header | + | |
- | and insert it into the metadata of the | + | |
- | record so that once it's been harvested | + | |
- | we can take advantage of it this is | + | |
- | pretty simple | + | |
- | i'm just going to edit my local | + | |
- | harvest oai.ini file which we set up | + | |
- | in that earlier video and go down to the | + | |
- | exposition section and | + | |
- | i'm just going to add the line inject | + | |
- | date | + | |
- | equals date stamp so in | + | |
- | every dublin core record that we harvest | + | |
- | from | + | |
- | expositions this will cause the | + | |
- | harvester | + | |
- | to add a date stamp tag which will | + | |
- | contain | + | |
- | the modification date of that record | + | |
- | this works very similar | + | |
- | to the inject id mechanism that we were | + | |
- | already using to get ids into our | + | |
- | harvested records | + | |
- | so now that i've changed my harvesting | + | |
- | rules i need to actually re-harvest | + | |
- | all of the records and if i were to | + | |
- | simply run the harvester now i wouldn' | + | |
- | get everything | + | |
- | because oai pmh harvesting is | + | |
- | incremental | + | |
- | so every time i run the harvest it will | + | |
- | only download the records that have been | + | |
- | added or changed | + | |
- | since my last harvest however in this | + | |
- | case because i've actually | + | |
- | changed the way i want to harvest i | + | |
- | really want to get everything again | + | |
- | i don't want just an incremental update | + | |
- | fortunately it's really easy to force if | + | |
- | you find to do a full re-harvest | + | |
- | we just need to delete the directory of | + | |
- | harvested records and start fresh | + | |
- | so i'm going to rm minus rf | + | |
- | local harvest expositions to recursively | + | |
- | force and delete of the exposition' | + | |
- | harvest directory | + | |
- | and now i can repeat my harvest | + | |
- | by running php harvest oai | + | |
- | dot php and tell it to harvest the | + | |
- | expositions repository | + | |
- | and now in just a few seconds i will | + | |
- | have my 293 records | + | |
- | downloaded again and just to show you | + | |
- | that this really | + | |
- | worked let me open up | + | |
- | one of these files | + | |
- | that we just harvested | + | |
- | and if i scroll to the end of the first | + | |
- | line | + | |
- | i will find that it has added a date | + | |
- | stamp tag | + | |
- | with a modification date for this record | + | |
- | so now that i have harvested data with | + | |
- | dates in it | + | |
- | i just need to index it and take | + | |
- | advantage | + | |
- | of those dates so that you find can | + | |
- | track changes | + | |
- | uh unfortunately vue finds example | + | |
- | uh xslt sheets for indexing xml | + | |
- | do not as of this recording | + | |
- | include modification | + | |
- | tracking support i have just opened | + | |
- | a jira ticket VuFind 1461 | + | |
- | which i will link to in the notes | + | |
- | accompanying this video | + | |
- | so that a future release if you find | + | |
- | will include examples to make this | + | |
- | a little bit more convenient but for | + | |
- | today | + | |
- | i'm going to take advantage of this | + | |
- | shortcoming to | + | |
- | give an example of customizing an xslt | + | |
- | and i'll just add the feature to the | + | |
- | ojs example which we're using for | + | |
- | harvesting | + | |
- | expositions so as with | + | |
- | just about everything else in VuFind | + | |
- | that you might want to customize | + | |
- | we can customize an xslt by copying it | + | |
- | from the core code | + | |
- | into our local custom directory | + | |
- | so in this instance the | + | |
- | existing xslt lives in the local import | + | |
- | xsl directory uh and so i need to | + | |
- | create a local equivalent of that so i | + | |
- | have a place to put my custom | + | |
- | so i'm just going to create the | + | |
- | directory local | + | |
- | import xsl | + | |
- | and then i'm going to copy | + | |
- | from import xsl ojs multirecord | + | |
- | xsl into | + | |
- | local import xsl | + | |
- | since ojs multi-record xsl is the | + | |
- | style sheet we're using for transforming | + | |
- | records harvested from ojs in this | + | |
- | example | + | |
- | then i just need to | + | |
- | edit the newly created copy of the xslt | + | |
- | and i'm just going to go down to the | + | |
- | bottom | + | |
- | and put some new fields following this | + | |
- | url here | + | |
- | so in xslt we're generating | + | |
- | document | + | |
- | in the case of indexing we're creating | + | |
- | the solar document that adds a record to | + | |
- | the solar index | + | |
- | so uh i can just put in | + | |
- | some solar xml | + | |
- | field name equals first indexed | + | |
- | in order to create the first index | + | |
- | value that we want to store i'm going to | + | |
- | close that | + | |
- | field tag and to fill in the actual | + | |
- | value | + | |
- | of the first index date VuFind | + | |
- | provides | + | |
- | a convenient helper function called get | + | |
- | first indexed | + | |
- | which takes a few parameters and uses | + | |
- | them to calculate the | + | |
- | first index date so here is where i need | + | |
- | to do a little bit of xslt | + | |
- | i use the xsl value of tag | + | |
- | to tell the xslt processor i want to | + | |
- | insert a dynamic value here | + | |
- | and for the select value i'm going to | + | |
- | use | + | |
- | the php function call which is how we | + | |
- | run custom php code from inside an | + | |
- | xslt the first parameter to this | + | |
- | is the name of the php function which in | + | |
- | this case | + | |
- | is you find get first indexed | + | |
- | and then we follow this with all of the | + | |
- | parameters that | + | |
- | the actual php function requires | + | |
- | in this case we it takes three first the | + | |
- | name of the solar core that we are | + | |
- | indexing into | + | |
- | which in this case is biblio next | + | |
- | it takes the id of the record that we | + | |
- | want to look up | + | |
- | and this we can get from the identifier | + | |
- | field | + | |
- | of our harvested records so i'm just | + | |
- | going to say | + | |
- | normalize space string identifier | + | |
- | which will take the contents of the | + | |
- | identifier tag | + | |
- | presented as a string which is the data | + | |
- | type that | + | |
- | VuFind expects and normalize all the | + | |
- | space | + | |
- | finally the third parameter is the last | + | |
- | change date of the record that we' | + | |
- | looking up | + | |
- | this is the reason we added that gate | + | |
- | stamp | + | |
- | tag to our harvested records so again | + | |
- | i'm going to | + | |
- | normalize space and convert to string | + | |
- | the date stamp | + | |
- | and that is my function fall then i can | + | |
- | just close | + | |
- | my value of tag and that should do the | + | |
- | job | + | |
- | of course i realize that if you're not | + | |
- | particularly familiar with xslt | + | |
- | or php this could be a bit intimidating | + | |
- | but even if you don't fully understand | + | |
- | it | + | |
- | you can copy and paste this out of the | + | |
- | jira ticket and | + | |
- | in a future if you find release this can | + | |
- | all be configured through your | + | |
- | properties file | + | |
- | instead of having to do this work now as | + | |
- | i mentioned there are | + | |
- | actually two fields we need not just | + | |
- | first index but also | + | |
- | last indexed but the code is extremely | + | |
- | similar for both of them so i'm just | + | |
- | going to copy and paste | + | |
- | what i wrote for first indexed | + | |
- | and change every instance of the word | + | |
- | first | + | |
- | to last and that should do the job | + | |
- | now it all comes down to whether i typed | + | |
- | that correctly | + | |
- | let's do a quick test to | + | |
- | see if i did i'm just going to run | + | |
- | the xml import | + | |
- | tool import slash import xsl.php | + | |
- | in test only mode to look at | + | |
- | some of my recently harvested records | + | |
- | um using this | + | |
- | new | + | |
- | set of changes | + | |
- | and if i did it correctly | + | |
- | after a moment i see | + | |
- | the generated solar xml | + | |
- | and sure enough it includes uh | + | |
- | first index and last index dates showing | + | |
- | the present time which means it worked | + | |
- | right because | + | |
- | all of these records have been indexed | + | |
- | and VuFind | + | |
- | as far as it's concerned | + | |
- | time today | + | |
- | because this is the first time i've | + | |
- | indexed these records with change | + | |
- | tracking turned on | + | |
- | uh if i were to re-index them in the | + | |
- | future | + | |
- | without the records themselves having | + | |
- | changed at all | + | |
- | um these dates would stay the same | + | |
- | that's how VuFind can detect | + | |
- | what changes similarly if i index new | + | |
- | copies of the records that have been | + | |
- | revised and have | + | |
- | newer modification dates then that last | + | |
- | index field is going to reflect | + | |
- | the date of the change | + | |
- | so having demonstrated in test mode that | + | |
- | this all works correctly | + | |
- | now all that's left to do is to actually | + | |
- | index all these records | + | |
- | i'm going to do that with the harvest | + | |
- | batch | + | |
- | import xsl script i just tell that that | + | |
- | i want to | + | |
- | index the expositions directory using | + | |
- | the ojs.properties configuration | + | |
- | and in just a few seconds | + | |
- | all of my records should be stored in | + | |
- | the index with | + | |
- | appropriate base | + | |
- | so that's it for change tracking | + | |
- | xml we now have everything we need to | + | |
- | turn | + | |
- | on the email alert feature and | + | |
- | fortunately this | + | |
- | is pretty simple in fact much simpler | + | |
- | than editing an xslt | + | |
- | if we look at our local config | + | |
- | you find config.ini file | + | |
- | there' | + | |
- | going to jump down to | + | |
- | and inside there there are a few | + | |
- | settings related to email alerts | + | |
- | the first is called schedule searches | + | |
- | which is turned off by default | + | |
- | until you turn it on all of the | + | |
- | functionality around email alerts | + | |
- | is hidden so i'm just going to turn it | + | |
- | on | + | |
- | the next setting is called force first | + | |
- | scheduled email | + | |
- | the purpose of this is that by default | + | |
- | if a user subscribes to a search | + | |
- | they won't get an email until there are | + | |
- | actually some new records | + | |
- | added to the search results but if you | + | |
- | turn this | + | |
- | on you can change the behavior so that | + | |
- | the first night after they' | + | |
- | subscribed they will receive an email | + | |
- | which sort of proves that the system is | + | |
- | working | + | |
- | i'm going to turn that on for today' | + | |
- | demo uh because otherwise we would have | + | |
- | to | + | |
- | uh wait some period of time for things | + | |
- | to change | + | |
- | uh before we could see the system | + | |
- | sending a message | + | |
- | the final setting here in config.ini | + | |
- | is scheduled search frequencies which | + | |
- | allows you to customize | + | |
- | the frequency at which notifications are | + | |
- | sent | + | |
- | by default the user can choose between | + | |
- | never being notified | + | |
- | being notified every day if there' | + | |
- | change or being notified every week | + | |
- | if there' | + | |
- | have different lengths of time between | + | |
- | notifications | + | |
- | you can customize this here but i'm | + | |
- | going to leave it at the defaults | + | |
- | for now also | + | |
- | while i'm in config.ini anyway uh | + | |
- | i should mention email configuration | + | |
- | since | + | |
- | obviously VuFind is only going to be | + | |
- | able to send notification emails if it | + | |
- | knows how to send email | + | |
- | and that may require some additional | + | |
- | configuration so i just want to bring | + | |
- | your attention | + | |
- | to the fact that there' | + | |
- | config.ini called | + | |
- | mail which is where you can set up uh | + | |
- | smtp settings standard mail transfer or | + | |
- | rather simple mail transfer protocol | + | |
- | to tell you find how to send mail | + | |
- | by default it assumes that there is an | + | |
- | smtp | + | |
- | process running on the VuFind server | + | |
- | but if there is not or if you have an | + | |
- | smtp service that requires | + | |
- | authentication | + | |
- | you can adjust these settings to point | + | |
- | to the right place | + | |
- | there are a number of services that will | + | |
- | allow you to | + | |
- | send emails using smtp in some cases | + | |
- | without cost if your volume is low or | + | |
- | you can run your own server | + | |
- | it's also worth noting that while | + | |
- | VuFind currently assumes you're using | + | |
- | smtp | + | |
- | the laminas mail module that it relies | + | |
- | on for sending messages | + | |
- | is quite flexible and can support other | + | |
- | methods as well | + | |
- | so with a little bit of custom code | + | |
- | it's possible to do things like use the | + | |
- | local send mail program | + | |
- | it would just require a custom factory | + | |
- | for your mail service | + | |
- | and if there' | + | |
- | would be useful for the community please | + | |
- | reach out and suggest it we can always | + | |
- | add more configuration options | + | |
- | in the future the other | + | |
- | possibility here is that you can set | + | |
- | your mail to test only | + | |
- | which will cause it to pretend to send | + | |
- | emails but not | + | |
- | actually send them obviously in a real | + | |
- | world situation you never want to do | + | |
- | that | + | |
- | but for this demo since i don't have an | + | |
- | smtp server | + | |
- | to show you right now i'm just going to | + | |
- | put this in test mode | + | |
- | so it will appear to be working even | + | |
- | though nothing will really be happening | + | |
- | so that's it we've configured email | + | |
- | alerts | + | |
- | so now let me show you what this looks | + | |
- | like | + | |
- | in the user interface | + | |
- | so i'm going to go to my catalog i'm | + | |
- | going to search for | + | |
- | test and having performed this search | + | |
- | it's now in my search history so if i | + | |
- | scroll down to search options and | + | |
- | go to the search history screen you' | + | |
- | see that | + | |
- | there' | + | |
- | that did not used to be here | + | |
- | this is how users are able to | + | |
- | uh get alerts so i'm going to set myself | + | |
- | up for daily alerts i want to know if | + | |
- | there are new tests | + | |
- | and of course i have to log in because | + | |
- | it needs to know my email address which | + | |
- | requires me to have an account so it | + | |
- | prompts me to do that | + | |
- | i log in and now as you can see | + | |
- | the search has moved from the recent | + | |
- | searches list into my saved searches | + | |
- | list | + | |
- | and it's scheduled for daily alerts | + | |
- | i could easily change this to | + | |
- | unsubscribe or improve the | + | |
- | increase the frequency if i wanted to | + | |
- | but for now i'll just leave it here | + | |
- | so how do the alerts get sent simple | + | |
- | there is a command line tool | + | |
- | which does the job and i will run that | + | |
- | right now | + | |
- | with php public index.php | + | |
- | scheduled search slash notify | + | |
- | and as you can see it was processing one | + | |
- | saved search | + | |
- | found my search because | + | |
- | i um had that setting to | + | |
- | always send an email the first time | + | |
- | someone subscribes | + | |
- | it treats this as new so it sent a | + | |
- | virtual message | + | |
- | and now it's done so | + | |
- | if you want to use this feature all you | + | |
- | need to do | + | |
- | is add the scheduled search notify | + | |
- | command to a cron job that runs | + | |
- | probably on a daily basis and then your | + | |
- | server will send out emails as things | + | |
- | change | + | |
- | if you already have a script set up to | + | |
- | uh index records on a regular basis | + | |
- | popping in this notification at the end | + | |
- | of that script would make a lot of sense | + | |
- | to ensure that the latest information | + | |
- | gets sent out | + | |
- | at the moment that it is fresh and | + | |
- | that' | + | |
- | all there is to email alerts uh thanks | + | |
- | for watching and as always feel free to | + | |
- | reach out with questions or suggestions | + | |
- | see you in the next video | + | |
---- struct data ---- | ---- struct data ---- | ||
+ | properties.Page Owner : | ||
---- | ---- | ||
videos/xml_change_tracking_and_email_alerts.1608820590.txt.gz · Last modified: 2020/12/24 14:36 by demiankatz