Warning: This page has not been updated in over over a year and may be outdated or deprecated.
videos:xml_change_tracking_and_email_alerts
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
videos:xml_change_tracking_and_email_alerts [2021/09/10 11:21] – [Related Resources] demiankatz | videos:xml_change_tracking_and_email_alerts [2023/05/09 18:16] (current) – [Transcript] crhallberg | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Video 13: XML Change Tracking and Email Alerts ====== | ====== Video 13: XML Change Tracking and Email Alerts ====== | ||
- | The thirteenth | + | The thirteenth |
Video is available as an [[https:// | Video is available as an [[https:// | ||
Line 17: | Line 17: | ||
===== Transcript ===== | ===== Transcript ===== | ||
- | // This is a raw machine-generated transcript; | + | Hello. In this month' |
- | hello in this month' | + | Fortunately, |
- | i'm going to tell you about | + | |
- | VuFind' | + | This is pretty simple. I'm just going to edit my local harvest oai.ini file which we set up in that earlier video and go down to the Expositions |
- | along the way i'm going to build on some | + | |
- | concepts that were previously discussed | + | So now that I've changed my harvesting rules, I need to actually re-harvest all of the records. And if I were to simply run the harvester now, I wouldn' |
- | in the videos about oai pmh and | + | |
- | indexing xml records if you haven' | + | So I'm going to RM minus RF local harvest |
- | already watched those videos you might | + | |
- | want to review them before proceeding | + | So now that I have harvested data with dates in it, I just need to index it and take advantage of those dates so that you can try and contract |
- | if you're ready to go forward i'll start | + | |
- | by pointing out that email alerts are a | + | I want to insert a dynamic value here. And for the select value, I'm going to use the PHP function call, which is how we run custom |
- | comparatively recent feature | + | |
- | in VuFind having been introduced in | + | I' |
- | release 6.1 | + | |
- | also known as selective dissemination of | + | I'm going to turn that on for today' |
- | information or | + | |
- | sdi if you want to be technical about it | + | The final setting here in config.ini is scheduled search frequencies, which allows you to customize the frequency at which notifications are sent. By default, the user can choose between never being notified, being notified every day if there' |
- | the email alerts | + | |
- | allow a user to subscribe to a search | + | Also, while I'm in config.ini anyway, I should mention email configuration, since obviously VuFind is only going to be able to send notification emails if it knows how to send email. And that may require some additional configuration. So I just want to bring your attention to the fact that there' |
- | query and receive emails notifying them | + | |
- | of any new results matching that query | + | It's also worth noting that while VuFind currently assumes you're using SMTP, the Laminas |
- | as they' | + | |
- | obviously if we want to be able to | + | The other possibility here is that you can set your mail to test only, which will cause it to pretend to send emails, but not actually send them. Obviously, |
- | notify users about new records | + | |
- | we need to be able to tell which records | + | But for this demo, since I don't have an SMTP server to show you right now, I'm just going to put this in test mode. So it will appear to be working, even though nothing will really be happening. So that's it. We've configured email alerts. So now, let me show you what this looks like in the user interface. So I'm going to go to my catalog. I'm going to search for test. And having performed this search, it's now in my search history. So if I scroll down to search options and go to the search history screen, you'll see that there' |
- | are new | + | |
- | and this requires us to use VuFinds | + | And that's all there is to email alerts. Thanks |
- | change tracking system | + | |
- | which we first talked about in the oai | + | // This is an edited version of an automated transcript. Apologies for any errors. // |
- | pmh video where we showed you how to set | + | |
- | up change tracking for marc records | + | |
- | since that time uh we've also gone | + | |
- | through how to index | + | |
- | xml records and so in this video i'm | + | |
- | going to show you how to set up change | + | |
- | tracking for xml records so that you can | + | |
- | set up email alerts | + | |
- | regardless of what type of record you' | + | |
- | working with | + | |
- | just to very quickly recap the | + | |
- | point of view finds change tracking | + | |
- | system | + | |
- | is to identify at index time uh which | + | |
- | records are new to the index | + | |
- | and which records have changed since the | + | |
- | last time they are indexed | + | |
- | this uses a couple of solar fields | + | |
- | called first | + | |
- | index and last indexed to keep track of | + | |
- | dates | + | |
- | and in order to function properly it | + | |
- | needs to know | + | |
- | the last change date of each record | + | |
- | that's being indexed | + | |
- | fortunately using our oai pmh harvesting | + | |
- | tools | + | |
- | it's not too difficult to do this | + | |
- | so i'm going to go to the terminal | + | |
- | and switch into my VuFind home | + | |
- | directory | + | |
- | so in the xml indexing video as an | + | |
- | example | + | |
- | we indexed the expositions open access | + | |
- | journal hosted at villanova | + | |
- | and so today i'm going to proceed with | + | |
- | that example but just make some | + | |
- | modifications | + | |
- | to enable change tracking the first | + | |
- | thing that we need to do | + | |
- | is as i mentioned we need to know the | + | |
- | modification dates of records | + | |
- | so that the change tracking system can | + | |
- | properly judge what has changed | + | |
- | and what hasn't when we re-index things | + | |
- | uh fortunately oai pmh provides date | + | |
- | information with every record | + | |
- | it just uh doesn' | + | |
- | record metadata itself | + | |
- | uh oai pmh package that's used to | + | |
- | deliver | + | |
- | the metadata | + | |
- | oai harvester | + | |
- | inject date which we can use to take | + | |
- | the modification date out of the oai pmh | + | |
- | header | + | |
- | and insert it into the metadata of the | + | |
- | record so that once it's been harvested | + | |
- | we can take advantage of it this is | + | |
- | pretty simple | + | |
- | i'm just going to edit my local | + | |
- | harvest oai.ini file which we set up | + | |
- | in that earlier video and go down to the | + | |
- | exposition | + | |
- | i'm just going to add the line inject | + | |
- | date | + | |
- | equals date stamp so in | + | |
- | every dublin | + | |
- | from | + | |
- | expositions | + | |
- | harvester | + | |
- | to add a date stamp tag which will | + | |
- | contain | + | |
- | the modification date of that record | + | |
- | this works very similar | + | |
- | to the inject id mechanism that we were | + | |
- | already using to get ids into our | + | |
- | harvested records | + | |
- | so now that i've changed my harvesting | + | |
- | rules i need to actually re-harvest | + | |
- | all of the records | + | |
- | simply run the harvester now i wouldn' | + | |
- | get everything | + | |
- | because | + | |
- | incremental | + | |
- | so every time i run the harvest it will | + | |
- | only download the records that have been | + | |
- | added or changed | + | |
- | since my last harvest | + | |
- | case because | + | |
- | changed the way i want to harvest | + | |
- | really want to get everything again | + | |
- | i don't want just an incremental update | + | |
- | fortunately | + | |
- | you find to do a full re-harvest | + | |
- | we just need to delete the directory of | + | |
- | harvested records and start fresh | + | |
- | so i'm going to rm minus rf | + | |
- | local harvest | + | |
- | force and delete of the exposition' | + | |
- | harvest directory | + | |
- | and now i can repeat my harvest | + | |
- | by running | + | |
- | dot php and tell it to harvest the | + | |
- | expositions | + | |
- | and now in just a few seconds | + | |
- | have my 293 records | + | |
- | downloaded again and just to show you | + | |
- | that this really | + | |
- | worked let me open up | + | |
- | one of these files | + | |
- | that we just harvested | + | |
- | and if i scroll to the end of the first | + | |
- | line | + | |
- | i will find that it has added a date | + | |
- | stamp tag | + | |
- | with a modification date for this record | + | |
- | so now that i have harvested data with | + | |
- | dates in it | + | |
- | i just need to index it and take | + | |
- | advantage | + | |
- | of those dates so that you find can | + | |
- | track changes | + | |
- | uh unfortunately vue finds example | + | |
- | uh xslt sheets for indexing | + | |
- | do not as of this recording | + | |
- | include modification | + | |
- | tracking support | + | |
- | a jira ticket VuFind 1461 | + | |
- | which i will link to in the notes | + | |
- | accompanying this video | + | |
- | so that a future release | + | |
- | will include examples to make this | + | |
- | a little bit more convenient | + | |
- | today | + | |
- | i'm going to take advantage of this | + | |
- | shortcoming to | + | |
- | give an example of customizing an xslt | + | |
- | and i'll just add the feature to the | + | |
- | ojs example which we're using for | + | |
- | harvesting | + | |
- | expositions so as with | + | |
- | just about everything else in VuFind | + | |
- | that you might want to customize | + | |
- | we can customize an xslt by copying it | + | |
- | from the core code | + | |
- | into our local custom directory | + | |
- | so in this instance the | + | |
- | existing | + | |
- | xsl directory | + | |
- | create a local equivalent of that so i | + | |
- | have a place to put my custom version | + | |
- | so i'm just going to create the | + | |
- | directory local | + | |
- | import | + | |
- | and then i'm going to copy | + | |
- | from import | + | |
- | xsl into | + | |
- | local import | + | |
- | since ojs multi-record | + | |
- | style sheet we're using for transforming | + | |
- | records harvested from ojs in this | + | |
- | example | + | |
- | then i just need to | + | |
- | edit the newly created copy of the xslt | + | |
- | and i'm just going to go down to the | + | |
- | bottom | + | |
- | and put some new fields following this | + | |
- | url here | + | |
- | so in xslt we're generating an xml | + | |
- | document | + | |
- | in the case of indexing we're creating | + | |
- | the solar document that adds a record to | + | |
- | the solar index | + | |
- | so uh i can just put in | + | |
- | some solar xml | + | |
- | field name equals first indexed | + | |
- | in order to create the first index | + | |
- | value that we want to store i'm going to | + | |
- | close that | + | |
- | field tag and to fill in the actual | + | |
- | value | + | |
- | of the first index date VuFind | + | |
- | provides | + | |
- | a convenient helper function called get | + | |
- | first indexed | + | |
- | which takes a few parameters and uses | + | |
- | them to calculate the | + | |
- | first index date so here is where i need | + | |
- | to do a little bit of xslt | + | |
- | i use the xsl value of tag | + | |
- | to tell the xslt processor | + | |
- | insert a dynamic value here | + | |
- | and for the select value i'm going to | + | |
- | use | + | |
- | the php function call which is how we | + | |
- | run custom | + | |
- | xslt the first parameter to this | + | |
- | is the name of the php function which in | + | |
- | this case | + | |
- | is you find get first indexed | + | |
- | and then we follow this with all of the | + | |
- | parameters that | + | |
- | the actual | + | |
- | in this case we it takes three first the | + | |
- | name of the solar core that we are | + | |
- | indexing into | + | |
- | which in this case is biblio next | + | |
- | it takes the id of the record that we | + | |
- | want to look up | + | |
- | and this we can get from the identifier | + | |
- | field | + | |
- | of our harvested records | + | |
- | going to say | + | |
- | normalize | + | |
- | which will take the contents of the | + | |
- | identifier tag | + | |
- | presented as a string which is the data | + | |
- | type that | + | |
- | VuFind expects and normalize all the | + | |
- | space | + | |
- | finally | + | |
- | change date of the record that we're | + | |
- | looking up | + | |
- | this is the reason we added that gate | + | |
- | stamp | + | |
- | tag to our harvested records | + | |
- | i'm going to | + | |
- | normalize space and convert | + | |
- | the date stamp | + | |
- | and that is my function | + | |
- | just close | + | |
- | my value of tag and that should do the | + | |
- | job | + | |
- | of course | + | |
- | particularly familiar with xslt | + | |
- | or php this could be a bit intimidating | + | |
- | but even if you don't fully understand | + | |
- | it | + | |
- | you can copy and paste this out of the | + | |
- | jira ticket | + | |
- | in a future if you find release this can | + | |
- | all be configured through your | + | |
- | properties | + | |
- | instead of having to do this work now as | + | |
- | i mentioned there are | + | |
- | actually two fields we need not just | + | |
- | first index but also | + | |
- | last indexed | + | |
- | similar for both of them so i'm just | + | |
- | going to copy and paste | + | |
- | what i wrote for first indexed | + | |
- | and change every instance of the word | + | |
- | first | + | |
- | to last and that should do the job | + | |
- | now it all comes down to whether | + | |
- | that correctly | + | |
- | let's do a quick test to | + | |
- | see if i did i'm just going to run | + | |
- | the xml import | + | |
- | tool import slash import | + | |
- | in test only mode to look at | + | |
- | some of my recently harvested records | + | |
- | um using this | + | |
- | new | + | |
- | set of changes | + | |
- | and if i did it correctly | + | |
- | after a moment | + | |
- | the generated | + | |
- | and sure enough it includes | + | |
- | first index and last index dates showing | + | |
- | the present time which means it worked | + | |
- | right because | + | |
- | all of these records have been indexed | + | |
- | and VuFind | + | |
- | as far as it's concerned for the first | + | |
- | time today | + | |
- | because this is the first time i've | + | |
- | indexed these records with change | + | |
- | tracking turned on | + | |
- | uh if i were to re-index | + | |
- | future | + | |
- | without the records themselves having | + | |
- | changed at all | + | |
- | um these dates would stay the same | + | |
- | that's how VuFind | + | |
- | what changes | + | |
- | copies of the records that have been | + | |
- | revised and have | + | |
- | newer modification dates then that last | + | |
- | index field is going to reflect | + | |
- | the date of the change | + | |
- | so having demonstrated in test mode that | + | |
- | this all works correctly | + | |
- | now all that's left to do is to actually | + | |
- | index all these records | + | |
- | i'm going to do that with the harvest | + | |
- | batch | + | |
- | import | + | |
- | i want to | + | |
- | index the expositions directory using | + | |
- | the ojs.properties configuration | + | |
- | and in just a few seconds | + | |
- | all of my records should be stored in | + | |
- | the index with | + | |
- | appropriate | + | |
- | so that's it for change tracking | + | |
- | xml we now have everything we need to | + | |
- | turn | + | |
- | on the email alert feature | + | |
- | fortunately this | + | |
- | is pretty simple | + | |
- | than editing an xslt | + | |
- | if we look at our local config | + | |
- | you find config.ini file | + | |
- | there' | + | |
- | going to jump down to | + | |
- | and inside there there are a few | + | |
- | settings related to email alerts | + | |
- | the first is called schedule searches | + | |
- | which is turned off by default | + | |
- | until you turn it on all of the | + | |
- | functionality around email alerts | + | |
- | is hidden | + | |
- | on | + | |
- | the next setting is called force first | + | |
- | scheduled email | + | |
- | the purpose of this is that by default | + | |
- | if a user subscribes to a search | + | |
- | they won't get an email until there are | + | |
- | actually some new records | + | |
- | added to the search results | + | |
- | turn this | + | |
- | on you can change the behavior so that | + | |
- | the first night after they' | + | |
- | subscribed they will receive an email | + | |
- | which sort of proves that the system is | + | |
- | working | + | |
- | i'm going to turn that on for today' | + | |
- | demo uh because otherwise we would have | + | |
- | to | + | |
- | uh wait some period of time for things | + | |
- | to change | + | |
- | uh before we could see the system | + | |
- | sending a message | + | |
- | the final setting here in config.ini | + | |
- | is scheduled search frequencies which | + | |
- | allows you to customize | + | |
- | the frequency at which notifications are | + | |
- | sent | + | |
- | by default the user can choose between | + | |
- | never being notified | + | |
- | being notified every day if there' | + | |
- | change or being notified every week | + | |
- | if there' | + | |
- | have different lengths of time between | + | |
- | notifications | + | |
- | you can customize this here but i'm | + | |
- | going to leave it at the defaults | + | |
- | for now also | + | |
- | while i'm in config.ini anyway | + | |
- | i should mention email configuration | + | |
- | since | + | |
- | obviously VuFind is only going to be | + | |
- | able to send notification emails if it | + | |
- | knows how to send email | + | |
- | and that may require some additional | + | |
- | configuration | + | |
- | your attention | + | |
- | to the fact that there' | + | |
- | config.ini called | + | |
- | mail which is where you can set up uh | + | |
- | smtp settings standard mail transfer | + | |
- | rather simple mail transfer protocol | + | |
- | to tell you find how to send mail | + | |
- | by default it assumes that there is an | + | |
- | smtp | + | |
- | process running on the VuFind server | + | |
- | but if there is not or if you have an | + | |
- | smtp service that requires | + | |
- | authentication | + | |
- | you can adjust these settings to point | + | |
- | to the right place | + | |
- | there are a number of services that will | + | |
- | allow you to | + | |
- | send emails using smtp in some cases | + | |
- | without cost if your volume is low or | + | |
- | you can run your own server | + | |
- | it's also worth noting that while | + | |
- | VuFind currently assumes you're using | + | |
- | smtp | + | |
- | the laminas | + | |
- | on for sending messages | + | |
- | is quite flexible and can support other | + | |
- | methods as well | + | |
- | so with a little bit of custom code | + | |
- | it's possible to do things like use the | + | |
- | local send mail program | + | |
- | it would just require a custom factory | + | |
- | for your mail service | + | |
- | and if there' | + | |
- | would be useful for the community please | + | |
- | reach out and suggest it we can always | + | |
- | add more configuration options | + | |
- | in the future | + | |
- | possibility here is that you can set | + | |
- | your mail to test only | + | |
- | which will cause it to pretend to send | + | |
- | emails but not | + | |
- | actually send them obviously | + | |
- | world situation you never want to do | + | |
- | that | + | |
- | but for this demo since i don't have an | + | |
- | smtp server | + | |
- | to show you right now i'm just going to | + | |
- | put this in test mode | + | |
- | so it will appear to be working even | + | |
- | though nothing will really be happening | + | |
- | so that's it we've configured email | + | |
- | alerts | + | |
- | so now let me show you what this looks | + | |
- | like | + | |
- | in the user interface | + | |
- | so i'm going to go to my catalog | + | |
- | going to search for | + | |
- | test and having performed this search | + | |
- | it's now in my search history | + | |
- | scroll down to search options and | + | |
- | go to the search history screen you'll | + | |
- | see that | + | |
- | there' | + | |
- | that did not used to be here | + | |
- | this is how users are able to | + | |
- | uh get alerts | + | |
- | up for daily alerts | + | |
- | there are new tests | + | |
- | and of course | + | |
- | it needs to know my email address which | + | |
- | requires me to have an account | + | |
- | prompts me to do that | + | |
- | i log in and now as you can see | + | |
- | the search has moved from the recent | + | |
- | searches list into my saved searches | + | |
- | list | + | |
- | and it's scheduled for daily alerts | + | |
- | i could easily change this to | + | |
- | unsubscribe or improve the | + | |
- | increase the frequency if i wanted to | + | |
- | but for now i'll just leave it here | + | |
- | so how do the alerts get sent simple | + | |
- | there is a command line tool | + | |
- | which does the job and i will run that | + | |
- | right now | + | |
- | with php public index.php | + | |
- | scheduled search slash notify | + | |
- | and as you can see it was processing one | + | |
- | saved search | + | |
- | found my search because | + | |
- | i um had that setting to | + | |
- | always send an email the first time | + | |
- | someone subscribes | + | |
- | it treats this as new so it sent a | + | |
- | virtual message | + | |
- | and now it's done so | + | |
- | if you want to use this feature all you | + | |
- | need to do | + | |
- | is add the scheduled search notify | + | |
- | command to a cron job that runs | + | |
- | probably on a daily basis and then your | + | |
- | server will send out emails as things | + | |
- | change | + | |
- | if you already have a script set up to | + | |
- | uh index records on a regular basis | + | |
- | popping in this notification at the end | + | |
- | of that script would make a lot of sense | + | |
- | to ensure that the latest information | + | |
- | gets sent out | + | |
- | at the moment that it is fresh and | + | |
- | that's | + | |
- | all there is to email alerts | + | |
- | for watching | + | |
- | reach out with questions or suggestions | + | |
- | see you in the next video | + | |
---- struct data ---- | ---- struct data ---- | ||
+ | properties.Page Owner : | ||
---- | ---- | ||
videos/xml_change_tracking_and_email_alerts.1631272914.txt.gz · Last modified: 2021/09/10 11:21 by demiankatz