Differences

This shows you the differences between two versions of the page.

--- videos:indexing_marc_records [2020/02/03 20:28] – [Transcript] demiankatz
+++ videos:indexing_marc_records [2023/04/26 13:26] (current) – crhallberg
@@ Line 1: / Line 1: @@
 ====== Video 2: Indexing MARC Records ======
-The second VuFind instructional video discusses how to index MARC records (and configure/customize the indexing process).
+The second VuFind® instructional video discusses how to index MARC records (and configure/customize the indexing process).
 Video is available as an [[https://vufind.org/video/Importing_MARC_Records.mp4|mp4 download]] or through [[https://www.youtube.com/watch?v=ayhWQFi3h_w&feature=youtu.be|YouTube]].
@@ Line 7: / Line 7: @@
 ===== Related Resources =====
+  - [[indexing:marc#related_video|Indexing MARC Records wiki page]]
   - [[indexing:solrmarc|SolrMarc wiki page]]
   - [[https://github.com/solrmarc/solrmarc|SolrMarc Documentation]]
@@ Line 13: / Line 14: @@
 ===== Transcript =====
-:!: Needs editing and cleanup
+Alright.
+So in this video, we are going to
- alright so in this video we are going to
 talk about loading mark records into
-view fines which is a good first step
+VuFind, which is a good first step
-after you have installed it as was
+after you have installed it, as was
-described in our previous video because
+described in our previous video. Because
 they are the easiest to load we are
-going to work exclusively with mark
+going to work exclusively with marc
 records today and I'm going to assume
-that you know what a mark record is but
+that you know what a marc record is but
-just the one sentence description is
+just the one-sentence description is
 it's a standard interchange format for
 library records where the data is split
 up into numeric fields with alphabetic
-subfields so mark records are included
+subfields. So marc records are included
-with view find as part of its test suite
+with VuFind as part of its test suite
-so if you just want to kick the tires
+so if you just wanted to kick the tires,
 it's really easy to get some records
-loaded in there you just switch into
+loaded in there. You just switch into
-your view finder Ector e and at the
+your VuFind directory and at the
-moment I'm actually looking at the
+the moment I'm actually looking at the
 virtual machine I set up in the previous
-video and I'm in the view finder Ector e
+video and I'm in the VuFind home directory.
-if I do an LS of tests data star dot MRC
+If I do an "ls tests/data/*.mrc",
-I can see all of the mark records that
+I can see all of the marc records that
-are available so first things first we
+are available.
-need to start up our solar index which
-is where if you find stores all of its
+So first things first we
-records we can't put things in the index
+need to start up our Solr index, which
-until the index is running so I do that
+is where VuFind stores all of its
-with solar Sh start and then it's going
+records. We can't put things in the index
-to tell me that solar is starting up on
+until the index is running. So I do that
-port 8080 a fact that it's useful to
+with "./solr.sh start" and then it's going
+to tell me that Solr is starting up on
+port 8080 a fact that is useful to
 remember for later since we might want
-to look at the solar index once it has
+to look at the Solr index once it has
-records in it and now that solar is
+records in it. And now that Solr is
 running there's just a handy script
-called import - mark dot SH that if we
+called "import-marc.sh" that if we
-feed it the name of a mark file which
+feed it the name of a marc file (which
-can be either binary mark or mark XML it
+can be either binary marc or marc XML) it
 will load everything into the index for
-us so I'm going to choose journals dot
+us. So I'm going to choose journals.mrc
-MRC which contains some journal records
+which contains some journal records
 and it takes just a couple seconds to
-load all of the data into the index once
+load all of the data into the index, once
-it gets going and there we go so I'm
+it gets going. And there we go! So I'm
 going to go to my web browser
-now and just do a blank search in view
+now and just do a blank search in
-find which is a good way to get a list
+VuFind which is a good way to get a list
 of all the records that have been
 indexed and I will see that I now have
 ten journal records appearing in the
-search results so what just happened it
+search results.
-might be useful to take a step back and
-talk a little bit about what solar is
+So what just happened?
-Solar is an open source search engine
+It might be useful to take a step back and
+talk a little bit about what Solr is.
+Solr is an open-source search engine
 which essentially takes a set of keys
 and values and applies some rules to
 those to make them very easy to search
-and perform faceted filtering on and so
+and perform faceted filtering on.
-what we did with our script was map some
+So what we did with our script was map some
-mark records into solar so I mentioned
+marc records into Solr.
-when I started up solar that it's useful
+I mentioned
+when I started up Solr that it's useful
 to make note of the port number that
 it's running on and that is because
-solar has a web interface that you can
+Solr has a web interface that you can
-actually look at if you just go to the
+actually look at. If you just go to the
 name of the server where it's running
 access the port it's running on and then
-go to slash solar you will you will find
+go to "/solr" you will find
-a handy administrative interface solar
+a handy administrative interface. Solr
 arranges its records into a set of cores
 so that you can store different types of
 records in different index schemas and
-in the case of you find the main records
+in the case of VuFind the main records
-are stored in a core named Biblio so if
+are stored in a core named "biblio".
-we go to Biblio and we go to the
+So if we go to biblio and we go to the
 querying tool we can run their default
-star colon star everything search and we
+"*:*" everything search and we
 will see that here indeed are the ten
 records that were put into the index and
 we can see some of the field names and
 values that got placed there so all of
-this data was mapped out of those mark
+this data was mapped out of those marc
-records that we loaded so how did the
+records that we loaded.
+So how did the
 indexing tool know what to do with the
-contents of the mark records the answer
+contents of the marc records?
-is there is a configuration file called
+The answer is there is a configuration file
-mark dot properties in view finds index
+called "marc.properties" in VuFind's
-sorry in view
 import directory which contains all of
-the rules for mapping mark fields into
+the rules for mapping marc fields into
-solar fields so if we edit this file we
+Solr fields. So if we edit this file we
 can browse through and see some of the
-mappings that are present this is using
+mappings that are present. This is using
-a separate open source tool called solar
+a separate open-source tool called SolrMarc
-mark which is special built for mapping
+which is special-built for mapping
-mark records into solar and so it
+marc records into Solr and so it
 contains a domain-specific language for
-specifying these mappings the simplest
+specifying these mappings. The simplest
 thing that you can possibly do is set a
 field to a fixed string so that every
 record you import will just have those
-particular values and in view finds
+particular values and in VuFind's
 default we set everything to a
-collection of catalog and institution of
+collection of catalog, an institution of
-my institution and a building of library
+"My Institution", and a building of "Library A".
-a these are obviously settings that you
+These are obviously settings that you
 will want to override and I'll show you
-how to do that in a little bit so mark
+how to do that in a little bit.
-records contain two types of fields the
-first nine fields go one through oo9 are
+So marc records contain two types of fields
-called control fields and they simply
+the first nine fields (001 through 009)
+are called control fields and they simply
 have numeric field designations without
-subfields you can see that by default of
+subfields you can see that by default VuFind
-you find takes the first value found in
+takes the first value found in an 001 field
-an oo 1 field
 and stores that as the unique identifier
-for the record IDs are really important
+for the record. IDs are really important
-in the solar index because that's how
+in the Solr index because that's how
-solar tells records apart every record
+Solr tells records apart. Every record
 must have a unique ID and if you index
-two records with the same ID the second
+two records with the same ID, the second
 record you index will replace the first
-one so if your mark records don't have
+one. So if your marc records don't have
-their ideas in the oo-one field this is
+their ideas in the 001 field this is
 another detail that you will need to
-customize so after those first 9 control
+customize.
-fields all of the other fields in a Marc
+So after those first 9 control
+fields all of the other fields in the marc
 record are called data fields and they
 have alphabetic subfields containing
 different bits of data and we store a
 lot of things from those data fields in
-the solar index so for example the
+the Solr index. So for example the
-second line here we get the library of
+second line here we get the Library of
-congress card number out of the Oh 100
+Congress card number out of
-field sub
+the 010 field, subfield a
-filled a and this first designation
+and this first designation
 again just stores only the first value
-in the solar index similarly here we
+in the Solr index.
-take the oath right o three five a
-control number but because we're not
+Similarly here we
+take the 035a control number
+but because we're not
 specifying only taking the first one
 that could potentially store multiple
-values in the index moving down a little
+values in the index.
-bit to how we index languages this is a
+Moving down a little
-little bit more complicated taking this
+bit to how we index languages. This is a
-part by part the solar mark field
+little bit more complicated. Taking this
+part by part, the SolrMarc field
 specifications can have a colon
 separated list of different things to
 pull out and it will take all of those
 values and store them in the index as
-separate values so here we are taking
+separate values. So here we are taking
-characters 35 through 37 of the oo 8
+characters 35-37 of the 008 control field
-control field which is usually a
+(which is usually a language code)
-language code as well as oh for one a D
+as well as 041a, d, h, and j
-H and J all as separate values and solar
+all as separate values and SolrMarc
-mark supports what it calls translation
+supports what it calls translation
 maps which can take one string and map
-it into another there's a folder in the
+it into another. There's a folder in the
-translation Maps subfolder of import
+translation maps subfolder of import
 called language map properties which
 translates all of the three-letter
-language codes that are found in mark
+language codes that are found in marc
 records into the full English names for
-those codes you'll also see in here that
+those codes.
+You'll also see in here that
 there are some custom methods and when
-solar mark sees the word custom that
+SolrMarc sees the word "custom" that
 means it's looking for a Java function
-with the name that follows so there's
+with the name that follows. So there's
 actually a Java class that ships with
-you find and contains a get formats
+VuFind and contains a getFormats
 method which contains all the rules for
-specifying record formats looking at
+specifying record formats. Looking at
 that is beyond the scope of this
 introductory video but by including
 custom methods in Java it is possible to
 write your own and to extend the ones
-that ship with you find and that might
+that ship with VuFind and that might
-be a good topic for a future video
+be a good topic for a future video.
-also you can apply translation maps to
+Also, you can apply translation maps to
 the output of custom methods so there's
 a format map properties which takes the
 very granular formats that come out of
 our Java code and translate them into a
-smaller set of more human readable
+smaller set of more human-readable
 values that will be displayed in the
-index one last example of some of the
+index. One last example of some of the
-power and flexibility of solar mark is
+power and flexibility of SolrMarc is
-here at the very bottom so the pattern
+here at the very bottom. So the pattern
 maps I mentioned before or rather the
 translation maps I mentioned before are
 separate files but you can also include
-translation maps in line in the mark dot
+translation maps inline in the
-properties file by giving them a name
+marc.properties file by giving them a name
-and wrapping them in parentheses and the
+and wrapping them in parentheses.
-translations don't have to be straight
+The translations don't have to be straight
-string to string you can also define a
+string-to-string you can also define a
-set of regular expression rules to apply
+set of regular expression rules to apply.
-so these last five lines of the file are
+So these last five lines of the file are
 the rules for extracting the numeric
-portion of OCLC numbers found in oh 3 5
+portion of OCLC numbers found in 035a.
-a so the rule says take all the values
+So the rule says take all the values
-from O 3 5 a and apply this pattern map
+from O35a and apply this pattern map
-and then this pattern 0 pattern 1
+and then this pattern_0, pattern_1,
-pattern 2 pattern 3 stuff is providing a
+pattern_2, pattern_3 stuff is providing a
 series of regular expressions that will
 be tested against all the strings that
 are extracted and if matches are found
-mappings will be made so this for
+mappings will be made. So this, for
-example takes the different case
+example, takes the different case
-variations of OC olc in parentheses and
+variations of O-C-O-L-C in parentheses and
 then extract just the numeric portion
-using a regular expression sub pattern
+using a regular expression subpattern
-matching to dollar 1 and then the step
+matching to $1. Then this gets the
-Co CM versions the OC n versions and the
+OCM versions, the OCN versions, and the
-O n versions and if you don't know what
+ON versions and if you don't know what
-I'm talking about there are a variety of
+I'm talking about: there are a variety of
 different ways that OCLC numbers are
 prefixed depending on the age and source
-of the records and for view fines
+of the records and for VuFind's
 purposes we only care about the numbers
-hence all of these mapping rules
+hence all of these mapping rules.
-so now that I've showed you how if you
-find works by default it would be useful
+So! Now that I've shown you how VuFind
-to know how that you how you can change
+works by default, it would be useful
-and customize these things so before I
+to know how you can change
-die of completely into that a quick
+and customize these things. Before I
-aside about the view find local
+dive completely into that, a quick
-directory which is an important concept
+aside about the VuFind local
+directory, which is an important concept
 for not just import but also
-configuration more generally of you find
+configuration more generally of VuFind.
-so when you setup you find the installer
+So when you setup VuFind, the installer
 gives you two environment variables
-there's few find home which in this
+there's VUFIND_HOME, which in this
-instance is user local view find and
+instance is "/usr/local/vufind" and
-there's view find local der which in
+there's VUFIND_LOCAL_DIR, which in
-this instance is user local view find
+this instance is "/usr/local/vufind/local".
-local the reason for this local
+The reason for this local
 directory is to give you a place to
 store all of your local files separately
 from the files that are distributed as
-part of you find this offers a couple of
+part of VuFind. This offers a couple of
-advantages one being that you can have
+advantages. One is that you can have
-multiple configurations of you find at
+multiple configurations of VuFind at
 once and switch between them by just
-changing the variable that tells you
+changing the variable that tells VuFind
-find where to load its configurations
+where to load its configurations
-the other is that when you upgrade you
+the other is that when you upgrade VuFind
-find you only need to keep track of a
+you only need to keep track of a
 couple of directories of your local
 files the rest you can delete and
 replace with a new version which makes
 the upgrade process a little bit more
-streamlined but in general the way if
+streamlined. In general, the way VuFind
-you find uses the local directory is
+uses the local directory is
 that when it's trying to load a
 configuration of any sort it will look
-in the local directory first if it finds
+in the local directory first.
-something there it will use it if it
+If it finds something there it will use it.
-does not find it there it will go up to
+If it does not find it there it will go up to
-view find home so this is why there are
+VUFIND_HOME. So this is why there are
 a lot of parallel structures so for
 example there's an import directory
-inside view find home which contains
+inside VUFIND_HOME which contains
 lots of default configurations but
 there's also a local import directory
 which contains a smaller number of files
-which are all local overrides and as you
+which are all local overrides. As you
-can see here when you first install view
+can see here when you first install
-find you actually get a couple of files
+VuFind you actually get a couple of files
-in there for free import properties and
+in there for free: "import.properties" and
-import off dot properties and these are
+"import_auth.properties". These are
-the files which tell Solr mark where to
+the files which tell SolrMarc where to
 find its configurations and
 other dependencies and so they are
-auto-generated for you by the Installer
+auto-generated for you by the installer
-to point at the directories that you
+to point at the directories that VuFind
-find is being installed into import dot
+is being installed into.
-properties is the primary configuration
+"import.properties" is the primary configuration
 file for loading into the biblio core
-and import auth is for loading authority
+and "import_auth" is for loading authority
 records which again may be a topic for a
-future video let's just take a quick
+future video.
-look at import dot properties so you can
-see what's in there it's not a whole lot
+Let's just take a quick
-we just specify which core name we're
+look at "import.properties" so you can
-loading things into we're specifying the
+see what's in there. It's not a whole lot.
+We just specify which core name we're
+loading things into. We're specifying the
 names of the properties files that we're
-using for mappings so as you can see
+using for mappings. So as you can see
-here is the mark dot properties file I
+here is the "marc.properties" file I
 showed you and also a second file called
-mark local dot properties which is a way
+"marc-local.properties" which is a way
 that you can create a file that
-overrides parts of mark dot properties
+overrides parts of "marc.properties"
 without overwriting all of it and we
 will be working with that in a little
-bit also we have the URL where solar is
+bit. Also we have the URL where Solr is
-running so that solar mark can push the
+running so that SolrMarc can push the
-records into its update Handler and then
+records into its update handler and then
-solar mark dot path which is the
+"solrmarc.path" which is the
-location of the solar mark files and as
+location of the SolrMarc files and as
 you can see we specify first the local
 import directory and then the main
 import directory which is how we load
-local files ahead of default files then
+local files ahead of default files. Then
 there are also a few defaults about
 error handling and character encodings
 that you should be able to leave alone
 but which are here for you to tweak if
-you ever need to so with all of that out
+you ever need to.
-of the way let's start customizing some
-things so the first thing that we should
+So with all of that out of the way,
-do is create a mark local dot properties
+let's start customizing some things.
+The first thing that we should
+do is create a "marc-local.properties"
 file in our local import directory so we
 have a local place to begin putting our
-customizations
+customizations.
-so the easiest way to do that is to copy
+So the easiest way to do that is to copy
-the standard mark local dot properties
+the standard "marc-local.properties"
-which contains a whole lot of commented
+which contains a whole lot of
-out examples that might be useful in to
+commented-out examples that might be useful
-local import and then edit it there so
+into local import and then edit it there.
-as I mentioned by default view find
+As I mentioned by default VuFind
-indexes everything to my institution and
+indexes everything to "My Institution" and
-library a and so some of the suggestions
+"Library A" and so some of the suggestions
-here in mark local are about changing
+here in marc-local are about changing
-that so I'm going to call the
+that. So I'm going to call the
-institution Damian's University and I'm
+institution "Demian's University" and I'm
-going to call the building
+going to call the building "Demian's house"
-Damian's house because I have started a
+because I have started a university
-University in which I record videos
+in which I record videos about VuFind.
-about view func so now having made that
+So now having made that
-change I need to re index all of my
+change, I need to re-index all of my
 records in order for the change to take
-effect whenever you change your import
+effect. Whenever you change your import
 rules you need to reload your records
-because the mappings occur at index time
+because the mappings occur at index time.
-so once again import mark Sh tests data
+So once again:
-journals mark we wait a few seconds
+"./import-marc.sh tests/data/journals.mrc".
+We wait a few seconds
 while the records all load and now if I
-go back to my web browser and my view
+go back to my web browser and my VuFind
-find search results you'll see that when
+search results you'll see that when
-I loaded things before my institution in
+I loaded things before "My Institution"
-library a were showing up in these
+and "Library A" were showing up in these
-facets but if i refresh the page my
+facets but if I refresh the page my
-changes have taken effect it's now
+changes have taken effect. It's now
-Damien's University and Damian's house
+Demian's University and Demian's House.
-please don't come to my house now going
+Please don't come to my house.
-back to the terminal let's look at
+Now going back to the terminal let's look at
-another thing we might want to change so
+another thing we might want to change.
-as I mentioned this is a file full of
+As I mentioned this is a file full of
-journals so if I look at the format
+journals, so if I look at the format
 facet I'll see there are 10 journals
 here but suppose that I don't like
 calling them journals
 I want to degrade them by calling them
-magazines so that is something I can
+magazines. That is something I can
-change by editing the format translation
+change by editing the
-Maps that I mentioned earlier
+format translation maps that I mentioned earlier.
-so once again we can create a copy of
+So once again we can create a copy of
 the map in our local directory and this
-will override the default map so first
+will override the default map. First
 let me create a directory to hold my
-custom Translation maps because you're
+custom translation maps because you're
 not going to have such a directory by
-default so we have to create local
+default. We have to create
-import translation Maps and then I can
+"local/import/translation_maps"
-copy import translation Maps format
+and then I can copy
-properties sorry
+"import/translation_maps/format_map.properties"
-format map dot properties into local
+into local "local/import/translation_maps".
-import translation Maps I've now created
+I've now created a local copy
-a local copy that I can customize so I
+that I can customize, so I
-will edit that and as you can see here
+will edit that.
+As you can see here
 is where we map the more technical terms
-that come out of the custom get format
+that come out of the custom getFormat
-method into more human readable versions
+method into more human-readable versions
 but down here we just map journal
 straight back into journal again but I
-can change this journal equals magazine
+can change this.
+"Journal = Magazine"
 and now if I just go back and import my
 records one more time now when I go back
 to my results and refresh the page
-Journal has just turned into magazine so
+"Journal" has just turned into "Magazine".
-as you can see it's pretty easy to
+As you can see it's pretty easy to
 customize some of these strings but
-let's look at some more complicated
+let's look at some more complicated things.
-things so as I mentioned languages are
+As I mentioned languages are
 currently pulled from a whole bunch of
 fields and run through a translation map
 but let's suppose that I don't want to
 display those English language names for
-some reason perhaps I would rather set
+some reason. Perhaps I would rather set
 up some settings in my code to change
-them into something on the fly what I
+them into something on the fly. What I
-can do is look at the rule in the core
+can do is look at the rule
-import mark properties and I am going to
+in the core import-marc.properties
-copy
+and I am going to copy this rule
-this rule then I'm going to edit my
+then I'm going to edit my
-local import mark local properties so I
+"local/import/marc-local.properties"
-can override it that's going to go to
+so I can override it.
-the bottom and I'm going to paste the
+Just going to go to the bottom
-rule in I'm actually going to paste it
+and I'm going to paste the rule in.
-in twice so I can keep the old version
+I'm actually going to paste it in twice
+so I can keep the old version
 commented out for reference and then I'm
-just going to get rid of the language
+just going to get rid of the language map.
-map so this will just take the codes and
+Now this will just take the codes and
 index them directly without doing any
-translation work on them
+translation work on them.
-so now I'm going to import the records
+So now I'm going to import the records
 one more time so again before I refresh
-the page to show the results let's see
+the page to show the results, let's see
 what this looked like before so here we
-have English French German Italian so
+have "English", "French", "German", "Italian".
-now I'm going to refresh the page and
+Now I'm going to refresh the page
-sure enough now we are only getting the
+and sure enough now we are only getting
-raw codes so one more interesting
+the raw codes.
-feature of solar mark is that you can
-actually combine together multiple rules
+One more interesting
-so I realize my example is getting a
+feature of SolrMarc is that you can
+actually combine together multiple rules.
+I realize my example is getting a
 little bit contrived but let's suppose
 that I actually wanted to see both the
-mapped versions and the raw versions
+mapped versions and the raw versions.
-what I can do is just put both versions
+What I can do is just put both versions
 of the rule in place but on the second
-one change the equals to a plus equals
+one change the "=" to a "+="
 and what this will do is it will take
 all the mapped versions put them in the
 language field and then also take all
 the raw versions and add those to the
-language field this kind of combination
+language field.
+This kind of combination
 can be really useful when you have
 information in multiple fields of the
-Marc record formatted in different ways
+marc record formatted in different ways
 and you want to normalize it in some
-fashion you can create a separate rule
+fashion. You can create a separate rule
 for each field and then combine all the
-fields together with plus equals but for
+fields together with "+=" but for
-this example we'll just do it this way
+this example we'll just do it this way.
-and now I will reindex one more time
+Now I will reindex one more time
-and now if i refresh this page one more
+and now if I refresh this page one more
-time and scroll down to language you'll
+time and scroll down to language, you'll
 see that we now have both the
-three-letter codes and the mapped names
+three-letter codes and the mapped names.
-so that covers the very basic ways that
-you can import mark records and
+So that covers the very basic ways that
-customize the way that they're mapped of
+you can import marc records and
-course there's a lot deeper dive
+customize the way that they're mapped.
+Of course, there's a lot deeper dive
 possible here because of looking at the
 custom Java code or the whole end-to-end
-experience of adding a new field to view
+experience of adding a new field to
-find not only indexing it but also
+VuFind not only indexing it but also
-making it display in the interface as I
+making it display in the interface.
-say these are good topics for the future
+As I said these are good topics for the future
 but for right now I'm just trying to
 introduce all of the basics if you want
-to learn more about solar mark and
+to learn more about SolrMarc and
 indexing be sure to take a look at the
-solar mark page in view finds wiki and
+SolrMarc page in VuFind's wiki and
-the solar mark github project both of
+the SolrMarc GitHub project, both of
 which have some more documentation on
 how to solve common problems using the
@@ Line 479: / Line 506: @@
 is actually quite rich and has quite a
 few features that can be strung together
-to do complicated transformations
+to do complicated transformations.
-there's also a solar mark tech mailing
+There's also a SolrMarc tech mailing
 list which is a great place to ask for
-help in addition to the usual view find
+help, in addition to the usual VuFind
-support venues so I hope this has been
+support venues.
+I hope this has been
 useful and I will share another video
-next month
+next month!
 ---- struct data ----
+properties.Page Owner :
 ----