Panl Fundamentals

At the heart of both Solr and Panl are text files which configure the respective servers.  In this chapter, the configuration files will be explained - whilst the main focus of this book is on the Panl configuration, there are parts of the Solr configuration which do impact the Panl server configuration, and do require some level of understanding (especially when things do not go as expected).

Configuration Files

There are four configuration files that have impact on the Panl server, two within Solr and, two (at a minimum) within Panl.

Solr Configuration:

1. solrconfig.xml - This file defines the default Solr field that will be used for a keyword search, and how the highlighting works.
   
   Whilst this file has a large list of configuration items, the only items that are covered by this book is the <requestHandler /> XML element with the name attribute's value of /query which defines the default Solr field to search upon.  See the line in the file^[21]:
   
           <requestHandler name="/query" class="solr.SearchHandler">
   
2. managed-schema.xml - This file defines the Solr fields, their names, field types, properties, and how they are processed (e.g. indexed, stored, analysed) by the Solr server.
   
   Each of these defined fields are then able to be configured for use by the Panl server in a variety of ways, the configuration options available depending on their Solr field type.

The Panl server relies on two files:

1. panl.properties file - This file configures the Panl server on

1. How to connect to the Solr instance,
2. Whether to enable the in-built web applications,
3. The verbosity of the error responses,
4. Decimal point formatting,
5. Removal of un-needed JSON keys from the response,
6. Extra information that can be passed back through the JSON response,
7. What Panl collections are available, and
8. Which Solr collections the configured Panl collections connect to.
   

2. <panl_collection_url>.panl.properties - There is one file for each Panl collection and its FieldSets (CaFUPs).  This file determines how the Panl URLs are generated and how each of the Solr fields are mapped and configured in the Panl server.

The Relationship Between Configuration Files

In the below diagram, the relationships and interactions between the files are briefly outlined:

Image: The Solr and Panl configuration files and the interaction points

Generating And Editing The Configuration Files

The process for generating and editing the configuration files is the same for every Solr collection and Panl CaFUP.

1. Copy the Solr configuration files for your Solr version to your project directory
2. Edit the solrconfig.xml to configure the default search, highlighting, and other options
3. Edit the managed-schema.xml files to configure the individual fields to be indexed, faceted, stored, and searched upon
4. Push the config and schema to the Solr server to create the collection
5. Upload the data to be indexed to the Solr server
6. Generate the Panl configuration files (Note: in most cases the generation will only be required to be performed once, to get you up and running quickly.  Further changes to the schema can be added manually.)
7. Edit the panl.properties and <panl_collection_url>.panl.properties files
8. Iterate over the process, either

1. Updating the Panl configuration and restarting the Panl server, or
2. Updating the managed schema file and going through the process from step 3.

Image: The dependencies and actions for the Solr and Panl configuration files.

Solr Configuration

The Solr configuration files MUST match the version for your solr installation. Both of the Solr files (and more) can be found in the

SOLR_INSTALL_DIRECTORY/server/solr/configsets/_default/conf/

In version 9.10.0, the following files and directories are present

lang/*.txt
managed-schema.xml
protwords.txt
solrconfig.xml
stopwords.txt
synonyms.txt

Copy these files and directories to your project directory (the below directory name is just a suggestion):

YOUR_PROJECT_DIRECTORY/config/solr/

Edit the Solr config and managed schema files to your project specifications and then run the Panl generator command line tool to output the generated files to (the below directory name is just a suggestion):

YOUR_PROJECT_DIRECTORY/config/panl/

Now you can edit the files and iterate your way to your solution.

Search Field Configuration

Being able to search on keywords within the indexed data is base (and rather obvious) functionality for any search engine.  

Within Solr, you may search either on:

1. A default search field
   (This is the default behaviour), or
2. A specific search field.
   (This is where a configured field, or fields are specifically searched upon)

Panl surfaces this functionality through its configuration.  To do this, both the Solr configuration and the Panl configuration must align.  This configuration spans across the two Solr configuration files, namely solrconfig.xml and managed-schema.xml, the configuration points and details of each file is explained below.

Image: The configuration differences between the default keyword search and the specific Solr keyword search.

In the above image, for the Solr Default Keyword Search, there is a defined default search field named text which is an analysed field.  All fields that require the default keyword search to be performed upon will be copied into this field.  When the default search is performed, the Solr query parameters will include q="keyword".

For the Specific Solr Search fields, the fields may still be copied into the default field, but if they are also to be used as a facet then they SHOULD NOT be analysed as well (however you may still wish to do this in a narrow set of use cases).

1. This is an analysed field which is NOT also configured to be a facet
   
   This can be configured as a specific search field and have the keyword searched upon it (and/or in conjunction with other specific fields)
2. This is an analysed field which should also be configured to be a facet.
   
   As the field is analysed, the words in the field will be broken into tokens (e.g. if the field was an author of "Joan Smith", two facets would be generated, namely "Joan" and "Smith").  From a faceting perspective, this is not desired, so the field is copied into another field (Field 3) that can then be used as the facet.
3. This is a facet field that is not analysed
   
   This is a copy of Field 2 without any analysis or tokenisation placed upon it.

When a specific field search is performed, the Solr query parameters will include q=field:"keyword", when multiple specific fields are targeted for the keyword, the Solr query parameters will include q=field1:"keyword"+OR+field2:"keyword".  (Note that the query operand is dependent on the configuration.)

The rules for the Default versus Specific Field Search are as follows:

• To be included in the default keyword search:

• Ensure the default field is analysed
• Copy the field into the defined default keyword search field

• To be included in a specific field keyword search:

• Ensure the field is analysed

• For the field to be also used as a facet field

• Copy the field to a non-analysed field

Keyword Search Configuration

The keyword search relies on a default field that is configured in Solr.  This default search field MUST always exist and MUST always be available.

solrconfig.xml

The solrconfig.xml file determines the default search field.  This default field is used if there is no configuration for Specific Solr Search Fields in both the Solr and Panl configuration files, and no specific field search is passed through as a Panl LPSE URL parameter.  For the Solr server, the snippet below shows the definition for the default field to search on, denoted by the <str /> XML element with the df (which stands for default field) attribute.

<str name="df">text</str>

The expanded XML definition including the default field to search upon is below:

This configures Solr to search for keywords ONLY within this field (i.e. text) if no specific search field is requested.  

managed-schema.xml

The text value of the <str name="df" /> element in the solrconfig.xml file is text which MUST map to a field in the managed-schema.xml Solr configuration file. This 'text' field snippet below is from the managed schema file:

The managed-schema.xml  (Bookstore sample file) configures the fields, their types (and consequently whether this field type is configured to be analysed), whether they are indexed, stored, and/or multivalued. A snippet of a managed schema file is below:

Throughout the book, the Solr field of text is used as the default field to be searched on, and any information that is required to have a keyword to be searched upon is copied to this field (see lines 9-13 above).  This can be thought of as a catch-all approach with the author, title, description, genre, and series values copied to the text field, which is then analysed and is searched upon for the default keyword search.

Lines 1-3 and Line 15 are used for the Specific Solr Search Fields configuration and are not used for the default search configuration.

For line 5 above, the type is text_general, which is defined by the fieldType Solr XML element below of :

Summary

To correctly configure the default keyword search field:

1. Check the solrconfig.xml for

1. the requestHandler element with value of the name attribute as /query,
2. and the child str element with the value of the name attribute as df
   The text of this element is the default field that Solr will use to perform keyword searches upon

2. Check the managed-schema.xml file for

1. The field element with the value for the Solr field with the name attribute as text,
2. Check the value of the type attribute against the fieldType element with the value of the name attribute and that this field type has child elements that are analysed.

The diagram below shows the relationship between configuration items for the default search:

Image: The Solr configuration items for the default search

The Solr text field above is used as a holding area to copy other fields into it so they may have the keyword search applied to them, whilst keeping the original un-analysed value. The below shows the relationship and elements of a cutdown version of the managed schema file for the mechanical pencils.

Image: The Solr fields and the copyField directive.

Specific Solr Field Keyword Search Configuration

To define specific fields to be searched upon, (either individually or collectively), rather than the default keyword search field, additional configuration is required in both the managed-schema.xml and <panl_collection_url>.panl.properties.

We will be using a different pattern from the default field configuration (which uses <copyField /> XML elements) to ensure that, where required, the specific field can be used as a facet AND as a Specific Solr Search Field. 

managed-schema.xml

Within Solr the three fields text_author, title, and description MUST be of a type that has an analyser so that they can be searched upon.  All three are set to the type text_general which is then indexed by Solr.

Line 9:

You may be wondering why there is a field definition for text_author and author, and why the field definition for author wasn't set to a type of text_general, which would enable the Specific Solr Search Field keyword search (as is done with the title and description fields).  The simple answer is that the author field is also going to be configured as a facet, and it it were set to t type that was text_general, then the individual words of the author would be tokenised and instead of having facets like the following:

• John Smith (1)
• Joan Smith (1)
• John Cavendish (1)

The tokenised facets would be displayed as:

• John (2)
• Smith (2)
• Joan (1)
• Cavendish (1)

<panl_collection_url>.panl.properties

To activate Specific Solr Search Fields, in the <panl_collection_url>.panl.properties, you will need to configure the list of analysed fields that Panl can work with by using the panl.search.fields: e.g.

panl.search.fields=title,\
 text_author,\
 description

Additionally, for any Solr fields listed within the panl.search.fields configuration above, There must be a corresponding Panl field definition of the form panl.search.<lpse_code>=<solr_field>.

Summary

To correctly configure a Specific Solr Search Field:

1. Ensure that the managed-schema.xml has

1. The Solr field element for the field which is analysed (e.g. text_general)

2. Configure the <panl_collection_url>.panl.properties file such that

1. The analysed field is included as a the Panl Search field configuration (i.e. add a property panl.search.<lpse_code> (Note that this is in addition to setting it as a facet or field)
2. The analysed field is also included in the comma separated list for the property panl.search.fields.

Below is an image of the configuration for Specific Solr Search Fields:

Image: The specific Solr search fields in the Panl configuration file

The above image shows the configuration for three fields to be Specific Solr Search Fields, namely, text_author, description, and title.

The Bookstore walkthrough goes into greater detail, which includes boosting query terms.

Panl Field Configuration

The <panl_collection_url>.panl.properties has a raft of configuration items that will impact the facets and results that are available and construction of the LPSE URL.  This part will focus on the configuration of the Panl fields, although the following items  are also configured within the properties file:

• LPSE code length,
• the

• query,
• sort,
• page,
• number of rows,
• query operand, and
• passthrough LPSE codes

• The URL parameter that the keyword search will respond to
• Whether to return facets that only have one result
• Whether to return facets which have the same number of results as returned documents
• Number of rows for default result, lookahead, and maximum,
• Highlighting,
• Ordering of the LPSE codes,
• FieldSets,
• Sort fields, and
• Search fields.

To configure the Panl fields, the configuration items will always contain the following properties:

panl.<field_type>.<lpse_code>=<solr_field>
panl.name.<lpse_code>=<panl_name>
panl.type.<lpse_code>=<solr_field_type_class>

And will optionally contain the following property (only if the Solr field definition element has the attribute and value: multiValued="true")

panl.multivalue.<lpse_code>=<true>

Where:

• <field_type> is one of

• facet, or
• field.

• <lpse_code> is the configured Panl LPSE code
• <solr_field> is the Solr field that matches the name as defined in the managed schema configuration file.
• <panl_name> is the more human readable text that can be used to be displayed in the UI.
• <solr_field_type_class> is the truncated Solr class name which is duplicated from the managed schema file.

Note: An additional property can also be added to either a facet or a field of the form panl.search.<lpse_code>=<solr_field> which will add this to the list of Specific Solr Search Fields.

Image: The Basic Panl field definitions which can be automatically generated by the in-built Panl Generator.

When configuring the Panl, there are two types of fields available, a regular field and a facet field.  

Panl Regular Field

A Panl regular field (or just a Panl field) is a field that can be sorted on, returned in the results, but it WILL NOT be returned as a facet.  These fields are useful to return additional information with the results.  For example, in the mechanical pencils configuration file, the 'Body Shape' and 'Grip Type' for a specific pencil are not facets, however they can be returned with the results, and can also be configured to be a Search field (if they are analysed by Solr).

All Panl Regular Fields have the configuration of the form:

panl.field.<lpse_code>

An example of a field from the mechanical pencils collection:

panl.field.b=book_image
panl.name.b=Book Image
panl.type.b=solr.StrField

Which defines the Solr field of book_image as a Panl Regular Field.  

You may also define any field as a Specific Solr Search Field by setting the property (however this must also be correctly configured in the Solr server as well):

panl.search.<lpse_code>=<solr_field_name>

Remember: The Solr field that this references MUST be configured to be analysed in the managed-schema.xml file.

Panl Facet Field

A Panl facet field is a field that can be faceted, sorted, and returned with the results.  It can also be configured to be a Specific Solr Search Field (although this is not recommended for the majority of use cases). When configured as a Facet Field, the options that are available are dependent on the Solr field type class as set in the property panl.type.<lpse_code>=<solr_field_type_class>.

All Panl Facet Fields have the configuration of the form:

panl.facet.<lpse_code>

An example of a field from the mechanical pencils collection:

panl.facet.D=decade_published
panl.name.D=Decade Published
panl.type.D=solr.IntPointField

Depending on the field type, various options are available for configuration.  For full details on configuration options for different facet types, see the section on Facet Definitions.

An example of a RANGE facet from the mechanical pencils configuration file.  The first three lines below are the standard field definitions, the rest of the lines define this Panl field as a RANGE facet with its various allowable configuration items.

panl.facet.w=weight
panl.name.w=Weight
panl.type.w=solr.IntPointField

panl.suffix.w=\ grams
panl.range.facet.w=true
panl.range.min.w=10
panl.range.max.w=50
panl.range.prefix.w=weighing from
panl.range.infix.w=\ to
panl.range.suffix.w=\ grams
panl.range.min.value.w=from light
panl.range.max.value.w=heavy pencils
panl.range.min.wildcard.w=true
panl.range.max.wildcard.w=true
panl.range.suppress.w=false

Setting The Field To Allow A Specific Search

If the Solr field is analysed, then the Panl field may also be configured to be a Specific Solr Search Field, an example from the Bookstore configurations file:

panl.search.T=text_author

See the section above on Specific Field Keyword Search Configuration details about how to ensure that these fields are correctly configured in both the Panl and Solr servers.

Summary of Panl Field Configuration Options

Summarised below are the Panl Field types and the available options and configuration items available.

(*) Provided that the underlying Solr field definition is analysed.  Note: Setting a Facet Field to be a Specific Solr Search Field is only useful in a narrow set of use-cases, and in general, this should not be done.

~ ~ ~ * ~ ~ ~