Panl Cookbook

The type of the Solr field and Panl type will determine the configuration options available and the information that is contained within the response object.

The Panl JSON response attempts to provide the implementer with enough information to determine the rendering to the interface, rather than hard-coding Solr field names and deciding on how they should be handled.

In this chapter, some 'recipes' for how to implement various faceted search queries that have been seen on other websites.

SEO Friendlier Canonical URLs

Returning to the previous Amazon example^[49] for the rOtring Rapid Pro Mechanical pencil, the URL is as follows:

https://www.amazon.com/rOtring-1904260-Rapid-Mechanical-Pencil/dp/B0055ZV8LK

Deconstructing the URL:

• rOtring - the manufacturer's name
• 1904620 - the product code (or item number) for rOtring - see
  https://www.rotring.com/pens-pencils/mechanical-pencils/rapid-pro-mechanical-pencil/SAP_1904260.html - and note the SEO friendlier URL - the only part which is of interest is the SAP_1904260.html file.) 
• Rapid-Mechanical-Pencil - a combination of a partial model name (Rapid Pro) and the type of instrument (Mechanical Pencil)
• /dp/ - the path separator
• B0055ZV8LK - the Amazon product code

With the only important part (for the Amazon back-end servers) being the product identifier which is B0055ZV8LK, and the URL of

https://www.amazon.com/dp/B0055ZV8LK/

Will link to exactly the same page.

To replicate this with the Panl server, the pencil that is referenced by the above is:

http://localhost:8181/panl-results-viewer/mechanical-pencils/default/Manufactured%20by%20Rotring%20Company/Rapid%20PRO/rotring/bNq/ 

Which is the search query that will only return one result:

Note at the bottom the id Solr field (with a LPSE code of i) with the value of 50 - which is the document id for the Solr result, which was generated from the Synapticloop database.  This ID is the unique reference to the Solr document, and the URL:

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/50/i/

With a passthrough parameter set (in the example this is 'z') then any string could be added to the URL and the LPSE included, for example:

 http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/rOtring-1904260-Rapid-Mechanical-Pencil/50/zi/

Configuration Details

In the <panl_collection_url>.panl.properties file ensure the following:

1. The panl.param.passthrough property is set (in the examples this is set to 'z')
   This will enable the functionality to build the SEO friendlier URLs
2. The panl.param.passthrough.canonical is set to true
   This will ensure that the passthrough parameter is used to generate the canonical URL
3. The id Solr field is configured as a Panl facet, i.e.: panl.facet.i=id
   This will enable Solr to look this up as a facet and pass the value to it
4. The LPSE code (i.e. 'i') for this facet is included in the panl.lpse.order property
   This will ensure that Panl has enabled the Solr field to be faceted upon and the value will be passed through to the Solr server (as opposed to ignoring it).
5. The Solr Field name (Panl LPSE code - 'id') is included in the panl.lpse.ignore property
   This will ensure that Panl does not include this LPSE code in the facetable fields (which are presented on the left hand side on the in-build Panl results viewer web app).

The mechanical-pencils.panl.properties file snippets for the above:

Recommendation

When attempting to use a passthrough parameter for a canonical URL, it may be better to have a service that automatically generates the canonical URL, rather than relying on passthrough parameters.  For example, for the mechanical pencil above, a service that generates and concatenates the brand name, model number, model name and associated colour from the unique identifier may be a better choice than using the Panl server.

This will ensure that the passthrough parameter canonical URL is generated correctly and uniquely, that way, no matter the passthrough parameter URL that is requested, the actual canonical URL remains the same.

BOOLEAN Facets

Apart from the boolean true/false value replacement, the BOOLEAN facet can be designated as a checkbox.

Displaying a BOOLEAN facet as a checkbox

This is only useful when you want to select only one of the true/false values or no value (i.e. both).

Even though a BOOLEAN facet has only two values, there are actually three states that a BOOLEAN facet can have:

1. 'True' selected - only those results with a true value will be returned
2. 'False' selected - only those results with a false value will be returned
3. Not selected - all results are returned which have either a true or false value.
   There is an additional state in which the document does not have any value assigned to this facet

Using a checkbox is different from using the in-built Panl functionality in that you may select 'true', 'false' or remove the selection.  In the case of a checkbox, you will only be able to select one of the true or false values, or select neither.

This is a good use case if you want to highlight only one of the values.  

True Value BOOLEAN Facet Checkbox Example

As an example shopping sites may have a 'Speedy' delivery checkbox, which will filter those results which are available for speedy delivery, however the shopping site does not wish to highlight the results that do not qualify for speedy delivery.  Hence the facet may be selected as either 'True' or no facet value at all.

False Value BOOLEAN Facet Checkbox Example

As an example, shopping sites may have items which are on backorder, thus making them unavailable for immediate delivery, but still deliverable once the backorder has been fulfilled.  For the user experience, you may wish to present a 'Exclude items on backorder' checkbox.  In effect this will set the boolean value for 'backorder' to 'false', if unchecked then all items will be shown, both those on backorder, and those not on backorder'.

In either of the above cases, any BOOLEAN facet can be turned into a checkbox, provided that you understand how this impacts the facet selection.

For the implementation, either the LPSE code, or the Solr field name must be known ahead of time.  This does tie the implementation to the data - (there is an additional piece of functionality which is under consideration to be able to set a 'checkbox' property on a BOOLEAN Facet), in the meantime, the implementation is as follows:

Using the contrived example of:

• Solr field name - JSON key name facet_name = in_built_eraser
• LPSE code - JSON key name panl_code = I

If the facet with the above Solr field name or LPSE code:

• Is in the panl.active.facets JSON Array, then the checkbox should be selected.
  
  To deselect the checkbox, use the remove_uri JSON property.
  
• Is in the panl.available.facets JSON Array, then render a checkbox with the value deselected.  
  
  To select the checkbox, use the uris.before and uris.after in conjunction with the searching through the values array for the value key equalling either "true" of "false" and use the value of the encoded key.

RANGE Facets

'Reverse' star Ratings

Many sites allow ratings of items (and comments), however, rather than searching for a rating between two values, they only permit a variable start range and upwards.  For the URL.

https://www.ryman.co.uk/stationery/pens/ballpoint?bv_rating=[2+TO+*]

The facet that is rendered:

Example Properties

To generate this example, the following <panl_collection_url>.panl.properties file could be used.

To generate the links, iterate from the panl.range.min.r property value to the panl.range.max.r property value and generate the link of

panl.prefix.r + <value> + panl.infix.r + panl.range.max.value

With the appropriate before and after links from the JSON response.

From the JSON response object, the above link generation would become:

uris.before + <value> + uris.during + uris.after_max_value

Which would generate a URL which would look like:

/Rating from:1-star-and-UP/r-/

REGULAR Facets

Condensed Multivalued Paths

This is especially useful where there is a prefix and/or a suffix that gets applied to every single facet value that is selected.  

Condensing the path is only available on REGULAR facets that are defined as multivalued in the Solr schema (i.e. on the XML field definition there is an attribute and value of multiValued="true") AND have the panl.multivalue.<lpse_code>=true property set in the <panl_collection_url>.panl.properties file.

In this example, the original mechanical-pencils.panl.properties file has been edited so that the Colours facet has a prefix of 'Colour:':

Example Properties - Original

Base URL:

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname 
(0 characters - not including the CaFUP)

One Facet, adding 'Black'

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/Colour:Black/W/ 
(15 characters - not including the CaFUP)

Two Facets, adding 'Blue'

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/Colour:Black/Colour:Blue/WW/ 
(28 characters - not including the CaFUP)

Three Facets, adding 'Red'

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/Colour:Black/Colour:Blue/Colour:Red/WWW/ 
(40 characters - not including the CaFUP)

Four Facets, adding 'Green'

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/Colour:Black/Colour:Blue/Colour:Red/Colour:Green/WWWW/ 
(54 characters - not including the CaFUP)

Five Facets, adding 'Yellow'

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/Colour:Black/Colour:Blue/Colour:Green/Colour:Red/Colour:Yellow/WWWWW/ 
(69 characters - not including the CaFUP)

Six Facets, adding 'Brown'

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/Colour:Black/Colour:Blue/Colour:Green/Colour:Red/Colour:Yellow/Colour:Brown/WWWWWW/ 
(83 characters - not including the CaFUP)

The  above URL will select pencil models that have colours in Black, Blue, Green, Red, Yellow, and Brown (There is only one result).  

Note the 6 'W' LPSE codes.

To condense the LPSE URL, configure Panl to have a multi value separator

Example Properties - Condensed

In the above configuration, note that the prefix is now 'Colours:' (plural) as opposed to the prefix of 'Colour:' in the previous configuration.  The impact of this is that the first facet will increase the LPSE URL length by 1, which has less of an impact as more facets are added.

Base URL:

http://localhost:8181/panl-results-viewer/mechanical-pencils-multi-separator/brandandname   
(0 characters - not including the CaFUP)

One Facet, adding 'Black'

http://localhost:8181/panl-results-viewer/mechanical-pencils-multi-separator/brandandname/Colours:Black/W/
(16 characters - not including the CaFUP)

Two Facets, adding 'Blue'

http://localhost:8181/panl-results-viewer/mechanical-pencils-multi-separator/brandandname/Colours:Black,Blue/W/
(21 characters - not including the CaFUP)

Three Facets, adding 'Red'

http://localhost:8181/panl-results-viewer/mechanical-pencils-multi-separator/brandandname/Colours:Black,Blue,Red/W/
(25 characters - not including the CaFUP)

Four Facets, adding 'Green'

http://localhost:8181/panl-results-viewer/mechanical-pencils-multi-separator/brandandname/Colours:Black,Blue,Red,Green/W/ 
(31 characters - not including the CaFUP)

Five Facets, adding 'Yellow'

http://localhost:8181/panl-results-viewer/mechanical-pencils-multi-separator/brandandname/Colours:Black,Blue,Green,Red,Yellow/W/ 
(38 characters - not including the CaFUP)

Six Facets, adding 'Brown'

http://localhost:8181/panl-results-viewer/mechanical-pencils-multi-separator/brandandname/Colours:Black,Blue,Green,Red,Yellow,Brown/W/
(44 characters - not including the CaFUP)

Which is slightly more readable, and saves space on the URL.

If the LPSE code is multivalued enabled (i.e the panl.multivalue.separator.<lpse_code> exists and is not an empty string), then the generated LPSE URLs will automatically include the correct information.

When rendering the value that the end user will see, you may choose either the encoded (which will include the prefix), or the encoded_multi (which will preclude the prefix) JSON key's value.

For the above example the compression rate - i.e. the ratio of condensed to normal as a percentage is as follows:

In the above table, remember that the condensed figure has a longer suffix that is applied to the values, hence the slight expansion for the addition of 1 colour.

OR Facets

Condensed OR Facets

Like the condensed multivalued facets, an OR Facet can have its LPSE path condensed by using only one prefix and one suffix irrespective of how many values are used within the URI.

In this example, the mechanical-pencils-or.panl.properties file for the 'brand' facet configuration is below:

Example Properties - Original

The above configures the brand facet to have a prefix of 'Manufactured by ' without any suffix.  When adding facets to the query, the URL quickly grows with the prefix being added to each facet value:

The landing page:

http://localhost:8181/panl-results-viewer/mechanical-pencils-or/brandandname
(0 characters - not including the CaFUP)

Adding the brand facet value 'Staedtler':

http://localhost:8181/panl-results-viewer/mechanical-pencils-or/brandandname/Manufactured by Staedtler/b/ 
(28 characters  - not including the CaFUP)

Adding another brand facet value of 'Mitsubishi'

http://localhost:8181/panl-results-viewer/mechanical-pencils-or/brandandname/Manufactured by Staedtler/Manufactured by Mitsubishi/bb/
(28 characters  - not including the CaFUP)

Adding a third brand facet value of 'Hightide Penco'

http://localhost:8181/panl-results-viewer/mechanical-pencils-or/brandandname/Manufactured by Mitsubishi/Manufactured by Staedtler/Manufactured by Hightide Penco/bbb/
(88 characters  - not including the CaFUP)

With a condensed LPSE path configuration (from the mechanical-pencils-or-separator.panl.properties file):

Example Properties - Condensed

The above configures the brand facet to have a prefix of 'Manufactured by ' (note the trailing space) and a suffix of ' Co.' (note the leading space) and an OR separator of ', or '.  When adding facets to the query, the URL will now have:

• Only one prefix
• Only one suffix
• Only one LPSE code

The landing page:

http://localhost:8181/panl-results-viewer/mechanical-pencils-or-separator/brandandname
(0 characters  - not including the CaFUP)

Adding the brand facet value 'Staedtler':

http://localhost:8181/panl-results-viewer/mechanical-pencils-or-separator/brandandname/Manufactured by Staedtler Co./b/
(32 characters  - not including the CaFUP)

Adding another brand facet value of 'Mitsubishi'

http://localhost:8181/panl-results-viewer/mechanical-pencils-or-separator/brandandname/Manufactured by Staedtler, or Mitsubishi Co./b/
(47 characters  - not including the CaFUP)

Adding a third brand facet value of 'Hightide Penco'

http://localhost:8181/panl-results-viewer/mechanical-pencils-or-separator/brandandname/Manufactured by Mitsubishi, or Staedtler, or Hightide Penco Co./b/
(66 characters  - not including the CaFUP)

For the above example the compression rate - i.e. the ratio of condensed to normal as a percentage is as follows:

In the above table, remember that the condensed figure has an additional prefix that is applied to the values, hence the slight expansion for the addition of 1 brand.

Analysed Facets (think Word Clouds)

Whilst Word Clouds have probably gone out of fashion, there is still value in being able to facet on individual words within a document.  The example that is used here is a subset of the text from the Book "Moby Dick" by Herman Melville - downloaded from the Project Gutenberg Website (https://www.gutenberg.org/cache/epub/2701/pg2701.txt) and is located in the src/test/resources/sample/book/data directory of the project.  The subsert is the first 14 chapters of the book.

Implementation Details

Step 1. Create the Solr Schema managed schema

The managed schema file for Solr is very simple in this example (a copy of which can be found here: src/test/resources/sample/book/solr/managed-schema.xml) - a snippet of which is below.

There is only one defined field for the contents - which is set to be of type text_general which will be analysed, and used as a facet.

Step 2. Generate the Panl Properties file

From the schema that was created in step 1 above, the Panl generator was used to output the panl.properties file and the book.panl.properties file.  

*NIX

Windows

The above generated both the files that were required.

Step 3. Edit the book.panl.properties file

Only one field was defined from the schema

And will be left as is

Step 4: Upload the schema to Solr

*NIX

Windows

Step 5: Index the data

As we wanted to index each word, a simple indexer was created, uploading the book paragraph by paragraph.  As the text file does not have an id (which is required), a simple indexer was written with a simple unique id generated for each paragraph added.

Step 6. Start the Panl server

*NIX

Windows

And browse the documents

http://localhost:8181/panl-results-viewer/book/empty/

This contents facet will show:

• the (1397)
• and (798)
• a (711)
• of (672)
• to (555)
• in (457)
• i (392)
• ...

Which has far to many small filler words, so they will need to be removed.

Step 7. Update the Solr Stop words file

In the src/test/resources/sample/book/solr/stopwords.txt file, add the stop words that you wish to filter out - the src/test/resources/sample/book/solr/stopwords-example.txt file has the words that were chosen for this implementation and may require more.

Step 8. Re-create the Solr Collection and Re-index the Data

For this implementation, the Solr Admin console will be used as an easy and straightforward way to delete a collection.

1. Open the Solr Admin console (http://localhost:7574/solr/#/)
2. Click on the Collections link which will take you to this page -
   http://localhost:7574/solr/#/~collections 
3. Click on the book collection
4. Click Delete Collection
5. Enter the collection name (i.e. "book")
6. Click Delete

Now

1. Re-create the Solr collection
2. Re-run the indexer

Using Panl for Dynamic Navigation Menus

Rather than having a separate configuration for the navigation, Panl can be used to drive the navigation menus.

For example, generating links to brand pages, or brand search results, create

Use the empty FieldSet (as you won't be rendering any of the documents) and set the panl.lpse.order=z,b,s,p,n,o,q to only include the b LPSE code (which is the brand name).

In the above properties, the facetsort property is set to index so that the results are returned in alphabetical order

Colour Swatches and Sort Names

Rather than within the UI code, searching through the various Solr facet names and then implementing.  Panl allows adding additional information through configuration (which is added to the Panl response object under the  'extra' JSON key) this enables the code to be decoupled from the Solr collection and field names.

In the sample directory (i.e. the sample/panl/mechanical-pencils/ directory) the mechanical-pencils-extra.panl.properties file configures this CaFUP to include an extra JSON object to pass additional information through to the front end UI.  

Which will add this JSON object (see Lines 9 and 20) in both the available and active facets Panl objects for their respective Solr facets.

The in-build Panl Results Viewer Web App uses this information to change the rendering.  In Line 9 the short_value property is used to render the value of the facet, rather than the value_encoded.  In Line 20 the swatch property is used to render colour swatches in addition to the name.  The below URL and image shows the rendering:

http://localhost:8181/panl-results-viewer/mechanical-pencils-extra/brandandname/

Which renders:

Without the extra JSON object, the rendering of the URL remains as per the previous examples in the book.

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/

Number Of Results Per Page

In the in-built Panl Results Viewer web app, the number of results per page that can be selected is hard-coded to 3, 5, or 10.  Should you wish to drive the values that are displayed on the page, then you can set additional information through the panl.properties file.

For an example, the

 PANL_INSTALL_DIRECTORY\sample\panl\mechanical-pencils-extra\panl.properties 

file defines a property of

panl.server.extra={ "num_per_page": [ 5, 10, 20 ] }

Which will return this property value in every response for the designated CaFUPs under the JSON key panl.extra.

This value can be overridden on a per collection basis by setting a property key of panl.collection.extra which  will send through the extra JSON object for that CaFUP only.

The file

PANL_INSTALL_DIRECTORY\sample\panl\mechanical-pencils-extra\mechanical-pencils-extra.panl.properties 

defines a property of

panl.collection.extra={ "num_per_page": [ 10, 20 ] }

Which overrides the server extra above and only returns the two values in the array.

More Like This Functionality

The More Like This (MLT) functionality in Solr returns documents that are similar to a specific document in their result list.  It does this by using terms from the unique specific document to find similar other documents in the index.

Within the Solr server, there are three implementations available, namely:

1. The MoreLikeThis Request Handler, (This is the recommended implementation)
2. The MoreLikeThis Search Component, and
3. The MoreLikeThis Query Parser (using either internal, or external content)

There are three ways to implement the More Like This (MLT) functionality, the Panl Server only supports two of these methods.

The Panl server only supports the first and third methods (the third method only performs on internal content, not external content).  There is an additional Query Parser that allows external content which is also not supported by the Panl server.

For More Like This requests that are passed to a replica shard, the Solr server will not return any results.  Consequently, Panl adds a Solr parameter of shards=shard1 and will attempt to retry the connection if it does not work up to 5 times.  There is a failure rate of up to 10% so be warned.

The recommendation would be to pre-cache the responses for every id - although this may be an expensive operation.

The difference in Results Between the Request Handler and the Query Parser

Using the following book details:

Title: The Black Echo
Author: Michael Connelly
Series: Harry Bosch
Genres: Mystery, Crime, Detective, Thriller
Id: 1

The Query Parser response - using q={!mlt qf=title,author,series,genre mintf=0 mindf=0 minwl=0}1

Returns the following book titles.

• Michael Connelly - The Last Coyote
• Michael Connelly - City of Bones
• Michael Connelly - The Black Ice
• Michael Connelly - The Concrete Blond
• Michael Connelly - Trunk Music

Whilst the Request Handler response - with the same URL parameters encoded - using mlt.fl=title,author,series,genre&mlt.mintf=0&mlt.mindf=0&q=id:1&mlt.minwl=0 returns a wider range of titles.

• Michael Connelly - The Last Coyote
• Michael Connelly - City of Bones
• Andy Weir - Artemis
• Andy Weir - Project Hail Mary
• Mary Wollstonecraft Shelley - Frankenstein

For the request handler example above, the interestingTerms configuration was set to 'details' and boost was set to true which also returned an Interesting Terms JSON Object in the response which gives an insight into how each of the titles were chosen.

Interesting Terms (with the boost value in brackets)

• genre:Crime (1.7234258651733398)
• genre:Detective (1.245267391204834)
• genre:Mystery (1.150124430656433)
• genre:Thriller (1)
• author:Michael Connelly (1.36171293258667)
• title:the (1)
• title:black (1.7234258651733398)
• title:echo (1.7234258651733398)
• series:Harry Bosch (1)

Why were they chosen:

The More Like This  Request Handler

To configure the Request Handler in the Panl server, use the following properties

When you are comfortable with the results that are being returned, you may then

panl.mlt.interestingTerms=none

panl.mlt.match.include=false

Or just not set the properties at all.

The More Like This Query Parser

To get better results, use the panl.mlt.qf property to base the query on specific fields to get a wider range of results

~ ~ ~ * ~ ~ ~