A Walkthrough Example - The Book Store

At this point, you should

1. Have a functional Solr server with indexed data.
2. Have a brief understanding how to add data to the Solr server (the Additional Data section is a good place to start).
3. Have a Panl server instance available and have looked at the functionality provided.
4. Also have a good idea as to how Panl interacts with Solr and the options that are available.

Working through this chapter of the book will provide all of the thinking, design, and steps required to index the required dataset, configure the Panl server, and get it all up and running.

This section runs through the setup and configuration of Panl and Solr for a distinct dataset showing the path from defining and setting up a Solr collection, to configuring the Panl server, to rendering the pages that are required for the user journey.  This dataset will be based on a book store^[11] with the underlying data containing 3 million records - this dataset is not included with the release package (however a cut-down, cleaned version is included for testing).  Running through this example will detail the decisions that are required and made and how the data was then formulated.

The process for pulling all of the components together is going to be the same regardless of the dataset that you are using.

1. Understand the dataset - Knowing what data is available, what cleanup process may be required, what data is missing and whether data should be derived or included from a separate source.
2. Configure the Solr index - Knowing how the underlying data will be indexed and surfaced to the Panl server.  In general one Solr collection, if set up correctly, will be able to drive multiple Panl configurations.
3. Configure the Panl server - This applies to both the panl.properties file and <panl_collection_url>.panl.properties files.  Remember: you may have multiple CaFUPs, so there may be multiple configuration files.  Panl allows you to 'slice and dice' the underlying Solr search collections for specific use cases. 

Finally, using all of the above information, and the functionality that each decision will provide:

4. Determine any additional web pages to render - Apart from the default (or main) search page which is the first page to be configured.

This, most likely, will be an iterative process, with any of the steps requiring updates to the other steps and configuration.  Having a good knowledge of the high-level requirements will enable you to quickly build up the configuration files for both the Solr and Panl servers.

For example when looking at the dataset, there are over 500,000 authors and I wanted to be able to have a page for each letter of the alphabet that would list the authors by their surname.  The dataset doesn't directly support this, so derived fields will need to be added (i.e. a field that is computed/derived from the dataset) which can then be indexed by the Solr server.

0. High Level Requirements

From a very high level perspective

• For this particular implementation, the keyword search should act upon the

• Title,
• Description,
• Author, and
• Genre

• From the dataset, there will be hundreds of thousands of Authors, consequently, immediately displaying all of the Authors within a search facet would not be useful.
• Have a unique link for every author being able to see the if the author has any series, with links to books in the series
• Users should be able to facet on specific fields that make sense from a search perspective

Note: As the dataset is discovered and understood and the way in which users will interact with the data, this may lead to other use-cases and requirements.  (See the section on The Iterative Implementation Process for more details).

1. Understanding the Dataset

Whether you are starting with an existing Solr managed schema, or you are looking to index a new dataset, understanding what the dataset contains is the MOST important part of the process.  This will drive all other decisions - after all, without the correct data, you won't be able index it, search on it, facet upon it, or present it to the user.

The above may seem obvious, however, understanding the data, in conjunction with the desired pages that you want to render, will inform whether you can derive data from the underlying dataset, combine the dataset with other external datasets, or need to use a back-end datastore to generate pages with links.

In this example we will be looking at setting a separate dataset containing just over 3 million records based on fiction books.

The dataset that I have access to has duplicates, mis-spellings, missing information, and information that is just plain wrong.  Before the indexing process, the data will need to be cleaned and any additional derived data generated.  Ignoring the previous data problems, the dataset contains the following information, with some notes about we would want to search and facet the dataset

Data

• Author

• Search on the author, or authors (the book may have a collaboration of authors)
• Facet on the author, however with around 500,000 authors, we don't want to display all of the facets on the initial search page
• See a page that lists all of the author's works (in order of date of publication) with links to the series that are available
• See the author in the returned documents

• Title

• Search on the title
• Will not be faceted, but will be displayed in the returned documents

• Description

• Search on the description
• Will not be faceted, but will be displayed in the returned documents

• Book Image

• See the image of the book
• Will not be faceted, but will be displayed in the returned documents

• Buy URL

• See the link to the URL to buy the book
• Will not be faceted, but will be displayed in the returned documents

• Genre

• Search on the genre
• Be able to select multiple genres as a facet

• Number of Pages

• See the number of pages in a book in the returned document
• Will not be faceted, but will be displayed in the returned documents

• First published year

• See the first published year of the book
• Will be faceted, but as there are close to 100 years of books, we don't want to display every year on the initial search page

• Language

• Select the language of the book
• Be able to select only a single language

• Paperback/Hardcover

• Select whether users want a hardcover, or paperback book

• Series

• Search on the series name
• Will be faceted, however as there will be many series of books, this should not appear with the initial results, but only when an author has been selected

• Price

• Select a price range for books
• Be able to sort by price range

Derived Data

• ID - derived from the database primary key

• This is the required primary key for the Solr search server and the database primary key has been chosen for this purpose

• Author A-Z index - derived from the first character of the 'Author' surname

• Used to present a list of pages by the first letter of the author's surname

• Decade published - derived from 'Year published'

• Can be used to help narrow down the decade in which the book is published, i.e. the user should be able to select the decade first, and then be able to select the year within that decade
• Will be used as a facet

• Book type - derived from the 'Number of pages' - one of 'Flash Fiction', 'Short Story' 'Novelette', 'Novella', or 'Novel'

• Will be used as a facet
• Will not be returned with the search documents.

• Text - A general purpose field that will have its contents analysed and used for search queries

• Will be used for the search query
• Will not be returned in the search documents
• Solr will generate this, rather than data being input.

Now that the data types are known and how we are going to use them, let's determine how we are going to index them in the Solr search engine.

2. Configure the Solr Index

Whilst we are defining how the data will be indexed, we need to keep in mind the configuration of the Panl server as well.  Looking at the data that you have indexed, or would like to index, the configuration of Panl is determined on two items:

1. Whether the indexed Solr field should be a facet, or just a field, and whether multiple values are available for the field, and
2. The data type of the Solr field, which will determine the configuration options that are available through the Panl server

REMEMBER

• Facets will allow you to filter the results.
• Fields will be returned with the search documents - you cannot filter the results of the field.

Whether you choose a Solr field to be configured in Panl to be a Facet, or a Field, both

• Can be returned with the search documents
• Are able to be used for sorting options

The difference with Facets, is that in addition to the above, Facets

• Are able to be selected to filter the results and, depending on the data type, can be further configured for prefixes, suffixes, ranges and value replacements

Noting the following rules for Solr configuration:

1. If we want to be able to search, sort, or facet on the data, then it must be indexed.  
2. If we want to see the results in the returned documents, then it must be stored
3. If the dataset field can only hold a single value, then multivalued is No, otherwise it is set to Yes.

Using information above and the requirements for the dataset, the Solr field definitions for importing and indexing is as follows:

The above table would lead to the following snippet of the Solr managed schema file with the Solr field names and values where set to true highlighted.

Working through the schema:

Line 1:

This is the schema name which maps to the Solr collection name and will be used by Panl for the CaFUPs.  WARNING: if the schema name starts with the string panl- then the Panl server may fail to start.

Line 2:

The _version_ field is required by a Solr Cloud deployment - this is an internal field, generated automatically by Solr and is used by the partial update procedure, the update log process.  You may not need to have this field, however it is mandatory for a Solr Cloud instance

Line 4:

The id field for uniquely identifying a Solr document within the collection

Lines 6 - 17:

The fields that come directly from the book store dataset, note the values of the XML element for the attributes multivalued, indexed, and stored

Lines 19-21:

The fields that are derived from the data.

Line 23:

This is a field that is used as a storage area for every other field that needs to be searched on - see lines 27-31 below.

Line 25:

This element tells Solr what the unique key is for this collection

Lines 27-31:

This copies the values from the required fields so that they can analysed by Solr and searched upon with a keyword or keywords.

Line 33:

For clarity and space, the Solr field definitions and additional XML elements were not included and replaced by ellipses.

Line 35:

The end of the Solr schema definition

3. Configure the Panl Server

Now that we understand the dataset, and the Solr search server is going to index the data, we can extend the Solr field definitions for the initial Panl configuration.

The above can be considered the default search page configuration, there will be other <panl_collection_url>.panl.properties files defined

The Default panl.properties file

For all of the Book Store URL path parts, the panl.properties file will be the same, all comments have been stripped from the file and the only incomplete line is the last line panl.collection.book-store=... which will have the <panl.collection_url>.panl.properties files added once we have configured them.

The Default Search Page Configuration

To generate the default search page configuration the in-built Panl utility is the quickest way to get up and running fast:

For more information on the available options, see the section for the command line options for the Panl Generator in the appendices.

*NIX command

Windows command

This will generate two files in the src/dist/sample/panl/book-store/ directory named panl.properties and book-store.panl.properties the text of which is not included - however the complete file can be seen in the GitHub repository

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/book-store/book-store.panl.properties

General Properties Configuration

Skipping over the defaults values for the panl.param.* properties (and no prefixes or suffixes were added to either the panl.param.page or panl.param.numrows properties) the following properties were changed:

solr.numrows.default=20

This was changed from the default value of 10 as 20 results seems to be a good starting point for such a large collection

solr.highlight=false

No changes for the default value of 'true', no highlighting will be required on the Book Store collection

panl.lpse.ignore=i

We still want to be able to search on this field and be able to pull out the results, generally with a canonical URL - e.g.

 /Michael+Connelly+Harry+Bosh+Series+The+Black+Echo/1678/zi/

And use the id Solr field (LPSE code 'i') as the lookup key with the value 1678, the rest of the URL path part will be ignored.

panl.sort.fields=price,a_to_z_index,first_published_year

These are the fields that are going to be able to be sortable - remember that relevancy is always available as a sort order and is the default sort order if no other sort order is selected.  The order of the value of this property is the order that Panl will return in the JSON response object.

The Generated Fields and Facets Configurations

Comments providing information about the settings have been removed from the examples below.

Solr Field 'id'

This facet will be left as it is for the moment, but this will be ignored by the Panl server as the property panl.lpse.ignore=i has this LPSE code.  

You can still return this as a field in the Panl results, so that if you need the unique id of the book for additional functionality (e.g. adding to a cart, linking to a separate page, looking up further details).  See the panl.results.fields.* properties.

Solr Field 'author'

A prefix has been added on line 6 of 'Author ' (note the ending whitespace).

There are too many authors to have this as a facet, and they will be ordered by the number of books that have been published, so this facet is configured to only appear if a search query is set, or if the first letter of the surname is selected.  Consequently line 8 has been un-commented so that the panl.when.a property has a value.  This facet will only appear if the search query (LPSE code 'q') or the a_to_z_index  facet (LPSE code 'A') has been selected.

Solr field 'title'

The generator has configured this Solr field as a Panl facet as it is both indexed and stored in Solr - this has been changed to a field, rather than a facet.

Solr field 'description'

The generator has configured this Solr field as a Panl facet as it is both indexed and stored in Solr - this has been changed to a field, rather than a facet.

Solr Field 'book_image'

No configuration changes made

Solr Field 'buy_url'

No configuration changes made

Solr field 'genre'

This will be an OR facet as it is configured with the property panl.or.facet.g=true, meaning that end users can select one or more of the facet values.  Note: This is already a multivalued Solr field, however using this as an OR facet means that you can select books which are 'Sci-Fi' OR 'Horror', rather than a book that is 'Sci-Fi' AND 'Horror'

Solr Fields 'num_pages'

No configuration changes made

Solr Fields 'first_published_year'

Add in a prefix of 'First published in ' - Line 5 - and this will only appear when the decade_published facet has been selected (LPSE code 'D')  - Line 7.

Solr Fields 'language'

No configuration changes made

Solr field 'is_paperback'

The display name has been changed to be 'Book Format' (Line 3) and a Boolean value replacement for both the true and false values (Lines 7 and 8).

Another way that this could have been index by Solr was to derive the data and store the book format as a string - i.e. type="string" with the values 'Paperback' and 'Hardcover', however keeping this as a boolean value with value replacements means that additional CaFUPs could be configured with different values for true and false if additional URLs were needed to be generated.

Solr field 'series'

This facet will only be passed through if an author facet (LPSE code 'a') has been selected (Line 7).  

Solr field 'price'

This facet is a RANGE facet (configured with the panl.range.facet.P=true property) - Lines 8 to 15. As an example, the configuration will generate the URL path part.

/From+5+to+100+dollars/P/

Additionally, with the wildcard properties set, it will generate a Solr query when the minimum or maximum values are passed through to use less than or greater than, respectively.  I.e. if the URL path part was used, as they are both a minimum and maximum value, the query would prices between 5 or below and 100 and greater.

For a URL path part of

/From+20+to+100+dollars/P/

It would return books greater than 20 (even if they are greater than 100)

For the URL path part of

/From+45+to+50+dollars/P/

It will only return values between 45 and 50 (inclusive)

Solr field 'a_to_z_index'

The Panl name of the facet has been changed to 'Authors (A-Z)' - (Line 3).

Normally Solr will return facets in order of decreasing count, however by setting Line 8 to index, this will sort the facets by index order, i.e. by the alphabetical order, rather than the number of returned documents for the specific facet. (Line 8)

Solr field 'decade_published'

No configuration changes made

Solr field 'text'

This is an internal Solr field that is used as a multi valued text field to store all fields that need to be searched against. As such, it is not going to be used as a facet, or a field, so the entire entry has been commented out.

Solr field 'book_length'

No configuration changes made

Panl LPSE Orders, FieldSets, and Sorting

The final part of the Panl configuration is the LPSE order, the FieldSets, and the available fields/facets to sort the results documents.

The Panl LPSE Order

Going through the field and facet configuration items, two of the facets were changed to be fields, and one of the facets was removed (commented out), they were:

1. The Solr field title (LPSE code 't') is now a field, not a facet
2. The Solr field description (LPSE code 'd') is now a field, not a facet
3. The Solr field text (LPSE code 'T') was commented out

So the original LPSE order (strikethrough text below)

The final version becomes

The Panl FieldSets

There are always going to be at least two FieldSets defined for any Panl collection, namely:

1. default - this is ALWAYS available, and if not set then it will return ALL fields in the Solr collection, whether you have defined them in the Panl configuration file or not. The recommendation is to either ignore this in your implementation, or edit this FieldSet to your purposes, as has been done below.
2. empty - this is ALWAYS available, and if it appears in the properties file, a warning will be printed and it will be ignored.  This will return no fields for the document (i.e. no documents at all).

Here, the only configured FieldSets is going to be the default, with no other FieldSets defined. I.e. the panl.results.fields.firstfive property has been removed.

Lines 14 to 17 have been removed.  The id field (Line 1) was kept in as it may be useful to link to the database for other purposes.  The final property looks thusly:

The Panl Sort Fields

To define the sort fields, use the panl.sort.fields property with a list of comma separated values.  Each of the sort fields must match the Solr field name, NOT the Panl LPSE code as these are passed directly through to the Solr server.

panl.sort.fields=price,a_to_z_index,first_published_year

Testing the Configuration

At this point (assuming that the data has been correctly added and indexed to the Solr Search server) you will be able to start the Panl server and view your single CaFUP on the Panl Results Viewer - http://localhost:8181/panl-results-viewer/book-store/default/.

Configuration Change Summary

4. Determine the Web Pages to Render

In addition to the default (i.e. main) search page with its functionality, additional page requirements are as follows.

1. SEO friendly URLs that list of all books published by author (in order of publication) along with the ability to facet by any series that the author has written.
2. SEO friendly URLs that lists all book series for Authors (in order of publication) that exist
3. A list of all Authors with their associated books and their series

Author and Author Series

Both of these pages can be generated with a single Panl Configuration (included as the dist/sample/panl/book-store/author-alphabetical.panl.properties file)  For each of the links to the authors, a link was generated in the format of /Author+<author_name/a/ and then the Panl server was left to do its work.

Author Listing

The indexing of the author listing page - i.e. a complete list of all authors within the dataset could not sufficiently be satisfied by the current dataset, so a new dataset was created and indexed by Solr (not included in the release package).  I was then able to produce the pages that were required by passing it through the Panl server to utilise the searching, sorting, pagination, and hierarchical facets.

A single Panl solution may not fit all use cases so you may need to look at additional datasets, or simply by using pages generated from a database.

The Iterative Implementation Process^[12]

When testing the configurations, the original implementation didn't quite make sense, so

1. The way the dataset was index by the Solr collection didn't suit all of the needs, so a separate collection was created to hold only an individual author with a multifield list of titles attached to it
2. Searching a book by decade didn't really make sense (or even having the hierarchical facet for first_published_in).  They were removed as facets and made to be fields.  The data was left in the Solr collection index, as they may be of use later
3. The site that was generated was a joining of the web application server, the database, and then the Panl configuration.  Some pages were generated by the database and served up by the web application which were then linked to the Panl implementation.
4. Any Solr fields that are of type float, when returned with the documents there may be storage errors, for example, each of the books are priced as a float as 19.99, when returned with the document, it comes back as 19.989999771118164 - which rounds to 19.99.  Instead I derived another field to have it as an integer for price in cents, then on the front end, I just formatted it to the correct decimal place.

The changes made and implementation details have not been provided in the included sample dataset.

The Panl server runs purely on configuration, so any changes that are made to either of the configurations will be utilised at runtime.  Provided that the Solr collection is set up to allow the broadest array of functionality, this becomes a very short iterative process.

~ ~ ~ * ~ ~ ~