Appendices

Panl URL Bindings

The following is a list of URLs that Panl binds to upon start up

User Defined URL Bindings

Panl has internal URL bindings which will always exist and then binds any user configured CAFUPs to the root of the server, in this book the defined URLs are:

• Panl in-built URL bindings (These bindings are ALWAYS available)

• http://localhost:8181/mechanical-pencils/default/ 
• http://localhost:8181/mechanical-pencils/empty/ 

• User defined through the CAFUP configuration

• http://localhost:8181/mechanical-pencils/firstfive/   
• http://localhost:8181/mechanical-pencils/brandandname/ 

Depending on the configuration of the Panl server, additional data sets may also have URL bindings:

• Book store

• http://localhost:8181/book-store/default/ 
• http://localhost:8181/book-store/empty/ 

• Mechanical Pencils (OR Facet)

• http://localhost:8181/mechanical-pencils-or/default/ 
• http://localhost:8181/mechanical-pencils-or/firstfive/ 
• http://localhost:8181/mechanical-pencils-or/brandandname/ 
• http://localhost:8181/mechanical-pencils-or/empty/ 

• Mechanical Pencils (More facets)

• http://localhost:8181/mechanical-pencils-more/default/ 
• http://localhost:8181/mechanical-pencils-more/firstfive/ 
• http://localhost:8181/mechanical-pencils-more/brandandname/ 
• http://localhost:8181/mechanical-pencils-more/empty/ 

• Simple Date

• http://localhost:8181/simple-date/default/
• http://localhost:8181/simple-date/firstfive/
• http://localhost:8181/simple-date/empty/

Panl Internal Functional URL Bindings

Additional functionality for a single search page implementation, along with retrieving more facets for a specific LPSE code are bound to URLs.

The Single Search Page JSON response is bound to the URL with the form

https://localhost:8181/panl-single-page/<panl_collection>

And the More Facets retrieval handler is bound to the URL with the form:

https://localhost:8181/panl-more-facets/<panl_collection>/<fieldset>/<lpse_path>/<lpse_codes>/?code=<lpse_code>&limit=<limit>

The following URLs (depending on the Panl configuration) will be bound for examples within this book:

• Single Search Page configuration information:

• http://localhost:8181/panl-single-page/mechanical-pencils/
• http://localhost:8181/panl-single-page/mechanical-pencils-or/
• http://localhost:8181/panl-single-page/mechanical-pencils-more/
• http://localhost:8181/panl-single-page/book-store/
• http://localhost:8181/panl-single-page/simple-date/

• More facets

• http://localhost:8181/panl-more-facets/book-store/default/ 
• http://localhost:8181/panl-more-facets/book-store/empty/ 
• http://localhost:8181/panl-more-facets/mechanical-pencils-or/default/ 
• http://localhost:8181/panl-more-facets/mechanical-pencils-or/firstfive/ 
• http://localhost:8181/panl-more-facets/mechanical-pencils-or/brandandname/ 
• http://localhost:8181/panl-more-facets/mechanical-pencils-or/empty/ 
• http://localhost:8181/panl-more-facets/mechanical-pencils-more/default/ 
• http://localhost:8181/panl-more-facets/mechanical-pencils-more/firstfive/ 
• http://localhost:8181/panl-more-facets/mechanical-pencils-more/brandandname/ 
• http://localhost:8181/panl-more-facets/mechanical-pencils-more/empty/ 
• http://localhost:8181/panl-more-facets/simple-date/default/ 
• http://localhost:8181/panl-more-facets/simple-date/firstfive/ 
• http://localhost:8181/panl-more-facets/simple-date/empty/ 

Internal Testing URL Bindings

The in-built example/testing  functionality is bound to the following URLs:

• http://localhost:8181/panl-results-viewer/
• http://localhost:8181/panl-results-explainer/
• http://localhost:8181/panl-single-page-search/

The above URLs can be disabled by by setting the panl.results.testing.urls=false property in the panl.properties file.

Definitions

#

42: The meaning of life

404: An HTTP status code returned when a resource could not be located by the server.

500: An HTTP status code returned by the server when an error occurred that stopped the processing of a request.

A

Apache: Refers to the Apache Software Foundation, the producers of an HTTP Server and a fine collection of software and libraries, also the author of the Solr search server.

B

Boolean facet: A Solr field which can only be one of two values, 'true' or 'false'.

Boolean value replacement: For a 'true' or 'false' value from a Solr field can have their 'true' or 'false' value replaced with more meaningful text, which will be converted by the Panl server.

C

CaFUPs: An acronym for Collections and FieldSet URL Paths.  This is the URL path that the Panl server will handle to return results from the Solr collection with the configured field names.

Collection: A collection is either a single logical search index that uses a single Solr configuration file (solrconfig.xml), or a collection of properties and field sets for the Panl server. (See also: Solr collection, and Panl collection).

D

DATE range facet: A Solr facet of type solr.DatePointField that can be used to filter results from the time period NOW to some arbitrary number of hours, days, months, or years, or from some arbitrary number of hours, days, months, years to NOW.

Document: A single result returned from the Solr search server.

E

F

FieldSet(s): A list of Solr field names that will be returned by the Panl server in the result documents.

Facet: Also referred to as a regular facet, a defined field in the Solr and Panl configuration that allows a user to filter the results by selecting a particular value for this field that the document must have.  The  returned facets have a Solr field name, a Panl field name, a value, and a count. (See also: Regular facets, OR facets, RANGE facets, DATE range facets, and Boolean facets)

Facet count: The number of documents which have a particular facet value assigned to them.

Facet index: Solr nomenclature for the value of a facet.

Facet value: The value of the Solr facet field as indexed by the Solr search server.

G

Hierarchical facet: A facet that will only appear if another facet or parameter has already been selected.  This is defined by the panl.when.<lpse_code> property.

H

Hierarchical facet: A facet that will only appear if another facet or parameter has already been selected.  This is defined by the panl.when.<lpse_code> property.

Highlighting: When configured, highlighting will return the matching text surrounded by markup (e.g. HTML) so that it may be rendered in a different fashion to bring attention to the text.

I

Infix: Text that is placed between the range values. Note: this is only available for RANGE facets.

J

JSON: JavaScript Object Notation, an open standard file and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute-value pairs.

K

L

LPSE code: The Last Path Segment Encoded string that the Panl server uses to parse the preceding URL path.

M

Mechanical Pencil: (or a clutch pencil) is a pencil with a replaceable and mechanically extendable graphite core (or "lead"). The lead is not attached to the outer casing, and can be  mechanically extended as its point is worn away from use.

N

Number per  page parameter: Defines the number of results to return per page from the Solr server.  This may be set to any positive integer value greater than 0 (zero).

O

OR facet: A configured facet that will allow returning multiple values from a facet that normally would allow the user to select only one.

P

Page parameter: A parameter which defines the page number of results that the user will be shown.

Panl collection: A collection of properties and field sets that control how the facets are configured and how the search results are returned.

Panl collection URL: Unlike a Solr collection of documents, a Panl collection is a collection of field sets and configuration.  There can be multiple field sets per configuration file, and multiple Collection URLs can be connected to a Solr search collection.  The Panl collection URL is made up of the selected Panl collection and selected FieldSet.

Panl field: A one-to-one mapping between a Solr field to a Panl field which is referenced throughout the configuration.

Panl generator: A utility function within the Panl server that will generate a starting panl.properties file and a  panl_collection_url.panl.properties file from a managed-schema.xml Solr collection configuration file.

Panl server: The HTTP server that acts as the proxy between LPSE encoded URL parts and the underlying Solr Server

Panl release package: A binary release as a .zip or .tgz file that can be downloaded from the Github releases page for Synapticloop Panl project.

Parameters: Parameters are specialised forms of LPSE codes that do not directly filter the results, but alter the way in which the results are returned.

Prefix: Text that is prepended to a facet value for the URL path, and optionally for the display of the value.

Properties: A text file format where each line defines a parameter which is stored as a pair of strings, one storing the name of the parameter (called the key), and the other storing the value

Q

q: The  query operator URL parameter that the Solr server responds to when searching text within the collection.

q.op: The query operand URL parameter that the Solr server responds to to determine whether the search term should be found in one of the fields (OR), or all of the fields (AND).

Query term: Text that is either a word or a phrase that is used to search across the Solr collection index.  Unlike facets, this is not limited to the values of the Solr facets.

Query parameter: The parameter that is passed through as a URL parameter from a text input field by an HTML form.

R

RANGE facet: A specialised form of facet which allows a search to be performed on a range of values.  If the Solr document value matches between the minimum and maximum range values, then it will be included with the results.

Replacement value: A value that replaces another value, used in BOOLEAN facets to replace a "true" or "false" value with another more human-readable value, and in RANGE facets for minimum and maximum values.

Regular facet: The default type of facet that allows filtering on a particular value.

S

SEO: An acronym for Search Engine Optimisation which is a set of practices designed to improve the appearance, positioning, and usefulness of content in the search results for search engines.

Solr collection: A collection of documents that Solr has indexed and is searched upon to return results. (See also: Panl collection)

Solr field: The name of a field that is defined within the managed-schema.xml file, which may be indexed, stored, and.or analysed.

Solr server: The Solr server

Solr schema: The XML file that defines the fields, their types, what

Sort order:  The field, or fields of a document which will be either ascending or descending ordered.

Suffix: Text that is appended to a facet value for the URL path, and optionally for the display of the value.  This also applies to a RANGE facet when appending the text to the end of a range query.

Synapticloop: The people behind the Panl server and generator.

T

.tar: A bundle of files placed together into the Tape ARchive format

Token: Panl parses and decodes the incoming request URL path and turns each of the path parts into a token. If there was an error in either the parsing or decoding, then this token will be marked as invalid and will not be passed through to the Solr server.

U

URI: Uniform Resource Identifier which is a unique sequence of characters that identifies an abstract or physical resource.

URL: Uniform Resource Locator is a subset of URI which specifically references a webpage.

URL path part: A part of the path of a URL, in the examples in the book the path part is taken as everything after the hostname.  A URL is of the form:

scheme ":" ["//" authority] path ["?" query] ["#" fragment]

V

W

Wildcard: Designates that a value that is used will match all values - used by the Panl server within a range query to indicate that it should be any number either below or above the selected minimum/maximum value.

X

x: x marks the spot.

XML: eXtensible Markup Language which is the format the Solr uses for the managed schema and configuration files.

Y

y: Why not?

Z

Zip: A file compression format allowing multiple files and directories to be easily packaged into a single file.

Command Line Options

The Panl release package has two modes

1. Panl server - serving up the search results, and
2. Panl generator - to generate configuration files for an existing collection.

For both of the modes, the usage describes the invocation through the Java JRE, rather than the supplied executable files (i.e. bin/panl and bin\panl.bat), however the command line options are the same.

Panl Server

The Panl server may be started by

bin/panl server

Which will start the server with default values:

• Looking for a panl.properties file in the directory that the command was executed
• Binding to the default port of 8181.

To set the panl.properties file that will be referenced, use the -properties command line option.

To set the port that you want to bind the properties to, use the -port command line option.

Panl Generator

The Panl generator can be invoked by

bin/panl generate -schema path/to/solr/managed-schema.xml

You MUST, at a minimum, pass through the path to the Solr managed schema file with the -schema command line option.

The default output directory and file name is the directory where this command was invoked and the filename will be panl.properties.  Should you wish to change this, use the -properties command line option with the filename.  If the option points to a directory, the default filename of panl.properties will be used.  Note: This will also be the directory that the <panl_collection_url>.panl.properties will be written to as well.

Panl generator will fail if the files already exist, however you may pass through the command line switch of -overwrite with a value of true to overwrite the files (the default is false).

Usage Text

You can see the complete usage (and possibly updated) instructions on the GitHub repository:

https://github.com/synapticloop/panl/blob/main/src/main/resources/usage.txt

Sample .properties Files

The Sample panl.properties Files

There are multiple sample properties files that are included in the downloaded packaged and can be viewed online:

Mechanical pencils

The default collection for running the examples through the book.

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/mechanical-pencils/panl.properties 

All

This file will start the Panl server with all included examples - however for it to display results correctly, the all datasets must be indexed.

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/all/panl.properties

Book store

Just the book store demo dataset with hierarchical and facet ordering.

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/book-store/panl.properties

Simple date

The simple date collection with DATE range facets.

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/simple-date/panl.properties

The contents of these files are unlikely to change between publishing this book and updates to the codebase.  To see the most up-to-date comments and instructions for usage, the template file that the Panl generator utility uses as a base can be found here:

https://github.com/synapticloop/panl/blob/main/src/main/resources/panl.properties.template

The Sample <panl_collection_url>.panl.properties Files

The sample file for the mechanical pencils example used in this book can be viewed online:

Mechanical pencils

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/mechanical-pencils/mechanical-pencils.panl.properties 

Additional properties files are also included in the download package

Book store (with Hierarchical facets and facet ordering)

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/book-store/book-store.panl.properties 

Mechanical pencils MORE faceting

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/mechanical-pencils/mechanical-pencils-more.panl.properties 

Mechanical pencils OR faceting

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/mechanical-pencils-or/mechanical-pencils-or.panl.properties 

Simple date (with DATE range facets)

https://github.com/synapticloop/panl/blob/main/src/dist/sample/panl/simple-date/simple-date.panl.properties 

The contents of this file is unlikely to change between publishing this book and updates to the codebase.  To see the most up-to-date comments and instructions for usage, the template file that the Panl generator utility uses as a base can be found here:

https://github.com/synapticloop/panl/blob/main/src/main/resources/panl_collection_url.panl.properties.template

Solr Version 8 & 7 Integration Notes

For integrations of the Solr server with the Panl Server, there are differences between the commands used to index the data by the Solr server, the response payload from the Solr server, and the connection strings to the Solr server which is configured in the panl.properties file.

Apart from the above, the Panl response payload will be identical irrespective of the Solr server version used.

SolrJ

The SolrJ connectors are changed, with the available list of SolrJ clients for version 8:

• HttpSolrClient
• Http2SolrClient
• LBHttpSolrClient
• LBHttp2SolrClient
• CloudSolrClient
• CloudHttp2SolrClient

And for version 7

• HttpSolrClient
• LBHttpSolrClient
• CloudSolrClient

Solr Configuration files

The managed-schema.xml file has now been renamed to simply managed-schema.

The Solr configuration file solrconfig.xml files change between versions.

JSON Response Object

The JSON response object has also changed

In Solr version 7 and 8:

Note:  The response key is a JSON array of results (line 3).  In version 7 and 8 the response array contains the documents, without any further keys.  Contrast this with the Solr 9 response which holds the documents in the response.docs JSON array and has additional keys containing information about the number of total documents:

Note:  The response key is a JSON Object of results (line 3 to 9 in bold above) with additional details.

Setting up a Solr 7 or 8 server

The main difference between Solr 7, 8 and Solr 9 is that the configuration MUST be uploaded to the zookeeper instance first, before the collection is created.

Additionally, there is no bin\post command for Windows machines so this is done through a Java command.

Windows Commands

1. Create an example cloud instance

This requires no interaction, will use the default setup, two replicas, and two shards under the 'example' cloud node.

2. Create the configuration for the mechanical pencils

This will create and set up the mechanical pencil schema so that a collection can be created and the data can be indexed.

3. Create the mechanical pencils collection

This will create and set up the mechanical pencil collection and schema so that the data can be indexed.

4. Index the mechanical pencils data

This will index the included sample mechanical pencil data into the Solr instance in the mechanical-pencils collection.

5. Start the Panl Server

This will start the server and be ready to accept requests.

6. Start searching and faceting

Open the link http://localhost:8181/panl-results-viewer/ in your favourite browser, choose a collection/FieldSet and search, facet, sort, paginate and view the results

For the simple-date dataset, the commands are almost identical, and, assuming that the Solr cloud is set up:

*NIX Commands

The *NIX commands are as per the windows section above with the file path delimiter changed from '\' to '/'

1. Create an example cloud instance

This requires no interaction, will use the default setup, two replicas, and two shards under the 'example' cloud node.

2. Create the configuration for the mechanical pencils

This will create and set up the mechanical pencil schema so that a collection can be created and the data can be indexed.

3. Create the mechanical pencils collection

This will create and set up the mechanical pencil collection and schema so that the data can be indexed.

4. Index the mechanical pencils data

This will index the included sample mechanical pencil data into the Solr instance in the mechanical-pencils collection.

5. Start the Panl Server

This will start the server and be ready to accept requests.

6. Start searching and faceting

Open the link http://localhost:8181/panl-results-viewer/ in your favourite browser, choose a collection/fieldset and search, facet, sort, paginate and view the results

For the simple-date dataset, the commands are almost identical, and, assuming that the Solr cloud is set up:

Additional Solr Version 7 Integration Notes

In the Panl response object, the num_results_exact (line 3 in bold below) will ALWAYS be true, as this version of Solr does not have this data available.

Solr Versions 6 and below

Unfortunately version 6 and earlier have no release packages to be built, the only way to have these built is to edit and compile the code from source.

https://github.com/synapticloop/panl/

Starting with the branch solr-panl-7 is probably the wisest - these are the integration points that will need to be looked at.

1. Create a new branch with your version of Panl - e.g. Version 6 would be branched as  solr-panl-6
2. Edit the /build.gradle file

1. change the distributionBaseName - e.g. solr-panl-6

2. Edit the Solrj dependencies, the dependency for SolrJ can be found on any maven repository:
   
   https://mvnrepository.com/artifact/org.apache.solr/solr-solrj 
   

3. Edit the Solr schema configuration and managed-schema

1. Copy over the version schema from the Solr version that you are using. (this can be found in the SOLR_INSTALL_DIRECTORY/example/files/conf directory  - or equivalent for your Solr version).
2. Edit the file to include the relevant fields that you want to index and search on

4. Look at the com.synapticloop.panl.server.client package, adding in the Solr Clients that are applicable to your version
5. Update the com.synapticloop.panl.server.client.PanlClient class to reference the correct clients.
6. Update the com.synapticloop.panl.server.handler.helper.CollectionHelper#getPanlClient() factory method to return the correct client

~ ~ ~ * ~ ~ ~