Welcome To Synapticloop Panl

Synapticloop Panl is a light-weight application server that is designed to sit between your web application and Solr search server instance(s) seamlessly converting human-readable, SEO friendly URLs into complex Solr search queries, and returning an enhanced JSON object for ease of integration and implementation.

It abstracts away the complexities of the Solr search parameters and building/translating of URLs so that you get the benefit of a human readable (and SEO friendlier) URLs without having to have a deep understanding of the mechanics behind it.

Some examples contained in this book also detail the conversions that Panl performs, and the Solr query that is executed, should you wish to delve deeper into the inner-workings of the Panl server's integration with Solr.

Apart from the default implementation that is expected of any search engine - i.e. keyword searching and faceting, Panl also provides some additional niceties for more human readable and SEO friendlier URLs.

Additional Panl Niceties

1. MULTIPLE ways to 'SLICE and DICE' - From one Solr collection, the Panl server can present the results and facets in multiple different ways, providing individual use cases for specific needs and search journeys.
2. PREFIXES and SUFFIXES - Panl can be configured to add prefixes and suffixes to the values within the URL path to increase readability, for example,
   
   The LPSE URL path of
   
           /Caran d'Ache/true/Black/bDW/ 
   
   could also have the brand Solr field prefixed with 'Manufactured By The ' and suffixed by ' Company' to produce the URL path
   
           /Manufactured By The Caran d'Ache Company/true/Black/bDW/ 
   
   Which will be parsed by the Panl server and converted to the correct value for the Solr server.
3. BOOLEAN value translations - For any Solr field that is defined as a solr.BoolField, then, in addition to prefixes and suffixes, the  'true' and 'false' values can be replaced with any arbitrary text, which will then be transparently converted between Panl and Solr.  
   
   For the LPSE URL path of
   
           /Caran d'Ache/true/Black/bDW/ 
   
   the 'true' value (which is defined as whether the mechanical pencil can be disassembled or not) could be replaced with 'Able to be disassembled' for true values, and 'Cannot be disassembled' for false values.  The above URL paths for 'true' and 'false' values would then become
   
           /Caran d'Ache/Able to be disassembled/Black/bDW/

and

        /Caran d'Ache/Cannot be disassembled/Black/bDW/

4. BOOLEAN checkboxes - This mechanism allows you to highlight one of the boolean states, whilst this may seem obvious to have a checkbox for a true/false value, the BOOLEAN checkboxes work in a subtly different way - only one of the boolean values is presented to the user.  By selecting the checkbox, one of either the 'true' or 'false' facet values will be selected. When deselected, the BOOLEAN facet is in a "don't care" state - i.e. the facet value can be either of the values (or even blank.
5. OR Facets - Where an item has only one facet value attached to it (for example a pencil will only be manufactured by one company), OR facets allow you to expand the selection to choose one or more of the single-valued facets.
6. CONDENSED multiple field values - Rather than having a forward slash URL path separator for multiple values of the same facet (used in OR Facets and Multivalued REGULAR facets), Panl can be configured to condense these values into a single path part, saving URL characters, and reducing URL length, and making the URL far more human readable.
   
   For example, selecting pencils manufactured by Faber-Castell OR Koh-i-Noor would have the URI path of
   
           /Manufactured by Koh-i-Noor Co./Manufactured by Faber-Castell Co./bb/
   
   with condensed multiple field values - this could be configured (with a value separator of ', or ') to become
   
           /Manufactured by Koh-i-Noor, or Faber-Castell Co./b/
   
   Saving 15 characters in the URL, the more multivalued fields values that are selected, the more URL space is saved (In the example, with 3 values selected, the saving becomes 30 characters).
7. SEARCH ALL OR SPECIFIC SOLR FIELDS - Any Solr field that is analysed can be selected to be searched on, for example, in the Bookstore Walkthrough, the user can select to search within the title, the author, the description, none (i.e. the default search) or all of them.  Additionally, you may choose to configure the Solr query time boost to put more weight on one (or more) of the fields.
8. MORE LIKE THIS - Return 'More Like This' results from the Solr server with your specific query, with the ability to configure the Solr query operands on the fly.
9. FIELD VALUE validation - By default, Solr will error (or give an erroneous result) when an invalid value is passed through - for example, if Solr is expecting a numeric value and it could not parse the passed in value, it will throw an exception.  Panl protects against this by attempting to parse the value as best it can, and silently dropping the parameter if it cannot be sensibly^[3] parsed. This is done for numeric types (integer, long, float, and double) and boolean values.
10. HIERARCHICAL facets - Only show facets if a separate facet is first selected, allowing you to lead users through the search journey, by only displaying facets that help the user narrow down their results.
    
    For example, you may be presenting a search page for Cars and you may only want to show the car models once the brand of cars has been selected first.
11. UNLESS facets^[4] - Continue to show a facet unless another specified facet is selected.  This can be thought of as the inverse of a hierarchical facet and is useful when a facet no longer becomes relevant as the user goes through the search journey.
12. SORTED facets - Each facet can be individually configured to order the facet results by either the facet count in descending order (which is the default), or the facet value (e.g. alphabetic/numeric based on the value of the facet - in either ascending or descending).  
13. MORE facets - Solr (and Panl) configures a limit for the maximum number of facet values that are returned with any query, this functionality enables you to dynamically load additional facet values if they are available but weren't returned with the results by default.
14. RESULTS SORTING options - Sort by any of the Solr fields that are configured in Panl, either ascending, or descending and with multiple sub-sorting available - e.g. sorting by a brand name, then the model number.  Additionally Panl generates URLs for the inverse of the sorting without impacting any sub-sorting.
15. INTEGRATED TYPEAHEAD/LOOKAHEAD - Retrieve results suggestions as you type in the query search box.  In the implementation included within this book the typeahead functionality is enabled after typing 3 or more characters and only returns valid results for the mechanical pencil collection dataset.
16. PAGINATION - Panl will return all of the variables required to easily generate pagination URL paths giving you options and control over your own implementation.  
    
    The returned information includes:

1. the number of pages of results,
2. the number of results per page,
3. the total number of results,
4. the current page number, and
5. whether the returned results are an exact number.

17. STATIC SITE GENERATION - With the exception of a query parameter, all available links for every conceivable URL path can be statically generated ahead of time, with canonical URLs.  Additionally, for search results which do not change frequently, the Panl JSON Response Object can be cached for faster lookups.
    
    Be warned that the number of possible pages that can be generated can quickly become incredibly large.
18. STATELESS - No state is stored in the Panl server. All state is held within the requested URL path part.  No sessions, no memory, nothing to backup or replicate across servers, easy to update, and quick to start and restart.
19. CACHE-ABLE - Unless the underlying Solr search document index changes,    each Solr and Panl responses can be cached.
20. 100% TEXT CONFIGURATION - All configuration for Panl is based on text files (Java .properties) files so they can be stored in a source code management system.  Additionally, upgrades to the Panl server are easy - just drop in the new Panl release package, use your existing configuration, and it will just work.  And with quick restart times, the configuration changes will be seen in an instant.

About Panl Server

The Panl server is an interface between your web app into the Solr search server converting human-readable, SEO friendly URL paths into complex Solr search queries. Rather than adding query parameters to the URL, Panl automatically generates and returns complete URL path links that can be rendered by your web application.

The Panl server uses last path segment encoding (LPSE) to parse and decode the full URL path, converting a URL path from

                                                |Last Path|
                                               | Segment |
/Manufactured by Koh-i-Noor Company/Clutch/Green/bmWsb-sN-/
|----------------LPSE PATH----------------------|  LPSE   |
                                               |  codes  |

To a search query that will return a list of mechanical pencils that

• Are manufactured (the brand) by Koh-i-Noor (b) - Note that the configured prefix of 'Manufactured by ' and suffix of ' Company' will be removed by the Panl server before being passed through to the Solr server.
• Have a Clutch mechanism (m), and
• Are Green (W) in colour

And, of the 8 results that are returned, the results will be sorted

• By brand name descending (sb-), then by
• Pencil Model (sN-)

Which is then passed through to the Solr search server as the following query:

q=*:*&q.op=OR&facet.limit=100&fl=brand,name&facet.mincount=1&rows=10&facet.field=le
ad_size_indicator&facet.field=colours&facet.field=brand&facet.field=mechanism_type&
facet.field=hardness_indicator&facet.field=lead_grade_indicator&facet.field=in_buil
t_sharpener&facet.field=disassemble&facet.field=category&facet.field=lead_length&fa
cet.field=in_built_eraser&facet.field=grip_shape&facet.field=weight&facet=true&fq=b
rand:"Koh-i-Noor"&fq=mechanism_type:"Clutch"&fq=colours:"Green"&sort=brand+desc,nam
e+desc&start=0

Panl presents a much cleaner URL than the above Solr search query.

When extracted, the Panl release package contains everything required for running the server including in-built web applications to view and test all functionality.

How Many Facets Does Panl Support?

The number of supported facets depends on the LPSE code length (which, if not set, defaults to 1). A LPSE code is a letter (either uppercase or lowercase) or number which maps to a parameter, operand, field, or filter.  There are five mandatory (and one optional) LPSE codes:

1. The query parameter,
2. The page number,
3. The number of results per page,
4. The query operand,
5. The sort order operand(s), and
6. (optionally) The pass through parameter

The above LPSE codes can be configured to your specifications and cannot be registered as facet LPSE codes.

With a LPSE length of 1:

• With the five mandatory codes, Panl will support up to 57 facets.
• With the five mandatory codes and the one optional code, Panl will support up to 56 facets.

With a LPSE length of 2:

• With the five mandatory codes, Panl will support up to 3,249 facets.
• With the five mandatory codes and the one optional code, Panl will support up to 3,136 facets.

The formula for working out what the maximum number of supported facets for the LPSE code is the number of available LPSE codes to the power of the LPSE length:

• With the five mandatory codes^[5]:
  (62 - 5)^lpse_length = 57^lpse_length
• With the five mandatory codes and the one optional code^[6]:
  (62 - 6)^lpse_length = 56^lpse_length

A LPSE length of 2 should provide more than enough facets for the majority of implementations.  Once the LPSE length gets above 2, the LPSE URL path becomes much longer, more quickly, subtly negating the value of the encoding of the URL to be compact and readable.  This growth of the URL length can be mitigated by configuring multi-value or OR separators.

Remember that you can define multiple Panl collections with CaFUPs for a Solr collection, and each of the CaFUPs can have different LPSE codes.  You may have over 56 fields for your documents in your indexed Solr collection, but you may wish to have a LPSE length of 1 and just use a subset of the fields for each of the CaFUPs.

Panl URL Structure

Collection Request Handlers

The URLs that the collection request handler responds to are determined by the Panl server configuration and returns an HTTP response of a JSON object that contains the results of the search query with all available facets and documents with the defined FieldSets.

The defined CaFUPs will respond to URLs in the form

/<panl_collection_url>/<fieldset>/<lpse_path>/<lpse_codes>/

Where:

• <panl_collection_url> Is the Panl collection (which is, in turn, configured to map to a specific Solr collection)
• <fieldset> Is the set of fields that are configured to be returned with the documents
• <lpse_path> Is the encoded values for the facts, queries, parameters, and operands, and
• <lpse_codes> Is the LPSe codes which determine how the LPSE path will be decoded by Panl

These URLs are configured through the <panl_collection_url>.panl.properties files and will be able to serve a virtually unlimited^[7] number of URLs.

Additionally, there are in-built Panl server URLs which provide additional functionality, the predefined URLs are as follows.

Panl Single Page Handler URLs

For each collection that is registered with Panl, the single page handler URLs return a JSON object that will allow building a single page search interface, they are bound to the following URLs:

/panl-single-page/<panl_collection>/

Where <panl_collection> is the Panl Collection that the single page search UI should be built from.

Panl More Facets Handler URLs

When there are more facets to be returned than the configured maximum (the  solr.facet.limit property in the panl.properties file) then, for each collection that is registered with Panl, the more facets handler URLs will return a JSON object that provides additional facet values for a specific facet and are bound to the following URLs in the form of:

/panl-more-facets/<panl_collection>/<fieldset>/<lpse_path>/<lpse_codes>/?code=<lpse_code>&limit=<limit>

Where:

• panl-more-facets is the start of the URL that the Panl server has bound the 'More Facets' handler to
• <panl_collection> is the Panl collection
• <fieldset> is the FieldSet for the fields that will be returned with the Solr documents Note: This FieldSet value must be passed through, however it is ignored and replaced by the More Facets handler with the 'empty' FieldSet as there is no need to return any documents
• <lpse_path> is the encoded values for the facets
• <lpse_codes> are the LPSE codes for the <lpse_path> above
• code=<lpse_code> is the query parameter LPSE code for the facet that the additional facet values are requested
• limit=<limit> is the query parameter for the maximum number of facet values to return.  Note: If this is set to -1, then all facets will be returned.

This will perform the search and only return the additional facet values for the Solr facet designated by the LPSE code <lpse_code> up to the limit designated by <limit> (or all if the limit is set to -1).

For example, if the solr.facet.limit is set to 100 and there are more than 100 available facets to be returned with the results, then the more facets handler will allow you to retrieve some or all of the additional values.

Panl Lookahead Handler URLs

For each collection that is registered with Panl, the lookahead handler URLs return a JSON object that provides a snippet of search results:

/panl-lookahead/<panl_collection>/<fieldset>/?search=<keywords>

Where:

• panl-lookahead is the start of the URL that the Panl server has bound the 'Lookahead' handler to
• <panl_collection> is the Panl collection
• <fieldset> is the FieldSet for the fields that will be returned with the Solr documents
• search is the URL parameter that the Panl server will respond to.  Note: this URL parameter name is configurable, however throughout the book it has been configured to be 'search'.
• <keywords> are the keywords that the user is searching for.

This will perform a search on the Solr index and return any results that it finds.  Note that it will not return any facets associated with the documents, only documents - and only the fields as defined by the <fieldset>.

Panl More Like This Handler URLs

For each collection that is registered with Panl and has enabled the More Like This handler in both the Solr and Panl servers, the handler URLs return a JSON object that provides a snippet of search results:

/panl-more-like-this/<panl_collection>/<fieldset>/<document_id>/

Where:

• panl-more-like-this is the start of the URL that the Panl server has bound the 'More Like This' handler to
• <panl_collection> is the Panl collection
• <fieldset> is the FieldSet for the fields that will be returned with the Solr documents
• <document_id> is the unique key of the Solr document to be used as the document to be based upon to return the More Like This.

This handler performs a More Like This query based on the specified document (using the unique Solr key).  It will not return any facets, only documents.

In-Built Web Apps (Viewer / Explainer / Single Search Page Examples)

For testing and debugging of the configured properties, the Panl Results Viewer, Panl Results Explainer, and Panl Single Search Page Example web apps are included in the Panl release package.  This surfaces all Panl functionality and allows integrators and implementers to understand and test the Panl configuration without having to integrate with a separate web application.

'Simple' Panl Results Viewer Web App

What started as a relatively simple page for testing and debugging turned into a page that has a fully functional faceted search interface, able to highlight all of the functionality of the Panl server and surface most of the Solr search server functionality.  It still remains an excellent way to test configuration options.

Below is a screenshot of the in-built Panl Results Viewer web demonstrating the features and functionality that you would expect from a search page implementation along with some additional features to make searching easier for you and your user.

When the Solr and Panl configuration is set up, the server is up and running, and the testing web app URLS are enabled, it is accessible at:

http://localhost:8181/panl-results-viewer/

With the image below showing the Panl results for the URL:

http://localhost:8181/panl-results-viewer/mechanical-pencils/brandandname/Manufactured%20by%20Koh-i-Noor%20Company/able%20to%20be%20disassembled/hexagonal/bDsN-sb+q/

Image: The In-Build Panl Results Viewer web app, highlighting all of the functionality.

1. A list of available Collections and FieldSet URL Paths (CaFUPs) that Panl is configured to serve.  CaFUPs enable different Solr fields and facets to be returned from the same Solr collection.
2. A textual representation of the CaFUP that the Panl Results Viewer web app currently is using.
3. The canonical URL path (which is returned with the Panl results JSON object) - An important part for search engines to de-duplicate URLs that return exactly the same information.  Multiple Panl LPSE URL paths WILL return exactly the same results.  You SHOULD use this link as either

• The rel="canonical" link element in the HTML, or
• The rel="canonical" link HTTP header

There is also an [explain] link that will take you to the in-built Panl Results Explainer web app for this particular canonical URL.

4. The search query box, by default, Panl responds to the same URL parameter name as The Solr server - i.e. 'q'.  This can be configured to be a different value through the Panl properties file.  In this book, it has been configured to respond to the 'search' query parameter.  Panl also supports passing through the Solr q.op parameter as a separate URL parameter for using the keywords as either OR, or AND.
   
   Specific Solr Search Field (not shown^[8]) Panl can be configured to search on specific Solr fields, rather than using the default search field.
   

Integrated Lookahead (not shown) With a simple HTTP call, Panl can return a lookahead for the search results.  In the example included in this book, the integrated lookahead is only activated after the user types in 3 characters.

5. Active filters, either queries or any of the selected facets that have been used to refine the search results - the  link is the URL path that will remove this query or facet
6. Active BOOLEAN filters, if the selected facet is a BOOLEAN facet (i.e. either true/false) then a link () can be included to invert this selection (i.e. change the value from true if currently false and vice versa).
7. BOOLEAN Checkboxes - any facets that have been defined as BOOLEAN   checkboxes, which allows the integrator to emphasise one of the values (either true or false).
8. Active Sorting, sorting options that are currently ordering the results - the  link is the URI path that will remove this sorting option from the results. If it is an active sorting filter, the Change to DESC or Change to ASC links will invert the sorting order without affecting any further sub-ordering.
9. RANGE filters, for facets that are defined as ranges - allowing users to select a range of values - the values are inclusive (i.e. include the minimum and maximum values).  RANGE filters also include dynamic maximum and minimum values so that the range that is rendered can be automatically updated.
   
   DATE Range filters (not shown^[9]), enabling searching over a range of dates (but not a specific date) in the form of:
   
           <next/previous> <any_integer> <hours/days/months/years>.
   
   For example:

Last 30 days
Previous 24 hours
Next 3 years

10. Available filters, additional facets with links  that can further refine and limit the Solr search results.  These facets can be sorted by the count descending (which is the default) and also by the index (or value) in either ascending or descending order. If the returned number of facets is not the complete set, then a link will be displayed to retrieve more facets.
11. Number of results found, and whether this is an exact match.
12. Query operand, whether the Solr search term query is OR, or AND - this affects the search query for Specific Solr Search Field functionality, not the faceting - i.e. the Solr server q.op parameter.  The eDisMax query analyser implementation (which is not included in this book) will also use this query operand to override the default query operand in the Solr configuration.
13. Page information, the number of pages, how many results are shown per page, and how many results are shown on this page.
14. Sorting options, whether to sort by relevance (the default) or by other configured sorting options with ascending and descending options available.  Any Solr field can be configured to be used as a sorting option, and multi-sort orders are available, allowing progressive sorting on more than one field.
15. Pagination options, the Panl server returns all information needed to build a pagination system - returning number of results, number of results shown per page, the current page number, and other information.
16. Number of results per page, Able to dynamically set the number of results to return for the query. Note: In the above image, the values 3, 5, and 10 are just examples that are hard-coded into the Panl Results Viewer and can be implemented with any positive integer number.
17. Timing information, About how long the Panl server took to build and return the results (including how much time the Solr server took to find and return the results).
18. The results, the fields that are returned with the documents and are shown in the results sections which are configured by the CaFUPs. Multiple field sets can be configured for the collection, allowing different groups of fields to be returned for different URL paths.  In the image, only two document fields are configured for this CaFUP, namely Brand, and Pencil Model.

'Simple'^[10] Panl Results Explainer

Again, this started as a relatively simple page for testing and debugging of the startup configuration options, rather than trawling through properties files and logs.

Below is a (cut-down) screenshot of the in-built Panl Results Explainer web app with explanations for canonical URL paths, the configuration of the Panl collection URL, and the individually configured properties for each of the fields and how this alters the Solr query. A useful page to see at-a-glance everything that a CaFUP is configured to do.

When the Solr and Panl configuration is set up, the server is up and running, and the testing web app URLS are enabled, it is accessible at:

http://localhost:8181/panl-results-explainer/ 

Image: The In-Build Panl Results Viewer web app

1. A list of available Collections and FieldSet URL Paths (CaFUPs) that Panl is configured to serve.  CaFUPs enable different Solr fields to be returned in the documents with the same search parameters.  Clicking on these links will populate the 'Configuration Parameters' and 'Field Configuration Explainer' sections.
2. A textual representation of the CaFUP that the Panl Results Explainer web app is currently using.
3. The canonical URL path entry field allows you to enter any canonical URL path and have the parsing and tokenising explained to you, including whether the parsed token was valid, the LPSE code found and the original value that Panl attempted to decode.  Note:  The CaFUP that the canonical URL path came from MUST match the CaFUP on the results viewer.
4. The request token explainer - for any canonical URL entered, this will list the parsing and decoding steps, with the following details

1. Whether the token is valid (if it is invalid, it will be ignored and not passed through to the Solr search server),
2. The type of Panl token that was found,
3. The LPSE code,
4. The parsed value,
5. The original value, and
6. Where pertinent, additional information pertaining to the specific code and its configuration.

5. Configuration parameters - parameters that are not fields or facets with information about the value, a description, and the property that set the value.
6. Field configuration explainer - for each of the fields or facets that are configured in the LPSE order an explanation of their configuration including:

1. The Java field type,
2. The LPSE code,
3. The Solr field name,
4. The Solr field type, the Panl field name, and
5. Additional configuration items which may include

1. Prefixes,
2. Suffixes,
3. Ranges,
4. Facet type, or
5. Minimum/maximum values

6. Any configuration warning messages that were found whilst parsing the properties files.

'Simple' Panl Single Search Page Example Web App

Panl also binds a URL path to enable the building of a single search page interface, and binds a URL path to view a working example of what the single search page could look like.

When the Solr and Panl configuration is set up, the server is up and running, and the testing web app URLS are enabled, it is accessible at:

http://localhost:8181/panl-single-page-search/

Image: The In-Build Panl Single Search Page Example interface web app for the mechanical pencils collection (Note: image split for sizing reasons)

1. A list of available example Single Search page interfaces that Panl is configured to serve.  CaFUPs enable different Solr fields to be returned in the documents with the same search parameters.  Clicking on these links will generate a working sample single search page.
2. The generated LPSE path that the selections from the search interface will apply.
3. All facets that can be selected, presented for the different types of facets, namely OR, RANGE, DATE Range, BOOLEAN, and REGULAR facets .
4. The generated LPSE path that the selections from the search interface will apply.
5. The search button that will take you to the in-built Panl Results Viewer web app so that you can view the results instantly.

About the Panl Generator

The Panl generator is a quick and interactive command line utility built into the Panl release package that, given a Solr managed schema file, generates the default panl.properties and <panl_collection_url>.panl.properties files.  This easily and quickly gets things up and running for your existing Solr schema from which you can iterate a solution from.

If you have an existing Solr schema and want to start testing the Panl server integration, then skip to the Integrating An Existing Solr Schema section.  If you are skipping ahead and diving straight into the Panl configuration generator, the rest of the sections of the book will give understanding on how to configure the Panl server to suit the requirements of your search page implementation.

~ ~ ~ * ~ ~ ~