Getting Started With Panl

A rather pleasing companion to the Apache® Solr® Faceted Search Engine

_____________

Author: Synapticloop - Version 2.2.0 (Release 1)


This book will rapidly get you up and running with a fully featured, SEO-friendly, keyword searchable, faceted search engine with an in-built, example search page to test it all out.



(Note: The latest release of this book can always be found here: https://github.com/synapticloop/panl/tree/main/src/book, and the online version here: https://synapticloop.github.io/panl/) 

WHY?

Because...


/Caran d'Ache/true/Black/bDW/


looks so much nicer than...


q=*:*&facet.mincount=1&rows=10&facet.field=lead_size_indicator&facet.fie
ld=grip_material&facet.field=
colours&facet.field=nib_shape&facet.field=d
iameter&facet.field=cap_shape&facet.field=brand&facet.field=mechanism_ty
pe&facet.field=length&facet.field=hardness_indicator&facet.field=grip_ty
pe&facet.field=cap_material&facet.field=lead_grade_indicator&facet.field
=tubing_material&facet.field=in_built_sharpener&facet.field=
disassemble&
facet.field=category&facet.field=body_shape&facet.field=clip_material&fa
cet.field=mechanism_material&facet.field=lead_length&facet.field=body_ma
terial&facet.field=in_built_eraser&facet.field=grip_shape&facet.field=re
lative_weight&facet.field=
name&facet.field=nib_material&facet.field=weig
ht&facet.field=
variants&facet=true&fq=brand:"Caran+d'Ache"&fq=disassembl
e:"true"
&fq=colours:"Black"&q.op=AND



TL;DR

If you are looking to get the quickest start on understanding how a Panl server is configured and up and running, then...

  1. GETTING UP TO SPEED

    Start with the Quick Start - The 5 Steps section to get the Solr and Panl servers up and running and browse the Panl Results Viewer.

    http://localhost:8181/panl-results-viewer/mechanical-pencils/default/ 

    Can't wait, or just need to see what it is all about? See the
    Quick Preview - Running Docker section.
  2. UNDERSTAND

    The Solr Fundamentals and Panl Fundamentals section introduces you to the concepts that lay the foundations for the rest of the book.  Once you have a good understanding of the configuration and functionality, then there are two example walkthroughs:
  1. A FULL EXAMPLE
    Work through the
    The Bookstore Walkthrough section to gain a deeper understanding of the complete process, from schema design, indexing, and configuration of both Panl and Solr.
  2. PROGRAMMATIC INDEXING
    Work through the
    Filesystem Indexing And Searching Walkthrough chapter to understand how to programmatically index files and commit them to Solr.
  1. INTEGRATE

    Use your existing Solr installation and dataset to set up Panl with
    Integrating An Existing Solr Schema.
  2. REFERENCE

    Need help understanding the options? See the Panl Configuration part.
  3. UPGRADE

    See the Upgrading Your Panl Server Version section.


QUICK TIP

Tip: Documentation for this book can also be read online, see:

https://synapticloop.github.io/panl/

or, to download the copy of this book for the Panl version that you are using, see:

https://github.com/synapticloop/panl/releases/

Corrections and updates to this book, when done, will always be reflected online and the main GitHub branch - see:

https://github.com/synapticloop/panl/blob/main/src/dist/book/Getting%20Started%20With%20Synapticloop%20Panl.pdf

~ ~ ~ * ~ ~ ~


Copyright Notices

MECHANICAL PENCIL DATA: The data used in this book is based on 2mm mechanical pencil data hand curated by the author.  The data and associated Solr search results utilised in this book references manufacturers of writing instruments, specifically 2mm mechanical pencils and is used for informational purposes.  The manufacturer's name, the name of the pencil (and/or model number) may be a trademark and/or is copyright of the manufacturer.

IMAGES, APPLICATIONS, AND SITE REFERENCES: Screenshots of applications and external websites are included in this book, they remain the copyright of the original owner and do not infer any association with, or endorsement of, Synapticloop, and/or vice versa.

Additional datasets may have information that is copyright or trademarked to the respective owners.


IMPORTANT: Whilst the project is released under the very liberal MIT licence, the sample mechanical data remains under the copyright of Synapticloop, and may not be used for commercial purposes.

A NOTE ON URL ENCODINGS: Within this book if you are using the Firefox browser then the address bar will automatically decode any URL encoded space characters ('%20') - other browsers will keep the encoding in the address bar.  Within this book, the majority of example URLs will not show the %20 encoding, however the associated href in the link will include these encodings.

~ ~ ~ * ~ ~ ~


TABLE OF CONTENTS

———————

PART 1:

ABOUT THIS BOOK

.        16

README.1ST        17

Book Versioning        17

Solr Schema Versions        17

About This Book        18

———————

PART 2:

INTRODUCTION

.        26

Welcome To Faceted Search        27

A Simple Faceted Search Example        29

A More Complex Faceted Search Example        30

SEO Friendlier URLs        32

About Apache Solr        33

Welcome To Synapticloop Panl        34

Additional Panl Niceties        34

About Panl Server        38

How Many Facets Does Panl Support?        40

Panl URL Structure        41

In-Built Web Apps (Viewer / Explainer / Single Search Page Examples)        44

About the Panl Generator        54

Why Synapticloop Panl?        56

What is a LPSE (pronounced 'lapse') code?        57

URL Path Nomenclature        59

Is Synapticloop Panl For Me?        59

———————

PART 3:

GETTING UP AND RUNNING

.        61

What You Will Need        62

Download The Resources        62

The Panl Directory Structure        63

Panl Server Versions        65

Version Numbering        65

Solr-Panl-10-x.x.x        65

Solr-Panl-9-x.x.x        66

Solr-Panl-8-x.x.x & Solr-Panl-7-x.x.x        66

Solr-Panl-6-x.x.x and earlier version        66

Quick Preview - Running Docker        68

Docker Hub        68

Building The Docker Image Locally        69

Quick Start - The 5 Steps        71

Operating System Dependent Notes        72

Windows Commands        73

*NIX Commands        75

Next Steps        77

Getting Started        78

Downloading the Resources        78

A Note On Running The Commands        79

Solr Versions Command Line Options        82

Creating and Starting a Solr Cloud Instance        83

Creating a Collection        86

Indexing the Data        87

Starting The Panl Server        88

When Something Goes Wrong        89

When Things go Horribly-Wrong        93

Next Steps        98

———————

PART 4:

FUNDAMENTALS

.        99

Solr Fundamentals        100

Solr Configuration File Changes        100

Fields and Field Types        102

Overview of Indexed / Stored / Analysed Fields        105

Highlighting        106

Panl Fundamentals        107

Configuration Files        107

Search Field Configuration        112

Specific Solr Field Keyword Search Configuration        121

Panl Field Configuration        124

———————

PART 5:

GOING DEEPER - PRACTICAL EXAMPLES
.        129

The Bookstore Walkthrough        131

0. High Level Requirements        133

1. Understanding the Dataset        133

2. Configure the Solr Index        136

3. Configure the Panl Server        143

4. Determine the Web Pages to Render        177

The Iterative Implementation Process        178

Filesystem Indexing And Searching Walkthrough        180

Indexing A Dataset        180

Project Requirements        181

Project Steps        181

Extending The Project        199

Additional Data        200

All Data Panl Server        200

Simple Date        201

Mechanical Pencils Extra        202

Mechanical Pencils OR Facet        203

Mechanical Pencils More Facets        204

Bookstore        205

Working With Any Dataset        206

Supported Solr Data Types        208

Unsupported/Partially Supported Solr Field Types        209

A Note on Prefixes, Suffixes, and Infixes        210

Facet and Field Types        212

Understanding Solr Configuration And Panl Integration        231

Querying Data        231

The Solr Managed Schema        240

The Solr Configuration File        259

———————

PART 6:

PANL CONFIGURATION IN DETAIL

.        264

Panl Configuration        265

The panl.properties file        265

The <panl_collection_url>.panl.properties file        266

The panl.properties Configuration File        268

Configuring The SolrJ Connector        269

Setting The Solr Server URL(s)        270

Enabling The Panl Results Testing URLs        271

Setting HTTP Status Message Verbosity        272

Setting The Decimal Point Separator        273

Binding Solr Collections to Panl URL Paths        273

The <panl_collection_url>.panl.properties Configuration File        277

Parameter and Operand Definitions        277

Search Queries and Search Operands        278

Field Definitions        290

Facet Definitions        292

REGULAR Facet        296

BOOLEAN Facet        298

OR Facet        303

RANGE Facet        313

DATE Range Facet        320

Some Notes        325

———————

PART 7:

INTEGRATION AND PRODUCTION

.        332

Putting It All Together (Technically)        333

A Brief Architecture        333

Production Readiness        334

Logging        334

Upgrading Your Panl Server Version        337

Generating SEO Friendly URLs        337

Standalone Solr Installations        339

Panl LPSE URL Paths Explained        344

Parameter Query        345

Parameter Query Operand        346

Parameter Number of Rows        347

Parameter Page Number        348

Parameter Pass Through        349

Parameter Sort Order        351

Facets        352

Explaining A More Complex Example        353

Search Integration And The Panl Response Object        356

Generating Panl Links        361

"response.docs" JSON Object Results Integration        364

"panl.search" Keyword Search Integration        368

"panl.search" Specific Solr Search Field Integration        371

"panl.active" JSON Object        375

"panl.active.facet" REGULAR facet object        378

"panl.active.facet" REGULAR (Multi-Valued) facet object        380

"panl.active.facet" BOOLEAN facet object        381

"panl.active.facet" BOOLEAN Checkbox facet object        383

"panl.active.facet" DATE Range facet object        386

"panl.active.facet" OR facet object        388

"panl.active.facet" RANGE facet object        389

"panl.active.numrows"        392

"panl.active.page"        393

"panl.active.search" JSON Object        394

"panl.active.query_operand" JSON Object        396

"panl.active.sort" JSON Array        397

"panl.available" JSON Object        400

"panl.available.date_range_facets" JSON Array        401

"panl.available.facets" JSON Array        402

"panl.available.range_facets" JSON Object        414

"panl.pagination" JSON Object        422

"panl.sorting" JSON Object        423

"response.highlighting" JSON Object        428

Single Search Page Integration And The Panl Response Object        431

URL Bindings        431

Error Responses        432

Success Responses        432

The "lpse_lookup" JSON Object        433

The "lpse_order" JSON Array        434

More Facets Integration And The Panl Response Object        435

URL bindings        435

Error Responses        438

Success Responses        438

The "facet" JSON Object        439

Lookahead Integration And The Panl Response Object        441

URL bindings        441

Error Responses        443

Success Responses        444

Integrating An Existing Solr Schema        445

Using the Panl Generator        445

Manually Creating the Configuration Files        449

———————

PART 8:

REFERENCE MATERIAL

.        453

Panl Cookbook        454

SEO Friendlier Canonical URLs        454

BOOLEAN Facets        459

RANGE Facets        461

REGULAR Facets        463

OR Facets        467

Analysed Facets (think Word Clouds)        469

Using Panl for Dynamic Navigation Menus        475

More Like This Functionality        481

Properties Quick Reference        486

Property References Format Explained        486

panl.bool.<lpse_code>.false        487

panl.bool.<lpse_code>.true        488

panl.bool.checkbox.<lpse_code>        489

panl.collection.<solr_collection_name>        490

panl.collection.extra        492

panl.date.<lpse_code>.days        492

panl.date.<lpse_code>.hours        493

panl.date.<lpse_code>.months        494

panl.date.<lpse_code>.next        496

panl.date.<lpse_code>.previous        497

panl.date.<lpse_code>.years        498

panl.decimal.point        499

panl.extra.<lpse_code>        500

panl.facet.<lpse_code>        500

panl.facetsort.<lpse_code>        502

panl.field.<lpse_code>        503

panl.form.query.operand.respondto        504

panl.form.query.respondto        504

panl.include.same.number.facets        505

panl.include.single.facets        505

panl.lpse.facetorder        506

panl.lpse.ignore        506

panl.lpse.length        507

panl.lpse.order        508

panl.mlt.boost        508

panl.mlt.enable        509

panl.mlt.fl        509

panl.mlt.handler        510

panl.mlt.interestingTerms        511

panl.mlt.match.include        511

panl.mlt.match.offset        512

panl.mlt.maxdf        512

panl.mlt.maxdfpct        513

panl.mlt.maxntp        514

panl.mlt.maxqt        514

panl.mlt.maxwl        515

panl.mlt.mindf        515

panl.mlt.mintf        516

panl.mlt.minwl        516

panl.mlt.numretries        517

panl.mlt.qf        518

panl.mlt.type        518

panl.multivalue.<lpse_code>        519

panl.multivalue.separator. <lpse_code>        519

panl.name.<lpse_code>        520

panl.or.always.<lpse_code>        521

panl.or.facet.<lpse_code>        522

panl.or.separator.<lpse_code>        523

panl.param.numrows        523

panl.param.numrows.prefix        524

panl.param.numrows.suffix        525

panl.param.page        525

panl.param.page.prefix        526

panl.param.page.suffix        527

panl.param.passthrough        527

panl.param.passthrough.canonical        528

panl.param.query        528

panl.param.query.operand        529

panl.param.sort        530

panl.prefix.<lpse_code>        530

panl.range.facet.<lpse_code>        531

panl.range.infix.<lpse_code>        532

panl.range.max.<lpse_code>        533

panl.range.max.value.<lpse_code>        533

panl.range.max.wildcard.<lpse_code>        534

panl.range.min.<lpse_code>        535

panl.range.min.value.<lpse_code>        535

panl.range.min.wildcard.<lpse_code>        536

panl.range.prefix.<lpse_code>        537

panl.range.suffix.<lpse_code>        537

panl.range.suppress.<lpse_code>        538

panl.remove.solr.json.keys        540

panl.results.fields.<field_set>        540

panl.results.fields.default        541

panl.results.fields.empty        541

panl.results.testing.urls        542

panl.search.<lpse_code>        542

panl.search.fields        543

panl.server.extra        544

panl.sort.fields        545

panl.status.404.verbose        545

panl.status.500.verbose        546

panl.suffix.<lpse_code>        546

panl.type.<lpse_code>        547

panl.uniquekey.<lpse_code>        547

panl.unless.<lpse_code>        548

panl.when.<lpse_code>        548

solr.default.query.operand        549

solr.facet.limit        550

solr.facet.min.count        550

solr.highlight        551

solr.numrows.default        551

solr.numrows.lookahead        552

solr.numrows.maximum        552

solr.numrows.morelikethis        553

solr.search.server.url        553

solrj.client        554

Decoding the Solr Query Parameters        555

facet=        555

facet.field=        556

facet.limit=        557

facet.mincount=        558

f.<solr_field>.facet.mincount=        560

f.<solr_field_name>.facet.sort=        560

fq=        560

hl=        564

hl.fl=        564

q=        564

q.op=        565

rows=        566

sort=        566

start=        566

stats=        567

stats.field=        567

Decoding the Solr More Like This Query Parameters        568

Translating The Parameters        568

qt=        570

q=        571

———————

PART 9:

WRAP UP

.        572

Design Considerations        573

Collections And FieldSet URL Paths (CaFUPs)        574

In-Built Panl Results Testing URLs        575

Panl Server Startup        576

Error Messaging        577

Testing vs. Production        580

Quick Middleware        580

Scalability        581

The Code        581

Afterword        583

On The Tag-Line        583

On Documentation        583

On This Book        585

Additional Functionality in the Pipeline        586

Real Life Implementations        587

Appendices        592

Panl Integration Versions        592

Panl URL Bindings        592

Definitions        597

Command Line Options        603

Sample .properties Files        608

Solr Version 8 & 7 Integration Notes        611

Solr Versions 6 and Below        618

End Plate        621

~ ~ ~ * ~ ~ ~