anl Server

Getting Started With Panl

A rather pleasing companion to the Apache® Solr® Faceted Search Engine

_____________

Author: Synapticloop - Version 2.0.0 (Release 1)


This book will rapidly get you up and running with a fully featured, SEO-friendly, keyword searchable, faceted search engine with an in-built, example search page to test it all out.



(Note: The most up-to-date version of this book can always be found here: https://github.com/synapticloop/panl/tree/main/src/book, and the online version here: https://synapticloop.github.io/panl/) 

WHY?

Because...


/Caran d'Ache/true/Black/bDW/


looks so much nicer than...


q=*:*&facet.mincount=1&rows=10&facet.field=lead_size_indicator&facet.fie
ld=grip_material&facet.field=
colours&facet.field=nib_shape&facet.field=d
iameter&facet.field=cap_shape&facet.field=brand&facet.field=mechanism_ty
pe&facet.field=length&facet.field=hardness_indicator&facet.field=grip_ty
pe&facet.field=cap_material&facet.field=lead_grade_indicator&facet.field
=tubing_material&facet.field=in_built_sharpener&facet.field=
disassemble&
facet.field=category&facet.field=body_shape&facet.field=clip_material&fa
cet.field=mechanism_material&facet.field=lead_length&facet.field=body_ma
terial&facet.field=in_built_eraser&facet.field=grip_shape&facet.field=re
lative_weight&facet.field=
name&facet.field=nib_material&facet.field=weig
ht&facet.field=
variants&facet=true&fq=brand:"Caran+d'Ache"&fq=disassembl
e:"true"
&fq=colours:"Black"&q.op=AND



TL;DR

If you are looking to get the quickest start on understanding how a Panl server is configured and up and running, then...

  1. GETTING UP TO SPEED
    Start with the Quick Start - The 5 Steps section to get the Solr and Panl servers up and running and browse the Panl Results Viewer.

    http://localhost:8181/panl-results-viewer/mechanical-pencils/default/ 
  2. UNDERSTAND
    The Solr Fundamentals and Panl Fundamentals section introduces you to the concepts that lay the foundations for the rest of the book.  Once you have a good understanding of the configuration and functionality, then there are two example walkthroughs:
  1. A FULL EXAMPLE
    Work through the
    The Bookstore Walkthrough section to gain a deeper understanding of the complete process, from schema design, indexing, and configuration of both Panl and Solr.
  2. PROGRAMMATIC INDEXING
    Work through the
    Filesystem Indexing And Searching Walkthrough chapter to understand how to programmatically index files and commit them to Solr.
  1. INTEGRATE
    Use your existing dataset to set up Panl with
    Integrating An Existing Solr Schema.
  2. REFERENCE
    Need help understanding the options? See the Panl Configuration part.
  3. UPGRADE
    See the Upgrading Your Panl Server Version section.

QUICK TIP

Tip: Documentation for this book can also be read online, see:

https://synapticloop.github.io/panl/

or, to download the copy of this book for the Panl version that you are using, see:

https://github.com/synapticloop/panl/releases/

Corrections and updates to this book, when done, will always be reflected online and the main GitHub branch - see:

https://github.com/synapticloop/panl/blob/main/src/dist/book/Getting%20Started%20With%20Synapticloop%20Panl.pdf

~ ~ ~ * ~ ~ ~


Copyright Notices

MECHANICAL PENCIL DATA: The data used in this book is based on 2mm mechanical pencil data hand curated by the author.  The data and associated Solr search results utilised in this book references manufacturers of writing instruments, specifically 2mm mechanical pencils.  The manufacturer's name, the name of the pencil (and/or model number) is a trademark and/or is copyright of the manufacturer.

IMAGES, APPLICATIONS, AND SITE REFERENCES: Screenshots of applications and external websites are included in this book, they remain the copyright of the original owner and do not infer any association with, or endorsement of, Synapticloop, and/or vice versa.

Additional datasets may have information that is copyright to the respective owners.


IMPORTANT: Whilst the project is released under the very liberal MIT licence, the sample mechanical data remains under the copyright of Synapticloop, and may not be used for commercial purposes.

A NOTE ON URL ENCODINGS: Within this book if you are using the Firefox browser then the address bar will automatically decode any URL encoded space characters ('%20') - other browsers will keep the encoding in the address bar.  Within this book, the majority of example URLs will not show the %20 encoding, however the associated href in the link will include these encodings.

~ ~ ~ * ~ ~ ~


TABLE OF CONTENTS

———————

PART 1:

INTRODUCTION

.        20

README.1ST        21

Book Versioning        21

Solr Schema Versions        21

About This Book        22

Nomenclature Used Throughout This Book        22

Book Format Conventions        26

Welcome To Faceted Search        29

A Simple Faceted Search Example        31

A More Complex Faceted Search Example        32

About Apache Solr        34

Welcome To Synapticloop Panl        35

Additional Panl Niceties        35

About Panl Server        38

How Many Facets Does Panl Support?        39

Panl URL Structure        41

In-Built Web Apps (Viewer / Explainer / Single Search Page)        43

About Panl Generator        52

Why Synapticloop Panl?        54

What is a LPSE (pronounced 'lapse') code?        55

The three types of LPSE codes        56

URL Path Nomenclature        56

Is Synapticloop Panl For Me?        57



PART 2:

GETTING UP AND RUNNING

.        59

What You Will Need        60

Download The Resources        60

The Panl Directory Structure        60

Panl Server Versions        63

Version Numbering        63

Solr-Panl-10-x.x.x        63

Solr-Panl-9-x.x.x        64

Solr-Panl-8-x.x.x & Solr-Panl-7-x.x.x        64

Solr-Panl-6-x.x.x and earlier version        64

Quick Start - The 5 Steps        65

A Note On Running The Commands        67

Latest Version Panl 2.0.0 and Solr 9.8.0        67

Previous Solr 9.*.* Versions Command Line Changes        67

Operating System Dependent        69

Windows Commands        69

*NIX Commands        71

Next Steps        72

Getting Started        74

Downloading the Resources        74

The Solr server        75

The Panl server        75

Solr Versions Command Line Options        76

Creating and Starting a Solr Cloud Instance        77

Creating a collection        79

Indexing the Data        80

Starting The Panl Server        81

When Something Goes Wrong        82

Checking the Logs        83

Deleting a collection        84

Stopping and Restarting the Solr Panl tutorial process        84

Stopping Solr        85

Re-Starting Solr        85

Worst-case scenario        86

Next Steps        86



PART 3:

GOING FURTHER

.        87

Solr Fundamentals        89

Solr Configuration File Changes        89

Fields and Field Types        91

Determining If A Solr Field Type Is Analysed        91

Overview of Indexed / Stored / Analysed Fields        93

Highlighting        94

Panl Fundamentals        95

Configuration Files        95

The Relationship Between Configuration Files        96

Generating And Editing The Configuration Files        97

Keyword Search Field Configuration        99

solrconfig.xml        99

managed-schema.xml        100

Summary        103

Specific Field Keyword Search Configuration        105

managed-schema.xml        106

<panl_collection_url>.panl.properties        106

Summary        107

Panl Field Configuration        108

Panl Field        110

Panl Facet Field        111

Setting The Field To Allow A Specific Search        112

Summary of Panl Field Configuration Options        112

The Bookstore Walkthrough        114

0. High Level Requirements        115

1. Understanding the Dataset        116

2. Configure the Solr Index        119

3. Configure the Panl Server        125

The Default Search Page Configuration        128

The Generated panl.properties File        130

The Generated <panl_collection_url>.panl.properties File        130

4. Determine the Web Pages to Render        156

Author and Author Series        156

Author Listing        157

The Iterative Implementation Process        157

Filesystem Indexing And Searching Walkthrough        159

Indexing A Dataset        159

Project Requirements        160

Project Steps        160

1. Initialise a new project        160

2. Download and Configure the Solr Server        162

3. Download and Configure the Panl Server        166

4. Write the Indexing Code        169

5. Run the servers        175

6. Search and facet on the results        176

Extending The Project        178

Additional Data        179

All Data Panl Server        179

Simple Date        180

Creating and Indexing the Data        180

panl.properties File Additions        181

Mechanical Pencils OR Facet        181

Creating and Indexing the Data        182

panl.properties File Additions        182

Mechanical Pencils More Facets        182

Creating and Indexing the Data        183

panl.properties File Additions        183

Bookstore        183

Working With Any Dataset        185

Supported Solr Data Types        187

Unsupported/Partially Supported Solr Field Types        188

A Note on Prefixes, Suffixes, and Infixes        189

Facet and Field Types        190

Search Fields        191

- The setup of a Specific Solr Search Search Field crosses both the <panl_collection_url>.panl.properties file and the Solr managed-schema.xml file and is used as an addition to a Field or Facet Field.        191

Fields        191

Facet Fields        192

Other Facet Options        203

Understanding Solr Configuration And Panl Integration        206

Querying Data        206

The Solr Query Operand        207

The Default Keyword Search        209

The Specific Solr Field Keyword Search        211

The Solr Managed Schema        215

Determining the Appropriate FieldType and Attributes        219

Testing The Impact Of Indexed / Stored / Analysed        224

The Impact Of docValues (Schema Version 1.7+)        230

The Solr Configuration File        233

Query Request Handler        233

Highlighting        234



PART 4:

PANL CONFIGURATION IN DETAIL

.        238

Panl Configuration        239

The panl.properties file        239

The <panl_collection_url>.panl.properties file        240

The panl.properties Configuration File        242

Configuring The SolrJ Connector        243

Setting The Solr Server URL(s)        244

Enabling The Panl Results Testing URLs        245

The Panl Results Viewer        246

The Panl Results Explainer        248

The Panl Single Page Search        248

Setting HTTP Status Message Verbosity        249

Binding Solr collections to Panl URL paths        250

The <panl_collection_url>.panl.properties Configuration File        252

Parameter and Operand Definitions        253

panl.param.query        253

panl.param.sort        254

panl.param.page        254

panl.param.numrows        256

panl.param.query.operand        256

panl.param.passthrough        257

panl.param.passthrough.canonical        258

panl.form.query.respondto        259

panl.include.single.facets        260

panl.include.same.number.facets        260

solr.default.query.operand        262

solr.facet.limit        262

solr.facet.min.count        262

solr.numrows.default        262

solr.numrows.lookahead        263

solr.numrows.maximum        263

solr.highlight        263

Field Definitions        263

Properties Available for Fields        264

Facet Definitions        265

Properties Available for Facets        266

REGULAR Facet        269

Example        270

Decoding the URL        271

BOOLEAN Facet        271

Additional Properties        272

Example        273

Example: Checkbox        274

Decoding the URL With a Replacement Value        275

Defining a Prefix and Suffix        275

Decoding the URL With a Value Replacement and a Prefix or Suffix        276

OR Facet        276

Additional Properties        276

The Difference Between Multi-valued Facets and OR Facets        277

Example        282

Decoding the URL        285

RANGE Facet        285

Additional Properties        286

Suppressing Individual Facet Values        288

Visualising the Properties        289

Decoding the URL - Range facet with an infix        291

Decoding the URL - Range facet without an infix        291

DATE Range Facet        292

Additional Properties        294

Example        295

Decoding the URL        296

Some Notes        297

About Value Replacements        297

On Field Validations        299

About Hierarchical Facets        299

About Unless Facets        301

About Sorting Facets        301



PART 5:

INTEGRATION AND PRODUCTION

.        304

Putting It All Together (Technically)        306

A Brief Architecture        306

Production Readiness        307

Logging        307

Setting the Logging Configuration and Levels        309

Upgrading Your Panl Server Version        310

Breaking Changes        310

Generating SEO Friendly URLs        310

Data Dependencies        310

Passthrough URLs        311

Handling Errors        311

Panl LPSE URL Paths Explained        312

Parameter Query        313

Parameter Query Operand        314

Parameter Number of Rows        314

Parameter Page Number        316

Parameter Pass Through        317

Parameter Sort Order        318

Facets        319

Explaining A More Complex Example        321

Search Integration And The Panl Response Object        323

Example JSON        323

URL Bindings        325

Error Responses        325

Success Responses        326

Generating Panl Links        328

Generating the HTML        329

Generating The href Attribute        330

Generating The Anchor Text        330

"response.docs" JSON Object Results Integration        330

Example docs JSON Response        330

Example panl.fields JSON Response        332

Example Rendering        332

Implementation and Integration        332

"panl.search" Keyword Search Integration        334

Example Rendering        335

Example JSON        335

Implementation and Integration        336

"panl.search" Specific Solr Search Field Integration        336

Example Rendering        338

Example JSON        338

Implementation and Integration        339

"panl.active" JSON Object        339

Integration and Implementation        342

"panl.active.facet" REGULAR facet object        343

Example Rendering        343

Example JSON        343

Integration and Implementation        344

"panl.active.facet" BOOLEAN facet object        344

Example Rendering        345

Example JSON        345

Integration and Implementation        345

"panl.active.facet" BOOLEAN Checkbox facet object        346

Example Rendering        347

Example JSON - active facets        347

Integration and Implementation        348

"panl.active.facet" DATE Range facet object        349

Example Rendering        349

Example JSON        350

Integration and Implementation        350

"panl.active.facet" OR facet object        351

Example Rendering        351

Example JSON        352

Integration and Implementation        352

"panl.active.facet" RANGE facet object        352

Example Rendering        353

Example JSON        354

Integration and Implementation        354

"panl.active.numrows"        355

Example Rendering        355

Example JSON        355

Integration and Implementation        355

"panl.active.page"        356

Example Rendering        356

Example JSON        357

Integration and Implementation        357

"panl.active.query" JSON Object        357

Example Rendering        358

Example JSON        358

Integration and Implementation        358

"panl.active.query_operand" JSON Object        359

Example Rendering        359

Example JSON        359

Integration and Implementation        360

"panl.active.sort" JSON Array        360

Example Rendering        361

Example JSON        361

Integration and Implementation        362

"panl.available" JSON Object        363

Example JSON        363

"panl.available.date_range_facets" JSON Array        364

Example Rendering        364

Example JSON        364

Integration and Implementation        365

"panl.available.facets" JSON Array        365

Example JSON - OR Separator        365

Integration and Implementation        366

"panl.available.range_facets" JSON Object        368

Integration and Implementation        368

"panl.pagination" JSON Object        375

Example JSON        375

Integration and Implementation        375

"panl.sorting" JSON Object        376

Example JSON        376

Integration and Implementation        378

"response.highlighting"        381

Single Search Page Integration And The Panl Response Object        382

URL Bindings        382

Error Responses        383

Success Responses        383

The "lpse_lookup" JSON Object        384

The "lpse_order" JSON Array        385

Generating the LPSE URL path        385

More Facets Integration And The Panl Response Object        386

URL bindings        386

Building the Request URL        386

Determining The Facet Limit        387

Error Responses        389

Success Responses        389

The "facet" JSON Object        390

Implementation notes        390

Lookahead Integration And The Panl Response Object        392

URL bindings        392

Building the Request URL        393

Error Responses        394

Success Responses        394

Integrating An Existing Solr Schema        396

Using the Panl Generator        396

Running the Command Line Utility        396

Editing the Generated Files        400

Manually Creating the Configuration Files        400

The panl.properties File        401

The <panl_collection_url>.panl.properties File        402



PART 6:

REFERENCE MATERIAL

.        404

Panl Cookbook        405

BOOLEAN Facets        405

Displaying a BOOLEAN facet as a checkbox        405

RANGE Facets        407

Minimum and Maximum values        407

'Reverse' star Ratings        407

Example Properties        407

REGULAR Facets        409

Condensed Multivalued Paths        409

Example Properties - Original        409

Example Properties - Condensed        410

OR Facets        413

Condensed OR Facets        413

Example Properties - Original        413

Example Properties - Condensed        414

Analysed Facets (think Word Clouds)        415

Properties Quick Reference        417

Property References Format Explained        417

panl.bool.<lpse_code>.false        418

panl.bool.<lpse_code>.true        419

panl.bool.checkbox.<lpse_code>        420

panl.collection.<solr_collection_name>        421

panl.date.<lpse_code>.days        422

panl.date.<lpse_code>.hours        423

panl.date.<lpse_code>.months        424

panl.date.<lpse_code>.next        425

panl.date.<lpse_code>.previous        427

panl.date.<lpse_code>.years        428

panl.decimal.point        429

panl.facet.<lpse_code>        430

panl.facetsort.<lpse_code>        431

panl.field.<lpse_code>        432

panl.form.query.respondto        433

panl.include.same.number.facets        434

panl.include.single.facets        434

panl.lpse.ignore        435

panl.lpse.length        435

panl.lpse.order        436

panl.multivalue.<lpse_code>        437

panl.multivalue.separator. <lpse_code>        437

panl.name.<lpse_code>        438

panl.or.always.<lpse_code>        439

panl.or.facet.<lpse_code>        439

panl.or.separator.<lpse_code>        440

panl.param.numrows        441

panl.param.numrows.prefix        442

panl.param.numrows.suffix        442

panl.param.page        443

panl.param.page.prefix        444

panl.param.page.suffix        444

panl.param.passthrough        445

panl.param.passthrough.canonical        445

panl.param.query        446

panl.param.query.operand        446

panl.param.sort        447

panl.prefix.<lpse_code>        448

panl.range.facet.<lpse_code>        448

panl.range.infix.<lpse_code>        449

panl.range.max.<lpse_code>        450

panl.range.max.value.<lpse_code>        451

panl.range.max.wildcard.<lpse_code>        451

panl.range.min.<lpse_code>        452

panl.range.min.value.<lpse_code>        453

panl.range.min.wildcard.<lpse_code>        454

panl.range.prefix.<lpse_code>        454

panl.range.suffix.<lpse_code>        455

panl.range.suppress.<lpse_code>        456

panl.results.fields.<field_set>        457

panl.results.fields.default        458

panl.results.fields.empty        458

panl.results.testing.urls        459

panl.search.<lpse_code>        459

panl.search.fields        460

panl.sort.fields        461

panl.status.404.verbose        462

panl.status.500.verbose        462

panl.suffix.<lpse_code>        463

panl.type.<lpse_code>        463

panl.unless.<lpse_code>        464

panl.when.<lpse_code>        464

solr.default.query.operand        465

solr.facet.limit        466

solr.facet.min.count        466

solr.highlight        467

solr.numrows.default        467

solr.numrows.lookahead        468

solr.numrows.maximum        468

solr.search.server.url        469

solrj.client        469

Decoding the Solr Query Parameters        471

facet=        471

facet.field=        472

facet.limit=        473

facet.mincount=        474

1. Default facet mincount        475

2. OR Facets mincount        475

f.<solr_field>.facet.mincount=        476

f.<solr_field_name>.facet.sort=        476

fq=        476

REGULAR Facets        477

REGULAR (multivalued) Facets        477

BOOLEAN Facets        478

RANGE Facets        478

OR Facets        479

DATE RANGE Facets        479

hl=        479

hl.fl=        480

q=        480

q.op=        481

rows=        481

sort=        482

start=        482

stats=        482

stats.field=        483



PART 7:

WRAP UP

.        484

Design Considerations        485

Collections And FieldSet URL Paths (CaFUPs)        486

In-Built Panl Results Testing URLs        487

Panl Server Startup        488

Error Messaging        489

404 - Not Found        489

500 - Internal Server Error        490

Testing vs. Production        492

Quick Middleware        492

The Code        492

Afterword        494

On The Tag-Line        494

On Documentation        494

On This Book        496

Additional Functionality in the Pipeline        497

Appendices        504

Panl URL Bindings        504

User Defined URL Bindings        504

Panl Internal Functional URL Bindings        505

Internal Testing URL Bindings        507

Definitions        508

Command Line Options        513

Panl Server        513

Panl Generator        514

Usage Text        514

Sample .properties Files        518

The Sample panl.properties Files        518

The Sample <panl_collection_url>.panl.properties Files        519

Solr Version 8 & 7 Integration Notes        521

SolrJ        521

Solr Configuration files        521

JSON Response Object        521

Setting up a Solr 7 or 8 server        522

Additional Solr Version 7 Integration Notes        527

Solr Versions 6 and below        528

End Plate        530

~ ~ ~ * ~ ~ ~