Automatic Indexing

Although the Smart Search index is automatically kept up-to-date whenever content items are amended, there are some circumstances where you need to re-run the Indexer. You can do this manually using the Index toolbar button in the Smart Search: Content Map page. However if you need to re-index content automatically then it is also possible to run the Indexer from the command line interface (CLI). This makes it particularly convenient to run the Indexer from a cron job.

From the CLI director the command line is:

php joomla.php finder:index

Typical output from the command line Indexer looks like this:

Smart Search INDEXER
============================

Starting Indexer
Setting up Finder plugins
Setup 154 items in 0.094 seconds.
 * Processed batch 1 in 0.213 seconds.
 * Processed batch 2 in 0.182 seconds.
 * Processed batch 3 in 0.177 seconds.
 * Processed batch 4 in 0.009 seconds.
Total Processing Time: 0.676 seconds.

Purging Before Indexing

Ordinarily, running the Indexer will do an incremental update of the index. That is, it will only update the index for those content items that have changed since the index was last updated. It will not, for instance, remove any items from prior indexing that came from plugins that were uninstalled. So, if you need to completely clear all the existing index entries before completely rebuilding the index, then you need to do a "purge and then index" operation. To do that you can add the --purge argument to the command line, like this

php joomla.php finder:index --purge

Note that this will attempt to preserve any static filters you may have set up, whereas selecting Maintenance→Clear Index on the toolbar will not preserve your static filters.

Setting Up a cron Job

Whilst the specifics are beyond the scope of this article, in general you will merely have to enter the above command into the cron job manager and specify the time or times at which the job is to be run. You will probably need to include the full path to the Joomla! CLI PHP file. For example, like this

php /var/www/myjoomla/cli/joomla.php finder:index --purge

It may also be necessary to specify the full path to the php command such as /usr/local/opt/php@8.2/bin/php.

Out of Memory Issues

If your site has particularly complex indexing requirements it is possible that the standard memory allocation will not be sufficient for the Indexer to run to completion. You can increase the memory allocated to the command-line Indexer using an extra parameter on the command-line, like this:

php -d memory_limit=256M joomla.php finder:index

Replace the 256M with whatever is appropriate for your circumstances.

The command-line Indexer uses the same parameters as the Indexer on the Global Configuration→Smart Search→Index page. You can access these parameters by clicking the Options toolbar button on any Smart Search menu page. The Indexer Batch Size field affects the amount of memory used by the Indexer.

Results Pages

The Smart Search component has five layout files that you can override to produce a layout to your own liking. To initiate the process from the Administrator menu:

• Log into the Administrator interface.
• Click the System menu item, then click Site Templates.
• Click the Cassiopeia Details and Files link.
• Select the Create Overrides tab.
• On the Components panel click com_finder, then click the Search item
• On the Editor panel select html→com_finder→search to see the list of override files created.

These files will not be affected by Joomla updates but you may be reminded to check them if the original sources are updated.

The Search View (Default Layout)

The default layout search view is divided into several parts:

• default layout
• form layout
• results layout
• sorting layout

The default layout (default.php)

This layout is very simple. It just defines the structure in which the search form and the search results are displayed. This layout is also responsible for loading the default CSS stylesheet for Smart Search which is located in media/com_finder/css/finder.css, so you might want to alter that to load your own CSS rules for Smart Search.

The form layout (default_form.php)

This layout defines the code required for the search form to operate correctly. The layout uses JavaScript code so you need to take care to preserve selectors that should not be altered unless you know what you are doing.

The view method getFields includes a number of hidden fields that are required for reliable searching. The search term is defined by the input field with the name of "q".

The results layout (default_results.php)

This layout produces the list of matching results for the search term. It also handles pagination and loads a layout for each individual search result.

The result layout (default_result.php)

This is the layout used to display a single result.

The sorting layout (default_sorting.php)

This item is only present if Show Sort Fields is set to Yes and one or more Additional Sort Fields are selected in the Menu Item Advanced tab.

Site Indexing

Smart Search works by creating and maintaining an independent index of search terms in a number of database tables. The problem for large sites is that the indexing process can be quite heavy in CPU, memory and disk usage. Even after the initial construction of the index, incremental updates can also be heavy. The good news is that querying the index is a relatively quick and lightweight operation.

Smart Search is suitable for the majority of Joomla sites. However, search presents particular challenges for large sites. It should be remembered that Smart Search is a pure PHP implementation of a search engine and particularly large sites may be better off using a standalone search engine such as Solr.

To use Smart Search on a large site you will probably need to adjust some of the configuration settings. What follows is some general advice on what to look out for and what to try tweaking. There are a number of known outstanding issues with regard to running Smart Search on large sites which will hopefully be addressed in future versions and these are also described here.

Always Use the CLI Indexer

Because the initial indexing process can take a long time, it is best to run the indexer from the command line to avoid any issues from browser sessions timing out. The CLI indexer will not timeout regardless of how long it takes to complete and it can be easily aborted if problems are encountered. Furthermore, error messages are visible with the CLI indexer, whereas they are hidden when running from the Administrator.

To use the CLI, open a terminal, switch to the cli folder in the root of your site and issue the following command:

php joomla.php finder:index

Batching

The indexer breaks the indexing job into batches of content items. By default the batch size is set at 30 meaning that up to 30 content items will be indexed per batch. Increasing the batch size will potentially make the indexing process faster, but it will use more memory and possibly more temporary disk space.

Out of Memory Issues

If the indexer is running out of memory, try making the following adjustments one at a time until the problem is resolved.

1. Decrease the batch size. If you have particularly large content items the indexer can run out of memory on even a single content item, so try dropping it to 5 initially and if you still run out of memory, drop it to 1.
2. If you are able to allocate more memory to the indexer, do so. You can increase the memory allocated to the command line indexer using an extra parameter on the command line. For example, to increase the memory limit to 256Mb use the following command, replacing the 256M with as much memory as you can safely allocate to a process on your system.
   php -d memory_limit=256M finder_indexer.php
3. Reduce the memory table limit. The default is 30,000 terms which means that as soon as the temporary in-memory #__finder_tokens table reaches this number of rows, the indexer will switch to using a disk table instead of a memory table. It may be that you don't have enough memory to handle a full or nearly full memory table so reducing the limit will tell the indexer to switch to disk sooner and so use less memory. Try 10,000 or even much smaller numbers.
4. Change the database engine used for the __finder_tokens and #__finder_tokens_aggregate tables from MEMORY to InnoDB. This could seriously impact performance as more of the indexing process will use the disk instead of memory, but it might allow the indexer to finish without running out of memory. Expect the indexing process to run for much longer. This will not affect search performance however.
5. Try to identify which content items are causing the indexer to run out of memory. If it's not obvious then you might try disabling all Smart Search plugins except one. Running the indexer with only one plugin enabled at a time should reveal which content type(s) are causing the issue. As a last resort consider breaking a few exceptionally large content items into separate items. If the problem is with a custom content type, look at the plugin code and consider indexing less of the available content per item.

Out of Disk Space Issues

The Smart Search index tables can grow quickly! The #__finder_links_termsX tables (where X is a single hexadecimal character) contain one row per term/phrase per content item and a single Joomla article containing 1,000 words will typically result in approximately 3,000 rows being added to these tables. A second article of a similar size will add a similar number of rows even if both articles contain the same words. A site with tens of thousands of articles, some of which may contain thousands of words, is likely to end up with these mapping tables containing millions of rows. It is not unusual for the index tables to occupy several gigabytes of disk space in such circumstances.

With the present version of Smart Search there isn't much you can do about this. However, it is hoped that in the near future you will be able to adjust the number of words per phrase that get indexed. For Joomla! 5 this is hard-wired at 3, meaning that every word that gets indexed is also indexed as part of a pair of adjacent words and as part of a triplet of adjacent words. This is useful for the auto-completion feature and generally improves the quality of search results. On sites where disk space is an issue, it would be good to reduce this to 2 or even 1, so that the mapping tables would be correspondingly smaller.

Notes

1. There is currently no concurrency locking to prevent more than one process running the indexer at the same time. This will almost certainly result in a corrupt index. Even someone saving changes to a content item while a full index is being carried out could potentially damage the index. As such, it may be safest to only run the indexer when the site can be scheduled to be offline.

About Options

Options allow you to manage the Smart Search global settings. The Option settings are inherited by views but the settings in a Menu Item will override the Global settings.

The Options panel is separated into two parts. When you are finished making adjustments, select the Save button. Otherwise, select the Cancel button to abandon any changes.

Search Options

The search options control the behavior of the search forms and search results for the Finder front-end.

• Result Taxonomy - This option controls whether search results should be shown with their corresponding taxonomy(ies). Defaults to Show.
• Result Description - This option controls whether search results should be shown with descriptions. Defaults to Show.
• Description Length - This option controls the maximum character length of the search result descriptions. The descriptions are truncated to the maximum length but the truncation is always done at the last space character so as not to truncate in the middle of a word. Only effective when the Result Description option is set to Show. Defaults to 255 characters.
• Result Image - This option controls whether images within the searched content are shown in the search results. Defaults to Hide.
• Result Date - This option controls whether the modification date of the searched content is shown in the search results. Defaults to Show.
• Result URL - This option controls whether search results should be shown with URLs below the description (if enabled). Defaults to Show.
• Gather Search Statistics - Defaults to Show.
• Allow Empty Search - Defaults to Show.
• Search Suggestions - Defaults to Show.
• Did You Mean - Defaults to Show.
• Query Explanation - Defaults to Show.
• Advanced Search - This option controls whether the advanced search options should be shown. Defaults to Show.
• Advanced Tips - This option controls whether advanced search tips should be shown. Defaults to Show.
• Expand Advanced Search - The advanced search options are contained in a collapsible container. This option controls whether the advanced search options container should be expanded by default. Defaults to No.
• Date Filters - This option controls whether the date filters should be shown in the advanced search options. Defaults to Hide.
• Sort Field - This option selects the taxonomy field on which to sort the results. Defaults to "Relevance".
• Sort Direction - This option selects the order on which to sort the results. Defaults to "Descending".
• Highlight Search Terms - This option controls whether search terms should be highlighted in search results. Defaults to Yes.
• Enable OpenSearch - Defaults to Yes.
• OpenSearch Name - Defaults to nothing.
• OpenSearch Description - Defaults to nothing.

Index Options

The index options control the behavior of the indexer used to process and analyse content for searching. Most settings such as the weight multiplier and stemmer options will not produce immediate changes as they are used during indexing and the effects of changing those settings will only be seen after the index has been purged and run again.

• Search For Phrases - This option selects whether the indexer should search for phrases as described above. Impacts performance. Defaults to Disabled.
• Indexer Batch Size - This option controls the indexer batch size. The Smart Search indexer breaks up the total indexing process into a number of batches to reduce server load and avoid PHP's memory and execution time limits. If the content items being indexed are particularly short and/or the server is particularly fast, the indexer can process more items per batch. However, if the content items being indexed are particularly long and/or the server is particularly slow, the indexer has to process fewer items per batch. Defaults to 50.
• Title Text Weight Multiplier - This option controls how much influence the title text has on the overall relevance score of a search result. Defaults to 1.7.
• Body Text Weight Multiplier - This option controls how much influence the body text has on the overall relevance score of a search result. Defaults to 0.7.
• Meta Data Weight Multiplier - This option controls how much influence the meta-data text has on the overall relevance score of a search result. Defaults to 1.2.
• Path Text Weight Multiplier - This option controls how much influence the path/URL text has on the overall relevance score of a search result. Defaults to 2.0.
• Misc Text Weight Multiplier - This option controls how much influence the miscellaneous text has on the overall relevance score of a search result. Defaults to 0.3.
• Enable Logging - creates a log file during the index process. Defaults to No.
• Default Language - Defaults to None.
• Filter Common Words - Defaults to No.
• Filter Numeric Terms - Defaults to No.