Inspecting, Monitoring, and Troubleshooting Axigen Indexes

For search and sort operations Axigen builds and maintains a set of indexes.

These indexes are used by various Axigen services (WebMail, IMAP) and are created and updated for each mailbox that exists within a domain. A number of SortIndex and SearchIndex jobs run on the Axigen server at a given time, each managing the updates for one mailbox at a time.

Indexing in Axigen X4 (10.4.x)

Starting with Axigen X4 all the server indexes used by Axigen have been converted to sqlite (fts) databases.

There are two major index types in axigen: sort indexes and search indexes.

Generally speaking, sort indexes are updated synchronously as clients expect them to mirror the state of the mailbox, whereas in axigen’s server model it is acceptable for search indexes to be updated asynchronously and thus fall behind mailbox content. This happens because computing search indexes is a relatively expensive operation that depends linearly to mailbox size.

The following index categories are used in X4:

• Calendar indexes (these are used to sort and search the calendar, compute recurrences for recurrent events, etc)

• Contact indexes (used to sort and search contact data)

• Conversation indexes (these are used when webmail uses the conversation based interface, where messages are grouped in threads)

• Email indexes (these are used for mail search)

• Sort indexes (these are used for sorting mail data, as required by the webmail interface but also other clients such as IMAP SORT commands, etc).

All indexes are saved on disk, at domain level, in their respective folders (e.g. calendar indexes in domain/calendar.idx, contact indexes in domain/contact.idx, etc).

Indexes are volatile information, so they are not part of the backups; if they are missing they will be recomputed at runtime.

Indexing Throughput

Axigen’s Processing Model: Services and Jobs

You can split Axigen’s processing model in two categories: services and jobs.

Services are used to implement client server interactions based on protocols: we have IMAP, WebMail, POP3, etc. services. Services are thus driven by client-server network interactions, where a client asks the server to execute a request and the server builds a response.

Jobs in contrast are used by Axigen for asynchronous processing; that is jobs compute data that will be required by services but they only do so when processing power is available. This is achieved using a job scheduler that decides which Axigen jobs should run and when, what number of jobs should be run in a axigen server at a given time, etc.

The job scheduler does its work using a number of 80 threads; this is the maximum number of threads that can execute jobs at any given time. Once the scheduler decides to run a specific job it will pass it to one of the threads for execution. Starting with Axigen X.3 the job scheduler decorates threads with a job id so thread names (as printed by top -H) can be used to inspect job status from outside Axigen.

Additionally the number of jobs that are to be executed in parallel at any time is also controlled by job category; if a certain job is scheduled to run and detects certain conditions are met (e.g. a calendar reminder job is about to execute and detects that too many calendar reminder jobs are already running) this job will give up its execution slot and postpone its own execution.

Controlling Indexing Throughput

Asynchronous indexes are updated by Axigen indexing jobs.

While an arbitrary number of indexing jobs may exist in memory at some point only a limited number of these can run in parallel in the same time.

The following asynchronous indexes: calendar search, contact search, mail search, mail sort index have their jobs controlled by the maxIndexingThreads configuration parameter.

To increase the speed in which indexing is performed for the above indexes, assuming the load average on your system is low enough, you can increase the parameter from default value 10 to a higher value such as 12 or 15. On the other hand, if your system cannot handle 10 parallel jobs you can decrease the number of indexing threads towards 5.

Monitoring the Progress of Index Jobs

Indexing jobs log their progress using a logger called "JOBLOG".

Here's a typical progress report for a SortIndex job:

Inspecting the Status of a Given Index

A set of CLI commands allow an Axigen admin to inspect the status of a given index:

In Axigen indexes are volatile, meaning that Axigen is able to recompute a missing index if needed.

A set of CLI commands, found under the root context, can be used to determine the disk location of a particular index and which folder is indexed by a particular database:

Indexes can be forcibly rebuilt by deleting them from disk.

Based on the example above, deleting /var/opt/axigen/domains/x.axigen.com/email.idx/2/2/2 will cause Axigen to recompute the email search index for user's Sent folder.