> ## Documentation Index
> Fetch the complete documentation index at: https://docs.neuroncx.in/llms.txt
> Use this file to discover all available pages before exploring further.

# CX Ops: Monitoring AI Service Level Indicators in Real Time

> Use the SLIs dashboard to measure LLM operation volumes, surface trending user intents, and track retrieval quality scores to keep your AI performing at its best.

The **CX Ops** dashboard — built around Service Level Indicators (SLIs) — gives you a quantitative view of how well your AI-powered conversation pipeline is performing end to end. Where other dashboards show you what users are doing, CX Ops shows you how well your AI is handling the workload: how many language model operations it's completing, which user intents are driving the most traffic, and whether the AI is actually retrieving the right answers when it matters. Monitoring these indicators regularly is the fastest way to catch degradation before it impacts your users.

## Completion Table

The Completion Table tracks the raw volume of four distinct language model operation types processed during the selected time period. Each row represents a different kind of AI compute happening inside your conversation pipeline.

| Operation | Description | Example Count |
| - | - | - |
| **Responses** | Full natural language replies generated and delivered to users | 2,164 |
| **Classifications** | Intent and topic tagging operations that route a message to the correct workflow | 2,156 |
| **Embeddings** | Vector representations created for semantic search and knowledge retrieval | 2,168 |
| **Transcriptions** | Speech-to-text conversions from voice channel interactions | 1,581 |

### What the numbers tell you

> [accordion] Responses vs. Classifications alignment
>
> In a healthy pipeline, the **Responses** count and the **Classifications** count should be close to each other — every message that gets classified should ultimately produce a response. A significant gap between the two (for example, far more classifications than responses) suggests that a portion of classified intents are not completing successfully. This could point to workflow errors, missing template mappings, or knowledge base gaps for those intents.

> [accordion] Embeddings as a retrieval health signal
>
> The **Embeddings** count reflects how often your system performed a semantic search against your vector store. A count notably higher than Responses suggests that some queries are triggering multiple retrieval passes before a response is generated — which can indicate low initial retrieval confidence. If embeddings are significantly lower than responses, some responses may be returning without a proper knowledge retrieval step, which often results in generic or hallucinated answers.

> [accordion] Transcriptions and voice pipeline load
>
> The **Transcriptions** count will naturally be lower than the others if only a portion of your traffic comes through voice channels. Cross-reference this number with the **Call Processing** workflow execution count in the Apps dashboard to verify that all voice calls are being transcribed. A large gap between the two may indicate dropped calls or transcription failures in your voice pipeline.

> [!NOTE]
>
> The Completion Table reflects operations at the backend level. One user conversation may generate multiple classification, embedding, and response operations depending on how many turns the dialogue contains.

***

## Trending Topics

The Trending Topics section surfaces the top 10 user intents and small-talk categories detected across all conversations during the selected period. Each entry corresponds to a classified intent label — such as `appointmentbook`, `reschedule`, or `cancel` — ranked by frequency.

This list is one of the most actionable outputs in the entire CX Ops dashboard because it tells you exactly what your users want, in aggregate, right now.

> [card] Validate your content coverage
>
> Compare the top trending topics against your Ontology dashboard's Topical Coverage charts. Any intent that ranks highly here but has thin knowledge base coverage is a content gap that is actively hurting your AI's performance today.

> [card] Prioritize template updates
>
> If a high-traffic intent lacks a mapped response template (visible in the Ontology dashboard's Topical Template Count), that intent is receiving a generic or fallback reply. Mapping a well-structured template to it can improve response quality immediately.

> [card] Detect emerging needs
>
> New topics appearing in the top 10 that weren't there in previous periods often signal a shift in user behavior — a new concern, a seasonal spike, or a recent event driving new questions. Treat new entries as early warnings to prepare content and templates before volume grows further.

> [card] Inform workflow design
>
> If a topic like `reschedule` or `cancel` consistently ranks in the top 10 but you don't have a dedicated workflow for it, that's a strong signal to build one. High intent frequency without a purpose-built workflow typically means those users are receiving lower-quality, patchwork responses.

> [!TIP]
>
> Export your Trending Topics list at the end of each month and compare it against the previous month. Consistent top-ranked topics should have mature, optimized templates and rich knowledge coverage. Rising topics are your next content and workflow priorities.

***

## LLM Retrieval Quality (SLI Graph)

The SLI Graph measures the **average retrieval quality score** for your AI's responses over time. This score reflects how successfully your LLM is finding and returning the correct answer from your vector stores and ontology layers — as opposed to generating a response that sounds plausible but isn't grounded in your knowledge base.

### Understanding the quality score

> [!NOTE]
>
> A score of **1** represents a successful, relevant response mapping — the AI retrieved the right content and produced a response that accurately addresses the user's query. Scores below 1 indicate partial matches, low-confidence retrievals, or responses generated without a strong knowledge base anchor.

The score is calculated based on retrieval matching quality and AI-side confidence signals. Monitoring this graph over time lets you see whether changes to your knowledge base, template mappings, or workflow routing are improving or degrading your AI's ability to give correct answers.

### Interpreting score trends

> [tab] Score consistently at 1
>
> Your retrieval pipeline is performing well — the AI is finding relevant content and mapping it accurately to user queries. Continue monitoring for any drops that follow content updates or workflow changes, but no immediate action is needed.

> [tab] Score gradually declining
>
> A slow decline often means your knowledge base is becoming stale relative to user needs. Your users may be asking questions that your content no longer answers well — perhaps due to outdated information or new topics emerging that haven't been indexed yet. Cross-reference with Trending Topics to identify which intents are driving the decline and update your KBase content accordingly.

> [tab] Score drops sharply after a change
>
> A sharp drop immediately following a content update, re-indexing event, or new deployment typically means something went wrong with that specific change. Check whether a template mapping was accidentally removed, whether a re-index completed successfully, or whether a new workflow is routing queries to the wrong knowledge index.

> [tab] Score fluctuates unpredictably
>
> Irregular fluctuations can indicate that your embeddings are inconsistent — some queries retrieve strong matches while others retrieve weak ones. This often points to uneven knowledge base coverage: well-covered topics score 1 reliably, while thin topics drag the average down. Use the Ontology dashboard's Topical Coverage view to find and fill the gaps.

### Improving retrieval quality: a continuous loop

**1. Monitor the SLI Graph weekly**

Set a regular cadence for reviewing your retrieval quality score. Weekly reviews give you enough data to spot trends without reacting to day-to-day noise.

**2. Cross-reference with Trending Topics**

When the score dips, look at which topics are trending. The most frequent intents with the thinnest knowledge coverage are the most likely culprits.

**3. Update the Ontology**

Add or revise Knowledge Blocks for the affected topics in the Content Console. Ensure that response templates are correctly mapped and that the KBase Index Summary reflects the new content after re-indexing.

**4. Validate the improvement**

After re-indexing and deploying updates, monitor the SLI Graph over the following week to confirm that the retrieval quality score recovers. If it doesn't, investigate whether the new content is being surfaced correctly by running test queries through your workflow.

> [!WARNING]
>
> Avoid making multiple simultaneous changes to your knowledge base, templates, and workflow routing at the same time. If the SLI score changes after a bundle of updates, you won't be able to determine which change caused the shift. Roll out changes incrementally and observe the SLI Graph between each change.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.