Ongoing Maintenance

• AI Librarian - Conveyor can automatically identify duplicate or conflicting sources, and flag them
  • The AI Librarian will automatically maintain past answers. If new sources (newer answers or curated Q&As) suggest that some past answers are out of date (or just duplicate), it will automatically retire them
  • If curated Q&As are suspected as out of date or duplicate, AI Librarian will recommend retiring or merging them (from the Conveyor app UI). It is recommended to periodically review these recommendations
• Periodic review of KBs and documents
  • Designate curators for sources maintained within Conveyor (curated Q&As, documents)
  • Set max age for sources (and receive reminders for verification)
    • Use shorter verification cycles (e.g., quarterly) for rapidly evolving areas
    • Use longer cycles (e.g., annual, never) for stable information
  • You can take source popularity into account (retire unused sources, beef up and update popular topics)
• To reduce maintenance overhead - opt for automatic syncing of documents (read directly from Google Drive or leveraging APIs)
• Monitor AI answering accuracy - Conveyor’s main dashboard shows answer rate and accuracy over time. Review periodically to ensure accuracy is preserved (or improving) over time. If it is not - your Knowledge Library may deserve a refresh.
• Managing Product Line Transitions
  • If your organization is offering a new product - consider defining a new product line in Conveyor. Typically most knowledge should be shared (labeled as ‘any product’) but you may want to add specific sources (documents or curated Q&As) for the new product. You may also do that over time (as you start receiving more questionnaires about the new product).

Troubleshooting

ConveyoyAI typically gets the answer right >90% of the time.
But in the cases it doesn’t - here are a few tips on how to troubleshoot, as in many cases the solution is in your own hands.

• Sources cited - whenever an answer is inaccurate - check which sources were cited (on the right)
  • Is there a source that’s out of date and needs to be updated or retired?
  • Is the answer taken from a source that is only relevant to other product lines? Was the correct product line selected when uploading the questionnaire?
    Alternatively, you may need to refine which product lines the cited source is labeled with (e.g. from ‘any product’ to a more specific subset)
  • Sources seem irrelevant - perhaps there is no good answer to be found in the knowledge library? It may be time to add more knowledge.
    (note that the answer to the specific question will be leveraged as a past answer in the future, so no need to do anything for that, but if there is a broader gap you may want to take the opportunity to add relevant documents or knowledge)
• History - You can also check the history tab (also on the right). Some organizations use a custom prompt to adjust tone or answer specific questions in a certain way. When used, ConveyorAI will use this prompt to make edits to the out-of-the-box answer. You will see such edits in the answer history. This can help understand if the inaccuracy is coming from the original ConveyorAI answer or from the custom prompt edits. If the latter - you may need to update your custom prompt.