diff --git a/docs/dev/plugins/ai_summary.rst b/docs/dev/plugins/ai_summary.rst index f965f1ff4..f8830bea5 100644 --- a/docs/dev/plugins/ai_summary.rst +++ b/docs/dev/plugins/ai_summary.rst @@ -10,15 +10,23 @@ AI Summary - :ref:`dev plugin` - :ref:`result types` -The AI Summary plugin shows a generated answer above the search results. The -text comes from an LLM server the administrator runs; see :ref:`its -configuration ` for how to set one up. +The AI Summary plugin shows a generated answer above the ordinary search +results, unless an engine has already answered the query directly -- with a +Wikipedia infobox, for instance, or an instant answer. It is meant to run +against a local LLM server and speaks the `OpenAI chat completions API`_, so it +works with `Ollama`_, Hugging Face TGI, LiteLLM, vLLM, llama.cpp and anything +else that implements that specification. See :ref:`its configuration ` for how to set one up. -The interesting part of this plugin is *when* things happen. Generating an -answer takes seconds, and a search engine that waits seconds before painting -anything is a broken search engine. So the plugin never blocks the result -page: it renders an empty box, and the browser fills that box afterwards from a -second, streaming request. +The purpose of the summary is not to tell you what the model memorised during +training, but to summarise the up to date results your query actually returned. +That is what the *grounding* setting does, and why it is enabled by default. + +Generating an answer takes seconds, and a search engine that waits seconds +before painting anything is a broken search engine. The summary is therefore +produced asynchronously: the plugin renders an empty box, the result page is +delivered immediately, and the browser fills that box from a second, streaming +request. The results below stay readable and scrollable the whole time. Request flow @@ -122,6 +130,8 @@ The streaming response uses ``direct_passthrough``, so the generator must yield ``bytes`` -- werkzeug asserts on ``str``, and the Flask test client does not catch it. +.. _Ollama: https://ollama.com/ +.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat .. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events .. _NDJSON: https://github.com/ndjson/ndjson-spec