jasonwitty
8d9bd59e19
PLACEHOLDER — rewrite this message before opening the PR. Adds a plugin that shows an AI generated summary above the search results, produced by an LLM server the administrator runs and reaches over the OpenAI chat completions API.
104 lines
4.0 KiB
ReStructuredText
104 lines
4.0 KiB
ReStructuredText
.. _ai_summary plugin:
|
|
|
|
==========
|
|
AI Summary
|
|
==========
|
|
|
|
.. sidebar:: Further reading ..
|
|
|
|
- :ref:`Configuration <settings ai_summary>`
|
|
- :ref:`dev plugin`
|
|
- :ref:`result types`
|
|
|
|
The AI Summary plugin shows a generated answer above the ordinary search
|
|
results. The text comes from an LLM server run by the administrator, which
|
|
speaks the `OpenAI chat completions API`_ -- `Ollama`_, Hugging Face TGI,
|
|
LiteLLM, vLLM, llama.cpp and anything else implementing that specification.
|
|
See :ref:`its configuration <settings ai_summary>` for how to set one up.
|
|
|
|
The summary is generated asynchronously: the result page is delivered without
|
|
delay and carries an empty placeholder, which the browser fills from a second,
|
|
streaming request.
|
|
|
|
Request flow
|
|
============
|
|
|
|
.. _ai_summary dataflow:
|
|
|
|
.. kernel-render:: DOT
|
|
:alt: Data flow between browser, SearXNG and the LLM server
|
|
:caption: A search that produces a summary: two requests, not one
|
|
|
|
digraph ai_summary {
|
|
rankdir=LR;
|
|
graph [fontname="sans-serif", ranksep=1.1, nodesep=0.4];
|
|
node [fontname="sans-serif", fontsize=11, shape=box, style="rounded,filled",
|
|
fillcolor="#f4f4f4", color="#999999"];
|
|
edge [fontname="sans-serif", fontsize=9, color="#666666"];
|
|
|
|
browser [label="browser"];
|
|
searxng [label="SearXNG"];
|
|
engines [label="search engines", fillcolor="#ffffff"];
|
|
llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"];
|
|
|
|
browser -> searxng [label=" 1 GET /search"];
|
|
searxng -> engines [label=" 2 query"];
|
|
searxng -> browser [label=" 3 result page,\l empty summary box\l", constraint=false];
|
|
browser -> searxng [label=" 4 POST /ai_summary\l (query + results)\l"];
|
|
searxng -> llm [label=" 5 POST /v1/chat/completions"];
|
|
llm -> searxng [label=" 6 SSE token stream", constraint=false];
|
|
searxng -> browser [label=" 7 NDJSON token stream", constraint=false];
|
|
}
|
|
|
|
Steps 1 to 3 are an ordinary SearXNG search. :py:obj:`SXNGPlugin.post_search
|
|
<searx.plugins.ai_summary.SXNGPlugin.post_search>` adds an empty
|
|
:py:obj:`searx.result_types.AiSummary` placeholder to the answer area and
|
|
returns; the result page is not delayed.
|
|
|
|
Steps 4 to 7 run in the browser once the page is rendered.
|
|
``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary``
|
|
endpoint (registered in :py:obj:`searx.webapp`), which opens a streaming
|
|
request to the LLM server and re-emits the tokens as they arrive.
|
|
|
|
The two streams use different formats. The LLM server sends `SSE`_ --
|
|
``data: {...}`` lines terminated by ``data: [DONE]``. SearXNG re-emits them to
|
|
the browser as `NDJSON`_: one JSON object per line, ``{"delta": "..."}`` for
|
|
each chunk of text and a final ``{"done": true}``.
|
|
|
|
When no summary is generated
|
|
============================
|
|
|
|
:py:obj:`SXNGPlugin.post_search
|
|
<searx.plugins.ai_summary.SXNGPlugin.post_search>` adds no placeholder for:
|
|
|
|
- page two and beyond
|
|
- categories other than *general*
|
|
- non-HTML output formats (the JSON, CSV and RSS APIs)
|
|
- queries an engine already answered with an infobox (wikipedia, wikidata) or
|
|
an instant answer (e.g. ddg definitions)
|
|
- an empty query, or no LLM server configured
|
|
|
|
API keys
|
|
========
|
|
|
|
The administrator's ``api_key`` is sent only to the configured ``base_url``.
|
|
Users who point the ``ai_summary_server`` preference at a server of their own
|
|
authenticate it with their own ``ai_summary_api_key`` preference, which is
|
|
stored in a cookie and excluded from the preferences URL; the administrator's
|
|
key is never sent to such a server. A server URL carrying credentials in its
|
|
userinfo is ignored, and the administrator's default is used instead.
|
|
|
|
:py:obj:`_server_api_key <searx.plugins.ai_summary._server_api_key>` implements
|
|
the rule.
|
|
|
|
Reference
|
|
=========
|
|
|
|
.. automodule:: searx.plugins.ai_summary
|
|
:members:
|
|
|
|
.. _Ollama: https://ollama.com/
|
|
.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat
|
|
.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
|
|
.. _NDJSON: https://github.com/ndjson/ndjson-spec
|