jasonwitty
b6eed9a993
The documentation was written from the perspective of someone who had just implemented the plugin: it opened with SSRF caveats and settings keys, and explained decisions rather than usage. The admin page now starts from what the feature is, followed by a four-step quickstart (install Ollama, pull a model, configure, restart) and a troubleshooting section for the failures that actually occur. The quickstart repeats the default plugins because a plugins: block replaces that list instead of merging into it -- following the short version of the instructions would otherwise switch every other plugin off. The developer page gains a rendered data flow diagram. The two-request design -- placeholder first, streamed answer second -- is the part of this plugin that is hard to convey in prose, and the SSE to NDJSON change is easier to see than to read about. The module docstring now describes the module and links to both pages, instead of restating administration guidance. Add Jason Witty to AUTHORS.rst. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
138 lines
5.4 KiB
ReStructuredText
138 lines
5.4 KiB
ReStructuredText
.. _ai_summary plugin:
|
|
|
|
==========
|
|
AI summary
|
|
==========
|
|
|
|
.. sidebar:: Further reading ..
|
|
|
|
- :ref:`settings ai_summary`
|
|
- :ref:`dev plugin`
|
|
- :ref:`result types`
|
|
|
|
.. contents::
|
|
:depth: 2
|
|
:local:
|
|
:backlinks: entry
|
|
|
|
The AI summary plugin shows a generated answer above the search results. The
|
|
text comes from an LLM server the administrator runs; see :ref:`settings
|
|
ai_summary` for how to set one up.
|
|
|
|
The interesting part of this plugin is *when* things happen. Generating an
|
|
answer takes seconds, and a search engine that waits seconds before painting
|
|
anything is a broken search engine. So the plugin never blocks the result
|
|
page: it renders an empty box, and the browser fills that box afterwards from a
|
|
second, streaming request.
|
|
|
|
|
|
Request flow
|
|
============
|
|
|
|
.. _ai_summary dataflow:
|
|
|
|
.. kernel-render:: DOT
|
|
:alt: Data flow between browser, SearXNG and the LLM server
|
|
:caption: A search that produces a summary: two requests, not one
|
|
|
|
digraph ai_summary {
|
|
rankdir=LR;
|
|
graph [fontname="sans-serif", ranksep=1.1, nodesep=0.4];
|
|
node [fontname="sans-serif", fontsize=11, shape=box, style="rounded,filled",
|
|
fillcolor="#f4f4f4", color="#999999"];
|
|
edge [fontname="sans-serif", fontsize=9, color="#666666"];
|
|
|
|
browser [label="browser\n(simple theme)"];
|
|
searxng [label="SearXNG"];
|
|
engines [label="search engines", fillcolor="#ffffff"];
|
|
llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"];
|
|
|
|
browser -> searxng [label=" 1 GET /search"];
|
|
searxng -> engines [label=" 2 query"];
|
|
searxng -> browser [label=" 3 result page,\l empty summary box\l", constraint=false];
|
|
browser -> searxng [label=" 4 POST /ai_summary\l (query + results)\l"];
|
|
searxng -> llm [label=" 5 POST /v1/chat/completions"];
|
|
llm -> searxng [label=" 6 SSE token stream", constraint=false];
|
|
searxng -> browser [label=" 7 NDJSON token stream", constraint=false];
|
|
}
|
|
|
|
Steps 1--3 are an ordinary SearXNG search. The plugin's
|
|
:py:obj:`post_search <searx.plugins.ai_summary.SXNGPlugin.post_search>` hook
|
|
adds an empty :py:obj:`searx.result_types.AiSummary` placeholder to the answer
|
|
area and returns immediately, so the page is not delayed.
|
|
|
|
Steps 4--7 happen in the browser after the page has painted.
|
|
``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary``
|
|
endpoint, which opens a streaming request to the LLM server and re-emits the
|
|
tokens as they arrive. The user sees the answer being written.
|
|
|
|
Two format changes happen along the way. The LLM server speaks `SSE`_
|
|
(``data: {...}`` lines, terminated by ``data: [DONE]``), because that is what
|
|
the OpenAI chat completions API specifies. SearXNG re-emits that to the
|
|
browser as `NDJSON`_ -- one JSON object per line, ``{"delta": "..."}`` for each
|
|
chunk of text and a final ``{"done": true}``. NDJSON is used because the
|
|
browser reads the body with ``fetch`` and a stream reader, where SSE's
|
|
``EventSource`` would be the wrong tool: ``EventSource`` cannot issue a POST.
|
|
|
|
|
|
When no summary is generated
|
|
============================
|
|
|
|
:py:obj:`post_search <searx.plugins.ai_summary.SXNGPlugin.post_search>` skips
|
|
the placeholder entirely for:
|
|
|
|
- page two and beyond -- a summary belongs with the first impression of a query
|
|
- anything but the *general* category
|
|
- non-HTML output formats (the JSON, CSV and RSS APIs)
|
|
- queries where an engine already produced an infobox (wikipedia, wikidata) or
|
|
an instant answer (e.g. ddg definitions)
|
|
- an empty query, or no LLM server configured
|
|
|
|
The infobox rule mirrors what the big engines do: if the query is a lookup of a
|
|
well known entity, that entity's own data is a better answer than a generated
|
|
paragraph.
|
|
|
|
|
|
Where the API key goes
|
|
======================
|
|
|
|
The administrator's ``api_key`` is only ever sent to the administrator's
|
|
``base_url``. This matters because users may set their own server in the
|
|
``ai_summary_server`` preference: without the check, any user of the instance
|
|
could point that preference at a host they control and collect the instance's
|
|
key from the ``Authorization`` header. Users authenticate to their own server
|
|
with their own ``ai_summary_api_key`` preference, which is stored in a cookie
|
|
and deliberately excluded from the shareable preferences URL.
|
|
|
|
:py:obj:`_server_api_key <searx.plugins.ai_summary._server_api_key>` implements
|
|
the rule; a URL carrying credentials in its userinfo is rejected outright,
|
|
because HTTP clients turn that into an ``Authorization`` header of the user's
|
|
choosing.
|
|
|
|
|
|
Implementation notes
|
|
====================
|
|
|
|
The ``/ai_summary`` route is registered in :py:obj:`searx.webapp`, next to the
|
|
favicon proxy, rather than in the plugin's ``init()``. Flask does not allow
|
|
``add_url_rule`` after the first request has been handled, and registering it
|
|
from a plugin breaks the test suite.
|
|
|
|
Requests to the LLM server bypass :py:obj:`searx.network` and are sent with a
|
|
plain :py:obj:`httpx.Client`. The outgoing proxy configuration is deliberately
|
|
not applied: an LLM server usually sits on localhost or in the local network,
|
|
which is exactly what an outgoing proxy is configured to avoid.
|
|
|
|
The streaming response uses ``direct_passthrough``, so the generator must yield
|
|
``bytes`` -- werkzeug asserts on ``str``, and the Flask test client does not
|
|
catch it.
|
|
|
|
.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
|
|
.. _NDJSON: https://github.com/ndjson/ndjson-spec
|
|
|
|
Reference
|
|
=========
|
|
|
|
.. automodule:: searx.plugins.ai_summary
|
|
:members:
|