jason/searxng
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[doc] ai_summary: describe the plugin, not the reasoning behind it

Browse Source 
Both pages carried design rationale and implementation notes that belong
in a review conversation rather than in documentation: why NDJSON is
sent to the browser instead of SSE, why the endpoint is registered in
webapp.py, why the outgoing proxy is bypassed. They are gone, along with
the troubleshooting section.

The section about public instances no longer opens by saying the plugin
is meant for private instances, which contradicted the instructions that
followed it. It now explains why the server, model and key are user
preferences at all -- so that they can be changed at home without
editing settings.yml -- and what that means once other people can reach
the instance.

"AI Summary" is the plugin, "AI summary" is the text it produces, and an
"instance" is always a SearXNG one; a page that also talks about LLM
servers cannot leave that to context.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
jasonwitty
2026-08-13 16:11:49 -07:00
parent 054b3a5c0d
commit 37d7d11959
2 changed files with 85 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/admin/settings/settings_ai_summary.rst
+47 -98
View File
@@ -13,11 +13,10 @@
Configuration of the :ref:`AI Summary plugin <ai_summary plugin>`, which shows a Configuration of the :ref:`AI Summary plugin <ai_summary plugin>`, which shows a
short AI generated answer above the search results. short AI generated answer above the search results.
The text is produced by an **LLM server that you run** -- SearXNG does not ship The text is produced by an LLM server that you run; SearXNG ships no model and
a model and does not talk to any AI provider on its own. Anything that speaks contacts no AI provider of its own. Any server implementing the `OpenAI chat
the `OpenAI chat completions API`_ works: `Ollama`_, vLLM, llama.cpp, LM Studio, completions API`_ works: `Ollama`_, vLLM, llama.cpp, LM Studio, Hugging Face
Hugging Face TGI and others. The plugin is not activated by default. TGI and others. The plugin is not activated by default.
.. _ai_summary quickstart: .. _ai_summary quickstart:
@@ -39,8 +38,8 @@ A local setup on the same machine as SearXNG, in four steps.
ollama pull gemma3:4b ollama pull gemma3:4b
``gemma3:4b`` needs roughly 4 GB of memory and runs on CPU if you have no GPU. ``gemma3:4b`` needs roughly 4 GB of memory and runs on CPU if you have no GPU.
On a small machine use ``gemma3:1b`` instead; any model in the `Ollama On a smaller machine use ``gemma3:1b``; any model in the `Ollama library`_
library`_ works. works.
**3. Configure SearXNG** **3. Configure SearXNG**
@@ -71,20 +70,18 @@ Add this to your ``settings.yml``:
.. warning:: .. warning::
A ``plugins:`` block **replaces** the default list, it is not merged into it A ``plugins:`` block **replaces** the default list, it is not merged into it
(:ref:`settings plugins`). If you list only the AI Summary plugin, every (:ref:`settings plugins`). Listing only the AI Summary plugin switches
other plugin is switched off. That is why the block above repeats the every other plugin off, which is why the block above repeats the defaults --
defaults -- drop the lines for plugins you do not want. drop the lines for plugins you do not want.
**4. Restart SearXNG** and search for something. **4. Restart SearXNG** and search for something.
The summary appears above the results while it is still being written. If The summary appears above the results while it is still being written.
nothing appears, see :ref:`ai_summary troubleshooting`.
Options Options
======= =======
Only ``base_url`` is required; a model is needed too, but if you leave ``model`` Only ``base_url`` is required. A model is needed too, but if ``model`` is left
empty the first entry of ``models`` is used. empty the first entry of ``models`` is used.
.. code:: yaml .. code:: yaml
@@ -97,13 +94,12 @@ empty the first entry of ``models`` is used.
.. autoclass:: searx.ai_summary.SettingsAISummary .. autoclass:: searx.ai_summary.SettingsAISummary
:members: :members:
Servers that require authentication
===================================
Servers that need a password vLLM and llama.cpp started with ``--api-key``, a gateway such as LiteLLM, or an
============================ LLM server behind an authenticating reverse proxy all expect a key. Set it
with ``api_key``:
Some servers require authentication -- vLLM and llama.cpp when started with
``--api-key``, a gateway such as LiteLLM, or any LLM server placed behind an
authenticating reverse proxy. Give SearXNG the key with ``api_key``:
.. code:: yaml .. code:: yaml
@@ -112,59 +108,47 @@ authenticating reverse proxy. Give SearXNG the key with ``api_key``:
api_key: "sk-..." api_key: "sk-..."
model: "gemma3:4b" model: "gemma3:4b"
The key is sent as an ``Authorization: Bearer`` header, and only to the server The key is sent as an ``Authorization: Bearer`` header and only to the server in
in ``base_url``. Users who point the ``ai_summary_server`` preference at a ``base_url``. Users who configure a server of their own in the
server of their own never receive it; they set their own key in the ``ai_summary_server`` preference never receive it; they set their own key in the
``ai_summary_api_key`` preference instead. ``ai_summary_api_key`` preference.
SearXNG has no separate secret store, so the key sits in ``settings.yml`` --
make sure that file is readable only by the user SearXNG runs as.
SearXNG has no separate secret store, so the key is held in ``settings.yml`` --
that file should be readable only by the user SearXNG runs as.
.. _ai_summary grounding: .. _ai_summary grounding:
Grounding Grounding
========= =========
Grounding decides *what the model is told*: With ``grounding`` enabled, which is the default, the query **and the top search
results** (title, URL and snippet, at most ``max_context_items`` of them) are
sent to the LLM server, and the answer reflects what the search found. With it
disabled only the query is sent and the model answers from its training data.
Users can change this in the ``ai_summary_grounding`` preference.
``grounding: true`` (the default) What leaves the network therefore depends on where the LLM server runs: with a
The query **and the top search results** (title, URL and snippet, at most server on localhost or in the local network, nothing does.
``max_context_items`` of them) are sent to the LLM server. Answers reflect
what the search actually found, so they are more accurate and more current.
``grounding: false`` Public SearXNG instances
Only the query is sent, and the model answers from its own training data. ========================
Faster and cheaper, but the answer can be outdated or invented.
Users can switch this in their preferences (``ai_summary_grounding``). The server URL, model and API key are user preferences so that someone running
SearXNG at home can switch models or debug their LLM server from the
preferences page, without editing ``settings.yml`` and restarting.
How much this matters for privacy depends entirely on where the LLM server runs. On public SearXNG instances, those same preferences let any visitor choose the
With a server on localhost or in your own network, nothing leaves that network. address the summary request is sent to. The request is made by the SearXNG
With a hosted server, the query and the result snippets are sent to the host, so a visitor can use it to reach machines on your network that they have
provider. This is the main reason a local server is recommended. no route to themselves -- an `SSRF`_ vector.
Public instances
================
.. attention:: .. attention::
The plugin is designed for private instances. Read this before enabling it Lock ``ai_summary_server`` on any SearXNG instance that is reachable by
on a public one. people outside your household.
Two things change on a public instance: Locking a preference makes SearXNG use your configured value and ignore the
user's (:ref:`settings preferences`):
**Every search costs real work.** A summary is a full LLM inference. Traffic
that was cheap to serve becomes expensive, on hardware you pay for.
**Users can choose the server SearXNG talks to.** The ``ai_summary_server``
preference makes your instance send requests to an address of the user's
choosing, which is a `SSRF`_ vector: it can be pointed at services inside your
network that are not reachable from outside.
Lock the preferences so that only your configuration is used
(:ref:`settings preferences`):
.. code:: yaml .. code:: yaml
@@ -175,46 +159,11 @@ Lock the preferences so that only your configuration is used
- ai_summary_model - ai_summary_model
- ai_summary_grounding - ai_summary_grounding
Locking ``ai_summary_server`` closes the SSRF vector; locking ``ai_summary_server`` is the one that matters: locking it closes the SSRF
``ai_summary_api_key`` stops users making your instance send an vector. Locking ``ai_summary_api_key`` additionally stops visitors making your
``Authorization`` header of their choosing to a host of their choosing. SearXNG instance send an ``Authorization`` header of their choosing to a host of
their choosing. ``ai_summary_model`` and ``ai_summary_grounding`` are about
cost and consistency rather than security.
.. _ai_summary troubleshooting:
Troubleshooting
===============
**No summary appears at all.**
The plugin deliberately stays quiet in several cases: on page two and beyond,
outside the *general* category, and when an engine already answered the query
with an infobox or an instant answer. Try a question-like query on the first
page. Check that the plugin is enabled both in ``settings.yml`` and in your
own preferences.
**The model list in the preferences is empty.**
SearXNG asks the server for its models (``GET /v1/models``) once at startup.
An empty list means that request failed -- the server was not running yet,
the URL is wrong, or it needs an ``api_key``. The log records the reason;
restart SearXNG after fixing it.
**The summary box shows an error.**
SearXNG could not reach the LLM server, or the server rejected the request.
Check ``base_url`` from the SearXNG machine, confirm the model name exists on
that server (``ollama list``), and check the SearXNG log -- it records the
status and the message the server replied with, which usually names the cause.
**The first search after a while fails, the next one works.**
An idle LLM server unloads the model and has to load it again, and it sends
nothing at all while doing so. If that takes longer than ``read_timeout``
(30 s by default) the request is abandoned. SearXNG repeats the request once,
which covers a normal load, but a large model on slow storage can need more:
raise ``read_timeout``, or keep the model in memory -- with Ollama, set
``OLLAMA_KEEP_ALIVE`` (for example ``-1`` to never unload it).
**The summary starts, then stops mid-sentence.**
The answer exceeded ``stream_timeout`` (120 s by default). Large models on
CPU are slow; either raise the limit or use a smaller model.
.. _Ollama: https://ollama.com/ .. _Ollama: https://ollama.com/
.. _Ollama library: https://ollama.com/library .. _Ollama library: https://ollama.com/library
docs/dev/plugins/ai_summary.rst
+38 -77
View File
@@ -11,23 +11,14 @@ AI Summary
- :ref:`result types` - :ref:`result types`
The AI Summary plugin shows a generated answer above the ordinary search The AI Summary plugin shows a generated answer above the ordinary search
results, unless an engine has already answered the query directly -- with a results. The text comes from an LLM server run by the administrator, which
Wikipedia infobox, for instance, or an instant answer. It is meant to run speaks the `OpenAI chat completions API`_ -- `Ollama`_, Hugging Face TGI,
against a local LLM server and speaks the `OpenAI chat completions API`_, so it LiteLLM, vLLM, llama.cpp and anything else implementing that specification.
works with `Ollama`_, Hugging Face TGI, LiteLLM, vLLM, llama.cpp and anything See :ref:`its configuration <settings ai_summary>` for how to set one up.
else that implements that specification. See :ref:`its configuration <settings
ai_summary>` for how to set one up.
The purpose of the summary is not to tell you what the model memorised during
training, but to summarise the up to date results your query actually returned.
That is what the *grounding* setting does, and why it is enabled by default.
Generating an answer takes seconds, and a search engine that waits seconds
before painting anything is a broken search engine. The summary is therefore
produced asynchronously: the plugin renders an empty box, the result page is
delivered immediately, and the browser fills that box from a second, streaming
request. The results below stay readable and scrollable the whole time.
The summary is generated asynchronously: the result page is delivered without
delay and carries an empty placeholder, which the browser fills from a second,
streaming request.
Request flow Request flow
============ ============
@@ -45,7 +36,7 @@ Request flow
fillcolor="#f4f4f4", color="#999999"]; fillcolor="#f4f4f4", color="#999999"];
edge [fontname="sans-serif", fontsize=9, color="#666666"]; edge [fontname="sans-serif", fontsize=9, color="#666666"];
browser [label="browser\n(simple theme)"]; browser [label="browser"];
searxng [label="SearXNG"]; searxng [label="SearXNG"];
engines [label="search engines", fillcolor="#ffffff"]; engines [label="search engines", fillcolor="#ffffff"];
llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"]; llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"];
@@ -59,84 +50,54 @@ Request flow
searxng -> browser [label=" 7 NDJSON token stream", constraint=false]; searxng -> browser [label=" 7 NDJSON token stream", constraint=false];
} }
Steps 1--3 are an ordinary SearXNG search. The plugin's Steps 1 to 3 are an ordinary SearXNG search. :py:obj:`SXNGPlugin.post_search
:py:obj:`post_search <searx.plugins.ai_summary.SXNGPlugin.post_search>` hook <searx.plugins.ai_summary.SXNGPlugin.post_search>` adds an empty
adds an empty :py:obj:`searx.result_types.AiSummary` placeholder to the answer :py:obj:`searx.result_types.AiSummary` placeholder to the answer area and
area and returns immediately, so the page is not delayed. returns; the result page is not delayed.
Steps 4--7 happen in the browser after the page has painted. Steps 4 to 7 run in the browser once the page is rendered.
``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary`` ``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary``
endpoint, which opens a streaming request to the LLM server and re-emits the endpoint (registered in :py:obj:`searx.webapp`), which opens a streaming
tokens as they arrive. The user sees the answer being written. request to the LLM server and re-emits the tokens as they arrive.
Two format changes happen along the way. The LLM server speaks `SSE`_
(``data: {...}`` lines, terminated by ``data: [DONE]``), because that is what
the OpenAI chat completions API specifies. SearXNG re-emits that to the
browser as `NDJSON`_ -- one JSON object per line, ``{"delta": "..."}`` for each
chunk of text and a final ``{"done": true}``. NDJSON is used because the
browser reads the body with ``fetch`` and a stream reader, where SSE's
``EventSource`` would be the wrong tool: ``EventSource`` cannot issue a POST.
The two streams use different formats. The LLM server sends `SSE`_ --
``data: {...}`` lines terminated by ``data: [DONE]``. SearXNG re-emits them to
the browser as `NDJSON`_: one JSON object per line, ``{"delta": "..."}`` for
each chunk of text and a final ``{"done": true}``.
When no summary is generated When no summary is generated
============================ ============================
:py:obj:`post_search <searx.plugins.ai_summary.SXNGPlugin.post_search>` skips :py:obj:`SXNGPlugin.post_search
the placeholder entirely for: <searx.plugins.ai_summary.SXNGPlugin.post_search>` adds no placeholder for:
- page two and beyond -- a summary belongs with the first impression of a query - page two and beyond
- anything but the *general* category - categories other than *general*
- non-HTML output formats (the JSON, CSV and RSS APIs) - non-HTML output formats (the JSON, CSV and RSS APIs)
- queries where an engine already produced an infobox (wikipedia, wikidata) or - queries an engine already answered with an infobox (wikipedia, wikidata) or
an instant answer (e.g. ddg definitions) an instant answer (e.g. ddg definitions)
- an empty query, or no LLM server configured - an empty query, or no LLM server configured
The infobox rule mirrors what the big engines do: if the query is a lookup of a API keys
well known entity, that entity's own data is a better answer than a generated ========
paragraph.
The administrator's ``api_key`` is sent only to the configured ``base_url``.
Where the API key goes Users who point the ``ai_summary_server`` preference at a server of their own
====================== authenticate it with their own ``ai_summary_api_key`` preference, which is
stored in a cookie and excluded from the preferences URL; the administrator's
The administrator's ``api_key`` is only ever sent to the administrator's key is never sent to such a server. A server URL carrying credentials in its
``base_url``. This matters because users may set their own server in the userinfo is ignored, and the administrator's default is used instead.
``ai_summary_server`` preference: without the check, any user of the instance
could point that preference at a host they control and collect the instance's
key from the ``Authorization`` header. Users authenticate to their own server
with their own ``ai_summary_api_key`` preference, which is stored in a cookie
and deliberately excluded from the shareable preferences URL.
:py:obj:`_server_api_key <searx.plugins.ai_summary._server_api_key>` implements :py:obj:`_server_api_key <searx.plugins.ai_summary._server_api_key>` implements
the rule; a URL carrying credentials in its userinfo is rejected outright, the rule.
because HTTP clients turn that into an ``Authorization`` header of the user's
choosing.
Implementation notes
====================
The ``/ai_summary`` route is registered in :py:obj:`searx.webapp`, next to the
favicon proxy, rather than in the plugin's ``init()``. Flask does not allow
``add_url_rule`` after the first request has been handled, and registering it
from a plugin breaks the test suite.
Requests to the LLM server bypass :py:obj:`searx.network` and are sent with a
plain :py:obj:`httpx.Client`. The outgoing proxy configuration is deliberately
not applied: an LLM server usually sits on localhost or in the local network,
which is exactly what an outgoing proxy is configured to avoid.
The streaming response uses ``direct_passthrough``, so the generator must yield
``bytes`` -- werkzeug asserts on ``str``, and the Flask test client does not
catch it.
.. _Ollama: https://ollama.com/
.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat
.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
.. _NDJSON: https://github.com/ndjson/ndjson-spec
Reference Reference
========= =========
.. automodule:: searx.plugins.ai_summary .. automodule:: searx.plugins.ai_summary
:members: :members:
.. _Ollama: https://ollama.com/
.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat
.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
.. _NDJSON: https://github.com/ndjson/ndjson-spec