[feat] plugin: AI Summary
PLACEHOLDER — rewrite this message before opening the PR. Adds a plugin that shows an AI generated summary above the search results, produced by an LLM server the administrator runs and reaches over the OpenAI chat completions API.
This commit is contained in:
jasonwitty
@@ -25,3 +25,4 @@ Settings
|
||||
settings_outgoing
|
||||
settings_categories_as_tabs
|
||||
settings_plugins
|
||||
settings_ai_summary
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
.. _settings ai_summary:
|
||||
|
||||
===============
|
||||
``ai_summary:``
|
||||
===============
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`ai_summary plugin`
|
||||
- :ref:`settings plugins`
|
||||
- :ref:`settings preferences`
|
||||
|
||||
Configuration of the :ref:`AI Summary plugin <ai_summary plugin>`, which shows a
|
||||
short AI generated answer above the search results.
|
||||
|
||||
The text is produced by an LLM server that you run; SearXNG ships no model and
|
||||
contacts no AI provider of its own. Any server implementing the `OpenAI chat
|
||||
completions API`_ works: `Ollama`_, vLLM, llama.cpp, LM Studio, Hugging Face
|
||||
TGI and others. The plugin is not activated by default.
|
||||
|
||||
.. _ai_summary quickstart:
|
||||
|
||||
Quickstart
|
||||
==========
|
||||
|
||||
A local setup on the same machine as SearXNG, in four steps.
|
||||
|
||||
**1. Install Ollama**
|
||||
|
||||
.. code:: sh
|
||||
|
||||
curl -fsSL https://ollama.com/install.sh | sh
|
||||
|
||||
**2. Download a model**
|
||||
|
||||
.. code:: sh
|
||||
|
||||
ollama pull gemma3:4b
|
||||
|
||||
``gemma3:4b`` needs roughly 4 GB of memory and runs on CPU if you have no GPU.
|
||||
On a smaller machine use ``gemma3:1b``; any model in the `Ollama library`_
|
||||
works.
|
||||
|
||||
**3. Configure SearXNG**
|
||||
|
||||
Add this to your ``settings.yml``:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
ai_summary:
|
||||
base_url: "http://127.0.0.1:11434"
|
||||
model: "gemma3:4b"
|
||||
|
||||
plugins:
|
||||
searx.plugins.ai_summary.SXNGPlugin:
|
||||
active: true
|
||||
# keep the plugins you already had, see the warning below
|
||||
searx.plugins.calculator.SXNGPlugin: {active: true}
|
||||
searx.plugins.hash_plugin.SXNGPlugin: {active: true}
|
||||
searx.plugins.self_info.SXNGPlugin: {active: true}
|
||||
searx.plugins.unit_converter.SXNGPlugin: {active: true}
|
||||
searx.plugins.ahmia_filter.SXNGPlugin: {active: true}
|
||||
searx.plugins.hostnames.SXNGPlugin: {active: true}
|
||||
searx.plugins.time_zone.SXNGPlugin: {active: true}
|
||||
searx.plugins.tracker_url_remover.SXNGPlugin: {active: true}
|
||||
searx.plugins.infinite_scroll.SXNGPlugin: {active: false}
|
||||
searx.plugins.oa_doi_rewrite.SXNGPlugin: {active: false}
|
||||
searx.plugins.tor_check.SXNGPlugin: {active: false}
|
||||
|
||||
.. warning::
|
||||
|
||||
A ``plugins:`` block **replaces** the default list, it is not merged into it
|
||||
(:ref:`settings plugins`). Listing only the AI Summary plugin switches
|
||||
every other plugin off, which is why the block above repeats the defaults --
|
||||
drop the lines for plugins you do not want.
|
||||
|
||||
**4. Restart SearXNG** and search for something.
|
||||
|
||||
The summary appears above the results while it is still being written.
|
||||
|
||||
Options
|
||||
=======
|
||||
|
||||
Only ``base_url`` is required. A model is needed too, but if ``model`` is left
|
||||
empty the first entry of ``models`` is used.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
ai_summary:
|
||||
base_url: "http://127.0.0.1:11434"
|
||||
model: "gemma3:4b"
|
||||
grounding: true
|
||||
|
||||
.. autoclass:: searx.ai_summary.SettingsAISummary
|
||||
:members:
|
||||
|
||||
Servers that require authentication
|
||||
===================================
|
||||
|
||||
vLLM and llama.cpp started with ``--api-key``, a gateway such as LiteLLM, or an
|
||||
LLM server behind an authenticating reverse proxy all expect a key. Set it
|
||||
with ``api_key``:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
ai_summary:
|
||||
base_url: "http://127.0.0.1:8000"
|
||||
api_key: "sk-..."
|
||||
model: "gemma3:4b"
|
||||
|
||||
The key is sent as an ``Authorization: Bearer`` header and only to the server in
|
||||
``base_url``. Users who configure a server of their own in the
|
||||
``ai_summary_server`` preference never receive it; they set their own key in the
|
||||
``ai_summary_api_key`` preference.
|
||||
|
||||
SearXNG has no separate secret store, so the key is held in ``settings.yml`` --
|
||||
that file should be readable only by the user SearXNG runs as.
|
||||
|
||||
.. _ai_summary grounding:
|
||||
|
||||
Grounding
|
||||
=========
|
||||
|
||||
With ``grounding`` enabled, which is the default, the query **and the top search
|
||||
results** (title, URL and snippet, at most ``max_context_items`` of them) are
|
||||
sent to the LLM server, and the answer reflects what the search found. With it
|
||||
disabled only the query is sent and the model answers from its training data.
|
||||
Users can change this in the ``ai_summary_grounding`` preference.
|
||||
|
||||
What leaves the network therefore depends on where the LLM server runs: with a
|
||||
server on localhost or in the local network, nothing does.
|
||||
|
||||
Public SearXNG instances
|
||||
========================
|
||||
|
||||
The server URL, model and API key are user preferences so that someone running
|
||||
SearXNG at home can switch models or debug their LLM server from the
|
||||
preferences page, without editing ``settings.yml`` and restarting.
|
||||
|
||||
On public SearXNG instances, those same preferences let any visitor choose the
|
||||
address the summary request is sent to. The request is made by the SearXNG
|
||||
host, so a visitor can use it to reach machines on your network that they have
|
||||
no route to themselves -- an `SSRF`_ vector.
|
||||
|
||||
.. attention::
|
||||
|
||||
Lock ``ai_summary_server`` on any SearXNG instance that is reachable by
|
||||
people outside your household.
|
||||
|
||||
Locking a preference makes SearXNG use your configured value and ignore the
|
||||
user's (:ref:`settings preferences`):
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
preferences:
|
||||
lock:
|
||||
- ai_summary_server
|
||||
- ai_summary_api_key
|
||||
- ai_summary_model
|
||||
- ai_summary_grounding
|
||||
|
||||
``ai_summary_server`` is the one that matters: locking it closes the SSRF
|
||||
vector. Locking ``ai_summary_api_key`` additionally stops visitors making your
|
||||
SearXNG instance send an ``Authorization`` header of their choosing to a host of
|
||||
their choosing. ``ai_summary_model`` and ``ai_summary_grounding`` are about
|
||||
cost and consistency rather than security.
|
||||
|
||||
.. _Ollama: https://ollama.com/
|
||||
.. _Ollama library: https://ollama.com/library
|
||||
.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat
|
||||
.. _SSRF: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
|
||||
@@ -0,0 +1,103 @@
|
||||
.. _ai_summary plugin:
|
||||
|
||||
==========
|
||||
AI Summary
|
||||
==========
|
||||
|
||||
.. sidebar:: Further reading ..
|
||||
|
||||
- :ref:`Configuration <settings ai_summary>`
|
||||
- :ref:`dev plugin`
|
||||
- :ref:`result types`
|
||||
|
||||
The AI Summary plugin shows a generated answer above the ordinary search
|
||||
results. The text comes from an LLM server run by the administrator, which
|
||||
speaks the `OpenAI chat completions API`_ -- `Ollama`_, Hugging Face TGI,
|
||||
LiteLLM, vLLM, llama.cpp and anything else implementing that specification.
|
||||
See :ref:`its configuration <settings ai_summary>` for how to set one up.
|
||||
|
||||
The summary is generated asynchronously: the result page is delivered without
|
||||
delay and carries an empty placeholder, which the browser fills from a second,
|
||||
streaming request.
|
||||
|
||||
Request flow
|
||||
============
|
||||
|
||||
.. _ai_summary dataflow:
|
||||
|
||||
.. kernel-render:: DOT
|
||||
:alt: Data flow between browser, SearXNG and the LLM server
|
||||
:caption: A search that produces a summary: two requests, not one
|
||||
|
||||
digraph ai_summary {
|
||||
rankdir=LR;
|
||||
graph [fontname="sans-serif", ranksep=1.1, nodesep=0.4];
|
||||
node [fontname="sans-serif", fontsize=11, shape=box, style="rounded,filled",
|
||||
fillcolor="#f4f4f4", color="#999999"];
|
||||
edge [fontname="sans-serif", fontsize=9, color="#666666"];
|
||||
|
||||
browser [label="browser"];
|
||||
searxng [label="SearXNG"];
|
||||
engines [label="search engines", fillcolor="#ffffff"];
|
||||
llm [label="LLM server\nOllama, vLLM, ...", fillcolor="#ffffff"];
|
||||
|
||||
browser -> searxng [label=" 1 GET /search"];
|
||||
searxng -> engines [label=" 2 query"];
|
||||
searxng -> browser [label=" 3 result page,\l empty summary box\l", constraint=false];
|
||||
browser -> searxng [label=" 4 POST /ai_summary\l (query + results)\l"];
|
||||
searxng -> llm [label=" 5 POST /v1/chat/completions"];
|
||||
llm -> searxng [label=" 6 SSE token stream", constraint=false];
|
||||
searxng -> browser [label=" 7 NDJSON token stream", constraint=false];
|
||||
}
|
||||
|
||||
Steps 1 to 3 are an ordinary SearXNG search. :py:obj:`SXNGPlugin.post_search
|
||||
<searx.plugins.ai_summary.SXNGPlugin.post_search>` adds an empty
|
||||
:py:obj:`searx.result_types.AiSummary` placeholder to the answer area and
|
||||
returns; the result page is not delayed.
|
||||
|
||||
Steps 4 to 7 run in the browser once the page is rendered.
|
||||
``client/simple/src/js/plugin/AiSummary.ts`` posts to the ``/ai_summary``
|
||||
endpoint (registered in :py:obj:`searx.webapp`), which opens a streaming
|
||||
request to the LLM server and re-emits the tokens as they arrive.
|
||||
|
||||
The two streams use different formats. The LLM server sends `SSE`_ --
|
||||
``data: {...}`` lines terminated by ``data: [DONE]``. SearXNG re-emits them to
|
||||
the browser as `NDJSON`_: one JSON object per line, ``{"delta": "..."}`` for
|
||||
each chunk of text and a final ``{"done": true}``.
|
||||
|
||||
When no summary is generated
|
||||
============================
|
||||
|
||||
:py:obj:`SXNGPlugin.post_search
|
||||
<searx.plugins.ai_summary.SXNGPlugin.post_search>` adds no placeholder for:
|
||||
|
||||
- page two and beyond
|
||||
- categories other than *general*
|
||||
- non-HTML output formats (the JSON, CSV and RSS APIs)
|
||||
- queries an engine already answered with an infobox (wikipedia, wikidata) or
|
||||
an instant answer (e.g. ddg definitions)
|
||||
- an empty query, or no LLM server configured
|
||||
|
||||
API keys
|
||||
========
|
||||
|
||||
The administrator's ``api_key`` is sent only to the configured ``base_url``.
|
||||
Users who point the ``ai_summary_server`` preference at a server of their own
|
||||
authenticate it with their own ``ai_summary_api_key`` preference, which is
|
||||
stored in a cookie and excluded from the preferences URL; the administrator's
|
||||
key is never sent to such a server. A server URL carrying credentials in its
|
||||
userinfo is ignored, and the administrator's default is used instead.
|
||||
|
||||
:py:obj:`_server_api_key <searx.plugins.ai_summary._server_api_key>` implements
|
||||
the rule.
|
||||
|
||||
Reference
|
||||
=========
|
||||
|
||||
.. automodule:: searx.plugins.ai_summary
|
||||
:members:
|
||||
|
||||
.. _Ollama: https://ollama.com/
|
||||
.. _OpenAI chat completions API: https://platform.openai.com/docs/api-reference/chat
|
||||
.. _SSE: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
|
||||
.. _NDJSON: https://github.com/ndjson/ndjson-spec
|
||||
@@ -7,6 +7,7 @@ Built-in Plugins
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
ai_summary
|
||||
calculator
|
||||
hash_plugin
|
||||
hostnames
|
||||
|
||||
Reference in New Issue
Block a user