diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..8465c25 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,48 @@ +name: CI + +on: + push: + branches: [master] + pull_request: + +permissions: + contents: read + +jobs: + check: + name: Lint, types, tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: astral-sh/setup-uv@v6 + with: + # Same uv as the Dockerfile, so CI resolves what production builds + version: "0.10.11" + enable-cache: true + + - name: Install + run: uv sync --locked --dev + + - name: Format + run: uv run ruff format --check . + + - name: Lint + run: uv run ruff check . + + - name: Type check + run: uv run ty check + + - name: Script syntax + run: node --check static/script.js + + - name: Tests + run: uv run pytest --cov=app --cov-report=term-missing --cov-fail-under=95 + + docker: + name: Docker image builds + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build + run: docker build --tag iris:ci . diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..0949974 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,21 @@ +# Runs the same checks as CI, with the tool versions pinned in uv.lock. +# Install once with: uvx pre-commit install +repos: + - repo: local + hooks: + - id: ruff-format + name: ruff format + entry: uv run ruff format + language: system + types: [python] + - id: ruff-check + name: ruff check + entry: uv run ruff check --fix + language: system + types: [python] + - id: ty + name: ty check + entry: uv run ty check + language: system + types: [python] + pass_filenames: false diff --git a/DESIGN.md b/DESIGN.md index b9e5571..4dc268b 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -216,10 +216,10 @@ The page is flat. Depth is conveyed by tone (Raised Ground for chips) and by the ### Shadow Vocabulary - **Focus ring** (`box-shadow: 0 0 0 2px #0d1117, 0 0 0 4px #58a6ff`): every `:focus-visible`; inputs instead swap their underline to blue with a 1px blue under-shadow. - **Knock-out ring** (`box-shadow: 0 0 0 3px #0d1117`): step marks, so the rail appears to pass behind them. -- **Tooltip lift** (`box-shadow: 0 8px 24px rgba(1, 4, 9, 0.6)`): the graph tooltip only. +- **Tooltip lift** (`box-shadow: 0 8px 24px rgba(1, 4, 9, 0.6)`): the graph tooltip and the title suggestion list, the two surfaces that float over other content. ### Named Rules -**The Flat Ground Rule.** Nothing on the page casts a shadow except the pointer tooltip. Do not lift sections, buttons or chips. +**The Flat Ground Rule.** Nothing on the page casts a shadow except what floats above it: the pointer tooltip and the suggestion list. Do not lift sections, buttons or chips. ## Shapes @@ -242,6 +242,12 @@ Restrained and exact: one filled action, everything else outlined or bare. - **Hover:** underline lifts to Faint Ink. - **Error:** `aria-invalid="true"` turns the underline To Red with the same 1px under-shadow. +### Suggestions +A listbox hung from the bottom of a route field while typing (ARIA combobox; the input keeps focus). Raised Ground with a 1px Strong Hairline border, no top border so it reads as the field's continuation, square corners and the tooltip lift. Each option is a 44px row ruled by a Hairline: the title at Body 500, the Wikipedia short description below in Label Faint Ink. The active option takes Sunken Fill and a Path Blue title. A disambiguation page in the To field is muted and says to pick a specific article. + +### Examples +Under the route actions, one Small line: "Try" in Faint Ink, three example routes as Path Blue in-text actions (shuffled per visit), and "Surprise me" in Muted Ink. Each is a 44px target. + ### Notice The error and "stopped" message, ruled not boxed. A three-column row: 20px stroked icon in the tone colour, body, actions. A tone rule on top (the tone at 55% via `color-mix`), a Hairline below, 16px vertical padding, no fill, no radius. The title names the problem (To Red Ink, 600), the message adds a new fact (Ink, Small), the hint gives recovery (Muted Ink, Small); empty lines collapse. The neutral variant swaps the tone to Faint Ink and the title to Ink. It enters with a 4px drop-in over 0.35s. @@ -252,13 +258,18 @@ The masthead is a 64px bar ruled below: the wordmark (24px logo plus "Iris" at T The signature of the instrument: a row of figures separated by vertical Hairlines, no boxes. Four equal columns plus a double-width "Reading" column holding the page currently being read as a Path Blue link (Faint Ink and inert when empty). Labels are 12px Faint Ink; values are Reading type. Depth is shown as a row of 8px circles: hollow (Strong Hairline), active (blue outline, beating), completed (blue fill). Above the readings a 1px activity track carries a blue sweep while searching. ### The Field (graph plot) -A dot field (1px dots at 16% grey on a 20px grid) framed by corner registration marks, 360px minimum height (300px on phones). While searching it shows only the two ends as label chips joined by a dashed 1px line with a travelling blue highlight; when the path arrives the D3 graph replaces it. Nodes are blue with a 2px ground stroke; the start and end nodes take their signal colour with a translucent 5px halo of the same hue. Path edges are 2.5px Path Blue with a single SVG arrowhead; other edges are 2px Strong Hairline at 60%. Label chips are Raised Ground with a 1px Strong Hairline border and 4px corners. +A dot field (1px dots at 16% grey on a 20px grid) framed by corner registration marks, 360px minimum height (300px on phones). While searching it shows the two frontiers: each end as a label chip, under it the pages that side read last (Label size, newest first in Ink, older ones fading to 26%), and a count ("214 pages · 2 links out") in Faint Ink. Between them a dashed 1px line carries a travelling blue highlight, and each side fills the line with solid Path Blue from its own end as it goes deeper, scaled to the whole depth budget so the two fills meet mid-line. Under 640px the scene stacks: start on top, end at the bottom reading upward, three pages per side. When the path arrives the D3 graph replaces it. Nodes are blue with a 2px ground stroke; the start and end nodes take their signal colour with a translucent 5px halo of the same hue. Path edges are 2.5px Path Blue with a single SVG arrowhead; other edges are 2px Strong Hairline at 60%. Label chips are Raised Ground with a 1px Strong Hairline border and 4px corners. The force simulation is tuned and fixed: link distance spans the available axis with strength 1; charge -240 (horizontal) or -160 (vertical); x/y position forces 0.22/0.10 (horizontal) or 0.16/0.22 (vertical); collision strength 0.8; alphaDecay 0.03; velocityDecay 0.45. These values are part of the system, not tuning knobs. ### Route List The found route as a ruled ordered list: each step is a 52px row ruled on top, with a 9px mark, a 36px index column, the title and a stroked open-arrow. A 1px blue rail at 45% opacity threads the marks, starting at the first mark and ending at the last; first mark green, last red, the rest blue. Hover turns the title Path Blue and nudges the arrow 2px up-right. +Above the list, the "Route" title shares its row with a "Copy link" text button (the system share sheet on touch devices). Under the summary, one Small Muted sentence explains the search: pages read on each side and where the frontiers met; that step carries a Faint "Frontiers met" tag. Under every step but the last hangs the sentence from that article which links to the next, as a quotation in Small Muted Ink aligned with the titles, the linked words in Path Blue and the section name in Faint Ink. A link found only in an infobox or navigation box says so in Faint Ink instead. + +### 404 +The same shell, masthead and colophon. The lead carries "404" at Reading size in To Red Ink, the display line "No page here.", one line of copy and the primary button back to the search. + ## Do's and Don'ts ### Do: diff --git a/Dockerfile b/Dockerfile index a521873..b61d255 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,7 +1,7 @@ FROM python:3.13-slim -# Install uv -COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ +# Pinned so a uv release can never change a build that the repo did not +COPY --from=ghcr.io/astral-sh/uv:0.10.11 /uv /uvx /bin/ WORKDIR /app @@ -9,15 +9,11 @@ WORKDIR /app COPY pyproject.toml uv.lock ./ RUN uv sync --frozen --no-dev -# Copy application code +# Copy application code (entrypoint.sh is executable in git) COPY . . -# Copy and set up entrypoint -COPY entrypoint.sh /entrypoint.sh -RUN chmod +x /entrypoint.sh - # Make the venv's binaries available (gunicorn, celery, etc.) ENV PATH="/app/.venv/bin:$PATH" # SERVICE_TYPE=worker runs celery, anything else runs gunicorn -CMD ["/entrypoint.sh"] +CMD ["/app/entrypoint.sh"] diff --git a/PRODUCT.md b/PRODUCT.md index 53d1cee..5ff3655 100644 --- a/PRODUCT.md +++ b/PRODUCT.md @@ -24,7 +24,7 @@ The implementation is itself part of the point. The project stands as a demonstr ## Operating Context -- Single page, no accounts, no persistence beyond the browser. A visitor types two titles and presses one button. +- Single page, no accounts. A visitor types two titles (with suggestions) or picks an example, and presses one button. A found route is kept for thirty days so a shared link can show it. - Searches run as background Celery tasks and are polled by the UI; a search survives a page refresh because the latest one is kept in `localStorage`. - Deployed publicly at `iris.hishaam.dev` (Railway, Docker; `entrypoint.sh` switches role on `SERVICE_TYPE`). It is live but not actively promoted, so traffic is incidental and stakes are low — strangers can still arrive cold with no context. - A run takes seconds to a couple of minutes. Waiting is a normal part of the experience, not an error state. @@ -46,7 +46,7 @@ The implementation is itself part of the point. The project stands as a demonstr ## Brand Commitments - Name: **Iris**; full title "Iris: Wikipedia Path Finder". -- Existing identity assets, all in `static/`: `logo.svg`, `favicon-dark.svg` / `favicon-light.svg` (switched on `prefers-color-scheme`), `og-image.svg` / `og-image.png`. +- Existing identity assets, all in `static/`: `logo.svg`, `favicon.svg` (its lines follow `prefers-color-scheme` from inside the SVG), `favicon.ico`, `apple-touch-icon.png`, `icon-192.png` / `icon-512.png` (web manifest), `og-image.svg` / `og-image.png`. - The GitHub-dark palette and JetBrains Mono are binding: the README and OG image are built on them, so a change to the theme is a change to published assets. - GPL-3.0 licensed (see `LICENSE`), open source at `github.com/mdhishaamakhtar/iris`. - Built by Md Hishaam Akhtar and Sharanya Mukherjee under DSC VIT; the README carries that attribution. diff --git a/README.md b/README.md index 4597527..a88d67b 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ ## Iris Wikipedia Pathfinder +[![CI](https://github.com/mdhishaamakhtar/iris/actions/workflows/ci.yml/badge.svg)](https://github.com/mdhishaamakhtar/iris/actions/workflows/ci.yml) + Iris finds a path between any two Wikipedia pages by traversing their links. You give it two page titles — say, "Banana" and "Shah Rukh Khan" — and it walks Wikipedia's link graph using BFS until it finds the connection. Results are visualized as an interactive graph you can drag around. @@ -21,9 +23,77 @@ It uses **bidirectional BFS** (default) which searches simultaneously from both ## What It Does - Find Wikipedia paths via **bidirectional BFS** (default) or standard forward-only BFS -- Async task processing — searches run in the background, results polled live -- Real-time progress updates during search (aggregated from both search frontiers) -- Interactive D3.js graph visualization of the path +- **Watch both frontiers grow** — each side shows the pages it is reading and how deep it has gone, streamed over Server-Sent Events +- **Every step explained** — the route quotes the sentence in each article that links to the next one +- **Shareable routes** — `/?from=Banana&to=Shah%20Rukh%20Khan` opens (or runs) that search, and unfurls in Slack, Discord or a social feed as an image of the route +- **Title suggestions** as you type, with disambiguation pages flagged before they fail +- Interactive D3.js graph of the path, and a ruled route list + +## How It Works + +```mermaid +flowchart LR + B[Browser] -- "POST /getPath" --> W[Flask / Gunicorn] + W -- "queue task" --> R[(Redis)] + R -- "task" --> C[Celery worker] + C -- "links + backlinks" --> WP[Wikipedia API] + C -- "frontiers, progress, caches" --> R + W -- "GET /tasks/stream/id (SSE)" --> B + R -- "task state" --> W +``` + +The web process never searches. It queues a Celery task and streams the task's +state back to the browser. The worker runs one breadth-first search over one +or two *frontiers*: forward from the start page through its links, and backward +from the end page through its backlinks, always expanding whichever side is +smaller. When a page turns up in both, the path is read back out of the parent +pointers. Every piece of search state — queues, visited sets, parent pointers — +lives in Redis, namespaced per search and deleted when it ends. + +Link lists are cached in Redis. Hub articles (500+ links: countries, years, +broad topics) are cached for a week rather than a day, because nearly every +search passes through them. + +The route found is the first one, not a proven shortest: Wikipedia caps how +many links one request returns, and Iris follows at most three continuations +per article. + +## Performance + +Search time is Wikipedia's request latency times the number of requests in +flight, not computation. Measured with `scripts/benchmark.py` on 24 October +2026: bidirectional search, production settings, a cold cache, one run each, +from a developer laptop. + +| Route | Time | Wikipedia requests | Pages explored | Path | +| --- | ---: | ---: | ---: | --- | +| Pizza → Byzantine Empire | 0.97 s | 2 | 1 | 2 pages | +| Mount Everest → Jazz | 2.79 s | 6 | 2 | 3 pages | +| Kangaroo → Fourier transform | 5.44 s | 33 | 22 | 4 pages | +| Pikachu → Quantum chromodynamics | 5.93 s | 37 | 22 | 4 pages | +| Taylor Swift → Photosynthesis | 7.37 s | 40 | 22 | 4 pages | +| Barack Obama → Mathematics | 8.70 s | 54 | 22 | 4 pages | + +A repeated search is answered from the path cache in milliseconds. Run +`uv run python scripts/benchmark.py` (needs local Redis and network) to +measure your own; `--compare` prints before/after. + +## API + +| Method | Path | Purpose | +| --- | --- | --- | +| `POST` | `/getPath` | Start a search: `{start, end, algorithm?}` → `202` with `task_id` | +| `GET` | `/tasks/stream/` | Progress and result as Server-Sent Events | +| `GET` | `/tasks/status/` | The same status body, polled | +| `DELETE` | `/tasks/` | Stop a search | +| `GET` | `/api/suggest?q=` | Title suggestions | +| `GET` | `/api/context?from=&to=` | The sentence in one article that links to the next | +| `GET` | `/api/og.png?from=&to=` | Share image for a route | +| `GET` | `/health` | Redis connectivity | + +Full schemas are in Swagger at `/api/docs`. The API is public and +unauthenticated, so it is rate-limited per client: 10 searches a minute, 120 +requests a minute to the lighter endpoints (`429` with `Retry-After` beyond). ## Tech Stack @@ -58,7 +128,7 @@ Swagger docs at [http://localhost:9020/api/docs](http://localhost:9020/api/docs) | Script | When to use | |--------|-------------| | `dev.sh` | Local development — starts Redis (if needed), Flask, and Celery in one terminal | -| `entrypoint.sh` | Docker/Railway container entry — switches on `SERVICE_TYPE` env var | +| `entrypoint.sh` | The Docker image's `CMD` — runs Gunicorn, or Celery when `SERVICE_TYPE=worker` | ## Dev @@ -69,8 +139,13 @@ uv run pytest --cov=app --cov-report=html # generate HTML coverage re uv run ruff format . # format uv run ruff check . # lint uv run ty check # type check +uvx pre-commit install # run format, lint and types on every commit ``` +CI runs the same checks on every push and pull request, requires 95% test +coverage, and builds the Docker image. Tests run against `fakeredis` with only +Wikipedia stubbed, so the search, the Redis store and the HTTP API run for real. + ## Contributors - **Md Hishaam Akhtar** — [GitHub](https://github.com/mdhishaamakhtar) · [LinkedIn](https://www.linkedin.com/in/md-hishaam-akhtar-812a3019a/) diff --git a/app/__init__.py b/app/__init__.py index 05ef26c..2628c97 100644 --- a/app/__init__.py +++ b/app/__init__.py @@ -18,6 +18,7 @@ from flask import Flask, Response, current_app, g, jsonify, request from marshmallow import ValidationError from werkzeug.exceptions import HTTPException +from werkzeug.middleware.proxy_fix import ProxyFix from app.config import Settings from app.errors import ErrorCode, IrisError @@ -77,6 +78,10 @@ def create_app(settings: Settings | None = None) -> Flask: settings = settings or Settings.from_env() app = Flask(__name__, static_folder="../static", static_url_path="/static") + # Railway terminates TLS in front of Gunicorn. Trusting one proxy hop is + # what makes request.remote_addr the client (for rate limits) and + # request.url_root https (for absolute share-preview URLs). + app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1) # type: ignore[method-assign] app.config.update( SECRET_KEY=settings.secret_key, DEBUG=settings.debug, @@ -119,6 +124,8 @@ def finish_request(response: Response) -> Response: response.headers["Access-Control-Allow-Origin"] = "*" response.headers["Access-Control-Allow-Methods"] = "GET, POST, DELETE, OPTIONS" response.headers["Access-Control-Allow-Headers"] = "Content-Type" + response.headers["X-Content-Type-Options"] = "nosniff" + response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin" if request.path.startswith(("/static/", "/flasgger_static/")): return response logger.info("request_completed", extra={"http_status": response.status_code}) @@ -150,7 +157,9 @@ def on_iris_error(exc: IrisError): logger.warning( "application_error", extra={"code": exc.code, "error": exc.message} ) - return problem(exc.message, exc.code, exc.status) + response, status = problem(exc.message, exc.code, exc.status) + response.headers.update(exc.headers) + return response, status @app.errorhandler(HTTPException) def on_http_error(exc: HTTPException): diff --git a/app/config.py b/app/config.py index 01f979c..53980af 100644 --- a/app/config.py +++ b/app/config.py @@ -69,10 +69,36 @@ class Settings: # Cache lifetimes, in seconds links_cache_ttl: int = 86_400 + hub_cache_ttl: int = 604_800 + """Link lists that reach ``hub_link_threshold`` belong to hub articles + (countries, years, broad topics). Nearly every search passes through them + and their link sets barely move, so they are kept a week, not a day.""" + hub_link_threshold: int = 500 path_cache_ttl: int = 3_600 page_cache_ttl: int = 7_200 search_state_ttl: int = 3_600 """Safety net so per-search Redis keys expire even if cleanup is skipped.""" + share_ttl: int = 2_592_000 + """How long a found route stays available to shared links and their + preview images: thirty days, long after the path cache has let it go.""" + context_cache_ttl: int = 604_800 + suggest_cache_ttl: int = 86_400 + + # Rate limits, per client IP per window + search_rate_limit: int = 10 + """Searches started per window. Each one fans out to hundreds of + Wikipedia requests, so this is what keeps one script from getting the + whole service throttled.""" + api_rate_limit: int = 120 + """Requests per window to the lighter endpoints (suggestions, link + context, progress streams, share images).""" + rate_window: int = 60 + + # Progress streaming + stream_interval: float = 0.5 + stream_max_seconds: int = 120 + """A stream closes after this long and the browser reconnects, so a + stream for a task that never starts cannot hold a thread forever.""" # Pathfinding max_search_depth: int = 6 diff --git a/app/errors.py b/app/errors.py index c4c9f08..82dbd81 100644 --- a/app/errors.py +++ b/app/errors.py @@ -19,6 +19,7 @@ class ErrorCode(StrEnum): PAGE_NOT_FOUND = "PAGE_NOT_FOUND" DISAMBIGUATION_PAGE = "DISAMBIGUATION_PAGE" PATH_NOT_FOUND = "PATH_NOT_FOUND" + RATE_LIMITED = "RATE_LIMITED" WIKIPEDIA_API_ERROR = "WIKIPEDIA_API_ERROR" CONFIGURATION_ERROR = "CONFIGURATION_ERROR" INTERNAL_ERROR = "INTERNAL_ERROR" @@ -36,6 +37,11 @@ def __init__(self, message: str, code: ErrorCode | None = None) -> None: if code is not None: self.code = code + @property + def headers(self) -> dict[str, str]: + """Extra response headers the error needs, such as ``Retry-After``.""" + return {} + class InvalidRequestError(IrisError): code = ErrorCode.INVALID_REQUEST @@ -52,6 +58,19 @@ class ConflictError(IrisError): status = 409 +class RateLimitedError(IrisError): + code = ErrorCode.RATE_LIMITED + status = 429 + + def __init__(self, retry_after: int) -> None: + super().__init__(f"Too many requests. Try again in {retry_after} seconds.") + self.retry_after = retry_after + + @property + def headers(self) -> dict[str, str]: + return {"Retry-After": str(self.retry_after)} + + class PageNotFoundError(IrisError): code = ErrorCode.PAGE_NOT_FOUND status = 404 diff --git a/app/fonts/JetBrainsMono-Regular.ttf b/app/fonts/JetBrainsMono-Regular.ttf new file mode 100644 index 0000000..dff66cc Binary files /dev/null and b/app/fonts/JetBrainsMono-Regular.ttf differ diff --git a/app/fonts/JetBrainsMono-SemiBold.ttf b/app/fonts/JetBrainsMono-SemiBold.ttf new file mode 100644 index 0000000..a70e69b Binary files /dev/null and b/app/fonts/JetBrainsMono-SemiBold.ttf differ diff --git a/app/fonts/OFL.txt b/app/fonts/OFL.txt new file mode 100644 index 0000000..8bee414 --- /dev/null +++ b/app/fonts/OFL.txt @@ -0,0 +1,93 @@ +Copyright 2020 The JetBrains Mono Project Authors (https://github.com/JetBrains/JetBrainsMono) + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +https://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/app/pathfinding.py b/app/pathfinding.py index a0760f0..b9ce935 100644 --- a/app/pathfinding.py +++ b/app/pathfinding.py @@ -41,6 +41,11 @@ FetchLinks = Callable[[list[str], PageFetched | None], dict[str, list[str]]] +SideStats = dict[str, dict[str, int]] +"""Per frontier ("forward", "backward"): pages ``explored`` so far, and the +``depth`` reached, in links out from that side's own end page.""" + + @dataclass(frozen=True, slots=True) class Progress: """A snapshot of an in-flight search, sent to the client as it runs.""" @@ -50,6 +55,9 @@ class Progress: current_depth: int last_node: str elapsed: float + side: str + """The frontier ``last_node`` was read by: "forward" or "backward".""" + sides: SideStats @dataclass(frozen=True, slots=True) @@ -57,6 +65,10 @@ class PathResult: path: list[str] nodes_explored: int search_time: float + # Defaulted so results cached before these fields existed still load. + meeting: str = "" + """The page where the two frontiers touched. In plain BFS, the target.""" + sides: SideStats = field(default_factory=dict) @property def length(self) -> int: @@ -87,9 +99,10 @@ class _Counters: lock: threading.Lock = field(default_factory=threading.Lock) nodes_explored: int = 0 depth_by_side: dict[str, int] = field(default_factory=dict) + explored_by_side: dict[str, int] = field(default_factory=dict) - def record(self, side: str, depth: int) -> tuple[int, int]: - """Count one fetched page; return (total explored, combined depth). + def record(self, side: str, depth: int) -> tuple[int, int, SideStats]: + """Count one fetched page; return (total explored, combined depth, sides). Combined depth sums both sides because a bidirectional search meeting at forward depth 2 and backward depth 2 has covered four hops of path. @@ -97,7 +110,19 @@ def record(self, side: str, depth: int) -> tuple[int, int]: with self.lock: self.nodes_explored += 1 self.depth_by_side[side] = max(self.depth_by_side.get(side, 0), depth) - return self.nodes_explored, sum(self.depth_by_side.values()) + self.explored_by_side[side] = self.explored_by_side.get(side, 0) + 1 + return ( + self.nodes_explored, + sum(self.depth_by_side.values()), + self.sides(), + ) + + def sides(self) -> SideStats: + # Expanding a page at depth d reaches pages d + 1 links out. + return { + side: {"explored": explored, "depth": self.depth_by_side[side] + 1} + for side, explored in self.explored_by_side.items() + } class PathFinder: @@ -127,7 +152,9 @@ def find( if not start or not end: raise InvalidRequestError("Start and end pages are required") if start == end: - return PathResult(path=[start], nodes_explored=1, search_time=0.0) + return PathResult( + path=[start], nodes_explored=1, search_time=0.0, meeting=start + ) started = time.monotonic() counters = _Counters() @@ -154,6 +181,8 @@ def find( path=path, nodes_explored=counters.nodes_explored, search_time=time.monotonic() - started, + meeting=meeting, + sides=counters.sides(), ) finally: self.store.delete(*all_keys) @@ -327,7 +356,7 @@ def _page_reporter( ) def page_fetched(title: str, _titles: list[str]) -> None: - explored, combined_depth = counters.record(side.name, depth) + explored, combined_depth, sides = counters.record(side.name, depth) if report is None: return report( @@ -337,6 +366,8 @@ def page_fetched(title: str, _titles: list[str]) -> None: current_depth=combined_depth, last_node=title, elapsed=round(time.monotonic() - started, 2), + side=side.name, + sides=sides, ) ) diff --git a/app/routes.py b/app/routes.py index 217cca1..b2cd8a2 100644 --- a/app/routes.py +++ b/app/routes.py @@ -7,8 +7,13 @@ from __future__ import annotations +import json import logging +import time import uuid +from collections.abc import Iterator +from dataclasses import asdict +from pathlib import Path from typing import Any from celery import states @@ -17,21 +22,32 @@ Response, current_app, jsonify, - redirect, request, url_for, ) from marshmallow import Schema, ValidationError, fields, validate, validates_schema from app import celery, services -from app.errors import ConflictError, ErrorCode, NotFoundError +from app.errors import ( + ConflictError, + ErrorCode, + InvalidRequestError, + NotFoundError, + RateLimitedError, +) from app.pathfinding import ALGORITHMS, BIDIRECTIONAL -from app.tasks import PROGRESS, find_path_task +from app.share import describe, render_card, with_share_meta +from app.tasks import PROGRESS, find_path_task, route_key logger = logging.getLogger(__name__) api = Blueprint("api", __name__) +MAX_TITLE = 255 +TERMINAL = (states.SUCCESS, states.FAILURE, states.REVOKED) +KEEPALIVE_SECONDS = 15 +"""An idle stream sends a comment this often, so proxies do not close it.""" + class SearchSchema(Schema): """Validates POST /getPath.""" @@ -83,7 +99,10 @@ def start_search() -> tuple[Response, int]: description: Search accepted; poll poll_url for progress and the result 400: description: Validation error + 429: + description: Too many searches from this address; see Retry-After """ + _rate_limit("search", services().settings.search_rate_limit) data = SearchSchema().load(request.get_json(silent=True) or {}) start, end = data["start"].strip(), data["end"].strip() @@ -122,8 +141,66 @@ def search_status(task_id: str) -> Response: description: Malformed task ID """ _require_task_id(task_id) + return jsonify(_status(task_id)) + + +@api.get("/tasks/stream/") +def stream_status(task_id: str) -> Response: + """Stream a search's progress as Server-Sent Events. + --- + tags: [Pathfinding] + summary: Stream task status + description: > + Each event's data is the same JSON body GET /tasks/status/{task_id} + returns, sent whenever it changes. The stream ends after a terminal + status (SUCCESS, FAILURE or REVOKED), or after a couple of minutes, when + an EventSource reconnects by itself. + produces: [text/event-stream] + parameters: + - name: task_id + in: path + required: true + type: string + responses: + 200: + description: An event stream of status bodies + 404: + description: Malformed task ID + 429: + description: Too many requests from this address + """ + _require_task_id(task_id) + _rate_limit("api", services().settings.api_rate_limit) + settings = services().settings + + def events() -> Iterator[str]: + yield f"retry: {int(settings.stream_interval * 2000)}\n\n" + last: dict[str, Any] | None = None + quiet = 0.0 + deadline = time.monotonic() + settings.stream_max_seconds + while True: + body = _status(task_id) + if body != last: + yield f"data: {json.dumps(body)}\n\n" + last, quiet = body, 0.0 + elif quiet >= KEEPALIVE_SECONDS: + yield ": keep-alive\n\n" + quiet = 0.0 + if body["status"] in TERMINAL or time.monotonic() >= deadline: + return + time.sleep(settings.stream_interval) + quiet += settings.stream_interval + + return Response( + events(), + mimetype="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) + + +def _status(task_id: str) -> dict[str, Any]: task = find_path_task.AsyncResult(task_id) - body: dict[str, object] = {"task_id": task_id} + body: dict[str, Any] = {"task_id": task_id} if task.state == states.PENDING: body |= {"status": states.PENDING, "message": "Task is waiting to be processed"} @@ -135,8 +212,7 @@ def search_status(task_id: str) -> Response: body |= {"status": states.FAILURE, "error": str(task.info)} else: body |= {"status": task.state} - - return jsonify(body) + return body def _result(payload: dict[str, Any]) -> dict[str, Any]: @@ -223,6 +299,109 @@ def health() -> tuple[Response, int]: ) +# --- Search helpers ------------------------------------------------------- + + +@api.get("/api/suggest") +def suggest() -> Response: + """Suggest article titles for what has been typed so far. + --- + tags: [Pathfinding] + summary: Title suggestions + parameters: + - name: q + in: query + required: true + type: string + description: The start of a title + responses: + 200: + description: > + Up to eight articles, best match first. Redirects are resolved; + disambiguation pages are flagged, since they cannot be an end page. + 429: + description: Too many requests from this address + """ + _rate_limit("api", services().settings.api_rate_limit) + query = request.args.get("q", "").strip()[:MAX_TITLE] + titles = services().wikipedia.suggest(query) if query else [] + return jsonify({"query": query, "titles": [asdict(item) for item in titles]}) + + +@api.get("/api/context") +def link_context() -> Response: + """The sentence in one article that links to the next. + --- + tags: [Pathfinding] + summary: Link context for one step of a path + parameters: + - name: from + in: query + required: true + type: string + description: The article that contains the link + - name: to + in: query + required: true + type: string + description: The article it links to + responses: + 200: + description: > + "where" is "prose" when the link sits in running text, and then + before + link + after is the sentence with link as the linked words. + "template" means the link is in an infobox, table or template; + "elsewhere" means it is not in the article's own text at all + (usually a navigation box). + 400: + description: Missing from or to + 429: + description: Too many requests from this address + """ + _rate_limit("api", services().settings.api_rate_limit) + source, target = _route_args() + context = services().wikipedia.link_context(source, target) + return jsonify({"from": source, "to": target, **asdict(context)}) + + +# --- Sharing -------------------------------------------------------------- + + +@api.get("/api/og.png") +def share_image() -> Response: + """The preview image a shared route link unfurls into. + --- + tags: [Pathfinding] + summary: Share image for a route + produces: [image/png] + parameters: + - name: from + in: query + required: true + type: string + - name: to + in: query + required: true + type: string + responses: + 200: + description: > + A 1200x630 PNG of the route if it has been found recently, or of the + question if not. + 400: + description: Missing from or to + """ + _rate_limit("api", services().settings.api_rate_limit) + start, end = _route_args() + path = services().store.get(route_key(start, end)) + return Response( + render_card(start, end, path), + mimetype="image/png", + # Short when the route is unknown: it may be found a minute from now. + headers={"Cache-Control": f"public, max-age={86_400 if path else 300}"}, + ) + + # --- UI ------------------------------------------------------------------- API_PREFIXES = ("getPath", "tasks", "health", "api", "apispec") @@ -230,16 +409,36 @@ def health() -> tuple[Response, int]: @api.get("/") def index() -> Response: - """Serve the visualisation UI.""" - return current_app.send_static_file("index.html") + """Serve the UI; a shared route link gets that route's preview tags.""" + start = request.args.get("from", "").strip()[:MAX_TITLE] + end = request.args.get("to", "").strip()[:MAX_TITLE] + if not (start and end): + return current_app.send_static_file("index.html") + + path = services().store.get(route_key(start, end)) + route: dict[str, Any] = {"from": start, "to": end} + page = with_share_meta( + Path(current_app.static_folder or "", "index.html").read_text("utf-8"), + describe(start, end, path), + url=request.url, + image=url_for("api.share_image", _external=True, **route), + ) + return Response(page, mimetype="text/html") + + +@api.get("/favicon.ico") +@api.get("/robots.txt") +def root_file() -> Response: + """Files browsers and crawlers ask for at the root, not under /static.""" + return current_app.send_static_file(request.path.lstrip("/")) @api.get("/") -def catch_all(path: str) -> Response: - """Send unknown API paths to a 404 and everything else to the UI.""" +def catch_all(path: str) -> tuple[Response, int]: + """Answer unknown API paths with JSON and everything else with a page.""" if path.startswith(API_PREFIXES): raise NotFoundError("Endpoint not found") - return redirect(url_for("api.index")) # type: ignore[return-value] + return current_app.send_static_file("404.html"), 404 # --- Helpers -------------------------------------------------------------- @@ -254,5 +453,26 @@ def _require_task_id(task_id: str) -> None: ) from exc +def _route_args() -> tuple[str, str]: + source = request.args.get("from", "").strip()[:MAX_TITLE] + target = request.args.get("to", "").strip()[:MAX_TITLE] + if not (source and target): + raise InvalidRequestError("Both 'from' and 'to' are required") + return source, target + + +def _rate_limit(bucket: str, limit: int) -> None: + """Allow ``limit`` requests per client per window, counted in Redis. + + Fixed windows, keyed by the window's number, so every key expires on its + own and nothing needs sweeping. + """ + window = services().settings.rate_window + now = int(time.time()) + key = f"ratelimit:{bucket}:{request.remote_addr}:{now // window}" + if services().store.increment(key, ttl=window) > limit: + raise RateLimitedError(retry_after=window - now % window) + + def _wants_terminate() -> bool: return request.args.get("terminate", "true").lower() != "false" diff --git a/app/share.py b/app/share.py new file mode 100644 index 0000000..aff90fd --- /dev/null +++ b/app/share.py @@ -0,0 +1,266 @@ +"""Share previews: the image and the words a link unfurls into. + +A shared link (``/?from=A&to=B``) unfurls in Slack, Discord or a social feed +as a card showing that route. Crawlers do not run JavaScript, so both the +```` tags and the image are produced here, from the route kept in Redis +under ``route:{start}:{end}`` when the search finished. A link to a route that +was never searched (or has expired) still gets a card: it asks the question +instead of answering it. + +The card follows the published OG image: GitHub-dark ground, JetBrains Mono, +green start, red end, blue path, on a faint dot field. +""" + +from __future__ import annotations + +import html +import io +import re +from dataclasses import dataclass +from functools import cache +from pathlib import Path + +from PIL import Image, ImageDraw, ImageFont + +WIDTH, HEIGHT = 1200, 630 +SCALE = 2 +"""Drawn at twice the size and downsampled: Pillow does not antialias shapes.""" + +FONTS = Path(__file__).parent / "fonts" + +GROUND = "#0d1117" +RAISED = "#161b22" +RULE = "#21262d" +RULE_STRONG = "#30363d" +INK = "#f0f6fc" +INK_MUTED = "#9198a1" +INK_FAINT = "#7d8590" +BLUE = "#58a6ff" +GREEN = "#3fb950" +RED = "#f85149" +DOT = (139, 148, 158, 40) + +MAX_ROWS = 7 +SITE = "IRIS.HISHAAM.DEV" + + +@dataclass(frozen=True, slots=True) +class ShareMeta: + title: str + description: str + + +def describe(start: str, end: str, path: list[str] | None) -> ShareMeta: + if path: + hops = len(path) - 1 + links = f"{hops} link{'s' if hops != 1 else ''}" + return ShareMeta( + title=f"{start} → {end} in {links} · Iris", + description=f"{' → '.join(path)}. A route through Wikipedia, found live by Iris.", + ) + return ShareMeta( + title=f"{start} → {end} · Iris", + description=f"How does {start} lead to {end} on Wikipedia? Iris follows the links live and draws the route.", + ) + + +def with_share_meta(page: str, meta: ShareMeta, url: str, image: str) -> str: + """Point the page's title and Open Graph / Twitter tags at one route.""" + values = { + "og:title": meta.title, + "twitter:title": meta.title, + "og:description": meta.description, + "twitter:description": meta.description, + "description": meta.description, + "og:url": url, + "og:image": image, + "twitter:image": image, + } + for key, value in values.items(): + page = re.sub( + rf'([^<]*", + lambda _: f"{html.escape(meta.title)}", + page, + count=1, + ) + + +# --- The card ------------------------------------------------------------- + + +@cache +def _font(weight: str, size: int) -> ImageFont.FreeTypeFont: + return ImageFont.truetype(str(FONTS / f"JetBrainsMono-{weight}.ttf"), size * SCALE) + + +def render_card(start: str, end: str, path: list[str] | None) -> bytes: + image = Image.new("RGB", (WIDTH * SCALE, HEIGHT * SCALE), GROUND) + draw = ImageDraw.Draw(image, "RGBA") + + _dot_field(draw) + _masthead(draw) + if path: + _route(draw, path) + else: + _question(draw, start, end) + _footer(draw, "WIKIPEDIA PATH • FOUND LIVE" if path else "WIKIPEDIA PATH") + + image = image.resize((WIDTH, HEIGHT), Image.Resampling.LANCZOS) + buffer = io.BytesIO() + image.save(buffer, format="PNG", optimize=True) + return buffer.getvalue() + + +def _s(*values: float) -> list[float]: + """Scale card coordinates to the drawing surface.""" + return [v * SCALE for v in values] + + +def _xy(x: float, y: float) -> tuple[float, float]: + return x * SCALE, y * SCALE + + +def _dot_field(draw: ImageDraw.ImageDraw) -> None: + for x in range(20, WIDTH, 20): + for y in range(20, HEIGHT, 20): + draw.ellipse(_s(x - 0.75, y - 0.75, x + 0.75, y + 0.75), fill=DOT) + + +def _masthead(draw: ImageDraw.ImageDraw) -> None: + # The logo: start and end on top, the path node below, joined by lines. + a, b, c = (78, 52), (100, 52), (89, 72) + for p, q in ((a, b), (a, c), (b, c)): + draw.line(_s(*p, *q), fill=INK_MUTED, width=2 * SCALE) + for (x, y), colour in ((a, GREEN), (b, RED), (c, BLUE)): + draw.ellipse(_s(x - 5, y - 5, x + 5, y + 5), fill=colour) + _tracked(draw, (118, 50), "IRIS", _font("Regular", 18), INK_MUTED, 0.18) + + +def _footer(draw: ImageDraw.ImageDraw, left: str) -> None: + draw.line(_s(72, 548, WIDTH - 72, 548), fill=RULE, width=SCALE) + font = _font("Regular", 14) + _tracked(draw, (72, 566), left, font, INK_FAINT, 0.14) + width = _tracked_width(SITE, font, 0.14) + _tracked(draw, (WIDTH - 72 - width / SCALE, 566), SITE, font, BLUE, 0.14) + + +def _route(draw: ImageDraw.ImageDraw, path: list[str]) -> None: + rows = _rows(path) + top, bottom = 118, 520 + step = min(62, (bottom - top) / max(len(rows), 1)) + first_y = (top + bottom) / 2 - step * (len(rows) - 1) / 2 + last_y = first_y + step * (len(rows) - 1) + + # The rail, then a ruled row and a mark per step, as in the route list. + rail_x = 80 + draw.line( + _s(rail_x, first_y, rail_x, last_y), fill=(88, 166, 255, 115), width=SCALE + ) + title_font = _font("SemiBold", 26 if len(rows) <= 5 else 23) + index_font = _font("Regular", 16) + for i, (number, title) in enumerate(rows): + y = first_y + i * step + if i: + draw.line(_s(104, y - step / 2, 760, y - step / 2), fill=RULE, width=SCALE) + colour = GREEN if i == 0 else RED if i == len(rows) - 1 else BLUE + if number is None: + colour = RULE_STRONG + r = 7 + draw.ellipse( + _s(rail_x - r - 4, y - r - 4, rail_x + r + 4, y + r + 4), fill=GROUND + ) + draw.ellipse(_s(rail_x - r, y - r, rail_x + r, y + r), fill=colour) + label = f"{number:02d}" if number is not None else "··" + draw.text(_xy(110, y), label, font=index_font, fill=INK_FAINT, anchor="lm") + text = _fit(title, title_font, 600) + draw.text( + _xy(160, y), + text, + font=title_font, + fill=INK if number is not None else INK_FAINT, + anchor="lm", + ) + + hops = len(path) - 1 + figure = str(hops) + draw.text( + _xy(1128, 300), figure, font=_font("SemiBold", 168), fill=INK, anchor="rs" + ) + caption = "link" if hops == 1 else "links" + draw.text( + _xy(1128, 344), caption, font=_font("Regular", 26), fill=INK_MUTED, anchor="rs" + ) + draw.text( + _xy(1128, 380), + f"{len(path)} pages", + font=_font("Regular", 18), + fill=INK_FAINT, + anchor="rs", + ) + + +def _rows(path: list[str]) -> list[tuple[int | None, str]]: + """Every step when they fit; otherwise both ends and an elision between.""" + numbered: list[tuple[int | None, str]] = [ + (i + 1, title) for i, title in enumerate(path) + ] + if len(numbered) <= MAX_ROWS: + return numbered + hidden = len(path) - (MAX_ROWS - 1) + head, tail = numbered[:3], numbered[-(MAX_ROWS - 4) :] + return [*head, (None, f"{hidden} more"), *tail] + + +def _question(draw: ImageDraw.ImageDraw, start: str, end: str) -> None: + font = _font("SemiBold", 40) + for y, title, colour in ((236, start, GREEN), (396, end, RED)): + draw.ellipse(_s(72, y - 8, 88, y + 8), fill=colour) + draw.text(_xy(110, y), _fit(title, font, 980), font=font, fill=INK, anchor="lm") + # The dashed connector the page draws while a search is running. + for y in range(262, 372, 12): + draw.line(_s(80, y, 80, y + 6), fill=RULE_STRONG, width=2 * SCALE) + draw.text( + _xy(110, 316), + "How does one lead to the other?", + font=_font("Regular", 22), + fill=INK_MUTED, + anchor="lm", + ) + + +# --- Text helpers ----------------------------------------------------------- + + +def _fit(text: str, font: ImageFont.FreeTypeFont, max_width: float) -> str: + if font.getlength(text) <= max_width * SCALE: + return text + while len(text) > 1 and font.getlength(text + "…") > max_width * SCALE: + text = text[:-1] + return text.rstrip() + "…" + + +def _tracked_width(text: str, font: ImageFont.FreeTypeFont, tracking: float) -> float: + gap = font.size * tracking + return sum(font.getlength(ch) for ch in text) + gap * (len(text) - 1) + + +def _tracked( + draw: ImageDraw.ImageDraw, + at: tuple[float, float], + text: str, + font: ImageFont.FreeTypeFont, + fill: str, + tracking: float, +) -> None: + """Letter-spaced capitals, as the published OG image sets its labels.""" + x, y = at[0] * SCALE, at[1] * SCALE + gap = font.size * tracking + for ch in text: + draw.text((x, y), ch, font=font, fill=fill) + x += font.getlength(ch) + gap diff --git a/app/store.py b/app/store.py index d946579..be2e971 100644 --- a/app/store.py +++ b/app/store.py @@ -49,6 +49,13 @@ def delete(self, *keys: str) -> None: def expire(self, key: str, seconds: int) -> None: self._redis.expire(key, seconds) + def increment(self, key: str, ttl: int) -> int: + """Add one to a counter, starting its expiry when it is created.""" + count = int(self._redis.incr(key)) # type: ignore[arg-type] + if count == 1: + self._redis.expire(key, ttl) + return count + # --- Sets ------------------------------------------------------------- def set_add(self, key: str, *values: str) -> None: diff --git a/app/tasks.py b/app/tasks.py index ae71d06..ac8114b 100644 --- a/app/tasks.py +++ b/app/tasks.py @@ -10,6 +10,7 @@ from __future__ import annotations import logging +from collections import deque from collections.abc import Callable from dataclasses import asdict from typing import Any, NotRequired, TypedDict @@ -41,6 +42,21 @@ RETRYABLE = (requests.RequestException, redis.RedisError, WikipediaAPIError) +RECENT_PER_SIDE = 6 +"""Pages each frontier shows the UI while it runs: the ones it read last.""" + + +def route_key(start_page: str, end_page: str) -> str: + """Where a found route is kept for shared links, whichever algorithm ran.""" + return f"route:{start_page}:{end_page}" + + +class FrontierStats(TypedDict): + explored: int + depth: int + recent: list[str] + """Newest first.""" + class SearchStats(TypedDict, total=False): """The progress payload the UI renders. Field names are part of the API.""" @@ -54,6 +70,12 @@ class SearchStats(TypedDict, total=False): end_page: str final_depth: int search_completed: bool + algorithm: str + frontier: dict[str, FrontierStats] + """In flight: per-side counts and the pages each side read last.""" + meeting_page: str + sides: dict[str, dict[str, int]] + """Finished: per-side ``explored`` and ``depth``.""" class TaskResult(TypedDict): @@ -96,6 +118,7 @@ def publish(stage: str, stats: SearchStats, elapsed: float = 0.0) -> None: "start_page": start_page, "end_page": end_page, "max_depth": settings.max_search_depth, + "algorithm": algorithm, **stats, }, "search_time_elapsed": elapsed, @@ -121,7 +144,13 @@ def publish(stage: str, stats: SearchStats, elapsed: float = 0.0) -> None: }, ) + recent: dict[str, deque[str]] = { + "forward": deque(maxlen=RECENT_PER_SIDE), + "backward": deque(maxlen=RECENT_PER_SIDE), + } + def on_progress(progress: Progress) -> None: + recent[progress.side].appendleft(progress.last_node) publish( "Searching...", { @@ -129,6 +158,14 @@ def on_progress(progress: Progress) -> None: "queue_size": progress.queue_size, "current_depth": progress.current_depth, "last_node": progress.last_node, + "frontier": { + side: { + "explored": stats["explored"], + "depth": stats["depth"], + "recent": list(recent[side]), + } + for side, stats in progress.sides.items() + }, }, elapsed=progress.elapsed, ) @@ -178,6 +215,9 @@ def on_progress(progress: Progress) -> None: "nodes_explored": result.nodes_explored, "final_depth": max(result.length - 1, 0), "search_completed": True, + "algorithm": algorithm, + "meeting_page": result.meeting, + "sides": result.sides, }, } @@ -233,9 +273,14 @@ def _find_path( logger.info( "path_cache_hit", extra={"start_page": start_page, "end_page": end_page} ) - return PathResult(**cached) - - finder = PathFinder(app.wikipedia, app.store, app.settings, on_progress) - result = finder.find(start_page, end_page, algorithm) - app.store.set(cache_key, asdict(result), ttl=app.settings.path_cache_ttl) + result = PathResult(**cached) + else: + finder = PathFinder(app.wikipedia, app.store, app.settings, on_progress) + result = finder.find(start_page, end_page, algorithm) + app.store.set(cache_key, asdict(result), ttl=app.settings.path_cache_ttl) + + # Shared links outlive the path cache; every hit renews the route. + app.store.set( + route_key(start_page, end_page), result.path, ttl=app.settings.share_ttl + ) return result diff --git a/app/wikipedia.py b/app/wikipedia.py index 5ba9e2b..7a48cc2 100644 --- a/app/wikipedia.py +++ b/app/wikipedia.py @@ -27,6 +27,7 @@ from app.config import Settings from app.errors import WikipediaAPIError from app.store import RedisStore +from app.wikitext import LinkContext, find_link_context logger = logging.getLogger(__name__) @@ -38,6 +39,8 @@ links out of the 500-link allowance, and keeps titles that merely contain a colon ("Star Wars: Episode IV – A New Hope") in.""" +SUGGESTION_LIMIT = 8 + PageFetched = Callable[[str, list[str]], None] """Called with (title, titles-found) as each page arrives. Runs on worker threads.""" @@ -49,6 +52,13 @@ class PageStatus: is_disambiguation: bool +@dataclass(frozen=True, slots=True) +class Suggestion: + title: str + description: str + is_disambiguation: bool + + @dataclass(frozen=True, slots=True) class LinkQuery: """One of the two directions the graph can be walked in.""" @@ -136,6 +146,10 @@ def backlinks( def page_status(self, title: str) -> PageStatus: ... + def suggest(self, prefix: str) -> list[Suggestion]: ... + + def link_context(self, source: str, target: str) -> LinkContext: ... + class WikipediaClient: """Fetches page links, with a shared rate limit and a Redis-backed cache.""" @@ -182,6 +196,87 @@ def page_status(self, title: str) -> PageStatus: self.cache.set(cache_key, asdict(status), ttl=self.settings.page_cache_ttl) return status + def suggest(self, prefix: str) -> list[Suggestion]: + """Articles whose titles start with ``prefix``, best match first. + + Redirects are resolved, so "usa" offers "United States" rather than the + redirect page; disambiguation pages are flagged so the UI can say so + before they are rejected as an end page. + """ + prefix = prefix.strip() + if not prefix: + return [] + cache_key = f"suggest:{prefix.casefold()}" + if self.cache and (cached := self.cache.get(cache_key)) is not None: + return [Suggestion(**item) for item in cached] + + query = self._request( + { + "generator": "prefixsearch", + "gpssearch": prefix, + "gpslimit": SUGGESTION_LIMIT, + "gpsnamespace": ARTICLE_NAMESPACE, + "prop": "description|pageprops", + "ppprop": "disambiguation", + "formatversion": 2, + } + ).get("query", {}) + + # Two prefix hits can redirect to one article; keep its best rank. + ranked = sorted(query.get("pages", []), key=lambda page: page.get("index", 0)) + seen: set[str] = set() + suggestions: list[Suggestion] = [] + for page in ranked: + title = page.get("title", "") + if not title or title in seen or "missing" in page: + continue + seen.add(title) + suggestions.append( + Suggestion( + title=title, + description=page.get("description", ""), + is_disambiguation="disambiguation" in page.get("pageprops", {}), + ) + ) + + if self.cache: + self.cache.set( + cache_key, + [asdict(item) for item in suggestions], + ttl=self.settings.suggest_cache_ttl, + ) + return suggestions + + def link_context(self, source: str, target: str) -> LinkContext: + """The sentence in ``source`` that carries its link to ``target``.""" + cache_key = f"context:{source}:{target}" + if self.cache and (cached := self.cache.get(cache_key)): + return LinkContext(**cached) + + response = self._request( + { + "titles": source, + "prop": "revisions", + "rvprop": "content", + "rvslots": "main", + "formatversion": 2, + } + ) + pages = response.get("query", {}).get("pages", []) + revisions = pages[0].get("revisions", []) if pages else [] + wikitext = ( + revisions[0].get("slots", {}).get("main", {}).get("content", "") + if revisions + else "" + ) + context = find_link_context(wikitext, target) + + if self.cache: + self.cache.set( + cache_key, asdict(context), ttl=self.settings.context_cache_ttl + ) + return context + def _page_status(self, title: str) -> PageStatus: query = self._request( {"titles": title, "prop": "info|categories", "cllimit": "max"} @@ -276,13 +371,19 @@ def _fetch_many( self.cache.set( f"{query.cache_prefix}:{title}", found, - ttl=self.settings.links_cache_ttl, + ttl=self._links_ttl(found), ) if on_page: on_page(title, found) return results + def _links_ttl(self, found: list[str]) -> int: + """Hubs are cached longer: see ``Settings.hub_cache_ttl``.""" + if len(found) >= self.settings.hub_link_threshold: + return self.settings.hub_cache_ttl + return self.settings.links_cache_ttl + def _fetch_one(self, query: LinkQuery, title: str) -> list[str]: """Fetch one page, following continuations up to the configured limit.""" params: dict[str, str | int] = {"titles": title, **query.params} diff --git a/app/wikitext.py b/app/wikitext.py new file mode 100644 index 0000000..c708fc3 --- /dev/null +++ b/app/wikitext.py @@ -0,0 +1,182 @@ +"""Find the sentence in an article's wikitext that links to another article. + +A path step says "A links to B"; this is what makes it say *why*. The link is +looked for in the article's prose, after templates, references, tables and +file captions are stripped away, and the sentence around it is returned as +plain text, split around the link's own words. + +Where the link is not in prose, the answer says so rather than guessing: +``template`` when it sits inside an infobox, table or other template in the +wikitext itself, and ``elsewhere`` when it is not in the wikitext at all — +the usual case for navigation boxes, which are transcluded at render time. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Literal + +Where = Literal["prose", "template", "elsewhere"] + +MAX_SENTENCE = 320 +"""Characters kept, link included, when a sentence runs long.""" + +ANCHOR_OPEN, ANCHOR_CLOSE = "\x01", "\x02" +"""Marks the target link's words while the line is reduced to plain text.""" + +_COMMENT = re.compile(r"", re.DOTALL) +_REF = re.compile(r"/]*/>|]*>.*?", re.DOTALL | re.IGNORECASE) +_TEMPLATE = re.compile(r"\{\{[^{}]*\}\}") +_TABLE = re.compile(r"\{\|[^{]*?\|\}", re.DOTALL) +_FILE = re.compile(r"\[\[(?:File|Image):[^\[\]]*(?:\[\[[^\]]*\]\][^\[\]]*)*\]\]", re.I) +_LINK = re.compile(r"\[\[([^\[\]|]+)(?:\|([^\[\]]*))?\]\]") +_LINK_WITH_TRAIL = re.compile(_LINK.pattern + r"([a-z]*)") +"""Letters straight after ``]]`` render as part of the link: [[plantain]]s.""" +_EXTERNAL = re.compile(r"\[https?://\S+\s*([^\]]*)\]") +_TAG = re.compile(r"<[^>]+>") +_HEADING = re.compile(r"^(={2,6})\s*(.*?)\s*\1\s*$") +_SENTENCE_END = re.compile(r"(?<=[.!?])[\"”’)]*\s+(?=[\"“(\[A-Z0-9])") + + +@dataclass(frozen=True, slots=True) +class LinkContext: + """The sentence as three runs, so no client has to count characters: + ``before + link + after`` is the sentence and ``link`` is the linked words.""" + + where: Where + before: str = "" + link: str = "" + after: str = "" + section: str | None = None + """The heading the sentence sits under; ``None`` for the lead.""" + + +def find_link_context(wikitext: str, target: str) -> LinkContext: + if not wikitext or not target: + return LinkContext(where="elsewhere") + + raw = _COMMENT.sub("", wikitext) + if not _links_to(raw, target): + return LinkContext(where="elsewhere") + + prose = _strip_structure(raw) + section: str | None = None + for line in prose.splitlines(): + if heading := _HEADING.match(line.strip()): + section = _plain(heading.group(2)) or None + continue + if not _links_to(line, target): + continue + text = _plain(_mark_target(line.lstrip("*#:; "), target)) + if runs := _sentence_around_anchor(text): + before, link, after = runs + return LinkContext( + where="prose", before=before, link=link, after=after, section=section + ) + + return LinkContext(where="template") + + +# --- Matching ------------------------------------------------------------- + + +def _same_title(written: str, target: str) -> bool: + """MediaWiki's title equality: first letter and spaces/underscores loose.""" + + def norm(title: str) -> str: + title = re.sub(r"[\s_]+", " ", title.split("#", 1)[0]).strip() + return title[:1].upper() + title[1:] + + return bool(written.strip()) and norm(written) == norm(target) + + +def _links_to(text: str, target: str) -> bool: + return any(_same_title(m.group(1), target) for m in _LINK.finditer(text)) + + +def _mark_target(line: str, target: str) -> str: + """Wrap the first link to ``target`` in anchor marks; leave the rest.""" + marked = False + + def replace(match: re.Match[str]) -> str: + nonlocal marked + if marked or not _same_title(match.group(1), target): + return match.group(0) + marked = True + words = (match.group(2) or match.group(1).split("#", 1)[0]) + match.group(3) + return f"{ANCHOR_OPEN}{words}{ANCHOR_CLOSE}" + + return _LINK_WITH_TRAIL.sub(replace, line) + + +# --- Reduction to plain text --------------------------------------------- + + +def _strip_structure(text: str) -> str: + """Remove everything that is not running prose.""" + text = _REF.sub("", text) + text = _remove_nested(_FILE, text) + text = _remove_nested(_TEMPLATE, text) + return _remove_nested(_TABLE, text) + + +def _remove_nested(pattern: re.Pattern[str], text: str) -> str: + """Delete innermost matches until none remain, so nesting unwinds.""" + while True: + reduced = pattern.sub("", text) + if reduced == text: + return text + text = reduced + + +def _plain(text: str) -> str: + text = _LINK.sub(lambda m: m.group(2) or m.group(1).split("#", 1)[0], text) + text = _EXTERNAL.sub(lambda m: m.group(1), text) + text = _TAG.sub("", text) + text = text.replace("'''", "").replace("''", "") + text = text.replace(" ", " ").replace("–", "–").replace("—", "—") + text = re.sub(r"\s+", " ", text) + # Removed references and templates leave stray gaps before punctuation. + text = re.sub(r"\s+([,.;:!?)])", r"\1", text) + text = re.sub(r"\(\s*\)", "", text) + return text.strip() + + +def _sentence_around_anchor(text: str) -> tuple[str, str, str] | None: + start = text.find(ANCHOR_OPEN) + end = text.find(ANCHOR_CLOSE, start) + if start < 0 or end < 0: + return None + + boundaries = [0, *(m.end() for m in _SENTENCE_END.finditer(text)), len(text)] + begin = max(b for b in boundaries if b <= start) + finish = min(b for b in boundaries if b > end) + + before = text[begin:start].lstrip() + link = text[start + 1 : end] + after = text[end + 1 : finish].rstrip() + if not link.strip(): + return None + return _trim(before, link, after) + + +def _trim(before: str, link: str, after: str) -> tuple[str, str, str]: + """Shorten a long sentence to a window around the link, at word breaks.""" + spare = MAX_SENTENCE - len(link) + if len(before) + len(after) <= spare: + return before, link, after + if spare <= 0: + return "…", link[:MAX_SENTENCE], "…" + + keep_before = min(len(before), max(spare // 2, spare - len(after))) + keep_after = spare - keep_before + if keep_before < len(before): + cut = before[-keep_before:] if keep_before else "" + cut = cut.split(" ", 1)[1] if " " in cut else cut + before = "…" + cut + if keep_after < len(after): + cut = after[:keep_after] + cut = cut.rsplit(" ", 1)[0] if " " in cut else cut + after = cut + "…" + return before, link, after diff --git a/entrypoint.sh b/entrypoint.sh old mode 100644 new mode 100755 index a9265f3..a31671a --- a/entrypoint.sh +++ b/entrypoint.sh @@ -1,13 +1,16 @@ #!/bin/sh # -# entrypoint.sh — Docker container entrypoint +# entrypoint.sh — the container's CMD (see Dockerfile) # -# Switches between web and worker modes based on the SERVICE_TYPE env var. -# Used by the Dockerfile and railway.toml for Railway deployments. +# One image runs both services; SERVICE_TYPE picks the role: # # SERVICE_TYPE=worker → Celery worker # SERVICE_TYPE= → Gunicorn web server (default) # +# The web server uses threaded workers because GET /tasks/stream/ holds a +# connection open for the length of a search; a sync worker would be pinned to +# one viewer. --timeout bounds a stuck worker, not a long request, under gthread. +# # Port is read from PORT env var (default: 8080, Railway sets this automatically). set -e @@ -20,5 +23,10 @@ if [ "$SERVICE_TYPE" = "worker" ]; then --concurrency=2 else echo "Starting Gunicorn on port ${PORT:-8080}..." - exec gunicorn --bind "0.0.0.0:${PORT:-8080}" --workers 2 --timeout 120 run:app + exec gunicorn --bind "0.0.0.0:${PORT:-8080}" \ + --workers 2 \ + --worker-class gthread \ + --threads 16 \ + --timeout 120 \ + run:app fi diff --git a/pyproject.toml b/pyproject.toml index cf5502f..c5afd55 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,6 +8,7 @@ dependencies = [ "flask>=3.1.1", "gunicorn>=23.0.0", "marshmallow>=3.22.0", + "pillow>=11.0.0", "python-dotenv>=1.1.1", "redis>=6.2.0", "requests>=2.32.4", diff --git a/railway.toml b/railway.toml index 7b5fa0e..c9a2770 100644 --- a/railway.toml +++ b/railway.toml @@ -1,8 +1,8 @@ +# The start command is the Dockerfile's CMD; it is not repeated here. [build] builder = "DOCKERFILE" dockerfilePath = "Dockerfile" [deploy] -startCommand = "/entrypoint.sh" restartPolicyType = "ON_FAILURE" restartPolicyMaxRetries = 3 diff --git a/static/404.html b/static/404.html new file mode 100644 index 0000000..c2e542e --- /dev/null +++ b/static/404.html @@ -0,0 +1,56 @@ + + + + + + Page not found · Iris + + + + + + + + + + + + +
+
+ + + Iris + + +
+ +
+
+

404

+

No page here.

+

+ Nothing links to this address. Iris is a single page, + and Wikipedia has the other seven million. +

+ +
+
+ + +
+ + diff --git a/static/apple-touch-icon.png b/static/apple-touch-icon.png new file mode 100644 index 0000000..0f4a821 Binary files /dev/null and b/static/apple-touch-icon.png differ diff --git a/static/favicon-dark.svg b/static/favicon-dark.svg deleted file mode 100644 index 3ba9d7b..0000000 --- a/static/favicon-dark.svg +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - diff --git a/static/favicon-light.svg b/static/favicon-light.svg deleted file mode 100644 index 7e506fc..0000000 --- a/static/favicon-light.svg +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - diff --git a/static/favicon.ico b/static/favicon.ico new file mode 100644 index 0000000..c268730 Binary files /dev/null and b/static/favicon.ico differ diff --git a/static/favicon.svg b/static/favicon.svg index 3ba9d7b..0508875 100644 --- a/static/favicon.svg +++ b/static/favicon.svg @@ -1,39 +1,15 @@ - - - - - - - - - - - - - + + + + + + + + + + diff --git a/static/icon-192.png b/static/icon-192.png new file mode 100644 index 0000000..037a1c3 Binary files /dev/null and b/static/icon-192.png differ diff --git a/static/icon-512.png b/static/icon-512.png new file mode 100644 index 0000000..b69dcfb Binary files /dev/null and b/static/icon-512.png differ diff --git a/static/index.html b/static/index.html index 7fb4df8..3284096 100644 --- a/static/index.html +++ b/static/index.html @@ -19,12 +19,11 @@ - + + + + + - @@ -108,17 +94,17 @@

Watch
- The readings update as each page is fetched: - depth reached, pages explored, the frontier - still queued, and the page being read now. + Each side shows the pages it is reading and + how deep it has gone; the readings count + pages explored and the frontier still queued.
Explore
- Drag nodes to rearrange the graph; open any - step on Wikipedia. Your last search survives - a refresh. + Each step quotes the sentence that links it + to the next. Drag the graph, open any step on + Wikipedia, or copy a link to the route.
@@ -141,7 +127,7 @@

>
- + autocapitalize="off" spellcheck="false" enterkeyhint="next" + role="combobox" + aria-autocomplete="list" + aria-expanded="false" + aria-controls="startPageSuggestions" required /> +
- + autocapitalize="off" spellcheck="false" enterkeyhint="go" + role="combobox" + aria-autocomplete="list" + aria-expanded="false" + aria-controls="endPageSuggestions" required /> +
@@ -195,6 +201,14 @@

+

+ Try + + +

+ diff --git a/static/robots.txt b/static/robots.txt index 669f404..009da2a 100644 --- a/static/robots.txt +++ b/static/robots.txt @@ -1,5 +1,7 @@ User-agent: * Allow: / -Disallow: /api/docs +Allow: /api/og.png +Disallow: /api/ Disallow: /apispec.json -Disallow: /cache/clear +Disallow: /tasks/ +Disallow: /getPath diff --git a/static/script.js b/static/script.js index db79f15..920ea81 100644 --- a/static/script.js +++ b/static/script.js @@ -13,6 +13,25 @@ const NODE_R = 14; const SVG_NS = "http://www.w3.org/2000/svg"; +// Suggestions start after two characters, once typing pauses +const SUGGEST_MIN_CHARS = 2; +const SUGGEST_DELAY_MS = 140; + +// Routes known to resolve, for a visitor who arrives with nothing in mind +const EXAMPLES = [ + ["Banana", "Shah Rukh Khan"], + ["Python (programming language)", "Monty Python"], + ["Coffee", "Moon"], + ["Chess", "Ancient Egypt"], + ["Jazz", "Mount Everest"], + ["Pizza", "Albert Einstein"], + ["Tea", "Black hole"], + ["Ballet", "Volcano"], +]; +const EXAMPLES_SHOWN = 3; + +const TERMINAL = new Set(["SUCCESS", "FAILURE", "REVOKED"]); + const $ = (id) => document.getElementById(id); const formatSeconds = (seconds) => @@ -21,6 +40,33 @@ const formatSeconds = (seconds) => const wikipediaUrl = (page) => `https://en.wikipedia.org/wiki/${encodeURIComponent(page)}`; +// A route as a link anyone can open: /?from=…&to=… +function shareUrl(start, end) { + const url = new URL("/", window.location.origin); + url.searchParams.set("from", start); + url.searchParams.set("to", end); + return url.toString(); +} + +function sharedRoute() { + const params = new URLSearchParams(window.location.search); + const start = params.get("from")?.trim(); + const end = params.get("to")?.trim(); + return start && end ? { start, end } : null; +} + +const plural = (count, word) => + `${count.toLocaleString()} ${count === 1 ? word : `${word}s`}`; + +function shuffled(items) { + const copy = [...items]; + for (let i = copy.length - 1; i > 0; i--) { + const j = Math.floor(Math.random() * (i + 1)); + [copy[i], copy[j]] = [copy[j], copy[i]]; + } + return copy; +} + function openWikipediaPage(page) { if (page && page !== "-") window.open(wikipediaUrl(page), "_blank"); } @@ -97,6 +143,189 @@ function debounce(fn, wait) { }; } +// Title suggestions under one route field: the ARIA combobox pattern, with +// the input keeping focus and the active option named by +// aria-activedescendant. Picking an option fills the field exactly, so the +// search starts from a real title instead of a guess. +class Autocomplete { + constructor(input, list, { onPick, isEnd }) { + this.input = input; + this.list = list; + this.onPick = onPick; + this.isEnd = isEnd; + this.options = []; + this.active = -1; + this.timer = null; + this.controller = null; + this.cache = new Map(); + + input.addEventListener("input", () => this.schedule()); + input.addEventListener("keydown", (e) => this.onKey(e)); + // iOS Safari can blur the input before a tapped option's click lands, + // so closing waits a beat; focus coming back cancels it. + input.addEventListener("blur", () => { + this.blurTimer = setTimeout(() => this.close(), 150); + }); + input.addEventListener("focus", () => clearTimeout(this.blurTimer)); + // Keep focus in the input while an option is pressed + list.addEventListener("pointerdown", (e) => e.preventDefault()); + } + + schedule() { + clearTimeout(this.timer); + const query = this.input.value.trim(); + if (query.length < SUGGEST_MIN_CHARS) { + this.close(); + return; + } + this.timer = setTimeout(() => this.fetch(query), SUGGEST_DELAY_MS); + } + + async fetch(query) { + const key = query.toLowerCase(); + if (this.cache.has(key)) { + this.render(this.cache.get(key)); + return; + } + this.controller?.abort(); + this.controller = new AbortController(); + try { + const response = await fetch( + `${API_BASE}/api/suggest?q=${encodeURIComponent(query)}`, + { signal: this.controller.signal }, + ); + if (!response.ok) return; + const { titles } = await response.json(); + this.cache.set(key, titles); + // Typing moved on while this was in flight + if (this.input.value.trim().toLowerCase() !== key) return; + this.render(titles); + } catch { + /* suggestions are a convenience; the field still works without them */ + } + } + + render(titles) { + if (document.activeElement !== this.input) return; + // Nothing to add when the only match is what is already typed + const typed = this.input.value.trim(); + if (!titles.length || (titles.length === 1 && titles[0].title === typed)) { + this.close(); + return; + } + + this.options = titles; + this.active = -1; + this.list.replaceChildren( + ...titles.map((item, index) => { + const option = document.createElement("li"); + option.id = `${this.list.id}-${index}`; + option.className = "suggestion"; + option.setAttribute("role", "option"); + option.setAttribute("aria-selected", "false"); + + const title = document.createElement("span"); + title.className = "suggestion-title"; + title.textContent = item.title; + option.appendChild(title); + + const note = item.is_disambiguation + ? this.isEnd + ? "Disambiguation page: pick a specific article" + : "Disambiguation page" + : item.description; + if (note) { + const detail = document.createElement("span"); + detail.className = "suggestion-note"; + detail.textContent = note; + option.appendChild(detail); + } + option.classList.toggle( + "is-ambiguous", + Boolean(item.is_disambiguation && this.isEnd), + ); + + option.addEventListener("click", () => this.pick(index)); + return option; + }), + ); + this.list.classList.remove("hidden"); + this.input.setAttribute("aria-expanded", "true"); + } + + onKey(event) { + const open = !this.list.classList.contains("hidden"); + switch (event.key) { + case "ArrowDown": + case "ArrowUp": { + if (!open) { + this.schedule(); + return; + } + event.preventDefault(); + const step = event.key === "ArrowDown" ? 1 : -1; + const count = this.options.length; + // Positions run 0..count, where 0 is the typed text itself + const position = (this.active + 1 + step + count + 1) % (count + 1); + this.highlight(position - 1); + break; + } + case "Enter": + if (open && this.active >= 0) { + event.preventDefault(); + this.pick(this.active); + } else { + this.close(); + } + break; + case "Escape": + if (open) { + event.preventDefault(); + this.close(); + } + break; + case "Tab": + this.close(); + break; + } + } + + // -1 means back to what was typed, with no option active + highlight(index) { + this.active = index; + [...this.list.children].forEach((option, i) => { + option.setAttribute("aria-selected", String(i === index)); + }); + const current = this.list.children[index]; + if (current) { + this.input.setAttribute("aria-activedescendant", current.id); + current.scrollIntoView({ block: "nearest" }); + } else { + this.input.removeAttribute("aria-activedescendant"); + } + } + + pick(index) { + clearTimeout(this.blurTimer); + const item = this.options[index]; + if (!item) return; + this.input.value = item.title; + this.close(); + this.onPick(item); + } + + close() { + clearTimeout(this.timer); + this.controller?.abort(); + this.options = []; + this.active = -1; + this.list.classList.add("hidden"); + this.list.replaceChildren(); + this.input.setAttribute("aria-expanded", "false"); + this.input.removeAttribute("aria-activedescendant"); + } +} + // Turn any failure into what the visitor needs: what happened, which field // (if one is at fault), and what to try next. Server messages are shown only // when they are written for people; internal ones stay in the server logs. @@ -146,6 +375,13 @@ function describeFailure({ code, message, details }, route) { hint: "This is usually brief. Try again in a minute.", retry: true, }; + case "RATE_LIMITED": + return { + title: "Too many searches at once", + message: "Iris limits how many searches one visitor can start each minute.", + hint: "Wait a minute, then try again.", + retry: true, + }; case "TIMEOUT": return { title: "The search took too long", @@ -180,12 +416,64 @@ class PathFinderUI { this.taskStartTime = null; this.pollTimeoutId = null; this.abortController = null; + this.eventSource = null; + this.watchdogId = null; + // Bumped per rendered route, so late link-context replies for an older + // route are dropped instead of written into the new one + this.routeGeneration = 0; this.initializeGraph(); + this.setupAutocomplete(); + this.renderExamples(); this.setupEventListeners(); this.restoreStateFromStorage(); } + setupAutocomplete() { + const fields = [ + ["startPage", false], + ["endPage", true], + ]; + for (const [id, isEnd] of fields) { + new Autocomplete($(id), $(`${id}Suggestions`), { + isEnd, + onPick: () => { + $(id).removeAttribute("aria-invalid"); + this.saveCurrentState(); + this.updateButtonState(); + // Picking the start moves on to the end; picking the end is ready + if (!isEnd && !$("endPage").value.trim()) $("endPage").focus(); + }, + }); + } + } + + renderExamples() { + $("examplesList").replaceChildren( + ...shuffled(EXAMPLES) + .slice(0, EXAMPLES_SHOWN) + .map(([start, end]) => { + const button = document.createElement("button"); + button.type = "button"; + button.className = "example"; + button.textContent = `${start} → ${end}`; + button.addEventListener("click", () => this.runRoute(start, end)); + return button; + }), + ); + } + + runRoute(start, end) { + $("startPage").value = start; + $("endPage").value = end; + for (const id of ["startPage", "endPage"]) { + $(id).removeAttribute("aria-invalid"); + } + this.saveCurrentState(); + this.updateButtonState(); + this.findPath(); + } + initializeGraph() { const svg = d3.select("#graph"); svg.selectAll("*").remove(); @@ -215,6 +503,13 @@ class PathFinderUI { $("clearBtn").addEventListener("click", () => this.clearVisualization()); $("retryBtn").addEventListener("click", () => this.findPath()); $("dismissBtn").addEventListener("click", () => this.hideNotice()); + $("shareBtn").addEventListener("click", () => this.shareRoute()); + $("surpriseBtn").addEventListener("click", () => { + const { start, end } = this.route(); + const choices = EXAMPLES.filter(([a, b]) => a !== start || b !== end); + const [a, b] = choices[Math.floor(Math.random() * choices.length)]; + this.runRoute(a, b); + }); for (const id of ["startPage", "endPage"]) { $(id).addEventListener("input", () => { @@ -236,6 +531,24 @@ class PathFinderUI { restoreStateFromStorage() { const saved = StateManager.load(); + const shared = sharedRoute(); + + // A shared link names its route; it wins over whatever this browser + // last did, unless that is the very same route already found. + if (shared) { + const path = saved?.result?.path; + $("startPage").value = shared.start; + $("endPage").value = shared.end; + if (path && path[0] === shared.start && path.at(-1) === shared.end) { + this.showVisualizationSection(); + this.handlePathFound(saved.result); + } else { + this.findPath(); + } + this.updateButtonState(); + return; + } + if (saved) { if (saved.startPage) $("startPage").value = saved.startPage; if (saved.endPage) $("endPage").value = saved.endPage; @@ -246,7 +559,7 @@ class PathFinderUI { this.taskStartTime = saved.taskStartTime || Date.now(); this.showVisualizationSection(); this.showLoading(); - this.pollTaskStatus(); + this.watchTask(); } else if (saved.result?.path) { this.showVisualizationSection(); this.handlePathFound(saved.result); @@ -286,6 +599,7 @@ class PathFinderUI { clearTimeout(this.pollTimeoutId); this.pollTimeoutId = null; } + this.closeStream(); // Abort any in-flight fetch requests if (this.abortController) { this.abortController.abort(); @@ -382,6 +696,7 @@ class PathFinderUI { // Reset depth. The real budget arrives with the first progress update; // until then keep whatever count is already rendered. this.updateDepthIndicator(0, $("depthDots").children.length || 6); + this.updateFrontiers({}); } setLastNode(title) { @@ -407,6 +722,49 @@ class PathFinderUI { progress.search_time_elapsed || 0, ); this.setLastNode(stats.last_node); + this.updateFrontiers(stats); + } + + // Each end's chip carries the pages that side read last, a count, and a + // fill along the line for how deep it has gone. The fills are measured + // against the whole depth budget, so two sides that meet touch mid-line. + updateFrontiers(stats) { + const frontier = stats.frontier || {}; + const maxDepth = stats.max_depth || $("depthDots").children.length || 6; + const forwardOnly = stats.algorithm === "bfs"; + + for (const [side, suffix] of [ + ["forward", "Forward"], + ["backward", "Backward"], + ]) { + const state = frontier[side]; + this.renderTrail($(`trail${suffix}`), state?.recent || []); + + let count = "Starting"; + if (state) { + count = `${plural(state.explored, "page")} · ${plural(state.depth, "link")} out`; + } else if (side === "backward" && forwardOnly) { + count = "Waiting to be reached"; + } else if (!stats.frontier) { + count = ""; + } + $(`count${suffix}`).textContent = count; + + const reach = Math.min(1, (state?.depth || 0) / maxDepth); + $(`reach${suffix}`).style.setProperty("--reach", reach.toFixed(3)); + } + } + + renderTrail(list, titles) { + const before = new Set([...list.children].map((li) => li.textContent)); + list.replaceChildren( + ...titles.map((title) => { + const item = document.createElement("li"); + item.textContent = title; + if (!before.has(title)) item.className = "is-new"; + return item; + }), + ); } updateDepthIndicator(currentDepth, maxDepth) { @@ -445,6 +803,12 @@ class PathFinderUI { async findPath() { const { start, end } = this.route(); + // A different route than the address bar names: stop advertising it + const shared = sharedRoute(); + if (shared && (shared.start !== start || shared.end !== end)) { + history.replaceState(null, "", "/"); + } + if (!start || !end) { this.showNotice({ title: "Enter both pages", @@ -489,27 +853,76 @@ class PathFinderUI { this.taskId = data.task_id; this.taskStartTime = Date.now(); this.saveCurrentState(); - this.pollTaskStatus(); + this.watchTask(); } catch (error) { if (error.name === "AbortError") return; this.fail({ code: "OFFLINE" }); } } - async pollTaskStatus(pollErrors = 0) { - // Capture the task this poll chain belongs to. After every await, a - // mismatch means a cancel or new search happened and this chain must - // die silently. + // Follow the task over Server-Sent Events, falling back to polling when + // the browser has no EventSource or the stream cannot be kept open. + watchTask() { + if (!this.taskId) return; + if (!("EventSource" in window)) { + this.pollTaskStatus(); + return; + } + const taskId = this.taskId; - if (!taskId) return; + const source = new EventSource(`${API_BASE}/tasks/stream/${taskId}`); + this.eventSource = source; + let errors = 0; + + // The stream only speaks when something changes, so the ten-minute + // limit needs its own clock. + this.watchdogId = setInterval(() => { + if (this.taskId === taskId) this.checkTimeout(); + }, 5000); + + source.onmessage = (event) => { + if (this.taskId !== taskId) return; + errors = 0; + this.handleStatus(JSON.parse(event.data)); + }; + // An error with the source still CONNECTING is the browser retrying by + // itself (it does after every stream the server rotates); CLOSED, or a + // run of failures, means the stream is not coming back. + source.onerror = () => { + if (this.taskId !== taskId) return; + errors += 1; + if (source.readyState === EventSource.CLOSED || errors >= MAX_POLL_ERRORS) { + this.closeStream(); + this.pollTaskStatus(); + } + }; + } + + closeStream() { + this.eventSource?.close(); + this.eventSource = null; + clearInterval(this.watchdogId); + this.watchdogId = null; + } + + checkTimeout() { if ( this.taskStartTime && Date.now() - this.taskStartTime > MAX_TASK_POLL_MS ) { this.fail({ code: "TIMEOUT" }); - return; + return true; } + return false; + } + + async pollTaskStatus(pollErrors = 0) { + // Capture the task this poll chain belongs to. After every await, a + // mismatch means a cancel or new search happened and this chain must + // die silently. + const taskId = this.taskId; + if (!taskId || this.checkTimeout()) return; const pollAgain = (delay, errors = 0) => { this.pollTimeoutId = setTimeout(() => this.pollTaskStatus(errors), delay); @@ -524,34 +937,8 @@ class PathFinderUI { const data = await response.json(); if (this.taskId !== taskId) return; - switch (data.status) { - case "PENDING": - // Waiting for a worker; the route is already on screen - this.showProgressLoader(); - pollAgain(1000); - break; - - case "IN_PROGRESS": - this.showProgressLoader(); - this.updateProgressDisplay(data.progress); - pollAgain(1000); - break; - - case "SUCCESS": - this.handlePathFound(data.result); - break; - - case "FAILURE": - this.fail({ code: data.code, message: data.error }); - break; - - case "REVOKED": - this.fail({ code: "STOPPED" }); - break; - - default: - this.fail({}); - } + this.handleStatus(data); + if (!TERMINAL.has(data.status)) pollAgain(1000); } catch (error) { if (error.name === "AbortError") return; // Stale chain — don't clobber the new search's UI @@ -567,6 +954,37 @@ class PathFinderUI { } } + // One status body, from the stream or a poll + handleStatus(data) { + switch (data.status) { + case "PENDING": + // Waiting for a worker; the route is already on screen + this.showProgressLoader(); + break; + + case "IN_PROGRESS": + this.showProgressLoader(); + this.updateProgressDisplay(data.progress); + break; + + case "SUCCESS": + this.closeStream(); + this.handlePathFound(data.result); + break; + + case "FAILURE": + this.fail({ code: data.code, message: data.error }); + break; + + case "REVOKED": + this.fail({ code: "STOPPED" }); + break; + + default: + this.fail({}); + } + } + handlePathFound(result) { this.hideLoading(); @@ -582,6 +1000,12 @@ class PathFinderUI { const path = result.path; this.setRoute(path[0], path[path.length - 1]); + // The address bar now names this route, ready to be copied or shared + try { + history.replaceState(null, "", shareUrl(path[0], path.at(-1))); + } catch { + /* some embedded browsers refuse history changes */ + } $("runState").dataset.state = "found"; $("runState").textContent = `Found in ${formatSeconds(result.search_time)}`; @@ -1048,6 +1472,7 @@ class PathFinderUI { } displayPathList(path, result) { + const meeting = result.search_stats?.meeting_page; const explored = result.search_stats?.nodes_explored || result.nodes_explored || 0; const hops = path.length - 1; @@ -1088,6 +1513,12 @@ class PathFinderUI { const title = document.createElement("span"); title.className = "step-title"; title.textContent = page; + if (page === meeting && index > 0 && index < path.length - 1) { + const tag = document.createElement("span"); + tag.className = "step-tag"; + tag.textContent = "Frontiers met"; + title.append(" ", tag); + } const open = document.createElementNS(SVG_NS, "svg"); open.setAttribute("class", "step-open"); @@ -1099,11 +1530,124 @@ class PathFinderUI { step.append(mark, number, title, open); item.appendChild(step); + + // The sentence that carries this page's link to the next one + if (index < path.length - 1) { + const context = document.createElement("p"); + context.className = "step-context is-loading"; + context.textContent = "Finding the link…"; + item.appendChild(context); + } return item; }), ); + const explain = this.explainRoute(path, result); + $("routeExplain").textContent = explain; + $("routeExplain").classList.toggle("hidden", !explain); + $("pathStepsContainer").classList.remove("hidden"); + this.loadLinkContexts(path); + } + + // Say what the two searches did, in a sentence + explainRoute(path, result) { + const stats = result.search_stats || {}; + const sides = stats.sides; + const meeting = stats.meeting_page; + if (!sides?.forward || !meeting || path.length < 2) return ""; + + const start = path[0]; + const end = path.at(-1); + const forward = sides.forward.explored || 0; + const backward = sides.backward?.explored || 0; + if (stats.algorithm === "bfs" || !backward) { + return `Searching forward from ${start}, Iris read ${plural(forward, "page")} before it reached ${end}.`; + } + + const fromStart = path.indexOf(meeting); + const where = + fromStart <= 0 || fromStart >= path.length - 1 + ? `The two searches met at ${meeting}.` + : `The two searches met at ${meeting}, ${plural(fromStart, "link")} from the start.`; + return `Iris read ${plural(forward, "page")} forward from ${start} and ${plural(backward, "page")} backward from ${end}. ${where}`; + } + + loadLinkContexts(path) { + const generation = ++this.routeGeneration; + const contexts = $("pathSteps").querySelectorAll(".step-context"); + + path.slice(0, -1).forEach(async (source, index) => { + const element = contexts[index]; + const target = path[index + 1]; + try { + const query = new URLSearchParams({ from: source, to: target }); + const response = await fetch(`${API_BASE}/api/context?${query}`); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + const context = await response.json(); + if (generation !== this.routeGeneration) return; + this.renderLinkContext(element, context); + } catch { + if (generation === this.routeGeneration) element.remove(); + } + }); + } + + renderLinkContext(element, context) { + element.classList.remove("is-loading"); + if (context.where !== "prose") { + element.classList.add("is-structural"); + element.textContent = + context.where === "template" + ? "Linked from an infobox, table or template on this page." + : "Linked from a navigation box or template on this page."; + return; + } + + const link = document.createElement("span"); + link.className = "step-context-link"; + link.textContent = context.link; + const quote = document.createElement("q"); + quote.append(context.before, link, context.after); + element.replaceChildren(quote); + + if (context.section) { + const section = document.createElement("span"); + section.className = "step-context-section"; + section.textContent = context.section; + element.append(" ", section); + } + } + + async shareRoute() { + const start = $("routeFrom").textContent; + const end = $("routeTo").textContent; + const url = shareUrl(start, end); + const touch = window.matchMedia("(pointer: coarse)").matches; + + // Phones get the system share sheet; desktops copy the link + if (touch && navigator.share) { + try { + await navigator.share({ title: `${start} → ${end} · Iris`, url }); + } catch { + /* dismissed */ + } + return; + } + + try { + await navigator.clipboard.writeText(url); + } catch { + window.prompt("Copy this link", url); + return; + } + $("shareLabel").textContent = "Link copied"; + $("shareStatus").textContent = `Link to ${start} → ${end} copied`; + clearTimeout(this.shareTimer); + this.shareTimer = setTimeout(() => { + $("shareLabel").textContent = "Copy link"; + $("shareStatus").textContent = ""; + }, 2000); } async cancelSearch() { @@ -1130,6 +1674,8 @@ class PathFinderUI { $("pathStepsContainer").classList.add("hidden"); $("visualizationSection").classList.remove("show"); + this.routeGeneration++; + history.replaceState(null, "", "/"); this.graph.svg.selectAll("*").remove(); this.graph.simulation?.stop(); this.graph.simulation = null; diff --git a/static/site.webmanifest b/static/site.webmanifest new file mode 100644 index 0000000..81a788a --- /dev/null +++ b/static/site.webmanifest @@ -0,0 +1,13 @@ +{ + "name": "Iris: Wikipedia Path Finder", + "short_name": "Iris", + "description": "Find a hyperlink path between any two Wikipedia pages.", + "start_url": "/", + "display": "standalone", + "background_color": "#0d1117", + "theme_color": "#0d1117", + "icons": [ + { "src": "/static/icon-192.png", "sizes": "192x192", "type": "image/png", "purpose": "any maskable" }, + { "src": "/static/icon-512.png", "sizes": "512x512", "type": "image/png", "purpose": "any maskable" } + ] +} diff --git a/static/styles.css b/static/styles.css index 883bc27..3f2e6f4 100644 --- a/static/styles.css +++ b/static/styles.css @@ -90,6 +90,15 @@ a { border-radius: 2px; } +.sr-only { + position: absolute; + width: 1px; + height: 1px; + overflow: hidden; + clip-path: inset(50%); + white-space: nowrap; +} + .skip-link { position: absolute; left: var(--sp-4); @@ -171,6 +180,14 @@ a { text-wrap: balance; } +/* 404: the code is a reading, so it takes the reading face */ +.lost-code { + margin-bottom: var(--sp-3); + font-size: var(--fs-reading); + font-weight: 500; + color: var(--danger-ink); +} + .lead-copy { margin-top: var(--sp-5); max-width: var(--measure); @@ -254,6 +271,10 @@ a { column-gap: var(--sp-6); } +.field { + position: relative; +} + .field label { display: inline-flex; align-items: center; @@ -322,6 +343,62 @@ a { box-shadow: 0 1px 0 var(--danger-red); } +/* Title suggestions float over the page like the graph tooltip: the one + raised, shadowed surface, because it sits above whatever is below it. */ +.suggestions { + position: absolute; + z-index: 20; + top: 100%; + left: 0; + right: 0; + max-height: min(360px, 60vh); + overflow-y: auto; + list-style: none; + border: 1px solid var(--rule-strong); + border-top: 0; + background: var(--bg-secondary); + box-shadow: 0 8px 24px rgba(1, 4, 9, 0.6); +} + +.suggestion { + display: flex; + flex-direction: column; + justify-content: center; + min-height: 44px; + padding: var(--sp-2) var(--sp-3); + border-top: 1px solid var(--rule); + cursor: pointer; +} + +.suggestion:first-child { + border-top: 0; +} + +.suggestion-title { + font-size: var(--fs-body); + font-weight: 500; + overflow-wrap: anywhere; +} + +.suggestion-note { + font-size: var(--fs-label); + color: var(--text-tertiary); + overflow-wrap: anywhere; +} + +.suggestion[aria-selected="true"], +.suggestion:hover { + background: var(--bg-tertiary); +} + +.suggestion[aria-selected="true"] .suggestion-title { + color: var(--accent-blue); +} + +.suggestion.is-ambiguous .suggestion-title { + color: var(--text-secondary); +} + .route-arrow { width: 22px; height: 22px; @@ -340,6 +417,44 @@ a { margin-top: var(--sp-6); } +/* Example routes: in-text actions, so Path Blue like "How it works" */ +.examples { + display: flex; + flex-wrap: wrap; + align-items: center; + gap: 0 var(--sp-4); + margin-top: var(--sp-3); + font-size: var(--fs-small); + color: var(--text-tertiary); +} + +.examples-list { + display: contents; +} + +.example, +.examples-surprise { + min-height: 44px; + border: 0; + background: none; + font: inherit; + color: var(--accent-blue); + text-align: left; + cursor: pointer; + transition: color 0.15s ease; +} + +.example:hover, +.examples-surprise:hover { + color: var(--accent-blue-hover); + text-decoration: underline; + text-underline-offset: 3px; +} + +.examples-surprise { + color: var(--text-secondary); +} + /* --- Buttons ------------------------------------------------------------- */ .btn { @@ -355,6 +470,7 @@ a { font: inherit; font-size: 14px; font-weight: 600; + text-decoration: none; cursor: pointer; transition: background-color 0.15s ease, @@ -781,23 +897,36 @@ a { border-right-width: 1px; } -/* While searching the field shows only the two ends, joined by a line - that is still being drawn. */ +/* While searching, the field shows the two frontiers: each end with the + pages that side read last, joined by a line each side fills as it goes + deeper. The fills are scaled to the whole depth budget, so they meet. */ .wait { position: absolute; inset: 0; - display: flex; + display: grid; + grid-template-columns: minmax(0, 1fr) minmax(64px, 320px) minmax(0, 1fr); align-items: center; - justify-content: center; gap: var(--sp-4); padding: var(--sp-10); } +.wait-side { + display: flex; + flex-direction: column; + align-items: flex-end; + gap: var(--sp-2); + min-width: 0; +} + +.wait-side.to { + align-items: flex-start; +} + .wait-end { display: inline-flex; align-items: center; gap: var(--sp-2); - max-width: 30%; + max-width: 100%; padding: var(--sp-1) var(--sp-3); border: 1px solid var(--rule-strong); border-radius: 4px; @@ -822,10 +951,52 @@ a { background: var(--danger-red); } +/* Newest first, fading with age */ +.wait-trail { + list-style: none; + display: flex; + flex-direction: column; + align-items: inherit; + max-width: 100%; + min-height: calc(6 * 20px); + font-size: var(--fs-label); + line-height: 20px; + color: var(--text-secondary); +} + +.wait-trail li { + max-width: 100%; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.wait-trail li:nth-child(2) { opacity: 0.8; } +.wait-trail li:nth-child(3) { opacity: 0.65; } +.wait-trail li:nth-child(4) { opacity: 0.5; } +.wait-trail li:nth-child(5) { opacity: 0.38; } +.wait-trail li:nth-child(6) { opacity: 0.26; } + +.wait-trail li.is-new:first-child { + color: var(--text-primary); + animation: trail-in 0.3s var(--ease-out); +} + +@keyframes trail-in { + from { + opacity: 0; + transform: translateY(-4px); + } +} + +.wait-count { + min-height: 20px; + font-size: var(--fs-label); + color: var(--text-tertiary); +} + .wait-line { position: relative; - flex: 1; - max-width: 320px; height: 1px; background-image: linear-gradient( 90deg, @@ -836,7 +1007,24 @@ a { overflow: clip; } -.wait-line span { +.wait-reach { + position: absolute; + top: 0; + bottom: 0; + width: calc(var(--reach, 0) * 100%); + background: var(--accent-blue); + transition: width 0.5s var(--ease-out); +} + +.wait-reach.from { + left: 0; +} + +.wait-reach.to { + right: 0; +} + +.wait-sweep { position: absolute; inset: 0; width: 30%; @@ -863,6 +1051,28 @@ a { margin-top: var(--sp-10); } +.route-list-head { + display: flex; + align-items: center; + justify-content: space-between; + gap: var(--sp-4); +} + +.share-btn { + gap: var(--sp-2); + margin-right: calc(-1 * var(--sp-3)); +} + +.share-btn svg { + width: 16px; + height: 16px; + fill: none; + stroke: currentColor; + stroke-width: 1.6; + stroke-linecap: round; + stroke-linejoin: round; +} + .route-list-title { font-size: 16px; font-weight: 600; @@ -875,6 +1085,14 @@ a { color: var(--text-secondary); } +.route-explain { + margin-top: var(--sp-2); + max-width: var(--measure); + font-size: var(--fs-small); + color: var(--text-secondary); + text-wrap: pretty; +} + .route-summary span + span::before { content: "·"; margin: 0 var(--sp-2); @@ -912,8 +1130,10 @@ a { opacity: 0.45; } +/* From the first mark, which sits mid-row: a step can be taller than its + row once its link context is below it, so 50% of the item would miss. */ .steps li:first-child::before { - top: 50%; + top: 26px; } .steps li:last-child::before { @@ -983,6 +1203,58 @@ a { border-radius: 0; } +.step-tag { + font-size: var(--fs-label); + font-weight: 400; + color: var(--text-tertiary); + white-space: nowrap; +} + +.step-tag::before { + content: "·"; + margin-right: var(--sp-2); +} + +/* The sentence linking a step to the next, hung under its title */ +.step-context { + max-width: var(--measure); + margin: calc(-1 * var(--sp-1)) 0 0 calc(9px + 36px + 2 * var(--sp-3)); + padding-bottom: var(--sp-3); + font-size: var(--fs-small); + line-height: 1.6; + color: var(--text-secondary); + overflow-wrap: anywhere; + animation: context-in 0.3s var(--ease-out); +} + +.step-context.is-loading, +.step-context.is-structural { + color: var(--text-tertiary); +} + +.step-context q { + quotes: "“" "”"; +} + +.step-context-link { + color: var(--accent-blue); +} + +.step-context-section { + color: var(--text-tertiary); + white-space: nowrap; +} + +.step-context-section::before { + content: "— "; +} + +@keyframes context-in { + from { + opacity: 0; + } +} + /* --- Colophon ------------------------------------------------------------ */ .colophon { @@ -1168,18 +1440,40 @@ a { } .wait { - flex-direction: column; + grid-template-columns: minmax(0, 1fr); + grid-template-rows: auto 56px auto; + justify-items: center; + align-content: center; + gap: var(--sp-3); padding: var(--sp-6); } - .wait-end { + .wait-side, + .wait-side.to { + align-items: center; max-width: 100%; } + /* The end reads bottom-up, so its newest page sits nearest the line */ + .wait-side.to { + flex-direction: column-reverse; + } + + .wait-trail { + min-height: calc(3 * 20px); + } + + .wait-side.to .wait-trail { + flex-direction: column-reverse; + } + + .wait-trail li:nth-child(n + 4) { + display: none; + } + .wait-line { - flex: none; width: 1px; - height: 72px; + height: 100%; background-image: linear-gradient( 180deg, var(--rule-strong) 50%, @@ -1188,7 +1482,25 @@ a { background-size: 1px 8px; } - .wait-line span { + .wait-reach { + left: 0; + right: 0; + width: auto; + height: calc(var(--reach, 0) * 100%); + transition-property: height; + } + + .wait-reach.from { + top: 0; + bottom: auto; + } + + .wait-reach.to { + top: auto; + bottom: 0; + } + + .wait-sweep { width: 100%; height: 30%; background: linear-gradient( @@ -1238,7 +1550,7 @@ a { /* Searching must stay perceivable: swap travel for a slow fade instead of freezing it, which reads as broken. */ .activity-fill, - .wait-line span { + .wait-sweep { width: 100% !important; height: 100% !important; transform: none !important; diff --git a/tests/conftest.py b/tests/conftest.py index 0632b7a..7ac730b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -17,7 +17,8 @@ from app import Services, celery, create_app from app.config import Settings from app.store import RedisStore -from app.wikipedia import PageFetched, PageStatus +from app.wikipedia import PageFetched, PageStatus, Suggestion +from app.wikitext import LinkContext # A small, deliberately asymmetric graph. # @@ -78,6 +79,25 @@ def page_status(self, title: str) -> PageStatus: is_disambiguation=resolved in DISAMBIGUATION_PAGES, ) + def suggest(self, prefix: str) -> list[Suggestion]: + titles = sorted([*self.graph, *DISAMBIGUATION_PAGES]) + return [ + Suggestion( + title=title, + description="", + is_disambiguation=title in DISAMBIGUATION_PAGES, + ) + for title in titles + if title.casefold().startswith(prefix.casefold()) + ] + + def link_context(self, source: str, target: str) -> LinkContext: + if target not in self.graph.get(source, []): + return LinkContext(where="elsewhere") + return LinkContext( + where="prose", before=f"{source} relates to ", link=target, after="." + ) + @staticmethod def _respond( found: dict[str, list[str]], on_page: PageFetched | None diff --git a/tests/test_api.py b/tests/test_api.py index bf4f1f8..e3e433d 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -6,6 +6,8 @@ from __future__ import annotations +import json +from dataclasses import replace from unittest.mock import patch import pytest @@ -61,6 +63,36 @@ def test_rejects_a_body_that_is_not_json(client): assert response.status_code == 400 +def use_settings(app, **changes): + current = app.extensions["iris"] + app.extensions["iris"] = replace( + current, settings=replace(current.settings, **changes) + ) + + +def test_limits_how_many_searches_one_client_starts(app, client): + use_settings(app, search_rate_limit=2) + + statuses = [search(client).status_code for _ in range(3)] + + assert statuses == [202, 202, 429] + refused = search(client) + assert refused.get_json()["code"] == "RATE_LIMITED" + assert 0 < int(refused.headers["Retry-After"]) <= 60 + + +def test_rate_limits_are_per_client(app, client): + use_settings(app, search_rate_limit=1) + + assert search(client).status_code == 202 + other = client.post( + "/getPath", + json={"start": "Python", "end": "Mathematics"}, + environ_base={"REMOTE_ADDR": "10.0.0.2"}, + ) + assert other.status_code == 202 + + # --- Polling -------------------------------------------------------------- # # The status endpoint is a translation layer over Celery's task states. Driving @@ -164,6 +196,57 @@ def test_a_revoked_task_is_reported_as_revoked(client, task_state): assert client.get(f"/tasks/status/{TASK_ID}").get_json()["status"] == "REVOKED" +# --- Streaming ------------------------------------------------------------ + + +def events(response) -> list[dict]: + return [ + json.loads(line.removeprefix("data: ")) + for line in response.get_data(as_text=True).splitlines() + if line.startswith("data: ") + ] + + +def test_streams_status_changes_until_the_search_ends(app, client, monkeypatch): + use_settings(app, stream_interval=0) + progress = {"search_stats": {"nodes_explored": 3}} + states = iter( + [ + StubResult("PENDING"), + StubResult("PROGRESS", info=progress), + StubResult("PROGRESS", info=progress), + StubResult("SUCCESS", result={"status": "SUCCESS", "path": ["A", "B"]}), + ] + ) + monkeypatch.setattr( + "app.routes.find_path_task.AsyncResult", lambda _id: next(states) + ) + + response = client.get(f"/tasks/stream/{TASK_ID}") + + assert response.mimetype == "text/event-stream" + assert [e["status"] for e in events(response)] == [ + "PENDING", + "IN_PROGRESS", + "SUCCESS", + ], "an unchanged status is not sent twice" + assert events(response)[-1]["result"]["path"] == ["A", "B"] + + +def test_a_stream_closes_after_its_time_limit(app, client, task_state): + use_settings(app, stream_interval=0, stream_max_seconds=0) + task_state(state="PENDING") + + response = client.get(f"/tasks/stream/{TASK_ID}") + + assert [e["status"] for e in events(response)] == ["PENDING"] + assert response.get_data(as_text=True).startswith("retry: ") + + +def test_streaming_a_malformed_task_id_is_a_404(client): + assert client.get("/tasks/stream/not-a-uuid").status_code == 404 + + def test_rejects_a_malformed_task_id(client): response = client.get("/tasks/status/not-a-uuid") @@ -207,6 +290,101 @@ def test_cancelling_a_malformed_task_id_is_a_404(client): assert client.delete("/tasks/not-a-uuid").status_code == 404 +# --- Suggestions and link context --------------------------------------- + + +def test_suggests_titles_for_a_prefix(client): + body = client.get("/api/suggest?q=mer").get_json() + + assert body["query"] == "mer" + assert body["titles"] == [ + {"title": "Mercury", "description": "", "is_disambiguation": True} + ] + + +def test_an_empty_suggestion_query_returns_nothing(client): + assert client.get("/api/suggest?q=").get_json()["titles"] == [] + + +def test_explains_a_step_with_the_sentence_that_links_it(client): + body = client.get("/api/context?from=Python&to=Programming").get_json() + + assert body["from"] == "Python" + assert body["to"] == "Programming" + assert body["where"] == "prose" + assert body["before"] + body["link"] + body["after"] == ( + "Python relates to Programming." + ) + + +def test_context_needs_both_ends(client): + response = client.get("/api/context?from=Python") + + assert response.status_code == 400 + assert response.get_json()["code"] == "INVALID_REQUEST" + + +def test_lighter_endpoints_share_their_own_rate_limit(app, client): + use_settings(app, api_rate_limit=1) + + assert client.get("/api/suggest?q=py").status_code == 200 + assert client.get("/api/suggest?q=py").status_code == 429 + assert search(client).status_code == 202, "searches are counted separately" + + +# --- Sharing ---------------------------------------------------------------- + +PNG = b"\x89PNG\r\n\x1a\n" + + +def test_draws_a_share_image_for_a_found_route(client, store): + store.set("route:Python:Mathematics", ["Python", "Monty Python", "Mathematics"]) + + response = client.get("/api/og.png?from=Python&to=Mathematics") + + assert response.status_code == 200 + assert response.mimetype == "image/png" + assert response.data.startswith(PNG) + assert "max-age=86400" in response.headers["Cache-Control"] + + +def test_draws_a_share_image_for_a_route_not_yet_found(client): + response = client.get("/api/og.png?from=Python&to=Orphan") + + assert response.data.startswith(PNG) + assert "max-age=300" in response.headers["Cache-Control"], "may be found soon" + + +def test_a_shared_link_unfurls_into_its_own_route(client, store): + store.set("route:Python:Mathematics", ["Python", "Monty Python", "Mathematics"]) + + page = client.get("/?from=Python&to=Mathematics").get_data(as_text=True) + + assert "Python → Mathematics in 2 links · Iris" in page + assert ( + 'property="og:image" content="http://localhost/api/og.png?from=Python&to=Mathematics"' + in page + ) + assert "Python → Monty Python → Mathematics" in page + + +def test_share_tags_escape_what_the_link_carries(client): + page = client.get('/?from=x&to=a"b').get_data(as_text=True) + + assert "x" not in page + assert "<i>x</i>" in page + assert "a"b" in page + + +def test_share_previews_use_the_public_scheme_behind_a_proxy(client): + page = client.get( + "/?from=Python&to=Mathematics", + headers={"X-Forwarded-Proto": "https", "X-Forwarded-Host": "iris.example"}, + ).get_data(as_text=True) + + assert 'content="https://iris.example/api/og.png?' in page + + # --- System --------------------------------------------------------------- @@ -239,11 +417,38 @@ def test_serves_the_ui_at_the_root(client): assert b"= len(result.path) - 1 + ) + + +def test_plain_bfs_meets_at_the_target_with_no_backward_side(finder): + result = finder.find("Python", "Mathematics", BFS) + + assert result.meeting == "Mathematics" + assert set(result.sides) == {"forward"} + assert result.sides["forward"]["explored"] == result.nodes_explored + + +def test_progress_names_the_side_that_read_each_page(wikipedia, store, settings): + seen: list[Progress] = [] + PathFinder(wikipedia, store, settings, seen.append).find( + "Python", "Mathematics", BIDIRECTIONAL + ) + + forward = {p.last_node for p in seen if p.side == "forward"} + backward = {p.last_node for p in seen if p.side == "backward"} + assert "Python" in forward and "Python" not in backward + assert "Mathematics" in backward and "Mathematics" not in forward + assert seen[-1].sides["forward"]["explored"] + seen[-1].sides["backward"][ + "explored" + ] == len(seen) diff --git a/tests/test_tasks.py b/tests/test_tasks.py index 4e6b8fc..d1523f6 100644 --- a/tests/test_tasks.py +++ b/tests/test_tasks.py @@ -78,6 +78,38 @@ def test_publishes_progress_while_searching(app): assert "max_depth" in meta["search_stats"] +def test_progress_shows_each_frontier_and_the_pages_it_read_last(app): + with patch.object(find_path_task, "update_state") as update_state: + find_path_task.apply(args=("Python", "Mathematics", BIDIRECTIONAL)) + + frontiers = [ + call.kwargs["meta"]["search_stats"]["frontier"] + for call in update_state.call_args_list + if "frontier" in call.kwargs["meta"]["search_stats"] + ] + last = frontiers[-1] + + assert set(last) == {"forward", "backward"} + assert last["forward"]["recent"][-1] == "Python", "oldest read is the start" + assert last["backward"]["recent"][-1] == "Mathematics" + assert all(side["explored"] >= 1 for side in last.values()) + assert all(len(side["recent"]) <= 6 for f in frontiers for side in f.values()) + + +def test_the_result_says_where_the_frontiers_met(app): + result = run() + stats = result["search_stats"] + + assert stats["algorithm"] == BIDIRECTIONAL + assert stats["meeting_page"] in result["path"][1:-1] + assert set(stats["sides"]) == {"forward", "backward"} + + +def test_a_found_route_is_kept_for_shared_links(app, store): + path = run()["path"] + assert store.get("route:Python:Mathematics") == path + + # --- Caching -------------------------------------------------------------- diff --git a/tests/test_wikipedia.py b/tests/test_wikipedia.py index 3a2a01d..c022db8 100644 --- a/tests/test_wikipedia.py +++ b/tests/test_wikipedia.py @@ -398,3 +398,117 @@ def test_spaces_requests_by_the_configured_delay(monkeypatch): client._await_rate_slot() assert waits and waits[0] > 0, "the second call must wait out the interval" + + +# --- Hubs ------------------------------------------------------------------- + + +def test_hub_link_lists_are_cached_for_longer(store): + settings = Settings( + env="testing", + secret_key="test", + wikipedia_request_delay=0, + hub_link_threshold=3, + links_cache_ttl=100, + hub_cache_ttl=10_000, + ) + session = FakeSession( + FakeResponse(links_payload("Hub", ["A", "B", "C"])), + FakeResponse(links_payload("Leaf", ["A"])), + ) + client = build(settings, session, cache=store) + + client.links(["Hub"]) + client.links(["Leaf"]) + + raw = store._redis + assert raw.ttl("wiki_links:Hub") > 100 + assert raw.ttl("wiki_links:Leaf") <= 100 + + +# --- Suggestions -------------------------------------------------------------- + + +def suggest_payload(*pages: dict) -> dict: + return {"query": {"pages": list(pages)}} + + +def test_suggests_titles_in_rank_order_with_descriptions(settings): + session = FakeSession( + FakeResponse( + suggest_payload( + {"title": "Shah Rukh", "index": 2, "description": "Timurid ruler"}, + {"title": "Shah Rukh Khan", "index": 1, "description": "Actor"}, + {"title": "Shah", "index": 3, "pageprops": {"disambiguation": ""}}, + ) + ) + ) + + found = build(settings, session).suggest("shah ru") + + assert [s.title for s in found] == ["Shah Rukh Khan", "Shah Rukh", "Shah"] + assert found[0].description == "Actor" + assert [s.is_disambiguation for s in found] == [False, False, True] + assert session.requests[0]["gpssearch"] == "shah ru" + + +def test_two_prefixes_redirecting_to_one_article_appear_once(settings): + session = FakeSession( + FakeResponse( + suggest_payload( + {"title": "United States", "index": 1}, + {"title": "United States", "index": 4}, + ) + ) + ) + assert len(build(settings, session).suggest("us")) == 1 + + +def test_caches_suggestions_case_insensitively(settings, store): + session = FakeSession(FakeResponse(suggest_payload({"title": "Banana"}))) + client = build(settings, session, cache=store) + + client.suggest("Ban") + client.suggest("ban") + + assert len(session.requests) == 1 + + +def test_an_empty_prefix_asks_nothing(settings): + session = FakeSession() + assert build(settings, session).suggest(" ") == [] + assert session.requests == [] + + +# --- Link context ------------------------------------------------------------- + + +def wikitext_payload(text: str) -> dict: + return { + "query": { + "pages": [ + {"title": "A", "revisions": [{"slots": {"main": {"content": text}}}]} + ] + } + } + + +def test_link_context_reads_the_article_wikitext(settings, store): + session = FakeSession(FakeResponse(wikitext_payload("A links to [[B]] here."))) + client = build(settings, session, cache=store) + + context = client.link_context("A", "B") + again = client.link_context("A", "B") + + assert (context.before, context.link, context.after) == ( + "A links to ", + "B", + " here.", + ) + assert again == context + assert len(session.requests) == 1, "the second lookup comes from the cache" + + +def test_link_context_of_a_missing_article_is_elsewhere(settings): + session = FakeSession(FakeResponse({"query": {"pages": [{"missing": True}]}})) + assert build(settings, session).link_context("Gone", "B").where == "elsewhere" diff --git a/tests/test_wikitext.py b/tests/test_wikitext.py new file mode 100644 index 0000000..67b6109 --- /dev/null +++ b/tests/test_wikitext.py @@ -0,0 +1,92 @@ +"""Finding the sentence that carries a link, from raw wikitext.""" + +from __future__ import annotations + +from app.wikitext import MAX_SENTENCE, find_link_context + +ARTICLE = """{{Short description|Edible fruit}} +{{Infobox plant +| name = Banana +| region = [[Southeast Asia]] +}} +A '''banana''' is an elongated, edible [[fruit]].{{cite web|title=x}} \ +It is grown in [[India|the Indian subcontinent]] and elsewhere. Bananas vary in size. + +== Taxonomy == +Linnaeus named [[Musa (genus)|''Musa'']] and placed [[plantain]]s in it. + +{| class="wikitable" +| [[Kerala]] || yields +|} +""" + + +def sentence(context) -> str: + return context.before + context.link + context.after + + +def test_finds_the_sentence_and_marks_the_linked_words(): + context = find_link_context(ARTICLE, "Fruit") + + assert context.where == "prose" + assert context.link == "fruit" + assert sentence(context) == "A banana is an elongated, edible fruit." + assert context.section is None + + +def test_a_piped_link_marks_the_words_shown_not_the_title(): + context = find_link_context(ARTICLE, "India") + + assert context.link == "the Indian subcontinent" + assert sentence(context) == ( + "It is grown in the Indian subcontinent and elsewhere." + ) + + +def test_letters_trailing_a_link_belong_to_it(): + assert find_link_context(ARTICLE, "Plantain").link == "plantains" + + +def test_reports_the_section_the_sentence_sits_in(): + context = find_link_context(ARTICLE, "Musa (genus)") + + assert context.section == "Taxonomy" + assert context.link == "Musa" + assert "