docs: convert reStructuredText sources to MyST markdown (#1579)

* docs: convert restructuredText sources to MyST markdown

Phase 2 of the documentation-site refresh. Run `rst2myst convert` over
every human-authored .rst file under docs/source/ and remove the
originals. The result:

- 33 .rst files become 33 .md files (user guide, contributor guide,
  index, links).
- Headings, paragraphs, hyperlinks, code blocks, admonitions, and
  toctree directives all map cleanly to MyST syntax.
- Cross-reference anchors round-trip through MyST as `(label)=`
  blocks. The converter kebab-cased the labels (e.g. `(io-csv)=`),
  but every `{ref}` target in the corpus still uses the underscore
  form from the original RST (`{ref}\`CSV <io_csv>\``) and so do the
  Python docstrings that AutoAPI pulls in. Rewrite the anchors back
  to the underscore form so the existing references resolve.
- 86 `{eval-rst}` blocks remain — they all wrap `.. ipython::`
  directives, which have no first-class MyST equivalent. They render
  identically and don't block the build.

conf.py changes:

- Enable `colon_fence` and `deflist` MyST extensions (rst-to-myst
  emits these on a few files, particularly execution-metrics.md).
- Keep `.rst` in `source_suffix` even though no human-authored RST
  remains: sphinx-autoapi generates RST under autoapi/ at build time
  and Sphinx needs the suffix registered to parse it.

AGENTS.md: update the two .rst paths called out under "Aggregate and
Window Function Documentation" to point at the .md equivalents.

Verified by building locally — `build succeeded`, no warnings, all
internal cross-references resolve, the ipython examples on the
landing page and basics page still execute.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: fix Apache license header format in converted markdown files

RST-to-MD conversion emitted MyST `%` comment syntax with blank line
between each header line, which renders as visible text. Replace with
canonical `<!--- ... -->` HTML comment block matching upstream
apache/datafusion and this repo's existing markdown files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: fix broken cross-reference links in distributing-work

The RST -> MyST conversion left two intra-page links as undefined
reference-style links, which CommonMark renders as literal bracketed
text (no Sphinx warning, so the --fail-on-warning build still passed).
Point both at the auto-generated heading anchors instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: execute examples via myst-nb; native tables and validated refs

Removes the last RST-syntax islands from the converted MyST markdown so
the docs are markdown-native for both human and LLM authors.

Executable examples (A): replace IPython.sphinxext.ipython_directive with
myst-nb. The 83 `{eval-rst}` + `.. ipython:: python` blocks become native
`{code-cell} ipython3` blocks, and the 14 pages that carry them gain
jupytext/kernelspec front matter so myst-nb runs them. conf.py routes .md
through myst-nb with nb_execution_mode="force" and
nb_execution_raise_on_error=True, so a failing example now fails the build.

myst-nb gives each page its own kernel instead of the IPython directive's
single namespace shared across all documents in build order. That isolation
surfaced expressions.md, which only ever worked by inheriting `col`/`lit`
from an earlier-built page — it now imports them itself. It also changes the
execution working directory to each page's own folder, so build.sh symlinks
the example data next to every page that reads it by relative name and
registers the python3 kernel; CI now calls build.sh so it matches local.

Tables (B): the 3 `.. list-table::` directives become GFM markdown tables.

Cross-references (C): the two intra-page links in distributing-work.md that
the conversion left as undefined markdown references (and that built green
while rendering literal brackets) become `{ref}` roles backed by explicit
`(label)=` targets, so a future break fails the build instead of shipping
silently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: render DataFrame cell outputs as text, not the HTML widget

myst-nb prefers a cell's `_repr_html_` over its text repr. A datafusion
DataFrame's HTML repr is a Jupyter-oriented widget — inline styles plus an
injected <script> — that renders at the wrong width in the docs theme.

Set nb_mime_priority_overrides so the html builder prefers text/plain. The
35 cells that end in a bare DataFrame now show the same readable ASCII
table the old IPython directive produced, with no per-cell `.show()` edits
and no dependence on the package-generated HTML staying theme-compatible.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(aggregations): use .alias() on grouping(), drop obsolete workaround

apache/datafusion#21411 is resolved — `.alias()` now works directly on a
`grouping()` expression. Removed the note describing the limitation and the
with_column_renamed workaround in the rollup and grouping_sets examples,
aliasing the grouping columns inline instead. Verified on the current
branch: the aliased aggregates execute and produce the named columns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: use a dark-mode variant of the logo

The header logo was the same SVG in both color modes; the light-colored
wordmark was hard to read on the dark theme. Point the theme's image_dark
at a new original_dark.svg whose wordmark uses light strokes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: restore right-hand on-this-page TOC, collapsible

The theme refresh emptied secondary_sidebar_items, dropping the
on-this-page table of contents that the previous site showed. Bring it
back on the right, wrapped in a native <details> so readers can fold it
away on the longer guide pages. Adds a custom page-toc-collapsible
secondary-sidebar template and styles the <summary> toggle (no JS).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: let readers hide the right TOC sidebar for full-width content

Follow-up to restoring the on-this-page TOC: "collapsible" should hide the
entire right-hand frame, not just fold the list. Replace the <details>
wrapper with a floating toggle button (toc-toggle.js) that hides the whole
secondary sidebar via a body class; the flex article container then
reclaims the width (its 60em cap is lifted while hidden). The preference is
remembered across pages in localStorage, and the button is suppressed below
the theme's breakpoint where the sidebar is already collapsed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(deps): pin typing-extensions to one version so uv.lock parses in CI

Adding the myst-nb docs stack pulled a newer typing-extensions only on
Python < 3.11, splitting it into two locked versions. Our own
`typing-extensions; python_full_version < '3.13'` dependency then spanned
that split, which uv recorded as a multi-version edge without a `version`
field — a form older uv builds (the one in CI's pinned setup-uv) reject
with "missing source field but has more than one matching package".

Add a [tool.uv] constraint-dependencies pin of typing-extensions>=4.15.0
so it resolves to a single version across all supported Pythons, removing
the fork and the under-specified edge. Relocked; uv lock --locked is clean
and no multi-version package has a marker-only edge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(deps): drop pickleshare and explicit ipython from docs group

Both were only needed by the old IPython.sphinxext.ipython_directive,
which myst-nb replaced. pickleshare (IPython %store, abandoned 2018) has
no remaining consumer. ipython is now pulled transitively by ipykernel
and myst-nb, so the explicit floor is redundant. Relocked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Apply reviewer's suggestion to fix CI error

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: timsaucer

.github/workflows/build.yml

@@ -552,9 +552,10 @@ jobs:
         run: |
           set -x
           cd docs
-          curl -O https://gist.githubusercontent.com/ritchie46/cac6b337ea52281aa23c049250a4ff03/raw/89a957ff3919d90e6ef2d34235e6bf22304f3366/pokemon.csv
-          curl -O https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet
-          uv run --no-project make html
+          # build.sh downloads the example data, registers the Jupyter kernel
+          # myst-nb needs, symlinks the data next to each executed page, and
+          # runs sphinx. Using it here keeps CI identical to a local build.
+          uv run --no-project bash ./build.sh
 
       - name: Copy & push the generated HTML
         if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref_type == 'tag')

AGENTS.md

@@ -84,9 +84,9 @@ Every Python function must include a docstring with usage examples.
 When adding or updating an aggregate or window function, ensure the corresponding
 site documentation is kept in sync:
 
-- **Aggregations**: `docs/source/user-guide/common-operations/aggregations.rst` —
+- **Aggregations**: `docs/source/user-guide/common-operations/aggregations.md` —
   add new aggregate functions to the "Aggregate Functions" list and include usage
   examples if appropriate.
-- **Window functions**: `docs/source/user-guide/common-operations/windows.rst` —
+- **Window functions**: `docs/source/user-guide/common-operations/windows.md` —
   add new window functions to the "Available Functions" list and include usage
   examples if appropriate.

Cargo.lock

@@ -36,6 +36,21 @@ rm -rf build 2> /dev/null
 rm -rf temp 2> /dev/null
 mkdir temp
 cp -rf source/* temp/
+
+# myst-nb executes each page as a notebook from the directory that page
+# lives in, so the example data files must sit alongside every page that
+# loads them by relative name (e.g. `ctx.read_csv("pokemon.csv")`). Symlink
+# them into each directory that has such a page rather than copying the
+# 20 MB parquet repeatedly.
+for d in temp temp/user-guide temp/user-guide/common-operations; do
+    ln -sf "$script_dir/pokemon.csv" "$d/pokemon.csv"
+    ln -sf "$script_dir/yellow_tripdata_2021-01.parquet" "$d/yellow_tripdata_2021-01.parquet"
+done
+
+# myst-nb runs `{code-cell}` blocks against a Jupyter kernel named "python3".
+# Register the active environment's interpreter as that kernel (idempotent).
+python -m ipykernel install --sys-prefix --name python3 --display-name "Python 3"
+
 make SOURCEDIR=`pwd`/temp html
 
 cd "$original_dir" || exit

docs/build.sh

@@ -63,3 +63,48 @@ html[data-theme="dark"] .table tbody tr:nth-of-type(odd) {
         white-space: normal !important;
     }
 }
+
+
+/* Hideable right-hand "On this page" sidebar.
+ * toc-toggle.js adds the button and toggles `pst-secondary-hidden` on <body>;
+ * hiding the sidebar lets the flex article container reclaim the width. */
+
+body.pst-secondary-hidden .bd-sidebar-secondary {
+  display: none;
+}
+
+/* Let the article use the freed space rather than just re-centering. */
+body.pst-secondary-hidden .bd-article-container {
+  max-width: none;
+}
+
+/* Floating toggle button, pinned to the top-right under the navbar. */
+#pst-secondary-toggle {
+  position: fixed;
+  top: 4.5rem;
+  right: 0.75rem;
+  z-index: 1020;
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  width: 2rem;
+  height: 2rem;
+  padding: 0;
+  border: 1px solid var(--pst-color-border, #ccc);
+  border-radius: 0.25rem;
+  background-color: var(--pst-color-surface, #fff);
+  color: var(--pst-color-text-base, #333);
+  cursor: pointer;
+}
+
+#pst-secondary-toggle:hover {
+  color: rgb(var(--pst-color-link-hover));
+}
+
+/* The toggle is only meaningful where the sidebar is shown (wide screens);
+ * below the theme's lg breakpoint the sidebar is already collapsed away. */
+@media (max-width: 959.98px) {
+  #pst-secondary-toggle {
+    display: none;
+  }
+}

docs/source/_static/images/original_dark.svg

@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/* Adds a button that hides the right-hand "On this page" sidebar so the
+ * article can use the full page width. The choice is remembered across
+ * pages via localStorage. */
+(function () {
+  "use strict";
+  var KEY = "pst-secondary-hidden";
+
+  function apply(hidden, btn) {
+    document.body.classList.toggle("pst-secondary-hidden", hidden);
+    if (btn) {
+      btn.setAttribute("aria-pressed", String(hidden));
+      btn.title = hidden ? "Show page contents" : "Hide page contents";
+    }
+  }
+
+  document.addEventListener("DOMContentLoaded", function () {
+    // Only offer the toggle on pages that actually have the sidebar.
+    if (!document.querySelector(".bd-sidebar-secondary")) {
+      return;
+    }
+
+    var btn = document.createElement("button");
+    btn.id = "pst-secondary-toggle";
+    btn.type = "button";
+    btn.setAttribute("aria-label", "Toggle page contents sidebar");
+    btn.innerHTML = '<i class="fa-solid fa-list" aria-hidden="true"></i>';
+    document.body.appendChild(btn);
+
+    btn.addEventListener("click", function () {
+      var hidden = !document.body.classList.contains("pst-secondary-hidden");
+      try {
+        localStorage.setItem(KEY, hidden ? "1" : "0");
+      } catch (e) {
+        /* localStorage may be unavailable; toggle still works for this page. */
+      }
+      apply(hidden, btn);
+    });
+
+    var stored = null;
+    try {
+      stored = localStorage.getItem(KEY);
+    } catch (e) {
+      /* ignore */
+    }
+    apply(stored === "1", btn);
+  });
+})();

docs/source/_static/theme_overrides.css

@@ -48,16 +48,40 @@
 extensions = [
     "sphinx.ext.mathjax",
     "sphinx.ext.napoleon",
-    "myst_parser",
-    "IPython.sphinxext.ipython_directive",
+    # myst_nb is a superset of myst_parser: it provides the MyST markdown
+    # parser plus executable `{code-cell}` notebook directives. Do NOT also
+    # list "myst_parser" — myst_nb activates it internally and listing both
+    # raises an extension conflict.
+    "myst_nb",
     "autoapi.extension",
 ]
 
+# NOTE: .rst stays alongside .md because sphinx-autoapi generates RST
+# under autoapi/ and Sphinx needs the suffix to parse it. The human-
+# authored docs are all MyST .md now. ".md" is routed through myst-nb so
+# pages carrying jupytext/kernelspec front matter execute their
+# `{code-cell}` blocks; pages without that front matter render as plain
+# MyST markdown. The ".rst" entry is only for the autoapi build artifacts.
 source_suffix = {
     ".rst": "restructuredtext",
-    ".md": "markdown",
+    ".md": "myst-nb",
 }
 
+# Execute notebook code cells at build time and fail the build if any cell
+# raises — this replaces the old IPython sphinx directive, whose executed
+# examples are now `{code-cell}` blocks. "force" re-executes every build so
+# stale cached output can never ship.
+nb_execution_mode = "force"
+nb_execution_timeout = 120
+nb_execution_raise_on_error = True
+
+# Prefer the plain-text repr of a cell's last expression over its rich
+# `_repr_html_`. A DataFrame's HTML repr is a self-contained widget (inline
+# styles + an injected <script>) built for Jupyter; in the docs theme it
+# renders at the wrong width. The text repr is the readable table the old
+# IPython directive showed and is stable across datafusion versions.
+nb_mime_priority_overrides = [("html", "text/plain", 0)]
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 
@@ -120,7 +144,7 @@ def setup(sphinx) -> None:
     "show_toc_level": 2,
     "logo": {
         "image_light": "_static/images/original.svg",
-        "image_dark": "_static/images/original.svg",
+        "image_dark": "_static/images/original_dark.svg",
         "alt_text": "Apache DataFusion in Python",
     },
     "navbar_start": ["navbar-logo"],
@@ -138,7 +162,10 @@ def setup(sphinx) -> None:
             "icon": "fa-brands fa-rust",
         },
     ],
-    "secondary_sidebar_items": [],
+    # Right-hand "On this page" TOC. A toggle button (added by
+    # _static/toc-toggle.js) lets the reader hide the whole sidebar and give
+    # the article full width.
+    "secondary_sidebar_items": ["page-toc"],
     "collapse_navigation": True,
     "show_nav_level": 2,
 }
@@ -164,12 +191,20 @@ def setup(sphinx) -> None:
 
 html_css_files = ["theme_overrides.css"]
 
+# Adds a button that hides the right-hand "On this page" sidebar so the
+# article can use the full width (see _static/toc-toggle.js).
+html_js_files = ["toc-toggle.js"]
+
 html_sidebars = {
     "**": ["sidebar-globaltoc.html"],
 }
 
 # tell myst_parser to auto-generate anchor links for headers h1, h2, h3
 myst_heading_anchors = 3
 
-# enable nice rendering of checkboxes for the task lists
-myst_enable_extensions = ["tasklist"]
+# MyST extensions:
+# - tasklist: GitHub-style `- [x]` checkboxes
+# - colon_fence: `:::{directive}` blocks (needed by execution-metrics.md
+#   after the RST -> MyST conversion)
+# - deflist: definition lists (used in a couple of converted pages)
+myst_enable_extensions = ["tasklist", "colon_fence", "deflist"]