From 30aa854e2a6105be3572965ddfe2add8869ff4a7 Mon Sep 17 00:00:00 2001 From: Sruthi Date: Wed, 26 Aug 2026 21:52:45 +0000 Subject: [PATCH 1/2] Add Jupyter JupySQL tutorial for querying Pinot. Data scientists need a local notebook path for SQL EDA and plots against the batch quickstart, not only the DB-API snippets on the Python client page. Co-authored-by: Cursor --- contrib/jupyter-jupysql/README.md | 76 +++++++ .../jupyter-jupysql/pinot_jupysql_eda.ipynb | 214 ++++++++++++++++++ contrib/jupyter-jupysql/requirements.txt | 25 ++ 3 files changed, 315 insertions(+) create mode 100644 contrib/jupyter-jupysql/README.md create mode 100644 contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb create mode 100644 contrib/jupyter-jupysql/requirements.txt diff --git a/contrib/jupyter-jupysql/README.md b/contrib/jupyter-jupysql/README.md new file mode 100644 index 000000000000..1626b799bb2b --- /dev/null +++ b/contrib/jupyter-jupysql/README.md @@ -0,0 +1,76 @@ + + +# Query Pinot from Jupyter with JupySQL + +This example shows how to query Apache Pinot from a Jupyter notebook using +[JupySQL](https://jupysql.ploomber.io/) and the [pinotdb](https://pypi.org/project/pinotdb/) +Python client. It is meant for local EDA: SQL magics, pandas DataFrames, and simple plots. + +Related issue: https://github.com/apache/pinot/issues/10160 + +## Prerequisites + +A running batch quickstart (loads the `baseballStats` table). The broker SQL +endpoint is **port 8000** (not 8099, which appears in some older client snippets). +The controller UI is port 9000. + +### Option A — local binary (this checkout) + +From the Pinot repo root, after `./mvnw clean install -DskipTests -Pbin-dist`: + +```bash +./build/bin/quick-start-batch.sh +``` + +### Option B — Docker + +```bash +docker run --name pinot-quickstart \ + -p 2123:2123 -p 9000:9000 -p 8000:8000 \ + -d apachepinot/pinot:latest QuickStart -type batch +``` + +Wait until the controller UI at http://localhost:9000 is up. + +## Run the notebook + +```bash +cd contrib/jupyter-jupysql +python3 -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +jupyter notebook pinot_jupysql_eda.ipynb +``` + +Connection string used in the notebook (broker **8000**, controller **9000**). +The engine is created with `use_multistage_engine=true` so JupySQL `%sqlplot` +CTEs are accepted: + +```text +pinot://localhost:8000/query/sql?controller=http://localhost:9000/ +``` + +To execute all cells headlessly (quickstart must already be running): + +```bash +jupyter nbconvert --to notebook --execute pinot_jupysql_eda.ipynb --output pinot_jupysql_eda.executed.ipynb +``` diff --git a/contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb b/contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb new file mode 100644 index 000000000000..9c262fa90bc0 --- /dev/null +++ b/contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb @@ -0,0 +1,214 @@ +{ + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "id": "license", + "metadata": {}, + "source": [ + "Licensed to the Apache Software Foundation (ASF) under one\n", + "or more contributor license agreements. See the NOTICE file\n", + "distributed with this work for additional information\n", + "regarding copyright ownership. The ASF licenses this file\n", + "to you under the Apache License, Version 2.0 (the\n", + "\"License\"); you may not use this file except in compliance\n", + "with the License. You may obtain a copy of the License at\n", + "\n", + " http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing,\n", + "software distributed under the License is distributed on an\n", + "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n", + "KIND, either express or implied. See the License for the\n", + "specific language governing permissions and limitations\n", + "under the License." + ] + }, + { + "cell_type": "markdown", + "id": "intro", + "metadata": {}, + "source": [ + "# Query Apache Pinot from Jupyter with JupySQL\n", + "\n", + "This notebook queries a local Pinot **batch quickstart** from Jupyter using\n", + "[JupySQL](https://jupysql.ploomber.io/) SQL magics and [pinotdb](https://pypi.org/project/pinotdb/).\n", + "\n", + "It covers:\n", + "\n", + "1. Connecting to Pinot from a notebook\n", + "2. Running SQL (`SELECT`, `GROUP BY`, `ORDER BY`)\n", + "3. Plotting query results\n", + "4. Keeping results as a pandas DataFrame for later EDA or modeling\n", + "\n", + "**Start Pinot first** (broker on port **8000**, controller on **9000**):\n", + "\n", + "```bash\n", + "./build/bin/quick-start-batch.sh\n", + "```\n", + "\n", + "or\n", + "\n", + "```bash\n", + "docker run --name pinot-quickstart -p 2123:2123 -p 9000:9000 -p 8000:8000 -d apachepinot/pinot:latest QuickStart -type batch\n", + "```\n", + "\n", + "The quickstart loads `baseballStats`. See `README.md` in this directory for install steps." + ] + }, + { + "cell_type": "code", + "id": "imports", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from sqlalchemy import create_engine\n", + "import matplotlib.pyplot as plt\n", + "\n", + "%load_ext sql\n", + "\n", + "%config SqlMagic.autopandas = True\n", + "%config SqlMagic.feedback = False\n", + "%config SqlMagic.displaycon = False\n", + "\n", + "# Batch / Docker quickstart: broker 8000, controller 9000 (not 8099).\n", + "# Multi-stage is required for JupySQL %sqlplot, which rewrites plots as CTEs.\n", + "engine = create_engine(\n", + " \"pinot://localhost:8000/query/sql?controller=http://localhost:9000/\",\n", + " connect_args={\"use_multistage_engine\": \"true\"},\n", + ")\n", + "%sql engine" + ] + }, + { + "cell_type": "markdown", + "id": "query-sql", + "metadata": {}, + "source": [ + "## Query Pinot with SQL magics\n", + "\n", + "`%%sql` sends the statement to the Pinot broker (`POST /query/sql`).\n", + "Use `LIMIT` on exploratory scans. Aggregations on `baseballStats` are cheap." + ] + }, + { + "cell_type": "code", + "id": "select-limit", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "%%sql\n", + "SELECT playerName, teamID, yearID, runs, homeRuns\n", + "FROM baseballStats\n", + "LIMIT 5" + ] + }, + { + "cell_type": "code", + "id": "group-by", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "%%sql\n", + "SELECT playerName, SUM(runs) AS sum_runs\n", + "FROM baseballStats\n", + "WHERE yearID >= 2000\n", + "GROUP BY playerName\n", + "ORDER BY sum_runs DESC\n", + "LIMIT 10" + ] + }, + { + "cell_type": "markdown", + "id": "plot-md", + "metadata": {}, + "source": [ + "## Plot query results\n", + "\n", + "Assign a `%sql` result to a variable. With `SqlMagic.autopandas = True` you get a\n", + "DataFrame you can plot with matplotlib (or pass into `%sqlplot`)." + ] + }, + { + "cell_type": "code", + "id": "plot-code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "top_teams = %sql SELECT teamID, SUM(runs) AS total_runs FROM baseballStats GROUP BY teamID ORDER BY total_runs DESC LIMIT 10\n", + "\n", + "ax = top_teams.plot.bar(x=\"teamID\", y=\"total_runs\", legend=False)\n", + "ax.set_title(\"Top 10 teams by total runs (baseballStats)\")\n", + "ax.set_xlabel(\"teamID\")\n", + "ax.set_ylabel(\"total runs\")\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "id": "sqlplot", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "%%sql --save top_teams_sql\n", + "SELECT teamID, SUM(runs) AS total_runs\n", + "FROM baseballStats\n", + "GROUP BY teamID\n", + "ORDER BY total_runs DESC\n", + "LIMIT 10" + ] + }, + { + "cell_type": "code", + "id": "sqlplot-bar", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "%sqlplot bar --table top_teams_sql --column teamID" + ] + }, + { + "cell_type": "markdown", + "id": "eda-md", + "metadata": {}, + "source": [ + "## Keep results for EDA or modeling\n", + "\n", + "The DataFrame is a normal pandas object. Use it for further EDA or as features\n", + "for a model — training a model is out of scope for this tutorial." + ] + }, + { + "cell_type": "code", + "id": "eda-code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "player_runs = %sql SELECT playerName, SUM(runs) AS sum_runs, SUM(homeRuns) AS sum_hr FROM baseballStats WHERE yearID >= 2000 GROUP BY playerName ORDER BY sum_runs DESC LIMIT 20\n", + "\n", + "print(player_runs.dtypes)\n", + "player_runs.head()" + ] + } + ] +} diff --git a/contrib/jupyter-jupysql/requirements.txt b/contrib/jupyter-jupysql/requirements.txt new file mode 100644 index 000000000000..06d783b68de7 --- /dev/null +++ b/contrib/jupyter-jupysql/requirements.txt @@ -0,0 +1,25 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +pinotdb>=9.1.0 +jupysql>=0.10.0 +pandas>=2.0 +matplotlib>=3.8 +jupyter>=1.0 +sqlalchemy>=2.0 +ipykernel>=6.0 +nbconvert>=7.0 From 2f5b301342536f4dbcd8cbafc6660150a008b8ab Mon Sep 17 00:00:00 2001 From: Sruthi Date: Wed, 26 Aug 2026 22:34:33 +0000 Subject: [PATCH 2/2] Show matplotlib figures inline in the JupySQL tutorial notebook. Jupyter does not render plt.show() without the inline backend, so the plotting section would look empty in a normal notebook session. Co-authored-by: Cursor --- .../jupyter-jupysql/pinot_jupysql_eda.ipynb | 107 +++++++++--------- 1 file changed, 54 insertions(+), 53 deletions(-) diff --git a/contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb b/contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb index 9c262fa90bc0..35a18088c246 100644 --- a/contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb +++ b/contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb @@ -1,21 +1,7 @@ { - "nbformat": 4, - "nbformat_minor": 5, - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "name": "python", - "pygments_lexer": "ipython3" - } - }, "cells": [ { "cell_type": "markdown", - "id": "license", "metadata": {}, "source": [ "Licensed to the Apache Software Foundation (ASF) under one\n", @@ -34,11 +20,11 @@ "KIND, either express or implied. See the License for the\n", "specific language governing permissions and limitations\n", "under the License." - ] + ], + "id": "license" }, { "cell_type": "markdown", - "id": "intro", "metadata": {}, "source": [ "# Query Apache Pinot from Jupyter with JupySQL\n", @@ -66,18 +52,17 @@ "```\n", "\n", "The quickstart loads `baseballStats`. See `README.md` in this directory for install steps." - ] + ], + "id": "intro" }, { "cell_type": "code", - "id": "imports", "metadata": {}, - "execution_count": null, - "outputs": [], "source": [ "from sqlalchemy import create_engine\n", "import matplotlib.pyplot as plt\n", "\n", + "%matplotlib inline\n", "%load_ext sql\n", "\n", "%config SqlMagic.autopandas = True\n", @@ -91,38 +76,38 @@ " connect_args={\"use_multistage_engine\": \"true\"},\n", ")\n", "%sql engine" - ] + ], + "execution_count": null, + "outputs": [], + "id": "imports" }, { "cell_type": "markdown", - "id": "query-sql", "metadata": {}, "source": [ "## Query Pinot with SQL magics\n", "\n", "`%%sql` sends the statement to the Pinot broker (`POST /query/sql`).\n", "Use `LIMIT` on exploratory scans. Aggregations on `baseballStats` are cheap." - ] + ], + "id": "query-sql" }, { "cell_type": "code", - "id": "select-limit", "metadata": {}, - "execution_count": null, - "outputs": [], "source": [ "%%sql\n", "SELECT playerName, teamID, yearID, runs, homeRuns\n", "FROM baseballStats\n", "LIMIT 5" - ] + ], + "execution_count": null, + "outputs": [], + "id": "select-limit" }, { "cell_type": "code", - "id": "group-by", "metadata": {}, - "execution_count": null, - "outputs": [], "source": [ "%%sql\n", "SELECT playerName, SUM(runs) AS sum_runs\n", @@ -131,25 +116,25 @@ "GROUP BY playerName\n", "ORDER BY sum_runs DESC\n", "LIMIT 10" - ] + ], + "execution_count": null, + "outputs": [], + "id": "group-by" }, { "cell_type": "markdown", - "id": "plot-md", "metadata": {}, "source": [ "## Plot query results\n", "\n", "Assign a `%sql` result to a variable. With `SqlMagic.autopandas = True` you get a\n", "DataFrame you can plot with matplotlib (or pass into `%sqlplot`)." - ] + ], + "id": "plot-md" }, { "cell_type": "code", - "id": "plot-code", "metadata": {}, - "execution_count": null, - "outputs": [], "source": [ "top_teams = %sql SELECT teamID, SUM(runs) AS total_runs FROM baseballStats GROUP BY teamID ORDER BY total_runs DESC LIMIT 10\n", "\n", @@ -159,14 +144,14 @@ "ax.set_ylabel(\"total runs\")\n", "plt.tight_layout()\n", "plt.show()" - ] + ], + "execution_count": null, + "outputs": [], + "id": "plot-code" }, { "cell_type": "code", - "id": "sqlplot", "metadata": {}, - "execution_count": null, - "outputs": [], "source": [ "%%sql --save top_teams_sql\n", "SELECT teamID, SUM(runs) AS total_runs\n", @@ -174,41 +159,57 @@ "GROUP BY teamID\n", "ORDER BY total_runs DESC\n", "LIMIT 10" - ] + ], + "execution_count": null, + "outputs": [], + "id": "sqlplot" }, { "cell_type": "code", - "id": "sqlplot-bar", "metadata": {}, - "execution_count": null, - "outputs": [], "source": [ "%sqlplot bar --table top_teams_sql --column teamID" - ] + ], + "execution_count": null, + "outputs": [], + "id": "sqlplot-bar" }, { "cell_type": "markdown", - "id": "eda-md", "metadata": {}, "source": [ "## Keep results for EDA or modeling\n", "\n", "The DataFrame is a normal pandas object. Use it for further EDA or as features\n", "for a model — training a model is out of scope for this tutorial." - ] + ], + "id": "eda-md" }, { "cell_type": "code", - "id": "eda-code", "metadata": {}, - "execution_count": null, - "outputs": [], "source": [ "player_runs = %sql SELECT playerName, SUM(runs) AS sum_runs, SUM(homeRuns) AS sum_hr FROM baseballStats WHERE yearID >= 2000 GROUP BY playerName ORDER BY sum_runs DESC LIMIT 20\n", "\n", "print(player_runs.dtypes)\n", "player_runs.head()" - ] + ], + "execution_count": null, + "outputs": [], + "id": "eda-code" } - ] -} + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file