Skip to content

Add Jupyter JupySQL tutorial for querying Pinot - #19374

Open
codewithsruthi wants to merge 2 commits into
apache:masterfrom
codewithsruthi:issue-10160-jupysql-tutorial
Open

Add Jupyter JupySQL tutorial for querying Pinot#19374
codewithsruthi wants to merge 2 commits into
apache:masterfrom
codewithsruthi:issue-10160-jupysql-tutorial

Conversation

@codewithsruthi

@codewithsruthi codewithsruthi commented Aug 26, 2026

Copy link
Copy Markdown

What this PR does

Adds a first-party Jupyter tutorial so Python / data-science users can query Apache Pinot from a notebook with JupySQL and pinotdb.

closes: #10160

That issue asked for three things that did not exist in the repo or the GitBook Python client page:

  1. Query Pinot via SQL (notebook magics)
  2. Plot results from SQL
  3. Use Pinot as part of Jupyter for EDA / modeling (keep a pandas DataFrame; this PR does not train a model)

The existing Python client docs only show DB-API / SQLAlchemy snippets. Dash / Streamlit / Redash tutorials exist; there was no Jupyter + JupySQL walkthrough.

Files

All new files are under contrib/jupyter-jupysql/ so GitHub Actions paths-ignore: contrib/** does not start the Java unit / integration / quickstart matrix.

File Role
contrib/jupyter-jupysql/README.md How to start batch quickstart (local ./build/bin/quick-start-batch.sh or Docker), install deps, run or headlessly execute the notebook
contrib/jupyter-jupysql/requirements.txt pinotdb, jupysql, pandas, matplotlib, jupyter, sqlalchemy, ipykernel, nbconvert
contrib/jupyter-jupysql/pinot_jupysql_eda.ipynb Runnable notebook against the baseballStats batch-quickstart table

Notebook contents

  • Apache 2.0 license in the first markdown cell
  • Connect with a SQLAlchemy engine and %sql engine (JupySQL’s reliable path for non-standard URLs):
    pinot://localhost:8000/query/sql?controller=http://localhost:9000/
  • connect_args={"use_multistage_engine": "true"} so JupySQL %sqlplot CTEs are accepted by Pinot
  • SQL: SELECT … LIMIT, GROUP BY / ORDER BY on playerName, teamID, yearID, SUM(runs)
  • Plotting: pandas/matplotlib bar chart and %sqlplot
  • %matplotlib inline so figures actually render in Jupyter (plt.show() is silent without it)
  • Last cell keeps a DataFrame for downstream EDA

Port: batch and Docker quickstarts expose the broker SQL API on 8000 (controller UI 9000). Older python.md snippets used 8099; the README calls that out.

Companion docs PR

GitBook lives in pinot-contrib/pinot-docs, not this repo:

pinot-contrib/pinot-docs#1010

That PR adds tutorials/getting-started/jupyter-jupysql.md and links it from the tutorials TOC and the Python client page.

Testing

No new Java tests. CI for this repo ignores contrib/**, so unit / integration / quickstart workflows are not expected to run (and “no checks” on this branch is expected).

Manual verification against a local batch quickstart (./build/bin/quick-start-batch.sh):

  • Controller GET /health on :9000OK
  • Broker POST /query/sql on :8000: SELECT count(*) FROM baseballStats97889 rows, no exceptions
  • jupyter nbconvert --execute pinot_jupysql_eda.ipynb: 7 code cells, 0 errors, 2 PNG plots
  • Same four SQL statements via SQLAlchemy/pinotdb (no Jupyter magics): all returned the expected row counts (5 / 10 / 10 / 20)

To reproduce:

./build/bin/quick-start-batch.sh   # wait until :9000 and :8000 are up
cd contrib/jupyter-jupysql
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
jupyter nbconvert --to notebook --execute pinot_jupysql_eda.ipynb \
  --output /tmp/pinot_jupysql_eda.executed.ipynb

Was generative AI tooling used to co-author this PR?
  • Yes — Cursor Grok 4.6

Generated-by: Cursor Grok 4.6

codewithsruthi and others added 2 commits August 26, 2026 21:52
Data scientists need a local notebook path for SQL EDA and plots against the batch quickstart, not only the DB-API snippets on the Python client page.

Co-authored-by: Cursor <cursoragent@cursor.com>
Jupyter does not render plt.show() without the inline backend, so the plotting section would look empty in a normal notebook session.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codewithsruthi
codewithsruthi marked this pull request as draft August 26, 2026 22:36
@codewithsruthi
codewithsruthi marked this pull request as ready for review August 27, 2026 21:15
@Jackie-Jiang Jackie-Jiang added the pinot-client Related to Pinot client libraries label Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pinot-client Related to Pinot client libraries

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FR] Jupysql integration

2 participants