docs: add sustainability benchmarking tutorial - #711
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
zinichakraborty
changed the title
| @@ -0,0 +1,433 @@ | |||
| { | |||
There was a problem hiding this comment.
Would it be possible to align the structure and getting started section with this tutorial (https://docs.pruna.ai/en/stable/docs_pruna/tutorials/reasoning_llm.html)? I mean, using the table and referencing the installation page if needed.
Reply via ReviewNB
| @@ -0,0 +1,433 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,433 @@ | |||
| { | |||
There was a problem hiding this comment.
Could you add a short sentence about this stable-difussion model and why it's used?
Reply via ReviewNB
| @@ -0,0 +1,433 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,433 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,433 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,433 @@ | |||
| { | |||
There was a problem hiding this comment.
sdiazlor
left a comment
sdiazlor
left a comment
There was a problem hiding this comment.
Overall, a really nice tutorial. Just some comments to improve the readability :)
590e2bc to
77cf274
Compare
|
Made the edits mentioned and added back in the figure outputs. Let me know if there are any other changes I can make! |
begumcig
left a comment
begumcig
left a comment
There was a problem hiding this comment.
This is truly amazing work thank you so much @zinichakraborty! I absolutely have no notes, and I think it's super ready to merge.
I noticed that you used 3 for your warm up iterations, have you ever experimented with other numbers? Sometimes with torch.compile, you may need to run the model a bit more to see the full benefits. I think 3 is probably good enough already, and this doesn't affect the clarity of the tutorial at all, but saying incase you will be using these numbers in your work :)
Thank you so much again for the tutorial!! 👑👑👑
|
Thank you I will try this! I set the number low for efficiency, but this is good to know. |
|
@zinichakraborty Thank you so much for the contribution, I'll merge now :) If you have any feedback using pruna (features, docs, usability, etc.), feel free to share here or on our Discord :) |
3 participants
Description
Adds a tutorial notebook demonstrating sustainability benchmarking with the
EvaluationAgent: astable-diffusion-v1-5baseline is compared againstdeepcacheandtorch_compileontotal_time,energy_consumed,co2_emissions, andclip_score, using a 10-sample subset ofLAION256on a T4. Includes side-by-side comparison figures and a takeaway on interpreting efficiency gains vs. quality cost (with a pointer to the CMMD tutorial for stricter quality evaluation). Also registers the notebook indocs/tutorials/index.rst.Related Issue
Fixes #602
Type of Change
Testing
uv run pytest -m "cpu and not slow")No library code is changed, so no unit tests were added. The notebook was executed end-to-end on a Colab T4 (pruna 0.3.4) with all cells completing; outputs are stripped per the convention in the existing tutorials. Comparison figures from the run are attached below for review.
ruff checkpasses on the notebook with the repo config.Checklist