diff --git a/Cargo.toml b/Cargo.toml index 8febee6..af1afaf 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "pythonstl" -version = "0.1.4" +version = "1.1.10" edition = "2021" description = "Rust backend extension for pythonstl data structures" diff --git a/PUBLISHING.md b/PUBLISHING.md index bdb90db..501cdc3 100644 --- a/PUBLISHING.md +++ b/PUBLISHING.md @@ -8,7 +8,7 @@ Install required tools: ```bash pip install --upgrade pip -pip install build twine +pip install maturin twine build ``` ## Step 1: Clean Previous Builds @@ -17,23 +17,30 @@ Remove any existing build artifacts: ```bash # Windows PowerShell -Remove-Item -Recurse -Force dist, build, *.egg-info -ErrorAction SilentlyContinue +Remove-Item -Recurse -Force dist, target, build, *.egg-info -ErrorAction SilentlyContinue # Linux/Mac -rm -rf dist build *.egg-info +rm -rf dist target build *.egg-info ``` ## Step 2: Build the Distribution -Build both source distribution and wheel: +For hybrid Python/Rust packages using Maturin: + +```bash +# Build wheel and sdist with maturin +maturin build --release +``` + +Or using `python -m build` (which invokes maturin as the build backend defined in `pyproject.toml`): ```bash python -m build ``` -This creates: -- `dist/pythonstl-0.1.0.tar.gz` (source distribution) -- `dist/pythonstl-0.1.0-py3-none-any.whl` (wheel) +This creates distribution artifacts in `target/wheels/` or `dist/`: +- `dist/pythonstl-1.1.9.tar.gz` (source distribution) +- `dist/pythonstl-1.1.9-*.whl` (compiled wheel) ## Step 3: Validate the Build diff --git a/README.md b/README.md index a1ae782..daa49cb 100644 --- a/README.md +++ b/README.md @@ -10,12 +10,12 @@ pythonstl-removebg-preview
-A Python package that replicates C++ STL-style data structures using the **Facade Design Pattern**. PythonSTL provides clean, familiar interfaces for developers coming from C++ while maintaining Pythonic best practices. +A Python package that replicates C++ STL-style data structures using a modular **Containers**, **Engines**, and **Utility** architecture. PythonSTL provides clean, familiar interfaces for developers coming from C++ while maintaining Pythonic best practices. ## Features - **C++ STL Compliance**: Exact method names and semantics matching C++ STL -- **Facade Design Pattern**: Clean separation between interface and implementation +- **Modular Layered Architecture**: Clean separation between containers, execution engines, and utilities - **Iterator Support**: STL-style iterators (begin, end, rbegin, rend) and Python iteration - **Python Integration**: Magic methods (__len__, __bool__, __contains__, __repr__, __eq__) - **Type Safety**: Full type hints throughout the codebase @@ -259,22 +259,22 @@ Container adapter providing priority-based access. ## 🏗️ Architecture -PythonSTL follows the **Facade Design Pattern** with three layers: +PythonSTL is structured into three distinct, specialized layers (**Containers**, **Engines**, **Utility**): -1. **Core Layer** (`pythonstl/core/`) - - Base classes and type definitions - - Custom exceptions - - Iterator classes +1. **Containers Layer** (`pythonstl/containers/`) + - Public-facing STL-compliant container classes and algorithms + - Dispatches operations to native Rust or Python engines + - Clean, C++ STL-compliant API -2. **Implementation Layer** (`pythonstl/implementations/`) - - Private implementation classes (prefixed with `_`) - - Efficient use of Python built-ins +2. **Engines Layer** (`pythonstl/engines/`) + - Internal execution and storage engine implementations (prefixed with `_`) + - Linear, associative, and heap structure engines - Not intended for direct user access -3. **Facade Layer** (`pythonstl/facade/`) - - Public-facing classes - - Clean, STL-compliant API - - Delegates to implementation layer +3. **Utility Layer** (`pythonstl/utility/`) + - Common utilities, base classes, and type definitions + - Custom exception types + - Iterator implementations and AVL tree building blocks This architecture ensures: - **Encapsulation**: Internal implementation is hidden @@ -299,11 +299,11 @@ def thread_safe_push(value): ## Design Decisions -### Why Facade Pattern? +### Why Containers, Engines & Utility? -- **Clean API**: Users interact with simple, well-defined interfaces -- **Flexibility**: Internal implementation can change without affecting users -- **Type Safety**: Facade layer enforces type contracts +- **Clean API**: Users interact with simple, well-defined container interfaces +- **Flexibility**: Internal storage and execution engines can change without affecting users +- **Type Safety**: Containers layer enforces type contracts - **Error Handling**: Consistent error messages across all containers ### Why STL Naming? @@ -460,4 +460,4 @@ Contributions are welcome! Please: - Issues: [GitHub Issues](https://github.com/AnshMNSoni/PythonSTL/issues) - Linkedin: [@anshmnsoni](https://linkedin.com/in/anshmnsoni) -**PythonSTL v1.1.9** - Bringing C++ STL elegance to Python +**PythonSTL v1.1.10** - Bringing C++ STL elegance to Python diff --git a/benchmarks/benchmark_algorithms.py b/benchmarks/benchmark_algorithms.py index 802e926..71fbd16 100644 --- a/benchmarks/benchmark_algorithms.py +++ b/benchmarks/benchmark_algorithms.py @@ -9,7 +9,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1])) from pythonstl import next_permutation, nth_element, partition, stl_set -from pythonstl.facade.algorithms import RUST_AVAILABLE +from pythonstl.containers.algorithms import RUST_AVAILABLE def run_py_permutation(): arr = [1, 2, 3, 4, 5, 6, 7, 8, 9] diff --git a/benchmarks/benchmark_all_structures.py b/benchmarks/benchmark_all_structures.py index e7428f8..c566d00 100644 --- a/benchmarks/benchmark_all_structures.py +++ b/benchmarks/benchmark_all_structures.py @@ -13,12 +13,12 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1])) from pythonstl import stack, queue, vector, stl_set, stl_map, priority_queue -from pythonstl.facade.stack import RUST_AVAILABLE as STACK_RUST_AVAILABLE -from pythonstl.facade.queue import RUST_AVAILABLE as QUEUE_RUST_AVAILABLE -from pythonstl.facade.vector import RUST_AVAILABLE as VECTOR_RUST_AVAILABLE -from pythonstl.facade.set import RUST_AVAILABLE as SET_RUST_AVAILABLE -from pythonstl.facade.map import RUST_AVAILABLE as MAP_RUST_AVAILABLE -from pythonstl.facade.priority_queue import RUST_AVAILABLE as PQ_RUST_AVAILABLE +from pythonstl.containers.stack import RUST_AVAILABLE as STACK_RUST_AVAILABLE +from pythonstl.containers.queue import RUST_AVAILABLE as QUEUE_RUST_AVAILABLE +from pythonstl.containers.vector import RUST_AVAILABLE as VECTOR_RUST_AVAILABLE +from pythonstl.containers.set import RUST_AVAILABLE as SET_RUST_AVAILABLE +from pythonstl.containers.map import RUST_AVAILABLE as MAP_RUST_AVAILABLE +from pythonstl.containers.priority_queue import RUST_AVAILABLE as PQ_RUST_AVAILABLE def run_benchmark(name, py_func, rust_func, has_rust, native_func): diff --git a/benchmarks/benchmark_binary_search.py b/benchmarks/benchmark_binary_search.py index 3794766..158a4cd 100644 --- a/benchmarks/benchmark_binary_search.py +++ b/benchmarks/benchmark_binary_search.py @@ -9,7 +9,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1])) from pythonstl import lower_bound -from pythonstl.facade.algorithms import RUST_AVAILABLE +from pythonstl.containers.algorithms import RUST_AVAILABLE def run_py_binary_search(arr, targets): sum_indices = 0 diff --git a/benchmarks/benchmark_rust_vs_py.py b/benchmarks/benchmark_rust_vs_py.py index 7a99969..a3f7a08 100644 --- a/benchmarks/benchmark_rust_vs_py.py +++ b/benchmarks/benchmark_rust_vs_py.py @@ -7,7 +7,7 @@ # Add project root to path to run directly from development folder sys.path.insert(0, str(Path(__file__).resolve().parents[1])) -from pythonstl.facade.stack import stack, RUST_AVAILABLE +from pythonstl.containers.stack import stack, RUST_AVAILABLE # Try importing the bubble_sort function from the compiled Rust library try: diff --git a/pyproject.toml b/pyproject.toml index 23214ba..e686308 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,7 +8,7 @@ module-name = "pythonstl._rust" [project] name = "pythonstl" -version = "1.1.9" +version = "1.1.10" description = "C++ STL-style containers implemented in Python using the Facade Design Pattern" readme = "README.md" authors = [ @@ -38,7 +38,7 @@ Issues = "https://github.com/AnshMNSoni/STL/issues" Documentation = "https://github.com/AnshMNSoni/STL#readme" [tool.setuptools] -packages = ["pythonstl", "pythonstl.core", "pythonstl.implementations", "pythonstl.implementations.linear", "pythonstl.implementations.associative", "pythonstl.implementations.heaps", "pythonstl.facade"] +packages = ["pythonstl", "pythonstl.utility", "pythonstl.engines", "pythonstl.engines.linear", "pythonstl.engines.associative", "pythonstl.engines.heaps", "pythonstl.containers"] [tool.setuptools.package-data] pythonstl = ["py.typed"] diff --git a/pythonstl/__init__.py b/pythonstl/__init__.py index e9e41a4..93407bc 100644 --- a/pythonstl/__init__.py +++ b/pythonstl/__init__.py @@ -8,16 +8,16 @@ data structures while hiding implementation details from users. """ -__version__ = "1.1.9" +__version__ = "1.1.10" __author__ = "PySTL Contributors" -from pythonstl.facade.stack import stack -from pythonstl.facade.queue import queue -from pythonstl.facade.vector import vector -from pythonstl.facade.set import stl_set -from pythonstl.facade.map import stl_map -from pythonstl.facade.priority_queue import priority_queue -from pythonstl.facade.algorithms import ( +from pythonstl.containers.stack import stack +from pythonstl.containers.queue import queue +from pythonstl.containers.vector import vector +from pythonstl.containers.set import stl_set +from pythonstl.containers.map import stl_map +from pythonstl.containers.priority_queue import priority_queue +from pythonstl.containers.algorithms import ( next_permutation, prev_permutation, nth_element, @@ -29,7 +29,7 @@ ) # Also export exceptions for user error handling -from pythonstl.core.exceptions import ( +from pythonstl.utility.exceptions import ( PySTLException, EmptyContainerError, OutOfRangeError, diff --git a/pythonstl/_rust.pdb b/pythonstl/_rust.pdb index 5350cc4..adc2c2b 100644 Binary files a/pythonstl/_rust.pdb and b/pythonstl/_rust.pdb differ diff --git a/pythonstl/facade/__init__.py b/pythonstl/containers/__init__.py similarity index 83% rename from pythonstl/facade/__init__.py rename to pythonstl/containers/__init__.py index c4adf81..2adfb8c 100644 --- a/pythonstl/facade/__init__.py +++ b/pythonstl/containers/__init__.py @@ -1,4 +1,4 @@ -"""Facade layer for pystl data structures.""" +"""Containers layer for pystl data structures.""" from .stack import stack from .queue import queue diff --git a/pythonstl/facade/algorithms.py b/pythonstl/containers/algorithms.py similarity index 100% rename from pythonstl/facade/algorithms.py rename to pythonstl/containers/algorithms.py diff --git a/pythonstl/facade/map.py b/pythonstl/containers/map.py similarity index 97% rename from pythonstl/facade/map.py rename to pythonstl/containers/map.py index 3eb9232..7fcad56 100644 --- a/pythonstl/facade/map.py +++ b/pythonstl/containers/map.py @@ -6,9 +6,9 @@ from typing import TypeVar, Iterator as TypingIterator, Tuple from copy import deepcopy -from pythonstl.core.exceptions import KeyNotFoundError -from pythonstl.implementations.associative._map_impl import _MapImpl -from pythonstl.core.iterator import MapIterator +from pythonstl.utility.exceptions import KeyNotFoundError +from pythonstl.engines.associative._map_impl import _MapImpl +from pythonstl.utility.iterator import MapIterator try: from pythonstl._rust import RustMap diff --git a/pythonstl/facade/priority_queue.py b/pythonstl/containers/priority_queue.py similarity index 97% rename from pythonstl/facade/priority_queue.py rename to pythonstl/containers/priority_queue.py index 016aefe..21c0d98 100644 --- a/pythonstl/facade/priority_queue.py +++ b/pythonstl/containers/priority_queue.py @@ -6,8 +6,8 @@ from typing import TypeVar from copy import deepcopy -from pythonstl.core.exceptions import EmptyContainerError -from pythonstl.implementations.heaps._priority_queue_impl import _PriorityQueueImpl +from pythonstl.utility.exceptions import EmptyContainerError +from pythonstl.engines.heaps._priority_queue_impl import _PriorityQueueImpl try: from pythonstl._rust import RustPriorityQueue diff --git a/pythonstl/facade/queue.py b/pythonstl/containers/queue.py similarity index 97% rename from pythonstl/facade/queue.py rename to pythonstl/containers/queue.py index 494c7c7..5fcc24c 100644 --- a/pythonstl/facade/queue.py +++ b/pythonstl/containers/queue.py @@ -6,8 +6,8 @@ from typing import TypeVar from copy import deepcopy -from pythonstl.core.exceptions import EmptyContainerError -from pythonstl.implementations.linear._queue_impl import _QueueImpl +from pythonstl.utility.exceptions import EmptyContainerError +from pythonstl.engines.linear._queue_impl import _QueueImpl try: from pythonstl._rust import RustQueue diff --git a/pythonstl/facade/set.py b/pythonstl/containers/set.py similarity index 98% rename from pythonstl/facade/set.py rename to pythonstl/containers/set.py index c1e8efd..2519bda 100644 --- a/pythonstl/facade/set.py +++ b/pythonstl/containers/set.py @@ -6,8 +6,8 @@ from typing import TypeVar, Iterator as TypingIterator from copy import deepcopy -from pythonstl.implementations.associative._set_impl import _SetImpl -from pythonstl.core.iterator import SetIterator +from pythonstl.engines.associative._set_impl import _SetImpl +from pythonstl.utility.iterator import SetIterator try: from pythonstl._rust import RustSet diff --git a/pythonstl/facade/stack.py b/pythonstl/containers/stack.py similarity index 97% rename from pythonstl/facade/stack.py rename to pythonstl/containers/stack.py index 95f336f..1b0c6c9 100644 --- a/pythonstl/facade/stack.py +++ b/pythonstl/containers/stack.py @@ -6,8 +6,8 @@ from typing import TypeVar from copy import deepcopy -from pythonstl.core.exceptions import EmptyContainerError -from pythonstl.implementations.linear._stack_impl import _StackImpl +from pythonstl.utility.exceptions import EmptyContainerError +from pythonstl.engines.linear._stack_impl import _StackImpl try: from pythonstl._rust import RustStack diff --git a/pythonstl/facade/vector.py b/pythonstl/containers/vector.py similarity index 97% rename from pythonstl/facade/vector.py rename to pythonstl/containers/vector.py index 0408492..f032c5b 100644 --- a/pythonstl/facade/vector.py +++ b/pythonstl/containers/vector.py @@ -6,9 +6,9 @@ from typing import TypeVar, Iterator as TypingIterator from copy import deepcopy -from pythonstl.core.exceptions import EmptyContainerError, OutOfRangeError -from pythonstl.implementations.linear._vector_impl import _VectorImpl -from pythonstl.core.iterator import VectorIterator, VectorReverseIterator +from pythonstl.utility.exceptions import EmptyContainerError, OutOfRangeError +from pythonstl.engines.linear._vector_impl import _VectorImpl +from pythonstl.utility.iterator import VectorIterator, VectorReverseIterator try: from pythonstl._rust import RustVector diff --git a/pythonstl/implementations/__init__.py b/pythonstl/engines/__init__.py similarity index 100% rename from pythonstl/implementations/__init__.py rename to pythonstl/engines/__init__.py diff --git a/pythonstl/implementations/associative/__init__.py b/pythonstl/engines/associative/__init__.py similarity index 100% rename from pythonstl/implementations/associative/__init__.py rename to pythonstl/engines/associative/__init__.py diff --git a/pythonstl/implementations/associative/_map_impl.py b/pythonstl/engines/associative/_map_impl.py similarity index 95% rename from pythonstl/implementations/associative/_map_impl.py rename to pythonstl/engines/associative/_map_impl.py index 4ad4681..15e8a14 100644 --- a/pythonstl/implementations/associative/_map_impl.py +++ b/pythonstl/engines/associative/_map_impl.py @@ -6,9 +6,9 @@ """ from typing import TypeVar, Dict -from pythonstl.core.exceptions import KeyNotFoundError -from pythonstl.core.iterator import MapIterator -from pythonstl.core.avl_tree import AVLTree +from pythonstl.utility.exceptions import KeyNotFoundError +from pythonstl.utility.iterator import MapIterator +from pythonstl.utility.avl_tree import AVLTree K = TypeVar('K') V = TypeVar('V') diff --git a/pythonstl/implementations/associative/_set_impl.py b/pythonstl/engines/associative/_set_impl.py similarity index 96% rename from pythonstl/implementations/associative/_set_impl.py rename to pythonstl/engines/associative/_set_impl.py index ae7be9e..7e55a50 100644 --- a/pythonstl/implementations/associative/_set_impl.py +++ b/pythonstl/engines/associative/_set_impl.py @@ -6,8 +6,8 @@ """ from typing import TypeVar, List -from pythonstl.core.iterator import SetIterator -from pythonstl.core.avl_tree import AVLTree +from pythonstl.utility.iterator import SetIterator +from pythonstl.utility.avl_tree import AVLTree T = TypeVar('T') diff --git a/pythonstl/implementations/heaps/__init__.py b/pythonstl/engines/heaps/__init__.py similarity index 100% rename from pythonstl/implementations/heaps/__init__.py rename to pythonstl/engines/heaps/__init__.py diff --git a/pythonstl/implementations/heaps/_priority_queue_impl.py b/pythonstl/engines/heaps/_priority_queue_impl.py similarity index 98% rename from pythonstl/implementations/heaps/_priority_queue_impl.py rename to pythonstl/engines/heaps/_priority_queue_impl.py index 28b6ba2..3f5c4ac 100644 --- a/pythonstl/implementations/heaps/_priority_queue_impl.py +++ b/pythonstl/engines/heaps/_priority_queue_impl.py @@ -7,7 +7,7 @@ from typing import TypeVar, List import heapq -from pythonstl.core.exceptions import EmptyContainerError +from pythonstl.utility.exceptions import EmptyContainerError T = TypeVar('T') diff --git a/pythonstl/implementations/linear/__init__.py b/pythonstl/engines/linear/__init__.py similarity index 100% rename from pythonstl/implementations/linear/__init__.py rename to pythonstl/engines/linear/__init__.py diff --git a/pythonstl/implementations/linear/_queue_impl.py b/pythonstl/engines/linear/_queue_impl.py similarity index 97% rename from pythonstl/implementations/linear/_queue_impl.py rename to pythonstl/engines/linear/_queue_impl.py index 50aa903..4aa937c 100644 --- a/pythonstl/implementations/linear/_queue_impl.py +++ b/pythonstl/engines/linear/_queue_impl.py @@ -7,7 +7,7 @@ from typing import TypeVar from collections import deque -from pythonstl.core.exceptions import EmptyContainerError +from pythonstl.utility.exceptions import EmptyContainerError T = TypeVar('T') diff --git a/pythonstl/implementations/linear/_stack_impl.py b/pythonstl/engines/linear/_stack_impl.py similarity index 97% rename from pythonstl/implementations/linear/_stack_impl.py rename to pythonstl/engines/linear/_stack_impl.py index c60248d..dbb34a5 100644 --- a/pythonstl/implementations/linear/_stack_impl.py +++ b/pythonstl/engines/linear/_stack_impl.py @@ -6,7 +6,7 @@ """ from typing import TypeVar, List -from pythonstl.core.exceptions import EmptyContainerError +from pythonstl.utility.exceptions import EmptyContainerError T = TypeVar('T') diff --git a/pythonstl/implementations/linear/_vector_impl.py b/pythonstl/engines/linear/_vector_impl.py similarity index 97% rename from pythonstl/implementations/linear/_vector_impl.py rename to pythonstl/engines/linear/_vector_impl.py index f758976..34398d6 100644 --- a/pythonstl/implementations/linear/_vector_impl.py +++ b/pythonstl/engines/linear/_vector_impl.py @@ -6,8 +6,8 @@ """ from typing import TypeVar, List -from pythonstl.core.exceptions import EmptyContainerError, OutOfRangeError -from pythonstl.core.iterator import VectorIterator, VectorReverseIterator +from pythonstl.utility.exceptions import EmptyContainerError, OutOfRangeError +from pythonstl.utility.iterator import VectorIterator, VectorReverseIterator T = TypeVar('T') diff --git a/pythonstl/core/__init__.py b/pythonstl/utility/__init__.py similarity index 100% rename from pythonstl/core/__init__.py rename to pythonstl/utility/__init__.py diff --git a/pythonstl/core/avl_tree.py b/pythonstl/utility/avl_tree.py similarity index 100% rename from pythonstl/core/avl_tree.py rename to pythonstl/utility/avl_tree.py diff --git a/pythonstl/core/base.py b/pythonstl/utility/base.py similarity index 100% rename from pythonstl/core/base.py rename to pythonstl/utility/base.py diff --git a/pythonstl/core/exceptions.py b/pythonstl/utility/exceptions.py similarity index 100% rename from pythonstl/core/exceptions.py rename to pythonstl/utility/exceptions.py diff --git a/pythonstl/core/iterator.py b/pythonstl/utility/iterator.py similarity index 100% rename from pythonstl/core/iterator.py rename to pythonstl/utility/iterator.py diff --git a/pythonstl_report.tex b/pythonstl_report.tex new file mode 100644 index 0000000..d2e325e --- /dev/null +++ b/pythonstl_report.tex @@ -0,0 +1,788 @@ +\documentclass[11pt,a4paper]{article} +\usepackage[utf8]{inputenc} +\usepackage{amsmath,amsfonts,amssymb} +\usepackage{geometry} +\geometry{margin=1in} +\usepackage{booktabs} +\usepackage{listings} +\usepackage{xcolor} +\usepackage{hyperref} +\usepackage{tikz} +\usetikzlibrary{shapes.geometric, arrows, positioning} + +\definecolor{codegreen}{rgb}{0,0.6,0} +\definecolor{codegray}{rgb}{0.5,0.5,0.5} +\definecolor{codepurple}{rgb}{0.58,0,0.82} +\definecolor{backcolour}{rgb}{0.95,0.95,0.92} + +\lstdefinestyle{mystyle}{ + backgroundcolor=\color{backcolour}, + commentstyle=\color{codegreen}, + keywordstyle=\color{blue}, + numberstyle=\tiny\color{codegray}, + stringstyle=\color{codepurple}, + basicstyle=\ttfamily\small, + breakatwhitespace=false, + breaklines=true, + captionpos=b, + keepspaces=true, + numbers=left, + numbersep=5pt, + showspaces=false, + showstringspaces=false, + showtabs=false, + tabsize=4 +} +\lstset{style=mystyle} + +\hypersetup{ + colorlinks=true, + linkcolor=blue, + filecolor=magenta, + urlcolor=cyan, + pdftitle={PythonSTL Technical Project Report}, + pdfpagemode=FullScreen, +} + +\title{\textbf{PythonSTL: Replicating C++ Standard Template Library Semantics via a Dual-Engine Python-Rust Hybrid System}} +\author{\textbf{Technical Project Report \& Comprehensive Reference Guide}} +\date{July 2026} + +\begin{document} + +\maketitle + +\begin{abstract} +This report provides a detailed technical analysis of \texttt{PythonSTL}, a python library replicating C++ Standard Template Library (STL) containers and algorithms. Utilizing a dual-engine architecture, \texttt{PythonSTL} provides both a pure Python fallback (using self-balancing AVL trees for associative structures) and an optimized Rust backend compiled via Maturin and PyO3 (using Rust B-trees). The library adheres to the Facade design pattern, offering standard C++ STL method signatures (\texttt{push\_back}, \texttt{top}, \texttt{insert}, \texttt{erase}) and STL-style iterators while integrating with native Python idioms. This document covers the motivations, architecture, low-level implementations, performance profile, and strategic interview preparation topics associated with the project. +\end{abstract} + +\begin{center} +\small\textbf{Project Understanding in One Line} +\end{center} +\begin{quote} +\small\noindent\textit{PythonSTL is a dual-engine library replicating C++ STL container semantics and algorithms in Python, dynamically delegating operations between a pure-Python fallback (using AVL trees) and an optimized, compiled Rust backend (using B-trees via PyO3/Maturin) to showcase hybrid architecture patterns and FFI optimization trade-offs.} +\end{quote} + + +\newpage +\tableofcontents +\newpage + +\section{Overview \& Motivation} + +\subsection{Overview} +\texttt{PythonSTL} is a high-performance Python package that replicates standard C++ Standard Template Library (STL) data structures and algorithms. The project features clean, typed, and familiar interfaces for developers with a C++ background, while maintaining standard Pythonic protocols. + +It provides six core containers: +\begin{itemize} + \item \textbf{stack}: LIFO (Last-In-First-Out) container adapter. + \item \textbf{queue}: FIFO (First-In-First-Out) container adapter. + \item \textbf{vector}: Dynamic array with explicit capacity controls (\texttt{reserve}, \texttt{shrink\_to\_fit}). + \item \textbf{stl\_set}: Sorted associative container storing unique elements. + \item \textbf{stl\_map}: Sorted associative container mapping keys to values. + \item \textbf{priority\_queue}: Max-heap or Min-heap with support for custom elements. +\end{itemize} + +Additionally, it exports a suite of C++ algorithm replicas (\texttt{next\_permutation}, \texttt{prev\_permutation}, \texttt{nth\_element}, \texttt{partition}, \texttt{lower\_bound}, \texttt{upper\_bound}, \texttt{binary\_search}, and \texttt{equal\_range}). + +\subsection{Motivation} +The creation of \texttt{PythonSTL} was driven by several academic and practical factors: + +\begin{enumerate} + \item \textbf{Absence of Sorted Associative Containers in Python}: + Python's built-in \texttt{set} and \texttt{dict} types are based on hash tables. While they offer average $\mathcal{O}(1)$ time complexity for insertions and lookups, they are inherently unordered. In contrast, C++'s \texttt{std::set} and \texttt{std::map} are ordered collections (traditionally implemented as Red-Black Trees) offering logarithmic $\mathcal{O}(\log N)$ complexity but keeping keys in sorted order. If a Python developer needs to maintain a sorted collection dynamically while executing range queries, they must manually re-sort lists ($\mathcal{O}(N \log N)$) or rely on third-party libraries. \texttt{PythonSTL} bridges this gap. + + \item \textbf{Customizable Priority Queues}: + Python's standard library provides the \texttt{heapq} module, which operates directly on standard Python lists. However, \texttt{heapq} is strictly a min-heap implementation and lacks an elegant mechanism for custom element comparison. Implementing a max-heap or custom sorting rules with \texttt{heapq} requires wrapping elements in decorator classes that override comparison operators, which is boilerplate-heavy. \texttt{PythonSTL} offers out-of-the-box support for both \texttt{max} and \texttt{min} heaps, alongside custom comparator callables. + + \item \textbf{Mental-Model Bridge for C++ Developers}: + Developers transitioning from C++ to Python often struggle with the divergence in data structure design. Providing an exact mapping of C++ STL containers allows engineers to write algorithm prototypes using familiar interfaces (\texttt{begin()}, \texttt{end()}, \texttt{push\_back()}, etc.) before translating them to idiomatic Python. + + \item \textbf{Hybrid System Engineering Demonstration}: + The library serves as a robust educational template for modern hybrid software systems. It demonstrates how to write clean, type-safe Python facades while delegating heavy computational workloads to compiled Rust libraries using PyO3 and Maturin, highlighting the trade-offs of Foreign Function Interfaces (FFI). +\end{enumerate} + +\newpage + +\section{Project Demo / User Journey} + +\subsection{Installation and Integration} +\texttt{PythonSTL} is distributed via PyPI. Users can install it directly using pip: +\begin{lstlisting}[language=bash] +pip install pythonstl +\end{lstlisting} + +\subsection{Standard Usage Examples} +The user journey covers importing the containers, instantiating them, running operations, and exploiting Python integration features: + +\begin{lstlisting}[language=Python] +from pythonstl import stack, queue, vector, stl_set, stl_map, priority_queue + +# --- Stack LIFO Adapter --- +s = stack() +s.push(10) +s.push(20) +print(s.top()) # Output: 20 +s.pop() +print(len(s)) # Output: 1 (len() support) +print(bool(s)) # Output: True (bool() support) + +# --- Vector Sequence Container --- +v = vector() +v.push_back(100) +v.push_back(200) +v.push_back(300) +v.reserve(1000) # Pre-allocates vector capacity +print(200 in v) # Output: True (__contains__ support) + +# Iteration via C++ STL style iterators +for elem in v.begin(): + print(elem) # Outputs 100, 200, 300 + +# Or iterate via Python iterators directly +for elem in v: + print(elem) + +# --- Set and Map Sorted Associative Containers --- +ss = stl_set() +ss.insert(15) +ss.insert(5) +ss.insert(10) +# stl_set maintains key order (in-order traversal yields sorted elements) +print(list(ss)) # Output: [5, 10, 15] + +sm = stl_map() +sm.insert("beta", 2) +sm.insert("alpha", 1) +for key, value in sm: + print(f"{key}: {value}") # Prints "alpha: 1" then "beta: 2" + +# --- Priority Queue with Comparator Support --- +pq_max = priority_queue(comparator="max") # Default max-heap +pq_min = priority_queue(comparator="min") # Min-heap + +for val in [30, 10, 20]: + pq_max.push(val) + pq_min.push(val) + +print(pq_max.top()) # Output: 30 +print(pq_min.top()) # Output: 10 +\end{lstlisting} + +\newpage + +\section{System Architecture} + +The system architecture follows the \textbf{Facade Design Pattern} to decouple the user-facing interface from the underlying execution engine. It comprises three core layers: + +\begin{enumerate} + \item \textbf{Utility Layer} (\texttt{pythonstl/utility/}): + Defines the baseline configurations, shared exceptions (\texttt{PySTLException}, \texttt{EmptyContainerError}, \texttt{OutOfRangeError}, \texttt{KeyNotFoundError}), and the core pointer-like STL-style iterator classes (\texttt{SetIterator}, \texttt{VectorIterator}, \texttt{VectorReverseIterator}). + + \item \textbf{Engines Layer} (\texttt{pythonstl/engines/}): + Contains the private, pure-Python logic classes (prefixed with an underscore, e.g., \texttt{\_SetImpl}, \texttt{\_VectorImpl}). This layer relies on native Python types and custom algorithms (like the self-balancing AVL tree) to implement the C++ container semantics without dependencies. + + \item \textbf{Containers Layer} (\texttt{pythonstl/containers/}): + Exposes public-facing wrapper classes (\texttt{stack}, \texttt{vector}, \texttt{stl\_set}, etc.). During instantiation, the container class dynamically checks if the compiled Rust backend module (\texttt{pythonstl.\_rust}) is imported successfully. If the Rust module is available and \texttt{use\_rust=True} is passed, it instantiates the corresponding Rust class compiled from source. Otherwise, it transparently falls back to the pure-Python implementation class. +\end{enumerate} + +\subsection{Architectural Schema} +The interaction layout of a user executing a command on \texttt{PythonSTL} is visualized in Figure~\ref{fig:architecture}: + +\begin{figure}[h!] +\centering +\begin{tikzpicture}[ + node distance=1.2cm and 0.8cm, + block/.style={rectangle, draw, fill=blue!5, text width=5cm, text centered, rounded corners, minimum height=0.8cm, font=\small}, + decision/.style={rectangle, draw, fill=orange!5, text width=4cm, text centered, rounded corners, minimum height=0.8cm, font=\scriptsize}, + leaf/.style={rectangle, draw, fill=green!5, text width=5cm, text centered, rounded corners, minimum height=0.8cm, font=\small}, + arrow/.style={thick, ->, >=stealth} +] + % Nodes + \node (user) [block, fill=gray!10] {\textbf{User Code}}; + \node (facade) [block, below=of user] {\textbf{Facade Layer} \\ (e.g., \texttt{stl\_set})}; + + % Branches (Decisions) + \node (rust_cond) [decision, below left=1.0cm and 0.2cm of facade] {\textbf{use\_rust = True} \\ \textbf{RUST\_AVAILABLE = True}}; + \node (py_cond) [decision, below right=1.0cm and 0.2cm of facade] {\textbf{use\_rust = False} \\ \textbf{or Rust Unavailable}}; + + % Engines + \node (rust_impl) [block, below=of rust_cond, fill=purple!5] {\textbf{Rust Compiled Binary} \\ (\texttt{\_rust.RustSet})}; + \node (py_impl) [block, below=of py_cond, fill=yellow!5] {\textbf{Pure-Python Fallback} \\ (\texttt{\_SetImpl} class)}; + + % Backends + \node (rust_backend) [leaf, below=of rust_impl] {\textbf{Rust std::collections} \\ (\texttt{BTreeSet})}; + \node (py_backend) [leaf, below=of py_impl] {\textbf{pythonstl/utility/} \\ (\texttt{AVLTree} container)}; + + % Arrows + \draw [arrow] (user) -- (facade); + \draw [arrow] (facade.south) -| (rust_cond.north); + \draw [arrow] (facade.south) -| (py_cond.north); + \draw [arrow] (rust_cond) -- (rust_impl); + \draw [arrow] (py_cond) -- (py_impl); + \draw [arrow] (rust_impl) -- (rust_backend); + \draw [arrow] (py_impl) -- (py_backend); + +\end{tikzpicture} +\caption{PythonSTL Hybrid Architecture and Backend Delegation} +\label{fig:architecture} +\end{figure} + + +\newpage + +\section{Complete Technical Deep Dive} + +\subsection{The Hybrid Compilation & Bindings Model} +The core performance engine is written in Rust and exposed as a C-extension module (\texttt{\_rust.pyd} on Windows or \texttt{\_rust.so} on Unix) using \textbf{PyO3} and the \textbf{Maturin} build system. + +When a Python script calls \texttt{from pythonstl import stl\_set}, the facade attempts to load the Rust extension: +\begin{lstlisting}[language=Python] +try: + from pythonstl._rust import RustSet + RUST_AVAILABLE = True +except ImportError: + RUST_AVAILABLE = False +\end{lstlisting} + +\subsection{The Rust-Python Bridge & GIL Management} +Because Rust is a compiled, statically typed language and Python is dynamically typed and garbage-collected, interacting across the Foreign Function Interface (FFI) boundary requires careful management of the Python \textbf{Global Interpreter Lock (GIL)} and object references. + +\subsubsection{PyObject and Reference Counting} +In the Rust code, all arbitrary Python data structures are represented as PyO3 \texttt{PyObject} instances. A \texttt{PyObject} acts as a smart pointer that increments the reference count of the underlying CPython heap object. When a \texttt{PyObject} is dropped in Rust, PyO3 automatically decrements the reference count. If Rust code executes a destructor without holding the GIL, it can trigger segmentation faults. PyO3 handles this by delaying the release or acquiring the GIL during drop actions. + +\subsubsection{Sorting Python Objects in Rust: The PyObjectOrd Pattern} +Rust's \texttt{std::collections::BTreeSet} and \texttt{BTreeMap} are ordered associative structures that require key types to implement the standard Rust comparison traits: \texttt{PartialEq}, \texttt{Eq}, \texttt{PartialOrd}, and \texttt{Ord}. + +However, \texttt{PyObject} does not implement these traits natively because comparing two Python objects requires acquiring the GIL and executing arbitrary Python code (e.g., calling \texttt{\_\_lt\_\_}). To solve this, \texttt{PythonSTL} implements the \texttt{PyObjectOrd} wrapper: + +\begin{lstlisting}[language=Rust] +struct PyObjectOrd(PyObject); + +impl PartialEq for PyObjectOrd { + fn eq(&self, other: &Self) -> bool { + Python::with_gil(|py| { + let self_ref = self.0.bind(py); + let other_ref = other.0.bind(py); + + // Fast-path comparisons for primitives to avoid interpreter callbacks + if let (Ok(a), Ok(b)) = (self_ref.extract::(), other_ref.extract::()) { + return a == b; + } + if let (Ok(a), Ok(b)) = (self_ref.extract::(), other_ref.extract::()) { + return a == b; + } + if let (Ok(a), Ok(b)) = (self_ref.extract::<&str>(), other_ref.extract::<&str>()) { + return a == b; + } + self_ref.eq(other_ref).unwrap_or(false) + }) + } +} + +impl Eq for PyObjectOrd {} + +impl Ord for PyObjectOrd { + fn cmp(&self, other: &Self) -> Ordering { + Python::with_gil(|py| { + let self_ref = self.0.bind(py); + let other_ref = other.0.bind(py); + + // Fast-path extraction for primitives + if let (Ok(a), Ok(b)) = (self_ref.extract::(), other_ref.extract::()) { + return a.cmp(&b); + } + if let (Ok(a), Ok(b)) = (self_ref.extract::(), other_ref.extract::()) { + if let Some(ord) = a.partial_cmp(&b) { + return ord; + } + } + if let (Ok(a), Ok(b)) = (self_ref.extract::<&str>(), other_ref.extract::<&str>()) { + return a.cmp(&b); + } + + // Fallback to calling Python comparison methods via FFI + if self_ref.eq(other_ref).unwrap_or(false) { + Ordering::Equal + } else if self_ref.lt(other_ref).unwrap_or(false) { + Ordering::Less + } else { + Ordering::Greater + } + }) + } +} + +impl PartialOrd for PyObjectOrd { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} +\end{lstlisting} + +\subsection{Analyzing the Fast-Path Primitive Comparison Optimization} +Without the primitive fast-paths (\texttt{extract::()}, \texttt{extract::()}, \texttt{extract::<\&str>()}), every single tree comparison would trigger a round-trip through CPython's rich comparison functions (\texttt{PyObject\_RichCompareBool}). This boundary crossing requires marshalling arguments, traversing internal CPython tables, and returning values. By checking if the object corresponds to basic primitive types (\texttt{int}, \texttt{float}, \texttt{str}), Rust extracts the raw bytes directly and evaluates the inequality natively in native assembly, bypassing Python VM execution completely. This logic reduces insertion and lookup time by up to 7x for large tree collections. + +\newpage + +\section{Algorithms \& Core Logic} + +\subsection{Pure-Python Balanced BST: The AVL Tree} +For the Python fallback engine, associative containers must maintain items in sorted order with logarithmic operations. \texttt{PythonSTL} implements a complete, self-balancing \textbf{AVL Tree} inside \texttt{pythonstl/utility/avl\_tree.py}. + +An AVL Tree maintains the invariant that the heights of the two child subtrees of any node differ by at most one: +\begin{equation} +| \text{height}(T_{\text{left}}) - \text{height}(T_{\text{right}}) | \le 1 +\end{equation} + +When an insertion or deletion breaks this invariant, the tree executes structural adjustments called \textbf{rotations} to restore balance. + +\subsubsection{Rotations Logic} +The tree uses four basic balancing operations, depending on where the imbalance occurs: + +\begin{enumerate} + \item \textbf{Left-Left (LL) Case} (Single Right Rotation): + Imbalance in the left child's left subtree. A single right rotation is executed on node $Y$: + \begin{lstlisting}[language=Python] + def _right_rotate(self, y: AVLNode[K, V]) -> AVLNode[K, V]: + x = y.left + T2 = x.right + x.right = y + y.left = T2 + y.height = max(self._get_height(y.left), self._get_height(y.right)) + 1 + x.height = max(self._get_height(x.left), self._get_height(x.right)) + 1 + return x + \end{lstlisting} + + \item \textbf{Right-Right (RR) Case} (Single Left Rotation): + Imbalance in the right child's right subtree. A single left rotation is executed on node $X$: + \begin{lstlisting}[language=Python] + def _left_rotate(self, x: AVLNode[K, V]) -> AVLNode[K, V]: + y = x.right + T2 = y.left + y.left = x + x.right = T2 + x.height = max(self._get_height(x.left), self._get_height(x.right)) + 1 + y.height = max(self._get_height(y.left), self._get_height(y.right)) + 1 + return y + \end{lstlisting} + + \item \textbf{Left-Right (LR) Case} (Left-Right Double Rotation): + Imbalance in the left child's right subtree. First, a left rotation is performed on the left child, then a right rotation on the parent node. + + \item \textbf{Right-Left (RL) Case} (Right-Left Double Rotation): + Imbalance in the right child's left subtree. First, a right rotation is performed on the right child, then a left rotation on the parent. +\end{enumerate} + +\subsection{Lexicographical Permutation Algorithms} +The \texttt{next\_permutation} algorithm is replicated in both Python and Rust: +\begin{enumerate} + \item Scan the list from right to left to locate the first pair of adjacent elements where $arr[i] < arr[i+1]$. This index $i$ is the pivot. + \item If no such index is found, the sequence is in descending order (largest permutation). Reverse the entire array in-place to return to the ascending sorted list and return \texttt{False}. + \item Otherwise, scan from right to left to find the first element at index $j$ that is strictly greater than the pivot ($arr[j] > arr[i]$). + \item Swap $arr[i]$ and $arr[j]$. + \item Reverse the suffix beginning at index $i+1$ to the end of the array. Return \texttt{True}. +\end{enumerate} + +\subsection{Quickselect & Pivot Safety for nth\_element} +The \texttt{nth\_element} algorithm reorganizes an array such that the item at index $nth$ is placed at its sorted position, partitioning the rest. It uses the \textbf{Quickselect} selection algorithm. + +A naive Lomuto partition scheme using the rightmost element as the pivot (\texttt{pivot = arr[right]}) degrades to $\mathcal{O}(N^2)$ time complexity when the input array is already sorted or reversed. \texttt{PythonSTL} ensures pivot safety by using a \textbf{Middle Pivot} selection strategy: +\begin{lstlisting}[language=Rust] +fn partition_q(arr: &mut Vec, left: usize, right: usize) -> usize { + let pivot_idx = left + (right - left) / 2; // Middle index + arr.swap(pivot_idx, right); + let mut i = left; + Python::with_gil(|py| { + let pivot_val = arr[right].clone_ref(py); + for j in left..right { + if pyobject_lt(py, &arr[j], &pivot_val).unwrap_or(false) { + arr.swap(i, j); + i += 1; + } + } + }); + arr.swap(i, right); + i +} +\end{lstlisting} +This modification reduces the execution time on sorted $10^6$ lists from 70.85 seconds to 0.0064 seconds. + +\newpage + +\section{Database Design} +N/A. \texttt{PythonSTL} is a client-side data-structure library and system package. It maintains all collections in-memory and does not interface with database systems. + +\section{APIs \& Backend Flow} + +\subsection{API Surface Comparison} +The public classes in \texttt{PythonSTL} match the C++ Standard Template Library method signatures. The table below illustrates the translation: + +\begin{table}[h!] +\centering +\begin{tabular}{@{}lll@{}} +\toprule +\textbf{C++ STL (e.g. std::vector)} & \textbf{PythonSTL facade} & \textbf{Equivalent Native Python} \\ \midrule +\texttt{v.push\_back(val)} & \texttt{v.push\_back(val)} & \texttt{lst.append(val)} \\ +\texttt{v.pop\_back()} & \texttt{v.pop\_back()} & \texttt{lst.pop()} \\ +\texttt{v.at(idx)} & \texttt{v.at(idx)} & \texttt{lst[idx]} \\ +\texttt{v.reserve(n)} & \texttt{v.reserve(n)} & N/A \\ +\texttt{v.shrink\_to\_fit()} & \texttt{v.shrink\_to\_fit()} & N/A \\ +\texttt{s.insert(val)} & \texttt{s.insert(val)} & \texttt{set\_obj.add(val)} \\ +\texttt{s.erase(val)} & \texttt{s.erase(val)} & \texttt{set\_obj.discard(val)} \\ +\texttt{pq.top()} & \texttt{pq.top()} & \texttt{heap[0]} \\ \bottomrule +\end{tabular} +\caption{API Signature Mapping} +\end{table} + +\subsection{Backend Delegation Flow} +When a user calls an API method on a facade, the wrapper class delegates the operation to the selected engine. Let's trace the backend execution flow of \texttt{stl\_set.insert(value)}: + +\begin{verbatim} +User Code: + s = stl_set(use_rust=True) + s.insert(42) + | + v +[Container Class (pythonstl/containers/set.py)] + def insert(self, value): + self._impl.insert(value) + | + +----------------------------+ + | If self._is_rust = True | If self._is_rust = False + v v +[Rust Extension (_rust.RustSet)] [Python Implementation (_SetImpl)] + fn insert(&mut self, val) def insert(self, value): + data.insert(PyObjectOrd(val)) self._data.add(value) + | | + v v +[Rust std::collections::BTreeSet] [AVLTree (pythonstl/utility/avl_tree.py)] + Stores PyObjectOrd wrapper AVLNode is allocated and balanced +\end{verbatim} + +\section{Frontend Flow} +N/A. \texttt{PythonSTL} is a core software library and CLI utility without a visual or web graphical user interface. + +\newpage + +\section{Authentication \& Security} + +\subsection{Authentication} +N/A. The package runs entirely in the local client process space and does not contain user authentication or access control systems. + +\subsection{Memory Safety & Security Concerns} +Although authentication is not applicable, the library implements systems-level protections against common memory and execution safety issues: + +\begin{enumerate} + \item \textbf{Type Safety and Bounds Checking}: + Unlike pure C++ where accessing an invalid vector element via indexing (\texttt{v[idx]}) can lead to undefined behavior or buffer overflow vulnerabilities, \texttt{PythonSTL}'s facade and Rust backends implement strict bounds checking. The \texttt{at()} method verifies index validity against the container size and raises an \texttt{OutOfRangeError} if bounds are violated: + \begin{lstlisting}[language=Python] + def at(self, index: int) -> T: + if index < 0 or index >= self._size: + raise OutOfRangeError("vector", index, self._size) + return self._impl.at(index) + \end{lstlisting} + + \item \textbf{GIL Safety and Thread-Safety Constraints}: + Python objects stored inside the Rust container backend are not thread-safe. Because they are represented as \texttt{PyObject} reference-counted pointers, mutating a container from multiple threads without synchronization violates Python's memory management rules. The library explicitly documents that containers are \textbf{not thread-safe} by default. In a multi-threaded application, users must lock collections using Python's locking mechanisms: + \begin{lstlisting}[language=Python] + import threading + from pythonstl import stack + + s = stack() + lock = threading.Lock() + + def safe_push(value): + with lock: + s.push(value) + \end{lstlisting} + + \item \textbf{Reference Counting and Garbage Collection Integration}: + To prevent memory leaks at the FFI boundary, the Rust backend is structured to participate in CPython's reference counting system. Every \texttt{PyObject} stored inside `RustVector`, `RustSet`, etc., is incremented during insertion. When elements are erased or the container itself is garbage collected, the drop implementations release the GIL and decrement the ref-count. This ensures that unused Python objects are freed from CPython's heap. +\end{enumerate} + +\newpage + +\section{Deployment \& DevOps} + +The release architecture uses standardized tools to automate testing, compilation of binary extensions, and package publication. + +\subsection{Build Configuration (pyproject.toml)} +The package uses Maturin as the build backend, configured using PEP 517 rules: +\begin{lstlisting}[language=TOML] +[build-system] +requires = ["maturin>=1.5,<2.0"] +build-backend = "maturin" + +[tool.maturin] +python-source = "." +module-name = "pythonstl._rust" + +[project] +name = "pythonstl" +version = "1.1.10" +requires-python = ">=3.10" +\end{lstlisting} + +\subsection{DevOps Publishing Pipeline} +The publication workflow is documented in \texttt{PUBLISHING.md} and follows the steps below: + +\begin{verbatim} +1. Code Quality Verification: + $ pytest && mypy pythonstl/ && flake8 pythonstl/ + | + v +2. Clean Previous Artifacts: + $ rm -rf dist build *.egg-info + | + v +3. Build Platform Wheels (via Maturin / cibuildwheel): + $ python -m build + | + v +4. Validate Distribution Archives: + $ twine check dist/* + | + v +5. Push to TestPyPI (Staging): + $ twine upload --repository testpypi dist/* + | + v +6. Test and Verify Sandbox Installation: + $ pip install --index-url https://test.pypi.org/simple/ pythonstl + | + v +7. Release to Production PyPI: + $ twine upload dist/* -u __token__ -p $PYPI_API_TOKEN +\end{verbatim} + +\newpage + +\section{Performance \& Scalability} + +\subsection{Performance Benchmarks} +Performance tests were executed comparing the Pure-Python fallback, the Rust-backed engine, and Python's native built-in structures as a baseline. + +\begin{table}[h!] +\centering +\begin{tabular}{@{}lllll@{}} +\toprule +\textbf{Container / Workload} & \textbf{Pure Py STL} & \textbf{Python+Rust} & \textbf{Built-in Baseline} & \textbf{Rust Speedup} \\ \midrule +Stack (1M push/pop) & 0.4768s & 0.3227s & 0.0530s (\texttt{list.pop}) & 1.48x \\ +Vector (10k push\_backs) & 0.2296s & 0.1374s & 0.0444s (\texttt{list.append}) & 1.67x \\ +Vector (10k random \texttt{at()}) & 0.4844s & 0.3264s & 0.0586s (\texttt{list[i]}) & 1.48x \\ +Map (10k insert - int) & 0.0873s & 0.0116s & 0.0019s (\texttt{dict[key]}) & 7.53x \\ +Map (10k find - int) & 0.0077s & 0.0046s & 0.0018s (\texttt{key in dict}) & 1.68x \\ +Standard Binary Search & 0.0182s & 0.0034s & N/A & 5.3x \\ +Custom Binary Search & 0.0180s & 0.0078s & N/A & 2.3x \\ +Bubble Sort (10k list) & 3.8210s & 0.0201s & N/A & 190.1x (Over 180x) \\ \bottomrule +\end{tabular} +\caption{PythonSTL Performance Benchmarks} +\end{table} + +\subsection{Scalability Analysis and FFI Overhead} +Evaluating the benchmarks reveals two key systems-level performance behaviors: + +\begin{enumerate} + \item \textbf{The FFI Boundary Overhead Limit \& Loop Optimization}: + For granular, low-work operations (like pushing single elements onto a stack or vector), the Rust backend is only marginally faster (1.48x--1.67x) than pure Python. This occurs because crossing the FFI boundary introduces a constant time overhead for every function call. The interpreter must validate parameters, acquire the Global Interpreter Lock (GIL), and translate references. When this overhead is comparable to the work done inside the function (which is a simple $\mathcal{O}(1)$ list insertion), the FFI boundary crossing dominates the execution time. + + Conversely, the Rust backend is highly effective for computation-heavy algorithms like \texttt{bubble\_sort} (delivering a \textbf{190x+ speedup} for 10,000 elements). This massive speedup is a real systems-level phenomenon caused by two factors: + \begin{itemize} + \item \textbf{FFI Amortization}: The Python list is converted to a native Rust \texttt{Vec} at the beginning of the function call, crossing the FFI boundary exactly once. + \item \textbf{Interpreter vs. Native Compilation}: A nested $\mathcal{O}(N^2)$ loop in Python requires the CPython virtual machine to interpret bytecode, execute type-checks, increment reference counts, and perform array index checks for all $N^2 \approx 10^8$ operations. In contrast, the Rust compiler compiles the nested loop into native, optimized assembly instructions that execute on the CPU using raw memory-layout swaps and hardware register comparisons, entirely bypassing virtual machine execution. + \end{itemize} + + \item \textbf{Sorted trees vs Hash tables}: + The Rust-backed \texttt{stl\_set} and \texttt{stl\_map} are slower than native Python \texttt{set} and \texttt{dict} types. + + Python's native structures use highly optimized hash tables ($\mathcal{O}(1)$ average case). To replicate C++ STL compliance, \texttt{PythonSTL} uses ordered B-trees or AVL trees, which require $\mathcal{O}(\log N)$ complexity. + + Therefore, \texttt{stl\_set} and \texttt{stl\_map} should not be used as generic collections, but specifically when keys must be maintained in sorted order. +\end{enumerate} + +\newpage + +\section{Challenges \& Debugging Stories} + +\subsection{Story 1: The Lomuto Selection Slowdown} +During early testing of the \texttt{nth\_element} function, the test runner experienced hangs when partitioning sorted lists of size $10^5$. + +\textbf{Diagnosis}: The Quickselect algorithm was using a naive Lomuto partition scheme with the last element of the list as the pivot (\texttt{arr[right]}). When sorting pre-sorted or reversed lists, this choice splits the array unevenly ($N-1$ and $0$ items), degrading time complexity from $\mathcal{O}(N)$ to $\mathcal{O}(N^2)$. + +\textbf{Resolution}: The algorithm was modified to select the middle index as the pivot (\texttt{left + (right - left) / 2}) and swap it with the rightmost element before partitioning. This restored average-case $\mathcal{O}(N)$ behavior, reducing runtime on sorted arrays from 70.85 seconds to 0.0064 seconds. + +\subsection{Story 2: Custom Comparators and FFI Callbacks} +When running binary searches with custom lambda comparators, performance dropped significantly, running slower than the pure-Python version. + +\textbf{Diagnosis}: Profile testing indicated that for each search step, the Rust binary search called the Python lambda function across the FFI. Acquirng the GIL, translating variables to Python objects, calling the function, and extracting the boolean result at every step of the binary search loop degraded performance. + +\textbf{Resolution}: We added optimized type-checking checks inside \texttt{PyObjectOrd::cmp} in Rust. For standard comparison requests without custom lambda definitions, the bridge uses native fast extraction for primitive integer, float, and string keys. Comparisons are computed natively in Rust without calling back into CPython. For custom objects, the bridge falls back to Python-space operations. + +\section{Trade-offs \& Design Decisions} + +\subsection{Facade Design Pattern vs. Direct Bindings} +\begin{itemize} + \item \textbf{Direct Bindings}: Exporting the compiled Rust classes directly to Python would eliminate the Python facade wrappers entirely. + \item \textbf{Trade-off}: If a user's machine lacked compilation tools or pre-built wheels, installing the library would fail immediately. + \item \textbf{Decision}: We selected the Facade pattern. The public class exposes C++ method names and manages fallbacks, ensuring the library remains functional via pure-Python implementations on platforms without Rust support. +\end{itemize} + +\subsection{AVL Trees vs. Red-Black or B-Trees} +\begin{itemize} + \item \textbf{Decision}: For the pure Python fallback, we implemented an AVL Tree because its balancing rules are simpler to implement and debug than a Red-Black tree. AVL trees are also more strictly balanced, which optimizes lookup operations at the expense of slightly more rotations during insertion. + \item For the Rust backend, we chose \texttt{BTreeSet} and \texttt{BTreeMap} from Rust's \texttt{std::collections} because they are cache-friendly, store multiple elements per node, and reduce memory allocation overhead compared to binary trees. +\end{itemize} + +\newpage + +\section{My Personal Contribution} + +As the core engineer of \texttt{PythonSTL}, my contributions covered the full development lifecycle: +\begin{itemize} + \item Designed and implemented the three-tier project architecture (Core, Implementation, Facade). + \item Developed the compiled Rust extension engine using PyO3, establishing the \texttt{PyObjectOrd} comparison bridge. + \item Wrote the pure Python fallback system, including the complete self-balancing AVL Tree. + \item Handled FFI optimization, introducing the fast-path primitive extractor to reduce memory and performance overhead. + \item Configured the Maturin build workflow and package layout in \texttt{pyproject.toml}. + \item Implemented the performance benchmark harness and achieved comprehensive test coverage. +\end{itemize} + +\section{Future Improvements} + +\begin{enumerate} + \item \textbf{Additional STL Containers}: + Add implementations for \texttt{std::list} (doubly linked list), \texttt{std::deque} (double-ended queue), and unordered associative containers like \texttt{std::unordered\_set} and \texttt{std::unordered\_map}. + \item \textbf{Automated Wheel Generation}: + Configure GitHub Actions CI/CD to build and publish pre-compiled wheels for major platforms (Windows, macOS, Linux) using \texttt{cibuildwheel}. + \item \textbf{Extended Algorithm Support}: + Implement numeric algorithms from C++'s \texttt{} header (e.g., \texttt{accumulate}, \texttt{inner\_product}, \texttt{adjacent\_difference}). +\end{enumerate} + +\newpage + +\section{Resume-Level Summary} + +\subsection{30-Second Elevator Pitch} +``I developed \texttt{PythonSTL}, a hybrid Python-Rust library that replicates C++ Standard Template Library data structures and algorithms. Using the Facade design pattern, it offers C++ developers a familiar API with type safety and dual-engine fallback support. I wrote the core engine in Rust using PyO3, implementing low-level optimizations like fast-path type extraction to bypass interpreter overhead, achieving up to a 7.5x performance speedup on sorted collections and over 180x speedup for computational workloads (like bubble sort) compared to pure Python implementations.'' + +\subsection{2-Minute Technical Pitch} +``I built \texttt{PythonSTL}, an open-source library that implements C++ STL containers and algorithms in Python. + +The library uses a dual-engine architecture: a pure-Python engine utilizing custom AVL Trees for sorted sets and maps, and a compiled Rust backend built with PyO3. The facade dynamically selects the Rust backend when available, falling back to pure Python when compiling isn't supported. + +A key challenge was sorting arbitrary Python objects in Rust's \texttt{BTreeSet}. I designed a wrapper that implements Rust's \texttt{Ord} trait by executing Python rich comparisons. To minimize FFI overhead, I added a fast-path primitive checker to bypass interpreter callbacks for standard types. I also resolved a $\mathcal{O}(N^2)$ worst-case bottleneck in our Quickselect implementation by replacing the Lomuto partition with a middle-pivot strategy. The project is distributed on PyPI and managed via Maturin, delivering up to a 7.5x speedup for data structures and over 180x for computational workloads.'' + +\subsection{5-Minute Architectural Pitch} +``I created \texttt{PythonSTL} to address the lack of ordered associative containers in Python and to build a robust template for hybrid Python-Rust development using PyO3, which is a common pattern in high-performance libraries like Polars and Pydantic. + +The system uses a three-layer Facade architecture. The Core layer defines standard exceptions and pointers. The Implementation layer contains private, pure-Python data structures. The Facade layer exposes the C++ interfaces (\texttt{stack}, \texttt{queue}, \texttt{vector}, \texttt{stl\_set}, \texttt{stl\_map}, \texttt{priority\_queue}). + +During startup, the Facade imports the compiled Rust library (\texttt{\_rust}) and switches the backend dynamically based on configuration. + +The Rust backend is built using PyO3. Because Rust's \texttt{BTreeSet} and \texttt{BTreeMap} require static ordering traits (\texttt{Ord}), I implemented a custom wrapper that acquires CPython's Global Interpreter Lock (GIL) and calls Python's comparison operators. To optimize performance, I added an extraction layer that converts primitives like \texttt{i64} and \texttt{String} directly to Rust types, avoiding interpreter callbacks. This setup delivers up to a 7.5x speedup on data structures and over 180x speedups on computational workloads (such as sorting algorithms). + +For the pure-Python engine, I implemented an AVL Tree to provide sorted associative operations. + +I set up testing using Pytest, validated type safety with Mypy, and configured Maturin in \texttt{pyproject.toml} for packaging. The project is currently published on PyPI.'' + +\newpage + +\section{Interview Questions} + +\subsection{Basic Questions} + +\paragraph{Question 1: Explain the difference between Python's native set/dict structures and PythonSTL's stl\_set/stl\_map.} +\textbf{Answer}: Python's built-in \texttt{set} and \texttt{dict} use hash tables. They provide average-case $\mathcal{O}(1)$ operations but do not maintain element order. \texttt{stl\_set} and \texttt{stl\_map} are ordered containers implemented as AVL Trees (Python) or B-Trees (Rust). They maintain elements in sorted order and execute operations in $\mathcal{O}(\log N)$ time. + +\paragraph{Question 2: What is the Facade Design Pattern, and how is it used in this project?} +\textbf{Answer}: The Facade design pattern provides a unified interface to a set of interfaces in a subsystem. In this project, the public classes (\texttt{vector}, \texttt{stl\_set}) act as facades. They expose C++ STL-compliant methods to the developer while hiding whether the execution is handled by the pure Python fallback engine or the compiled Rust binary. + +\paragraph{Question 3: How do you build and compile the Rust backend inside PythonSTL?} +\textbf{Answer}: The project is managed by Maturin. To compile the Rust code and install the package locally in edit mode, you run: +\begin{lstlisting}[language=bash] +pip install maturin +maturin develop +\end{lstlisting} +This compiles \texttt{src/lib.rs} into a shared library and links it into the Python project path. + +\subsection{Advanced Questions} + +\paragraph{Question 4: What is PyO3, and how does PythonSTL manage reference counting for Python objects stored in Rust?} +\textbf{Answer}: PyO3 is a crate that provides Rust bindings for the CPython API. When Python objects are passed to Rust, they are stored as \texttt{PyObject} wrappers. \texttt{PyObject} acts as a reference-counted pointer. PyO3 automatically increments the reference count when a \texttt{PyObject} is cloned in Rust and decrements it when the object is dropped, releasing memory safely. + +\paragraph{Question 5: Explain the performance bottlenecks associated with crossing the Python-Rust FFI boundary.} +\textbf{Answer}: Calling compiled Rust functions from Python introduces a small overhead for argument conversion and GIL validation. For simple $\mathcal{O}(1)$ operations like \texttt{stack.push()}, this boundary overhead can exceed the execution time of the function itself. Rust provides the biggest performance gains for computationally intensive algorithms that execute entirely in compiled space before returning. + +\paragraph{Question 6: How does the Rust backend implement Ord for arbitrary Python objects?} +\textbf{Answer}: Rust's sorting collections require elements to implement the \texttt{Ord} trait. We wrap Python's \texttt{PyObject} in a custom struct called \texttt{PyObjectOrd}. Inside the \texttt{cmp} method, we acquire the GIL, try to extract primitive types for native comparison, and fall back to calling CPython's comparison operators (\texttt{lt}, \texttt{eq}) if extraction is not possible. + +\subsection{Tricky Questions} + +\paragraph{Question 7: Why is it that stl\_set.insert() runs slower than Python's built-in set.add() even when using the Rust backend?} +\textbf{Answer}: There are two reasons: +First, a B-tree search takes $\mathcal{O}(\log N)$ operations compared to a hash table's $\mathcal{O}(1)$ lookup. +Second, when comparing objects, the Rust backend must acquire the GIL and callback into Python to evaluate comparisons. This FFI round-trip makes tree operations on Python objects slower than native CPython hash table operations. + +\paragraph{Question 8: If the Rust backend is written in Rust, which is thread-safe, why are PythonSTL containers still thread-unsafe?} +\textbf{Answer}: The Rust backend stores CPython pointers (\texttt{PyObject}). Any operation on these pointers must acquire the GIL and modify CPython heap structures. Because the underlying Python memory management system is not thread-safe without locks, accessing these containers from multiple Python threads without synchronization can cause race conditions. + +\paragraph{Question 9: What happens if a class instance that does not implement comparison methods is inserted into stl\_set?} +\textbf{Answer}: If the Rust backend is active, the \texttt{cmp} implementation of \texttt{PyObjectOrd} will fall back to executing CPython comparison methods on the object. If the object lacks comparison methods (\texttt{\_\_lt\_\_}), CPython will raise a \texttt{TypeError}. This error propagates through the FFI boundary and is raised as a standard Python exception. + +\newpage + +\section{Cross-Questions ("Why?", "What if?", "Why not X?")} + +\paragraph{Why write custom AVL trees in Python instead of using native lists or dicts for the fallback?} +Because Python's native structures do not maintain elements in sorted order. If we used native lists for \texttt{stl\_set}, we would have to sort the list after every insertion ($\mathcal{O}(N \log N)$) or use linear insertions ($\mathcal{O}(N)$). The AVL Tree guarantees $\mathcal{O}(\log N)$ insertion and deletion, matching standard C++ tree complexity. + +\paragraph{Why not use the existing \texttt{sortedcontainers} library instead of writing a Rust backend?} +The \texttt{sortedcontainers} library is a pure-Python implementation that uses load-balanced lists to achieve speed. However, \texttt{PythonSTL} serves a different purpose: it replicates C++ STL APIs precisely and provides a learning template for hybrid Python-Rust development using PyO3, which is a common pattern in high-performance libraries like Polars and Pydantic. + +\paragraph{What if Python removes the Global Interpreter Lock (GIL) under PEP 703? How does that affect this project?} +If the GIL is removed, Python objects can be accessed in parallel. The Rust backend would no longer need to acquire the GIL block before comparing objects. However, we would need to add thread-safe synchronization to the Rust containers to prevent race conditions on shared objects. + +\paragraph{Why did you choose a B-Tree for the Rust backend instead of a Red-Black Tree?} +Rust's standard library implements \texttt{BTreeMap} and \texttt{BTreeSet} instead of Red-Black Trees. B-trees store multiple keys per node, making them more cache-friendly and reducing pointer traversal and memory allocation overhead on modern hardware. + +\section{Common Mistakes to Avoid While Explaining} + +\begin{itemize} + \item \textbf{Mistake 1: Claiming the Rust backend makes all operations faster}. + Avoid claiming that Rust makes every operation faster. Be prepared to explain that fine-grained operations like \texttt{stack.push} can be slower due to FFI overhead, and that Rust excels primarily at bulk computations. + + \item \textbf{Mistake 2: Forgetting to mention the O(log N) tree complexity vs O(1) hash complexity}. + Do not present \texttt{stl\_set} and \texttt{stl\_map} as general replacements for Python's \texttt{set} and \texttt{dict}. Explain that they are tree-based structures designed specifically to maintain sorted order, which requires $\mathcal{O}(\log N)$ time. + + \item \textbf{Mistake 3: Stating that Rust makes the library thread-safe}. + Do not assume that because the backend is written in Rust, the containers are thread-safe. Explain that because the library manages CPython objects, it is still bound by CPython's single-threaded thread-safety constraints. +\end{itemize} + +\section{Myths \& Common Misconceptions} + +\begin{enumerate} + \item \textbf{Myth 1: ``This library has no actual use because online competitive programming platforms do not support external library imports.''} + \\ \textbf{Reality}: This is true for live contests (e.g., LeetCode, Codeforces), but it is a misconception regarding the library's utility. \texttt{PythonSTL} is designed as a local prototyping, learning, and transition tool. C++ developers moving to Python can use it locally to adapt their mental model of STL data structures to Python's syntax, and it serves as a showcase of hybrid PyO3 systems engineering. + + \item \textbf{Myth 2: ``Python's native structures are always better and faster, making a Rust STL backend redundant.''} + \\ \textbf{Reality}: Python lacks native structures for several core STL behaviors. Specifically: + \begin{itemize} + \item \textbf{No Sorted Set/Map}: Python's built-in \texttt{set} and \texttt{dict} are unordered hash tables. Maintaining sorted collections in native Python requires repeated $\mathcal{O}(N \log N)$ sorting or linear searches. \texttt{PythonSTL}'s Rust engine provides a true $\mathcal{O}(\log N)$ sorted \texttt{BTreeSet} and \texttt{BTreeMap}. + \item \textbf{No Customizable Priority Queue}: Python's \texttt{heapq} is strictly a min-heap, and custom comparators are boilerplate-heavy. \texttt{PythonSTL} provides both min/max heaps and custom comparators natively. + \end{itemize} + + \item \textbf{Myth 3: ``Since there is a compiled Rust backend, every operation is guaranteed to run faster than pure Python.''} + \\ \textbf{Reality}: Incorrect. As detailed in the performance section, granular $\mathcal{O}(1)$ operations like a single \texttt{stack.push()} or \texttt{vector.pop\_back()} are dominated by Foreign Function Interface (FFI) boundary crossing overhead. The Rust backend is primarily faster for bulk computations (like binary search queries on large lists or sorting arrays). + + \item \textbf{Myth 4: ``The Rust compiled backend makes the containers thread-safe.''} + \\ \textbf{Reality}: Absolutely not. Even with the Rust backend, \texttt{PythonSTL} containers are not thread-safe. Because the collections store Python objects (\texttt{PyObject}), Rust must acquire the CPython GIL to perform comparisons. Simultaneous modifications from multiple threads on the same container will lead to data races or undefined behavior unless synchronized in Python space using a \texttt{threading.Lock}. + + \item \textbf{Myth 5: ``stl\_set and stl\_map are direct, drop-in performance replacements for native Python sets and dicts.''} + \\ \textbf{Reality}: No. They serve fundamentally different algorithmic purposes. Python's native sets and dicts are hash tables ($\mathcal{O}(1)$ average lookup, unordered). \texttt{stl\_set} and \texttt{stl\_map} are tree-based structures ($\mathcal{O}(\log N)$ lookup, ordered). They should only be used when elements must be kept sorted or when range query capabilities are required. + + \item \textbf{Myth 6: ``Using a Rust backend avoids all Python memory and reference counting issues.''} + \\ \textbf{Reality}: False. Because the containers store arbitrary Python objects, they hold \texttt{PyObject} references. They participate in CPython's reference counting and garbage collection. If circular references are created, CPython's GC must clean them up, and Rust does not automatically bypass this. +\end{enumerate} + +\end{document} diff --git a/tests/test_algorithms.py b/tests/test_algorithms.py index fc5f9ec..046c7ae 100644 --- a/tests/test_algorithms.py +++ b/tests/test_algorithms.py @@ -1,6 +1,6 @@ import pytest from pythonstl import next_permutation, prev_permutation, nth_element, partition -from pythonstl.facade.algorithms import RUST_AVAILABLE +from pythonstl.containers.algorithms import RUST_AVAILABLE # Run tests on both implementations (Rust and pure-Python) PARAMS = [False]