[FEA] Add native GPU Gradient Boosting (GBM) estimators to cuML

[sauravsingla]

Feature request

cuML provides GPU-accelerated implementations for many classical machine-learning algorithms, but there is currently no native cuML training implementation corresponding to the scikit-learn Gradient Boosting family, such as:

• GradientBoostingClassifier
• GradientBoostingRegressor
• HistGradientBoostingClassifier
• HistGradientBoostingRegressor

I would like to propose adding native GPU Gradient Boosting Machine (GBM) estimators to cuML.

Motivation

Gradient-boosted decision trees remain one of the most widely used model families for structured/tabular machine learning.

cuML already provides a strong GPU-accelerated workflow for many classical estimators. Native GBM support would make it easier to keep complete tabular ML pipelines within cuML/RAPIDS without moving to another library solely for gradient boosting.

It could also improve scikit-learn API coverage and potentially expand the usefulness of cuml.accel for pipelines that rely on sklearn gradient boosting estimators.

Suggested scope

A staged implementation could begin with:

• GradientBoostingClassifier
• GradientBoostingRegressor

and potentially extend to:

• HistGradientBoostingClassifier
• HistGradientBoostingRegressor

Useful compatibility areas could include:

• binary and multiclass classification
• regression
• sample_weight
• common loss functions
• learning rate
• boosting iterations / n_estimators
• depth and leaf controls
• early stopping where applicable
• deterministic random_state behavior where possible
• sklearn-style fit, predict, and predict_proba
• cuDF/CuPy and CPU/GPU input interoperability consistent with cuML conventions

Why native cuML support?

XGBoost, LightGBM and CatBoost already provide GPU gradient boosting, but native cuML support could serve a different purpose:

1. Closer scikit-learn API compatibility.
2. Seamless integration with existing cuML pipelines.
3. Potential acceleration through cuml.accel.
4. Consistent RAPIDS memory and dataframe interoperability.
5. A common estimator and benchmarking interface alongside other cuML algorithms.

The goal would not necessarily be to reproduce every capability of dedicated boosting libraries, but to provide a strong GPU-native implementation of the standard sklearn GBM family.

Possible implementation direction

It may be useful to first decide whether the initial target should be:

• classic Gradient Boosting for sklearn API parity, or
• histogram-based Gradient Boosting as the more GPU-friendly/performance-oriented starting point.

A benchmark suite could compare:

• scikit-learn CPU implementation
• proposed cuML GPU implementation
• training time
• inference time
• GPU memory consumption
• predictive/numerical parity
• scaling with rows, columns, classes and boosting iterations

[btepera]

Hi @sauravsingla - thanks for opening this issue. Historically, we haven't prioritized a GBM implementation in cuML due to the presence of other mature and popular GBM libraries (XGBoost, LightGBM, etc.) with GPU-accelerated paths that we help maintain and the lack of a real use case for a GBM implementation in cuML. I'm curious to learn more about your workflow and better understand what problem a GBM implementation in cuML would help you solve that can't be done with one of these other libraries.

e.g.:

• Are there limitations in your environment preventing you from installing xgboost, lightgbm, etc. in addition to cuML?
• Are you using sklearn GBM estimators today and wanting to switch to a GPU-accelerated version with something like cuml.accel? If so, is XGB's sklearn API a potential fit here?
• Are you using a different GBM library today side-by-side with cuML and running into compatibility issues in your broader workflow? If so, is there a reproducer you can share?
• Something else?

[sauravsingla]

Thanks @btepera — that distinction is helpful.

The use case I had in mind is specifically cuml.accel coverage for existing scikit-learn workflows, rather than a need for another general-purpose GBM library.

There is no environment restriction preventing the use of XGBoost or LightGBM, and I agree that both already provide mature GPU-accelerated boosting implementations.

The gap I was thinking about is an existing sklearn application that already uses, for example:

from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(...)

Today, cuml.accel can transparently accelerate a growing set of sklearn estimators while preserving the existing application code. GradientBoostingClassifier / GradientBoostingRegressor are currently outside that accelerated set, so those estimators remain on the CPU.

The alternative is to change the application/model definition to something such as:

from xgboost import XGBClassifier
model = XGBClassifier(...)

XGBoost’s sklearn API is certainly a good option, but I see a distinction between providing an sklearn-compatible interface to XGBoost and transparently accelerating an application that specifically uses sklearn’s GradientBoostingClassifier/Regressor.

So the practical question behind this request is:

Would extending cuml.accel coverage to sklearn’s GradientBoostingClassifier / GradientBoostingRegressor be valuable for users who want to GPU-accelerate existing sklearn applications without replacing the estimator or rewriting that part of the pipeline?

This can also matter in mixed pipelines where several stages are GPU-accelerated through cuml.accel, but an unsupported estimator falls back to CPU. Avoiding unnecessary CPU/GPU transitions may improve the end-to-end acceleration story in addition to preserving the existing sklearn API.

I would therefore frame the value of native support primarily around sklearn API coverage, zero/low-code-change migration, and avoiding fallback within cuml.accel workflows, rather than competing with XGBoost or LightGBM on boosting functionality.

If the intended RAPIDS direction is instead that sklearn GradientBoosting estimators should remain CPU fallbacks and users should explicitly migrate those workloads to XGBoost’s sklearn API, that clarification would also be very useful and would answer the original feature request.