feat: unified Transformers4Rec with CLM/MLM/PLM/RTD training objective

README.md

@@ -155,7 +155,7 @@ The table below lists the recommendation models/algorithms featured in Cornac. E
 | 2023 | [Scalable Approximate NonSymmetric Autoencoder (SANSA)](cornac/models/sansa), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.sansa.recom_sansa), [paper](https://dl.acm.org/doi/10.1145/3604915.3608827) | Collaborative Filtering | [requirements](cornac/models/sansa/requirements.txt), CPU | [quick-start](examples/sansa_movielens.py), [150k-items](examples/sansa_tradesy.py)
 | 2022 | [Disentangled Multimodal Representation Learning for Recommendation (DMRL)](cornac/models/dmrl), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.dmrl.recom_dmrl), [paper](https://arxiv.org/pdf/2203.05406.pdf) | Content-Based / Text & Image | [requirements](cornac/models/dmrl/requirements.txt), CPU / GPU | [quick-start](examples/dmrl_example.py)
 | 2021 | [Bilateral Variational Autoencoder for Collaborative Filtering (BiVAECF)](cornac/models/bivaecf), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.bivaecf.recom_bivaecf), [paper](https://dl.acm.org/doi/pdf/10.1145/3437963.3441759) | Collaborative Filtering / Content-Based | [requirements](cornac/models/bivaecf/requirements.txt), CPU / GPU | [quick-start](https://github.com/PreferredAI/bi-vae), [deep-dive](https://github.com/recommenders-team/recommenders/blob/main/examples/02_model_collaborative_filtering/cornac_bivae_deep_dive.ipynb)
-|      | [GPT-2 for Sequential Recommendation (GPT2Rec)](cornac/models/gpt2rec), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.gpt2rec.recom_gpt2rec), [paper](https://dl.acm.org/doi/10.1145/3460231.3474255) | Next-Item | [requirements](cornac/models/gpt2rec/requirements.txt), CPU / GPU | [quick-start](examples/transformer_rec_diginetica.py)
+|      | [Transformers4Rec-style Unified Transformer Recommender (TransformerRec)](cornac/models/transformer_rec), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.transformer_rec.recom_transformer_rec), [paper](https://dl.acm.org/doi/10.1145/3460231.3474255) | Next-Item | [requirements](cornac/models/transformer_rec/requirements.txt), CPU / GPU | [quick-start](examples/transformer_rec_diginetica.py)
 |      | [Causal Inference for Visual Debiasing in Visually-Aware Recommendation (CausalRec)](cornac/models/causalrec), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.causalrec.recom_causalrec), [paper](https://arxiv.org/abs/2107.02390) | Content-Based / Image | [requirements](cornac/models/causalrec/requirements.txt), CPU / GPU | [quick-start](examples/causalrec_clothing.py)
 |      | [Explainable Recommendation with Comparative Constraints on Product Aspects (ComparER)](cornac/models/comparer), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.comparer.recom_comparer_sub), [paper](https://dl.acm.org/doi/pdf/10.1145/3437963.3441754) | Explainable | CPU | [quick-start](https://github.com/PreferredAI/ComparER)
 | 2020 | [Adversarial Multimedia Recommendation (AMR)](cornac/models/amr), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.amr.recom_amr), [paper](https://ieeexplore.ieee.org/document/8618394) | Content-Based / Image | [requirements](cornac/models/amr/requirements.txt), CPU / GPU | [quick-start](examples/amr_clothing.py)

cornac/models/__init__.py

@@ -50,7 +50,6 @@
 from .gcmc import GCMC
 from .global_avg import GlobalAvg
 from .gp_top import GPTop
-from .gpt2rec import GPT2Rec
 from .gru4rec import GRU4Rec
 from .hft import HFT
 from .hpf import HPF
@@ -84,6 +83,7 @@
 from .spop import SPop
 from .svd import SVD
 from .tifuknn import TIFUKNN
+from .transformer_rec import TransformerRec
 from .trirank import TriRank
 from .upcf import UPCF
 from .vaecf import VAECF

cornac/models/bert4rec/recom_bert4rec.py

@@ -13,42 +13,25 @@
 # limitations under the License.
 # ============================================================================
 
-import numpy as np
-from tqdm.auto import trange
+from ..transformer_rec import TransformerRec
 
-from cornac.models.recommender import NextItemRecommender
 
-from ...utils import get_rng
-from ..seq_utils import session_seq_iter, val_score
-
-SUPPORTED_LOSSES = (
-    "bce",
-    "ce",
-    "bpr",
-    "bpr-max",
-    "softmax",
-    "cross-entropy",
-    "xe_softmax",
-    "top1",
-)
-
-
-class BERT4Rec(NextItemRecommender):
+class BERT4Rec(TransformerRec):
     """BERT4Rec: a bidirectional transformer encoder for sequential rec.
 
-    Wraps HuggingFace's :class:`~transformers.BertModel` as the sequence
-    encoder; the last-position hidden state scores candidate items by dot
-    product, sharing the ``(B, B+N)`` loss contract of
-    :mod:`cornac.models.seq_utils`. Parameters mirror
-    :class:`cornac.models.SASRec` (minus ``use_pos_emb`` — the backbone
-    provides its own positional embeddings); see the SASRec docstring for
-    details about ``loss``, ``model_selection``, and the rest.
+    A light interface over :class:`~cornac.models.TransformerRec` fixed to
+    ``backbone='bert', objective='clm', loss_at='last'``, i.e. the
+    next-item-at-last-position training shared by the transformer family in
+    Cornac. See the :class:`~cornac.models.TransformerRec` docstring for the
+    shared parameters (``loss``, ``model_selection``, negative sampling,
+    etc.).
 
     Note
     ----
-    This uses the next-item-at-last-position objective shared by the
-    transformer family in Cornac, *not* the canonical masked-language-model
-    (MLM) objective of the original paper.
+    The original paper trains with the masked-language-model (Cloze) objective;
+    that canonical setting is available as
+    ``TransformerRec(backbone='bert', objective='mlm')``.
+    In our experiments, the last-position objective gives better performance.
 
     References
     ----------
@@ -83,145 +66,31 @@ def __init__(
         val_k=20,
         val_metric="recall",
     ):
-        super().__init__(name, trainable=trainable, verbose=verbose)
-        if loss not in SUPPORTED_LOSSES:
-            raise ValueError(
-                f"loss='{loss}' not supported; choose from {SUPPORTED_LOSSES}"
-            )
-        if model_selection not in ("last", "best"):
-            raise ValueError(
-                f"model_selection='{model_selection}' not supported; choose 'last' or 'best'"
-            )
-        self.embedding_dim = embedding_dim
-        self.loss = loss
-        self.batch_size = batch_size
-        self.learning_rate = learning_rate
-        self.n_sample = n_sample
-        self.sample_alpha = sample_alpha
-        self.n_epochs = n_epochs
-        self.max_len = max_len
-        self.num_blocks = num_blocks
-        self.num_heads = num_heads
-        self.dropout = dropout
-        self.l2_reg = l2_reg
-        self.bpreg = bpreg
-        self.elu_param = elu_param
-        self.device = device
-        self.seed = seed
-        self.rng = get_rng(seed)
-        self.model_selection = model_selection
-        self.val_eval_every = val_eval_every
-        self.val_k = val_k
-        self.val_metric = val_metric
-
-    def fit(self, train_set, val_set=None):
-        super().fit(train_set, val_set)
-        if not self.trainable:
-            return self
-
-        import torch
-
-        from .bert4rec import BERT4RecModel
-        from ..seq_utils.losses import get_loss_function
-
-        torch.manual_seed(self.seed if self.seed is not None else 0)
-
-        self.pad_idx = self.total_items
-        self.model = BERT4RecModel(
-            item_num=self.total_items,
-            embedding_dim=self.embedding_dim,
-            maxlen=self.max_len,
-            n_layers=self.num_blocks,
-            n_heads=self.num_heads,
-            dropout=self.dropout,
-            pad_idx=self.pad_idx,
-            device=self.device,
-        )
-
-        loss_fn = get_loss_function(self.loss)
-        loss_kwargs = dict(
-            bpreg=self.bpreg, elu_param=self.elu_param, n_sample=self.n_sample
-        )
-        opt = torch.optim.Adam(
-            self.model.parameters(), lr=self.learning_rate, betas=(0.9, 0.98)
-        )
-
-        best_val = -float("inf")
-        best_state = None
-        progress_bar = trange(1, self.n_epochs + 1, disable=not self.verbose)
-        for epoch_id in progress_bar:
-            self.model.train()
-            total_loss = 0.0
-            cnt = 0
-            for inc, (in_uids, hist_iids, out_iids) in enumerate(
-                session_seq_iter(
-                    self.train_set,
-                    pad_index=self.pad_idx,
-                    batch_size=self.batch_size,
-                    max_len=self.max_len,
-                    n_sample=self.n_sample,
-                    sample_alpha=self.sample_alpha,
-                    rng=self.rng,
-                    shuffle=True,
-                )
-            ):
-                if len(hist_iids) < 2:
-                    continue
-                hist_iids_t = torch.tensor(
-                    hist_iids, dtype=torch.long, device=self.device, requires_grad=False
-                )
-                out_iids_t = torch.tensor(
-                    out_iids, dtype=torch.long, device=self.device, requires_grad=False
-                )
-
-                self.model.zero_grad()
-                item_scores = self.model(None, hist_iids_t, out_iids_t)
-                L = loss_fn(
-                    item_scores,
-                    out_iids=out_iids_t,
-                    batch_size=len(hist_iids),
-                    **loss_kwargs,
-                )
-                if self.l2_reg > 0:
-                    for p in self.model.parameters():
-                        L = L + self.l2_reg * torch.norm(p)
-
-                L.backward()
-                opt.step()
-
-                total_loss += L.cpu().detach().numpy() * len(hist_iids)
-                cnt += len(hist_iids)
-                if inc % 10 == 0 and cnt > 0:
-                    progress_bar.set_postfix(loss=(total_loss / cnt))
-
-            if (
-                self.model_selection == "best"
-                and val_set is not None
-                and epoch_id % self.val_eval_every == 0
-            ):
-                score = val_score(
-                    self, self.train_set, val_set, metric=self.val_metric, k=self.val_k
-                )
-                if score is not None and score > best_val:
-                    best_val = score
-                    best_state = {
-                        n: p.detach().clone()
-                        for n, p in self.model.state_dict().items()
-                    }
-
-        if self.model_selection == "best" and best_state is not None:
-            self.model.load_state_dict(best_state)
-        return self
-
-    def score(self, user_idx, history_items, **kwargs):
-        import torch
-
-        if len(history_items) == 0:
-            return np.ones(self.total_items, dtype="float")
-        log_seq = [self.pad_idx] * (self.max_len - len(history_items)) + list(
-            history_items
+        super().__init__(
+            name=name,
+            backbone="bert",
+            objective="clm",
+            loss_at="last",
+            embedding_dim=embedding_dim,
+            loss=loss,
+            batch_size=batch_size,
+            learning_rate=learning_rate,
+            n_sample=n_sample,
+            sample_alpha=sample_alpha,
+            n_epochs=n_epochs,
+            max_len=max_len,
+            num_blocks=num_blocks,
+            num_heads=num_heads,
+            dropout=dropout,
+            l2_reg=l2_reg,
+            bpreg=bpreg,
+            elu_param=elu_param,
+            device=device,
+            trainable=trainable,
+            verbose=verbose,
+            seed=seed,
+            model_selection=model_selection,
+            val_eval_every=val_eval_every,
+            val_k=val_k,
+            val_metric=val_metric,
         )
-        log_seq = log_seq[-self.max_len :]
-        log_seq_t = torch.tensor([log_seq], dtype=torch.long, device=self.device)
-        self.model.eval()
-        return self.model.predict(user_idx, log_seq_t)