Dataset Indexing Issue

[ming024]

In the data collator of AddLabelDataset

slue-toolkit/slue_toolkit/fairseq_addon/data/add_label_dataset.py

Lines 28 to 33 in ea0939b

	def collater(self, samples):
	collated = self.dataset.collater(samples)
	if len(collated) == 0:
	return collated
	indices = set(collated["id"].tolist())
	target = torch.LongTensor([s["label"] for s in samples if s["id"] in indices])

If self.dataset.collater sort the collated data, target and collated will become mismatched.