This study explores the impact of various layers of the XLS-R encoder on speaker recognition accuracy using CNNs and logistic regression. Findings indicate that earlier layers yield higher accuracy, highlighting their importance in feature capture. The study also reveals a significant gender disparity in accuracy. These results suggest the need for further investigation into model biases and optimizations.