Training deep learning models effectively requires large datasets, but many of these datasets contain label noise, which can significantly impair classification performance. To tackle this challenge, a research team from Yildiz Technical University—comprising Enes Dedeoglu, H. Toprak Kesgin, and Prof. Dr. M. Fatih Amasyali—has developed a method known as Adaptive-k, aimed at improving optimization in the presence of label noise. Their findings were published in Frontiers of Computer Science.
The Adaptive-k method uniquely determines the number of samples selected from mini-batches for updating, which enhances the separation of noisy samples and boosts training success in datasets affected by label noise. This approach is straightforward and does not require prior knowledge of the noise ratio, additional model training, or significant increases in training duration. Adaptive-k has shown performance results that are comparable to the Oracle method, which entirely excludes noisy samples.
In their research, the team compared Adaptive-k to several well-known algorithms, including Vanilla, MKL, Vanilla-MKL, and Trimloss, evaluating its effectiveness against the Oracle scenario. Experiments were conducted on three image datasets and four text datasets, consistently demonstrating that Adaptive-k outperforms its counterparts in label-noisy conditions. Additionally, Adaptive-k is compatible with various optimization algorithms like SGD, SGDM, and Adam.
The key contributions of this research include:
- The introduction of Adaptive-k, a novel and easy-to-implement algorithm for robust training in noisy label datasets, requiring no additional model training or data augmentation.
- A theoretical analysis of Adaptive-k compared to MKL and SGD.
- High-accuracy estimation of clean sample ratios during training without needing prior dataset knowledge or hyperparameter adjustments.
- Empirical evaluations showing Adaptive-k's superior performance against Oracle, Vanilla, MKL, Vanilla-MKL, and Trimloss on multiple datasets.
Future work will focus on refining the Adaptive-k method, exploring additional applications, and further enhancing its performance.