We propose a general framework for assessing sparsity robustness in modern LLMs and conduct a systematic study of activation sparsity such models. Our study reveals universal patterns of sparsity in LLMs and provides practical guidelines for model acceleration and design.
Dec 6, 2025
We challenge the use of calibration metrics in early-exit models and show cases where calibration fails to accurately reflect the network performance. We argue for failure prediction as a more reliable performance proxy that better correlates with efficiency gains in early-exit networks.
Dec 6, 2025
We propose a novel generative mixture-of-GANs approach for accelerating particle detector simulations that maintains high fidelity while achieving significant computational speedups compared to traditional methods.
Aug 30, 2025
We investigates intermediate representations in neural networks during class-incremental learning and propose to leverage them via auxiliary early-exit classifiers. Interestingly, we find out that in continual learning scenarios networks enhanced with such classiers are not only more efficient, but also show improved performance and reduced forgetting across task sequences.
Jul 1, 2025
We develop an efficient approach LLM input safety moderation using latent prototypes and demonstrate that safe and unsafe inputs are separable in the model's latent space.
Feb 22, 2025
We conduct an investigation into the stability gap in continual learning and identify the critical role of the classification head in continual learning. We then suggest nearest mean classifer as a potential solution for improved model stability.
Jan 1, 2025
We propose a method to convert dense transformers to dynamic Mixture-of-Experts models, which leverages natural activation sparsity in the neural networks. Crucially, we propose to enforce activation sparsity during short (continual) training process via additional sparsity regularization, and argue for use of dynamic-k expert routing in MoEfied models. Finally, we show how with efficient implementation our method achieves computational efficiency while maintaining the performance.
Dec 1, 2024
We develop an exact orthogonal initialization technique for static sparse training that enables more robust sparse neural network training.
Jul 1, 2024
We examine knowledge distillation in exemplar-free continual learning and find out that allowing the adaptation of teacher network during the learning process through batch normalization updates improves knowledge transfer across several continual learning methods.
Jan 1, 2024
We propose Zero-Time Waste, an early exit network architecture that reduces computational waste via cascading connections between early-exit classifiers and ensembling mechanism. ZTW achieves better efficiency-accuracy trade-offs in pre-trained models and offers a practical architectural solution for deployment of early exit neural networks.
Dec 1, 2023
We propose to use hypernetworks to generate implicit neural representations of sound signals, enabling efficient audio compression and high-quality reconstruction.
Sep 1, 2023
We propose progressive latent replay mechanism that enhances generative rehearsal in continual learning by efficiently managing memory and computational resources while maintaining model performance.
Nov 1, 2022