posts
Decoupled Embeddings for Pre-training (DEPT)
A blog write-up of DEPT, a pre-training framework that decouples token embeddings from the transformer body so they can be optimized separately per data source. This tackles the challenges of training on heterogeneous datasets, cutting memory consumption by up to 24% and communication costs by up to 714x, while matching or improving downstream task performance versus standard training.