Lorenzo Sani

Sitemap

A list of all the posts and pages found on the site. For you robots out there is an XML version available for digesting as well.

Pages

Lorenzo Sani

I work on the systems and optimization that make large-scale federated and distributed training of foundation models practical.

Posts

Introducing Endeavor 1.0 from Flower Labs Permalink

Published:

Flower Labs launched Endeavor 1.0, a frontier-class AI model designed to stay deployable within organizations’ own infrastructure rather than locking them into a managed API. It achieves competitive performance across key benchmarks and can be accessed through Flower’s managed service or through private deployment, giving organizations a path from API dependency to independent operation as their needs evolve.

Cambridge Collaborative Agent Hackathon Permalink

Published:

Flower Labs announced a one-day hackathon at the University of Cambridge for building collaborative AI agents with Flower. Participants competed in a SuperGrid or an Infrastructure track to build prototype agent systems that operate safely under human oversight, with a combined prize pool of £1,500.

The Red Queen Hypothesis - A New Way Forward for Self-Improving AI Permalink

Published:

A Cambridge Department of Computer Science and Technology news piece covering the Red Queen Gödel Machine, a framework that lets AI agents and their evaluators co-evolve rather than compete against a fixed benchmark. The approach overcomes performance plateaus by continuously raising evaluation standards, showing gains in scientific paper writing and grading while potentially cutting computational cost through a hybrid of open-source and proprietary models.

Lizzy-7B: A UK-built Open Frontier LLM for Sovereign AI Permalink

Published:

Flower Labs released Lizzy-7B, an open-weight language model built in the UK to support sovereign AI infrastructure. The preview release is tuned for UK-specific applications in finance, healthcare, and government, and demonstrates strong performance against European competitors while emphasizing practical deployment, building on Flower’s research into distributed AI systems.

Photon: Federated Pre-training of Large Language Models Permalink

Published:

A blog write-up of Photon, an extension to the Flower framework that enables decentralized pre-training of large language models across multiple independent data centers. Photon reaches training speeds comparable to or faster than centralized approaches while cutting communication overhead by 64-512x, letting institutions collaboratively train billion-parameter models without sharing sensitive data.

Decoupled Embeddings for Pre-training (DEPT) Permalink

Published:

A blog write-up of DEPT, a pre-training framework that decouples token embeddings from the transformer body so they can be optimized separately per data source. This tackles the challenges of training on heterogeneous datasets, cutting memory consumption by up to 24% and communication costs by up to 714x, while matching or improving downstream task performance versus standard training.

Introducing FlowerLLM Permalink

Published:

Flower Labs and Cambridge’s CaMLSys lab trained a 1.3-billion-parameter language model using federated learning, surpassing the previous record set by Google DeepMind by more than a factor of three. This shows that LLM pre-training can scale across distributed, heterogeneous hardware and private data sources, opening the door to foundation model development that taps into previously unused compute and data.

portfolio

publications

Flower: A Friendly Federated Learning Research Framework

Published in arXiv, 2022

This paper is about Flower, the federated learning framework developed at the University of Cambridge by the MLSys group.

Recommended citation: Beutel, D., Topal, T., Mathur, A., Qiu, X., Fernandez-Marques, J., Gao, Y., Sani, L., Kwing, H., Parcollet, T., Gusmão, P., & Lane, N. (2020). Flower: A Friendly Federated Learning Research Framework. arXiv preprint arXiv:2007.14390.
Download Paper

The Future of Large Language Model Pre-training is Federated

Published in FL@FM-NeurIPS'24 Workshop (Oral), 2024

We present in this work the first fully federated system for the federated pre-training of large language models.

Recommended citation: Lorenzo Sani, Alex Iacob, Zeyu Cao, Bill Marino, Yan Gao, Tomas Paulik, Wanru Zhao, William F. Shen, Preslav Aleksandrov, Xinchi Qiu, & Nicholas D. Lane. (2024). The Future of Large Language Model Pre-training is Federated.
Download Paper

Worldwide Federated Training of Language Models

Published in FL@FM-NeurIPS'24 Workshop (Oral, Outstanding Paper Award), 2024

WorldLM is a novel approach to federated training of language models exploiting federations of federations in a hierarchical setting.

Recommended citation: Alex Iacob, Lorenzo Sani, Bill Marino, Preslav Aleksandrov, William F. Shen, & Nicholas Donald Lane. (2024). Worldwide Federated Training of Language Models.
Download Paper

Sheaf HyperNetworks for Personalized Federated Learning

Published in arXiv, 2024

Sheaf HyperNetworks is a novel HyperNetwork-based methodology to improve the performance of any personalized federated learning setting.

Recommended citation: Bao Nguyen, Lorenzo Sani, Xinchi Qiu, Pietro Liò, & Nicholas D. Lane. (2024). Sheaf HyperNetworks for Personalized Federated Learning.
Download Paper

DEPT: Decoupled Embeddings for Pre-training Language Models.

Published in ICLR 2025 - The Thirteenth International Conference on Learning Representations (Oral), 2024

This work proposes a novel pre-training method for language models that decouples the embeddings from the rest of the model.

Recommended citation: Alex Iacob*, Lorenzo Sani*, Meghdad Kurmanji, William F. Shen, Xinchi Qiu, Dongqi Cai, Yan Gao, & Nicholas Donald Lane (2025). DEPT: Decoupled Embeddings for Pre-training Language Models. In The Thirteenth International Conference on Learning Representations.
Download Paper

Photon: Federated LLM Pre-Training

Published in MLSys 2025 - Proceedings of Machine Learning and Systems 7, 2024

We present Photon, the first fully federated system for the federated pre-training of large language models.

Recommended citation: Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, & Nicholas D. Lane. (2024). Photon: Federated LLM Pre-Training.
Download Paper

SparsyFed: Sparse Adaptive Federated Training

Published in ICLR 2025 - The Thirteenth International Conference on Learning Representations, 2025

This work proposes a novel approach to sparse federated training that does not require to fix a global mask to be efficient.

Recommended citation: Adriano Guastella*, Lorenzo Sani*, Alex Iacob, Alessio Mora, Paolo Bellavista, & Nicholas Donald Lane (2025). SparsyFed: Sparse Adaptive Federated Learning. In The Thirteenth International Conference on Learning Representations.
Download Paper

LUNAR: LLM Unlearning via Neural Activation Redirection

Published in NeurIPS 2025 - The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025

We present LUNAR, a novel unlearning methodology grounded in the Linear Representation Hypothesis.

Recommended citation: William F. Shen, Xinchi Qiu, Meghdad Kurmanji, Alex Iacob, Lorenzo Sani, Yihong Chen, Nicola Cancedda, & Nicholas D. Lane. (2025). LUNAR: LLM Unlearning via Neural Activation Redirection. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems.
Download Paper

DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models

Published in ICLR 2026 - The Fourteenth International Conference on Learning Representations, 2025

DES-LOC desynchronises the communication of optimizer states, cutting bandwidth for foundation-model training without sacrificing convergence.

Recommended citation: Alex Iacob, Lorenzo Sani, Mher Safaryan, Paris Giampouras, Samuel Horváth, Andrej Jovanovic, Meghdad Kurmanji, Preslav Aleksandrov, William F. Shen, Xinchi Qiu, & Nicholas D. Lane. (2026). DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models. In The Fourteenth International Conference on Learning Representations.
Download Paper

AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling

Published in arXiv, 2025

AbbIE is an autoregressive block-based iterative encoder that recursively refines representations, improving perplexity at a fixed parameter budget.

Recommended citation: Preslav Aleksandrov, Meghdad Kurmanji, Fernando Garcia Redondo, David O'Shea, William Shen, Alex Iacob, Lorenzo Sani, Xinchi Qiu, Nicola Cancedda, & Nicholas D. Lane. (2025). AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling.
Download Paper

MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates

Published in ICLR 2026 - The Fourteenth International Conference on Learning Representations, 2025

MT-DAO combines optimizer moments tracked on multiple timescales with local updates, closing the gap to fully synchronous training under infrequent communication.

Recommended citation: Alex Iacob, Andrej Jovanovic, Mher Safaryan, Meghdad Kurmanji, Lorenzo Sani, Samuel Horváth, William F. Shen, Xinchi Qiu, & Nicholas D. Lane. (2026). MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates. In The Fourteenth International Conference on Learning Representations.
Download Paper

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

Published in ICML 2026 - The Forty-Third International Conference on Machine Learning, 2026

LoRDO performs distributed low-rank optimization with infrequent communication, matching full-rank quality at a fraction of the synchronisation cost.

Recommended citation: Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, & Nicholas D. Lane. (2026). LoRDO: Distributed Low-Rank Optimization with Infrequent Communication. In International Conference on Machine Learning (ICML 2026).
Download Paper

Response-Conditioned Parallel-to-Sequential Orchestration for Multi-Agent Systems

Published in arXiv, 2026

A multi-agent orchestration scheme that adaptively transitions between parallel and sequential execution based on intermediate agent responses.

Recommended citation: Nurbek Tastan, Alex Iacob, Lorenzo Sani, Meghdad Kurmanji, Nicholas D. Lane, Samuel Horvath, & Karthik Nandakumar. (2026). Response-Conditioned Parallel-to-Sequential Orchestration for Multi-Agent Systems.
Download Paper

FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

Published in arXiv, 2026

FoMoE breaks the full-replica barrier in distributed training by federating mixtures of experts, so no participant needs to hold the entire model.

Recommended citation: Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji, Alex Iacob, Andrej Jovanovic, Yan Gao, Wanru Zhao, & Nicholas D. Lane. (2026). FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs.
Download Paper

The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators

Published in arXiv, 2026

A self-improving system in which agents and the evaluators that score them co-evolve, avoiding the reward hacking that fixed evaluators invite.

Recommended citation: Alex Iacob, Andrej Jovanović, William F. Shen, Daniel Burkhardt, Meghdad Kurmanji, Nurbek Tastan, Lorenzo Sani, Niccolò Alberto Elia Venanzi, Ambroise Odonnat, Zeyu Cao, Bill Marino, Xinchi Qiu, & Nicholas D. Lane. (2026). The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators.
Download Paper

Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment

Published in arXiv, 2026

Flower Hub is a platform for publishing, discovering, and executing decentralized and federated applications, enabling reproducible benchmarking across simulation and deployment.

Recommended citation: Yan Gao, Mohammad Naseri, Javier Fernandez-Marques, Dimitris Stripelis, Lorenzo Sani, Davide Eynard, Fan Zhang, Hong Jia, Ting Dang, D. B. Emerson, Fatemeh Tavakoli, Ole Werger, Lars Wulfert, Petros Demetrakopoulos, Sofia Tsekeridou, InSeo Song, KangYoon Lee, Honghao Li, Lingjuan Lyu, John P Dickerson, Daniel Janes Beutel, & Nicholas D. Lane. (2026). Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment.
Download Paper

talks

Federated Learning @ Ammagamma

Published:

On the 18th of November 2021, Giovanni Anceschi invited me to Ammagamma, a data science company based in Modena, to give an informal talk to his team about federated learning and my master’s thesis work with the Flower framework. For more details regarding Flower please visit the project page.

FlowerLLM AMA @ Flower Labs

Published:

On the 2nd of August 2024, I took part in the first FlowerLLM AMA, an online Ask-Me-Anything session run by Flower Labs after the release of FlowerLLM-3B, answering questions from the Flower community about our work on federated pre-training of large language models. For more details regarding this project please visit the project page.

Federating Large Language Models from Scratch @ Alan Turing Institute

Published:

The Research Engineering Team at the Alan Turing Institute invited me to present our work on federated learning of large language models. This talk was inserted in the seminar series titled “Robots in Disguise”. I presented our work on federated learning of large language models, in particular, how we can train large language models from scratch in a federated manner. A wider introduction to federate learning and its main challenges was also presented.

Flower AMA: Pre-training 7B LLMs made easy with Flower

Published:

On the 20th of November 2024, I presented our work on federated learning of large language models at the Flower AMA session. With the help of Prof. Nicholas D. Lane, Alexandru-Andrei Iacob and Preslav Aleksandrov, we discussed some critical aspects of pre-training language models and their relevancy to the federated approach.

Federated Pre-Training of Large Language Models

Published:

On the 15th of December 2024, I presented our work on federated pre-training of large language models at the NeurIPS workshop on Federated Learning for Foundational Models @ Federated Laerning.

Photon: Federated LLM Pre-Training @ MLSys 2025

Published:

On the 15th of May 2025, I presented Photon, our system for federated pre-training of large language models, in the Federated Learning session of the Eighth Conference on Machine Learning and Systems (MLSys 2025). For more details regarding this project please visit the project page.

teaching

Summer intern supervisor for CaMLSys@UniOfCam

Intern supervisor, University of Cambridge, Department of Computer Science and Technology, 2023

I supervised Allen Cong during his Summer Internship. The project was related to optimising a CV task on a Rock64 Rock Pi 4 SE equipped with an Intel RealSense camera.

Master Thesis Supervisor

Thesis supervisor, University of Cambridge, Department of Computer Science and Technology, 2023

I supervised Adriano Guastella (from the University of Bologna) during his Master’s Thesis project when he was visiting the Computer Laboratory. The work investigated the intersection between Powerpropagation, Sparse Weight Activation Training, and federated learning.

MPhil Thesis Supervisor

Thesis supervisor, University of Cambridge, Department of Computer Science and Technology, 2023

I co-supervised Bao Nguyen during his MPhil Thesis project. The work investigated the possibility of applying sheaf neural networks in the context of federated learning. A research paper describing our approach is available on arXiv.