March 20, 2025

The AI4EOSC team has contributed to this leaderboard by training a large language model (LLM) in a federated manner for medical applications, achieving the top position in the FlowerTune LLM leaderboard in the medical field. Below are the key technical aspects of this achievement.

Federated Training with Flower

The model was trained using Flower, a federated learning framework that enables multiple data owners to collaborate in model training without sharing sensitive data. This methodology is particularly relevant in the medical field, where privacy and legal restrictions prevent centralized processing of clinical data. By using federated learning, each institution trains the model locally with its data and only shares updated parameters, which are then aggregated to build a more robust global model.

Implementation of the FedAvgOpt Function

A key innovation in this work is the implementation of the FedAvgOpt aggregation function. This strategy, previously proposed by the team in the preprint Enhancing the Convergence of Federated Learning Aggregation Strategies with Limited Data. FedAvgOpt dynamically adjusts the weights of the clients solving an optimization problem, and it aims to improve model convergence specifically in scenarios with limited or heterogeneous data. However, the effectiveness of this approach is also shown when dealing with huge amounts of data and large models as LLMs. 

Results in the FlowerTune LLM Leaderboard

The federated medical LLM trained and aggregated using FedAvgOpt  achieved an accuracy of 70.80% on PubMedQA, 58.04% on MedMCQA, and 62.84% on MedQA Additionally, the communication budget was slightly larger than 2GB when training the model in a NVIDIA GPU V100, available from the AI4EOSC platform. This demonstrates the model’s efficiency and cost-effective computational resource usage.

Additional Technical Details

• Base Model: The model ContactDoctor/Bio-Medical-Llama-3-8B was used as the starting point for federated training.
• Dataset: The model was trained on a medical dataset available on Hugging Face.
• Simulation: The Flower simulation engine was used to emulate the fine-tuning process of the LLM in a federated manner, allowing training to be performed on a single GPU.

For more information and access to the source code, visit the project’s GitHub repository: https://github.com/judithspd/ai4os-fedllm-medical.

This achievement highlights AI4EOSC’s capability to lead innovations in federated learning applied to LLM, providing solutions that respect privacy and are more efficient with the management of computing resources.