Federated learning^20-24 develops a decentralized training schema for privacy-preserving learning, enabling multiple clients to learn a network collaboratively without sharing their private data. Note that conventional federated learning frameworks require the models of different clients to have the same architectures²⁵, which hinders the model customization for different applications. Li et al.²⁶ proposed FedMD to tackle the heterogeneous models through knowledge distillation. In some cases, data from different clients can be non-i.i.d, making conventional federated learning challenging to converge. Li et al.²⁷ introduced FedProx to tackle the heterogeneity in data and systems. Smith et al.²⁸ introduced federated multi-task learning and designed a robust and systems-aware optimization method to address this problem, termed MOCHA, to handle practical systems issues. Further, federated transfer learning²⁹ is another topic related to this work, which is proposed to transfer knowledge across domains in federated settings. This work focused on a more challenging setting, where both model-privacy and data-privacy were taken into consideration, and all information, including data domain, tasks and model architectures, are agnostic.

Knowledge amalgamation focuses on a student learning problem from multiple teachers models in different domains^10-12,30-34. It aims to train a versatile student model that can handle all tasks from teachers. For example, Shen et al.¹⁰ proposed a layer-wise coding approach to fuse the representation of different tasks and learn a comprehensive classifier from multiple teacher classifiers. Luo et al.¹¹ introduced the idea of common feature learning to extract shared knowledge between teachers. Ye et al.³⁰ proposed a multi-task amalgamation method, which learns a superior student model from teachers to handle various tasks, including semantic segmentation, depth estimation and surface normal estimation. Luo et al.¹² introduced the dual-step adaptive competitive-cooperation training approach for learning the multi-talent student model. Knowledge amalgamation has recently been extended to non-euclidean data like the graph, where a multi-talent student model was trained for point cloud classification and segmentation³⁴. Zhang et al.³³ introduced a new knowledge amalgamation method for transformer, where the student model was trained for object detection. In the current work, a new setting was investigated for knowledge amalgamation, in which the teacher models were not directly available.

As an increasing number of pre-trained models have been released by developers, reusing pre-trained models has become a common practice in the community for solving downstream tasks. The seminal work of Hinton⁴ established the basic framework of knowledge distillation, which aimed to learn a lightweight student model from cumbersome teachers. Following this pioneering framework, many algorithms have been proposed for model reusing. In particular, Ahn et al.⁸ proposed VKD to transfer the representative ability of a pre-trained model to a student. Romero et al.⁵ proposed FitNets to transfer the feature extraction ability of the pre-trained model, which adopted the feature map as the supervision. Lopes et al.³⁵ proposed data-free knowledge distillation to reuse public models without accessing the original training data. Jiao et al.³⁶ utilized knowledge distillation to compress a transformer model for efficient inference. In the current work, a selective model reusing problems in federated settings was taken into consideration, and FedSA was proposed to train a student model on a different task.

M= {M1,M2,…,MN} denotes a set of pre-trained models. The models are trained on the local private datasets D={D1,D2,…,DN}. The models and datasets can only be accessed on their servers. The goal of FedSA is to obtain a model MT that excels on the target task T with the help of several decentralized pre-trained models. For ease of description, θ‾n indicates the parameters of nth pre-trained model, and θT indicates the parameters of the target model. The annotated data of target task T is DT={{x1,y1},{x2,y2},…,{xM,yM}}. The expensive labeling costs cause the amount of annotated data to be usually limited. Besides, the method was performed on resource limited device.

A novel framework named Federated Knowledge Amalgamation (FedSA) was proposed. An overall framework of FedSA is shown in Fig. 2. With saliency-based method, the proper models S^ are chosen for further selective aggregation. Owing to the progressive learning of the target model, the process of knowledge selection and selective aggregation is iterative. After several rounds of selective aggregation, the target model was adapted to the target data. The target model learned task-related knowledge through target task adaptation, and achieved competitive performance.

With massive variation in knowledge among models, the proper model is more beneficial to the target model, so it is essential to determine a way to select the proper models. The knowledge about target task is pivotal before selection, so the target model was trained with limited target data.

Then the saliency-based method was adopted to choose proper models. The saliency map is widely used to explain the attention of models, it reflects the knowledge of models, so the saliency map was adopted by the knowledge selection to measure the knowledge transferability. In detail, the representation of models were calculated by averaging the saliency map of target data. The transferability scores are pair-wise similarity of model’s representation.

Let $a_{n,j}^k\in {\mathbb{R}}^{WHC}$ indicate saliency map for one input image xj in the probe data ${\mathcal{D}}_{\mathcal{T}}$. It can be computed through one single forward and backward propagation,

where, rk is the network’s hidden representation of the k-th layer. Let $A_n^k$ denote the overall saliency map for the k-th layer of n-th model Mn. It can be formed by averaging the all saliency maps $a_{n,j}^k{|}_{j=1}^M$. Formally, we have $A_n^k=\frac{1}{M}{\sum }_{j=1}^M{a}_{n,j}^k$. After measuring N attribution maps for the models in M, we get $\mathcal{A}=\{A_1^k,A_2^k,\dots ,A_N^k\}$. The transferability score of two models can be estimated as follows:

where, $cos\_sim(A_n^k,A_{\mathcal{T}}^k)=\frac{A_n^k·A_{\mathcal{T}}^k}{\parallel A_n^k\parallel ·\parallel A_{\mathcal{T}}^k\parallel }$, M is the number of target data. Then the selection probabilities are calculated by normalizing the transferability scores between the target model MT and candidate models ${\left\{M_n\right\}}_{n=1}^N$. The probabilities are formed as:

The selected model indexes set $\widehat{\mathcal{S}}$ is established by replacement sampling with probabilities $\{p_1,p_2,\dots ,p_N\}$. The theoretical analysis in the supplemental materials indicates that this replacement sample policy guarantees the convergence of the algorithm.

After the knowledge selection, the selected model set $\widehat{\mathcal{S}}$ has been established. The selective aggregation distills the knowledge from the selected decentralized models to the target model. Due to privacy issues, models and data may only be allowed for internal usage.

In FedSA, a local student model was employed as the messenger of communication between the target and pre-trained models. It learns the knowledge from the pre-trained models and is uploaded to the central device for further aggregation. Most neural networks can be divided into the encoder and the decoder, and the output features of the encoder include rich and semantic information. Also, the feature extraction ability of the teacher model is more generalizable. Therefore, the local student learns the knowledge from the local teacher model through the feature-based knowledge distillation^5-37.

Due to the heterogeneous network architectures, the output feature dimensions of teacher and student models could be distinct. A translator block formed by three convolutions with 1×1 kernel was used to align the teacher and student output dimensions. The student output was converted through the block to a predefined output length. Let ${\mathbb{F}}_n^s$ and ${\mathbb{F}}_n^p$ indicate the aligned features of the student model and teacher model Mn respectively. The X denotes the training data of model Mn. The θnt indicate the parameters of the n-th local student in t-th iteration. In order to encourage the intermediate output of the student model to imitate that of the teacher model, the loss of feature knowledge distillation is computed as follows:

After the local knowledge distillation, the parameters of local students θnt were uploaded to the center server.

To aggregate the knowledge of different local student models, the update of the target model’s parameters with K selected teachers is as follows:

Direct access to pre-trained models was avoided by the local student models, ensuring the model and data privacy. Besides, the most of the calculations was performed on the server, and the aggregation on the central device was designed to be very efficient, so it is easy to be performed on device.

In the selective aggregation, the target model MT only learns the knowledge about generalizable feature extraction, so it is natural to make target task adaptation for more incredible performance. In detail, the decoder part of the target model was trained with the annotated target data. The annotated target data is usually limited, so it can be performed efficiently on device.

The proposed method is summarized in Algorithm 1, which includes three stages: 1) saliency-based knowledge selection; 2) selective aggregation; 3) task adaptation. The overall objective function of FedSA is as follows:

where, ${\mathcal{L}}_{task}$ refers to the loss function of target task.

The single-task amalgamation experiments were conducted with the adoption of three public benchmark datasets, i.e., CIFAR10, CIFAR100¹⁷ and ImageNet32¹⁸. Among them, CIFAR10 and CIFAR100 were used as the source dataset to construct the target or probe dataset, and ImageNet employed to construct the local private dataset. More specifically, to compose the target dataset, {1%, 5%, 10%, 100%} images were randomly extracted from each class of the source dataset CIFAR10 or CIFAR100. ImageNet32 was divided into ten subsets since we have ten local teachers, which contain 100 classes each without overlap, and those sub-datasets were used as the local private datasets. ResNet34³⁸ was adopted as the network structure of local private teachers, and those local teacher models were pre-trained on their corresponding private datasets. ResNet34, ResNet18³⁸, VGG19³⁹, WRN_16_2⁴⁰ and MobileNetV2⁴¹ were utilized as the option of network structures of local student models and target model for homogeneous and heterogeneous experiment settings. Experiments for multi-task amalgamation were performed on Taskonomy dataset¹⁹. The taskonomy dataset includes over 4 million images of indoor scenes from around 600 buildings, and each image has 26 annotations for all kinds of visual tasks. Moreover, Taskonomy also provided 26 pre-trained models focusing on different visual tasks. The network structure of these models can be unified into two parts, namely, the encoder and the decoder. A visual task was selected from the task dictionary as the target task and the rest 25 off-the-shelf ResNet50 with different decoders used as local teacher models to help train the target model for the chosen task.

All the methods were implemented in PyTorch on a Quadro P6000 GPU in the experiment. Single-task amalgamation experiments focused on classification tasks, where pre-trained local private teacher models and the target student model were responsible for different classes. As for multi-task amalgamation, the target student model was expected to solve a new visual task. A task from the task dictionary was chosen as the target task, and the pre-trained models of the remaining tasks used as the local teacher models. As for fair comparison, all compared methods utilized the knowledge of the pre-trained models used in FedSA, and experiment settings like the network architecture and target data were kept the same. The hyperparameters settings in the experiment will be unified according to the impact results in the algorithm. For more details, refer to the supplementary information.

Single-task amalgamation focuses on the classification task. Different local tasks refer to non-overlap classification classes, and the classification ability of local teacher models is transferred to help classify the new categories in the target task. To show the effectiveness of the proposed method, the following experiments were conducted.

Eight methods Scratch Training, Transfer, KACC¹⁰, KACFL¹¹, SOKA-Net¹², FedAvg+²⁰, FedMD+²⁶ and FedProx+²⁷ were adopted as baselines in experiments to validate the effectiveness of the proposed method. For Scratch Training, the target dataset was taken to train the models directly. The Transfer method adopted the target dataset to train the models pre-trained with subsets of the ImageNet32 (local private datasets). As no method directly addresses the problem, three federated methods, FedAvg+, FedMD+ and FedProx+ were modified for fair comparison. In order to keep the same privacy implication, the feature-based knowledge distillation step was also in the local update phase of FedAvg+, FedMD+ and FedProx+, but there is no knowledge selection phase in these methods. After the iterative federated learning phrase, the model from three federated methods was fine-tuned on the downstream target data. Three knowledge amalgamation methods, KACC¹⁰, KACFL¹¹ and SOKA-Net¹² were also selected to validate the performance of FedSA further. We relaxed the data privacy constraints in these knowledge amalgamation methods.

Table 1 shows the results of the proposed FedSA and baselines, the proposed FedSA achieved the SOTA results in all the experimental settings. The transfer methods achieved better results than the scratch training method, owning to the knowledge in the pre-trained model. Compared with the modified federated learning methods, the results are improved due to the effectiveness of collaboration, but differences still exist. In addition, several state-of-art knowledge amalgamation methods are inferior to our innovative method FedSA. In order to further complete the conception of model privacy protection, extensive experiments on heterogeneous networks were conducted, in which different nodes could exhibit diverged model architectures. FedSA also achieved the best performance.

In order to investigate the effectiveness of the selection and aggregation strategy, several methods with different knowledge selection and aggregation strategies were implemented for comparison. Apart from the proposed method FedSA, four knowledge selection strategies and three knowledge aggregation ways were taken into consideration. The following are the knowledge selection strategies. Random means model selection with equal probability. Fixed means model selection with the highest transferability scores, and the selected models set are immutable. Top-K means the selection of the k models with the highest transferability scores, while Least-K with the lowest scores. We have three different aggregation ways: weighted with positive correlation to transferability scores (referred to as Positive Weighted), weighted with negative correlation to similarity (referred to as Negative Weighted), and Unweighted.

Table 2 shows the results of the methods with the different knowledge selection and amalgamation strategies. Overall, the performance of FedSA, with high accuracy and stability advantages, is quite competitive for all settings. Fixed gets the worst results due to the overfitting problem. Random sometimes may exhibit good performance but is not applicable for practical applications since it is unstable. Least-K knowledge selection method is less competitive than Top-K and the selection method of FedSA for fewer similarities in saliency maps. However, with the updation of the target model, the selected models tend to be fixed with Positive Weighted, which may also cause the overfitting problem and is detrimental to improving the effectiveness of the global model. As a result, FedSA, with stability and mobility, can select models with high similarity and avoid the problem of over-fitting to some degree.

In this part, the transferability scores of different stages when training on CIFAR10 and CIFAR100 are visualized in Fig. 3. It indicates that, as the knowledge of the target model increases, the most proper models for the target model are changed, especially from the first phase of knowledge selection to the second phase. Therefore, it indicates that knowledge selection needs to be dynamic. It is more in line with the characteristics of progressive learning.

Analytical experiments Several experiments were conducted on the different number of nodes selected from clients to explore the factors affecting the results of the proposed method. The experiment settings of this section are the same as those of the single-task experiments, and the number of chosen nodes ranges from one to five. The Results are shown in Fig. 3. From the results, fewer nodes may lead to limited knowledge transfer; more nodes may result in off-task knowledge transfer and unpredictable results.

The effectiveness of transferability scores Experiments were conducted to demonstrate the effectiveness of transferability scores in this section. 10% of CIFAR data were used as the target dataset. Teacher models were pre-trained by local private datasets. Transferability scores were calculated between the target and local teacher models, respectively, and then normalized to 0 to 1 for contrast. All parameters of the local teacher models, except the last layer, were frozen, and the teacher models were trained to evaluate the transfer accuracy. Table 3 shows that the transfer accuracy and transferability score generally correlate positively.

In order to show the effectiveness of the proposed method on the cross-task settings, experiments on Taskonomy dataset were conducted¹⁹. Taskonomy provides 26 types of annotations for different visual tasks and the corresponding pre-trained models. In this section, FedSA was deployed to amalgamate knowledge from different models to resolve a different task.

The taskonomy dataset was divided into training, validation and test set. The current experiment adopted these three parts for the local private, target and test datasets. In the experiment, one of the tasks in taskonomy was selected as the target task, while others were regarded as the task of the pre-trained teacher models. In this experiment, semantic segmentation and object classification were selected as the target task to validate the effectiveness of FedSA. Although the proposed FedSA was only verified on object classification and semantic segmentation, the method can be easily adapted to other visual tasks. As shown in Table 4, the results indicate that the proposed FedSA is quite competitive compared with others, especially on semantic segmentation tasks. The results of Scratch Training are unsatisfactory for the limited labeled data. As for the Transfer method, the result was improved effectively with the prior knowledge from the pre-trained models. Owing to the redundancy of knowledge, FedAvg+ does not exhibit excellent performance. The KACC method amalgamates multiple pre-trained models so that the target model can make an accurate prediction. In the proposed method FedSA, the appropriate knowledge from different models was amalgamated into the target model with task adaption. Therefore, more accurate results were achieved by the proposed method FedSA.

In this section, three images from the test set in taskonomy were chosen as the input of the target model, and the semantic segmentation predictions from the target models are visualized in Fig. 4. The figure shows the effect of the proposed FedSA more intuitively. In Fig. 4, the proposed FedSA predicts some small objects correctly, while other methods ignore or make wrong predictions. The visualization results indicate that the proposed FedSA can achieve a more accurate target model than other methods.

The computation resources and network communication ability of the device are usually limited. This part compared the computation and communication costs of some SOTA knowledge amalgamation methods and FedSA. FLOPs, the abbreviation of floating point operations, means floating-point operands and is regarded as the amount of calculation that can measure the complexity of the algorithm or model. Here, in order to show the feasibility of the proposed FedSA on practical deployment, we compared and analyzed the FLOPs. C.B. means the total number of bytes that needs to be communicated. It includes device-to-server and server-to-device communication.

For simple comparison, ResNet18 was taken as the network architecture of the target model and CIFAR10 as the target dataset. Here, it is assumed that the number of the target dataset is 5000 (10% of the whole CIFAR10 dataset), the number of selective models is 3, the communication rounds are 50, and the epochs for fine-tuning is 30. The number of pre-trained model data is 383,450 (30% of ImageNet32 dataset). The ratio of backward-forward FLOPs in neural networks is typically between 1:1 and 3:1 and almost often close to 2:1⁴². We take 2:1 in the current work.

The amalgamation procedure of KACC follows two steps: feature amalgamation and parameter learning. Thus, we get $7.39M\times 383,450\times 50=141.68T$ FLOPs for feature amalgamation and (556.71M×3+7.39M×2)×383,450×50=32303.94T FLOPs for parameter learning. The fine-tuning of target model is $(556.71M+5.12K\times 2)\times 5000\times 30=83.51T$, as the ratio of backward-forward FLOPs for linear layer is 1:1. The middle feature maps are required to be communicated between the device and servers, so the total number of communication bytes is $8K\times 3\times 383,450\times 50=438.82G$.

There are individual adaption layers and shared extractor in the feature amalgamation of KACFL. Therefore, we get $(5.54M+1.39M)\times 4\times 383,450\times 50=531.46T$ FLOPs for feature amalgamation and $(556.71M\times 3+5.54M\times 4\times 2+1.39M\times \times 2)\times 383,450\times 50=33083.49T$ FLOPs for the parameter learning. The fine-tuning of target model is $(556.71M+5.12K\times 2)\times 5000\times 30=83.51T$. The output feature maps of pre-trained teachers need communication in KACFL, so the total number of communication bytes is $8K\times 3\times 383,450\times 50=438.82G$.

In SOKA-Net, the feature are amalgamated with MIA module. We get $4.19M\times 3\times 383,450\times 50=240.99T$ FLOPs for feature amalgamation and $(556.71M\times 3+4.19M\times 3\times 2)\times 383,450\times 50=32502.56T$ FLOPs for target model parameter learning. The fine-tuning of target model is $(556.71M+5.12K\times 2)\times 5000\times 30=83.51T$. The output feature maps of pre-trained teachers also need communication in SOKA-Net, so the total number of communication bytes is $8K\times 3\times 383,450\times 50=438.82G$.

FedSA mainly has two steps for the knowledge amalgamation procedure: aggregate the knowledge from selective clients and utilize the target dataset to fine-tune the last linear layer of the target model. Thus, we get $11.17M\times 3\times 50=1675.5M$ FLOPs for knowledge aggregation, as only the parameters of agent models need averaging, and $(556.71M+5.12K\times 2)\times 5000\times 30=83.51T$ FLOPs for fine-tuning. Besides, the parameters of target models are delivered, so the total number of communication bytes is $11.17M\times 3\times 2\times 50=3.43G$.

The computation and communication cost analysis results for our proposed method, which are compared with some current SOTA knowledge amalgamation methods, are shown in Table 5. It indicates that the framework in the current work exhibits far lower computation (390×) and communication costs (128×) than other knowledge amalgamation methods, as we can obtain a target student without any assistance from the teachers’ training data, and demonstrating the feasibility of FedSA for practical deployment on the device.