HER2 Expression Prediction with Flexible Multi-Modal Inputs via Dynamic Bidirectional Reconstruction

摘要

In the field of HER2 expression level assessment for breast cancer, clinical evaluations often rely on the synergistic analysis of both H&E and IHC stained images. However, acquiring dual-modality images for the same patient is frequently hindered by complex clinical workflows and high costs, resulting in missing modalities. To address this challenge, we propose an adaptive bimodal input prediction framework that flexibly supports both single-modality and dual-modality inputs. This framework employs a dynamic branch selection mechanism to overcome the rigid dependency of existing models on complete inputs, enabling accurate predictions using either H&E or IHC images alone, while retaining the ability for joint inference when both modalities are available. The core technical innovations include: a missing modality branch selector that dynamically activates either a modality completion process or an end-to-end dual-modality inference pipeline based on the available input; and a cross-modal generative adversarial network (CM-GAN) that facilitates context-aware reconstruction of the missing modality in the feature space. This design improves the prediction accuracy from 71.44% to 94.25% when using single-modality H&E images, significantly mitigating performance degradation caused by incomplete information. Experimental results demonstrate that the proposed framework achieves a prediction accuracy of 95.09% with full dual-modality input and maintains a high reliability of 90.28% under single-modality conditions. By adopting this ''dual-modality preferred, single-modality compatible'' flexible architecture, healthcare institutions can achieve near dual-modality accuracy without mandating synchronized acquisition of both image types. This is particularly valuable for regions with limited IHC staining infrastructure, offering a cost-effective clinical solution and substantially enhancing the accessibility of HER2 expression level assessment.