Skip to content
2000
Volume 18, Issue 10
  • ISSN: 2352-0965
  • E-ISSN: 2352-0973

Abstract

Background

Multimodal medical image fusion is a core tool to enhance the clinical utility of medical images by integrating complementary information from multiple images. However, the existing deep learning-based fusion methods are not good at effectively extracting the key target features, and easy to make the results blurry.

Objective

The main objective of the paper is to propose a medical image fusion method that effectively extracts features from source images and preserves them in the fused results.

Methods

The prior knowledge and a dual-branch U-shaped structure are employed by the proposed method to extract both the local and global features from images of different modalities. A novel Transformer module is designed to capture the global correlations at the super-pixel level. Each feature extraction module uses the Haar Wavelet downsampling to reduce the spatial resolution of the feature maps while preserving as much information as possible, effectively reducing the information uncertainty.

Results

Extensive experiments on public medical image datasets and a biological image dataset demonstrated that the proposed method achieves superior performance in both qualitative and quantitative evaluations.

Conclusion

This paper applies prior knowledge to medical image fusion and proposes a novel dual-branch U-shaped medical image fusion network. Compared with nine state-of-the-art fusion methods, the proposed method produces better-fused results with richer texture details and better visual quality.

© 2025 Bentham Science Publishers
Loading

Article metrics loading...

/content/journals/raeeng/10.2174/0123520965344127250314232839
2025-04-28
2026-01-01

  • From This Site

    /content/journals/raeeng/10.2174/0123520965344127250314232839
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. ZhangW. LuY. ZhengH. YuL. MBRARN: Multibranch residual attention reconstruction network for medical image fusion.Med. Biol. Eng. Comput.202361113067308510.1007/s11517‑023‑02902‑2 37624534
     [Google Scholar]
  2. MaJ. TangL. FanF. HuangJ. MeiX. MaY. SwinFusion: Cross-domain long-range learning for general image fusion via swin transformerIEEE/CAA Journal of Automatica Sinica2022971200121710.1109/JAS.2022.105686
     [Google Scholar]
  3. ZhuR. LiX. ZhangX. WangJ. HID: The hybrid image decomposition model for MRI and CT fusion.IEEE J. Biomed. Health Inform.202226272773910.1109/JBHI.2021.3097374 34270437
     [Google Scholar]
  4. XuY. WangZ. WuS. ZhanX. MPCT: A medical image fusion method based on multi-scale pyramid convolution and Transformer.Biomed. Signal Process. Control202510110712910.1016/j.bspc.2024.107129
     [Google Scholar]
  5. WuJ. GeA. LiuS. WangQ. ZhuD. ChenX. An optimized inversion method for hyperspectral image fusion based on a hue–intensity–saturation, wavelet, and trust-region conjugate gradient method.Electronics202413225210.3390/electronics13020252
     [Google Scholar]
  6. MaT. GeS. WangC. ZhuH. LiDAR and hyperspectral remote sensing image fusion using extended extrema morphological profiles for semantic segmentationSixteenth Int. Conf. Digit. Image Process, vol. Vol. 13274, 2024pp. 153-16010.1117/12.3037704
     [Google Scholar]
  7. SinghS. SinghH. BuenoG. DenizO. SinghS. MongaH. HrisheekeshaP.N. PedrazaA. A review of image fusion: Methods, applications and performance metrics.Digit. Signal Process.202313710402010.1016/j.dsp.2023.104020
     [Google Scholar]
  8. LiL. ShiY. LvM. JiaZ. LiuM. ZhaoX. ZhangX. MaH. Infrared and visible image fusion via sparse representation and guided filtering in Laplacian pyramid domain.Remote Sens.20241620380410.3390/rs16203804
     [Google Scholar]
  9. PanY. ZhaoH. Color polarization image fusion algorithm based on wavelet transformFourth International Conference on Advanced Algorithms and Signal Image Processing (AASIP 2024), Kuala Lumpur, Malaysia, 2024, pp. 22-26.10.1117/12.3045493
     [Google Scholar]
  10. LiL. ZhaoX. HouH. ZhangX. LvM. JiaZ. MaH. Fractal dimension-based multi-focus image fusion via coupled neural P systems in NSCT domain.Fractal and Fractional202481055410.3390/fractalfract8100554
     [Google Scholar]
  11. ZhouT. LiQ. LuH. ChengQ. ZhangX. GAN review: Models and medical image fusion applications.Inf. Fusion20239113414810.1016/j.inffus.2022.10.017
     [Google Scholar]
  12. LiuY. ChenX. ChengJ. PengH. A medical image fusion method based on convolutional neural networks20th International Conference on Information Fusion Xi’an, China, 2017, pp. 1-7.10.23919/ICIF.2017.8009769
     [Google Scholar]
  13. FuJ. LiW. DuJ. HuangY. A multiscale residual pyramid attention network for medical image fusion.Biomed. Signal Process. Control20216610248810.1016/j.bspc.2021.102488
     [Google Scholar]
  14. LiangX. HuP. ZhangL. SunJ. YinG. MCFNet: Multi-layer concatenation fusion network for medical images fusion.IEEE Sens. J.201919167107711910.1109/JSEN.2019.2913281
     [Google Scholar]
  15. XuH. MaJ. JiangJ. GuoX. LingH. U2Fusion: A unified unsupervised image fusion network.IEEE Trans. Pattern Anal. Mach. Intell.202244150251810.1109/TPAMI.2020.3012548 32750838
     [Google Scholar]
  16. ZhangY. LiuY. SunP. YanH. ZhaoX. ZhangL. IFCNN: A general image fusion framework based on convolutional neural network.Inf. Fusion2020549911810.1016/j.inffus.2019.07.011
     [Google Scholar]
  17. XieX. ZhangX. TangX. ZhaoJ. XiongD. OuyangL. YangB. ZhouH. LingB.W.K. TeoK.L. MACTFusion: Lightweight cross transformer for adaptive multimodal medical image fusionIEEE J. Biomed. Health Inform.2024PP11210.1109/JBHI.2024.339162038640042
     [Google Scholar]
  18. MaJ. YuW. LiangP. LiC. JiangJ. FusionGAN: A generative adversarial network for infrared and visible image fusion.Inf. Fusion201948112610.1016/j.inffus.2018.09.004
     [Google Scholar]
  19. MaJ. XuH. JiangJ. MeiX. ZhangX.P. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion.IEEE Trans. Image Process.2020294980499510.1109/TIP.2020.2977573 32167894
     [Google Scholar]
  20. LiJ. HuoH. LiC. WangR. SuiC. LiuZ. Multigrained attention network for infrared and visible image fusion.IEEE Trans. Instrum. Meas.20217011210.1109/TIM.2020.3029360
     [Google Scholar]
  21. LiJ. HuoH. LiC. WangR. FengQ. AttentionFGAN: Infrared and visible image fusion using attention-based generative adversarial networks.IEEE Trans. Multimedia2021231383139610.1109/TMM.2020.2997127
     [Google Scholar]
  22. ZhouT. ChengQ. LuH. LiQ. ZhangX. QiuS. Deep learning methods for medical image fusion: A review.Comput. Biol. Med.202316010695910.1016/j.compbiomed.2023.106959 37141652
     [Google Scholar]
  23. LiuY. ZangY. ZhouD. CaoJ. NieR. HouR. DingZ. MeiJ. An improved hybrid network with a transformer module for medical image fusion.IEEE J. Biomed. Health Inform.20232773489350010.1109/JBHI.2023.3264819 37023161
     [Google Scholar]
  24. DosovitskiyA. BeyerL. KolesnikovA. WeissenbornD. ZhaiX. UnterthinerT. DehghaniM. MindererM. HeigoldG. GellyS. UszkoreitJ. HoulsbyN. An image is worth 16x16 words: Transformers for image recognition at scaleProceedings of the International Conference on Learning Representations (ICLR) 2021, pp. 1-3.
     [Google Scholar]
  25. VsV. ValanarasuJ.M.J. OzaP. PatelV.M. Image fusion transformer2022 IEEE International Conference on Image Processing (ICIP) Bordeaux, France, 16-19 October 2022, pp. 3566-3570.10.1109/ICIP46576.2022.9897280
     [Google Scholar]
  26. WangZ. ChenY. ShaoW. LiH. ZhangL. SwinFuse: A residual swin transformer fusion network for infrared and visible images.IEEE Trans. Instrum. Meas.20227111210.1109/TIM.2022.3216413
     [Google Scholar]
  27. ZhangJ. LiuA. WangD. LiuY. WangZ.J. ChenX. Transformer-based end-to-end anatomical and functional image fusion.IEEE Trans. Instrum. Meas.20227111110.1109/TIM.2022.3200426
     [Google Scholar]
  28. TangW. HeF. LiuY. DuanY. MATR: Multimodal medical image fusion via multiscale adaptive transformer.IEEE Trans. Image Process.2022315134514910.1109/TIP.2022.3193288 35901003
     [Google Scholar]
  29. LinX. MaL. LiuW. ChangS.F. Context-gated convolution. Computer Vision—ECCV 2020. ECCV 2020. VedaldiA. BischofH. BroxT. Frahmand J.M. ChamSpringer20201236310.1007/978‑3‑030‑58523‑5_41
     [Google Scholar]
  30. TangW. HeF. LiuY. TCCFusion: An infrared and visible image fusion method based on transformer and cross correlation.Pattern Recognit.202313710929510.1016/j.patcog.2022.109295
     [Google Scholar]
  31. LiK. WangY. GaoP. SongG. LiuY. LiH. QiaoY. Uniformer: Unified transformer for efficient spatiotemporal representation learningarXiv preprint20222201.04676
     [Google Scholar]
  32. RaghuM. UnterthinerT. KornblithS. ZhangC. DosovitskiyA. Do vision transformers see like convolutional neural networks?Adv. Neural Inf. Process. Syst.2021341211612128
     [Google Scholar]
  33. LiuZ. LinY. CaoY. HuH. WeiY. ZhangZ. LinS. GuoB. Swin transformer: Hierarchical vision transformer using shifted windowsProceedings of the IEEE/CVF International Conference on Computer Vision Montreal, QC, Canada, 10-17 October 2021, pp. 9992-10002.10.1109/ICCV48922.2021.00986
     [Google Scholar]
  34. Hyeon-WooN. Yu-JiK. HeoB. HanD. OhS.J. OhT.H. Scratching visual transformer’s back with uniform attentionProceedings of the IEEE/CVF International Conference on Computer Vision Paris, France, 01-06 October 2023, pp. 5784-5795.10.1109/ICCV51070.2023.00534
     [Google Scholar]
  35. ZhaoH. Equivariant multi-modality image fusionProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Seattle, WA, USA, 16-22 June 2024, pp. 25912-25921.10.1109/CVPR52733.2024.02448
     [Google Scholar]
  36. XuG. LiaoW. ZhangX. LiC. HeX. WuX. Haar wavelet downsampling: A simple but effective downsampling module for semantic segmentation.Pattern Recognit.202314310981910.1016/j.patcog.2023.109819
     [Google Scholar]
  37. ZamirS.W. AroraA. KhanS. HayatM. KhanF.S. YangM.H. Restormer: Efficient transformer for high-resolution image restorationProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) New Orleans, LA, USA, 18-24 June 2022, pp. 5718-5729.10.1109/CVPR52688.2022.00564
     [Google Scholar]
  38. ZhaoH. GouY. LiB. PengD. LvJ. PengX. Comprehensive and delicate: An efficient transformer for image restorationProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17-24 June 2023, pp. 14122-14132.10.1109/CVPR52729.2023.01357
     [Google Scholar]
  39. RonnebergerO. FischerP. BroxT. U-net: Convolutional networks for biomedical image segmentationMedical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference Munich, Germany, October 5-9 2015, pp. 234-241.10.1007/978‑3‑319‑24574‑4_28
     [Google Scholar]
  40. ZhaoZ. BaiH. ZhangJ. ZhangY. XuS. LinZ. TimofteR. VanL. CDDFuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusionProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17-24 June 2023, pp. 5906-5916.10.1109/CVPR52729.2023.00572
     [Google Scholar]
  41. LiangP. JiangJ. LiuX. MaJ. Fusion from decomposition: A self-supervised decomposition approach for image fusionEuropean Conference on Computer Vision Springer, Cham, 03 November 2022, pp 719–735.10.1007/978‑3‑031‑19797‑0_41
     [Google Scholar]
  42. JohnsonK.A. BeckerJ.A. The whole brain atlasAvailable from: http://www.med.harvard.edu/aanlib
  43. TsienR.Y. The green fluorescent protein.Annu. Rev. Biochem.199867150954410.1146/annurev.biochem.67.1.509 9759496
     [Google Scholar]
  44. MaJ. MaY. LiC. Infrared and visible image fusion methods and applications: A survey.Inf. Fusion20194515317810.1016/j.inffus.2018.02.004
     [Google Scholar]
  45. GaoY. MaS. LiuJ. LiuY. ZhangX. Fusion of medical images based on salient features extraction by PSO optimized fuzzy logic in NSST domain.Biomed. Signal Process. Control20216910285210.1016/j.bspc.2021.102852
     [Google Scholar]
  46. LiW. DuJ. ZhaoZ. LongJ. Fusion of medical sensors using adaptive cloud model in local Laplacian pyramid domain.IEEE Trans. Biomed. Eng.20196641172118310.1109/TBME.2018.2869432 30207947
     [Google Scholar]
  47. TanW. TiwariP. PandeyH.M. MoreiraC. JaiswalA.K. Multimodal medical image fusion algorithm in the era of big data.Neural Comput. Appl.202012110.1007/s00521‑020‑05173‑2
     [Google Scholar]
  48. ZhangH. MaJ. SDNet: A versatile squeeze-and-decomposition network for real-time image fusion.Int. J. Comput. Vis.2021129102761278510.1007/s11263‑021‑01501‑8
     [Google Scholar]
  49. XuH. MaJ. EMFusion: An unsupervised enhanced medical image fusion network.Inf. Fusion20217617718610.1016/j.inffus.2021.06.001
     [Google Scholar]
  50. LiuJ. LinR. WuG. LiuR. LuoZ. FanX. CocoNet: Coupled contrastive learning network with multi-level feature ensemble for multi-modality image fusion.Int. J. Comput. Vis.2023128
     [Google Scholar]
/content/journals/raeeng/10.2174/0123520965344127250314232839
Loading
EMIF: Equivariant Multimodal Medical Image Fusion Network Via Super Token and Haar Wavelet Downsampling
Recent Advances in Electrical & Electronic Engineering 18, 2011 (2025); https://doi.org/10.2174/0123520965344127250314232839
/content/journals/raeeng/10.2174/0123520965344127250314232839
/content/journals/raeeng/10.2174/0123520965344127250314232839
Loading

Data & Media loading...


  • Article Type:     Research Article
Keyword(s): image fusion; image processing; medical diagnostic imaging; Medical image fusion; MRI; transformer

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error
Please enter a valid_number test