close
Skip to main content
Springer Nature Link
Log in

Multi-modal deep-fusion network for meningioma presurgical grading with integrative imaging and clinical data

  • Original article
  • Published:
Save article
View saved research
BERJAYA The Visual Computer Aims and scope Submit manuscript

Abstract

Predicting meningioma grade before surgery is crucial to making decisions on therapy planning and prognosis prediction. Prior works mainly investigate traditional classification techniques with hand-crafted features or rely on image data only, thus having limited accuracy. In this study, we propose a novel multi-modal classification method, i.e., multi-modal deep-fusion network (MMDF), integrating high-dimensional 3D MRI imaging information and low-dimensional tabular clinical data to classify low-grade and high-grade meningiomas. Specifically, the MMDF adopts two modality-specific branches to extract image and clinical features, respectively, and leverages an image-clinical integration module in the shared branch to fuse cross-model features, while specifically considering the impact of low-dimensional clinical data. Besides, we propose a multi-modal image feature aggregation module to integrate three image modalities in the image-specific branch, which can compensate for the feature distribution gaps among the contrast-enhanced T1, contrast-enhanced T2-FLAIR and ADC modalities. Comprehensive experiments show that our approach significantly outperforms the SOTA methods using imaging data only and those combining image and tabular data both, with an AUC of 0.958, sensitivity of 0.877, specificity of 0.926, and accuracy of 0.921. Our approach holds high potential to aid radiologists in doing the presurgical evaluation for clinical decision making.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
BERJAYA
Fig. 2
BERJAYA
Fig. 3
BERJAYA
Fig. 4
BERJAYA
Fig. 5
BERJAYA
Fig. 6
BERJAYA

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data availability

The dataset cannot be released at present because of the hospital’s requirements for data protection.

Notes

  1. We also found that ADC modality achieves the best results among using the three modalities individually.

References

  1. Abiwinanda, N., Hanif, M., Hesaputra, S.T., Handayani, A., Mengko, T.R.: Brain tumor classification using convolutional neural network. In: World congress on medical physics and biomedical engineering 2018, pp. 183–189. Springer (2019)

  2. Banerjee, S., Mitra, S., Masulli, F., Rovetta, S.: Deep radiomics for brain tumor detection and classification from multi-sequence mri. arXiv preprint arXiv:1903.09240 (2019)

  3. Banerjee, S., Mitra, S., Masulli, F., Rovetta, S.: Deep radiomics for brain tumor detection and classification from multi-sequence mri. arXiv preprint arXiv:1903.09240 (2019)

  4. Cornelius, J.F., Slotty, P.J., Steiger, H.J., Hänggi, D., Polivka, M., George, B.: Malignant potential of skull base versus non-skull base meningiomas: clinical series of 1,663 cases. Acta Neurochirurgica 155(3), 407–413 (2013)

    Article  Google Scholar 

  5. Dai, Z., Liu, H., Le, Q.V., Tan, M.: Coatnet: marrying convolution and attention for all data sizes. Adv. Neural Inf. Process. Syst. 34, 3965–3977 (2021)

    Google Scholar 

  6. Deepak, S., Ameer, P.M.: Brain tumor classification using deep CNN features via transfer learning. Comput. Biol. Med. 111, 103345 (2019)

    Article  Google Scholar 

  7. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  8. El-Dahshan, E.S.A., Mohsen, H.M., Revett, K., Salem, A.B.M.: Computer-aided diagnosis of human brain tumor through mri: a survey and a new algorithm. Expert Syst. Appl. 41(11), 5526–5545 (2014)

    Article  Google Scholar 

  9. Guo, Z., Li, X., Huang, H., Guo, N., Li, Q.: Medical image segmentation based on multi-modal convolutional neural network: study on image fusion schemes. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 903–907. IEEE (2018)

  10. Hawkins-Daarud, A., Rockne, R.C., Anderson, A.R., Swanson, K.R.: Modeling tumor-associated edema in gliomas during anti-angiogenic therapy and its impact on imageable tumor. Front. Oncol. 3, 66 (2013)

    Article  Google Scholar 

  11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition pp. 770–778 (2016)

  12. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708 (2017)

  13. Jie, H., Li, S., Gang, S., Albanie, S.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. PP(99) (2017)

  14. Kunyi, Z.: Imaging principles and clinical applications of magnetic resonance imaging (mri). J. Med. Equip. (2008)

  15. Park, Y.W., Oh, J., You, S.C., Han, K., Ahn, S.S., Choi, Y.S., Chang, J.H., Kim, S.H., Lee, S.K.: Radiomics and machine learning may accurately predict the grade and histological subtype in meningiomas using conventional and diffusion tensor imaging. Eur. Radiol. 29(8), 4068–4076 (2019)

    Article  Google Scholar 

  16. Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: Visual reasoning with a general conditioning layer. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

  17. Pölsterl, S., Wolf, T.N., Wachinger, C.: Combining 3d image and tabular data via the dynamic affine feature map transform. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 688–698. Springer (2021)

  18. Qu, Y., Zhu, H., Cao, K., Li, X., Sun, Y.: Prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer using a deep learning (dl) method. Thorac. Cancer 11(3), 651 (2020)

    Article  Google Scholar 

  19. Saraf, S., McCarthy, B.J., Villano, J.L.: Update on meningiomas. The Oncologist 16(11), 1604–1613 (2011)

    Article  Google Scholar 

  20. Shaowei, Z., Liujun, H., Shaolei, G., Xioayi, L., Zhunyi, Z., Shaochun, S., Shujia, C.: Analysis of meningioma recurrence rates following treatment. Lingnan Modern Clin. Surg. 17(06), 742 (2017)

    Google Scholar 

  21. Srinivas, A., Lin, T.Y., Parmar, N., Shlens, J., Abbeel, P., Vaswani, A.: Bottleneck transformers for visual recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16,519–16,529 (2021)

  22. Surov, A., Gottschling, S., Mawrin, C., Prell, J., Spielmann, R.P., Wienke, A., Fiedler, E.: Diffusion-weighted imaging in meningioma: prediction of tumor grade and association with histopathological parameters. Transl. Oncol. 8(6), 517–523 (2015)

    Article  Google Scholar 

  23. Talo, M., Baloglu, U.B., Yıldırım, Ö., Acharya, U.R.: Application of deep transfer learning for automated brain abnormality classification using mr images. Cogn. Syst. Res. 54, 176–188 (2019)

    Article  Google Scholar 

  24. Tseng, K.L., Lin, Y.L., Hsu, W., Huang, C.Y.: Joint sequence learning and cross-modality convolution for 3d biomedical segmentation. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6393–6400 (2017)

  25. Yan, P.F., Yan, L., Hu, T.T., Xiao, D.D., Feng, J.: The potential value of preoperative mri texture and shape analysis in grading meningiomas: a preliminary investigation. Transl. Oncol. 10(4), 570–577 (2017)

    Article  Google Scholar 

  26. Yang, S.Y., Park, C.K., Park, S.H., Kim, D.G., Chung, Y.S., Jung, H.W.: Atypical and anaplastic meningiomas: prognostic implications of clinicopathological features. J. Neurol. Neurosurg. Psychiatry 79(5), 574–580 (2008)

    Article  Google Scholar 

  27. Yin, B., Liu, L., Zhang, B.Y., Li, Y.X., Li, Y., Geng, D.Y.: Correlating apparent diffusion coefficients with histopathologic findings on meningiomas. Eur. J. Radiol. 81(12), 4050–4056 (2012)

    Article  Google Scholar 

  28. Zeng, Z., Tong, Z., Han, Z., Zhang, Y., Zwiggelaar, R.: The classification of meningioma subtypes based on the color segmentation and shape features. In: Frontier and future development of information technology in medicine and education, pp. 2669–2674. Springer (2014)

  29. Zhang, H., Berg, A.C., Maire, M., Malik, J.: Svm-knn: discriminative nearest neighbor classification for visual category recognition. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 2, pp. 2126–2136. IEEE (2006)

  30. Zhang, H., Mo, J., Jiang, H., Li, Z., Hu, W., Zhang, C., Wang, Y., Wang, X., Liu, C., Zhao, B., et al.: Deep learning model for the automated detection and histopathological prediction of meningioma. Neuroinformatics 19(3), 393–402 (2021)

    Article  Google Scholar 

  31. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921–2929 (2016)

  32. Zhu, H., Fang, Q., He, H., Hu, J., Xu, K.: Automatic prediction of meningioma grade image based on data amplification and improved convolutional neural network. Comput. Math. Methods Med. 2019(33), 1–9 (2019)

    MATH  Google Scholar 

  33. Zhu, Y., Man, C., Gong, L., Dong, D., Tian, J.: A deep learning radiomics model for preoperative grading in meningioma. Eur. J. Radiol. 116, 128–134 (2019)

  34. Zhu, Y., Man, C., Gong, L., Dong, D., Yu, X., Wang, S., Fang, M., Wang, S., Fang, X., Chen, X., et al.: A deep learning radiomics model for preoperative grading in meningioma. Eur. J. Radiol. 116, 128–134 (2019)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the grant from Tianjin Natural Science Foundation (Grant No. 20JCYBJC00960).

Author information

Authors and Affiliations

  1. Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, People’s Republic of China

    Wennan Liu & Liang Wan

  2. College of Intelligence and Computing, Tianjin University, Tianjin, People’s Republic of China

    Tianling Liu

  3. Tianjin Huanhu Hospital, Tianjin University, Tianjin, People’s Republic of China

    Tong Han

Authors
  1. Wennan Liu
    View author publications

    Search author on:PubMed Google Scholar

  2. Tianling Liu
    View author publications

    Search author on:PubMed Google Scholar

  3. Tong Han
    View author publications

    Search author on:PubMed Google Scholar

  4. Liang Wan
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Liang Wan.

Ethics declarations

Conflict of interest of potential conflicts of interest.

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, W., Liu, T., Han, T. et al. Multi-modal deep-fusion network for meningioma presurgical grading with integrative imaging and clinical data. Vis Comput 39, 3561–3571 (2023). https://doi.org/10.1007/s00371-023-02978-9

Download citation

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s00371-023-02978-9

Keywords