UW Madison researchers propose a three modality, early fusion data integration model to capture prognostic information from genomics, pathology, and radiology. This early fusion multimodal deep learning model for survival prognosis involves a “micro” to “macro” approach that progressively combines information across genomics, pathology, and radiology. Incongruence between data type representations is addressed by organizing genomics data into related disease-specific gene-family sets and the imagining data (histology whole slide images, MRI) into smaller patch images. By employing attention mechanisms between data types, deep learning embeddings representing each image patch can be acquired as directed by the other contributing sources. Sequentially informed representations are created by first acquiring genomic-directed histology embeddings, which are then used to direct the embeddings of radiology data. While the methodology is applied in the context of developing integrated prognostic biomarkers for glioblastoma across radiology, pathology, genomics and patient demographics, the proposed framework is applicable to building prognostic and predictive biomarkers across different types of cancers.