Deep Learning Methods for Precise Cardiac Image Segmentation Review

Segmentation with MRI images

MRI images of cardiac regions are a non-invasive approach that shows structural visualization of the heart. MRI does not need any ionizing radiation. Cardiac MRI image facilitates appropriate quantification of both functions and anatomy, and pathological tissues such as scars. Based on this MRI image segmentation, a standardized quantitative cardiac analysis is conducted. The process of MRI cardiac image segmentation is illustrated in Figure 2.^[10]

Close

Figure 2:

Magnetic resonance imaging–based cardiac image segmentation. The schematic (left panel) demonstrates labeled regions: right ventricle (1), left ventricular cavity (2), left ventricular myocardium (3), and papillary muscles/trabeculations (4). The corresponding cardiac MRI slice (right panel) illustrates segmentation contours, with the red dotted line indicating the left ventricular endocardial border and the green dotted line representing the epicardial border; additional outer contour (blue, if present) denotes the right ventricular or overall cardiac boundary. Internal white markings highlight papillary muscles or intracavitary structures within the left ventricle.

Export to PPT

Trans^[10] used a fully convolutional network (FCN) to segment RV, LV, and myocardium appropriately with shorter axis cardiac MRI. This approach is based on FCN to attain accuracy in the segmentation process, which outperforms conventional techniques in accuracy and speed. Many notable works have been done on optimizing network structure to improve feature-based learning ability during the segmentation process. Bernard et al.^[11] modeled a dense U-net to merge multi-scale features for proper segmentation with larger anatomical variability. Zheng et al.^[12] Examined various loss functions such as weighted dice loss, weighted cross entropy, focal loss, and supervision loss, to enhance the segmentation process. Based on FCN, two-dimensional networks are considered indeed of three-dimensional networks. This is due to typical low planar resolution and artefacts identified in cardiac magnetic resonance (MR) scans that restrict the 3D network availability. The main limitation of this approach is the slice-by-slice processing without leveraging the inter-slice dependencies. Therefore, the 2D network fails in segmenting and locating heart slices such as basal slices and apical slices, where ventricles are not determined properly. To combat this issue, various works are done in obtaining contextual information, such as shapes from multi-view images and labels in 2D FCN. Alternatively, spatial information can also be extracted from slices to perform segmentation using recurrent units. These units can then be used for leveraging information over diverse temporal frames in cardiac cycles to enhance temporal and spatial consistency for attaining the finest segmentation.

Other predominant issues associated with 2D and 2D FCN segmentation processes are that the network is trained with a pixel-wise loss function, thus making it unsuitable for learning features specific to the anatomical structures.^[13] Various approaches focus on applying and modeling anatomical constraints to train the network to enhance robustness and accuracy. These are specified in terms of regularization by considering contour, topology, and shape or region information to boost the network to produce plausible anatomical segmentation. The proposed network regularization provides more appropriate segmentation during post-processing stages.

Multitask learning is augmented with the FCN-based cardiac segmentation, to regularise partitioning by training auxiliary tasks that are more appropriate for the segmentation process, such as motion estimation, cardiac function estimation, image reconstruction, and ventricle size classification. Multitask-based network training assists in extracting features that result in enhancing prediction accuracy and learning efficiency. Some of the researchers concentrate on multi-stage pipelining that breaks down the segmentation process into multiple sub-tasks. For instance, localization with a region of interest (ROI) is adopted in some cases. The author in Dou et al.^[14] proposed a network model termed OmegaNet, which may include a U-net for the localization of the cardiac chamber. Here, learning modules are normalized as image orientation, and U-net series are provided for fine-grained segmentation. With ROI localization and input image rotation for canonical orientation, the anticipated model may provide superior outcomes in orientation and in various sizes.

Detailed analysis has been done by merging traditional segmentation techniques with neural networks (NNs), for instance, atlas-based models, deformable models, level sets, and graph-based cut methods. Here, NNs are used in initialization and feature extraction to diminish manual interactions and enhance the segmentation accuracy of traditional segmentation for deployment. For LV segmentation, DL-based models are used in cardiac MRI images. The author used a convolutional NN (CNN) to detect LV automatically and then used an auto-encoder to evaluate LV shape. The estimated shape was utilized to initialize the deformation model for shape refinement. This model converges faster than traditional deformable models, thus achieving better accuracy.^[15] This idea is extended for segmenting RV. Here, a hybrid model is deployed that shows superior segmentation accuracy than non-DL methods. The segmentation process is used for bi-ventricular segmentation with FCNs for simultaneous segmentation of RV and LV. These methods achieve superior accuracy than the traditional models in confronting segmentation regions such as ventricular myocardium. This projects the superiority of DL models.

Next, atrial fibrillation (AF) is measured as a common cardiac electrical disorder that affects more than 1 million people. Based on this clinical practice segmentation process, the atrial anatomy process is used in both AF pre-operative planning and postoperative evaluation. The atrium segmentation is used for scar segmentation. Conventional methods like region growing, non-rigid registration, and atlas-based label fusion are applied for LA segmentation. As well, the accuracy of this method is superior with better initialization and pre-processing techniques. Similarly, scar segmentation is done with contrast-enhanced MRI and late gadolinium enhancement imaging. This facilitates segmentation over myocardial scar identification and fibrosis. In DL, scar segmentation is done with clustering techniques and threshold methods that are used for local intensity changes. The foremost limitations associated with these techniques are the need for manual segmentation in the ROI to diminish computational costs and search space. As an outcome, semi-automated techniques are not appropriate for clinical deployment. Some DL techniques are merged with traditional approaches for scar segmentation purposes. Dou et al.^[15] used an atlas model to recognise the LA and used a deep NN (DNN) to identify fibrotic tissues over those regions. Multi-view CNN is used to fuse features for superior segmentation accuracy. This model provides a 0.90 mean Dice score for LA and a 0.78 mean Dice score for atrial scars. The aortic lumen segmentation from MR images is needed for appropriate hemodynamic and mechanical aorta characterization. The important challenge associated with this is annotation sparsity in image sequences, where some frames are not annotated. This limitation is an open-ended research problem that attracts researchers’ attention.

Segmentation with CT images

CT imaging is a non-invasive method utilized for disease prediction and treatment planning, particularly in evaluating cardiac anatomy and coronary arteries. It includes two modalities: contrast-enhanced CT angiography, which provides superior visualization of cardiac chambers and vessels after contrast agent injection, and non-contrast CT imaging, which uses tissue density to generate images based on varying attenuation values of materials such as calcium and fat. This review also discusses prevalent DL techniques used for cardiac CT segmentation. The emphasis on diagnosis and treatment planning is crucial for managing critical care patients. As shown in Figure 3, CT image segmentation can be performed using deep learning approach.^[16]

Close

Figure 3:

Segmentation with computed tomography images. RA: Right atrium; RV: Right ventricle, LA: Left atrium; RV: Left ventricle.

Export to PPT

DL techniques for segmentation involve extracting ROI and classifying them using CNNs. Various methods, including 3D FCN and bounding boxes for LV detection, enhance heart segmentation accuracy, particularly in CT images compared to MR images due to differences in intensity distributions and image quality. A volumetric representation of the heart is derived from training multiple CNN models on different views. In addition, advancements in segmentation refinement utilize shape context and adaptive fusion strategies, while hybrid loss functions are employed to address class imbalance and improve performance.

The author in Simonyan and Zisserman^[17] suggested an automatic patient identification approach by segmenting and analyzing the LV myocardium using a multi-scale FCN for segmentation and a convolutional autoencoder for characterization. Analyzing coronary arteries is crucial for diagnosing CVD and planning surgeries, with the segmentation process involving lumen and centerline extraction, which faces challenges due to motion artefacts in CT imaging. DL models, particularly CNNs, are applied in both pre- and post-processing stages, with 3D dilated CNNs trained on posterior probability distributions to enhance extraction. A multi-task segmentation strategy across different modalities improves performance, while a multi-scale supervision method integrated into the U-net architecture demonstrates better voxel-level predictions compared to baseline models, producing smooth surfaces without the need for post-processing.^[18]

Many works focus on directly segmenting chest CT and non-contrast cardiac CT using a combination of Dense Net and U-net for improved accuracy. A two-stage pipelining approach is utilized for localizing the target ROI through rigid transformation, enhancing segmentation robustness with a DL framework enhanced by LV segmentation in echocardiograms. The method features a deep belief network (DBN) for identifying rigid transformations and localization, highlighting that DBN-based feature extraction is more resilient to image appearance variations, which reduces the complexity of inference and training with sparse learning in the detection process.

Some works focus on segmentation without restoring two-stage processes to decrease computational complexity, using sparse manifold learning to maintain segmentation accuracy while reducing training and search complexity. The FCN author applies coarse segmentation to refine level set-based methods. Cardiac ultrasound images record temporal sequences, with various methods enhancing coherence among frames for improved segmentation robustness. A common approach is dynamic modeling with particle filtering, which yields better outcomes by disregarding temporal information. The integration of long-short-term memory and U-Net considers multiple frames, enhancing segmentation accuracy and robustness against image quality variations compared to using a single U-Net frame. Moreover, machine learning models efficiently produce both unlabeled and labeled images by expanding training data, employing a deep network for initializing a small set of labeled data, and utilizing U-Net with Kalman filtering for pre-training annotations to validate external classifiers.

Hu et al.^[19] proposed a model to train a CNN for partial labelling of multiple sequences and fine-tuned the network for manual segmentation. Semi-supervised framework facilitates training on both unlabeled and labeled images. This uses a generative network to produce an ultrasound image for segmentation of unlabeled frames and to identify segmentation to resourcefully generate input for ultrasound images. Unlabeled data may manually improve annotated data from multiple domains and assist in improving the segmentation of certain fields.

Segmentation with ultrasound images

Ultrasound cardiac imaging, known as echocardiography, is a key tool for assessing cardiovascular function due to its low cost and portability. Automation of anatomical structure segmentation employs methods like level sets, active contours, and active shapes, though accuracy suffers from issues like speckle noise and image contrast. To enhance image quality for tissue classification, DL techniques combined with deformable models improve 2D segmentation. Features extracted through DNN advance robustness, while one-shot segmentation reduces computational complexity in two-stage approaches.^[20] Temporal ultrasound data recording aids coherence, enhancing segmentation accuracy. CNN effectively manages segmentation without post-processing complications, leveraging U-Net for annotation in 2D ultrasound images, which improves accuracy through extensive training data and real-time processing capabilities. Focusing on clinical situations involves the application of critical care monitoring and decision-making tailored to individual patients.

3D cardiac segmentation presents greater challenges than 2D segmentation due to issues like temporal resolution and image quality. The increased dimensionality of 3D images complicates DL processing. A two-stage approach is employed, utilizing a 2D CNN for initial coarse segmentation, which aids in refining the 3D model. Transfer learning is implemented to optimize training by utilizing a segmented dataset while addressing challenges related to device variation, patient conditions, and protocols. Figure 4 presents ultrasound image-based segmentation.^[21]

Close

Figure 4:

Ultrasound image–based cardiac segmentation. Representative transthoracic echocardiographic frames from standard views illustrating variability in anatomy and image quality.

Export to PPT

To encourage and to resemble real-time clinical situations, the evaluation and development of clinical segmentation must be done in a large set of publicly available datasets. The benchmarking dataset shows that an auto-encoder-based CNN may reduce the inter-observer and lower error conditions in cardiologists. A sparsely connected NN with a lesser number of parameters is used to identify a bounding box to locate the target structure, where learning with a bounding box reduces computational complexity. To enhance clarity and reproducibility, detailing dataset acquisition, preprocessing steps (normalization, resizing, augmentation, slice selection), network architecture, training parameters (learning rate, batch size, epochs, optimizer, loss function), and performance evaluation metrics.

DL network models

DL network (DLN) models are deep artificial neural networks (D-ANN). Every NN comprises of input layer, hidden layers and output layers [Figure 5].^[3] This section presents a detailed analysis of DLN models and the techniques associated with this for segmentation algorithms has been analyzed. DL networks, known as DLN, consist of a structure called a D-ANN, which comprises three main components: an input layer, hidden layers, and an output layer.

Close

Figure 5:

Generic view of convolutional neural network.

Export to PPT

The methodology employed for dataset acquisition and preprocessing is detailed, encompassing normalization, resizing, augmentation, and slice selection. It describes the network architectures utilized alongside the training parameters, including learning rate, batch size, epoch count, optimizer, and loss function. Consistency in pre-processing was maintained across all datasets. The network architectures referenced are consistent with their original publications, and all training parameters are provided to enable replication of results. Publicly available datasets were prioritized to enhance replicability, and standardized evaluation metrics in medical imaging segmentation were used for performance comparison with previous studies.

Two-dimensional (2D) CNN

The concept of segmentation in image processing, particularly using 2D images and filters within a CNN framework. It highlights the advantages of using multi-modality images, specifically through RGB channels, over single modality inputs for improved segmentation results. Techniques such as transfer learning are employed for training low-level and high-level features. In addition, 2D CNNs efficiently extract spatial information with reduced computational costs, and multi-task segmentation assesses network designs for various organ segmentation. The training model benefits from 2D labelled data, which is generally more accessible than 3D data, while also noting that volumetric image decomposition aims to alleviate dimensionality issues. Challenges associated with 2D convolution and lower resolution depth are also mentioned. Figure 6 shows the segmentation using 2D CNN architecture.^[12]

Close

Figure 6:

2D convolutional neural network.

Export to PPT

Three-dimensional (3D) CNN

3D techniques are utilized to extract volumetric data across the X, Y, and Z axes, focusing on the labels of central voxels. The 2D model emphasizes higher spatial information while the 3D model focuses on volumetric data across the X, Y, and Z axes. A 3D model predicts central voxel labels using 3D patches and has an enhanced structure compared to 2D models. 3D medical imaging benefits from improved spatial information and utilizes a multi-scale 3D CNN to enhance resolution through a receptive field. This model employs 3 × 3 kernels for better accuracy and addresses dimensionality issues with shared spatial weights, reducing parameters and processing time. Despite its effectiveness in segmenting volumetric images and organs, the training of 3D models poses challenges.^[23] The use of 3D pooling stabilizes features across 3D space, promoting quicker convergence rates compared to traditional 3D CNNs. While the model excels at segmenting large organs, it struggles with brain boundary detection and faces drawbacks when juxtaposed with 2D models.

U-Net

U-Net utilizes the concept of deconvolution and employs skip connections to address localization issues. The U-Net is a significant structural model for medical image segmentation, employing deconvolution and consisting of 19 layers with skip connections to address localization issues [Figure 8].^[13] It balances patch size and pooling layers to enhance accuracy, utilizing connections between resolution layers to improve feature extraction in deconvolution layers. The model demonstrates efficient and quick image segmentation, with advancements in segmentation accuracy over the traditional U-Net. The 3D U-Net variant further enhances spatial information, enabling volumetric segmentation from 2D slices and utilizing a softmax loss function for improved sample annotation and densification.

Close

Figure 8:

U-Net model.

Export to PPT

V-Net model

V-Net employs feature extraction and reduces resolution, facilitating probabilistic segmentation through softmaxThis text describes a model resembling the U-Net, known as V-Net, which focuses on feature extraction and resolution reduction through convolutional pathways [Figure 9].^[15] It functions similarly to pooling, reducing memory requirements while lacking backpropagation input. The process concatenates lower resolution feature maps to enable probabilistic segmentation and employs softmax for distinguishing between foreground and background. The V-Net model potentially offers a higher dice coefficient and presents advantages over traditional methods, specifically the original V-Net model.

Close

Figure 9:

V-Net model.

Export to PPT

Recurrent NNs (RNN)

RNN models utilize recurrent connections for pattern memorization, but long short-term memory (LSTM) networks can lose spatial information. In contrast, gated recurrent unit (GRU) models simplify this by removing memory cells. This RNN model facilitates pattern memorization with recurrent connections, using multiple adjacent slices to extract contextual information from sequential data [Figure 10].^[12] LSTM, a subset of RNN, is employed for image segmentation, though it risks losing spatial information. The contextual LSTM enhances the U-Net structure by capturing slice context for improved segmentation^[24] GRUs simplify performance but result in memory cell removal. The clockwork RNN optimises modeling with fewer parameters and better long-term dependency. While RNNs allow for parallel processing, they can be challenging and time-consuming; their performance is notably superior in segmenting organs, such as the liver, compared to inter-slice information.^[24,25]

Close

Figure 10:

Recurrent neural network.

Export to PPT

Challenges

Design challenges in DL models include overfitting, primarily caused by small dataset sizes. Solutions involve training with larger datasets, creating multiple patch views, using drop-outs, and applying pooling layers to reduce parameter dimensionality. Batch normalization enhances convergence, but downsampling leads to information loss. Deeper networks may face exploding signal issues addressed with supervised models. In medical image segmentation, challenges include the appearance of target organs and contrast boundaries, especially with 3D models, which complicate training and increase computational demands. Future analysis is required to identify effective solutions to these challenges for critical care cardiac imaging.