Europhysics News (2000) Vol. 31 No. 4

Medical image registration

Karin Kneöaurek¹, Marija Ivanovic², Josef Machac¹, David A. Weber^2
1 Division of Nuclear Medicine, The Mount Sinai Medical Center, New York, USA
² Department of Radiology, University of California at Davis, California, USA

Functional imaging using single photon emission computed tomography (SPECT) and positron emission tomography (PET) is extremely valuable in the diagnosis of various disorders. Uncertainty in the anatomic definition on SPECT and PET images, however, sometimes limits their usefulness. To overcome this problem, a combination of magnetic resonance images (MRI) and X-ray computed tomography (CT) images with functional SPECT or PET images of the same sections of the body, is used. This provides complementary anatomic (MRI or CT) and physiological (SPECT or PET) information that is of great importance to research, diagnosis, and treatment.

Two basic types of medical images are made: functional body images (such as SPECT or PET scans), which provide physiological information, and structural images (such as CT or MRI), which provide an anatomic map of the body. Different medical imaging techniques may provide scans with complementary and occasionally conflicting information. The combination of images can often lead to additional clinical information not apparent in the separate images. The goal of image fusion is to impose a structural anatomic framework on functional images. Often in a functional image, there simply isn't enough anatomic detail to determine the position of a tumor or other lesion.

   Although, the construction of a composite, overlapping medical image - described in the field as medical image registration has been primarily used in the fusion of functional and anatomical images, it has also been applied to a series of the same modality images. Examples of this are registration of SPECT images of the same subject in follow-up studies or in a comparison of an image with normal uptake properties to an image with suspected abnormalities. In addition, image registration of SPECT and PET images and the registration of SPECT and PET images with anatomic atlases, provide an important means to evaluate comparative uptake properties of SPECT and PET radiopharmaceuticals, and to correlate uptake properties with anatomy.

   Medical image registration has been applied to the diagnosis of breast cancer, colon cancer, cardiac studies, wrist and other injuries, inflammatory diseases and different neurological disorders including brain tumors, Alzheimer's disease and schizophrenia. This method has also been utilized in radiotherapy, mostly for brain tumors, and by cranio-facial surgeons to prepare for and simulate complex surgical procedures.

   One area where image registration plays an important role is in medical imaging in the early detection of cancers. Radiologists often have difficulty locating and accurately identifying cancer tissue, even with the aid of structural information such as CT and MRI because of the low contrast between the cancer and the surrounding tissues in CT and MRI images. Using SPECT and radioactively labeled monoclonal antibodies it is possible to obtain high contrast images of the concentration of antibodies in tumors. However, sometimes it is difficult to determine the precise location of the high concentration of the radioactive isotope in SPECT or PET images in relation to anatomic structures, such as vital organs and surrounding healthy tissue. Image registration is a visualization tool that can significantly aid in the early detection of tumors and other diseases, and aid in improving the accuracy of diagnosis.

   The problems related to multimodality imaging can be separated into three groups; a) problems related to file structures, transfer and networking, b) registration, and c) visualization of the composite images.

Fig 1 MRI and SPECT head sagittal slices of the same patient and the coregistrated (MRI + SPECT) image. The lesion on the top of the skull is more prominent in the composite image, although it can be visualized in both modalities.

  
Problems related to file structures, transfer and networking
The main problem in networking of diagnostic images from different modalities is the need to merge unknown or unlike image file structures. To illustrate the problem, suppose you wanted to find out whether a man has grown a bigger mustache by inspecting two pictures taken at different times. One is a photograph, and the other on a video camera. And you want to do it precisely. Imagine the problems in composing the pictures in the two media, the size of the pictures, their quality, different viewing angles, etc. The problems with medical images are much more severe. Different vendors use different ways of writing patient data on the hard disk. Usually patient data are split into two parts; header and binary image data. The two parts can be one file or two separate files. The header contains general information such as patient name, age, matrix size used in imaging, number of images acquired, depth of each image, i.e. 8 or 16 bit deep images, diagnostic comments and the date of examination. The binary image or images are usually one file. However, on some computers the way of writing the data can be different, e.g., the order of high and low bytes can be reversed. The worse case is facing an unknown format of patient data, e.g., the compressed data. Handling the different patient file structures may be the most difficult issue in applying medical imaging registration but it obviously is not the only one. There is now a big effort in the medical imaging community to define certain standards for patient imaging files. There are several proposals, although still in the development stage, such as the DICOM standard being developed in the USA and the Interfile standard being developed in Europe. But in spite of the general approach and standards, individual manufacturers still maintain their own patient file formats. Therefore, there remains a need for software to transfer information from one format to another. We hope that medical equipment companies will provide basic information on their patient formats and allow direct access to the raw and uncompressed patient images. Networking patient images and their physical transfer to the common processing system is also an obstacle facing many clinical institutions. The main problem is the transfer of huge amounts of data at a reasonable speed using affordable hardware and software. There is a concern about the transferring of confidential patient data using the main hospital Ethernet backbone, which is accessible to the general public. The optimal solution is to run a local network which is connected to the outside world by a bridge or router, providing on-line transfer of patient data. The router or bridge should also serve as a gatekeeper, and protect unauthorized access to sensitive patient data. As a backup, it is advisable to use tape cartridges or other common media (optical, floppy disk, etc.).

Registration
The first step in correlating images of the different modalities is to address differences in the acquisition parameters between different modalities. Those parameters are different pixel† or voxel†† size, different matrix size and different orientation in acquired images. For example, one of the first steps in registration SPECT with MRI or CT images is to expand the 64x64 SPECT image to the 256x256 or even to 512x512 matrix, the usual size of the MRI or CT images. In order to preserve the quality of images enlarged to this extent, certain interpolation has to be used. Mostly bilinear and B-spline interpolations have been used in medical image registration

   The process of registration, which is also referred to as image fusion, superimposition, matching or merge, is based on the definition of the transformation that transforms an image of one modality to the image of the another modality. Image registration should map each point in one image onto the corresponding point in the second image.

   Mostly it is assumed that transformation between the images of the different modalities is isotropic. It is assumed that neither image is skewed and that pin cushion or barrel distortion for each modality is negligible. Thus, linear transformations can been used, such as rotation, scaling, reflection and translation. In addition to the linear transformation, affine transformations that include uniform and nonuniform scaling and shearing in addition to linear transformations have been used. When there is variability in geometric structures between images, e.g. distortion in the MRI image because of the variable magnetic field, curved transformations, referred as warping, may be applied.

   There are four general approaches to the registration problem; control-point based registration, moment- based registration, edge-based registration, and optimization of a similarity measurement.

Fig 2 Coregistrated SPECT-MRI image. The SPECT image was pasted in "opaque" mode on the top of the black-and-white MRI image, which provided an anatomical template.

  
Control-point based registration methods
efining control-points consists of choosing some characteristic points in images. Usually features and/or geometric properties that are unique to a set of images define these points or landmarks. Control-points must lead to an accurate description of the image transformation. These landmarks can be used as extrinsic, intrinsic or combination of both. They can be internal anatomic landmarks (rib cage, ventricles, bone surfaces, etc.) or external point sources attached to the patient, or external fiducial markers in stereotactic studies.

   The registration step consists of defining the transformation, which maps the control-points of the first image onto the ones in the second image (an MSE, mean square error). Preprocessing performed using control- points is not always an easy operation. Using internal anatomic landmarks requires considerable operator expertise. However, user provided control-points usually lead to satisfactory and fast registration. Also, a priori information from the user's knowledge is straightforwardly introduced in the process.

   The process of registration can be accomplished manually, semiautomatically or automatically. The most common approach is semiautomatical, although the degree of interaction can vary considerably among different applications.

Moment-based registration
The basic idea here is to extract common information from images without or with minimal participation of the user.

   Gray-level and geometric properties of images are characterized by the center of gravity, principal axis and more complex moments. Parameters (translation, rotation, scaling, etc.) of the transform leading to "standard" images are computed by normalizing moments in each image. The main property of the normalized moments is that they are invariant features of images.

   For moment-based registration techniques noise tolerance is rather weak since noise can lead to imperfect moment estimates and large errors in parameter determination. Also, moments do not involve enough knowledge about a priori information that is present only through the registration model. These techniques usually do not permit robust registration and field of use is restricted to the matching of simple objects in image pairs.

Edge-based registration methods
This class of methods has usually been performed to register images where neat contours exist. The preprocessing stage consists of discarding all information except edges in each image.

   The edge extraction step can be solved in many different ways: template matching, zero-crossing, etc. Nevertheless, contours are not easily extracted from noisy images. Resulting contours must be characterized properly for the associated matching algorithm.

   Minimizing MSE and comparing intensity values of edge pixels is frequently used to match edges. Available information about edges is incompletely used since only the appearance of edges is involved in this method and the optimization criterion often presents a very sharp minimum. Because of noise in contour images, this minimum can be missed and a local minimum selected instead, leading to a totally meaningless registration. In order to overcome this failure, and to ensure high reliability, contour-based methods must involve systematic search, symbolic representation or global optimization techniques. In the three-dimensional case (3D), edge detection techniques match the images by minimizing the MSE distance between the surfaces of an anatomical structure visible on both modalities (e.g., head and hat method).

   Potential use in medical imaging is significant, because in most instances edge information is the only common feature found in each image.

Fig 3 Coregistrated 3D image of the brain and basal ganglia. The brain image was obtained from the MRI study, while basal ganglia image was created as a result of the PET study of the metabolic rate using F-18 FDG (Courtesy of PET Laboratory at Mount Sinai Medical Center, New York).

  
Optimization of a similarity measurement as a registration method
The goal here is not the extraction of features but rather transforming one of the images and measuring the similarity between images until, the best fit is achieved.

   Registration is usually performed by optimizing a similarity criterion, such as a correlation coefficient, a correlation function, or a sum of absolute differences. However, in some situations using these similarity measurements for the registration of images may lead to misregistration. The reason for this is that criterion values may not take into account variations in the amount of contrast medium during angiography or the presence of a tumor in only one image. This class of method usually works well with the images from the same modality, but it is not as useful for registering the images from different modalities because the pixel intensity values are usually not related in different modalities.

   The above methods have been implemented on X-rays, radionuclide (SPECT and PET) scans, CT scans, ultrasound, and MRI images. Registration parameters used have usually been translation, rotation, gray-scale and spatial scaling; in some cases, it is necessary to include more parameters such as shear and warping. A priori information, usually as landmarks, has been used in almost every instance to extract information relevant to registration. Only in simple cases, for which images are very similar, has the preprocessing been of little importance. The choice of the applied method usually depends on the existence of features in medical images, such as neat contours.

   In clinical practice the methods are usually combined. In Fig. 1 are shown the sagittal MRI, SPECT, and the combined head sagittal slice of the same patient. The lesion on the top of the scull is visible in both modalities, but is much more enhanced on the composite image. The MRI and SPECT images were centered and oriented by using center of gravity and principal axis. This was followed by extracting the edges that are used to size the images. The final stage was to use the control-points in fine-tuning in the matching process. The point sources were made by filling capillary tubes with a drop of concentrated solution of Tc-99m and copper sulfide. The markers were about 1x1x1 mm. These point sources were on the patient's head. The activity was strong enough to be visible in the SPECT image, but weak enough not to contaminate the SPECT brain image. For MRI imaging, copper sulfide is a contrast medium. The point sources were visible in both modalities, and matching the control points was performed manually.

   Most errors, which propagate through the matching process, are due to sample size, imperfection of the acquisition system, noise and interpolations used. Accuracy of the registration process tested in the phantom studies has shown that is better, or at worse, equal to 2 mm.

Visualization
After matching, a problem of visualization of the composite image occurs. It is not clear how to create the composite image, i.e. how to combine images of the different modalities in the process of creation of a single composite image. More precisely, the question is what should be the value in each image location, i.e. in each pixel in the composite image. The main problem in creating a composite image is how to retain information from both individual images in addition to giving correlation information of the functional (SPECT and PET) and anatomical (MRI and CT) features. The way of creating the composite image depends on the clinical demands. In certain situations retaining anatomical structures in the MRI image may be essential, while SPECT information may be barely visible. The opposite may be desirable in some other situations in which functional (SPECT) information can be pasted on barely visible anatomic background provided by MRI or CT study. Usually, MRI (or CT) image is scaled to one color range and a SPECT image to another. A very attractive approach is to keep the MRI (or CT) image black-and-white as an anatomic template, while making the SPECT image transparent and colored (Fig. 2). The new techniques of 3D presentations using different shadings and transparencies can also be utilized in the visualization (Fig. 3).

   Future work involves producing better markers that would be visible in both modalities, i.e. SPECT- MRI and SPECT-CT studies. The next problem is placing the markers on the patients. They can be attached directly to the patients skin, as we did, or face masks and holders can be designed and used. Such masks and holders are usually part of the patient immobilization system, which allows reproducible positioning. There are several companies making facial masks and plastic holders, but some additional engineering may be necessary for modification for a particular application. These masks and holders should also be transparent to the image modalities used, which is obviously not an easy task to achieve.

   In addition to the use of different scales of chromacity and intensity, as well as hue and saturation in the creation of a composite image, some other approaches seem to be worth trying. One promising method consists in creating a composite 512x512 image from two 256x256 images. In the composite image each second pixel can contain information from the MRI image and the rest can be filled with SPECT image values. It will be interesting to investigate other combinations in creating combined images from the value of each pixel in individual modality images.

   In conclusion, image registration has been used in many clinical situations; radiotherapy, tumor, stroke, blood flow and other diagnostic procedures, as well as in plastic and cranio-planar surgery. Mostly, image registration was applied in merging the functional SPECT or PET data with anatomical CT and MRI images, providing additional useful clinical information. With further development of the computer technology and physical methods for registration and visualization, medical image fusion will definitely find an even wider clinical application.

Further reading
[1] D.A. Weber, M. Ivanovic, Semin. Nucl. Med. 4,311 (1994)

[2] C.A. Pelizzari, G.T.Y. Chen, D.R. Spelbring, J. Comput. Assist. Tomogr. 13, 20 (1989)

[3] P. Gerlot, Y. Bizais, In: Information processing in medical imaging, Eds. K. DeGraaf, M.A. Viergever, p 81 (Plenum, New York,1988)

  
For correspondence and reprint requests:
Dr. Karin Kneöaurek
Division of Nuclear Medicine
Box 1141, One Gustave L. Levy Place
New York, NY 10029, USA
Tel. US-(212) 241-9368
Fax. US-(212) 831-2851
E-mail:kknesaurek@smtplink.mssm.edu