Abstract: In this paper, we advocate for Generalized Assorted Camera (GAC) arrays
for multi-modal imaging—i.e., a camera array with filters of different characteristics
placed in front of each camera aperture. GAC provides us with three distinct advantages
over GAP: ease of implementation, flexible application dependent imaging since filters
are external and can be changed and depth information that can be used for enabling novel
applications (e.g. post-capture refocusing). The primary challenge in GAC arrays is that
since the different modalities are obtained from different viewpoints, there is a need
for accurate and efficient cross-channel registration. Traditional approaches such as
SSD, SAD, and mutual information all result in multi-modal registration errors. Here,
we propose a robust cross-channel matching cost function, based on aligning normalized
gradients, that allows us to compute cross-channel sub-pixel correspondences for scenes
exhibiting non-trivial geometry. We highlight the promise of GAC arrays with our cross-channel
normalized gradient cost for several applications such as low light imaging, post-capture
refocusing, skin perfusion imaging using RGB+NIR and hyperspectral imaging.
