TY - JOUR
T1 - SHS-GAN
T2 - Synthetic Enhancement of a Natural Hyperspectral Database
AU - Hauser, Jonathan
AU - Shtendel, Gal
AU - Zeligman, Amit
AU - Averbuch, Amir
AU - Nathan, Menachem
N1 - Publisher Copyright:
IEEE
PY - 2021/5/12
Y1 - 2021/5/12
N2 - Deep Learning frameworks are gaining increased popularity in image processing tasks such as computational hyperspectral imaging. While these frameworks achieve state-of-the-art results in terms of reconstruction quality and run time, they often require massive databases of hyperspectral cubes for training the reconstruction algorithms. Unfortunately, such databases are usually hard to acquire due to complexity and cost considerations. To mitigate these challenges, we propose a method for generating a synthetic database of hyperspectral cubes in the visible range using a limited number of natural hyperspectral cubes, an unlimited number of RGB images, and a Generative Adversarial Network model. The suggested algorithm, dubbed SHS-GAN, is trained to get a query RGB image and to output a synthetic hyperspectral cube. While the spectral domain of the synthetic hyperspectral cube shares similar statistical properties as the natural hyperspectral cubes used in the training process, the SHS-GAN is trained to preserve the spatial characteristics of the query RGB image, whereas the R, G, B values provide an additional constraint along with the spectral domain. Our suggested framework was utilized for performing Snapshot Spectral Imaging (SSI) from a single monochromatic dispersed and diffused snapshot using the DD-Net reconstruction neural network. We demonstrate, by simulations and lab experiments, that enhancing the training database with synthetic data from the SHS-GAN improves the reconstruction quality of the hyperspectral cube. In addition, we share a new original database of more than 10,000 hyperspectral cubes of real objects of size 256x256x29 in the 420-700 nm visible range.
AB - Deep Learning frameworks are gaining increased popularity in image processing tasks such as computational hyperspectral imaging. While these frameworks achieve state-of-the-art results in terms of reconstruction quality and run time, they often require massive databases of hyperspectral cubes for training the reconstruction algorithms. Unfortunately, such databases are usually hard to acquire due to complexity and cost considerations. To mitigate these challenges, we propose a method for generating a synthetic database of hyperspectral cubes in the visible range using a limited number of natural hyperspectral cubes, an unlimited number of RGB images, and a Generative Adversarial Network model. The suggested algorithm, dubbed SHS-GAN, is trained to get a query RGB image and to output a synthetic hyperspectral cube. While the spectral domain of the synthetic hyperspectral cube shares similar statistical properties as the natural hyperspectral cubes used in the training process, the SHS-GAN is trained to preserve the spatial characteristics of the query RGB image, whereas the R, G, B values provide an additional constraint along with the spectral domain. Our suggested framework was utilized for performing Snapshot Spectral Imaging (SSI) from a single monochromatic dispersed and diffused snapshot using the DD-Net reconstruction neural network. We demonstrate, by simulations and lab experiments, that enhancing the training database with synthetic data from the SHS-GAN improves the reconstruction quality of the hyperspectral cube. In addition, we share a new original database of more than 10,000 hyperspectral cubes of real objects of size 256x256x29 in the 420-700 nm visible range.
KW - Computational hyperspectral imaging
KW - deep learning
KW - image processing
KW - neural networks
UR - http://www.scopus.com/inward/record.url?scp=85105869174&partnerID=8YFLogxK
U2 - 10.1109/TCI.2021.3079818
DO - 10.1109/TCI.2021.3079818
M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???
AN - SCOPUS:85105869174
SN - 2573-0436
VL - 7
SP - 505
EP - 517
JO - IEEE Transactions on Computational Imaging
JF - IEEE Transactions on Computational Imaging
M1 - 9429891
ER -