Spectral imaging technology aims to capture spectral information for each two-dimensional spatial point to form a spectral data cube. It has been applied in a broad range of fields such as remote sensing^1,2, precision agriculture³, medical diagnostics^4,5, food inspection⁶, environmental monitoring⁷, art conservation^8,9 and astronomy¹⁰. Traditional spectral imagers rely on either spatial scanning, such as whiskbroom scanning¹¹ and pushbroom scanning¹², or spectral scanning, such as filter wheels¹³ and tunable filters^14,15. However, scanning methods suffer from low acquisition speed, which is not applicable for dynamic recording of moving targets. To overcome this limitation, snapshot spectral imaging methods¹⁶ are explored. Early snapshot techniques, such as integral field spectrometry^17-19, multispectral beam splitting²⁰, and image-replicating imaging spectrometer²¹, still rely on light splitting, also the optical systems of which are bulky. With the development of compressive sensing (CS)^22,23, growing research interests have been attracted by the computational snapshot spectral imaging technique²⁴, which can be categorized into three groups: coded aperture, speckle-based and spectral filter array methods. For coded aperture methods, the classical system is coded aperture snapshot spectral imager (CASSI)^25-32, which uses fixed masks and dispersive elements to implement band-wise modulation. CASSI is capable of capturing and reconstructing the hyperspectral images rapidly with deep-learning techniques. However, the complicated optical components lead to large system volume, which cannot meet the growing demand for portable applications. Speckle-based methods^33-38 utilize the wavelength dependence of speckle from a scattering media or diffractive optical element to achieve spectral imaging. Although the systems can be compact, the spectral resolution is limited by the speckle correlation through wavelengths. The spectral filter array methods can be viewed as an extension of Bayer filters, which adopt a super-pixel containing a group of spectral filters for spectral recovery. Even though the methods of this class are endowed with the advantages of compact device size and high spectral accuracy, there exist mosaic effect in the reconstructed spectral images, where the recovered spectra for the edge points are inaccurate. Recently, our group demonstrated a snapshot spectral imaging chip based on metasurface-based spectral filter arrays with the ultra-high center-wavelength accuracy of 0.04 nm and the spectral resolution of 0.8 nm³⁹. Furthermore, the spectral resolution was improved to 0.5 nm when adopting metasurfaces with freeform shaped meta-atoms in our latest work⁴⁰. However, in addition to the Mosaic effect mentioned above, the classical iterative CS algorithm adopted in the current work makes the computation time of data cube reconstruction still remain very long, which limits its application in the mobile systems with speed requirements, such as pilotless automobile.

In the current work, on-chip mosaic-free rapid spectral imaging was demonstrated by employing advanced deep-learning-empowered algorithms developed for CASSI to the metasurface spectral imager reported in our previous work⁴⁰. The metasurface spectral imager produces different amplitude modulation patterns in different spectral bands, which plays the role of a fixed mask plus a disperser in CASSI. Specifically, the spectral imager was designed by integrating a metasurface layer composed of 360 × 440 metasurface units with freeform shaped meta-atoms onto a CMOS image sensor (CIS). There are totally 400 kinds of metasurface units, each of which is of a distinctive spectral response function. As a proof of principle, 256 × 256 metasurface units and 26 wavelength bands were selected from 450 to 700 nm with a step of 10 nm. A deep unfolding network based on the alternating direction method of multipliers (ADMM) algorithm, which is called ADMM-net⁴¹, was adopted for the fast reconstruction of spectral images. Here, the network was trained on a synthetic dataset containing 750,000 spectral data cubes with the size of 256 × 256 × 26, which are generated from the CAVE⁴¹ and KAIST⁴² datasets. Besides, additive white Gaussian noise was imposed on the measurements to mimic real-test cases. On-chip rapid spectral imaging eliminating the mosaic effect was realized by applying the ADMM-net to reconstruct the spectral data cube directly. Compared with the point-by-point iterative spectral reconstruction, four orders of magnitude speed improvement was achieved by the ADMM-net, which enables a spectral data cube reconstruction rate of 55 per second, and the average spectral reconstruction fidelity exceeds 99% for a 24-patch Macbeth color checker. In practice, video-rate spectral image reconstruction was demonstrated for moving objects and outdoor driving scenes. It is found that the concolorous sky and moving white cars can be effectively distinguished by their spectra, while the existing driverless vehicles can easily mistake a white truck for the sky and cause a crash. The approach adopted in the current work is capable of solving the huge safety problem caused by the defect in metamerism recognition for not only autonomous driving⁴³ but also other fields of intelligent perception, and shows great potential for various applications.

The schematic of the metasurface-based spectral imager is shown in Fig. 1a, as reported in ref.⁴⁰. A silicon metasurface layer was integrated on the image sensor with a microlens layer. The metasurface layer was composed of 360 × 440 metasurface units, which were obtained by arranging 20 × 20 kinds of metasurface units repeatedly 18 × 22 times. Different metasurface units exhibit distinctive transmission spectra. Each metasurface unit is a periodic array with freeform shaped meta-atoms. The periods and shapes of the corresponding 400 kinds of meta-atoms were optimized to maximize the mutual differences of the transmission spectra (see ref.⁴⁰ for details). Each metasurface unit represents a spatial point, and light will be detected after being modulated by each unit. For a certain point (i,j), the spectrum of incident light can be reconstructed from the $N={\left(2n+1\right)}^2$ detected signals at the surrounding N points (that is, $\left(i-n,j-n\right),\left(i-n,j-n+1\right),\dots ,\left(i+n,j+n\right)$, typically, we set n=2). The spectral reconstruction was implemented by solving such a system of linear equations:

Where, $y^{\left[i,j\right]}$ denotes the measured signal at the (i,j) point, $M_k^{\left[i,j\right]}$ represents the modulation intensity (i.e., transmittance of the metasurface unit) at the (i,j) point for the k-th spectral channel, $x_k^{\left[i,j\right]}$ is the k-th element of the target spectrum vector at the (i,j) point, $N_{\lambda }$ is the total number of spectral channels. Since there are a total of 360 × 440 points, 158,400 groups of equations like Eq. (1) are needed to be solved to reconstruct the spectra of all points to recover the whole spectral data cube, which is time-consuming. For the iterative CS algorithm, it is assumed that the spectra of incident light at the N points around (i,j) are the same with those in Eq. (1), which results in mosaic effect in the reconstructed spectral images. In order to address the above problems, the deep-learning algorithms⁴⁴ were proposed to be exploited for directly reconstructing the data cube inspired by CASSI (see Supplement 1, S1 for details), as indicated in Fig. 1b.

The basic principle of CASSI is presented in Fig. 1c, where the spectral data cube is spatially coded by a fixed physical mask (coded aperture), and then spectrally sheared by a dispersive element, and finally measured by a detector. Therefore, for CASSI, a fixed mask and a disperser were used to achieve different masks at different spectral channels. The fixed mask can be traditional blocking-unblocking coded aperture^25,26 or colored coded aperture27, 28, 29, 30. In the current work, the metasurface layer was adopted to achieve different masks at different wavelengths, and its mathematical model can be written as:

Where, $x_k={\left[\begin{array}{cccc}{x}_k^{\left[1,1\right]}& x_k^{\left[2,1\right]}& \cdots & x_k^{\left[N_x,N_y\right]}\end{array}\right]}^T$ denotes the k-th frame of the spectral data cube, $N_x,N_y$ represent the number of points along the horizontal and vertical dimensions, respectively. When taking the measurement noise into consideration, the Eq. (2) can be expressed in a vectorized formulation as follows:

Where, $y\in {\mathbb{R}}^{N_x{N}_y}$ is the compressed measurement, $\Phi \in {\mathbb{R}}^{N_x{N}_y\times N_x{N}_y{N}_{\lambda }}$ is the sensing matrix, $x\in {\mathbb{R}}^{N_x{N}_y{N}_{\lambda }}$ is the target signal and $e\in {\mathbb{R}}^{N_x{N}_y}$ is the measurement noise. As a proof of principle, 256 × 256 metasurface units were selected for spectral imaging with 26 wavelengths from 450 nm to 700 nm (that is, $N_x=N_y=256,N_{\lambda }=26$).

In the current work, a deep unfolding network based on the ADMM algorithm dubbed ADMM-net was employed for data cube reconstruction. The framework of ADMM-net with K stages (K = 12) is depicted in Fig. 2a. As in ref.⁴¹, let v denote an estimate of the desired signal, and by introducing two auxiliary variables x,u, the three steps for updating variables in each stage are listed as follows:

Here, ${\gamma }^{\left(k\right)}>0$ is another auxiliary trainable parameter, I is an identity matrix, D_k denotes a denoiser. Eq. (4) represents a linear projection. Since $\Phi$ is a concatenation of $N_{\lambda }$ diagonal matrices, it can be seen that $\Phi {\Phi }^T$ is a diagonal matrix. Therefore, Eq. (4) can be solved efficiently via element-wise operation instead of calculating the huge matrix inversion as derived in ref.⁴⁵. Eq. (5) is a denoising process performed by the CNN, as shown in Fig. 2a. A 15-layer U-net⁴⁶ was adopted as the denoiser, and its architecture is described in Fig. 2b, where the skip-connection can be regarded as residual learning, which is shown to be necessary for the denoiser⁴¹.

A basic dataset containing 262 scenes with a size of 512 × 512 × 26 was constructed from the publicly available hyperspectral dataset CAVE and KAIST (see Supplement 1, S2 for details). The RGB images of the selected 15 scenes in the basic dataset, which are converted from the spectral images via the International Commission on Illumination color-matching function⁴⁸, are shown in Fig. 2c. For model training, 252 scenes were randomly selected from the basic dataset, implementing data augmentation to obtain 750,000 samples with the size of 256 × 256 × 26. The operations of data augmentation include random cropping, rotation and multiplying with the spectra of LED light or sunlight sources (see Supplement 1, S3 for details). The remaining 10 scenes were downsampled to the size of 256 × 256 × 26 for testing. In addition, the loss function is the root mean square error between the ground truth and the output result of the network. The network was trained by the Adam optimizer on Pytorch using NVIDIA GeForce RTX 3080 GPUs. The total number of epochs is 300, and the batch size was set to 4 due to the limitation of GPU memory. The initial learning rate was set to 0.001, and scaled to 90% of the previous one every 20 epochs.

Firstly, network training and test were conducted under noisy conditions. Since it takes a lot of time to acquire measured images for real objects to train the network, the method of simulated measurement was adopted, that is, using the calculated measured images y through Eq. (3) with pre-calibrated Φ, spectral images x in the dataset and suitable noise distribution e. Both x and y were normalized to [0,1]. It is assumed that each element of the measurement noise vector e in Eq. (3) follows an independent zero-mean Gaussian distribution⁴⁷, that is, ei∼N(0,σn2). Here, the standard deviation ${\sigma }_n$ represents the noise level, and it is randomly chosen between 0 and 0.05 to increase the robustness to noise of different levels. The reconstruction results of two test scenes under 2% and 4% noise levels are given in Fig. 3. For the crayon scene with 2% noise, it can be seen that the average fidelity of the recovered spectra for the selected three points exceeds 99.97%, as indicated in the legends of Fig. 3a. When the noise level is increased to 4%, it can be seen that the ADMM-net can still recover the spectra reliably with an average fidelity of 99.87%. Here the fidelity is defined as follows:

where f₁, f₂ are the l₂-normalized ground truth and reconstructed result, respectively, and 〈〉 represents the inner product. The four reconstructed exemplar frames of the crayon hyperspectral data are shown in Fig. 3b, which are highly consistent with the ground truth under different noise levels. For the toy scene, the ADMM-net can also provide accurate reconstructed results, as shown in Figs. 3c and 3d. When the noise level is increased from 2% to 4%, the spectral reconstruction quality is degraded more seriously compared with the crayon scene since there are more fine spatial details in Fig. 3c, while the four frames in Fig. 3d are still recovered with high quality.

Then, the ADMM-net trained with noise was applied to reconstruct the real data from snapshot measurements captured by the spectral imager, as shown in Fig. 4a. The metasurface layer with the size of 8 mm × 6.4 mm was integrated on top of a CIS (Thorlabs, CS235MU). The spectral imager is assembled with a 50 mm-length fixed focal lens (Thorlabs, MVL50M23) for imaging. The recovered results for a standard 24-patch Macbeth color checker are displayed in Fig. 4b. In order to better show the reconstruction performance, a commercial spectral camera (Dualix Instruments, GaiaField Pro-V10) based on line scanning was employed to capture the spectral image as a reference. Moreover, the results of ADMM-net were compared with those obtained by traditional iterative algorithm GAP-TV⁴⁵, end-to-end CNN λ-net⁴⁸ and point-by-point spectral reconstruction using the iterative CS algorithm implemented by CVX⁴⁸, an open-source package for convex optimization. For CVX method, 25 metasurface units (N=25) in Eq. (1) were adopted for spectral reonstruction with 601 wavelength channels (from 450 to 750 nm with a step of 0.5 nm), and then the result was downsampled to 26 channels (from 450 to 700 nm with a step of 10 nm). The CVX method used l₁-norm regularization based on sparsity in the spectral transformation domain via dictionary learning. It can be clearly seen from Fig. 4b that higher spectral reconstruction accuracy is achieved by ADMM-net than other algorithms. For the selected four points, the average fidelities of the reconstructed spectra using ADMM-net, GAP-TV, λ-net and CVX are 99.53%, 97.18%, 97.72% and 97.32%, respectively. Besides, ADMM-net is also superior to the CVX in spatial details, which eliminates the mosaic effect. Additional reconstruction results for a Thorlabs box are provided in Supplement 1, S4.

In addition, the running time was compared using different methods so as to obtain a spectral data cube, as displayed in Table 1. For the commercial spectral camera based on line scanning, it takes about 1 minute to capture a data cube with a size of 256 × 256 × 26. For the computational methods, about 110 s are required for the GAP-TV to recover the spectral data cube of size 256 × 256 × 26, since it is an iterative method. The ADMM-net and λ-net spend It takes 1.72 and 2.44 s for the ADMM-net and λ-net on CPU (Intel Xeon Gold 6226R), or 0.018 and 0.095 s on GPU (NVIDIA GeForce RTX 3080), respectively. The CVX is the slowest, which takes 7767 and 4854 s for the spectral cube of size 256 × 256 × 601 and 256 × 256 × 26, respectively. It is clear that ADMM-net is the most efficient. The running time was divided by the number of wavelength channels, as indicated in the last row of Table 1. It can be seen that the reconstruction speed of ADMM-net is over four orders of magnitude faster than that of CVX, which enables 256 × 256 × 26 × 55 4D data cube reconstruction per second for rapid spectral imaging in real applications.

Video spectral imaging experiments were carried out to verify the capability of the proposed approach for high-speed spectral image reconstruction. The first example was a moving Thorlabs box indoors with an LED light source. The reconstructed hyperspectral video contains a total of 200 frames (19.84 s), from which 10 frames are extracted and shown in Fig. 5. From the results of recovered RGB images and spectral images at four wavelengths, it can be seen that the spatial, spectral and motion details are reconstructed with high quality. The full video is provided in the Supplement 2.

The second example is outdoor driving scene under the sunlight. The reconstructed video contains 300 frames (8.38 s) with a shorter exposure time, compared with the first example. That is, the approach employed in the current work can support a refresh rate of about 2 data cubes per meter for autonomous vehicles running at 60 km/h. Note that the refresh rate contains both the capture and reconstruction process, while the aforementioned 55 spectral data cube per second only takes the reconstruction process into account. Fig. 6 shows the recovered RGB images and spectral images at three wavelengths of the extracted 8 frames, as well as the recovered spectra for the selected three points. It can be seen that the driving cars with different colors can be reconstructed with fine spatial details, and the spectral characteristics are also recovered with high quality. In particular, from the spectra of points A and B in the frame 20 and 100, it can be observed that there exist obvious differences between the spectra of the sky and white car. Hence, the sky and white cars can be distinguished adopting the approach, which is significant for autonomous driving with the defect in metamerism recognition as a huge security concern⁴³. The full video of this example is provided in the Supplement 3.

There are still some aspects remain to be improved in the work. Firstly, the wavelength sampling interval was set to 10 nm due to the limitation of GPU memory. Reconstruction of spectral images with more wavelength bands can be anticipated with improved computational power and GPU memory. Secondly, the capacity of the training dataset is still insufficient. Spectral image datasets with large scale, accuracy and diversity are required to improve the network performance. Thirdly, the structure of the ADMM-net can be further improved in several aspects such as increasing the number of stages, and using deeper and more advanced CNN denoisers. Besides, only Gaussian white noise was considered and employed to simulate the real situations in the network training. A more complete and accurate noise model which takes both Gaussian white noise and shot noise into account is required for better performance.

In summary, a deep unfolding network called ADMM-net was proposed in the manuscript and employed to fastly reconstruct the spectral image from the snapshot measurement of the metasurface-based spectral imager. The approach employed in the current work shows excellent performance in both simulated and real data reconstruction. It is worth noting that, compared with conventional point-by-point iterative spectral reconstruction, the reconstruction speed was improved by four orders of magnitudein the real experiment with high spectral accuracy and without mosaic effect. Moreover,video-rate spectral imaging was also demonstrated for moving objects and outdoor driving scenes with good performance of metamerism recognition, in which the concolorous sky and white cars can be effectively distinguished according to their spectra. The approach adopted in the current work could provide real-time capture and reconstruction of hyperspectral images, paving the way for autonomous driving and other various real-time applications in the field of intelligent perception.

The metasurface-based spectral imager was fabricated on a Silicon-On-Insulator (SOI) wafer with a 220-nm silicon layer. Firstly, the metasurface patterns were defined via electron beam lithography (EBL) using ZEP resist, and transferred onto the top silicon layer through reactive ion etching (RIE). Subsequently, the silicon dioxide under the patterned area was removed through immersion in buffered hydrofluoric (HF) acid in a water bath at the temperature of 40 °C, and maintained for approximately 3 min. Finally, the suspended silicon metasurface layer was peeled off and transferred onto a CIS chip via a polydimethylsiloxane (PDMS) adhesion layer.

Authors’ contributions J. Yang conceived the study and composed the manuscript. K. Cui and Y. Huang supervised the project, provided much support on experimentation, and reviewed the manuscript with contributions from all other co-authors. All authors read and approved the manuscript.

Availability of data and materials The data and materials that support the findings of this study and custom codes are available from the corresponding author upon reasonable request.

Supplementary materials Supplementary material associated with this arti- cle can be found, in the online version, at doi:10.1016/j.chip.2023.100045.

Funding The National Natural Science Foundation of China (Grant No. U22A6004); The National Key Research and Development Program of China (2022YFF1501600).

Acknowledgments The authors would like to express their sincere gratitude to Beijing Seetrum Technology Co., Ltd. for their valuable discussions, and Tianjin H Chip Technology Group Corporation, Innovation Center of Advanced Optoelectronic Chip and Institute for Electronics and Information Technology in Tianjin, Tsinghua University for their fabrication support with SEM and ICP etching.

Declaration of Competing Interests The authors declare no conflicts of interest.