Aflatoxin is a kind of virulent and strong carcinogenic substance, and it is found widely in peanuts, maize, and other agricultural products. In order to detect aflatoxin in peanut, we proposed a spectral–spatial (SS) combinative deep learning (DL) method based on hyperspectral images. First, we construct a push-broom hyperspectral imaging system using a grating spectrometer module (GSM), a SCMOS CCD, and an electric displacement platform. Using this system, we collect the hyperspectral images of 73 peanut samples before and after contamination by aflatoxin under 365 nm ultraviolet (UV) illumination. Then, based on Convolutional Neural Network (CNN), we propose the SS combinative DL method. In this method, we select principal components analysis (PCA) images or key band images as the spatial data and calculate the mean of each spectral band (including the derivative of consecutive spectral bands) as the spectral data. It is found that, because the combination of spectral and spatial data takes into account both spectral and spatial information, it obviously improves the performance of recognition rate. The total fivefold cross-validation recognition rate is about 95%, which is higher than k-nearest neighbor (KNN), support vector machine (SVM), and back propagation artificial neural network (BP-ANN). The combination of PCA image and spectral image is better than the combination of key band image and spectral image. Our results suggest that the method we proposed is a rapid, precise, and nondestructive technique for the detection of toxic metabolites in peanuts, and it could be an alternative to manual techniques.