Aflatoxin is a kind of virulent and strong carcinogenic substance, and it is found widely in peanuts, maize, and other agricultural products. In order to detect aflatoxin in peanuts, we first built a hyperspectral imagery system using grating module, an SCOMS camera, and an electric displacement platform, and acquired 146 hyperspectral image cubes of 73 peanut samples before and after contaminated with aflatoxin. Then, we proposed a reshaped image method of pixel spectral for the convolutional neural network (CNN) method. By studying random selection data sets and comparing them with different identification models, we found that (1) reshape image established by the pixel-level spectral is good enough for aflatoxin-detected problems, and the overall recognition rate reached above 95% on pixel level. (2) Deep learning (DL) method is worked well, and it is better than traditional identification models, not only on the pixel level but also on the kernel identification. The recognition rate of above 90% on the kernel level can quickly be used in sorting machine’s design.