Aflatoxin is a kind of highly toxic and carcinogenic substance with the characteristic of ultraviolet (UV) fluorescence. To explore the application of hyperspectral imaging technology in aflatoxin detection, a hyperspectral imaging system is built under 365 nm UV light. The hyperspectral images of 250 peanut kernel samples in 33 bands (400–720 nm) with five kinds of concentrations are collected. An object-oriented point-source illumination compensation method is proposed for compensating hyperspectral illumination and four fluorescence indexes such as Radiation Index (RI), Difference Radiation Indexes (DRIs), Ratio Radiation Index (RRI), and Normalized Difference Radiation Index (NDRI) are constructed. Radial basis function–support vector machine (RBF–SVM) model is constructed based on grid search to recognize and make regression analysis on the degree of aflatoxin contamination. The DRIs have the optimal performance, the accuracy rate of fivefold cross validation of SVM is 95.5%, and the mean square error (MSE) and R are 0.0223 and 0.9785, respectively, for testing data. Based on DRIs, narrowband spectra are searched by Fisher, Plus-l-remove-r, and band correlation coefficient. The narrowband spectrum obtained by Fisher’s optimization research is optimal, and the spectrum band is 410–430 nm. On this narrowband, the accuracy rate, MSE, and R are 87.2%, 0.27418, and 0.86732, respectively, for testing data using the same model of SVM. Perhaps this narrowband spectrum can be easily transplanted to the online aflatoxin detection production line. The results of the current research are of positive significance for the research of agricultural products such as aflatoxin grain sorting and online fast detection device.