The detection of peanut cultivars and quality is an important composition for peanut breeding and quality testing. In order to evaluate the feasibility of mass peanut seed detection via appearance characteristics, first we take pictures of 48 varieties and 6 different qualities of each variety with digital camera, and then we use the method of principle component analysis and artificial neural network to establish a seed recognition model which is made up of 49 distinct appearance characteristics with regard to their shape, texture, and color and optimize the model. The testing result indicates: after the model optimization, variety recognition rate and quality recognition rate reach 91.2% and 93.0%, respectively; the color characteristic plays an impactful role in the variety and quality detection; and the appearance characteristic is more helpful in detecting the quality than in differentiating between varieties. The detection method based on the machine vision possesses the cost and speed advantages, and it can be used in identification of peanut cultivars and evaluation of their quality.