Pest recognition is very important to crops growing healthily, and this in turn affects crop yields and quality. At present, it is a great challenge to obtain accurate and reliable pest identification. In this study, we put forward a diagnostic system based on transfer learning for pest detection and recognition. On ten types of pests, the transfer learning method have achieved an accuracy of 93.8%. We compared the transfer learning method with human experts and a traditional neural network model. Experimental results show that the performance of the proposed method is comparable to human experts and a traditional neural network. To verify the general adaptability of this model, we used our model to recognize two types of weeds: Sisymbrium Sophia and Procumbent Speedwell, and achieved an accuracy of 99.46%. The proposed method can provide evidence for the control of pests and weeds and the precise spraying of pesticides.