It is extremely important to correctly identify the carrot appearance quality in design and manufacture of carrot sorter. In this chapter, we have established a carrot appearance quality control system based on deep learning framework. The information of carrot is collected using the image, and thereafter, the recognition model is erect on AlexNet network, which is pretrained by a large-scale computer vision database (Image-Net). Our framework uses transfer learning, which trains neural networks with small amounts of data compared to the traditional convolutional neural network (CNN). Applying this approach to the dataset of carrot images, we demonstrate the performance of the proposed model by comparing it with the performance of human experts. The different grades can be recognized with great accuracy from a large amount of carrots under different surface conditions. Further, we demonstrate the general applicability of our system in potato quality recognition. At the same time, when the number of training samples is small, high recognition accuracy can still be achieved through transfer learning. The model can not only meet the requirements of classification recognition but can also greatly reduce the amount of cost spent in sample collection.