To speed up the process of carrot grading and realize automatic carrot grading, we have designed and implemented an automatic system based on computer vision, which consists of a feeding machine, a conveyor system, an image processing system, an image acquisition system, and a grading control system. Carrots were transferred based on three-level accelerated relay transmission techniques, and the ultimate speed is about 4 m s−1. To obtain a high-quality image of carrot moving at high speed, frame-trigger linear push-broom imaging technology was used to acquire the carrot image. The system was controlled by the single-chip microcomputer (SCM) and a computer. Pneumatic control and time-delay techniques were applied to control the whole process of carrot grading. Detection methods were proposed to detect surface defects like cracks, fracture, abnormity, and so on. Experimental results show that the proposed defect detection method can achieve an overall accuracy rate of 90% and about 15 carrots could be graded per second. The constructed grading system could meet the demand of carrot grading and sorting.