Considering the needs of aerodynamic layout and high temperature protection, the star sensor is usually installed inside the near space hypersonic vehicle, and implements attitude measurement through porthole attached to the surface of the vehicle. Ground simulation experiments have shown that the detection capability of the star sensor is reduced, due to the increased stray light received at the detector, after the porthole structure is introduced into the optical path of the star sensor. In this paper, the phenomenon of structural stray light caused by the porthole is simulated. Stray light sources, system geometry and surface optical properties are modeled respectively in the simulation scenario. Point source transmittance is used as evaluation indices for the stray light suppression effect. The simulation calculations found that the surface scattering of the porthole glasses is the main reason that the porthole structure causes the stray light suppression effect to decrease. This structural stray light is mainly composed of scattered light formed by multiple scattering between porthole glasses and scattered light formed by scattering between the inner wall of the system and the porthole glasses. Compared with the case without porthole structure, the point source transmittance of the star sensor can reach a maximum of 10−4. The star sensor has been unable to meet the detection limit requirements. From the viewpoint of improving the surface optical properties of the system and optimizing the system structure, the suppression method for porthole structural stray light is studied. The results show that reducing the total integrated scatter of the system structure surface (including the system inner wall surface and the porthole glass surface) and using a single-layer porthole can decrease the influence of structural stray light caused by the porthole. The two methods are used to determine the final optimization scheme. The optimized system satisfies the detection requirements of the star sensor. Outside the stray light protection angle, the point source transmittance of the optimized system is below 3 × 10−5.