Early detection of infants with autism spectrum disorder (ASD) can lead to effective developmental support. In clinical practice, early screening of 18-month-old infants is implemented using a parent-completed questionnaire. However, research has suggested that signs of ASD may also appear in movement characteristics during the first few months of age. In this paper, we propose a method to evaluate infant movement from videos based on non-negative tensor factorization (NTF) and apply it to ASD risk assessment in the neonatal period. The proposed method applies NTF to pose data estimated from videos, and decomposes the infant’s movements into multiple components while considering the linkage between body parts. In the experiment, we evaluated the effectiveness of the proposed method using 36 low-risk infants and 13 high-risk infants for ASD, with the aim of applying the method to ASD risk assessment. The results showed that the proposed method captured the tendency for infants to perform different movements depending on their risk level. Machine learning analysis revealed that the proposed method identified ASD risk with an accuracy exceeding 70%, which was comparable or superior to the existing video evaluation method based on heuristically designed indicators.