TRANSFORMER-BASED AUDIO-VISUAL AUTISM RECOGNITION SYSTEM BASED ON FAMILY OBSERVATION SCHEDULE

Abstract

A behavior recognition system for analyzing an audio-video signal to detect challenging behaviors in autism via behavioral features. The system includes a processor configured to segment the audio-video signal into clips of audio data and video data, each of said clips having a predefined duration and annotated with interaction styles. The processor samples and preprocesses the audio data and video data of said clips to provide square video patches and square audio patches. The processor tokenizes the square video patches to embed video positional information and video modality information, and tokenize the square audio patches to embed audio positional information and audio modality information. And, the processor predicts behaviors based on the tokenized square video patches and the tokenized square audio patches.

CPC Classifications