Cross-Modal Metrics for Capturing Correspondences Between Music Audio and Stage Lighting Signals

摘要

In the performing arts, the interplay between visual and auditory stimuli is of substantial importance. This also applies to stage lighting and its correspondence with the music audio in live music performances. In this paper, we aim for measuring cross-modal correlations between music audio signals and stage lighting signals and formalize these correlations by introducing specific metrics. Our metrics capture the temporal concurrence of musical beats and lighting changes, the correlation of loudness and brightness, and the correspondence of structural parts in music and lighting, respectively. We achieve this by relating features extracted from the audio signal with music information retrieval techniques to features extracted from the lighting control parameters. We demonstrate that the proposed metrics effectively differentiate real-world lighting signals from randomly generated signals in relation to the real-world audio signal. Moreover, we show that the metrics capture cross-modal correspondences of artistic relevance and serve to provide interesting analyses of four distinct lighting show styles. In addition, we show that our metrics are a valuable tool for evaluating generative systems that produce lighting signals from audio input and, therefore, can be crucial for developing such systems.