Skip to content

FilmComposer: LLM-Driven Music Production for Silent Film Clips

Conference: CVPR 2025
arXiv: 2503.08147
Code: https://apple-jun.github.io/FilmComposer.github.io/
Area: Image Generation
Keywords: Film Scoring, LLM Multi-Agent, Rhythm Control, MusicGen, Digital Audio Workstation

TL;DR

Proposes FilmComposer, which simulates the workflow of professional musicians. Through three major modules—visual processing, rhythm-controllable MusicGen, and multi-agent arranging/mixing—it achieves the first automatic generation of high-quality professional soundtracks for silent film clips.

Background & Motivation

Background: AI music generation has made progress in waveform quality (MusicGen) and symbolic music control, but a significant gap still remains before meeting the professional standards of film scoring (48kHz/24bit, musicality, thematic development).

Limitations of Prior Work: Existing video soundtrack generation methods (such as CMT, VidMuse) primarily target short videos, overlooking three core aspects of film music: audio quality, musicality, and musical development.

Core Idea: Combines the richness of waveform music generation with the high quality of symbolic music generation, utilizing a multi-agent system for arranging and mixing to achieve cinema-grade scoring.

Method

Overall Architecture

The three modules correspond to the three-step workflow of professional musicians: (1) Visual processing \(\rightarrow\) analysis/annotation of beat points and semantics; (2) Rhythm-controllable MusicGen \(\rightarrow\) composition to generate the main melody; (3) Multi-agent evaluation/arranging/mixing \(\rightarrow\) producing the final cinema-grade audio.

Key Designs

  1. Rhythm-Controllable MusicGen:

    • Function: Generates melodies synchronized with film clips based on beat points and visual-linguistic descriptions.
    • Mechanism: Introduces a rhythm conditioner (chromagram features) into MusicGen, which is prepended to the Transformer decoder along with the visual description text conditions. Fine-tuned on the custom MusicPro-7k dataset.
    • Design Motivation: The first large language model capable of directly generating rhythm-aligned music from visual inputs.

  2. Multi-Agent Evaluation System:

    • Function: Evaluates the musicality of the generated melody and decides whether to regenerate.
    • Mechanism: Based on the AutoGen framework, five reviewer agents (Mode/Melody/Harmony/Rhythm/Emotion) sequentially assess the melody through a chat-based discussion, scoring it based on music theory standards.
    • Design Motivation: Ensures that only high-quality melodies proceed to the subsequent arranging phase.

  3. Multi-Agent Arranging & Mixing:

    • Function: Orchestrates the melody into a complete piece and mixes/renders it within a Digital Audio Workstation (DAW).
    • Mechanism: Six agents (Analyze/Arrange/Instrument/Volume/Mixing/Reviewer) collaborate in a group chat, receiving motion descriptions and ABC notation melodies to design arranging plans and drive the DAW to execute them.
    • Design Motivation: Leverages the music theory knowledge and reasoning capabilities of LLMs to replace manual orchestrating.

Loss & Training

Constructs the MusicPro-7k dataset: 7418 film-clip-to-music pairs, containing descriptions, beat points, and main melodies. Training labels are generated using a proposed main melody extraction algorithm (based on track coverage and note ratios).

Key Experimental Results

Main Results

Achieves SOTA across five dimensions: audio quality, video alignment/consistency, diversity, musicality, and musical development, while proposing new evaluation metrics tailored for film scoring.

Key Findings

  • Multi-agent arranging significantly enhances the quality of musical development.
  • Rhythm control allows the generated music to align highly synchronously with video beats.
  • The DAW mixed output reaches the professional standard of 48kHz/24bit.

Highlights & Insights

  • Comprehensively simulates the workflow of professional human musicians.
  • The framework is highly interactive, allowing user intervention at each step.
  • Combines the respective advantages of waveform generation and symbolic music processing.

Limitations & Future Work

  • The multi-agent system relies on the music theory knowledge of LLMs, which may introduce bias.
  • The DAW operational steps are complex, presenting room for improvement in end-to-end automation.
  • The size of the MusicPro-7k dataset remains relatively limited.

Rating

  • Novelty: 8/10 — The first complete AI system tailored for film scoring.
  • Technical Depth: 7/10 — Multi-module integration.
  • Experimental Thoroughness: 7/10 — Introduces new evaluation metrics.
  • Writing Quality: 7/10 — Clear structure.