A research team led by Professor Junyong Noh from the Graduate School of Culture Technology introduced a method of generating a virtual camera layout of a 3D animation scene using a reference video. The team believes that the proposed method can provide effective assistance to 3D artists when positioning a camera in a virtual scene.

Framing and camera movement are essential components of cinematography. These tools assist artists in focusing a viewer’s attention on a particular object and shaping the emotional impact of a given scene. However, the process of creating a camera layout that produces the desired aesthetic effect in a virtual scene is often complex and time-consuming. A high degree of freedom of a virtual camera makes it difficult for creators to deliver their intentions precisely; positioning numerous cameras throughout the scene to capture actions from different angles is a possible solution, but this is burdensome and hard to manipulate.

As a solution to this problem, Professor Noh’s team developed a computer vision-based framework that automatically generates a virtual camera layout by analyzing the cinematic intention of a reference video. Firstly, the framework segments a reference video into a sequence of shots with distinct camera layouts. With the detected shot boundaries, each shot is analyzed to extract the scene layout — in particular, framing, camera movement, and its subjects’ visual features. All of the extracted data is stored in a shot list, a document that describes in detail every camera shot that needs to be captured in a video production, which is used by the framework to calculate a layout for a given scene. Additionally, the subjects are manually associated with staged 3D assets of characters of the new project. The framework is able to determine the desired positioning even for stylized characters with non-human proportions without compromising their visibility.

According to the results of the conducted user studies, the team confirmed that the output of this framework is comparable to layouts composed by a human artist. The team believes that the system can be of use to both novice and professional 3D creators. On one hand, beginners with limited knowledge of 3D art can rapidly test out different virtual camera layouts in the pre-visualization stage by utilizing this framework. On the other, in professional projects, such as animated series creation, where artists have to work on an extensive number of shots regularly, the framework can improve productivity by preventing artists from performing repetitive work and allowing them to focus more on the creative side of their work.