Zuoyue Li - 李 作越

I am a Ph.D. student in the Computer Vision and Geometry (CVG) group at ETH Zurich, supervised by Prof. Marc Pollefeys. My research interests focus on 3D vision and 3D generative models, and I collaborate closely with Prof. Martin R. Oswald and Prof. Zhaopeng Cui. My doctoral research was mainly funded by the Swiss Data Science Center (SDSC) fellowships.　

I am currently a research intern at Google Zurich with a topic on generative AI and digital humans. I was a research engineer intern at Meta Zurich with a topic on 3D object detection and scene understanding, and was an overseas researcher in the Computer Vision Group at the Institute of Industrial Science (IIS), The University of Tokyo (東京大学), supervised by Prof. Yoichi Sato, funded by Japan Society for the Promotion of Science (JSPS) fellowships.　

I obtained my M.Sc. degree in Computer Science with distinction at ETH Zurich. I completed my B.Eng. degree in Electronic and Information Engineering as an outstanding graduate at Zhejiang University (浙江大学).　

Email  /  GitHub  /  Google Scholar  /  LinkedIn

3D Urban Scene Generation from Satellite Images with Diffusion
Zuoyue Li, Zhenqiang Li, Zhaopeng Cui, Marc Pollefeys, Martin R. Oswald.
CVPR 2024 (Highlight)  /  Paper  /  Project Page  /  Code

Generalize diffusion models to 3D sparse space and perform urban scene generation on a given or predicted geometry, followed by neural rendering techniques to render arbitrary views with excellence in both single-frame quality and inter-frame consistency.

CompNVS: Novel View Synthesis with Scene Completion
Zuoyue Li, Tianxing Fan, Zhenqiang Li, Zhaopeng Cui, Yoichi Sato, Marc Pollefeys, Martin R. Oswald.
ECCV 2022  /  Paper  /  Code

Synthesize novel views from RGB-D images with largely incomplete scene coverage. Perform generation on a sparse grid-based neural representation to complete unobserved scene parts. Extrapolate the missing area and render consistent photorealistic image sequences.

Factorized and Controllable Neural Re-rendering of Outdoor Scene for Photo Extrapolation
Boming Zhao, Bangbang Yang, Zhenyang Li, Zuoyue Li, Guofeng Zhang, Jiashu Zhao, Dawei Yin, Zhaopeng Cui, Hujun Bao.
ACM Multimedia 2022 (Oral)  /  Paper  /  Project page

Expand tourist photos from a narrow field of view to a wider one while maintaining a similar visual style. Propose factorized neural re-rendering model to produce photorealistic novel views from cluttered outdoor Internet photo collections, which enables applications such as controllable scene re-rendering, photo extrapolation, and 3D photo generation.

Sat2Vid: Street-view Panoramic Video Synthesis from a Single Satellite Image
Zuoyue Li, Zhenqiang Li, Zhaopeng Cui, Rongjun Qin, Marc Pollefeys, Martin R. Oswald.
ICCV 2021  /  Paper

Synthesize both temporally and geometrically consistent street-view panoramic video from a single satellite image and camera trajectory. Explicitly create a 3D point cloud representation of the scene and maintain dense 3D-2D correspondences across frames that reflect the geometric scene configuration inferred from the satellite view. Generation adopts GAN-based methods in the 3D sparse space.

NVS-MonoDepth: Improving Monocular Depth Prediction with Novel View Synthesis
Zuria Bauer, Zuoyue Li, Sergio Orts Escolano, Miguel Cazorla, Marc Pollefeys, Martin R. Oswald.
3DV 2021  /  Paper

Application of novel view synthesis to improve monocular depth estimation, with a wrapping scheme using the estimated depth to an additional viewpoint. The same depth network is applied to the synthesized view and provides another supervision.

Spatio-Temporal Perturbations for Video Attribution
Zhenqiang Li, Weimin Wang, Zuoyue Li, Yifei Huang, Yoichi Sato.
TCSVT 2021  /  Paper

Take extra attention to the evaluation metrics for video attribution methods. Specifically, a new reliability measurement method is proposed, by which the reliable and objective metrics are screened. The effectiveness of the proposed attribution method is extensively investigated by both subjective and objective evaluation, and comparison with multiple significant baseline attribution methods.

Towards Visually Explaining Video Understanding Networks with Perturbation
Zhenqiang Li, Weimin Wang, Zuoyue Li, Yifei Huang, Yoichi Sato.
WACV 2021  /  Paper

Aim to provide an easy-to-use visual explanation method for video understanding networks with diversified structures. Propose a generic perturbation-based visual explanation method, enhanced by a novel spatiotemporal smoothness constraint. The method enables the comparison of explanation results between different video classification networks and avoids generating pathological adversarial explanations for video inputs.

Geometry-Aware Satellite-to-Ground Image Synthesis for Urban Areas
Xiaohu Lu*, Zuoyue Li*, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys, Rongjun Qin.
*Equal contribution. CVPR 2020  /  Paper  /  Code

Generate panoramic street-view images that are geometrically consistent with a given satellite image via a GAN-based network with the proposed geo-transformation layer that retains the physical satellite-to-ground relation. The synthesized images retain well-articulated and authentic geometric shapes, as well as the texture richness of the street view in various scenarios.

Topological Map Extraction from Overhead Images
Zuoyue Li, Jan Dirk Wegner, Aurelien Lucchi.
ICCV 2019  /  Paper  /  Code

Circumvent the conventional pixel-wise segmentation of aerial images and predict objects in a vector representation directly. Directly extracts the topological map of a city from overhead images as collections of building footprints and road networks.