ICCV 2022 Open Access Repository
Zekun Hao; Arun Mallya; Serge Belongie Ming-Yu Liu. MINECRAFT Proceedings of the IEEE/CVF Internation conference on Computer Vision (ICCV) 2021, pages. 14072-14082
We present GANcraft, a neural unsupervised rendering framework for generating realistic images of large 3D block worlds, such as those created in Minecraft. Our method takes the semantic block world as input, where each block is assigned a semantic label , such as dirt, grass, or water. The world is represented as an ongoing volumetric function. We train our model to render consistent, view-consistent photos using a camera controlled by the user. We designed a training technique that relies on adversarial and pseudo-ground truth learning, in the absence of actual images from the block world. This is in contrast to prior work on neural rendering for view synthesis, which requires ground truth images to calculate the geometry of the scene as well as the view-dependent appearance. GANcraft allows users to control both scene semantics as well as output style. Experimental results with comparison to solid baselines prove the effectiveness of GANcraft on this novel task of photorealistic 3D block synthesis.