The 21th International Symposium on Mixed and Augmented Reality (ISMAR 2022) has recently been successfully completed in Singapore. The paper “FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality” contributed by Prof. Xubo Yang’s team from Digital ART Laboratory, School of Software Engineering, SEIEE, won the Best IEEE ISMAR 2022 Journal Paper Award. The paper is published in IEEE TVCG (CCF Recommended Class A Journal), the top journal in the field of visualization and computer graphics. The first three authors of the paper are Nianchen Deng (Ph.D. candidate of SJTU), Zhenyi He (M.S. of SJTU, Ph.D. of New York University), and Jiannan Ye (Ph.D. candidate of SJTU), with Prof. Xubo Yang as the joint corresponding author.

Contributions of the FoV-NeRF Paper

An essential need for future VR/AR is to teleport different users and environments for collaborative scenarios, requiring high-quality and low latency for real-time rendering. The neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes could be achieved without loss of view-dependent effects. While NeRF can significantly benefit rendering for VR applications, it faces unique challenges posed by high field-of-view, high resolution, and stereoscopic/egocentric viewing of VR, typically causing low quality and high latency of the rendered images.

FoV-NeRF presents the first gaze-contingent 3D neural representation and view synthesis method. It incorporates the human psychophysics of visual- and stereo-acuity into an egocentric neural representation of 3D scenery. It then jointly optimizes the latency/performance and visual quality while mutually bridging human perception and neural scene synthesis to achieve perceptually high-quality immersive interaction. The results showed that the proposed method significantly reduces latency (up to 99% time reduction compared with NeRF) without loss of high-fidelity rendering (perceptually identical to full-resolution ground truth). The presented approach may serve as the first step toward future VR/AR systems that capture, teleport, and visualize remote environments in real time.

Scene representation and rendering results by gaze-contingent neural radiance field for VR

FoV-NeRF Visual acuity adaptive synthesis and rendering mechanism

About “IEEE ISMAR”

IEEE ISMAR is the top international academic conference in the field of augmented and mixed reality, jointly organized by IEEE Computer Society, IEEE VGTC, and ACM SIGGRAPH. IEEE ISMAR 2022 was held in Singapore, covering two keynotes, over 100 posters and demos, and about 20 workshops held by experts in the field. The conference received more than 130 high-quality papers on over 20 topics, including interaction techniques, rendering, VR/AR applications, modeling, etc, and one best journal paper was selected by the award committee.

Preview of “IEEE VR 2023”

IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR) is the premier international event for presenting research results in the broad areas of virtual, augmented, and mixed reality (VR/AR/MR). Since 1993, IEEE VR has presented groundbreaking research and accomplishments by virtual reality pioneers: scientists, engineers, designers, and artists, paving the way for the future. IEEE VR 2023will be held in Shanghai, China, on March 25th-29th, 2023, the first time the IEEE VR conference will be held in China. Professor Xubo Yang of the School of Software Engineering, SEIEE, will serve as one of the general chairs of IEEE VR 2023 to lead the conference organization. Welcome researchers and students in the related VR/AR research field to join IEEE VR 2023!

Website: http://ieeevr.org/2023

WeChat Subscription: IEEEVR