Tutorial on 

Representation, Evaluation and Utilities of Point Clouds

IEEE Int'l Conf. on Visual Communications and Image Processing (VCIP), Dec 13 – 16, 2022

Speaker: Prof Weisi Lin
School of Computer Science and Engineering, Nanyang Technological University, Singapore

Brief Description: 

Point clouds (PCs) increasingly become available and indispensable for diversified applications in our work and life. They contain information from any viewpoint, and therefore brings about both more opportunities and challenges. Creation of digital twins for everything in physical worlds has been enabled by increasingly economical availability of 3D point clouds (3D PCs), as a result of rapid development of Lidar, RGB-D sensing and computational structure-from-motion (SfM) technology. This offers unprecedented opportunities in digital transformation and smart cities. PCs can scale from a single object (like a desk or statue) to an entire city, for VR/AR/MR, BIM (building-information model), facial recognition access control, smart manufacturing, urban surveillance and planning, cultural heritage preservation, crime investigation, navigation of robots, autonomous driving and flying, and medical and biology science. 
  
This tutorial will present the recent research and development for PC representation, evaluation, and utilities. Related important topics include but are not limited to various filtering and processing, simplification, compression, shape/mesh construction, and image-based localization. To achieve economical and green computing, PC saliency and quality modelling can be formulated; it may aim at human uses or machine uses (essential in the AI Era). Utility-oriented evaluation is especially useful due to the big dynamics of scalability and diversification of PCs in practice. More exciting possibilities are expected because PCs provide bridges among computer vison, computer graphics and multimedia interaction. Future research directions will be substantially analysed and further discussed, including those for the emerging metaverse.

  Outline of the tutorial 

  1. Introduction        
       1.1 Basic concept definition & scopes of this tutorial        
       1.2 Demand and potential of PCs 
       1.3 Current and emerging utilities          
       1.4 Overview of the accomplished work 
  2. Representation of PCs         
       2.1 3D data acquisition                 
          Direct acquisition 
          Acquisition via computing 
          General data structure        
       2.2 PC compression                 
          Standards                 
          Non-standardized methods        
       2.3 PC simplification 
       2.4 Mesh construction from a PC 
  3. Modelling PC Saliency         
       3.1 Saliency for human uses 
       3.2 Keypoint detection and feature descriptors 
          Handcrafted approaches 
          Learning-based approaches 
       3.3 Utility-oriented saliency 
  4. PC quality Evaluation 
       4.1 Evaluation for signal fidelity 
       4.2 Evaluation for human uses 
       4.3 Evaluation with feature analysis 
       4.4 Utility-oriented evaluation
  5.  ​Possible Future Research 
       5.1 Further work for important existing topics 
       5.2 Exploring more advancement        
          Physical world        
          Metaverse  
  6. Final Discussion & Remarks 

About the speaker

Weisi Lin is an active researcher and research leader in image processing, perception-based signal modelling and assessment, video compression, and multimedia communication systems. He had been the Lab Head, Visual Processing, Institute for Infocomm Research (I2R). He is currently a Professor in School of Computer Science and Engineering, Nanyang Technological University, Singapore, where he also serves as the Associate Chair (Research). 

He is a Fellow of IEEE and IET, and an Honorary Fellow of Singapore Institute of Engineering Technologists. He has been awarded Highly Cited Researcher 2019, 2020 and 2021 by Clarivate Analytics, and elected as a Distinguished Lecturer in both IEEE Circuits and Systems Society (2016-17) and Asia-Pacific Signal and Information Processing Association (2012-13). He has been an Associate Editor for EEE Trans. Image Process., IEEE Trans. Circuits Syst. Video Technol., IEEE Trans. Multimedia, IEEE Sig. Proc. Letters, Quality and User Experience, and J. Visual Commun. Image Represent. He has been a TP Chair for several international conferences. He believes that good theory is practical, and has delivered 10 major systems and modules for industrial deployment with the technology developed.

Relevant Recent publications (download for copies): 
C. Lv, W. Lin, B. Zhao, “Voxel Structure-based Mesh Reconstruction from a 3D Point Cloud”, IEEE Transactions on Multimedia, accepted 

J. Xiong, H. Gao, M. Wang, H. Li, K. N. Ngan, W. Lin,  “Efficient Geometry Surface Coding in V-PCC”, IEEE Transactions on Multimedia, accepted.

C. Lv, W. Lin, B. Zhao, "Approximate Intrinsic Voxel Structure for Point Cloud Simplification", IEEE Transactions on Image Processing, 30(9): 7241 – 7255, 2021 

J. Xiong, H. Gao, M. Wang, H. Li, W. Lin, “Occupancy Map Guided Fast Video based Dynamic Point Cloud Coding”, IEEE Transactions on Circuits and Systems for Video Technology, accepted

B. Zhao, W. Lin, C. Lv, “Fine-Grained Patch Segmentation and Rasterization for 3D Point Cloud Attribute Compression”,IEEE Trans. on Circuits and Systems for Video Technology, 31(12): 4590-4602, 2021 

W. Cheng, W. Lin, K. Chen, X. Zhang, “Cascaded Parallel Filtering for Memory Efficient Image-based Localization”, International Conference on Computer Vision (ICCV), 2019 

X. Ding, W. Lin, Z. Chen, X. Zhang, “Point Cloud Saliency Detection by Local and Global Feature Fusion”, IEEE Transactions on Image Processing, 28(11): 5379–5393, 2019 

W. Cheng, K. Chen, W. Lin, M. Goesele, X. Zhang, Y. Zhang, "A Two-stage Outlier Filtering Framework for City-Scale Localization using 3D SfM Point Clouds", IEEE Transactions on Image Processing, 28(10): 4857 - 4869, 2019   

W. Cheng, W. Lin, X. Zhang, M. Goesele, M-T Sun, “A Data-driven Point Cloud Simplification Framework for City-scale Image-based Localization”, IEEE Transactions on Image Processing, 26(1): 262-275, 2017 

S. M. Prakhya, B. Liu, W. Lin, V. Jakhetiya, S. C. Guntuku, “B-SHOT: A Binary 3D Feature Descriptor for Fast Keypoint Matching on 3D Point Clouds”, Autonomous Robots,  41(7):1501–1520, 2017

S. M Prakhya, J. Lin, V. Chandrasekhar, W. Lin, B. Liu, "3DHoPD: A Fast Low Dimensional 3D Descriptor", IEEE Robotics and Automation Letters, 2(3): 1472-1479, 2017

S. M. Prakhya, W. Lin, V. Chandrasekhar, B. Liu, J. Lin, “Low Bit-rate 3D Feature Descriptors for Depth Data from Kinect-style Sensors”, Signal Processing: Image Communication, 51: 40–49, 2017 

S.  M. Prakhya, B. Liu, W. Lin, “Detecting Keypoint Sets on 3D Point Clouds via Histogram of Normal Orientations”, Pattern Recognition Letters, 83 (Part 1): 42–48, 2016

S. M. Prakhya, B. Liu, W. Lin, “B-SHOT: A binary feature descriptor for fast and efficient keypoint matching on 3D point clouds”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015

Expected audience:
Researchers, engineers and graduate students from universities, research centers and companies, in areas related to point clouds, 3D meshes, Lidar, signal and visual processing, video compression, vision-based modelling, computer graphics and animation, machine learning, VR/AR/MR, BIM, smart manufacturing, urban surveillance and planning, cultural heritage preservation, crime investigation, navigation of robots, autonomous driving and flying, multimedia communication and networking, cloud and mobile computing, medical and biology science, and the list may go on.