-- CS635 (Spring 2009) --

Capturing and Rendering Real-World Scenes

Have you ever wondered how to create models of 3D objects? Have you ever wanted to create a model of an entire room, floor, or building? Have you ever wanted to add real-world environments and objects to your games and virtual worlds? If so, this is the course for you!!!

pic5  pic4  pic1  pic6

pic3  pic2

1. Overview

[Course Summary]

The objective of this course/seminar is to understand the fundamental problems and challenges encountered when capturing, modeling, and rendering 3D structures and objects. The course covers several subjects within computer graphics, computer vision, and computer science so as to provide to the student a full understanding of the capture/model/render pipeline. From this understanding and cross-fertilization of ideas, it is expected that students will in the future be able to develop new and improved approaches.


A short list of the topics to be covered in this course, based on the latest research results in the field, are:


-          Geometry-based Acquisition

o   Stereo Methods

o   Structured Light Methods

-          Image-based Acquisition

o   Image-based Rendering Algorithms (Lumigraphs/Lightfields)

o   Photometric Stereo Methods

-          Photo-Geometric Methods

o   Latest hybrid algorithms combining both of the above

-          Calibration and other Fundamentals

o   Review of Optimization Methods

o   Pose Estimation

o   Pose-Free Calibration


The course is divided into two parts.


-          The first part describes, during the lectures, the aforementioned research methods to be presented by reviewing the latest works in the field. The emphasis is on making the student aware of the fundamental problems in the 3D acquisition pipeline. The lectures will describe in detail a few selected and important topics, giving the student a broad overview yet in depth knowledge of key topics.


-          The second part of the course consists of a guided research programming project that progressively implements one of the above methods and/or a derivative to be determined based on student expertise. The intent is to give the students a hands-on experience; depending on the particular case, necessary hardware can be provided, and a publication outlet for the semester project would be an ideal goal.


Classroom: LWSN 3151A

Time: TTh 9-10:15am

Office hours: by appointment


Students are required to have previous programming experience and are highly recommended to have previous computer graphics experience.



The course grade is determined by the performance in a short series of small programming assignments at the beginning and a final project. Each component will be evaluated during an interactive session with the instructor. The grade depends on a combination of meeting the requirements, the presentation, and the sophistication of the solution. There will be no final exam but rather a public demo day at the end of the semester with all projects.


[Spring 2007 Webpage – NOTE: the course will be significantly revised this year due to new research material in the field!]


For more information about the general type of research, I recommend looking at:


CGVLab Webpage: http://www.cs.purdue.edu/cgvlab

My webpage: http://www.cs.purdue.edu/~aliaga

For graphics in general: http://kesen.huang.googlepages.com/


A subset of relevant conferences that could be targeted with this semester’s work include:


ACM SIGGRAPH Asia 2009 (May submission deadline, Conference in December, Toyko)

IEEE International Conference on Computer Vision 2009 (March submission deadline, Conference in October, Kyoto)

Eurographics 2010 (September submission deadline, Conference in March, somewhere in Europe)

IEEE Visualization (maybe relevant) 2009 (March submission deadline, Conference in October, somewhere in the US)


3. Administrative Issues



All assignments must be handed-in by the specified due date/time. An assignment late by up to one day receives a 50% penalty (e.g., if maximum score is 10, it will be a maximum of 5), by up to two days a 75% penalty and after that a 100% penalty. The final project consists of 2 formal presentations (background research and final project presentation) and one informal presentation (intermediate progress report). These dates will be established once students and projects are settled. The final project presentations and demonstrations must be on time; otherwise a grade of 0 is given.


Code and Collaboration

All assignments, presentations, and projects must be done individually unless otherwise indicated by instructor. In research, it is highly encouraged to “build upon the shoulders” of others, however due credit must be given to the sources. Unreported copying or plagiarism will give you a failing grade in the course and you will be subject to standard departmental and University policies. For the programming assignments, code obtained from the Internet, books, or other sources may *not* be used. For the final project, previously-written code is permissible pending instructor approval.



4. Handouts, Code, Data


Assignment #0 – Camera Calibration

Nonlinear Least Squares Optimization Package: lmdif



Lectures Notes:

1/13 – Introduction

1/15 – OpenGL and Graphics

1/20 and 1/22 – Cameras and Calibration

1/27 – Pose Estimation, Art Gallery Problem, and Aspect Graph

1/29 – Features

“Good Features to Track”, J. Shi, C. Tomasi, IEEE CVPR 1994

Linear 3D Reconstruction I

Excerpts from ‘Computer Vision’ by Forsynth and Ponce, 2003

2/3   – Linear 3D Reconstruction II

Excerpts from ‘Computer Vision’ by Forsynth and Ponce, 2003

            Epipolar Geometry

Excerpts from ‘Computer Vision’ by Forsynth and Ponce, 2003)


Assignment #1 – 3D Scene Reconstruction, Linear pose estimation by Ess et al. (optional paper to read)

Triangulation Package: [FortuneDelaunay]

Basic 3D Rendering System: [BasicModeler] [Sample Models]


2/7  - Structured Light I

“High-Accuracy Stereo Depth Maps using Structured Light”, D. Scharstein, R. Szeliski, IEEE CVPR 2003.

2/10 – Structured Light II

“Pattern Codification Strategies in Structured Light Systems”, J. Salvi, J. Pages, J. Batlle, Pattern Recognition, 2004.

“Real-time 3D Model Acquisition”, S. Rusinkiewicz, O. Hall-Holt, M. Levoy, SIGGRAPH 2001.

2/12 – Lightfields and Lumigraphs I

                        “The plenoptic function and the elements of early vision”, E. Adelson, J. Bergen,

Chapter in Computational Models of Visual Processing, 1991.

“Rendering Trees from Precomputed Z-buffer Views”, N. Max, K. Ohsaki, Eurographics Rendering Workshop 1995.

                        “Plenoptic Modeling: An Image-Based Rendering System”, L. McMillan, G. Bishop, SIGGRAPH 1995.

                        “Light Field Rendering”, M. Levoy, P. Hanrahan, SIGGRAPH 1996.

                        “Lumigraph”, S. Gortler, R. Grzeszczuk, R. Szeliski, M. Cohen, SIGGRAPH 1996.

2/17 – Lightfields and Lumigraphs II

                        “Dynamically Reparameterized Light Fields”, A. Isaksen, L. McMillan, S. Gortler, SIGGRAPH 2000.

                        “Unstructured Lumigraph Rendering”, M. Bosse, L. McMillan, S. Gortler, M. Cohen, SIGGRAPH 2001.


Assignment #2 – Lumigraph++

Barycentric coordinates: [code fragment]


2/19, 2/24 – Photometric Stereo I

                        “Photometric Stereo: Lambertian Reflectance and Light Sources with Unknown Direction and Strength”,

Woodham R., Iwahori Y., Barman R., UBC, TR-91-18, 1991.

                        “Photometric Stereo with General, Unknown Lighting”, R. Basri, D. Jacobs, I. Kemelmacher, IJCV, 2007.

                        “Passive Photometric Stereo from Motion”, Lim et al., ICCV, 2005.


2/26 – Non-pinhole Camera Models


3/5 – Geometric + Photometric Stereo (hybrid)

                        “Photogeometric Structured Light: A Self-Calibrating and Multi-Viewpoint

Framework for Accurate 3D Modeling”, Aliaga and Xu, CVPR, 2008.


3/5 – Pose Free Acquisition

                        “Pictures and Trails”, Shi and Tomasi, CVPR 1994.

“A Framework for 3D Modeling using Pose-free Equations”, Aliaga et al. 2009



3/10, 3/12 – Light Transport and Inverse Light Transport

                        “Dual Photography”, Sen et al. SIGGRAPH 2005.

                        “Helmholtz Stereopsis”, Zickler et al. ECCV 2002.

                        “A Theory of Inverse Light Transport”, Seitz et al., ICCV 2005.


3/17 – Spring Break


3/19 – Spring Break


3/24 – Student Presentations

                        Topic: improving photometric solutions using geometric data


3/26 – Student Presentations

                        Topic: extracting structure and normals from temporal image sequences of urban landscapes


3/31 – Camera Registration


4/2 – Acquisition and Visualization


4/7, 4/9 – Student Presentations

                        Topic: building multi-perspective images


4/14, 4/16 – Radiometric Compensation

                        “Radiometric Self Calibration”, Mitsunaga and Nayar, CVPR 1999.

                        “A Projection System with Radiometric Compensation for Screen Imperfections”, Nayar et al, PROCAMS 2003.

                        “Making One Object Look Like Another: Controlling Appearance Using a Projector-Camera System”,

Grossberg et al., CVPR 2004.

                        “Radiometric Compensation using Inverse Light Transport”, Wetzstein et al, Pacific Graphics 2007.

                        “Virtual Restoration of Real-World 3D Objects”, Aliaga et al., SIGGRAPH Asia 2008.


4/21, 4/23 –     Acquisition and Compensation of Complex Non-Lambertian Surfaces.

                                    “Recovering Shape in the Presence of Interreflections”, Nayar, 1991.

                                    “Better Optical Triangulation though Space Time Analysis”, Curless and Levoy, 1995.

                                    “3D Modeling of Optically-Challenging Objects”, Park and Kak, 2008.

Tentative: Acquisition and Compensation of Deforming and Dynamic Surfaces.

            (various one-shot structured light methods)


4/28 – TBA + demo preparations


4/30 – Demo day!


Orientation-Free 3D Reconstruction: Traditionally, 3D reconstruction is performed given initial guesses of camera positions, camera orientations and point positions. Observing that camera orientation error makes things worse than camera position error, we eliminate its effect with a new formulation without camera orientation. The experiments show that our formulation is more robust and accurate to error in initial guesses comparing to the traditional 3D reconstruction method.


Single Image 3D Lightfield Compression Using Epipolar Occlusion Cameras: Lightfields are very powerful representations of 3D scenes but require vast amounts of data to represent all of the rays of a scene.  The epipolar occlusion camera (EOC) is able to efficiently encode a series of images (a 3D lightfield) by exploiting depth discontinuities and epipolar geometry. The existing implementation of the EOC takes the 3D scene as input and produces an EOC as output.  For my project, we have attempted to take pinhole camera images as input in order to assemble them into an EOC image.


Inferring building heights: A first step towards image-based modeling of building masses: We present a method for estimating the height of buildings from bird's eye images and building footprint geometries. Our approach consists of finding the vertical translation of the building footprint that optimizes the similarity between a set of edges extracted from the image, and the projection of the translated footprint onto the camera plane. While solutions are available for the problem of inferring building facades from images, little work has been done in image-based modeling of building masses. Solving the height estimation problem is the initial step of our method to address the mass modeling problem.