-- CS635 (Spring 2007) --

Capturing and Rendering Real-World Scenes

Have you ever wondered how to create models of 3D objects? Have you ever wanted to create a model of an entire room, floor, or building? Have you ever wanted to add real-world environments and objects to your games and virtual worlds? If so, this is the course for you!!!


Which of these two images represent the environment quality you'd rather see in your virtual world or game?

1. Overview

The objective of this course/seminar is to understand the fundamental problems and challenges encountered when capturing, modeling, and rendering 3D structures and objects. The course covers several subjects within computer graphics and computer science so as to provide to the student a full understanding of the capture/model/render pipeline. From this understanding and cross-fertilization of ideas, it is expected that students will in the future be able to develop new and improved approaches. The course is divided into three parts.

-          The first part describes, in a series of lectures, the basics of 3D reconstruction and several approaches to the problem. The emphasis of this section of the course is on making the student aware of the fundamental problems in the 3D acquisition pipeline. The lectures will describe in detail a few selected and important topics, giving the student a broad overview yet in depth knowledge of important topics.

-          The second part of the course consists of guided programming projects that progressively implement a basic 3D reconstruction pipeline. The intent is to give the student a practical application of the concepts covered during the lectures. Students will be (optionally) provided with digital cameras.

-          The third, and crucial part of the course, is to give the student a brief glimpse of researching and developing a novel project in 3D acquisition. The student will be guided through this process with several checkpoints and intermediate presentations. Pending approval by the instructor, small team projects are permissible but the contribution of each student must be clear – each student will still present their component individually.

[Fall 2004 webpage]

Classroom: LSWN 3151A

Time: MWF 9:30-10:20am

Office hours: by appointment


Students are required to have previous programming experience and are highly recommended to have previous computer graphics experience.


The course grade is determined by the performance of the programming assignments and the final project. Each component will be evaluated during an interactive session with the instructor. The grade depends on a combination of meeting the requirements, the presentation, and the sophistication of the solution. The grade percentage is divided as follows: programming assignments #0 to #2 (10%, 15%, 15%) and final project (60%). The final project grade is further subdivided into background research (15%), final presentation (15%), and project quality (30%).

Assignment #0 – Synthetic-World 3D Reconstruction

Given an implementation of a simple 3D synthetic renderer, augment the program to “reconstruct” synthetic scene points. First, render the scene from several viewpoints and save all camera and model parameters. Then, reconstruct the scene points from their projections using the standard OpenGL camera model. Since this is all done synthetically, the correct reconstruction is actually known.

Assignment #1 – Real-world 3D Reconstruction

Using the previous assignment, replace the synthetic data with real-world data. First calibrate either a course-provided digital camera or one of your own.  Then, “click” on corresponded scene points and on known scene-point fiducials (to estimate pose). Finally, reconstruct the observed scene points and project and blend the captured images onto the recovered model.

Assignment #2 - Lumigraph++

Further extend the previous assignment to implement a Lumigraph with varying focal-plane. Capture a large set of images of a scene and support varying the apparent “focus” of the reconstructed scene.

Final Project

Perform background research, develop, and present a project in 3D acquisition. Students may select from the following project-topic suggestions. Details and actual project descriptions will be determined during the course.

a)      3D Reconstruction: propose a robust method to do pose-estimation for reconstruction, e.g. reverse-pose estimation, viewpoint-planning.

b)      Lightfield/lumigraph rendering: improve upon a particular aspect of this approach.

c)      Point-based rendering: capture or render a model using points.

d)      Your own project.


2. Schedule

January 8/10/12 (Basics)

-          Introduction

-          Linear Algebra and Geometry Review

-          Graphics Pipeline Review (“Feed-forward Graphics Pipeline”)

o        Assignment #0 out: Synthetic-World 3D Reconstruction

January 17/19 (3D Reconstruction)

-          Linear/Nonlinear 3D Point Reconstruction (“Inverse Graphics Pipeline”)

-          Reconstruction/Pose Estimation

-    Slides: pose

January 22/24/26 (Camera Models and Calibration)

-          Triangulation

o        Voronoi/Delaunay/3D-Triangulation

-          Camera Models:

o        Single COP/Multiple COP

o        Omnidirectional: spherical, conical, hyperboloidal, and paraboloidal

-          Camera Calibration

o        Tsai/Function Optimization

o        Assignment #0 in

o        Assignment #1 out: Real-world 3D Reconstruction

-          Slides: omni-cameras, cameras, calibration

January 29/31/2 (Computer Vision)

-          Computer Vision Fundamentals

o        Low/middle/high-level vision

o        Feature tracking

-          Epipolar Geometry

-          Bundle Adjustment

-    Slides: computer-vision

February 5/7/9 (On Removing Intermediate Acquisition Steps)

-          Lightfields/Lumigraphs (removing correspondence)

o        Lightfield/SOI

-          Lumigraph/ULR

o        Changing focus plane

o        Changing aperture

-          Pose-free Reconstruction (removing pose)

o        (Brief explanation)

o        Assignment #1 in

o        Assignment #2 out: Lumigraph++

-         Slides: Lightfields/Lumigraphs

 February 12/14/16 (Point-based Rendering)

-          Simplified mathematical formulation

-          Two-pass rendering algorithm: estimate normals and splat size

-          TBA

February 19/21/23 (Other Reconstruction Approaches)

-          (View-dependent) Texture Mapping

-          Image Morphing/Image Warping

-          Space Carving/Voxel Coloring

o        Assignment #2 in

o        Final project out: overview/brainstorm

February 26/28/2 (Final Projects Brainstorm)

-          Informal discussion/Special Topics

-          Informal discussion/Special Topics

-          How to do background research

March 5/7/9 (Background Research Presentations: What is out there?)

-          20-30 mins/student

(Spring Break)

March 19/21/23 (Special Topics I - TBA)

-          Project dependent topics

March 26/28/30 (Initial Project Presentations: What is novel to it?)

-          20-30 mins/student

-          Brainstorming/feedback

April 2/4/6 (Special Topics II - TBA)

-          Project dependent topics

April 9/11/13 (Special Topics III - TBA)

-          Project dependent topics

April 16/18/20 (Preliminary Demo – mandatory)

-          How to give a demo

-          20-30 mins/student

April 23/25/27 (Final project presentations)

-          23 – Practice Presentations

-          25 – Practice Presentations

-          27 – Demo day!


3. Administrative Issues


All assignments must be handed-in by the specified due date/time. An assignment late by up to one day receives a 50% penalty (e.g., if maximum score is 10, it will be a maximum of 5), by up to two days a 75% penalty and after that a 100% penalty. The final project consists of 2 formal presentations (background research and final project presentation) and one informal presentation (intermediate progress report). These dates will be established once students and projects are settled. The final project presentations and demonstrations must be on time; otherwise a grade of 0 is given.

Code and Collaboration

All assignments, presentations, and projects must be done individually unless otherwise indicated by instructor. In research, it is highly encouraged to “build upon the shoulders” of others, however due credit must be given to the sources. Unreported copying or plagiarism will give you a failing grade in the course and you will be subject to standard departmental and University policies. For the programming assignments, code obtained from the Internet, books, or other sources may *not* be used. For the final project, previously-written code is permissible pending instructor approval.


4. Handouts, Code, Data


    Course Summary (same as webpage but in printable form)

    Assignment #0 - Synthetic-World Reconstruction

    Assignment #1 - Real-World  Reconstruction

    Assignment #2 - Lumigraph++

    Final Project - Overview