Lectures: on *Tuesdays, 8:30-10:00 *, lecture hall 0-805 ‘Fejér Lipót’

Practices: On *Tueasdays 14.15-15.45*, laboratory room 00-803 ‘Programozási Nyelvi labor’ and 00-807 ‘Adatbázis labor’

Teachers:

- Levente Hajder
- Tamás Tófalvi

Demonstrators:

- Botond Nás
- Illés Sas

## Dates for Assignment Presentation

- TDL

## Dates for Oral Exams

- TDL

## Schedule for the oral exam

- Then a timetable is published here with the NEPTUN ID of the examinees.
- Each student has 20 minutes for the oral exam, and another 20 minutes to get ready.
- The question, selected by a random number generator, is announced in the chatbox, 20 minutes before the personal timeslot.
- The final mark is given at the end of the oral exam. It is immediately written into NEPTUN.
- Due to the university rules, another students must listen the exam. The NEPTUN ID of the listeners are also announced in the timetable.
- Students are not allowed to use headsets during the exam.
- EVERYTHING can be used during the exam except the help of other person(s).

## Course materials:

- Derivatives with respect to vectors
- Lagrange multiplier
- Robust model fitting: RANSAC
- Multi-model fitting
- Camera models
- RQ-factorization
- Numerical optimization
- Point Registration

Presentations:

Materials for practices:

- GUI by OpenCV
- Affine transformations
- Random Sampling Consensus (RANSAC)
- Point Visualisation
- Multi-model fitting
- Homography Estimation
- Camera Calibration
- Stereo Vision
- Mid-point Triangulation
- Tomasi-Kanade factorization
- Point Registration
- Object Detection on LiDAR point clouds
- Chessboard-based calibration

**Videos for the labs can be downloaded from here.**

Latest version of OpenCV can be downloaded from project’s website or our page.

Questions for the oral exam:

- Basic estimation theory: solution for homogeneous and inhomogeneous linear system of equations.
- Robust fitting methods: RANSAC, LO-RANSAC.
- Multi-model fitting.
- Camera models: perspective camera, orthogonal projection, weak-perspective camera.
- Calibration of perspective camera using a spatial calibration object: Estimation and decomposition of projection matrix.
- Chessboard-based calibration of perspective cameras. Radial and tangential distortion.
- Plane-plane homography. Panoramic images by homographies.
- Estimation of homography. Data normalization.
- Basics of epipolar geometry. Essential and fundamental matrices. Rectification.
- Estimation of essential and fundamental matrices. Data normalization. Decomposition of essential matrix.
- Triangulation for both standard and general stereo vision.
- Stereo vision for planar motion.
- Tomasi-Kanade factorization: multi-view reconstruction by orthogonal and weak-perspective camera models.
- Reconstruction by merging stereo reconstructions. Registration of two point sets by a similarity transformation.
- Numerical multi-view reconstruction: bundle adjustment.
- Tomasi-Kanade factorization with missing data.
- Reconstruction by special devices: laser scanning, depth camera, LiDAR

**Assignments **(2021 fall, obsolet; visible only for over-viewing the difficulty)

1. **Robust plane detection on LiDAR point clouds by SeqRANSAC (30%). **Implement the Sequential RANSAC (SeqRANSAC) algorithm with optimal plane detection. The method for optimal plane detection is overviewed in the lecture on 27th September, you can find the details here. Find the three most dominant planes by SeqRANSAC. Try to tune the RANSAC parameter to find good planes, however, do not worry if the detection is not perfect. Run the method on all the data (except plane.xyz) given in this ZIP file of RANSAC project . Upload the source codes and the results in PLY format (coloured point clouds) to Canvas. Colour the three detected planes by red, green and pink.

Sequential RANSAC is the iterative modification of RANSAC: it finds a plane by RANSAC, then the plane points are removed from the dataset, and RANSAC method is run again on the rest of the points. The plane detection and removal can be iterated many times. In this assignment, the three most dominant plane should be found.

It is allowed to modify the basic RANSAC project, you do not have to start as an empty project.

**Submission deadline: 12th November**

2. **Image Stitching using a homography** **(40%)**. There is a car, and it is equipped with two digital cameras. The cameras are synchronized in time. We have uploaded ten sample images for both cameras. The task of the second assignment is to stitch the images of the two cameras by a homography that should be estimated as:

(i) Take two images that were taken at the same time. The homography should be estimated only for one image pair.

(ii) Do feature matching by OpenCV or other tools like ASIFT . (5%)

(iii) Implement the normalized linear homography estimation. (The original, unnormalized implementation is allowed to be modified. Hint: Source code in camera calibration can be easily modified for data normalization) (15%)

(iv) Homography estimation should be robustified by the standard RANSAC method. (20%)

The images can be downloaded from our webpage.

Transform the images from the second camera to fit the first ones and stitch the videos.

Upload the source code and the ten stitched images in a ZIP file to Canvas.

**Submission deadline: 17th December **

3. **Triangulation for standard stereo (30%) **

In this assignment, there are also two car-mounted cameras given. However, they are next to each other, thus the camera setup is a ‘standard stereo’. The goal is to retrieve the 3D point cloud of the environment. The following subtask should be solved:

- Do feature matching by OpenCV or other tools like ASIFT .
- Discard the outlier matchings. You can use any robust methods including the well-known RANSAC method. Feel free to implement your own ideas if you have.
- Implement the special triangulation method for standard stereo.
- Save the obtained point cloud in XYZ or PLY format.
- Upload the source code and the resulting point clouds to Canvas.

The stereo images (five pairs) can be downloaded from our webpage. The camera intrinsic matrix (malaga.mat) is also available there. (The parameters are the same for the cameras.) The baseline is 12cm. The pixel size is 166.67 pixel/millimeter, the focal length is 3.8mm. Run your implementation for all the five image pairs.

**Submission deadline: Two days before oral exam**

Final mark: oral exam (0-100%) + homework (0-100%) + extra scores (optional, given by the teachers)

Marks:

- Excellent (5): 170%-
- Good (4): 140-169%
- Satisfactory (3): 110-139%
- Pass (2): 80-109%
- Fail (1): 0-79%