Fisheye Calibration Basics

Camera calibration is the process of computing the extrinsic and intrinsic parameters of a camera. Once you calibrate a camera, you can use the image information to recover 3-D information from 2-D images. You can also undistort images taken with a fisheye camera.

Fisheye cameras are used in odometry and to solve the simultaneous localization and mapping (SLAM) problems visually. Other applications include, surveillance systems, GoPro, virtual reality (VR) to capture 360 degree field of view (FOV), and stitching algorithms. These cameras use a complex series of lenses to enlarge the camera's field of view, enabling it to capture wide panoramic or hemispherical images. However, the lenses achieve this extremely wide angle view by distorting the lines of perspective in the images.

The Computer Vision Toolbox™ contains calibration algorithms for the pinhole camera model and the fisheye camera model. You can use the fisheye model with cameras up to a field of view (FOV) of 195 degrees. The field-of-view range for general classes of lenses are:

Normal lens — 50° - 80°
Wide angle lens — 80° - 120°
Ultra-wide angle lens — 120° - 180°
Fisheye lens — 180° - 220°
Super fisheye lens — greater than 220°

The Computer Vision Toolbox provides functionality to model the pinhole camera model up to 95°, the Kannala-Brandt Model [2] up to 115°, and the Scaramuzza Model up to 195°. The table lists the features you can use for these models.

Model Feature Maximum FOV

Pinhole Model

Model	Feature	Maximum FOV
Pinhole Model	`cameraIntrinsics`	Up to 95° For greater accuracy as you get close to 95° , increase the number of distortion coefficients. Can be used in all CVT camera calibration workflows.
Kannala-Brandt Model (OpenCV fisheye model)	`cameraIntrinsicsKB`	Up to 115° Import fisheye parameters from OpenCV using the `cameraIntrinsicsFromOpenCV` function.
Scaramuzza Model	`fisheyeIntrinsics`	Up to 195°

cameraIntrinsics

Up to 95°

For greater accuracy as you get close to 95° , increase the number of distortion coefficients.
Can be used in all CVT camera calibration workflows.

Kannala-Brandt Model (OpenCV fisheye model)

cameraIntrinsicsKB

Up to 115°

Import fisheye parameters from OpenCV using the cameraIntrinsicsFromOpenCV function.

Scaramuzza Model fisheyeIntrinsics Up to 195°

Because of the extreme distortion a fisheye lens produces, the pinhole model cannot model a fisheye camera.

The pinhole model and the fisheye model side-by-side

Fisheye Camera Model

The Computer Vision Toolbox calibration algorithm uses the fisheye camera model proposed by Scaramuzza [1]. The model uses an omnidirectional camera model. The process treats the imaging system as a compact system. In order to relate a 3-D world point on to a 2-D image, you must obtain the camera extrinsic and intrinsic parameters. World points are transformed to camera coordinates using the extrinsic parameters. The camera coordinates are mapped into the image plane using the intrinsic parameters.

Extrinsic Parameters

The extrinsic parameters consist of a rotation, R, and a translation, t. The origin of the camera's coordinate system is at its optical center and its x- and y-axis define the image plane.

The transformation from world points to camera points is:

Matrix operations showing w, (the scale factor) times the column vector x,y,1 (the image points) equal to P times the column vector x,y,z,1 (the world points). Also, P, (the camera matrix) equal to K, (the intrinsics matrix) times the row vector R, t,(the extrinsics, rotation and translation).

Intrinsic Parameters

For the fisheye camera model, the intrinsic parameters include the polynomial mapping coefficients of the projection function. The alignment coefficients are related to sensor alignment and the transformation from the sensor plane to a pixel location in the camera image plane.

The following equation maps an image point into its corresponding 3-D vector.

are the ideal image projections of the real-world points.
represents a scalar factor.
are polynomial coefficients described by the Scaramuzza model, where .
is a function of (u,v) and depends only on the distance of a point from the image center: .

The intrinsic parameters also account for stretching and distortion. The stretch matrix compensates for the sensor-to-lens misalignment, and the distortion vector adjusts the (0,0) location of the image plane.

The following equation relates the real distorted coordinates (u'',v'') to the ideal distorted coordinates (u,v).

Fisheye Camera Calibration in MATLAB

To remove lens distortion from a fisheye image, you can detect a checkerboard calibration pattern and then calibrate the camera. You can find the checkerboard points using the detectCheckerboardPoints and generateCheckerboardPoints functions. The estimateFisheyeParameters function uses the detected points and returns the fisheyeParameters object that contains the intrinsic and extrinsic parameters of a fisheye camera. You can use the fisheyeCalibrationErrors object to check the accuracy of the calibration.

Correct Fisheye Image for Lens Distortion

Open Live Script

Remove lens distortion from a fisheye image by detecting a checkboard calibration pattern and calibrating the camera. Then, display the results.

Gather a set of checkerboard calibration images.

images = imageDatastore('calibrationImages');

Detect the calibration pattern from the images. The 'PartialDetections' Name-Value argument is set to true by default allowing detection of partial checkerboards.

[imagePoints,boardSize] = detectCheckerboardPoints(images.Files, 'HighDistortion', true);

Generate world coordinates for the corners of the checkerboard squares.

squareSize = 20; % millimeters
worldPoints = patternWorldPoints('checkerboard',boardSize,squareSize);

Estimate the fisheye camera calibration parameters based on the image and world points. Use the first image to get the image size.

I = readimage(images,10); 
imageSize = [size(I,1) size(I,2)];
params = estimateFisheyeParameters(imagePoints,worldPoints,imageSize);

Remove lens distortion from the first image I and display the results.

J1 = undistortFisheyeImage(I,params.Intrinsics);
figure
imshowpair(I,J1,'montage')
title('Original Image (left) vs. Corrected Image (right)')

Figure contains an axes object. The hidden axes object with title Original Image (left) vs. Corrected Image (right) contains an object of type image.

J2 = undistortFisheyeImage(I,params.Intrinsics,'OutputView','same', 'ScaleFactor', 0.2);
figure
imshow(J2)
title('Output View with low Scale Factor')

Figure contains an axes object. The hidden axes object with title Output View with low Scale Factor contains an object of type image.

References

[1] Scaramuzza, D., A. Martinelli, and R. Siegwart. "A Toolbox for Easily Calibrating Omnidirectional Cameras." Proceedings to IEEE International Conference on Intelligent Robots and Systems, (IROS). Beijing, China, October 7–15, 2006.

[2] Juho Kannala and Sami Brandt. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE transactions on pattern analysis and machine intelligence, 28:1335–40, 09 2006.