블로그 이미지
Leeway is... the freedom that someone has to take the action they want to or to change their plans.
maetel

Notice

Recent Post

Recent Comment

Recent Trackback

Archive

calendar

1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31
  • total
  • today
  • yesterday

Category

2010. 4. 28. 12:54 Computer Vision
Rahbar, K. and Pourreza, H. R. 2008. Inside looking out camera pose estimation for virtual studio. Graph. Models 70, 4 (Jul. 2008), 57-75. DOI= http://dx.doi.org/10.1016/j.gmod.2008.01.001


posted by maetel
2010. 3. 3. 19:54 Computer Vision
http://www.hitl.washington.edu/artoolkit/

ARToolKit Patternmaker
Automatically create large numbers of target patterns for the ARToolKit, by the University of Utah.


ARToolKit-2.72.tgz 다운로드

http://www.openvrml.org/

DSVideoLib
A DirectShow wrapper supporting concurrent access to framebuffers from multiple threads. Useful for developing applications that require live video input from a variety of capture devices (frame grabbers, IEEE-1394 DV camcorders, USB webcams).


openvrml on macports
http://trac.macports.org/browser/trunk/dports/graphics/openvrml/Portfile


galaxy:~ lym$ port search openvrml
openvrml @0.17.12 (graphics, x11)
    a cross-platform VRML and X3D browser and C++ runtime library
galaxy:~ lym$ port info openvrml
openvrml @0.17.12 (graphics, x11)
Variants:    js_mozilla, mozilla_plugin, no_opengl, no_x11, player, universal,
             xembed

OpenVRML is a free cross-platform runtime for VRML and X3D available under the
GNU Lesser General Public License. The OpenVRML distribution includes libraries
you can use to add VRML/X3D support to an application. On platforms where GTK+
is available, OpenVRML also provides a plug-in to render VRML/X3D worlds in Web
browsers.
Homepage:    http://www.openvrml.org/

Build Dependencies:   pkgconfig
Library Dependencies: boost, libpng, jpeg, fontconfig, mesa, libsdl
Platforms:            darwin
Maintainers:          raphael@ira.uka.de openmaintainer@macports.org
galaxy:~ lym$ port deps openvrml
openvrml has build dependencies on:
    pkgconfig
openvrml has library dependencies on:
    boost
    libpng
    jpeg
    fontconfig
    mesa
    libsdl
galaxy:~ lym$ port variants openvrml
openvrml has the variants:
    js_mozilla: Enable support for JavaScript in the Script node with Mozilla
    no_opengl: Do not build the GL renderer
    xembed: Build the XEmbed control
    player: Build the GNOME openvrml-player
    mozilla_plugin: Build the Mozilla plug-in
    no_x11: Disable support for X11
    universal: Build for multiple architectures


openvrml 설치



ARToolKit-2.72.1 설치 후 테스트

graphicsTest on the bin directory
-> This test confirms that your camera support ARToolKit graphics module with OpenGL.

videoTest on the bin directory
-> This test confirms that your camera supports ARToolKit video module and ARToolKit graphics module.

simpleTest on the bin directory
-> You need to notice that better the format is similar to ARToolKit tracking format, faster is the acquisition (RGB more efficient).


"hiro" 패턴을 쓰지 않으면, 아래와 같은 에러가 난다.

/Users/lym/ARToolKit/build/ARToolKit.build/Development/simpleTest.build/Objects-normal/i386/simpleTest ; exit;
galaxy:~ lym$ /Users/lym/ARToolKit/build/ARToolKit.build/Development/simpleTest.build/Objects-normal/i386/simpleTest ; exit;
Using default video config.
Opening sequence grabber 1 of 1.
vid->milliSecPerFrame: 200 forcing timer period to 100ms
Video cType is raw , size is 320x240.
Image size (x,y) = (320,240)
Camera parameter load error !!
logout


Using default video config.
Opening sequence grabber 1 of 1.
vid->milliSecPerFrame: 200 forcing timer period to 100ms
Video cType is raw , size is 320x240.
Image size (x,y) = (320,240)
*** Camera Parameter ***
--------------------------------------
SIZE = 320, 240
Distortion factor = 159.250000 131.750000 104.800000 1.012757
350.47574 0.00000 158.25000 0.00000
0.00000 363.04709 120.75000 0.00000
0.00000 0.00000 1.00000 0.00000
--------------------------------------
Opening Data File Data/object_data2
About to load 2 Models
Read in No.1
Read in No.2
Objectfile num = 2


arGetTransMat() 안에서 다음과 같이 pattern의 transformation 값을 출력해 보면,
    // http://www.hitl.washington.edu/artoolkit/documentation/tutorialcamera.htm
    printf("camera transformation: %f  %f  %f\n",conv[0][3],conv[1][3],conv[2][3]);

결과:


Feature List     
* A simple framework for creating real-time augmented reality applications    
* A multiplatform library (Windows, Linux, Mac OS X, SGI)    
* Overlays 3D virtual objects on real markers ( based on computer vision algorithm)    
* A multi platform video library with:          
o multiple input sources (USB, Firewire, capture card) supported          
o multiple format (RGB/YUV420P, YUV) supported          
o multiple camera tracking supported          
o GUI initializing interface    
* A fast and cheap 6D marker tracking (real-time planar detection)    
* An extensible markers patterns approach (number of markers fct of efficency)    
* An easy calibration routine    
* A simple graphic library (based on GLUT)    
* A fast rendering based on OpenGL    
* A 3D VRML support    
* A simple and modular API (in C)    
* Other language supported (JAVA, Matlab)    
* A complete set of samples and utilities    
* A good solution for tangible interaction metaphor    
* OpenSource with GPL license for non-commercial usage


framework



"ARToolKit is able to perform this camera tracking in real time, ensuring that the virtual objects always appear overlaid on the tracking markers."

how to
1. 매 비디오 프레임 마다 사각형 모양을 찾기
2. 검은색 사각형에 대한 카메라의 상대적 위치를 계산
3. 그 위치로부터 컴퓨터 그래픽 모델이 어떻게 그려질지를 계산
4. 실제 영상의 마커 위에 모델을 그림

limitations
1. 추적하는 마커가 영상 안에 보일 때에만 가상 물체를 합성할 수 있음
2. 이 때문에 가상 물체들의 크기나 이동이 제한됨
3. 마커의 패턴의 일부가 가려지는 경우 가상 물체를 합성할 수 없음
4. range(거리)의 제한: 마커의 모양이 클수록 멀리 떨어진 패턴까지 감지할 수 있으므로 추적할 수 있는 volume(범위)이 더 커짐
(이때 거리는  pattern complexity (패턴의 복잡도)에 따라 달라짐: 패턴이 단순할수록 한계 거리가 길어짐)
5. 추적 성능이 카메라에 대한 마커의 상대적인 orientation(방향)에 따라 달라짐
: 마커가 많이 기울어 수평에 가까워질수록 보이는 패턴의 부분이 줄어들기 때문에 recognition(인식)이 잘 되지 않음(신뢰도가 떨어짐)
6. 추적 성능이 lighting conditions (조명 상태)에 따라 달라짐
: 조명에 의해 종이 마커 위에 reflection and glare spots (반사)가 생기면 마커의 사각형을 찾기가 어려워짐
: 종이 대신 반사도가 적은 재료를 쓸 수 있음


ARToolKit Vision Algorithm



Development
Initialization    
1. Initialize the video capture and read in the marker pattern files and camera parameters. -> init()
Main Loop    
2. Grab a video input frame. -> arVideoGetImage()
3. Detect the markers and recognized patterns in the video input frame. -> arDetectMarker()
4. Calculate the camera transformation relative to the detected patterns. -> arGetTransMat)
5. Draw the virtual objects on the detected patterns. -> draw()
Shutdown    
6. Close the video capture down. -> cleanup()

ref.
http://king8028.tistory.com/entry/ARToolkit-simpletestc-%EC%84%A4%EB%AA%8512
http://kougaku-navi.net/ARToolKit.html



ARToolKit video configuration



camera calibration

Default camera properties are contained in the camera parameter file camera_para.dat, that is read in each time an application is started.

The program calib_dist is used to measure the image center point and lens distortion, while calib_param produces the other camera properties. (Both of these programs can be found in the bin directory and their source is in the utils/calib_dist and utils/calib_cparam directories.)



ARToolKit gives the position of the marker in the camera coordinate system, and uses OpenGL matrix system for the position of the virtual object.


ARToolKit API Documentation
http://artoolkit.sourceforge.net/apidoc/


ARMarkerInfo Main structure for detected marker
ARMarkerInfo2 Internal structure use for marker detection
ARMat Matrix structure
ARMultiEachMarkerInfoT Multi-marker structure
ARMultiMarkerInfoT Global multi-marker structure
ARParam Camera intrinsic parameters
arPrevInfo Structure for temporal continuity of tracking
ARVec Vector structure


arVideoGetImage()

video.h
/**
 * \brief get the video image.
 *
 * This function returns a buffer with a captured video image.
 * The returned data consists of a tightly-packed array of
 * pixels, beginning with the first component of the leftmost
 * pixel of the topmost row, and continuing with the remaining
 * components of that pixel, followed by the remaining pixels
 * in the topmost row, followed by the leftmost pixel of the
 * second row, and so on.
 * The arrangement of components of the pixels in the buffer is
 * determined by the configuration string passed in to the driver
 * at the time the video stream was opened. If no pixel format
 * was specified in the configuration string, then an operating-
 * system dependent default, defined in <AR/config.h> is used.
 * The memory occupied by the pixel data is owned by the video
 * driver and should not be freed by your program.
 * The pixels in the buffer remain valid until the next call to
 * arVideoCapNext, or the next call to arVideoGetImage which
 * returns a non-NULL pointer, or any call to arVideoCapStop or
 * arVideoClose.
 * \return A pointer to the pixel data of the captured video frame,
 * or NULL if no new pixel data was available at the time of calling.
 */
AR_DLL_API  ARUint8*        arVideoGetImage(void);


ARParam

param.h
/** \struct ARParam
* \brief camera intrinsic parameters.
*
* This structure contains the main parameters for
* the intrinsic parameters of the camera
* representation. The camera used is a pinhole
* camera with standard parameters. User should
* consult a computer vision reference for more
* information. (e.g. Three-Dimensional Computer Vision
* (Artificial Intelligence) by Olivier Faugeras).
* \param xsize length of the image (in pixels).
* \param ysize height of the image (in pixels).
* \param mat perspective matrix (K).
* \param dist_factor radial distortions factor
*          dist_factor[0]=x center of distortion
*          dist_factor[1]=y center of distortion
*          dist_factor[2]=distortion factor
*          dist_factor[3]=scale factor
*/
typedef struct {
    int      xsize, ysize;
    double   mat[3][4];
    double   dist_factor[4];
} ARParam;

typedef struct {
    int      xsize, ysize;
    double   matL[3][4];
    double   matR[3][4];
    double   matL2R[3][4];
    double   dist_factorL[4];
    double   dist_factorR[4];
} ARSParam;




arDetectMarker()

ar.h 헤더 파일의 설명:
/**
* \brief main function to detect the square markers in the video input frame.
*
* This function proceeds to thresholding, labeling, contour extraction and line corner estimation
* (and maintains an history).
* It's one of the main function of the detection routine with arGetTransMat.
* \param dataPtr a pointer to the color image which is to be searched for square markers.
*                The pixel format depend of your architecture. Generally ABGR, but the images
*                are treated as a gray scale, so the order of BGR components does not matter.
*                However the ordering of the alpha comp, A, is important.
* \param thresh  specifies the threshold value (between 0-255) to be used to convert
*                the input image into a binary image.
* \param marker_info a pointer to an array of ARMarkerInfo structures returned
*                    which contain all the information about the detected squares in the image
* \param marker_num the number of detected markers in the image.
* \return 0 when the function completes normally, -1 otherwise
*/
int arDetectMarker( ARUint8 *dataPtr, int thresh,
                    ARMarkerInfo **marker_info, int *marker_num );


You need to notice that arGetTransMat give the position of the marker in the camera coordinate system (not the reverse). If you want the position of the camera in the marker coordinate system you need to inverse this transformation (arMatrixInverse()).



XXXBK: not be sure of this function: this function must just convert 3x4 matrix to classical perspective openGL matrix. But in the code, you used arParamDecompMat that seem decomposed K and R,t, aren't it ? why do this decomposition since we want just intrinsic parameters ? and if not what is arDecomp ?




double arGetTransMat()

ar.h 헤더 파일의 설명:
/**
* \brief compute camera position in function of detected markers.
*
* calculate the transformation between a detected marker and the real camera,
* i.e. the position and orientation of the camera relative to the tracking mark.
* \param marker_info the structure containing the parameters for the marker for
*                    which the camera position and orientation is to be found relative to.
*                    This structure is found using arDetectMarker.
* \param center the physical center of the marker. arGetTransMat assumes that the marker
*              is in x-y plane, and z axis is pointing downwards from marker plane.
*              So vertex positions can be represented in 2D coordinates by ignoring the
*              z axis information. The marker vertices are specified in order of clockwise.
* \param width the size of the marker (in mm).
* \param conv the transformation matrix from the marker coordinates to camera coordinate frame,
*             that is the relative position of real camera to the real marker
* \return always 0.
*/
double arGetTransMat( ARMarkerInfo *marker_info,
                      double center[2], double width, double conv[3][4] )



arUtilMatInv()

ar.h 헤더 파일의 설명:
/**
* \brief Inverse a non-square matrix.
*
* Inverse a matrix in a non homogeneous format. The matrix
* need to be euclidian.
* \param s matrix input   
* \param d resulted inverse matrix.
* \return 0 if the inversion success, -1 otherwise
* \remark input matrix can be also output matrix
*/
int    arUtilMatInv( double s[3][4], double d[3][4] );






posted by maetel
2010. 2. 10. 15:47 Computer Vision
Seong-Woo Park, Yongduek Seo, Ki-Sang Hong: Real-Time Camera Calibration for Virtual Studio. Real-Time Imaging 6(6): 433-448 (2000)
doi:10.1006/rtim.1999.0199

Seong-Woo Park, Yongduek Seo and Ki-Sang Hong1

Dept. of E.E. POSTECH, San 31, Hyojadong, Namku, Pohang, Kyungbuk, 790-784, Korea


Abstract

In this paper, we present an overall algorithm for real-time camera parameter extraction, which is one of the key elements in implementing virtual studio, and we also present a new method for calculating the lens distortion parameter in real time. In a virtual studio, the motion of a virtual camera generating a graphic studio must follow the motion of the real camera in order to generate a realistic video product. This requires the calculation of camera parameters in real-time by analyzing the positions of feature points in the input video. Towards this goal, we first design a special calibration pattern utilizing the concept of cross-ratio, which makes it easy to extract and identify feature points, so that we can calculate the camera parameters from the visible portion of the pattern in real-time. It is important to consider the lens distortion when zoom lenses are used because it causes nonnegligible errors in the computation of the camera parameters. However, the Tsai algorithm, adopted for camera calibration, calculates the lens distortion through nonlinear optimization in triple parameter space, which is inappropriate for our real-time system. Thus, we propose a new linear method by calculating the lens distortion parameter independently, which can be computed fast enough for our real-time application. We implement the whole algorithm using a Pentium PC and Matrox Genesis boards with five processing nodes in order to obtain the processing rate of 30 frames per second, which is the minimum requirement for TV broadcasting. Experimental results show this system can be used practically for realizing a virtual studio.


전자공학회논문지 제36권 S편 제7호, 1999. 7 
가상스튜디오 구현을 위한 실시간 카메라 추적 ( Real-Time Camera Tracking for Virtual Studio )   
박성우 · 서용덕 · 홍기상 저 pp. 90~103 (14 pages)
http://uci.or.kr/G300-j12265837.v36n07p90

서지링크     한국과학기술정보연구원
가상스튜디오의 구현을 위해서 카메라의 움직임을 실시간으로 알아내는 것이 필수적이다. 기존의 가상스튜디어 구현에 사용되는 기계적인 방법을 이용한 카메라의 움직임 추적하는 방법에서 나타나는 단점들을 해결하기 위해 본 논문에서는 카메라로부터 얻어진 영상을 이용해 컴퓨터비전 기술을 응용하여 실시간으로 카메라변수들을 알아내기 위한 전체적인 알고리듬을 제안하고 실제 구현을 위한 시스템의 구성 방법에 대해 다룬다. 본 연구에서는 실시간 카메라변수 추출을 위해 영상에서 특징점을 자동으로 추출하고 인식하기 위한 방법과, 카메라 캘리브레이션 과정에서 렌즈의 왜곡특성 계산에 따른 계산량 문제를 해결하기 위한 방법을 제안한다.



Practical ways to calculate camera lens distortion for real-time camera calibration
Pattern Recognition, Volume 34, Issue 6, June 2001, Pages 1199-1206
Seong-Woo Park, Ki-Sang Hong




generating virtual studio




Matrox Genesis boards
http://www.matrox.com/imaging/en/support/legacy/

http://en.wikipedia.org/wiki/Virtual_studio
http://en.wikipedia.org/wiki/Chroma_key

camera tracking system : electromechanical / optical
pattern recognition
2D-3D pattern matches
planar pattern


feature extraction -> image-model matching & identification -> camera calibration
: to design the pattern by applying the concept of cross-ratio and to identify the pattern automatically


영상에서 찾아진 특징점을 자동으로 인식하기 위해서는 공간 상의 점들과 영상에 나타난 그것들의 대응점에 대해서 같은 값을 갖는 성질이 필요한데 이것을 기하적 불변량 (Geometric Invariant)이라고 한다. 본 연구에서는 여러 불변량 가운데 cross-ratio를 이용하여 패턴을 제작하고, 영상에서 불변량의 성질을 이용하여 패턴을 자동으로 찾고 인식할 수 있게 하는 방법을 제안한다.


Tsai's algorithm
R. Y. Tsai, A Versatile Camera Calibration Technique for High Accuracy 3-D Maching Vision Metrology Using Off-the-shelf TV Cameras and Lenses. IEEE Journal of Robotics & Automation 3 (1987), pp. 323–344.

direct image mosaic method
Sawhney, H. S. and Kumar, R. 1999. True Multi-Image Alignment and Its Application to Mosaicing and Lens Distortion Correction. IEEE Trans. Pattern Anal. Mach. Intell. 21, 3 (Mar. 1999), 235-243. DOI= http://dx.doi.org/10.1109/34.754589

Lens distortion
Richard Szeliski, Computer Vision: Algorithms and Applications: 2.1.6 Lens distortions & 6.3.5 Radial distortion

radial alignment constraint
"If we presume that the lens has only radial distortion, the direction of a distorted point is the same as the direction of an undistorted point."

cross-ratio  http://en.wikipedia.org/wiki/Cross_ratio
: planar projective geometric invariance
 - "pencil of lines"
http://mathworld.wolfram.com/CrossRatio.html
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/MOHR_TRIGGS/node25.html
http://www.cut-the-knot.org/pythagoras/Cross-Ratio.shtml
http://web.science.mq.edu.au/~chris/geometry/


pattern identification

 카메라의 움직임을 알아내기 위해서는 공간상에 인식이 가능한 물체가 있어야 한다. 즉, 어느 위치에서 보더라도 영상에 나타난 특징점을 찾을 수 있고, 공간상의 어느 점에 대응되는 점인지를 알 수 있어야 한다.

패턴이 인식 가능하기 위해서는 카메라가 어느 위치, 어느 자세로 보던지 항상 같은 값을 갖는 기하적 불변량 (Geometric Invariant)이 필요하다.

Coelho, C., Heller, A., Mundy, J. L., Forsyth, D. A., and Zisserman, A.1992. An experimental evaluation of projective invariants. In Geometric invariance in Computer Vision, J. L. Mundy and A. Zisserman, Eds. Mit Press Series Of Artificial Intelligence Series. MIT Press, Cambridge, MA, 87-104.


> initial identification process
extracting the pattern in an image: chromakeying -> gradient filtering: a first-order derivative of Gaussian (DoG) -> line fitting: deriving a distorted line (that is actually a curve) equation -> feature point tracking (using intersection filter)


R1x = 0



http://en.wikipedia.org/wiki/Difference_of_Gaussians



real-time camera parameter extraction

이상적인 렌즈의 optical axis가 영상면에 수직이고 변하지 않는다고 할 때, 영상 중심은 카메라의 줌 동작 동안 고정된 값으로 계산된다. (그러나 실제 렌즈의 불완전한 특성 때문에 카메라의 줌 동작 동안 영상 중심 역시 변하게 되는데, 이 변화량은 적용 범위 이내에서 2픽셀 이하이다. 따라서 본 연구에서는 이러한 변화를 무시하고 이상적인 렌즈를 가정하여 줌동작에 의한 영상 중심을 구하게 된다.)

For zoom lenses, the image centers vary as the camera zooms because the zooming operation is executed by a composite combination of several lenses. However, when we examined the location of the image centers, its standard deviation was about 2 pixels; thus we ignored the effect of the image center change.


calculating lens distortion coefficient

Zoom lenses are zoomed by a complicated combination of several lenses so that the effective focal length and distortion coefficient vary during zooming operations.

When using the coplanar pattern with small depth variation, it turns out that focal length and z-translation cannot be separated exactly and reliably even with small noise.

카메라 변수 추출에 있어서 공간상의 특징점들이 모두 하나의 평면상에 존재할 때는 초점거리와 z 방향으로의 이동이 상호 연관 (coupling)되어 계산값의 안정성이 결여되기 쉽다.


collinearity

Collinearity represents a property when the line in the world coordinate is also shown as a line in the image. This property is not preserved when the lens has a distortion.


Once the lens distortion is calculated, we can execute camera calibration using linear methods.


filtering

가상 스튜디오 구현에 있어서는 시간 지연이 항상 같은 값을 가지게 하는 것이 필수적이므로, 실제 적용에서는 예측 (prediction)이 들어가는 필터링 방법(예를 들면, Kalman filter)은 사용할 수가 없었다.

averaging filter 평균 필터








Orad  http://www.orad.co.il

Evans & Sutherland http://www.es.com









posted by maetel