블로그 이미지
Leeway is... the freedom that someone has to take the action they want to or to change their plans.
maetel

Notice

Recent Post

Recent Comment

Recent Trackback

Archive

calendar

1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31
  • total
  • today
  • yesterday

Category

2010. 9. 18. 11:58 Footmarks

4th Open Lab


September 14, 2010 | Written by admin

마침내 오픈랩이 다시 돌아왔습니다. 벌써 4번째 행사네요.
꼭 방문하셔서 제니텀의 최근 작업들을 경험하면서 즐거운시간 가지시길 바랍니다.

Zenitum’s Open Lab is back! Our Open Lab 4 will showcase our latest work in the field of augmented reality, including 3D reconstruction, and various techniques for recognizing and tracking images.


전시내용:


1. 영상기반의 모바일 증강현실 트래킹 엔진 & GPS기반 모바일 증강현실 트래킹 엔진

http://youtu.be/OvLTOWoze0A
http://youtu.be/YcgebgYeU5M
http://youtu.be/ibWnY9ZXKzk
http://youtu.be/7jUaxlS52tU
http://youtu.be/O-myIJboPn0


2. 4Cast: Full 3D 재구성 시스템

http://youtu.be/LByly6rlZMg
http://youtu.be/577gv_xeWPU
http://youtu.be/xL8YSgdQEXM

- 원하시는 분은 자신의 Full 3D 재구성 모델을 만들어 드립니다.


3. Media Art Project: iWall

- 3D 질감을 표현하는 대형 액티브 미디어 월과 iPhone과의 만남

- 기존(작년의 프로토타입)의 Active Media Wall 프로젝트 동영상은

http://youtu.be/wLlAfTa2lVg

posted by maetel
2010. 5. 30. 01:59

보호되어 있는 글입니다.
내용을 보시려면 비밀번호를 입력하세요.

2010. 3. 3. 19:54 Computer Vision
http://www.hitl.washington.edu/artoolkit/

ARToolKit Patternmaker
Automatically create large numbers of target patterns for the ARToolKit, by the University of Utah.


ARToolKit-2.72.tgz 다운로드

http://www.openvrml.org/

DSVideoLib
A DirectShow wrapper supporting concurrent access to framebuffers from multiple threads. Useful for developing applications that require live video input from a variety of capture devices (frame grabbers, IEEE-1394 DV camcorders, USB webcams).


openvrml on macports
http://trac.macports.org/browser/trunk/dports/graphics/openvrml/Portfile


galaxy:~ lym$ port search openvrml
openvrml @0.17.12 (graphics, x11)
    a cross-platform VRML and X3D browser and C++ runtime library
galaxy:~ lym$ port info openvrml
openvrml @0.17.12 (graphics, x11)
Variants:    js_mozilla, mozilla_plugin, no_opengl, no_x11, player, universal,
             xembed

OpenVRML is a free cross-platform runtime for VRML and X3D available under the
GNU Lesser General Public License. The OpenVRML distribution includes libraries
you can use to add VRML/X3D support to an application. On platforms where GTK+
is available, OpenVRML also provides a plug-in to render VRML/X3D worlds in Web
browsers.
Homepage:    http://www.openvrml.org/

Build Dependencies:   pkgconfig
Library Dependencies: boost, libpng, jpeg, fontconfig, mesa, libsdl
Platforms:            darwin
Maintainers:          raphael@ira.uka.de openmaintainer@macports.org
galaxy:~ lym$ port deps openvrml
openvrml has build dependencies on:
    pkgconfig
openvrml has library dependencies on:
    boost
    libpng
    jpeg
    fontconfig
    mesa
    libsdl
galaxy:~ lym$ port variants openvrml
openvrml has the variants:
    js_mozilla: Enable support for JavaScript in the Script node with Mozilla
    no_opengl: Do not build the GL renderer
    xembed: Build the XEmbed control
    player: Build the GNOME openvrml-player
    mozilla_plugin: Build the Mozilla plug-in
    no_x11: Disable support for X11
    universal: Build for multiple architectures


openvrml 설치



ARToolKit-2.72.1 설치 후 테스트

graphicsTest on the bin directory
-> This test confirms that your camera support ARToolKit graphics module with OpenGL.

videoTest on the bin directory
-> This test confirms that your camera supports ARToolKit video module and ARToolKit graphics module.

simpleTest on the bin directory
-> You need to notice that better the format is similar to ARToolKit tracking format, faster is the acquisition (RGB more efficient).


"hiro" 패턴을 쓰지 않으면, 아래와 같은 에러가 난다.

/Users/lym/ARToolKit/build/ARToolKit.build/Development/simpleTest.build/Objects-normal/i386/simpleTest ; exit;
galaxy:~ lym$ /Users/lym/ARToolKit/build/ARToolKit.build/Development/simpleTest.build/Objects-normal/i386/simpleTest ; exit;
Using default video config.
Opening sequence grabber 1 of 1.
vid->milliSecPerFrame: 200 forcing timer period to 100ms
Video cType is raw , size is 320x240.
Image size (x,y) = (320,240)
Camera parameter load error !!
logout


Using default video config.
Opening sequence grabber 1 of 1.
vid->milliSecPerFrame: 200 forcing timer period to 100ms
Video cType is raw , size is 320x240.
Image size (x,y) = (320,240)
*** Camera Parameter ***
--------------------------------------
SIZE = 320, 240
Distortion factor = 159.250000 131.750000 104.800000 1.012757
350.47574 0.00000 158.25000 0.00000
0.00000 363.04709 120.75000 0.00000
0.00000 0.00000 1.00000 0.00000
--------------------------------------
Opening Data File Data/object_data2
About to load 2 Models
Read in No.1
Read in No.2
Objectfile num = 2


arGetTransMat() 안에서 다음과 같이 pattern의 transformation 값을 출력해 보면,
    // http://www.hitl.washington.edu/artoolkit/documentation/tutorialcamera.htm
    printf("camera transformation: %f  %f  %f\n",conv[0][3],conv[1][3],conv[2][3]);

결과:


Feature List     
* A simple framework for creating real-time augmented reality applications    
* A multiplatform library (Windows, Linux, Mac OS X, SGI)    
* Overlays 3D virtual objects on real markers ( based on computer vision algorithm)    
* A multi platform video library with:          
o multiple input sources (USB, Firewire, capture card) supported          
o multiple format (RGB/YUV420P, YUV) supported          
o multiple camera tracking supported          
o GUI initializing interface    
* A fast and cheap 6D marker tracking (real-time planar detection)    
* An extensible markers patterns approach (number of markers fct of efficency)    
* An easy calibration routine    
* A simple graphic library (based on GLUT)    
* A fast rendering based on OpenGL    
* A 3D VRML support    
* A simple and modular API (in C)    
* Other language supported (JAVA, Matlab)    
* A complete set of samples and utilities    
* A good solution for tangible interaction metaphor    
* OpenSource with GPL license for non-commercial usage


framework



"ARToolKit is able to perform this camera tracking in real time, ensuring that the virtual objects always appear overlaid on the tracking markers."

how to
1. 매 비디오 프레임 마다 사각형 모양을 찾기
2. 검은색 사각형에 대한 카메라의 상대적 위치를 계산
3. 그 위치로부터 컴퓨터 그래픽 모델이 어떻게 그려질지를 계산
4. 실제 영상의 마커 위에 모델을 그림

limitations
1. 추적하는 마커가 영상 안에 보일 때에만 가상 물체를 합성할 수 있음
2. 이 때문에 가상 물체들의 크기나 이동이 제한됨
3. 마커의 패턴의 일부가 가려지는 경우 가상 물체를 합성할 수 없음
4. range(거리)의 제한: 마커의 모양이 클수록 멀리 떨어진 패턴까지 감지할 수 있으므로 추적할 수 있는 volume(범위)이 더 커짐
(이때 거리는  pattern complexity (패턴의 복잡도)에 따라 달라짐: 패턴이 단순할수록 한계 거리가 길어짐)
5. 추적 성능이 카메라에 대한 마커의 상대적인 orientation(방향)에 따라 달라짐
: 마커가 많이 기울어 수평에 가까워질수록 보이는 패턴의 부분이 줄어들기 때문에 recognition(인식)이 잘 되지 않음(신뢰도가 떨어짐)
6. 추적 성능이 lighting conditions (조명 상태)에 따라 달라짐
: 조명에 의해 종이 마커 위에 reflection and glare spots (반사)가 생기면 마커의 사각형을 찾기가 어려워짐
: 종이 대신 반사도가 적은 재료를 쓸 수 있음


ARToolKit Vision Algorithm



Development
Initialization    
1. Initialize the video capture and read in the marker pattern files and camera parameters. -> init()
Main Loop    
2. Grab a video input frame. -> arVideoGetImage()
3. Detect the markers and recognized patterns in the video input frame. -> arDetectMarker()
4. Calculate the camera transformation relative to the detected patterns. -> arGetTransMat)
5. Draw the virtual objects on the detected patterns. -> draw()
Shutdown    
6. Close the video capture down. -> cleanup()

ref.
http://king8028.tistory.com/entry/ARToolkit-simpletestc-%EC%84%A4%EB%AA%8512
http://kougaku-navi.net/ARToolKit.html



ARToolKit video configuration



camera calibration

Default camera properties are contained in the camera parameter file camera_para.dat, that is read in each time an application is started.

The program calib_dist is used to measure the image center point and lens distortion, while calib_param produces the other camera properties. (Both of these programs can be found in the bin directory and their source is in the utils/calib_dist and utils/calib_cparam directories.)



ARToolKit gives the position of the marker in the camera coordinate system, and uses OpenGL matrix system for the position of the virtual object.


ARToolKit API Documentation
http://artoolkit.sourceforge.net/apidoc/


ARMarkerInfo Main structure for detected marker
ARMarkerInfo2 Internal structure use for marker detection
ARMat Matrix structure
ARMultiEachMarkerInfoT Multi-marker structure
ARMultiMarkerInfoT Global multi-marker structure
ARParam Camera intrinsic parameters
arPrevInfo Structure for temporal continuity of tracking
ARVec Vector structure


arVideoGetImage()

video.h
/**
 * \brief get the video image.
 *
 * This function returns a buffer with a captured video image.
 * The returned data consists of a tightly-packed array of
 * pixels, beginning with the first component of the leftmost
 * pixel of the topmost row, and continuing with the remaining
 * components of that pixel, followed by the remaining pixels
 * in the topmost row, followed by the leftmost pixel of the
 * second row, and so on.
 * The arrangement of components of the pixels in the buffer is
 * determined by the configuration string passed in to the driver
 * at the time the video stream was opened. If no pixel format
 * was specified in the configuration string, then an operating-
 * system dependent default, defined in <AR/config.h> is used.
 * The memory occupied by the pixel data is owned by the video
 * driver and should not be freed by your program.
 * The pixels in the buffer remain valid until the next call to
 * arVideoCapNext, or the next call to arVideoGetImage which
 * returns a non-NULL pointer, or any call to arVideoCapStop or
 * arVideoClose.
 * \return A pointer to the pixel data of the captured video frame,
 * or NULL if no new pixel data was available at the time of calling.
 */
AR_DLL_API  ARUint8*        arVideoGetImage(void);


ARParam

param.h
/** \struct ARParam
* \brief camera intrinsic parameters.
*
* This structure contains the main parameters for
* the intrinsic parameters of the camera
* representation. The camera used is a pinhole
* camera with standard parameters. User should
* consult a computer vision reference for more
* information. (e.g. Three-Dimensional Computer Vision
* (Artificial Intelligence) by Olivier Faugeras).
* \param xsize length of the image (in pixels).
* \param ysize height of the image (in pixels).
* \param mat perspective matrix (K).
* \param dist_factor radial distortions factor
*          dist_factor[0]=x center of distortion
*          dist_factor[1]=y center of distortion
*          dist_factor[2]=distortion factor
*          dist_factor[3]=scale factor
*/
typedef struct {
    int      xsize, ysize;
    double   mat[3][4];
    double   dist_factor[4];
} ARParam;

typedef struct {
    int      xsize, ysize;
    double   matL[3][4];
    double   matR[3][4];
    double   matL2R[3][4];
    double   dist_factorL[4];
    double   dist_factorR[4];
} ARSParam;




arDetectMarker()

ar.h 헤더 파일의 설명:
/**
* \brief main function to detect the square markers in the video input frame.
*
* This function proceeds to thresholding, labeling, contour extraction and line corner estimation
* (and maintains an history).
* It's one of the main function of the detection routine with arGetTransMat.
* \param dataPtr a pointer to the color image which is to be searched for square markers.
*                The pixel format depend of your architecture. Generally ABGR, but the images
*                are treated as a gray scale, so the order of BGR components does not matter.
*                However the ordering of the alpha comp, A, is important.
* \param thresh  specifies the threshold value (between 0-255) to be used to convert
*                the input image into a binary image.
* \param marker_info a pointer to an array of ARMarkerInfo structures returned
*                    which contain all the information about the detected squares in the image
* \param marker_num the number of detected markers in the image.
* \return 0 when the function completes normally, -1 otherwise
*/
int arDetectMarker( ARUint8 *dataPtr, int thresh,
                    ARMarkerInfo **marker_info, int *marker_num );


You need to notice that arGetTransMat give the position of the marker in the camera coordinate system (not the reverse). If you want the position of the camera in the marker coordinate system you need to inverse this transformation (arMatrixInverse()).



XXXBK: not be sure of this function: this function must just convert 3x4 matrix to classical perspective openGL matrix. But in the code, you used arParamDecompMat that seem decomposed K and R,t, aren't it ? why do this decomposition since we want just intrinsic parameters ? and if not what is arDecomp ?




double arGetTransMat()

ar.h 헤더 파일의 설명:
/**
* \brief compute camera position in function of detected markers.
*
* calculate the transformation between a detected marker and the real camera,
* i.e. the position and orientation of the camera relative to the tracking mark.
* \param marker_info the structure containing the parameters for the marker for
*                    which the camera position and orientation is to be found relative to.
*                    This structure is found using arDetectMarker.
* \param center the physical center of the marker. arGetTransMat assumes that the marker
*              is in x-y plane, and z axis is pointing downwards from marker plane.
*              So vertex positions can be represented in 2D coordinates by ignoring the
*              z axis information. The marker vertices are specified in order of clockwise.
* \param width the size of the marker (in mm).
* \param conv the transformation matrix from the marker coordinates to camera coordinate frame,
*             that is the relative position of real camera to the real marker
* \return always 0.
*/
double arGetTransMat( ARMarkerInfo *marker_info,
                      double center[2], double width, double conv[3][4] )



arUtilMatInv()

ar.h 헤더 파일의 설명:
/**
* \brief Inverse a non-square matrix.
*
* Inverse a matrix in a non homogeneous format. The matrix
* need to be euclidian.
* \param s matrix input   
* \param d resulted inverse matrix.
* \return 0 if the inversion success, -1 otherwise
* \remark input matrix can be also output matrix
*/
int    arUtilMatInv( double s[3][4], double d[3][4] );






posted by maetel
2010. 2. 23. 00:47 Computer Vision
1> pattern identification 패턴 인식

rough preview
1) 무늬의 deep/light 색의 경계점들 찾기 edge detection
2) 찾은 점들을 직선으로 연결
3) 검출된 가로선과 세로선의 cross ratio와 실제 무늬의 cross ratio를 비교하여, 몇 번째 선인지 인식

detailed preview
1. initial identification process 초기 인식 과정 (특징점 인식)

1) chroma keying:  RGB -> YUV 변환

2) gradient filtering: first-order derivative Gaussian filter (length = 7)
 -1) 세로축에 대해 영상 축소 (1/4)하여 필터링
 -2) Gx, Gy 절대값 비교하여 vertical / horizontal direction 판별
 -3) 가로축에 대해

3) line fitting: lens distortion coefficient을 고려하여 이차곡선으로 피팅

4) identification
 -1) 영상에서 찾아진 선들이 실제 무늬에서 몇 번째 선인지 인식
 -2) feature points는 직선 식에 의해 피팅된 선들의 교점으로 정확하게 구할 수 있음

2. feature point tracking 실제 동작 과정 (특징점 위치 추적)
: feature points corresponding 검출된 특징점을 무늬의 교점과 매칭

  1) intersection filter H (교점 필터)로 local maximum & minimum를 가지는 교점 검출

  2) 검출된 교점의 부호를 판별하여 두 부류로 나눔

  3) 이전 프레임에서의 교점의 위치를 기준으로 현재 프레임에서 검출된 교점에 대해 가장 가까운 이전 점을 찾음

  * 다음 프레임에서 새로 나타난 특징점에 대해서도 이전 프레임에서의 카메라 변수를 이용해 실제 패턴 상의 교점을 영상으로 투영시켜 기준점으로 삼을 수 있음




2> real-time camera parameter extraction 실시간 카메라 변수 추출: Tsai's algorithm

1. determining image center 영상 중심 구하기: zooming
: using the center of expansion as a constant image-center

1) (lens distortion을 구하기 위한 초기화 과정에서) 정지된 카메라의 maximum zoom-out과 maximum zoom-in 상태에서 찾아서 인식한 특징점들을 저장

2) 두 개의 프레임에서 같은 점으로 나타난 특징점들을 연결한 line segments의 common intersection 교점을 계산

* 실제로 zooming은 여러 개의 lens들의 조합으로 작동하기 때문에 카메라의 zoom에 따라서 image center가 변하게 되지만, 이에 대한 표준 편차가 작으므로 무시하기로 함

2. lens distortion coefficient 계산
zooming이 없다면 고정된 값이 되므로 이하와 같이 매번 계산해 줄 필요가 없어짐

(1) f-k1 look-up table을 참조하는 방법
: zooming하는 과정에서 초점 거리 f와 렌즈 왜곡 변수 k1이 계속 변하게 되므로, 이에 대한 참조표를 미리 만들어 두고 나서 실제 동작 과정에서 참조
* 특징점들이 모두 하나의 평면에 존재하는 경우에는 초점거리 f와 카메라의 z 방향으로의 이동 Tz가 서로 coupled되기 때문에 카메라 변수가 제대로 계산되기 어렵다는 점을 고려하여 평면 상의 특징점들에 대해서 Tz/f를 인덱스로 사용하는 편법을 쓴다면, 카메라가 z 방향으로는 이동하지 않고 고정되어 있어야 한다는 (T1z = 0)조건이 붙게 됨

(2) collinearity를 이용하는 방법
: searching for k1 which maximally preserves collinearity 인식된 교점들에 대해 원래 하나의 직선에 속하는 점들이 왜곡 보상 되었을 때 가장 직선이 되게 하는 왜곡변수를 구함

  1) 영상에서 같은 가로선에 속하는 교점들 (Xf, Yf) 가운데 세 개를 고름

  2) 식7로부터 왜곡된 영상면 좌표 (Xd, Yd)를 구함
 
  3) 식5로부터 왜곡 보상된 영상면 좌표 (Xu, Yu)를 구함

  4) 식21과 같은 에러 함수 E(k1)를 정의

  5) 영상에 나타난 N개의 가로선들에 대해서 E(k1) 값을 최소화하는 k1을 구함 (식 23) -> 비선형 최적화이나 iteration은 한 번
 
3. Tsai's algorithm
렌즈 왜곡 변수를 알면 카메라 캘리브레이션은 선형적 방법으로 구할 수 있게 됨




3> filtering
잡음으로 인해 검출된 교점에 오차가 생기므로 카메라변수가 틀려지게 됨
(->카메라가 정지해 있어도 카메라변수에 변화가 생겨 결과적으로 그래픽으로 생성된 가상의 무대에 떨림이 나타나게 됨)

averaging filter 평균 필터 (전자공학회논문지 제36권 S편 제7호 식19)









posted by maetel
2010. 2. 10. 15:47 Computer Vision
Seong-Woo Park, Yongduek Seo, Ki-Sang Hong: Real-Time Camera Calibration for Virtual Studio. Real-Time Imaging 6(6): 433-448 (2000)
doi:10.1006/rtim.1999.0199

Seong-Woo Park, Yongduek Seo and Ki-Sang Hong1

Dept. of E.E. POSTECH, San 31, Hyojadong, Namku, Pohang, Kyungbuk, 790-784, Korea


Abstract

In this paper, we present an overall algorithm for real-time camera parameter extraction, which is one of the key elements in implementing virtual studio, and we also present a new method for calculating the lens distortion parameter in real time. In a virtual studio, the motion of a virtual camera generating a graphic studio must follow the motion of the real camera in order to generate a realistic video product. This requires the calculation of camera parameters in real-time by analyzing the positions of feature points in the input video. Towards this goal, we first design a special calibration pattern utilizing the concept of cross-ratio, which makes it easy to extract and identify feature points, so that we can calculate the camera parameters from the visible portion of the pattern in real-time. It is important to consider the lens distortion when zoom lenses are used because it causes nonnegligible errors in the computation of the camera parameters. However, the Tsai algorithm, adopted for camera calibration, calculates the lens distortion through nonlinear optimization in triple parameter space, which is inappropriate for our real-time system. Thus, we propose a new linear method by calculating the lens distortion parameter independently, which can be computed fast enough for our real-time application. We implement the whole algorithm using a Pentium PC and Matrox Genesis boards with five processing nodes in order to obtain the processing rate of 30 frames per second, which is the minimum requirement for TV broadcasting. Experimental results show this system can be used practically for realizing a virtual studio.


전자공학회논문지 제36권 S편 제7호, 1999. 7 
가상스튜디오 구현을 위한 실시간 카메라 추적 ( Real-Time Camera Tracking for Virtual Studio )   
박성우 · 서용덕 · 홍기상 저 pp. 90~103 (14 pages)
http://uci.or.kr/G300-j12265837.v36n07p90

서지링크     한국과학기술정보연구원
가상스튜디오의 구현을 위해서 카메라의 움직임을 실시간으로 알아내는 것이 필수적이다. 기존의 가상스튜디어 구현에 사용되는 기계적인 방법을 이용한 카메라의 움직임 추적하는 방법에서 나타나는 단점들을 해결하기 위해 본 논문에서는 카메라로부터 얻어진 영상을 이용해 컴퓨터비전 기술을 응용하여 실시간으로 카메라변수들을 알아내기 위한 전체적인 알고리듬을 제안하고 실제 구현을 위한 시스템의 구성 방법에 대해 다룬다. 본 연구에서는 실시간 카메라변수 추출을 위해 영상에서 특징점을 자동으로 추출하고 인식하기 위한 방법과, 카메라 캘리브레이션 과정에서 렌즈의 왜곡특성 계산에 따른 계산량 문제를 해결하기 위한 방법을 제안한다.



Practical ways to calculate camera lens distortion for real-time camera calibration
Pattern Recognition, Volume 34, Issue 6, June 2001, Pages 1199-1206
Seong-Woo Park, Ki-Sang Hong




generating virtual studio




Matrox Genesis boards
http://www.matrox.com/imaging/en/support/legacy/

http://en.wikipedia.org/wiki/Virtual_studio
http://en.wikipedia.org/wiki/Chroma_key

camera tracking system : electromechanical / optical
pattern recognition
2D-3D pattern matches
planar pattern


feature extraction -> image-model matching & identification -> camera calibration
: to design the pattern by applying the concept of cross-ratio and to identify the pattern automatically


영상에서 찾아진 특징점을 자동으로 인식하기 위해서는 공간 상의 점들과 영상에 나타난 그것들의 대응점에 대해서 같은 값을 갖는 성질이 필요한데 이것을 기하적 불변량 (Geometric Invariant)이라고 한다. 본 연구에서는 여러 불변량 가운데 cross-ratio를 이용하여 패턴을 제작하고, 영상에서 불변량의 성질을 이용하여 패턴을 자동으로 찾고 인식할 수 있게 하는 방법을 제안한다.


Tsai's algorithm
R. Y. Tsai, A Versatile Camera Calibration Technique for High Accuracy 3-D Maching Vision Metrology Using Off-the-shelf TV Cameras and Lenses. IEEE Journal of Robotics & Automation 3 (1987), pp. 323–344.

direct image mosaic method
Sawhney, H. S. and Kumar, R. 1999. True Multi-Image Alignment and Its Application to Mosaicing and Lens Distortion Correction. IEEE Trans. Pattern Anal. Mach. Intell. 21, 3 (Mar. 1999), 235-243. DOI= http://dx.doi.org/10.1109/34.754589

Lens distortion
Richard Szeliski, Computer Vision: Algorithms and Applications: 2.1.6 Lens distortions & 6.3.5 Radial distortion

radial alignment constraint
"If we presume that the lens has only radial distortion, the direction of a distorted point is the same as the direction of an undistorted point."

cross-ratio  http://en.wikipedia.org/wiki/Cross_ratio
: planar projective geometric invariance
 - "pencil of lines"
http://mathworld.wolfram.com/CrossRatio.html
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/MOHR_TRIGGS/node25.html
http://www.cut-the-knot.org/pythagoras/Cross-Ratio.shtml
http://web.science.mq.edu.au/~chris/geometry/


pattern identification

 카메라의 움직임을 알아내기 위해서는 공간상에 인식이 가능한 물체가 있어야 한다. 즉, 어느 위치에서 보더라도 영상에 나타난 특징점을 찾을 수 있고, 공간상의 어느 점에 대응되는 점인지를 알 수 있어야 한다.

패턴이 인식 가능하기 위해서는 카메라가 어느 위치, 어느 자세로 보던지 항상 같은 값을 갖는 기하적 불변량 (Geometric Invariant)이 필요하다.

Coelho, C., Heller, A., Mundy, J. L., Forsyth, D. A., and Zisserman, A.1992. An experimental evaluation of projective invariants. In Geometric invariance in Computer Vision, J. L. Mundy and A. Zisserman, Eds. Mit Press Series Of Artificial Intelligence Series. MIT Press, Cambridge, MA, 87-104.


> initial identification process
extracting the pattern in an image: chromakeying -> gradient filtering: a first-order derivative of Gaussian (DoG) -> line fitting: deriving a distorted line (that is actually a curve) equation -> feature point tracking (using intersection filter)


R1x = 0



http://en.wikipedia.org/wiki/Difference_of_Gaussians



real-time camera parameter extraction

이상적인 렌즈의 optical axis가 영상면에 수직이고 변하지 않는다고 할 때, 영상 중심은 카메라의 줌 동작 동안 고정된 값으로 계산된다. (그러나 실제 렌즈의 불완전한 특성 때문에 카메라의 줌 동작 동안 영상 중심 역시 변하게 되는데, 이 변화량은 적용 범위 이내에서 2픽셀 이하이다. 따라서 본 연구에서는 이러한 변화를 무시하고 이상적인 렌즈를 가정하여 줌동작에 의한 영상 중심을 구하게 된다.)

For zoom lenses, the image centers vary as the camera zooms because the zooming operation is executed by a composite combination of several lenses. However, when we examined the location of the image centers, its standard deviation was about 2 pixels; thus we ignored the effect of the image center change.


calculating lens distortion coefficient

Zoom lenses are zoomed by a complicated combination of several lenses so that the effective focal length and distortion coefficient vary during zooming operations.

When using the coplanar pattern with small depth variation, it turns out that focal length and z-translation cannot be separated exactly and reliably even with small noise.

카메라 변수 추출에 있어서 공간상의 특징점들이 모두 하나의 평면상에 존재할 때는 초점거리와 z 방향으로의 이동이 상호 연관 (coupling)되어 계산값의 안정성이 결여되기 쉽다.


collinearity

Collinearity represents a property when the line in the world coordinate is also shown as a line in the image. This property is not preserved when the lens has a distortion.


Once the lens distortion is calculated, we can execute camera calibration using linear methods.


filtering

가상 스튜디오 구현에 있어서는 시간 지연이 항상 같은 값을 가지게 하는 것이 필수적이므로, 실제 적용에서는 예측 (prediction)이 들어가는 필터링 방법(예를 들면, Kalman filter)은 사용할 수가 없었다.

averaging filter 평균 필터








Orad  http://www.orad.co.il

Evans & Sutherland http://www.es.com









posted by maetel
2009. 11. 8. 16:31 Computer Vision
Branislav Kisačanin & Vladimir Pavlović & Thomas S. Huang
Real-Time Vision for Human-Computer Interaction
(RTV4HCI)
Springer, 2005
(google book's overview)

2004 IEEE CVPR Workshop on RTV4HCI - Papers
http://rtv4hci.rutgers.edu/04/


Computer vision and pattern recognition continue to play a dominant role in the HCI realm. However, computer vision methods often fail to become pervasive in the field due to the lack of real-time, robust algorithms, and novel and convincing applications.

Keywords:
head and face modeling
map building
pervasive computing
real-time detection

Contents:
RTV4HCI: A Historical Overview.
- Real-Time Algorithms: From Signal Processing to Computer Vision.
- Recognition of Isolated Fingerspelling Gestures Using Depth Edges.
- Appearance-Based Real-Time Understanding of Gestures Using Projected Euler Angles.
- Flocks of Features for Tracking Articulated Objects.
- Static Hand Posture Recognition Based on Okapi-Chamfer Matching.
- Visual Modeling of Dynamic Gestures Using 3D Appearance and Motion Features.
- Head and Facial Animation Tracking Using Appearance-Adaptive Models and Particle Filters.
- A Real-Time Vision Interface Based on Gaze Detection -- EyeKeys.
- Map Building from Human-Computer Interactions.
- Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures.
- Epipolar Constrained User Pushbutton Selection in Projected Interfaces.
- Vision-Based HCI Applications.
- The Office of the Past.
- MPEG-4 Face and Body Animation Coding Applied to HCI.
- Multimodal Human-Computer Interaction.
- Smart Camera Systems Technology Roadmap.
- Index.




RTV4HCI: A Historical Overview
Matthew Turk (mturk@cs.ucsb.edu)
University of California, Santa Barbara
http://www.stanford.edu/~mturk/
http://www.cs.ucsb.edu/~mturk/

The goal of research in real-time vision for human-computer interaction is to develop algorithms and systems that sense and perceive humans and human activity, in order to enable more natural, powerful, and effective computer interfaces.

Computers in the Human Interaction Loop (CHIL)

perceptual interfaces
multimodal interfaces
post-WIMP(windows, icons, menus, pointer) interfaces

implicit user awareness or explicit user control

The user interface
- the software and devices that implement a particular model (or set of models) of HCI

Computer vision technologies must ultimately deliver a better "user experience".

B Shneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Third Edition, Addison-Wesley, 1998.
: 1) time to learn 2) speed of performance 3) user error rates 4) retention over time 5) subjective satisfaction

- Presence and location (Face and body detection, head and body tracking)
- Identity (Face recognition, gait recognition)
- Expression (Facial feature tracking, expression modeling and analysis)
- Focus of attention (Head/face tracking, eye gaze tracking)
- Body posture and movement (Body modeling and tracking)
- Gesture (Gesture recognition, hand tracking)
- Activity (Analysis of body movement)

eg.
VIDEOPLACE (M W Krueger, Artificial Reality II, Addison-Wesley, 1991)
Magic Morphin Mirror / Mass Hallucinations (T Darrell et al., SIGGRAPH Visual Proc, 1997)

Principal Component Analysis (PCA)
Linear Discriminant Analysis (LDA)
Gabor Wavelet Networks (GWNs)
Active Appearance Models (AAMs)
Hidden Markov Models (HMMs)

Identix Inc.
Viisage Technology Inc.
Cognitec Systems


- MIT Medial Lab
ALIVE system (P Maes et al., The ALIVE system: wireless, full-body interaction with autonomous agents, ACM Multimedia Systems, 1996)
PFinder system (C R Wren et al., Pfinder: Real-time tracking of the human body, IEEE Trans PAMI, pp 780-785, 1997)
KidsRoom project (A Bobick et al., The KidsRoom: A perceptually-based interactive and immersive story environment, PRESENCE: Teleoperators and Virtual Environments, pp 367-391, 1999)




Flocks of Features for Tracking Articulated Objects
Mathias Kolsch (kolsch@nps.edu
Computer Science Department, Naval Postgraduate School, Monterey
Matthew Turk (mturk@cs.ucsb.edu)
Computer Science Department, University of California, Santa Barbara




Visual Modeling of Dynamic Gestures Using 3D Appearance and Motion Features
Guangqi Ye (grant@cs.jhu.edu), Jason J. Corso, Gregory D. Hager
Computational Interaction and Robotics Laboratory
The Johns Hopkins University



Map Building from Human-Computer Interactions
http://groups.csail.mit.edu/lbr/mars/pubs/pubs.html#publications
Artur M. Arsenio (arsenio@csail.mit.edu)
Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology



Vision-Based HCI Applications
Eric Petajan (eric@f2f-inc.com)
face2face animation, inc.
eric@f2f-inc.com



The Office of the Past
Jiwon Kim (jwkim@cs.washington.edu), Steven M. Seitz (seitz@cs.washington.edu)
University of Washington
Maneesh Agrawala (maneesh@microsoft.com)
Microsoft Research
Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 10 - Volume 10  Page: 157   Year of Publication: 2004
http://desktop.google.com
http://grail.cs.washington.edu/projects/office/
http://www.realvnc.com/



Smart Camera Systems Technology Roadmap
Bruce Flinchbaugh (b-flinchbaugh@ti.com)
Texas Instruments

posted by maetel
2009. 7. 23. 18:53 Computer Vision
Brian Williams, Georg Klein and Ian Reid
(Department of Engineering Science, University of Oxford, UK)
Real-Time SLAM Relocalisation
In Proceedings of the International Conference on Computer Vision, Rio de Janeiro, Brazil, 2007
demo 1
demo 2


• real-time, high-accuracy localisation and mapping during tracking
• real-time (re-)localisation when when tracking fails
• on-line learning of image patch appearance so that no prior training or map structure is required and features are added and removed during operation.


Lepetit's image patch classifier (feature appearance learning)
=> integrating the classifier more closely into the process of map-building
(by using classification results to aid in the selection of new points to add to the map)


> recovery from tracking failure: local vs. global
local -  particle filter -> rich feature descriptor
global - proximity using previous key frames


- based on SceneLib (Extended Kalman Filter)
- rotational (and a degree of perspective) invariance via local patch warping
- assuming the patch is fronto-parallel when first seen
http://freshmeat.net/projects/scenelib/

active search

innovation covariance

joint compatibility test

randomized lists key-point recognition algorithm
1. randomized: (2^D  - 1) tests -> D tests
2. independent treatment of classes
3. binary leaf scores (2^D * C * N bits for all scores)
4. intensity offset
5. explicit noise handing

training the classifier

The RANSAC (Random Sample Consensus) Algorithm




ref.
Davison, A. J. and Molton, N. D. 2007.
MonoSLAM: Real-Time Single Camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 29, 6 (Jun. 2007), 1052-1067. DOI= http://dx.doi.org/10.1109/TPAMI.2007.1049

Vision-based global localization and mapping for mobile robots
Se, S.   Lowe, D.G.   Little, J.J.   (MD Robotics, Brampton, Ont., Canada)

Lepetit, V. 2006.
Keypoint Recognition Using Randomized Trees. IEEE Trans. Pattern Anal. Mach. Intell. 28, 9 (Sep. 2006), 1465-1479. DOI= http://dx.doi.org/10.1109/TPAMI.2006.188

Lepetit, V., Lagger, P., and Fua, P. 2005.
Randomized Trees for Real-Time Keypoint Recognition. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cvpr'05) - Volume 2 - Volume 02 (June 20 - 26, 2005). CVPR. IEEE Computer Society, Washington, DC, 775-781. DOI= http://dx.doi.org/10.1109/CVPR.2005.288

Fischler, M. A. and Bolles, R. C. 1981.
Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (Jun. 1981), 381-395. DOI= http://doi.acm.org/10.1145/358669.358692
posted by maetel
2009. 3. 31. 21:10 Computer Vision

Real-time simultaneous localisation and mapping with a single camera

Davison, A.J.  
Dept. of Eng. Sci., Oxford Univ., UK;

This paper appears in: Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on
Publication Date: 13-16 Oct. 2003
On page(s): 1403-1410 vol.2
ISBN: 0-7695-1950-4
INSPEC Accession Number: 7971070
Digital Object Identifier: 10.1109/ICCV.2003.1238654
Current Version Published: 2008-04-03


 

posted by maetel
2009. 3. 26. 19:56 Computer Vision

Inverse Depth Parametrization for Monocular SLAM
Civera, J.   Davison, A.J.   Montiel, J. 


This paper appears in: Robotics, IEEE Transactions on
Publication Date: Oct. 2008
Volume: 24,  Issue: 5
On page(s): 932-945
ISSN: 1552-3098
INSPEC Accession Number: 10301459
Digital Object Identifier: 10.1109/TRO.2008.2003276
First Published: 2008-10-03
Current Version Published: 2008-10-31

Javier Civera, Departamento de Informática e Ingeniería de Sistemas, Universidad de Zaragoza

Andrew J. Davison, Reader in Robot Vision at the Department of Computing, Imperial College London

Jose Maria Martinez Montiel, Robotics and Real Time Group, Universidad de Zaragoza




monocular simultaneous localization and mapping  (SLAM)

representation of uncertainty

the standard extended Kalman filter (EKF)

direct parametrization of the inverse depth of features

feature initialization

camera motion estimates

6-D state vector --> converted to the Euclidean XYZ form

linearity index => automatic detection and conversion to maintain maximum efficiency



I. Introduction


monocular camera
: projective sensor measuring the beairng of image features

monocular (adj) 단안(單眼)(용)의, 외눈의

A stereo camera is a type of camera with two or more lenses. This allows the camera to simulate human binocular vision.

structure from motion = SFM
1) feature matching
2) global camera location & scene feature position estimates

sliding window processing

Sliding Window Protocol is a bi-directional data transmission protocol used in the data link layer (OSI model) as well as in TCP (transport layer of the OSI model). It is used to keep a record of the frame sequences sent and their respective acknowledgements received by both the users.

In robotics and computer vision, visual odometry is the process of determining the position and orientation of a robot by analyzing the associated camera images.

Odometry is the use of data from the movement of actuators to estimate change in position over time. Odometry is used by some robots, whether they be legged or wheeled, to estimate (not determine) their position relative to a starting location.

visual SLAM

probabilistic filtering approach

initializing uncertain depth estimates for distance features

Gaussian distributions implicit in the EKF

a new feature parametrization that is able to smoothly cope with initialization of features at all depth - even up to "infinity" - within the standard EKF framework: direct parametrization of inverse depth relative to the camera position from which a feature was first observed


A. Delayed and Undelayed Initialization

main map; main probabilistic state; main state vector

test for inclusion

delayed initialization
> treating newly detected features separately from the main map to reduce depth uncertainty before insertion into the full filter (with a standard XYZ representation)
- Features that retain low parallax over many frames (those very far from the camera or close to the motion epipole) are usually rejected completely because they never pass the test for inclusion
> (in 2-D and simulation) Initialization is delayed until the measurement equation is approximately Gaussian and the point can be safely triangulated.
> 3-D monocular vision with inertial sensing + auxiliary particle filter (in high frame rate sequence)

undelayed initialization
> While features with highly uncertain depths provide little information on camera translation, they are extremely useful as bearing references for orientation estimation.
: a multiple hypothesis scheme, initializing features at various depths and pruning those not reobserved in subsequent images
> Gaussian sum filter approximated by a federated information sharing method to keep the computational overhead low
-> to spread the Gaussian depth hypotheses along the ray according to inverse depth

Davision's particle method --> (Sola et al.) Gaussian sum filter --> (Civera at al.) new inverse depth scheme

 

A Gaussian sum is more efficient representation than particles (efficient enough that the separate Gaussians can call be put into the main state vector), but not as efficient as the single Gaussian representation that the inverse depth parametrization aalows.



B. Points at Infinity

efficient undelayed initialization + features at all depths (in outdoor scenes)


Point at infinity: a feature that exhibits no parallax during camera motion due to its extreme depth
-> not used for estimating camera translationm but for estimating rotation

The homogeneous coordinate systems of visual projective geometry used normally in SFM allow explicit representation of points at infinity(, and they have proven to play an important role during offline structure and motion estimation).

sequential SLAM system

Montiel and Davison: In special case where all features are known to be infinite -- in very-large-scale outdoor scenes or when the camera rotates on a tripod -- SLAM in pure angular coordinates turns the camera into a real-time visual compass.


Our probabilistic SLAM algorithm must be able to represent the uncertainty in depth of seemingly infinite features. Observing no parallax for a feature after 10 units of camera translation does tell us something about its depth -- it gives a reliable lower bound, which depends on the amount of motion made by the camera (if the feature had been closer than this, we would have observed parallax).

The explicit consideration of uncertainty in the locations of points has not been previously required in offline computer vision algorithms, but is very important in a more difficult online case.



C. Inverse Depth Representation

There is a unified and straightforward parametrization for feature locations that can handle both initialization and standard tracking of both close and very distant features within the standard EKF framework.


standard tracking

An explicit parametrization of the inverse depth of a feature along a semiinfinite ray from the position from which it was first viewed allows a Gaussian distribution to cover uncertainty in depth that spans a depth range from nearby to infinity, and permits seamless crossing over to finite depth estimates of features that have been apparently infinite for long periods of time.

linearity index + inverse depth parametrization

The projective nature of a camera means that the image measurement process is nearly linear in this inverse depth coordinate.


Inverse depth appears in the relation between image disparity and point depth in a stereo vision; it is interpreted as the parallax with respect to the plane at infinity. (Hartley and Zisserman)

Inverse depth is used to relate the motion field induced by scene points with the camera velocity in optical flow analysis. 

modified polar coordinates

target motion analysis = TMA

EKF-based sequential depth estimation from camera-known motion

multibaseline stereo

matching robustness for scene symmetries

sequential EKF process using inverse depth
( ref. Stochastic Approximation and Rate-Distortion Analysis for Robust Structure and Motion Estimation )

undelayed initialization for 2-D monocular SLAM 
( ref. A unified framework for nearby and distant landmarks in bearing-only SLAM )

FastSLAM-based system for monocular SLAM
( ref. Ethan Eade &  Tom Drummond,  Scalable Monocular SLAM )

special epipolar update step

FastSLAM

( ref. Civera, J.   Davison, A.J.   Montiel, J.M.M., Inverse Depth to Depth Conversion for Monocular SLAM 
J. Montiel and A. J. Davison “A visual compass based on SLAM,” )

loop-closing



II. State Vector Definition


handheld camera motion
> constant angular and linear velocity model

quaternion








posted by maetel