Sports Information Retrieval for Video Annotation makes manual annotation and indexing no more practical. The urgent requirements for multimedia applications therefore motivate the researches in various aspects of video analysis for developing automatic systems and tools for content-based multimedia information retrieval. Recently, sports video is attracting considerable attention due to potential commercial benefits and entertaining functionalities. There has been lots of research directed toward automatic indexing and summarization of broadcast sports video. As the pace of life in the information society accelerates, most viewers desire to retrieve the significant events or designated scenes and players, rather than watch a whole game in a sequential way. Various algorithms of shot classification and highlight extraction in sports video have been developed based on the combination of low-level visual/auditory features and game-specific rules. Some research efforts focus on ball/player track- In this paper, we propose to review current research and take baseball for example to give an insight into sports video analysis. The rest of this paper is organized as follows. Section 2 reviews the related work on sports video analysis. Section 3 elaborates the baseball video analysis work with applications (Chen et. al., 2006b, 2008a). Section 4 gives the further research directions and finally, Section 5 concludes this paper. RELATED WORK ON SPORT VIDEO ANALYSIS The contents of sports video are well-structured since the broadcasters present the game process in similar ways due to the game rules. Therefore, many domain-specific features and knowledge can be employed and incorporated into sports video analysis. The possible applications have