Hierarchical Semantic Processing Architecture for Smart Sensors in Surveillance Networks

II Abstract - Data acquisition by multi-domain data acquisition provides means for environment perception usable for detecting unusual and possibly dangerous situations. When being automated, this approach can simplify surveillance tasks required in, for example, airports or other security sensitive infrastructures. This paper describes a novel architecture for surveillance networks based on combining multimodal sensor information. Compared to previous methodologies using only video information, the proposed approach also uses audio data thus increasing its ability to obtain valuable information about the sensed environment. A hierarchical processing architecture for observation and surveillance systems is proposed, which reco-gnizes a set of pre-defined behaviors and learns about normal behaviors. Deviations from "normality" are reported in a way understandable even for staff without special training. The processing architecture, including the physical sensor nodes, is called SENSE (smart embedded network of sensing entities). Parts of this work have been published previously; the main enhancements of this paper compared to previous publications are detailed descriptions of the layers 1 and 4, "pre-processing including plausibility checks" and "parameter inference". In the other layers, details not necessary for a general understanding of the approach have been omitted.