Our research will start by investigating a new hierarchical, multi-level approach for dynamic fusion of multiple sensing modalities, specifically based on wearables and static IoT sensors of the smart home environment. Each intermediate level of our hierarchical fusion scheme will comprise synthetic features that will be formulated either (a) through the probabilistic, weighted fusion of the lower-level features, or (b) by transforming the essence of lower level features into higher-level semantic representations, in respect of the classification problem per se; i.e. the target activity to be understood, based on the monitoring context. In this scope, we will extend state of art approaches of sensor abstraction –based multimodal fusion, by focusing specifically on key challenges of applying corresponding HAR systems in realistic settings, i.e. resilience to sensor noise and opportunistic changes in the involvement of different sensors (e.g. a wearable gets disconnected), as well as on adaptive context dependent modality fusion towards more effective, detailed activity recognition and understanding. In this scope, we will extend state-of-art approaches of in-place oriented adaptive fusion4 so as to embrace further contextual cues than user position, such as temporal relationships between the measurements and inferred environment and user states, including an approach for modality boosting respective to its potential of further augmenting the effectiveness of observing activity details, given inferred increased probabilities for a specific activity to have been established. At the higher end, features of the different levels will be fed to the classification scheme; both simple and complex classifiers will be examined, while key to our scope will be Dynamic Bayesian Network (DBN) -based models and deep learning oriented, Autoregressive Networks.
The aim here will be to develop a novel goal-oriented, sensing ensemble coordination framework, capable to steer the efforts of the integrated system to enhance both (a) the effectiveness of activity recognition and (b) the level of detail and robustness in behaviour understanding. In essence, this will act as a second fusion layer, extending the fusion framework of Objective 1, which deals with wearables and static IoT sensors, to one that also fuses data derived from dynamic, active sensing modalities seamlessly coordinated though our proposed approach, i.e. in our specific case, a mobile robot vision module. Our research to this end will start by defining how decisions shall be made from the coordination module during runtime on (a) whether the dynamic modalities, i.e. the robot camera, must be engaged in the user monitoring task and (b) how it should be engaged. These decisions will be based on the processing of multi-level features of the wearables/static sensors (Obj. 1), and specifically on the semantic representation of their values into inference on three core factors: (a) whether a user activity of interest may have started based on IoT-based sensing, (b) whether, given the monitoring capabilities of the IoT-based system part, the detailed monitoring of this plausible activity calls for dynamic vision-based monitoring to understand key specificities and (c) whether the robot navigation and positioning at a place that allows effective robot vision –based monitoring, would be welcome by the user, trying to avoid cases of no socially-acceptable robot behaviour. Key starting point to the research towards the decision-making mechanisms involved in this cognitive fusion part of our approach will be Partially Observable Markov Decision Processes (POMDP)-based models. The outcome of robot-vision based monitoring will be fed to the multimodal fusion-based framework of Obj. 1.
The proposed methods for human activity recognition and behaviour understanding through the fusion of (a) wearables and IoT data and (b) wearables, IoT and mobile robot vision, will be systematically evaluated in the course of the project. This will be done through multi-level evaluation, starting from performance evaluation of individual fusion components of the first-level fusion approach, gradually moving to the evaluation of the active sensing fusion and finally the overall, integrated approach. By employing different classifiers on top of the proposed fusion methods, we will compare performance on both activity recognition and detection of key activity specificities for behaviour analysis, against a set of state-of-art methods, employing either wearables/IoT/robot modalities separately and in combination. Evaluation will be performed both on (a) publicly available datasets and (b) datasets that will be collected in the premises of the CERTH-ITI smart home. The latter evaluation will be performed in parallel to a further key project aim, that of establishing a proof-of-concept integrated wearables/IoT/robot system, capable to operate based on our proposed methods within a real smart home. In this respect, evaluation will also involve, further to HAR performance, also usability and user acceptance of the overall system. The proof-of-concept system will be realized based on the CERTH-ITI smarthome infrastructure and the RAMCIP robot of CERTH-ITI.