Top In The Lab, Flop In The Field? Evaluation Of A Sensor-based Travel Activity Classifier With The SHL Dataset

We present a solution to the Sussex-Huawei Locomotion- Transportation (SHL) recognition challenge (team "S304"). Our experiments reveal two potential pitfalls in the evalu- ation of activity recognition algorithms: 1) unnoticed over- fitting due to autocorrelation (i.e. dependencies between temporally close samples), and 2) the accuracy/generality trade-off due to idealized conditions and lack of variation in the data. We show that evaluation with a random train- ing/test split suggests highly accurate recognition of eight different travel activities with an average F1 score of 96% for single-participant/fixed-position data, whereas with proper backtesting the F1 score drops to 84%, for data of different participants in the SHL Dataset to 61%, and for different carrying positions to 54%. Our experiments demonstrate that results achieved `in-the-lab´ can easily become sub- ject to an upward bias and cannot always serve as reliable indicators for the future performance `in-the-field´, where generality and robustness are essential.