Alina Karakanta

ALT A poster titled "Evaluating automatic subtitling: Correlating Post-editing effor and automatic metrics". The main research questions is how well automatic machine translation evaluation metrics correlate with temporal and technical post-editing effort in automatic subtitling. The contributions are two. First, we release a subtitle post-editing corpus. Three subtitlers (one for English into German and two for English into Italian) post-edited 9 talks in Matesub. We collected product, process and participant-based data. This means the automatic and post-edited versions of the subtitles, keystrokes and the time spent on editing each subtitle, and user experience ratings. Second, we performed a correlation analysis with automatic subtitling and MT metrics at the level of subtitles and the level of talks. We found that edit-distance metrics such as SubER and HTER correlate best with temporal effort at the subtitle level, even though correlations are such a small granularity are moderate.