TY - GEN
T1 - An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation
AU - Moryossef, Amit
AU - Müller, Mathias
AU - Göhring, Anne
AU - Jiang, Zifan
AU - Goldberg, Yoav
AU - Ebling, Sarah
N1 - Publisher Copyright:
© 2023 2023 VFS Handling Qualities Technical Meeting. All rights reserved.
PY - 2023/1/1
Y1 - 2023/1/1
N2 - Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.
AB - Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.
UR - https://www.scopus.com/pages/publications/85174784958
M3 - Conference contribution
AN - SCOPUS:85174784958
T3 - AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages
SP - 25
EP - 36
BT - AT4SSL 2023 - Proceedings of the 2nd International Workshop on Automatic Translation for Signed and Spoken Languages
A2 - Shterionov, Dimitar
A2 - De Sisto, Mirella
A2 - Muller, Mathias
A2 - Van Landuyt, Davy
A2 - Omardeen, Rehana
A2 - O'boyle, Shaun
A2 - Braffort, Annelies
A2 - Roelofsen, Floris
A2 - Blain, Frederic
A2 - Vanroy, Bram
A2 - Avramidis, Eleftherios
PB - European Association for Machine Translation
T2 - 2nd International Workshop on Automatic Translation for Signed and Spoken Languages, AT4SSL 2023
Y2 - 15 June 2023
ER -