Utilize este identificador para referenciar este registo:
https://hdl.handle.net/1822/88997
Registo completo
Campo DC | Valor | Idioma |
---|---|---|
dc.contributor.author | Adão, Telmo | por |
dc.contributor.author | Oliveira, José A. | por |
dc.contributor.author | Shahrabadi, Somayeh | por |
dc.contributor.author | Jesus, Hugo | por |
dc.contributor.author | Fernandes, Marco | por |
dc.contributor.author | Costa, Ângelo | por |
dc.contributor.author | Ferreira, Vânia | por |
dc.contributor.author | Gonçalves, Martinho Fradeira | por |
dc.contributor.author | Lopéz, Miguel A.Guevara | por |
dc.contributor.author | Peres, Emanuel | por |
dc.contributor.author | Magalhães, Luís Gonzaga Mendes | por |
dc.date.accessioned | 2024-02-22T16:58:14Z | - |
dc.date.available | 2024-02-22T16:58:14Z | - |
dc.date.issued | 2023-11-01 | - |
dc.identifier.uri | https://hdl.handle.net/1822/88997 | - |
dc.description.abstract | Communication between Deaf and hearing individuals remains a persistent challenge requiring attention to foster inclusivity. Despite notable efforts in the development of digital solutions for sign language recognition (SLR), several issues persist, such as cross-platform interoperability and strategies for tokenizing signs to enable continuous conversations and coherent sentence construction. To address such issues, this paper proposes a non-invasive Portuguese Sign Language (Língua Gestual Portuguesa or LGP) interpretation system-as-a-service, leveraging skeletal posture sequence inference powered by long-short term memory (LSTM) architectures. To address the scarcity of examples during machine learning (ML) model training, dataset augmentation strategies are explored. Additionally, a buffer-based interaction technique is introduced to facilitate LGP terms tokenization. This technique provides real-time feedback to users, allowing them to gauge the time remaining to complete a sign, which aids in the construction of grammatically coherent sentences based on inferred terms/words. To support human-like conditioning rules for interpretation, a large language model (LLM) service is integrated. Experiments reveal that LSTM-based neural networks, trained with 50 LGP terms and subjected to data augmentation, achieved accuracy levels ranging from 80% to 95.6%. Users unanimously reported a high level of intuition when using the buffer-based interaction strategy for terms/words tokenization. Furthermore, tests with an LLM—specifically ChatGPT—demonstrated promising semantic correlation rates in generated sentences, comparable to expected sentences. | por |
dc.description.sponsorship | FCT -Fundação para a Ciência e a Tecnologia(C644866286-00000011) | por |
dc.language.iso | eng | por |
dc.publisher | MDPI | por |
dc.relation | POCI-01-0247-FEDER-068605 | por |
dc.relation | C644866286-00000011 | por |
dc.relation | info:eu-repo/grantAgreement/FCT/6817 - DCRRNI ID/UIDB%2F04033%2F2020/PT | por |
dc.relation | LA/P/0126/2020 | por |
dc.rights | openAccess | por |
dc.subject | Deaf-hearing communication | por |
dc.subject | Generative pre-trained transformer (GPT) | por |
dc.subject | Inclusion | por |
dc.subject | Large language models (LLM) | por |
dc.subject | Long-short term memory (LSTM) | por |
dc.subject | Machine learning (ML) | por |
dc.subject | Portuguese sign language | por |
dc.subject | Sign language recognition (SLR) | por |
dc.subject | Video-based motion analytics | por |
dc.title | Empowering deaf-hearing communication: Exploring synergies between predictive and generative AI-based strategies towards (Portuguese) sign language interpretation | por |
dc.type | article | por |
dc.peerreviewed | yes | por |
oaire.citationIssue | 11 | por |
oaire.citationVolume | 9 | por |
dc.date.updated | 2024-02-09T12:03:22Z | - |
dc.identifier.doi | 10.3390/jimaging9110235 | por |
sdum.export.identifier | 13211 | - |
sdum.journal | Journal of Imaging | por |
Aparece nas coleções: | CAlg - Artigos em revistas internacionais / Papers in international journals |
Ficheiros deste registo:
Ficheiro | Descrição | Tamanho | Formato | |
---|---|---|---|---|
jimaging-09-00235-v2 (1).pdf | 7,02 MB | Adobe PDF | Ver/Abrir |