Foundation Models for Natural Language Processing
This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts. Over the recent years, a revolutionary new paradigm has been developed for tra...
I tiakina i:
| Ngā kaituhi matua: | , |
|---|---|
| Hōputu: | Online |
| Reo: | Ingarihi |
| I whakaputaina: |
Springer Nature
2023
|
| Ngā marau: | |
| Urunga tuihono: | ONIX_20230620_9783031231902_10 |
| Ngā Tūtohu: |
Kāore He Tūtohu, Me noho koe te mea tuatahi ki te tūtohu i tēnei pūkete!
|
| _version_ | 1869520437781725184 |
|---|---|
| author | Paaß, Gerhard Giesselbach, Sven |
| author_browse | Giesselbach, Sven Paaß, Gerhard |
| author_facet | Paaß, Gerhard Giesselbach, Sven |
| author_sort | Paaß, Gerhard |
| collection | Directory of Open Access Books |
| description | This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts. Over the recent years, a revolutionary new paradigm has been developed for training models for NLP. These models are first pre-trained on large collections of text documents to acquire general syntactic knowledge and semantic information. Then, they are fine-tuned for specific tasks, which they can often solve with superhuman accuracy. When the models are large enough, they can be instructed by prompts to solve new tasks without any fine-tuning. Moreover, they can be applied to a wide range of different media and problem domains, ranging from image and video processing to robot control learning. Because they provide a blueprint for solving many tasks in artificial intelligence, they have been called Foundation Models. After a brief introduction to basic NLP models the main pre-trained language models BERT, GPT and sequence-to-sequence transformer are described, as well as the concepts of self-attention and context-sensitive embedding. Then, different approaches to improving these models are discussed, such as expanding the pre-training criteria, increasing the length of input texts, or including extra knowledge. An overview of the best-performing models for about twenty application areas is then presented, e.g., question answering, translation, story generation, dialog systems, generating images from text, etc. For each application area, the strengths and weaknesses of current models are discussed, and an outlook on further developments is given. In addition, links are provided to freely available program code. A concluding chapter summarizes the economic opportunities, mitigation of risks, and potential developments of AI. |
| format | Online |
| id | doab-20.500.12854ir-107926 |
| institution | Directory of Open Access Books |
| language | eng |
| publishDate | 2023 |
| publishDateRange | 2023 |
| publishDateSort | 2023 |
| publisher | Springer Nature |
| publisherStr | Springer Nature |
| record_format | ojs |
| spelling | doab-20.500.12854ir-1079262025-07-17T10:01:22Z Foundation Models for Natural Language Processing Paaß, Gerhard Giesselbach, Sven Pre-trained Language Models Deep Learning Natural Language Processing Transformer Models BERT GPT Attention Models Natural Language Understanding Multilingual Models Natural Language Generation Chatbot Foundation Models Information Extraction Text Generation thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQL Natural language and machine translation thema EDItEUR::C Language and Linguistics::CF Linguistics::CFX Computational and corpus linguistics thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQE Expert systems / knowledge-based systems thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQM Machine learning thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQL Natural language and machine translation thema EDItEUR::C Language and Linguistics::CF Linguistics::CFX Computational and corpus linguistics thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQE Expert systems / knowledge-based systems thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQM Machine learning This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts. Over the recent years, a revolutionary new paradigm has been developed for training models for NLP. These models are first pre-trained on large collections of text documents to acquire general syntactic knowledge and semantic information. Then, they are fine-tuned for specific tasks, which they can often solve with superhuman accuracy. When the models are large enough, they can be instructed by prompts to solve new tasks without any fine-tuning. Moreover, they can be applied to a wide range of different media and problem domains, ranging from image and video processing to robot control learning. Because they provide a blueprint for solving many tasks in artificial intelligence, they have been called Foundation Models. After a brief introduction to basic NLP models the main pre-trained language models BERT, GPT and sequence-to-sequence transformer are described, as well as the concepts of self-attention and context-sensitive embedding. Then, different approaches to improving these models are discussed, such as expanding the pre-training criteria, increasing the length of input texts, or including extra knowledge. An overview of the best-performing models for about twenty application areas is then presented, e.g., question answering, translation, story generation, dialog systems, generating images from text, etc. For each application area, the strengths and weaknesses of current models are discussed, and an outlook on further developments is given. In addition, links are provided to freely available program code. A concluding chapter summarizes the economic opportunities, mitigation of risks, and potential developments of AI. 2023-07-26T21:45:54Z 2023-07-26T21:45:54Z 2023-06-20T10:23:35Z 2023 book ONIX_20230620_9783031231902_10 https://library.oapen.org/handle/20.500.12657/63548 9783031231902 9783031231896 https://directory.doabooks.org/handle/20.500.12854/107926 eng Artificial Intelligence: Foundations, Theory, and Algorithms open access image/jpeg image/jpeg n/a n/a https://library.oapen.org/bitstream/20.500.12657/63548/1/978-3-031-23190-2.pdf https://library.oapen.org/bitstream/20.500.12657/63548/1/978-3-031-23190-2.pdf Springer Nature Springer International Publishing 10.1007/978-3-031-23190-2 10.1007/978-3-031-23190-2 9fa3421d-f917-4153-b9ab-fc337c396b5a 66282fa2-c4b9-4457-9645-79730d2e7aeb 9783031231902 9783031231896 Springer International Publishing 436 Cham [...] open access |
| spellingShingle | Pre-trained Language Models Deep Learning Natural Language Processing Transformer Models BERT GPT Attention Models Natural Language Understanding Multilingual Models Natural Language Generation Chatbot Foundation Models Information Extraction Text Generation thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQL Natural language and machine translation thema EDItEUR::C Language and Linguistics::CF Linguistics::CFX Computational and corpus linguistics thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQE Expert systems / knowledge-based systems thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQM Machine learning thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQL Natural language and machine translation thema EDItEUR::C Language and Linguistics::CF Linguistics::CFX Computational and corpus linguistics thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQE Expert systems / knowledge-based systems thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQM Machine learning Paaß, Gerhard Giesselbach, Sven Foundation Models for Natural Language Processing |
| title | Foundation Models for Natural Language Processing |
| title_full | Foundation Models for Natural Language Processing |
| title_fullStr | Foundation Models for Natural Language Processing |
| title_full_unstemmed | Foundation Models for Natural Language Processing |
| title_short | Foundation Models for Natural Language Processing |
| title_sort | foundation models for natural language processing |
| topic | Pre-trained Language Models Deep Learning Natural Language Processing Transformer Models BERT GPT Attention Models Natural Language Understanding Multilingual Models Natural Language Generation Chatbot Foundation Models Information Extraction Text Generation thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQL Natural language and machine translation thema EDItEUR::C Language and Linguistics::CF Linguistics::CFX Computational and corpus linguistics thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQE Expert systems / knowledge-based systems thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQM Machine learning thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQL Natural language and machine translation thema EDItEUR::C Language and Linguistics::CF Linguistics::CFX Computational and corpus linguistics thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQE Expert systems / knowledge-based systems thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQM Machine learning |
| topic_facet | Pre-trained Language Models Deep Learning Natural Language Processing Transformer Models BERT GPT Attention Models Natural Language Understanding Multilingual Models Natural Language Generation Chatbot Foundation Models Information Extraction Text Generation thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQL Natural language and machine translation thema EDItEUR::C Language and Linguistics::CF Linguistics::CFX Computational and corpus linguistics thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQE Expert systems / knowledge-based systems thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQM Machine learning thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQL Natural language and machine translation thema EDItEUR::C Language and Linguistics::CF Linguistics::CFX Computational and corpus linguistics thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQE Expert systems / knowledge-based systems thema EDItEUR::U Computing and Information Technology::UY Computer science::UYQ Artificial intelligence::UYQM Machine learning |
| url | ONIX_20230620_9783031231902_10 |
| work_keys_str_mv | AT paaßgerhard foundationmodelsfornaturallanguageprocessing AT giesselbachsven foundationmodelsfornaturallanguageprocessing |