LaMDA: Language Models for Dialog Applications
LaMDA: Language Models for Dialog Applications
We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency.
Marcelo Menegali、Blaise Aguera-Arcas、Ray Kurzweil、Yanping Huang、Yuanzhong Xu、Joe Fenton、Ben Hutchinson、Erin Hoffman-John、Alicia Jin、Hongrae Lee、Kathleen Meier-Hellstern、Jamie Hall、Zhifeng Chen、Renelito Delos Santos、Amin Ghafouri、Pranesh Srinivasan、Huaixiu Steven Zheng、Vinodkumar Prabhakaran、Daniel De Freitas、Lora Aroyo、Matthew Lamm、Meredith Ringel Morris、Romal Thoppilan、Vincent Zhao、Johnny Soraker、Yu Du、Alena Butryna、Claire Cui、Chung-Ching Chang、Toju Duke、Ben Zevenbergen、Viktoriya Kuzmina、Taylor Bos、Alejandra Molina、Marc Pickett、James Qin、Mark Diaz、Yanqi Zhou、Noam Shazeer、Rachel Bernstein、YaGuang Li、Maxim Krikun、Dehao Chen、Adam Roberts、Aaron Cohen、Maarten Bosma、Igor Krivokon、Quoc Le、Heng-Tze Cheng、Ravi Rajakumar、Marian Croak、Dmitry Lepikhin、Josh Lee、Laichee Man、Tulsee Doshi、Ed Chi、Kristen Olson、Apoorv Kulshreshtha、Leslie Baker、Will Rusch
计算技术、计算机技术
Marcelo Menegali,Blaise Aguera-Arcas,Ray Kurzweil,Yanping Huang,Yuanzhong Xu,Joe Fenton,Ben Hutchinson,Erin Hoffman-John,Alicia Jin,Hongrae Lee,Kathleen Meier-Hellstern,Jamie Hall,Zhifeng Chen,Renelito Delos Santos,Amin Ghafouri,Pranesh Srinivasan,Huaixiu Steven Zheng,Vinodkumar Prabhakaran,Daniel De Freitas,Lora Aroyo,Matthew Lamm,Meredith Ringel Morris,Romal Thoppilan,Vincent Zhao,Johnny Soraker,Yu Du,Alena Butryna,Claire Cui,Chung-Ching Chang,Toju Duke,Ben Zevenbergen,Viktoriya Kuzmina,Taylor Bos,Alejandra Molina,Marc Pickett,James Qin,Mark Diaz,Yanqi Zhou,Noam Shazeer,Rachel Bernstein,YaGuang Li,Maxim Krikun,Dehao Chen,Adam Roberts,Aaron Cohen,Maarten Bosma,Igor Krivokon,Quoc Le,Heng-Tze Cheng,Ravi Rajakumar,Marian Croak,Dmitry Lepikhin,Josh Lee,Laichee Man,Tulsee Doshi,Ed Chi,Kristen Olson,Apoorv Kulshreshtha,Leslie Baker,Will Rusch.LaMDA: Language Models for Dialog Applications[EB/OL].(2022-01-20)[2025-05-16].https://arxiv.org/abs/2201.08239.点此复制
评论