Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
Priti Oli、James B. Simon、Nathan A. Chi、Peter Chang、Leo Gao、John Miller、Maria Jose Ram¨arez Quintana、Rif A. Saurous、Samuel Gruetter、Siamak Shakeri、Karthik Gopalakrishnan、Daniel Levy、Jason Wei、Evgenii Zheltonozhskii、Michael Cohen、Elizabeth Donoway、Michael Ivanitskiy、Maartje ter Hoeve、Moin Aminnaseri、Isaac Noble、B. Ryan Roberts、Lucy Noble、Sharon Zhou、Ian Ng、Ruslan Salakhutdinov、Liam Dugan、Daniel Mosegu¨a Gonz¨￠lez、Dan Garrette、Karl Krauth、Andrew Dai、Debnath、Lianhui Qin、Timofei Kornev、Sebastian Gehrmann、Agnieszka Kluska、Hayden Bogar、Giorgio Mariani、Rosanne Liu、Yifu Chen、Mitch Walker、Clemencia Siro、Ludwig Schmidt、Stefano Ermon、Shadi Hamdan、Vivek Srikumar、Chris Callison-Burch、Wout Vossen、Zachary Seid、Kyle Richardson、Neta Gur-Ari Krakover、Zijian Wang、Simone Melzi、Jesse Engel、Michael Strube、Titus Tunduny、Qinlang Chen、Chenlin Meng、Andrew La、Yair Lakretz、Pablo Antonio Moreno Casares、Summer Misherghi、Marie Tolkiehn、Jiacheng Xu、Nicole Martinez、Jose Hernandez-Orallo、James Zou、Mantas Mazeika、Michael Starritt、Shubham Toshniwal、Paul Vicol、Adam Fisch、Clara E. Rivera、Roman Sitelew、Sriharsha Hatwar、Francesca Happ¨|、Erkut Erdem、Theodore Rothschild、Arun Kirubarajan、Jeroen Taal、Xinyi Wu、Jos Rozen、Adam Santoro、Lucas Lam、Melvin McElrath、Cameron Dour、Jascha Sohl-Dickstein、Bao Sheng Loe、Nicholas Roberts、Cristina Garbacea、Shikhar Singh、Jonathan Batchelder、David Drakard、Xinran Zhao、Kevin Omondi、Omar Elbaghdadi、Uri Shaham、Danielle Perszyk、Dimitri Coelho Mollo、Eleanor Hagerman、Stephanie Lin、Kamil Kanclerz、Michael A. Yee、Ziyi Wu、Luis Oliveros Col¨?n、Jennifer Marsh、Germ¨￠n Kruszewski、Rabin Banjade、Cameron Diao、Ryan Stovall、James Zheng、Sanjeev Kwatra、Eunice Engefu Manyasi、Gaurav Mishra、Abhinav Rastogi、Mohit Bansal、Ryan Lee、Hannah Kim、Arash Gholamidavoodi、Xiang Ren、Vinay Uday Prabhu、Maheen Farooqi、Sanghyun Han、Yejin Choi、Michele Bevilacqua、Joshua B. Tenenbaum、Andrew Lampinen、Aman Hussain、Christian Voigt、Yufang Hou、David Dohan、Kumar Shridhar、Dan Kilman、Steven T. Piantadosi、Stanislas Dehaene、Yiding Hao、Aarohi Srivastava、Harsh Mehta、James Koppel、Laria Reynolds、Soo-Hwan Lee、Frieda Rong、Deniz Yuret、Katerina Ignatyeva、Yu Hou、Chiyu Wu、Abu Awal Md Shoeb、Shixiang Shane Gu、M¨￠ty¨￠s Schubert、Tianle Wang、Genta Indra Winata、Karen Livescu、Vedant Misra、Xiaoyu Tong、Mihir Kale、Anh Vuong、Jiaming Song、Ryan Chi、L¨1tfi Kerem ?enel、Elad Segal、Zirui Wang、Abubakar Abid、Dieuwke Hupkes、Sarik Ghazarian、Daniel Freeman、Stefan Divic、Jesujoba Alabi、Mor Geva、Kevin Gimpel、Chandan Singh、Aitor Lewkowycz、Dan Hendrycks、Abhishek Rao、Ellie Pavlick、Trenton Chang、Shima Asaadi、Rachel Etta Rudolph、Oliver Zhang、Matthias Hagen、Owain Evans、Sherry Shi、Dilyar Buzan、Giambattista Parascandolo、Yasaman Bahri、Austin Herrick、Joseph Guerr、Sneha Priscilla Makini、Rohan Sikand、Danny Hernandez、Jacob Hilton、Bryan Orinion、Te-Lin Wu、Ethan A. Chi、Berk Ekmekci、William Zhang、Louis-Philippe Morency、Tyler Shultz、Piyush Patil、Luke Metz、Sam Shleifer、Ali Tazarv、Behnam Hedayatnia、Stephen Prasad、Omer Levy、Piotr Mi?kowski、Jana Thompson、Gregor Betz、Pinyu Hwang、William Saunders、Luheng He、Hannah Rashkin、Ali Safaya、C¨|sar Ferri Ram¨arez、Thomas Phan、Sebastian Schuster、Sayan Ghosh、Vinay Ramasesh、Mike Cain、Anu Venkatesh、Michael Gu、Denis Kleyko、Christopher Potts、Yonatan Belinkov、Nayeon Lee、Jelle Bosscher、Roman Novak、Vera Demberg、Parth Doshi、Andrea Santilli、Jekaterina Novikova、Deep Ganguli、Chris Waites、Ashish Sabharwal、Vishakh Padmakumar、Ekaterina Shutova、Nick Doiron、Kory Mathewson、Hugh Mee Wong、Ronan LeBras、Rylan Yang、Niklas Deckers、Rahel Habacker、Fanyue Xia、Gerard de Melo、Dan Roth、Francois Chollet、Adri¨¤ Garriga-Alonso、Tariq Ali、Hongming Zhang、Noah Constant、Dong-Ho Lee、Joshua S. Rule、Eric Tang、Amanda Askell、Mimee Xu、Samuel R. Bowman、Jonathan Berant、Manaal Faruqui、Ekin Dogus Cubuk、Akshat Agarwal、Andreas Stuhlm¨1ller、Ameet Rahane、Victoria Nyamai、Sepideh Sadeghi、Joan Waweru、Nikita Nangia、Sahib Singh、J?rg Frohberg、Amanda Dsouza、Sebastian Bischoff、Ethan Dyer、Alethea Power、Martin Potthast、Robbe Raymaekers、Ambrose Slone、Damien Sileo、Avia Efrat、Anders Andreassen、Aykut Erdem、Alex Ray、Yuntao Bai、Henry Shevlin、Rowan Jacobs、Ethan Kim、Jared Kaplan、Lidia Contreras-Ochando、Vikas Raunak、Zijie J. Wang、Tatsu Hashimoto、Jan Koco¨?、Sarah A. Rous、Derek Chen、Ethan Jerzak、Rapha?l Milli¨¨re、Micha? Sw?drowski、Jaap Jumelet、Tao Yu、Sam Wiseman、Trishala Neeraj、Shubh Pachchigar、Ayla Karaka?、Jarema Radom、Xudong Shen、Nitish Shirish Keskar、Kristen Chiafullo、Mo Tiwari、Mario Giulianelli、Denis Emelin、Mirac Suzgun、Li Zhang、Benno Stein、Jackson Kernion、Niklas Muennighoff、Andy Zou、Allen Nie、Dar Gilboa、Sam Dillavou、Siva Reddy、Christopher D. Manning、Maarten Sap、Arul Menezes、Sean Casey、Jaehoon Lee、Tal Linzen、Rui Zhang、Tobias Gerstenberg、Joseph Jones、Andrea Madotto、Jack Geissinger、William Fedus、Janelle Wingfield、Daphne Ippolito、Kyle McDonell、Marco Marelli、Jaime Fern¨￠ndez Fisac、Mukund Varma T、Gloria Wang、Joseph Boudeman、Medina Orduna Baitemirova、Bart?omiej Bojanowski、Saif M. Mohammad、Melody Arnaud、David Jurgens、Luca Moschella、Marco Maru、Nuan Wen、Kaustubh D. Dhole、Blake Howald、Alex Warstadt、Tal Schuster、Percy Liang、Angela Jiang、Ramon Risco、Jason Phang、Aditya Gupta、Yichi Yang、Bill Yuchen Lin、Anna Gottardi、Derek Tam、Stella Biderman、Batuhan ?zyurt、Anantharaman S. Iyer、Sajant Anand、Katja Markert、Debajyoti Datta、Nanyun Peng、John U. Balis、Shyamolima、Courtney Ashcraft、Nicholas Cameron、Qiaozhu Mei、Adam R. Brown、Fatemeh Siar、Svetlana Kiritchenko、John Burden、Shashank Srivastava、Michihiro Yasunaga、Paul Pu Liang、Joyce Chua、Emma Lam、Colin Raffel、Robert Frank、Hinrich Sch¨1tze、Yadollah Yaghoobzadeh、Yangqiu Song、Ksenia Shkaruta、Riku Arakawa、Jason Yosinski、Swaroop Mishra、Pouya Pezeshkpour、Noah Fiedel、Barret Zoph、Pascale Fung、Asher Mullokandov、Peter Eckersley、Cedrick Argueta、Animesh Gupta、Peiyuan Liao、Alicia Parrish、Raefer Gabriel、Antonio Norelli、Stuart M. Shieber、Ryan Teehan、Cindy Ramirez、Behnam Neyshabur、Daniel Khashabi、Eric Chu、Angelica Chen、Guy Gur-Ari、Dylan Schrader、Hiromu Yakura、Niveditha S. Iyer、Simon Thormeyer、Rhythm Garg、Shyam Upadhyay、Danqi Chen、Qing Lyu、Richard Barnes、Maarten Bosma、Ernie Chang、Diganta Misra、Pegah Alipoormolabashi、Zhuoye Zhao、Alexander W. Kocurek、Charles Rathkopf、Emanuele Rodola、Diyi Yang、Mozhdeh Gheini、Jillian Tang、Phu Mon Htut、Hana Galijasevic、Tiberius Nkinyili、Gonzalo Jaimovitch-L¨?pez、Catherine Stinson、Tao Li、Marco Baturan、Elizabeth Barnes、Benjamin Inden、Arfa Tabassum、Th¨|o Desbordes、Jeremy Kim、Timo Schick、Chitta Baral、Fernando Mart¨anez-Plumed、Hannaneh Hajishirzi、Matthew L. Leavitt、Tushar Khot、Alice Xiang、Samuel S. Schoenholz、Martha Lewis、Spencer Torene、Omar Agha
ShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammieShammie
计算技术、计算机技术数学物理学
Priti Oli,James B. Simon,Nathan A. Chi,Peter Chang,Leo Gao,John Miller,Maria Jose Ram¨arez Quintana,Rif A. Saurous,Samuel Gruetter,Siamak Shakeri,Karthik Gopalakrishnan,Daniel Levy,Jason Wei,Evgenii Zheltonozhskii,Michael Cohen,Elizabeth Donoway,Michael Ivanitskiy,Maartje ter Hoeve,Moin Aminnaseri,Isaac Noble,B. Ryan Roberts,Lucy Noble,Sharon Zhou,Ian Ng,Ruslan Salakhutdinov,Liam Dugan,Daniel Mosegu¨a Gonz¨￠lez,Dan Garrette,Karl Krauth,Andrew Dai,Debnath,Lianhui Qin,Timofei Kornev,Sebastian Gehrmann,Agnieszka Kluska,Hayden Bogar,Giorgio Mariani,Rosanne Liu,Yifu Chen,Mitch Walker,Clemencia Siro,Ludwig Schmidt,Stefano Ermon,Shadi Hamdan,Vivek Srikumar,Chris Callison-Burch,Wout Vossen,Zachary Seid,Kyle Richardson,Neta Gur-Ari Krakover,Zijian Wang,Simone Melzi,Jesse Engel,Michael Strube,Titus Tunduny,Qinlang Chen,Chenlin Meng,Andrew La,Yair Lakretz,Pablo Antonio Moreno Casares,Summer Misherghi,Marie Tolkiehn,Jiacheng Xu,Nicole Martinez,Jose Hernandez-Orallo,James Zou,Mantas Mazeika,Michael Starritt,Shubham Toshniwal,Paul Vicol,Adam Fisch,Clara E. Rivera,Roman Sitelew,Sriharsha Hatwar,Francesca Happ¨|,Erkut Erdem,Theodore Rothschild,Arun Kirubarajan,Jeroen Taal,Xinyi Wu,Jos Rozen,Adam Santoro,Lucas Lam,Melvin McElrath,Cameron Dour,Jascha Sohl-Dickstein,Bao Sheng Loe,Nicholas Roberts,Cristina Garbacea,Shikhar Singh,Jonathan Batchelder,David Drakard,Xinran Zhao,Kevin Omondi,Omar Elbaghdadi,Uri Shaham,Danielle Perszyk,Dimitri Coelho Mollo,Eleanor Hagerman,Stephanie Lin,Kamil Kanclerz,Michael A. Yee,Ziyi Wu,Luis Oliveros Col¨?n,Jennifer Marsh,Germ¨￠n Kruszewski,Rabin Banjade,Cameron Diao,Ryan Stovall,James Zheng,Sanjeev Kwatra,Eunice Engefu Manyasi,Gaurav Mishra,Abhinav Rastogi,Mohit Bansal,Ryan Lee,Hannah Kim,Arash Gholamidavoodi,Xiang Ren,Vinay Uday Prabhu,Maheen Farooqi,Sanghyun Han,Yejin Choi,Michele Bevilacqua,Joshua B. Tenenbaum,Andrew Lampinen,Aman Hussain,Christian Voigt,Yufang Hou,David Dohan,Kumar Shridhar,Dan Kilman,Steven T. Piantadosi,Stanislas Dehaene,Yiding Hao,Aarohi Srivastava,Harsh Mehta,James Koppel,Laria Reynolds,Soo-Hwan Lee,Frieda Rong,Deniz Yuret,Katerina Ignatyeva,Yu Hou,Chiyu Wu,Abu Awal Md Shoeb,Shixiang Shane Gu,M¨￠ty¨￠s Schubert,Tianle Wang,Genta Indra Winata,Karen Livescu,Vedant Misra,Xiaoyu Tong,Mihir Kale,Anh Vuong,Jiaming Song,Ryan Chi,L¨1tfi Kerem ?enel,Elad Segal,Zirui Wang,Abubakar Abid,Dieuwke Hupkes,Sarik Ghazarian,Daniel Freeman,Stefan Divic,Jesujoba Alabi,Mor Geva,Kevin Gimpel,Chandan Singh,Aitor Lewkowycz,Dan Hendrycks,Abhishek Rao,Ellie Pavlick,Trenton Chang,Shima Asaadi,Rachel Etta Rudolph,Oliver Zhang,Matthias Hagen,Owain Evans,Sherry Shi,Dilyar Buzan,Giambattista Parascandolo,Yasaman Bahri,Austin Herrick,Joseph Guerr,Sneha Priscilla Makini,Rohan Sikand,Danny Hernandez,Jacob Hilton,Bryan Orinion,Te-Lin Wu,Ethan A. Chi,Berk Ekmekci,William Zhang,Louis-Philippe Morency,Tyler Shultz,Piyush Patil,Luke Metz,Sam Shleifer,Ali Tazarv,Behnam Hedayatnia,Stephen Prasad,Omer Levy,Piotr Mi?kowski,Jana Thompson,Gregor Betz,Pinyu Hwang,William Saunders,Luheng He,Hannah Rashkin,Ali Safaya,C¨|sar Ferri Ram¨arez,Thomas Phan,Sebastian Schuster,Sayan Ghosh,Vinay Ramasesh,Mike Cain,Anu Venkatesh,Michael Gu,Denis Kleyko,Christopher Potts,Yonatan Belinkov,Nayeon Lee,Jelle Bosscher,Roman Novak,Vera Demberg,Parth Doshi,Andrea Santilli,Jekaterina Novikova,Deep Ganguli,Chris Waites,Ashish Sabharwal,Vishakh Padmakumar,Ekaterina Shutova,Nick Doiron,Kory Mathewson,Hugh Mee Wong,Ronan LeBras,Rylan Yang,Niklas Deckers,Rahel Habacker,Fanyue Xia,Gerard de Melo,Dan Roth,Francois Chollet,Adri¨¤ Garriga-Alonso,Tariq Ali,Hongming Zhang,Noah Constant,Dong-Ho Lee,Joshua S. Rule,Eric Tang,Amanda Askell,Mimee Xu,Samuel R. Bowman,Jonathan Berant,Manaal Faruqui,Ekin Dogus Cubuk,Akshat Agarwal,Andreas Stuhlm¨1ller,Ameet Rahane,Victoria Nyamai,Sepideh Sadeghi,Joan Waweru,Nikita Nangia,Sahib Singh,J?rg Frohberg,Amanda Dsouza,Sebastian Bischoff,Ethan Dyer,Alethea Power,Martin Potthast,Robbe Raymaekers,Ambrose Slone,Damien Sileo,Avia Efrat,Anders Andreassen,Aykut Erdem,Alex Ray,Yuntao Bai,Henry Shevlin,Rowan Jacobs,Ethan Kim,Jared Kaplan,Lidia Contreras-Ochando,Vikas Raunak,Zijie J. Wang,Tatsu Hashimoto,Jan Koco¨?,Sarah A. Rous,Derek Chen,Ethan Jerzak,Rapha?l Milli¨¨re,Micha? Sw?drowski,Jaap Jumelet,Tao Yu,Sam Wiseman,Trishala Neeraj,Shubh Pachchigar,Ayla Karaka?,Jarema Radom,Xudong Shen,Nitish Shirish Keskar,Kristen Chiafullo,Mo Tiwari,Mario Giulianelli,Denis Emelin,Mirac Suzgun,Li Zhang,Benno Stein,Jackson Kernion,Niklas Muennighoff,Andy Zou,Allen Nie,Dar Gilboa,Sam Dillavou,Siva Reddy,Christopher D. Manning,Maarten Sap,Arul Menezes,Sean Casey,Jaehoon Lee,Tal Linzen,Rui Zhang,Tobias Gerstenberg,Joseph Jones,Andrea Madotto,Jack Geissinger,William Fedus,Janelle Wingfield,Daphne Ippolito,Kyle McDonell,Marco Marelli,Jaime Fern¨￠ndez Fisac,Mukund Varma T,Gloria Wang,Joseph Boudeman,Medina Orduna Baitemirova,Bart?omiej Bojanowski,Saif M. Mohammad,Melody Arnaud,David Jurgens,Luca Moschella,Marco Maru,Nuan Wen,Kaustubh D. Dhole,Blake Howald,Alex Warstadt,Tal Schuster,Percy Liang,Angela Jiang,Ramon Risco,Jason Phang,Aditya Gupta,Yichi Yang,Bill Yuchen Lin,Anna Gottardi,Derek Tam,Stella Biderman,Batuhan ?zyurt,Anantharaman S. Iyer,Sajant Anand,Katja Markert,Debajyoti Datta,Nanyun Peng,John U. Balis,Shyamolima,Courtney Ashcraft,Nicholas Cameron,Qiaozhu Mei,Adam R. Brown,Fatemeh Siar,Svetlana Kiritchenko,John Burden,Shashank Srivastava,Michihiro Yasunaga,Paul Pu Liang,Joyce Chua,Emma Lam,Colin Raffel,Robert Frank,Hinrich Sch¨1tze,Yadollah Yaghoobzadeh,Yangqiu Song,Ksenia Shkaruta,Riku Arakawa,Jason Yosinski,Swaroop Mishra,Pouya Pezeshkpour,Noah Fiedel,Barret Zoph,Pascale Fung,Asher Mullokandov,Peter Eckersley,Cedrick Argueta,Animesh Gupta,Peiyuan Liao,Alicia Parrish,Raefer Gabriel,Antonio Norelli,Stuart M. Shieber,Ryan Teehan,Cindy Ramirez,Behnam Neyshabur,Daniel Khashabi,Eric Chu,Angelica Chen,Guy Gur-Ari,Dylan Schrader,Hiromu Yakura,Niveditha S. Iyer,Simon Thormeyer,Rhythm Garg,Shyam Upadhyay,Danqi Chen,Qing Lyu,Richard Barnes,Maarten Bosma,Ernie Chang,Diganta Misra,Pegah Alipoormolabashi,Zhuoye Zhao,Alexander W. Kocurek,Charles Rathkopf,Emanuele Rodola,Diyi Yang,Mozhdeh Gheini,Jillian Tang,Phu Mon Htut,Hana Galijasevic,Tiberius Nkinyili,Gonzalo Jaimovitch-L¨?pez,Catherine Stinson,Tao Li,Marco Baturan,Elizabeth Barnes,Benjamin Inden,Arfa Tabassum,Th¨|o Desbordes,Jeremy Kim,Timo Schick,Chitta Baral,Fernando Mart¨anez-Plumed,Hannaneh Hajishirzi,Matthew L. Leavitt,Tushar Khot,Alice Xiang,Samuel S. Schoenholz,Martha Lewis,Spencer Torene,Omar Agha.Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models[EB/OL].(2022-06-09)[2025-06-29].https://arxiv.org/abs/2206.04615.点此复制
评论