Generative Pre-trained Transformer 3 (GPT-3)
Original author(s) OpenAI[1]
Initial release June 11, 2020 (beta)
Repository
Predecessor GPT-2
Successor GPT-4
Type
Website openai.com/blog/openai-api

Generative Pre-trained Transformer 3 (GPT-3) is an autoregressive language model released in 2020 that uses deep learning to produce human-like text. When given a prompt, it will generate text that continues the prompt.

The architecture is a decoder-only transformer network with a 2048-token-long context and then-unprecedented size of 175 billion parameters, requiring 800GB to store. The model was trained using generative pre-training; it is trained to predict what the next token is based on previous tokens. The model demonstrated strong zero-shot and few-shot learning on many tasks.[2]

The successor to GPT-2, GPT-3 is the third-generation language prediction model in a GPT series created by OpenAI, a San Francisco-based artificial intelligence research laboratory.[3] GPT-3, which was introduced in May 2020, and was in beta testing as of July 2020,[4] is part of a trend in natural language processing (NLP) systems of pre-trained language representations.[1]

The quality of the text generated by GPT-3 is so high that it can be difficult to determine whether or not it was written by a human, which has both benefits and risks.[5] Thirty-one OpenAI researchers and engineers presented the original May 28, 2020 paper introducing GPT-3. In their paper, they warned of GPT-3's potential dangers and called for research to mitigate risk.[1]: 34  David Chalmers, an Australian philosopher, described GPT-3 as "one of the most interesting and important AI systems ever produced."[6] An April 2022 review in The New York Times described GPT-3's capabilities as being able to write original prose with fluency equivalent to that of a human.[7]

Microsoft announced on September 22, 2020, that it had licensed "exclusive" use of GPT-3; others can still use the public API to receive output, but only Microsoft has access to GPT-3's underlying model.[8]

Background

Further information: GPT-2 § Background

According to The Economist, improved algorithms, powerful computers, and an increase in digitized data have fueled a revolution in machine learning, with new techniques in the 2010s resulting in "rapid improvements in tasks" including manipulating language.[9] Software models are trained to learn by using thousands or millions of examples in a "structure ... loosely based on the neural architecture of the brain".[9] One architecture used in natural language processing (NLP) is a neural network based on a deep learning model that was first introduced in 2017—the transformer architecture.[10] There are a number of NLP systems capable of processing, mining, organizing, connecting and contrasting textual input, as well as correctly answering questions.[11]

On June 11, 2018, OpenAI researchers and engineers posted their original paper introducing the first generative pre-trained transformer (GPT)—a type of generative large language model that is pre-trained with an enormous and diverse corpus of text via datasets, followed by discriminative fine-tuning to focus on a specific task. GPT models are transformer-based deep learning neural network architectures. Up to that point, the best-performing neural NLP models commonly employed supervised learning from large amounts of manually-labeled data, which made it prohibitively expensive and time-consuming to train extremely large language models.[2]

That first GPT model is known as "GPT-1," and it was then followed by "GPT-2" in February 2019. GPT-2 was created as a direct scale-up of GPT-1, with both its parameter count and dataset size increased by a factor of 10. It had 1.5 billion parameters, and was trained on a dataset of 8 million web pages.[12]

In February 2020, Microsoft introduced its Turing Natural Language Generation (T-NLG), which was claimed to be the "largest language model ever published at 17 billion parameters."[13] It performed better than any other language model at a variety of tasks which included summarizing texts and answering questions.

Training and capabilities

A sample student essay about pedagogy written by GPT-3

The construct of “learning styles” is problematic because it fails to account for the processes through which learning styles are shaped. Some students might develop a particular learning style because they have had particular experiences. Others might develop a particular learning style by trying to accommodate to a learning environment that was not well suited to their learning needs. Ultimately, we need to understand the interactions among learning styles and environmental and personal factors, and how these shape how we learn and the kinds of learning we experience.

– Text generated by Mike Sharples[14]

On May 28, 2020, an arXiv preprint by a group of 31 engineers and researchers at OpenAI described the development of GPT-3, a third-generation "state-of-the-art language model".[1][5] The team increased the capacity of GPT-3 by over two orders of magnitude from that of its predecessor, GPT-2,[15] making GPT-3 the largest non-sparse language model to date.[1]: 14 [3] Because GPT-3 is structurally similar to its predecessors,[1] its greater accuracy is attributed to its increased capacity and greater number of parameters.[16] GPT-3's capacity is ten times larger than that of Microsoft's Turing NLG, the next largest NLP model known at the time.[5]

Lambdalabs estimated a hypothetical cost of around $4.6 million US dollars and 355 years to train GPT-3 on a single GPU in 2020,[17] with lower actual training time by using more GPUs in parallel.

Sixty percent of the weighted pre-training dataset for GPT-3 comes from a filtered version of Common Crawl consisting of 410 billion byte-pair-encoded tokens.[1]: 9  Other sources are 19 billion tokens from WebText2 representing 22% of the weighted total, 12 billion tokens from Books1 representing 8%, 55 billion tokens from Books2 representing 8%, and 3 billion tokens from Wikipedia representing 3%.[1]: 9  GPT-3 was trained on hundreds of billions of words and is also capable of coding in CSS, JSX, and Python, among others.[4]

GPT-3 training data[1]: 9 
Dataset # tokens Proportion
within training
Common Crawl 410 billion 60%
WebText2 19 billion 22%
Books1 12 billion 8%
Books2 55 billion 8%
Wikipedia 3 billion 3%

Since GPT-3's training data was all-encompassing, it does not require further training for distinct language tasks.[4] The training data contains occasional toxic language and GPT-3 occasionally generates toxic language as a result of mimicking its training data. A study from the University of Washington found that GPT-3 produced toxic language at a toxicity level comparable to the similar natural language processing models of GPT-2 and CTRL. OpenAI has implemented several strategies to limit the amount of toxic language generated by GPT-3. As a result, GPT-3 produced less toxic language compared to its predecessor model, GPT-1, although it produced both more generations and a higher toxicity of toxic language compared to CTRL Wiki, a language model trained entirely on Wikipedia data.[18]

On June 11, 2020, OpenAI announced that users could request access to its user-friendly GPT-3 API—a "machine learning toolset"—to help OpenAI "explore the strengths and limits" of this new technology.[19][20] The invitation described how this API had a general-purpose "text in, text out" interface that can complete almost "any English language task", instead of the usual single use-case.[19] According to one user, who had access to a private early release of the OpenAI GPT-3 API, GPT-3 was "eerily good" at writing "amazingly coherent text" with only a few simple prompts.[21] In an initial experiment 80 US subjects were asked to judge if short ~200 word articles were written by humans or GPT-3. The participants judged correctly 52% of the time, doing only slightly better than random guessing.[1]

On November 18, 2021, OpenAI announced that enough safeguards had been implemented that access to its API would be unrestricted.[22] OpenAI provided developers with a content moderation tool that helps them abide by OpenAI's content policy.[23] On January 27, 2022, OpenAI announced that its newest GPT-3 language models, collectively referred to as InstructGPT, was now the default language model used on their API. According to OpenAI, InstructGPT produced content that was better aligned to user intentions by following instructions better, generating fewer made-up facts, and producing somewhat less toxic content.[24]

Because GPT-3 can "generate news articles which human evaluators have difficulty distinguishing from articles written by humans,"[5] GPT-3 has the "potential to advance both the beneficial and harmful applications of language models."[1]: 34  In their May 28, 2020 paper, the researchers described in detail the potential "harmful effects of GPT-3"[5] which include "misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing and social engineering pretexting".[1] The authors draw attention to these dangers to call for research on risk mitigation.[1]: 34 

GPT-3 is capable of performing zero-shot and few-shot learning (including one-shot).[1]

In June 2022, Almira Osmanovic Thunström wrote that GPT-3 was the primary author on an article on itself, that they had submitted it for publication,[25] and that it had been pre-published while waiting for completion of its review.[26]

GPT-3.5

On March 15, 2022, OpenAI made available new versions of GPT-3 and Codex in its API with edit and insert capabilities under the names "text-davinci-002" and "code-davinci-002".[27] These models were described as more capable than previous versions and were trained on data up to June 2021.[28] On November 30, 2022, OpenAI began referring to these models as belonging to the "GPT-3.5" series,[28] and released ChatGPT, which was fine-tuned from a model in the GPT-3.5 series.[29]

GPT-3.5 with browsing (ALPHA)

On April 10, 2023, OpenAI introduced an advanced version of its GPT-3.5 series model, known as GPT-3.5 with Browsing (ALPHA).[30] This innovative model builds upon the capabilities of its predecessor, released on March 15, 2022, as "text-davinci-002" and "code-davinci-002".[31] The GPT-3.5 with Browsing (ALPHA) model enhances its performance by incorporating the ability to access and browse online information, leading to more accurate and up-to-date responses to user queries.[30]

Designed to improve user experience, the GPT-3.5 with Browsing (ALPHA) model delivers more precise and contextually relevant information. It has been trained on data up to September 2021, resulting in better performance compared to the earlier GPT-3.5 series models, which were trained on data up until June 2021.[31] OpenAI launched this cutting-edge model to provide developers and users with an advanced natural language processing tool capable of effectively retrieving and synthesizing online information.[30]

To enable browsing capabilities, OpenAI implemented a new API that allows the GPT-3.5 with Browsing (ALPHA) model to access selected online resources during operation.[32] This feature empowers users to ask questions or request information with the expectation that the model will deliver updated, accurate, and relevant answers based on the latest online sources.

On April 27, 2023, OpenAI made the GPT-3.5 with Browsing (ALPHA) model publicly available to GPT Plus users, broadening access to its state-of-the-art capabilities and features.[32]

Reception

Applications

Reviews

Criticism

GPT-3's builder, OpenAI, was initially founded as a non-profit in 2015.[56] In 2019, OpenAI broke from its usual open-source standards by not publicly releasing GPT-3's predecessor model, citing concerns that the model could facilitate the propagation of fake news. OpenAI eventually released a version of GPT-2 that was 8% of the original model's size.[57] In the same year, OpenAI restructured to be a for-profit company.[58] In 2020, Microsoft announced the company had exclusive licensing of GPT-3 for Microsoft's products and services following a multi-billion dollar investment in OpenAI. The agreement permits OpenAI to offer a public-facing API such that users can send text to GPT-3 to receive the model's output, but only Microsoft will have access to GPT-3's source code.[8]

Large language models, such as GPT-3, have come under criticism from a few of Google's AI ethics researchers for the environmental impact of training and storing the models, detailed in a paper co-authored by Timnit Gebru and Emily M. Bender in 2021.[59]

The growing[when?] use of automated writing technologies based on GPT-3 and other language generators, has raised concerns regarding academic integrity[60] and raised the stakes of how universities and schools will gauge what constitutes academic misconduct such as plagiarism.[61]

OpenAI's GPT series was built with data from the Common Crawl dataset, a conglomerate of copyrighted articles, internet posts, web pages, and books scraped from 60 million domains over a period of 12 years. TechCrunch reports this training data includes copyrighted material from the BBC, The New York Times, Reddit, the full text of online books, and more.[62] In its response to a 2019 Request for Comments on Intellectual Property Protection for Artificial Intelligence Innovation from the United States Patent and Trademark Office (USPTO), OpenAI argued that "Under current law, training AI systems [such as its GPT models] constitutes fair use," but that "given the lack of case law on point, OpenAI and other AI developers like us face substantial legal uncertainty and compliance costs."[63]

See also

References

  1. ^ a b c d e f g h i j k l m n Brown, Tom B.; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini; Herbert-Voss, Ariel; Krueger, Gretchen; Henighan, Tom; Child, Rewon; Ramesh, Aditya; Ziegler, Daniel M.; Wu, Jeffrey; Winter, Clemens; Hesse, Christopher; Chen, Mark; Sigler, Eric; Litwin, Mateusz; Gray, Scott; Chess, Benjamin; Clark, Jack; Berner, Christopher; McCandlish, Sam; Radford, Alec; Sutskever, Ilya; Amodei, Dario (May 28, 2020). "Language Models are Few-Shot Learners". arXiv:2005.14165.
  2. ^ a b Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (June 11, 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). p. 12. Archived (PDF) from the original on January 26, 2021. Retrieved July 31, 2020.
  3. ^ a b Shead, Sam (July 23, 2020). "Why everyone is talking about the A.I. text generator released by an Elon Musk-backed lab". CNBC. Archived from the original on July 30, 2020. Retrieved July 31, 2020. Four preprints were released between May 28 and July 22, 2020.
  4. ^ a b c Bussler, Frederik (July 21, 2020). "Will GPT-3 Kill Coding?". Towards Data Science. Archived from the original on August 19, 2020. Retrieved August 1, 2020.
  5. ^ a b c d e Sagar, Ram (June 3, 2020). "OpenAI Releases GPT-3, The Largest Model So Far". Analytics India Magazine. Archived from the original on August 4, 2020. Retrieved July 31, 2020.
  6. ^ a b Chalmers, David (July 30, 2020). Weinberg, Justin (ed.). "GPT-3 and General Intelligence". Daily Nous. Philosophers On GPT-3 (updated with replies by GPT-3). Archived from the original on August 4, 2020. Retrieved August 4, 2020.
  7. ^ Johnson, Steven; Iziev, Nikita (April 15, 2022). "A.I. Is Mastering Language. Should We Trust What It Says?". The New York Times. Archived from the original on November 24, 2022. Retrieved April 23, 2022.
  8. ^ a b Hao, Karen (September 23, 2020). "OpenAI is giving Microsoft exclusive access to its GPT-3 language model". MIT Technology Review. Archived from the original on February 5, 2021. Retrieved September 25, 2020. The companies say OpenAI will continue to offer its public-facing API, which allows chosen users to send text to GPT-3 or OpenAI's other models and receive its output. Only Microsoft, however, will have access to GPT-3's underlying code, allowing it to embed, repurpose, and modify the model as it pleases.
  9. ^ a b "An understanding of AI's limitations is starting to sink in". The Economist. June 11, 2020. ISSN 0013-0613. Archived from the original on July 31, 2020. Retrieved July 31, 2020.
  10. ^ Polosukhin, Illia; Kaiser, Lukasz; Gomez, Aidan N.; Jones, Llion; Uszkoreit, Jakob; Parmar, Niki; Shazeer, Noam; Vaswani, Ashish (June 12, 2017). "Attention Is All You Need". arXiv:1706.03762 [cs.CL].
  11. ^ "Natural Language Processing". Archived from the original on August 22, 2020. Retrieved July 31, 2020.
  12. ^ https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
  13. ^ Sterling, Bruce (February 13, 2020). "Web Semantics: Microsoft Project Turing introduces Turing Natural Language Generation (T-NLG)". Wired. ISSN 1059-1028. Archived from the original on November 4, 2020. Retrieved July 31, 2020.
  14. ^ Marche, Stephen (December 6, 2022). "The College Essay Is Dead". The Atlantic. Archived from the original on January 24, 2023. Retrieved December 8, 2022.
  15. ^ "Language Models are Unsupervised Multitask Learners" (PDF). openai.com. Archived (PDF) from the original on December 12, 2019. Retrieved December 4, 2019. GPT-2, is a 1.5B parameter Transformer
  16. ^ Ray, Tiernan (June 1, 2020). "OpenAI's gigantic GPT-3 hints at the limits of language models for AI". ZDNet. Archived from the original on June 1, 2020. Retrieved July 31, 2020.
  17. ^ Li, Chuan (June 3, 2020), OpenAI's GPT-3 Language Model: A Technical Overview, archived from the original on March 27, 2023, retrieved March 27, 2023
  18. ^ Gehman, Samuel; Gururangan, Suchin; Sap, Maarten; Choi, Yejin; Smith, Noah A. (November 16–20, 2020), REALTOXICITYPROMPTS: Evaluating Neural Toxic Degeneration in Language Models, Association for Computational Linguistics, pp. 3356–3369, arXiv:2009.11462
  19. ^ a b "OpenAI API". OpenAI. June 11, 2020. Archived from the original on June 11, 2020. Retrieved July 31, 2020.
  20. ^ Coldewey, Devin (June 11, 2020). "OpenAI makes an all-purpose API for its text-based AI capabilities". TechCrunch. Archived from the original on October 27, 2021. Retrieved July 31, 2020. If you've ever wanted to try out OpenAI's vaunted machine learning toolset, it just got a lot easier. The company has released an API that lets developers call its AI tools in on "virtually any English language task."
  21. ^ Arram (July 9, 2020). "GPT-3: An AI that's eerily good at writing almost anything". Arram Sabeti. Archived from the original on July 20, 2020. Retrieved July 31, 2020.
  22. ^ "OpenAI's API Now Available with No Waitlist". OpenAI. November 18, 2021. Archived from the original on November 5, 2022. Retrieved November 5, 2022.
  23. ^ "OpenAI API". beta.openai.com. Archived from the original on December 23, 2022. Retrieved November 5, 2022.
  24. ^ "Aligning Language Models to Follow Instructions". OpenAI. January 27, 2022. Archived from the original on November 5, 2022. Retrieved November 5, 2022.
  25. ^ Thunström, Almira Osmanovic (June 30, 2022). "We Asked GPT-3 to Write an Academic Paper about Itself – Then We Tried to Get It Published". Scientific American. Archived from the original on June 30, 2022. Retrieved June 30, 2022.
  26. ^ Transformer, Gpt Generative Pretrained; Thunström, Almira Osmanovic; Steingrimsson, Steinn (June 21, 2022). "Can GPT-3 write an academic paper on itself, with minimal human input?". Archive ouverte HAL (in French). Archived from the original on June 30, 2022. Retrieved June 30, 2022.
  27. ^ "New GPT-3 Capabilities: Edit & Insert". OpenAI. March 15, 2022. Archived from the original on January 13, 2023. Retrieved January 13, 2023.
  28. ^ a b "OpenAI API". platform.openai.com. Archived from the original on March 20, 2023. Retrieved March 15, 2023.
  29. ^ "ChatGPT: Optimizing Language Models for Dialogue". OpenAI. November 30, 2022. Archived from the original on November 30, 2022. Retrieved January 13, 2023.
  30. ^ a b c tingetici (April 10, 2023). "Default (GPT-3.5) with browsing ALPHA -- NEW Model showed up just now". r/OpenAI. Retrieved April 27, 2023.
  31. ^ a b "Introducing GPT-3.5 Series: text-davinci-002 and code-davinci-002 Models". OPEN AI. March 15, 2022. Retrieved April 27, 2023.
  32. ^ a b "GPT-3.5 with Browsing (ALPHA) Now Available for GPT Plus Users". OPEN AI. April 27, 2023. Retrieved April 27, 2023.
  33. ^ "OpenAI Codex". OpenAI. August 10, 2021. Archived from the original on February 3, 2023. Retrieved December 23, 2022.
  34. ^ Thompson, Clive (March 15, 2022). "How an AI Became My Code-Writing Genie". Wired. Archived from the original on December 23, 2022. Retrieved December 23, 2022.
  35. ^ "Microsoft announced its first customer product features powered by GPT-3 and @Azure". The AI Blog. May 25, 2021. Archived from the original on May 26, 2021. Retrieved May 26, 2021.
  36. ^ Vincent, James (May 25, 2021). "Microsoft has built an AI-powered autocomplete for code using GPT-3". The Verge. Archived from the original on December 23, 2022. Retrieved December 23, 2022.
  37. ^ "CodexDB - SQL Processing Powered by GPT-3". CodexDB - SQL Processing Powered by GPT-3. Archived from the original on December 7, 2022. Retrieved December 7, 2022.
  38. ^ Fagone, Jason (July 23, 2021). "The Jessica Simulation: Love and loss in the age of A.I." San Francisco Chronicle. Archived from the original on July 28, 2021. Retrieved July 29, 2021.
  39. ^ GPT-3 (September 8, 2020). "A robot wrote this entire article. Are you scared yet, human? | GPT-3". The Guardian. ISSN 0261-3077. Archived from the original on September 8, 2020. Retrieved September 15, 2020.
  40. ^ "Update: Language Models and Dragon". Latitude blog. December 8, 2021. Archived from the original on April 25, 2022. Retrieved March 22, 2022.
  41. ^ "This Mystical Book Was Co-Authored by a Disturbingly Realistic AI". www.vice.com. 2022. Archived from the original on December 23, 2022. Retrieved December 23, 2022.
  42. ^ GPT-3 (February 24, 2023). "38 Prompt Examples in 10 Different Categories | GPT-3". GiPiTi Chat. Archived from the original on April 8, 2023. Retrieved February 24, 2023.
  43. ^ "Can ChatGPT AI chatbot spot early stages of Alzheimer's? - study". The Jerusalem Post. 2022. Archived from the original on February 10, 2023. Retrieved February 10, 2023.
  44. ^ Agbavor, Felix; Liang, Hualou (December 22, 2022). "Predicting dementia from spontaneous speech using large language models". PLOS Digital Health. 1 (12): e0000168. doi:10.1371/journal.pdig.0000168. PMID 36812634. S2CID 255029590.
  45. ^ Manjoo, Farhad (July 29, 2020). "How Do You Know a Human Wrote This?". The New York Times. ISSN 0362-4331. Archived from the original on October 29, 2020. Retrieved August 4, 2020.
  46. ^ Weinberg, Justin, ed. (July 30, 2020). "Philosophers On GPT-3 (updated with replies by GPT-3)". Daily Nous. Archived from the original on October 30, 2020. Retrieved July 31, 2020.
  47. ^ Simonite, Tom (July 22, 2020). "Did a Person Write This Headline, or a Machine?". Wired. ISSN 1059-1028. Archived from the original on November 1, 2020. Retrieved July 31, 2020.
  48. ^ Claypoole, Theodore (July 30, 2020). "New AI Tool GPT-3 Ascends to New Peaks, But Proves How Far We Still Need to Travel". The National Law Review. Archived from the original on October 30, 2020. Retrieved August 4, 2020.
  49. ^ Marcus, Gary (December 1, 2018). "The deepest problem with deep learning". Medium. Archived from the original on August 1, 2019. Retrieved September 29, 2020.
  50. ^ Marcus, Gary; Davis, Ernest (August 22, 2020). "GPT-3, Bloviator: OpenAI's language generator has no idea what it's talking about". MIT Technology Review. Archived from the original on August 23, 2020. Retrieved August 23, 2020.
  51. ^ Metz, Cade (November 24, 2020). "Meet GPT-3. It Has Learned to Code (and Blog and Argue)". The New York Times. ISSN 0362-4331. Archived from the original on December 6, 2020. Retrieved November 24, 2020.
  52. ^ "Medical chatbot using OpenAI's GPT-3 told a fake patient to kill themselves". AI News. October 28, 2020. Archived from the original on January 10, 2021. Retrieved January 8, 2021.
  53. ^ Chomsky on Terence McKenna, Sam Harris, GPT3, Cryptocurrencies, Kierkegaard, Neuralink, & Hofstadter. March 24, 2021. Event occurs at 1:11:44. Archived from the original on April 29, 2021. Retrieved April 29, 2021.
  54. ^ Floridi, Luciano; Chiriatti, Massimo (November 1, 2020). "GPT‑3: Its Nature, Scope, Limits, and Consequences". Minds and Machines. 30 (4): 681–694. doi:10.1007/s11023-020-09548-1. S2CID 228954221.
  55. ^ Vincent, James (July 30, 2020). "OpenAI's latest breakthrough is astonishingly powerful, but still fighting its flaws". The Verge. Archived from the original on July 30, 2020. Retrieved November 9, 2022.
  56. ^ Olanoff, Drew (December 11, 2015). "Artificial Intelligence Nonprofit OpenAI Launches With Backing From Elon Musk And Sam Altman". Tech Crunch. Archived from the original on October 20, 2022. Retrieved May 31, 2021.
  57. ^ Hao, Karen (August 29, 2019). "OpenAI has released the largest version yet of its fake-news-spewing AI". MIT Technology Review. Archived from the original on May 9, 2021. Retrieved May 31, 2021.
  58. ^ Coldewey, Devin (March 11, 2019). "OpenAI shifts from nonprofit to 'capped-profit' to attract capital". Tech Crunch. Archived from the original on January 4, 2023. Retrieved May 31, 2021.
  59. ^ Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Shmitchell, Shmargaret (March 3, 2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. pp. 610–623. doi:10.1145/3442188.3445922.
  60. ^ Mindzak, Michael; Eaton, Sarah Elaine. "Artificial intelligence is getting better at writing, and universities should worry about plagiarism". The Conversation. Archived from the original on November 7, 2021. Retrieved November 6, 2021.
  61. ^ Rogerson, Ann M.; McCarthy, Grace (December 2017). "Using Internet based paraphrasing tools: Original work, patchwriting or facilitated plagiarism?". International Journal for Educational Integrity. 13 (1): 1–15. doi:10.1007/s40979-016-0013-y. ISSN 1833-2595. S2CID 9473217.
  62. ^ Here are a few ways GPT-3 can go wrong. TechCrunch. Archived from the original on November 26, 2021. Retrieved November 26, 2021.
  63. ^ Comment Regarding Request for Comments on Intellectual Property Protection for Artificial Intelligence Innovation (PDF). USPTO. Archived (PDF) from the original on October 16, 2021. Retrieved November 30, 2021.