By Nicole Cooper
Could artificial intelligence systems replace lawyers in the future? The short answer is no. However, the capabilities of these artificial intelligence (AI) systems are concerning. The most popular AI system in the world’s view right now is ChatGPT. ChatGPT is an AI chatbot system that OpenAI, a company in San Francisco, released in November 2022.[1] ChatGPT is a generative AI that allows users to enter written prompts and receive human-like text or images and videos generated by the AI.[2] This system can produce human-like responses to just about any request or prompt from a user.[3]
One reason people are concerned ChatGPT could replace jobs is that it has passed a range of tests and exams, including law exams and portions of the bar exam.[4] Daniel Katz, a professor at Chicago Kent College of Law and Michael Bommarito, a professor at Michigan State University College of Law, tested OpenAI’s GPT3.5 model on the multistate multiple choice (MBE) section of the Bar exam.[5] Overall, it was correct 50.3% of the time as compared to regular test-takers, who get 68% of the questions right.
Recently, professors at the University of Minnesota Law School graded tests blindly. After completing 95 multiple-choice questions and 12 essay questions, the bot performed on average at the level of a C+ student, achieving a passing score.[6] Although this system is smart enough to pass complicated exams, its scores aren’t particularly high enough to show that it can perform better than a law student. It would be more concerning if the exam produced perfect scores.
ChatGPT also has severe limitations. It cannot perform the complex tasks that lawyers do daily such as applying facts to the law, providing advice that is in the best interests of clients, interviewing witnesses, negotiating settlements, and much more. It also struggles to know what the user really wants from it as its answers to questions vary widely to how the prompts are phrased.[7] Its inability to perform interpersonal tasks will keep its potential advancements into the legal field limited to a research tool similar to Westlaw and LexisNexis.[8]
Lastly, I actually tested out ChatGPT myself and asked the AI a wide range of requests/questions. These prompts included a recipe for chocolate chip cookies, a poem about dogs, and lastly a question: can lawyers be replaced by ChatGPT? After typing out a great chocolate chip cookie recipe, and a heart-warming poem about dogs, the response to my last question was interesting as it came to the same conclusion that I did. ChatGPT states that “No, lawyers cannot be replaced by ChatGPT or any other AI language model. While AI technology has made significant strides in recent years, there are still many areas where human lawyers have critical advantage.”[9] I included the rest of the response in a screenshot below.
It is much appreciated that this AI does not think it will take over the legal field, however, lawyers should be careful if they choose to use this technology in practice. While it may be able to draft legal documents and conduct legal research faster than humanly possible, it can still make mistakes and provide misinformation. Even OpenAI stated on their website, “ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.”[10] If lawyers rely on AI to do the work for them, it will likely create a host of liabilities and malpractice concerns.
[1] Ryan Browne, All you need to know about ChatGPT, the A.I. chatbot that’s got the world talking and tech giants clashing, cnbc (Feb. 98, 2023, 7:37 AM EST), https://www.cnbc.com/2023/02/08/what-is-chatgpt-viral-ai-chatbot-at-heart-of-microsoft-google-fight.html#:~:text=In%20its%20own%20description%2C%20ChatGPT,talk%20of%20the%20business%20world.
[2] Id.
[3] Id.
[4] Samantha Murphy Kelly, ChatGPT passes exams from law and business schools, cnn business (Jan. 26, 2023, 1:35 PM EST), https://www.cnn.com/2023/01/26/tech/chatgpt-passes-exams/index.html.
[5] Michael Bommarito & Daniel Katz, GPT Takes the Bar Exam, ssrn (Jan. 03, 2023), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4314839.
[6] Kelly, supra note 4.
[7] Jason Hennessey, How Law Firms Can Leverage ChatGPT To Get More Cases, the national law review (Jan. 17, 2023), https://www.natlawreview.com/article/how-law-firms-can-leverage-chatgpt-to-get-more-cases#:~:text=Recent%20adopters%20have%20been%20impressed,would%20typically%20be%20manual%20tasks.
[8] Eran Kahana, ChatGPT Is a Cool Trick, But AI Won’t Replace Lawyers, lexis+ (Feb. 07, 2023, 3:39 PM EST), https://plus.lexis.com/newsstand#/law360-pulse/article/1573178.
[9] ChatGPT, openai, https://chat.openai.com/chat (last visited Feb 23, 2023).
[10] ChatGPT: Optimizing Language Models for Dialogue, openai, https://openai.com/blog/chatgpt/ (last visited Feb. 23, 2023).