Can ChatGPT pass the Bar Exam?

ChatGPT and the Bar Exam

The bar exam is a demanding test that evaluates candidates' knowledge and reasoning skills through essays, performance tests, and multiple-choice questions. It is a standardized examination used by numerous U.S. jurisdictions as part of the bar admission process. The exam consists of three sections that gauge a candidate's grasp of common law principles, ability to apply legal principles to specific scenarios, and aptitude for completing legal tasks.

This exam is designed to allow attorneys to obtain licenses in multiple states without having to retake the bar exam for each state. Each jurisdiction sets its passing score, and exam scores can be transferred to other jurisdictions, subject to the rules and requirements of each state.

ChatGPT, built on the powerful OpenAI language models GPT-3.5 and GPT-4, possesses the ability to effectively tackle complex tasks that require extensive knowledge of the law, reading comprehension, and writing proficiency. These models come close to meeting the standards of human lawyers in nearly all U.S. jurisdictions. However, the National Conference of Bar Examiners, responsible for the creation of the multiple-choice section, recognizes the distinct abilities that lawyers acquire through education and practice—a level that current artificial intelligence cannot match.

Performance of ChatGPT in the Bar Exam

In recent research conducted by professors from Illinois Tech and Michigan State College of Law, ChatGPT 3.5 attained passing scores in two categories of the multistate bar exam (MBE) and performed similarly to human test-takers in another category. This demonstrated its comprehensive understanding of the legal domain. Additionally, OpenAI researchers conducted their own study, in which ChatGPT excelled in the bar exam, achieving a score in the 90th percentile. In this research, ChatGPT correctly answered almost 76% of the multiple-choice questions—an improvement from its previous rate of 50%—and outperformed the average human test-taker by more than 7%.

ChatGPT displayed the ability to generate essay and performance test responses that were highly relevant and logically connected. The threshold for passing the bar exam varies among states. For instance, in New York, a score of 266, approximately the 50th percentile, is required. ChatGPT's results suggest that large language models possess the capabilities to handle intricate tasks demanding significant legal knowledge, comprehension, and writing skills. These models meet or exceed the standards applied to human lawyers in almost all U.S. jurisdictions.

Did ChatGPT Pass a Bar Exam?

Yes, ChatGPT successfully passed the Uniform Bar Examination.

What is the Top 1% Bar Exam Score?

A score of 300 roughly corresponds to the top 90th percentile, while a score of 330 places test-takers around the top 1%.