As a junior lawyer, you are probably familiar with phrases like “many thanks for this analysis, now where is the summary?”
Almost every good legal memorandum includes a summary. Why? Because most clients are not going to read the whole thing. It’s fast and it’s what clients want. Same goes for lawyers, if they need to digest complex content such as legal literature or case law, it's always helpful to start with reading a summary as it gives the reader a mental framework when digging into the details.
Now, it turns out that GPT-3.5 and GPT-4 language models (the AI of Chat GPT) are better at summarising than humans are.
In a comprehensive study, released a couple of days ago, a professor in Computer Science, an Associate Professor and a PhD student in Computer Vision compared summaries generated by GPT-3.5 and GPT-4 to summaries generated by people.
Consistently, the persons evaluating the summaries prefer AI generated summaries over human generated summaries. Important reasons why are:
- AI generated summaries are more coherent and are better at covering all topics human generated summaries
- Human generated summaries contain more inconsistencies between the factual information in the summary and the source text, compared to AI generated summaries
An important factor to take into account as well when judging whether a summary is helpful is the speed by which it is generated. The usefulness of summaries is not only measured by its accuracy but also by its ability to save time. In that sense, AI generated summaries have a clear preference over human generated summaries as well because for AI it's a matter of seconds to generate a summary.
In legal, summaries come in handy for purposes like gaining quick insights, completing a memorandum or communicating comprehensively about complex topics. Soon, it will be unthinkable for lawyers to see text summarization as a purely human task. Instead, we expect lawyers to cooperate with AI like GPT-3.5 and GPT-4 to increase the summaries’ accuracy ánd the speed by which they are generated.
Please see the link to the research paper here.
