Getting started

This page provides an overview of how to get started with LegalBench.

Downloading data

• LegalBench task data is available on Huggingface.
• If you would like task data as TSV files, please email us.
• If you would like to access an older version of the benchmark, please email us.

Accessing prompts

• Prompt templates for each task are accessible on the Github repository.
• If you would like to add a prompt template, then please create a pull-request on that repository.
• See here for an example of how to generate prompts for a dataset using a template.

Evaluation

• Additional information on the metrics used for each LegalBench task can be found in the paper.
• Code for automatically computing metrics is available on the Github repository.
• A small number of LegalBench tasks (~7) require manual evaluation. We have released a gradebook to guide this evaluation, which can be downloaded here. If you have any questions regarding these tasks, please reach out.