The launch of SolidityBench, a new benchmark for evaluating large language models (LLMs) in Solidity code generation, aims to address the growing demand for secure and efficient smart contracts within the blockchain ecosystem.

SolidityBench promotes the development of sophisticated AI models for smart contracts while providing insights into their current capabilities and limitations.

It features two innovative benchmarks, NaïveJudge and HumanEval, designed to assess the proficiency of AI models in generating smart contract code.

The HumanEval benchmark adapts OpenAI’s original HumanEval from Python to Solidity, consisting of 25 tasks of varying difficulty that are compatible with the Hardhat development environment.

NaïveJudge evaluates LLMs by implementing smart contracts based on specifications derived from audited OpenZeppelin contracts, focusing on correctness and efficiency.

Developers and researchers are encouraged to explore and contribute to SolidityBench to refine AI models and promote best practices.

Scores for the models are based on a scale from 0 to 100, reflecting a comprehensive assessment across functionality, security, and efficiency.

OpenAI's GPT-4o has been ranked as the best AI model for writing Solidity smart contract code, achieving an overall score of 80.05.

OpenAI's newer reasoning models, o1-preview and o1-mini, scored 77.61 and 75.08 respectively, falling short of GPT-4o's top score.

Models from Anthropic and XAI, including Claude 3.5 Sonnet and grok-2, showed competitive performance with scores around 74.

In contrast, Nvidia's Llama-3.1-Nemotron-70B scored the lowest in the top 10 at 52.54.

Advanced LLMs, including OpenAI's GPT-4 and Claude 3.5 Sonnet, serve as impartial code reviewers, assessing key functionalities, edge cases, error management, and overall code structure.

The evaluation criteria for generated code include functional completeness, adherence to Solidity best practices, security standards, and optimization efficiency.