NaturalSQL

NaturalSQL is a series of large language models fine tuned for Natural Language to SQL generation.

Benchmarks

NaturalSQL-6.7B-v0 surpasses comparable models in performance on the SQL-Eval benchmark, outshining even its peers in size and scope. NaturalSQL even beats models such as Claude and GPT-3.5-Turbo on Text to SQL instructions for novel datasets.

Percentage of Correct SQL Queries on Novel Schemas

NaturalSQL-6.7B-v0 outperforms Text to SQL models double its size in the categories of ratio, joins, and order_by, queries.

Date queries will be a strong area of focus in the next iteration.

SQL-Eval Results by Category

Complex SQL Instructions

NaturalSQL was finetuned on complex questions and shows great results for its size on multi-part questions that entail multiple joins, sub queries, etc. Here is a post comparing NaturalSQL-6.7B against a similar sized model with 5 compound questions.

Example Question

Determine the customer with the largest total order value last month, and list all the products and their quantities they ordered.

sql
WITH
  LastMonthOrders AS (
    SELECT
      o.customerid,
      o.orderid,
      op.productid,
      op.quantity,
      p.productname,
      op.quantity * p.unitprice AS total_order_value
    FROM
      orders o
      JOIN order_details op ON o.orderid = op.orderid
      JOIN products p ON op.productid = p.productid
    WHERE
      o.orderdate >= date_trunc('month', CURRENT_DATE) - INTERVAL '1 month'
      AND o.orderdate < date_trunc('month', CURRENT_DATE)
  ),
  CustomerTotalOrderValue AS (
    SELECT
      customerid,
      SUM(total_order_value) AS total_value
    FROM
      LastMonthOrders
    GROUP BY
      customerid
    ORDER BY
      total_value DESC
    LIMIT
      1
  )
SELECT
  c.customerid,
  c.companyname,
  lm.productid,
  lm.productname,
  lm.quantity
FROM
  CustomerTotalOrderValue ctov
  JOIN LastMonthOrders lm ON ctov.customerid = lm.customerid
  JOIN customers c ON c.customerid = lm.customerid;