You can’t really use a diffusion image generation model to create charts. There are a lot of reasons for this, but the most important ones are that, one, the model doesn’t actually know anything about you’re trying to plot, and two, they’re built to turn random noise into an image plausibly described by the prompt.

If I ask DALL·E 2 for a bar chart showing the number of properties for sale in Seattle by bedroom count, it returns this:

It’s immediately obvious that this chart is not based on any actual real-world data. DALL·E didn’t read the data and carefully plot accurate lines, labels, and values. Instead, the model related my prompt to a glossy infographic. “Bar chart” leads to colorful vertical lines and two axes. “Seattle” ends up in the title. Bed, side table, and lamp illustrations as suggested by the word “bedroom.” An interesting result, possibly useful as inspiration, but ultimately junk.

So how can we go from the bad example to something more like this?

Charting requires precision and direct access to the underlying data. There are plenty of tools and libraries allowing us to create charts using programming languages like Javascript, languages that can conveniently also be used to transform and prepare the data that we’re charting.

Luckily for us, programming languages are just text, and AI models are really good at generating text. By guiding a model to generate text that is specifically Javascript, and to use a popular charting library like Chart.js, it can reliably return code representing the chart we described.

There are a couple obstacles. It’s unreasonable to simply drop the data into the model prompt. Doing so would put an artificial ceiling on the amount of data we can visualize because each model has a maximum context size. These values are far, far lower than the limits imposed by browser performance or computer hardware.

Loading a model’s prompt with a bunch of data not only increases the chance that it will mangle the values when generating code, it’s also pretty expensive. Model inference is priced as a function of prompt size, so it would be pretty pricey. If you’re running your project locally, you’re probably not worried about billing issues, but providing all the data in the prompt would still greatly reduce performance and open the door for inaccuracies.

Instead, we can describe the data to the model by providing its schema. The model now knows which properties each element of the set has and how to access them — all without seeing the actual values. This is especially important if you’re concerned about exposing sensitive information to third-party services.

Another obstacle: chat history. It’s one thing to be able to describe a chart and get a result in a few seconds, but it’s much more powerful to be able to have a back-and-forth to refine the results. We need to be able to inject the previous prompts and responses to the model to provide context in what’s already happened, so you can implicitly refer to previous responses without having to write an exhaustive prompt specify everything you want to see in the next response.

I work at Griptape, so it’s only natural that I would host this experiment on Griptape Cloud, our platform for building and hosting AI applications like this one. It just so happens that configuring the assistant managing interactions with the underlying model with a thread will provide me with exactly this: a conversation history that I can tie to a user session on the browser front-end. Configuring my assistant with rulesets allows me to address a few other issues, including how AI models are chatty and want to bury code responses in cheery prose, wrapped in Markdown. These also provide a convenient place to describe the data schema, request Javascript code, and tell the model to use Chart.js instead of other libraries.

There’s a thin layer of business logic hosted in an AWS Lambda managing things like creating new threads and polling for assistant run completion, making the front-end development experience very simple. Make a request, receive code, evaluate it.

The result is a really cool tool that allows a user to almost have a conversation with data — to ask for a visualization, filter the data, revise the result, and iterate until they’re satisfied. Once they’re happy with the result, they download the chart or view and copy the code defining the transformation and configuration to recreate this chart elsewhere.

Here are a few examples:

Iteratively building a chart showing average price as a function of distance from the Space Needle.

Visualizing home affordability across a range of incomes and number of bedrooms.

Play with it yourself here.