Introducing Artifacts
A step forward in structured output generation.
This is an analysis of the system prompt generation for Claude 3.5 Sonnet. The link to the code for this analysis is available at the end of this article published alongside the source. The main part of this analysis regards the introduction of artifacts, and how this might work as part of Anthropics' categorization and retrieval system that is currently beating all LLM benchmarks for code generation and other tasks.
Artifacts are for substantial, self-contained content that users might modify or reuse.
An artifact is a paradigm change because it formalizes a new concept. The concept of persistent data. An artifact is a base object or example for each task the LLM is going to complete. By persisting data, we are able to iterate and remember changes.
By providing fixed references, we unblock the ability to incrementally improve and refine output. This is a step towards controlling the ephemeral nature of verbose LLM output and generating standarized output of a higher quality.
Autocomplete and diff
An expert foundation for Generative AI.
Some of the inherent problems with Generative AI for functional tasks such as code completion is that they can often repeat entire files for simple changes. Othertimes they will shift from one coding style to another within the same tasks. This is a problem of focus. As such, there is a need for a ‘diff’ feature, where we compare the difference between before and after as opposed to repeating the same content in each response.
Artifacts serve a dual purpose; first they act as a reference point for how and where we need output. This is like the setting of the scope. This will help stop the LLM from losing focus of the original problem and also keeps persistent structure and categorization in the output.
As a follow on, we also have an autocomplete feature. We have now directed our LLM to focus on a specific task or problem in an opinionated and curated way. Artifacts provide the entire work in progress to the prompt. Any engineer who has accidentally wiped their code with "Rest of code here" thanks you.
We can see the setting of the scope here:
Self-contained, complex content that can be understood on its own, without context from the conversation
We are shifting focus to a concrete artifact. It is worth noting the explicit instruction to ignore the context from the conversation. This is a method of ensuring quality by reference to curated data. It is a quality control mechanism that controls the verbose and potentially random characteristics of the input.
Here is an example of some of the exclusion criteria:
Content that is dependent on the current conversational context to be useful.
Content that is unlikely to be modified or iterated upon by the user.
Request from users that appears to be a one-off question.
All of this fits nicely inside an architecture for retrieval. By having a library of curated artifacts, we can now direct our system to retrieve from a controlled dataset.
We know that all AI developers are focusing heavily on investing in high quality curated data. Artifacts are a step towards framing verbose input and output with a structure that is standardized and controlled.
The prompt is focusing on the system context as opposed to the user input. So the artifact acts as a concrete reference point both in the generated text, and as structured data behind the scenes. This gives us fast and accurate retrieval, and focus. Something that is very helpful for...
Thinking
Logical thinking is a key part of the generation process.
Prompt engineers have long been telling us that one of the keys to reliable output is guiding LLMs to form logicial multi-step generation processes. We see formal recognition of this in the prompt.
1. Briefly before invoking an artifact, think for one sentence in <antthinking> tags about how it evaluates against the criteria for a good and bad artifact. Consider if the content would work just fine without an artifact. If it's artifact-worthy, in another sentence determine if it's a new artifact or an update to an existing one (most common). For updates, reuse the prior identifier.
Here we are obligating our system to take a structured multi-step process to analyse the task and the output. Again, moving towards a strong casting of otherwise unstructured verbose content and alluding to a search and retrieval system for artifacts.
<antthinking>Creating a Python script to calculate factorials meets the criteria for a good artifact. It's a self-contained piece of code that can be understood on its own and is likely to be reused or modified. This is a new conversation, so there are no pre-existing artifacts. Therefore, I'm creating a new artifact.</antthinking>
<antthinking>This request is a direct modification of the existing factorial-calculator artifact. It's not a new artifact but an update to make the script more robust. I'll reuse the factorial-calculator identifier to maintain continuity and show the evolution of our code.</antthinking>
We can see the implementation of a logical thought process for the generation of defined outputs. By ensuring that our algorithm goes through the same logical steps, we have the seeds of an intelligent and repeatable generation process. Defined thought processes, e.g. thinking.
We can compare this logical approach to the thought process of a person. First of all we have a rational problem solving approach. The LLM data set is the brain, but artifacts are the skills and knowledge enabling us to arrive at a certain output of a certain quality. They are our skills helping to guide the output.
If we imagine all the competing LLM currently being released, we can now derive that they are relying on their own intepretation and implementation of logical thought processes. LLMs are essentially creating a system to mimic the logical thought process of a human. We are building the missing parts, the knowledge, memories and retrieval processes that fuel the brain.
This makes systems prompts and instructions incredibly valuable assets. The understanding and refinement of "logical thinking" is a key part of the generation process.
We can see some basic implementations of this structured thinking in the code...
Identifiers and Search
Search and retrieval of artifacts is a key part of the system prompt.
<antartifact identifier="factorial-script" type="application/vnd.ant.code" language="python" title="Simple Python factorial script"> def factorial(n): if n == 0: return 1 else: return n * factorial(n - 1)
So what is application/vnd.ant.code. Application is simple enough, VND is vendor, ANT will be Anthropic (the creators of Claude Sonnet 3.5) and code; that's an insight into their architecture. I would expect some kind of taxonomy and structured data that lists the tasks that people trying to achieve with LLMs.
We could for example create some psuedo code for an attempt at a powerpoint presentation.
<antartifact
identifier="powerpoint-presentation"
type="application/vnd.ant.presentation"
purpose="business"
title="Simple powerpoint presentation">
Slide 1: Title slide
Slide 2: Introduction
Slide 3: Problem statement
Slide 4: Solution
</antartifact>
This is almost certainly nothing like the production code, but an interesting mental paradigm. To control and structure verbose output, we have to encounter logical and rational processes for categorizing and standardizing the input and output.
I suspect this means that when inputs come in, they run separate battle-hardened algorithms that run entity extraction and categorization. This structured data is then run through an asset search and retrieval process. Where for text we use vector databases; for other defined outputs we have now introduced this concept of artifacts. For example, a React Code task could go something like this.
"INPUT: Create a react component for a metrics dashboard",
"ENTITY_EXTRACTION: Coding, React, Metrics Dashboard",
"ENTITY_SEARCH: Retrieve code artifacts for Metrics Dashboard where type = React",
"SYSTEM_PROMPT: create_system_prompt(artifact_id='metrics-dashboard-component', type='application/vnd.ant.code', language='react')"
There is a lot going on, and we can see the hard yards that are needed behind the scenes to curate high quality examples and taxonomies for what is essentially an unlimited theoretical pool of tasks.
But it is at its core, as far we can see, a search and retrieval system, based on a proprietary templating language.
Templating language structure
A rendering template that will shift based on input variables
I started my career many years ago as a Drupal developer. Reading the prompt, the word that jumped out at me was TWIG. Twig is a html templating language that was commonly used for rendering templates in HTML from PHP.
It looks like Claude Sonnet 3.5 uses something similar, which makes perfect sense. Given the text input to the LLM, we need to systematically generate blocks of text. These are the dynamic tags that are put together to generate the prompt.
This will leverage a kind of function calling approach. Each tag has a specific purpose. This then serves as an abstraction as we direct our model to find the right category and type for each specific purpose. I imagine prompt construction goes something like this. Massively speculative psuedo code incoming…
function generate_reference() {
context=artifact_type, artifact_id, task
examples = search_examples_by_type(‘react’, ‘hooks’, ‘current_ids’)
return structured_data
}
function generate_system_instruction(structured_data) {
<antartifact>
<artifacts_info {{ attribute(version, '1.00') }}>
<% artifact_introduction %>
<% artifact_selection %>
<% artifact_elimination %>
<% artifact_usage %>
<% artifact_instruction {{ type(code, 'data-code') }} {{ type(react, 'data-react') }} %>
</ artifacts_info/>
<example {{ for react_example in react_code_artifact }}>{react_example }</example>
<example_docstring {{ for example_thought_process in curated_artifact_generation_example }}>{example_thought_process }<example_docstring />
<user_query>{sanitized_user_query}</user_query>
<assistant_response>{sanitized_user_query}</assistant_response>
</antartifact
}
So there we have it, a thought process broken into blocks. Entity extraction mapped with advanced search and retrieval. The building blocks for a logical thought process. The underpinning data is key to the quality of the output.
Conclusion
One small artifact for Claude, a giant leap for AI.
Artifacts are to structured output what vector search is to rag. It is the search and retrieval system for Generative AI, the memories for our LLM brain.
As we read this blog, we can imagine armies of developers and marketeers, building libraries of curated artifacts. This content library is the fuel that is driving Generative AI. This library is accessed via classification and then search and retrieval tasks.
By working with artifacts we have reference points that exist beyond the ephemeral. Ones that can be refined and re-reused. We already had thought and verbose output. Now we've got memories and expertise...
Claude 3.5 system
The system prompt in full
