2. Machine intelligence Narrows the language gap between tools and intent

Mind the gap

There’s a language gap present in today’s creative landscape. One that separates our creative applications from our creative intent. The gap is the reason we are required to on-board, the reason we find ourselves disoriented when a button moves after a software update, and the reason we watch online videos to learn features our tools have in fact offered for years. We accept the gap, because the historic nature of computers—binary execution of pre-programmed functions—has caused it to always be there. This gap demands that users speak like a computer so that they might effectively use a computer.

Exposing our creative limitations

Today’s creative applications could be described as a collection of digital tools: a digital paintbrush, a digital typesetter, a digital brightness control, et cetera. Together, they deliver an experience that asks users to translate their creative intent into a sequence of tools. For example: to “highlight the colors of the sky” in an application such as Photoshop might translate into the following tool sequence: Magnetic Lasso Tool → Hue/Saturation Tool → Brightness/Contrast Tool.

There are multiple tool sequences that could achieve any one of a user’s creative intents. Common sequences can even turn into their own vocabularies and language shortcuts around certain creative tasks. Single tools can also achieve a wide array of outputs depending on how they’re used and manipulated.

The language gap reveals limitations that users persist through as they exercise their intent within creative applications. Here’s three in particular:

• First, that their ability to enact creative intent is limited to the language of tools. Users cannot successfully use an application unless they interact with it using the language of tools.
• Second, that their ability to enact creative intent is limited to their knowledge of the tools available to them, including their ability and availability to learn new tools. If a user’s intent extends beyond their knowledge of the available tools, they must either consult documentation outside of the application, or alter course in the direction of more familiar tools.
• Third, that their ability to enact creative intent is limited to the capability of the tools available to them. If a user’s intent is greater than the function of their tools, they must either change their intent or look for new tools elsewhere.

Inspiring creative sameness

Two decades ago, creative technologist John Maeda was interviewed by The New York Times. He talked about the language gap in relation to Adobe tools:

“The problem is that most people can’t just ‘finish’ things with [an Adobe product]. They have to use it to start them, also… , the styles that emerge are homogeneous because the software is universal. Without being able to know how to program, you can’t break out of the technology.”¹

Maeda recognized a consequence of the gap: that the language of tools can subvert the language of creative intent. Rather than continue translating intent across the gap, users can instead develop a habit of formulating their intent directly in the language of the tool. Maeda describes the outcomes of this phenomenon as “homogeneous styles”, a sameness that emerges when creativity starts at the opening of an application, and is then led only by the tools that the user knows. Maeda’s sentiments are echoed in this common quote:

“We shape our tools, and thereafter, they shape us”.²

Our language of intent

If the language of tools is by nature, binary and mechanical, the language of intent is contextual and human. Where a creative application hears an intent of “select the brush tool and place four brushstrokes on the artboard”, a user might speak that same intent as “turn a chair into a barstool”. Intent can also be vague and multi-layered. Vague, because another user could achieve the same outcome with “make the seat taller”. Multi-layered, because the task might contribute to a greater intent of “re-designing a restaurant interior”.

When directing his junior designers, graphic design luminary Massimo Vignelli would famously describe a visual layout as needing to “bang” or have “reverence”.³ His intent could simultaneously be classed as human, contextual, vague, and multi-layered all at the same time, where his designers could only learn the true meaning of the direction through experience. It highlights the multi-dimensional complexity of creative intent, and raises question as to whether a creative tool is capable of accepting and facilitating bang in the way Vignelli meant? Will technology allow for the language gap to be narrowed?

Narrowing the gap with machine intelligence

At the beginning of this essay, computing was described as a historically “binary execution of pre-programmed functions”. In this style of computing functions are rules-based, and rules are pre-programmed into software before their release. The resulting functions are defined, engineered, and inflexible to adaptation.

Machine learning offers an alternative style of computing that takes a different approach. Here, intelligent computing is still rules-based, but the rules are not pre-programmed. Instead, machine intelligence infers what the rules should be after recognizing patterns and relationships in a dataset. This method allows intelligent functionality to be adaptive to the individual user, following rules that are calculated to best meet each individual request. This logic allows a pair of users to give the same search term and receive different results, or a single user to input the same search term at different points in time and be given different results. In both instances, a greater number of inputs beyond the single-given search term are utilized in calculating the most relevant result.

Through contextual learning and dynamically refined outputs, machine intelligence offers to learn the language of the user, and in doing so observe the subject and context of a user’s content. In recognizing the human language of intent, users are empowered to engage their creative applications without first needing to learn the language of tools, and creative applications are empowered to “highlight the colors of the sky” or “turn a chair into a bar stool”.

Above and below are visualizations of “turn a chair into a bar stool” and “highlight the colors of the sky”. The first image shows the steps taken when translating creative intent into the language of tools. The second suggests what’s possible as the language gap is narrowed.

Narrowing already

We’re already starting to see the language gap narrowing in today’s creative applications. Below, Photoshop’s “Face-Aware Liquify” feature allows for the semantic editing of human faces.⁴

Thinking beyond “smart” tools

There’s a language gap present in today’s creative landscape. One that separates our creative applications from our creative intent. But advances in machine intelligence are narrowing the gap and allowing us to reimagine experiences of the next creative wave. These applications won’t just offer a collection of enhanced “smart” tools. They’ll offer a deeper dialogue between a user and their creative intent. They’ll be active facilitators in the creative process and of our creative navigation. They’ll bring new creative possibilities for application users and application makers alike. They’ll bring challenges too.