How DALL-E could power a creative revolution

Programming note: I’ll mention this again next week, but giving you a heads up that I’ve added Platformer’s summer break to the posting schedule. After next week, I’ll be off the following two weeks to recharge and work on some longer-term projects. As always, thanks for your help in making this one-person shop a sustainable endeavor.

Also: all images in this post were generated using artificial intelligence.

I.

Every few years, a technology comes along that splits the world neatly into before and after. I remember the first time I saw a YouTube video embedded on a web page; the first time I synced Evernote files between devices; the first time I scanned tweets from people nearby to see what they were saying about a concert I was attending.

I remember the first time I Shazam’d a song, summoned an Uber, and streamed myself live using Meerkat. What makes these moments stand out, I think, is the sense that some unpredictable set of new possibilities had been unlocked. What would the web become when you could easily add video clips to it? When you could summon any file to your phone from the cloud? When you could broadcast yourself to the world?

It’s been a few years since I saw the sort of nascent technology that made me call my friends and say: you’ve got to see this. But this week I did, because I have a new one to add to the list. It’s an image generation tool called DALL-E, and while I have very little idea of how it will eventually be used, it’s one of the most compelling new products I’ve seen since I started writing this newsletter.

Technically, the technology in question is DALL-E 2. It was created by OpenAI, a 7-year-old San Francisco company whose mission is to create a safe and useful artificial general intelligence. OpenAI is already well known in its field for creating GPT-3, a powerful tool for generating sophisticated text passages from simple prompts, and Copilot, a tool that helps automate writing code for software engineers.

DALL-E — a portmanteau of the surrealist Salvador Dalí and Pixar’s WALL-E — takes text prompts and generates images from them. In January 2021, the company introduced the first version of the tool, which was limited to 256-by-256 pixel squares.

But the second version, which entered a private research beta in April, feels like a radical leap forward. The images are now 1,024 by 1,024 pixels, and can incorporate new techniques such as “inpainting” — replacing one or more elements of an image with another. (Imagine taking a photo of an orange in a bowl and replacing it with an apple.) DALL-E has also improved at understanding the relationship between objects, which helps it depict increasingly fantastic scenes — a koala dunking a basketball, an astronaut riding a horse.

For weeks now, threads of DALL-E-generated images have been taking over my Twitter timeline. And after I mused about what I might do with the technology — namely, waste countless hours on it — a very nice person at OpenAI took pity on me and invited me into the private research beta. The number of people who have access is now in the low thousands, a spokeswoman told me today; the company is hoping to add 1,000 people a week. 

II.

Upon creating an account, OpenAI makes you agree to DALL-E’s content policy, which is designed to prevent most of the obvious potential abuses of the platform. There is no hate, harassment, violence, sex, or nudity allowed, and the company also asks you not to create images related to politics or politicians. (Here it seems worth noting that among OpenAI’s co-founders is Elon Musk, who is famously mad at Twitter for a much less restrictive set of policies. He left its board in 2018.)

DALL-E also prevents a lot of potential image creation by adding keywords (“shooting,” for example) to a block list. You’re also not allowed to use it to create images intended to deceive — no deepfakes allowed. And while there’s no prohibition against trying to make images based on public figures, you can’t upload photos of people without their permission, and the technology seems to slightly blur most faces slightly to make it clear that the images have been manipulated.

Once you’ve agreed to that, you’re presented with DALL-E’s delightfully simple interface: a text box inviting you to create whatever you can think of, content policy permitting. Imagine using the Google search bar like it was Photoshop — that’s DALL-E. Borrowing some inspiration from the search engine, DALL-E includes a “surprise me” button that pre-populates the text with a suggested query, based on past successes. I’ve often used this to get ideas for trying artistic styles I might never have considered otherwise — a “macro 35mm photograph,” for example, or pixel art.

For each of my initial queries, DALL-E would take around 15 seconds to generate 10 images. (Earlier this week, the number of images was reduced to six, to allow more people access.) Nearly every time, I would find myself cursing out loud and laughing at how good the results were.

For example, here’s a result from “a shiba inu dog dressed as a firefighter."

And here’s one from “a bulldog dressed as a wizard, digital art."

I love these fake AI dogs so much. I want to adopt them and then write children’s books about them. If the metaverse ever exists, I want them to join me there.

You know who else can come? “Frog wearing a hat, digital art."

Why is he literally perfect?

Over on our Sidechannel Discord server, I began taking requests. Someone asked to depict “the metaverse at night, digital art.” What came back, I thought, was suitably grand and abstract:

I won’t attempt to explain here how DALL-E is making these images, in part because I’m still working to understand it myself. (One of the core technologies involved, “diffusion,” is explained helpfully in this blog post last year from Google AI.) But I have been repeatedly struck by how creative this image-generation technology can seem.

Take, for example, two examples shared in my Discord by another reader with DALL-E access. First, look at the set of results for “A bear economist in front of a stock chart crashing, digital art.”

And second, “A bull economist in front of a graph of a surging stock market with up line, synthwave, digital art.”

It’s striking the degree to which DALL-E captures emotion here: the fright and exasperation of the bear, and the aggression of the bull. It seems wrong to describe any of this as “creative” — what we’re looking at here are nothing more than probabilistic guesses — and yet they have on me the same effect that looking at something truly creative would.

Another compelling aspect of DALL-E is the way it will attempt to solve a single problem in a variety of ways. For example, when I asked it to show me “a delicious cinnamon bun with googly eyes,” it had to figure out how to depict the eyes.

Sometimes DALL-E added a pair of plastic-looking eyes to a roll, as I would have done. Other times it created eyes out of negative space in the frosting. And in one case it made the eyes out of miniature cinnamon rolls.

That was one of the times I cursed out loud and started laughing.

III.

What becomes of a technology like this?

DALL-E is the most advanced image generation tool I’ve seen to date, but it’s far from the only one. I’ve also experimented lightly with a similar tool named Midjourney, which is also in beta; Google has announced another, named Imagen, but has let to yet outsiders try it. A third tool, DALL-E Mini, has generated a series of viral images over the past few days; it has no relation to OpenAI or DALL-E, though, and I imagine the developer will get hit with a cease-and-desist letter shortly.

OpenAI told me that it hasn’t yet made any decisions about whether and how DALL-E might someday become available more generally. The point of the current research beta is to how people use this technology, adapting both the tool and content policies as necessary.

And yet already, the number of use cases artists have discovered for DALL-E is surprising. One artist is using DALL-E to create augmented reality filters for social apps. A chef in Miami is using it to get new ideas for how to plate his dishes. Ben Thompson wrote a prescient piece about how DALL-E could be used to create extremely cheap environments and objects in the metaverse.

It’s natural, and appropriate, to worry about what this sort of automation might do to professional illustrators. It may well be that many jobs are lost. And yet I can’t help but think tools like DALL-E could be useful in their workflows. What if they asked DALL-E to sketch out a few concepts for them before they got started, for example? The tool lets you create variations of any image; I used it to suggest alternate Platformer logos:

I’ll stick with the one I’ve got. But if I were an illustrator, I might appreciate the alternate suggestions, if only for the inspiration.

It’s also worth considering what creative potential these tools might open up for people who would never think (or could afford) to hire an illustrator. As a kid I wrote my own comic books, but my illustration skills never progressed very far. What if I could have instructed DALL-E to draw all my superheroes for me instead?

On one hand, this doesn’t seem like the sort of tool that most people would use every day. And yet I imagine that in the coming months and years we’ll find ever-more creative applications of tech like this: in e-commerce, in social apps, in the home and at work. For artists, it looks like it could be one of the most powerful tools for remixing culture that we’ve ever seen — assuming the copyright issues get sorted out. (It’s not entirely clear whether using AI to generate images of protected works is considered fair use or not, I’m told. If you want to see DALL-E’s take on “Batman eating a sandwich,” DM me.)

I suspect we’ll see some harmful applications of this tool as well. While I trust OpenAI to enforce strong policies against the misuse of DALL-E, surely similar tools will emerge and take more of an anything-goes approach to content moderation. People are already creating malicious, often pornogaphic deepfakes to harass their exes using the crude tools available today; that technology is only going to get better.

It’s often the case that, when a new technology emerges, we focus on its happier and more whimsical uses, only to ignore how it might be misused in the future. As thrilled as I have been to use DALL-E, I’m also quite anxious about what similar tools could do in the hands of less scrupulous companies.

It’s also worth thinking about what even positive uses of this technology could do at scale. When most images we encounter online are created by AI, what does that do to our sense of reality? How will we know what anything we are seeing is real?

For now, DALL-E feels like a breakthrough in the history of consumer tech. The question is whether in a few years we’ll think of it as the start of a creative revolution, or something more worrisome. The future is already here, and it’s adding 1,000 users a week. The time to discuss its implications is now, before the rest of the world gets its hands on it.

Have a DALL-E image request for me? Paid Platformer subscribers can come on into our Sidechannel, where I’m taking requests in the #daily-dall-e channel. I’ll get to as many as I can — time, content policies, and GPU usage permitting. (OpenAI limits testers to 50 requests per day.)

Governing

• Meta was hit with at least eight lawsuits this week alleging that the company’s recommendation algorithms led to addiction and self-harm in young people. (Malathi Nayak / Bloomberg)
  • I can’t tell if this story is among the eight, but here’s another lawsuit alleging that Instagram addiction led to eating disorders and thoughts of suicide. Notably, it cites last year’s Facebook Papers as evidence the company knew about the risks. (Jesselyn Cook / NBC)
• People selling guns on Facebook Marketplace can break its rules up to 10 times before being removed from the platform, sparking internal complaints. (Elizabeth Dwoskin and Naomi Nix / Washington Post)
• Activist groups submitted 12 ads to Facebook calling for the murder of different ethnic groups in Ethiopia — and all were approved. The company said it was a mistake. (Barbara Ortutay / AP)
• After a change to Facebook’s recommendation algorithm in 2018, conservative political groups saw outsized increases in their engagement, according to new academic research. The researcher “said he considered that perhaps the Republican messages were just resonating more, but the same pattern wasn’t found on Twitter, and it was consistent on Facebook across over 1,000 Republican pages.” (Brandy Zadrozny / NBC)
• Microsoft made a series of worker-friendly changes, some of them required by state law, including an end to non-compete agreements (in most cases) and to non-disclosure agreements in harassment case settlements. (Todd Bishop / GeekWire)
• A small network of pro-Kremlin creators have seen their audiences surge since Russia’s invasion of Ukraine, particularly on YouTube and Telegram. (Brandy Zadrozny / NBC)
• The Securities and Exchange Commission is investigating whether TerraUSD, the “stablecoin” that plotzed last month, violated federal investor-protection guidelines. If it didn’t violate any regulations, what does that say about the SEC? (Matt Robinson / Bloomberg)

Industry

• ⭐ Meta halted development of a dual-camera smartwatch amid a broader pullback in its augmented-reality hardware product roadmap. (Kurt Wagner / Bloomberg)
  • The company will also stop making Portal smart displays aimed at consumers, in favor of attempting to sell them to businesses. (Janko Roettgers / Protocol)
  • And Meta won’t release what was supposed to be the first version of its AR glasses, and may not commercially release a product for “several years.” (Sylvia Varnham O'Regan / The Information)
• David Mortenson, Meta’s top engineering leader, is stepping down. He’ll be replaced by Santosh Janardhan, a VP of engineering. (Alex Heath / The Verge)
• Meta also began trading under the META ticker symbol today. The stock is down 37 percent since announcing the name change from Facebook. (Subrat Patnaik / Bloomberg)
• Twitter executives told staff to expect a shareholder vote on the Musk acquisition by August. That seems ambitious to me? (Dana Hull, Michelle F Davis, and Maxwell Adler / Bloomberg)
  • Or what if Musk somehow blows up the deal financing, allowing him to get away for $1 billion? (Matt Levine / Bloomberg)
• Apple will handle its own lending for its forthcoming buy now, pay later service, a sign of the company’s growing ambitions in banking. (Mark Gurman / Bloomberg)
• TikTok added new screen time controls to remind users to take a break. (Karissa Bell / Engadget)
• Spotify forecast that its podcasting business will be profitable within one to two years. (Ashley Carman / Bloomberg)
• ApeCoin holders voted for the coin to continue on the Ethereum blockchain despite its high transaction fees, rather than moving to another chain. (Olga Kharif / Bloomberg)
• Seth Green got his Bored Ape back. And it only cost him (another) $300,000. (Sarah Emerson / BuzzFeed)

Those good tweets

Talk to me

Send me tips, comments, questions, and DALL-E requests: casey@platformer.news.