Lee Dong-ik, who was an engineering student at Seoul National University, did not know what to expect from a machine learning course he took for fun. This very class later became a start to his future career and making a ground-breaking artificial intelligence-based tool.

“I did not know I would end up becoming an AI developer because I was majoring in engineering, not AI,” Lee told 4i-mag. “While I was studying technologies to control robots during my graduate programme, my supervisor had founded a start-up of machine learning-based products. This sparked my interest [of AI again] and led me to work in the related fields.”

As of today, Lee is working as an AI researcher at a Seoul-based research centre Saige Research. However, the AI-based product that he developed and received a huge response from mainstream media outlets recently has nothing to do with his full-time job.

It was a tool based on his graduate course dissertation, designed to change real human’s faces into cartoon representations.

AI Cartoon Artist

Many people would have dreamed of becoming a cartoon character or being a part of their favourite animation or cartoons. Lee’s tool can realise those people’s imaginations with the power of science and technology.

The tool — trained with images of online cartoons uploaded on a South Korean platform Webtoon — captures distinct characteristics of real human faces and recreates them into various cartoon styled representations.

“The tool is built on generative image modelling technology,” Lee said. “The tool is trained with images of cartoons to transform real humans to have attributes of cartoon characters.”

The technology used in the tool is called the generative adversarial network, or GAN for short, a deep-learning technology that generates fake images by using two sub-models. The first sub-model is the generator model, which is trained with examples, and the second one is the discriminator model, which decides whether those examples look original (or real) or generated (or fake).

The generator’s goal is to fool the discriminator by making plausible examples. The interaction between the two sub-models goes on until the discriminator says that more than 50 per cent of the provided fake examples are real. This means the generator is creating original-like examples.

The Start of AI Cartoon Artist

Lee explains that his idea to develop the tool had first started with scholastic curiosity. The basic principles and the functioning model of the tool, which runs on Python, was first introduced in his paper published after his graduate programme.

“At the end of my graduate programme, I was writing a dissertation on generative image modelling technology,” he said. “Then I started to wonder to what extent can this ultramodern technology be applied.”

When 4i-mag asked Lee why he chose cartoon images over any other datasets in testing the applicability of his tool, he replied that he did not want to use the “same conventional datasets that have been adopted by many other researchers already”

One of the most successful cartoon styles that he could recreate was Malnyun Series on Webtoon. “I was a big fan of this cartoon series,” Lee was quoted as saying by Dong-A Science. “This cartoon also had quite distinctive styles of art, so I wanted to try recreating it by using AI.”

As his recreation of Malnyun Series cartoons received a huge reaction from the public, Lee felt obliged to ask the original cartoon artist to settle down any concerns of copyright infringement.

“I sent an email to Lee Malnyun [the original artist], saying that I am sorry for any inconvenience that I may have caused with the use of his artworks in my research without his consent,” he told Dong-A Science. “But the artist replied that he enjoyed my works as well and it was okay to use his works as the purpose of the use was for research.”

Future Prospects

Lee says he currently does not have any plans for the official publication of the tool, although there would be many people who would like to try it out.

“This project of mine remained at a hobby-level, so currently there is no plan to introduce it to the market,” Lee said. “I am interested in using AI technology in the creation and utilisation of visual content, so I will continue to try and build many other similar tools relevant to the industry.”

Especially for the online cartoon industry, Lee believes there are many sectors where AI tools can be a significant help.

“I think technological tools can improve the productivity of cartoon artists — for example, colouring, refining the pen drawings, and changing the styles,” he said. “Collecting samples of photos or drawings that artists can get inspiration from or creating other forms of artworks based on the cartoons, such as short-formed animations, can be also done with the help of AI.”

Lee added that AI tools may be soon able to carry out all steps of the cartoon-making process, from writing the storyline to drawing continuity drafts.

“For instance, a tool that writes a story on its own or a model that transforms text-based content into images has already been out in the market for a while,” Lee said.

Not the Same, but Similar

Lee’s product is not the only AI tool that makes or draws cartoons.

In 2018, a group of researchers from the University of Illinois and the Allen Institue for Artificial Intelligence came up with an AI model that makes cartoons based on text descriptions. The developed tool, dubbed CRAFT (Composition, Retrieval, and Fusion Network), is trained with over 25,000 animation clips and creates a set of parameters of items, humans, and backgrounds among others.

For example, if the user types in the description of a character ‘A’ doing a certain action in a certain set of time and space, the tool gives back the result based on what it has learned from its clip dataset.

Although the users have to label correctly to make the outcome more correct and the returned animation clips still look raw rather than completed, the research showed the usability and potentials of AI tools in making simple animations.

Compared to CRAFT, an AI tool named Cartoonizer is more similar to Lee’s work. Cartoonizer, an app using a graphics processing unit for the video inference and central processing units to fetch the images, is based on a research paper published by the University of Tokyo researchers in 2020.

The app turns any photos and videos of real humans or settings into cartoon-like illustrations. The demo version of this app, which is available online, currently can be used for free for up to 10-second videos and any photos in jpeg and png formats. The operators think that Cartoonizer can be used in making simplistic arts or basic sketches of artworks.

Also last year, another AI tool dubbed Toonify has taken the internet by storm, with its function to “toonify” humans. Toonify, developed by software engineers Doron Adler and Justin Pinkney, also uses GAN to change real human faces into cartoon-like representations, as Lee did for his tool.

The tool, which is available here, can change people’s faces in many styles of animations and cartoons. Using this app is not completely free — asking users to pay from £3 to £25 depending on the number of photos they want to transform. However, the website still is visited by thousands of visitors every day, showing what contributions can AI make in the process of creating cartoon artworks.

Who knows, at this pace, our next generation cartoon artists can be all replaced with computers.