AI Layout: The dream of dynamic flexibility

More Flexible Dream

A Big fan of Auto Layout

I've always been a big fan of Figma's Auto Layout.

Before the launch of Auto Layout, resizing designs was a manual task. For instance, when you're designing two buttons and you want to make the text in one of the buttons longer, you'd have to manually resize that button to to accommodate the extra characters. Then, you'd move the other button a certain distance in the same direction to keep the spacing consistent.

It is tedious, but it's a repetitive task that designers often need to undertake. What's even more frustrating is that when the design content is nearly completed, a small change can require you to manually adjust many layers, and you have no choice.

Therefore, when I first learned about Auto Layout and saw that buttons with Auto Layout can resize with their text, it was truly exciting. I will always be grateful to Figma for opening the door to a new flexible world for designers.

However, as designers continue to use Auto Layout in their designs, many have become believers of Auto Layout. They prefer to use Auto Layout in their designs, which has caused some dilemmas.

• When the current Auto Layout does not support the adjustments you want, we could only manually remove it layer by layer. At this point, Auto Layout has turned into a constraint for freeform design.

• When designers collaborate on design files, their habits of adding Auto Layout can differ among individuals. They often have to inspect the Auto Layout properties layer by layer before beginning work on that file. Alternatively, you may have to choose to remove all Auto Layout added by others.

It was not until one day, when I saw Sho Kuwamoto's presentation on Opening Keynote - Conference Kickoff - Sho Kuwamoto (Schema 2021) at Schema 2021, that I was inspired. Drag & Drop represents Free-form design, while the use of Auto Layout represents Structured Design. The correct way to work involves flexibly using both methods according to work needs. It is not advisable to prefer Auto Layout for everything because it makes edits easier.

But a question emerged in my mind. It's certain that designers have the option to decide whether to add Auto Layout to layers, thus categorizing them as Free-form or Structured. However, it seems that categorizing into Free-form and Structured isn't dictated by the content of the layers. Instead, it's more influenced by the adjustments that are intended to be made.

For instance, when a designer is exploring multiple layout changes for a card, the card should be categorized as Free-form, allowing for more free-form exploration. However, if the designer wishes to make some “structured” adjustments, such as adjusting the spacing of layers within the card, then the card should be categorized as Structured.

I suddenly realized that what designers want is the flexibility to switch between Free-form and Structured based on the adjustments that are intended to be made.

Before We Decide To Set Off

After the initial inspiration, the idea of creating a flexible switch between Free-form and Structured design was prioritized.

We have envisioned a preliminary concept. Imagine entering a temporary structured mode where all edits become structured adjustments. In this mode, you'd have the flexibility to adjust spacing and padding, reorder and duplicate items, among others. This enables you to create designs that expand or contract to fit their contents, and reflow as their contents change. After completing these edits, you exit this mode, and everything switches back to Free-form Design, where you can drag & drop designs. Doesn't that sound cool?

However, the challenge we face is that if we want to create a flexible switch between Free-form and Structured, we need the capability to automatically generate Auto Layouts that can take us into the world of Structured design.

Despite the complexity of layouts, they can actually be abstracted into patterns of alignment, spacing, and padding values between layers. Learning these precise, abstracted rules is something AI can achieve.

Therefore, we decided to start building AI Layout. Once AI has learned from enough pages and recognized enough layouts, it can apply structure to designs. These tasks, accomplished by AI, are not different from those undertaken by designers who invest significant time and effort. AI can serve as a bridge between Free-form Design and Structured Design.

This was the beginning of our journey to create the AI Layout.

Building AI Layout

From the presentation Designing Auto Layout V4 by Joel Miller and Oscar Nilsson at Figma's Config 2022, I gained more insight into how users are utilizing the Auto Layout feature:

• In Figma, only 20% of users use Auto Layout for most of their work, according to data from Config 2022—though this percentage may be higher now.
• The barrier to use auto layout is high, making it difficult for many designers to understand complex layout structures, especially those created by others. As a result, many designers completely remove auto layout just to make simple edits.

Therefore, we have defined the profiles of two user groups:

• Users who do not use Auto Layout: They are not very familiar with its properties and rules. We aim to provide them with a Smarter Temporary Structure to boost efficiency through AI Layout.
• Auto Layout users: They are familiar with the properties and rules of Auto Layout. When they need to make temporary structural adjustments to designs that have not previously had auto layout added, AI layout allows them to make these adjustments with a single click without having to add auto layout layer by layer.

Determining the Scope of Effect

Before building AI Layout - AI builds temporary structure, the first thing we need to do is determine the scope of effect.

Initially, my intuition was that applying effects to selection would offer users more control and a superior experience, so we then created a prototype based on this approach. However, in practical experience, we discovered some problems with it. For instance, when we wanted to adjust the spacing or padding of the first card on the following screen.

• For Auto Layout users, familiarity with the properties and rules of Auto Layout simplifies tasks like those depicted below. They can select the first card along with any other layers that will be adjusted. Upon entering a temporary structure mode, they are then able to make the desired edits.

• However, for users who do not use Auto Layout, everything becomes less clear. Their instinct is often to select the card they want to edit. However, when they attempt to edit a card, they find that the content below the card does not keep spacing and move down as expected. Instead, it starts to overlap the content below. This indicates that the adjustment is still operating in a design without Auto Layout.

We ultimately decided that when users enter temporary structure mode, AI will identify the entire screen that should be effected based on the layers selected by the user, building temporary structure. In this way, users who do not use Auto Layout no longer need to understand the rules of how Auto Layout affects other layers. Meanwhile, for Auto Layout users, as long as the AI results are accurate, they can still make the adjustments they desire.

Under AI Layout

AI builds temporary structure:

We've implemented “AI builds temporary structure” by using AI to automatically create Auto Layouts, functioning within the scope of the entire screen. This allows you to quickly perform adjustments, such as adjusting spacing and padding, reordering, and duplicating items, without having to manually add Auto Layout layer by layer.

After you've made your structured adjustments, you simply remove the temporary structure with one click. The design is back to Free-form design, unrestricted by the temporary structure.

From then on, switch freely between Free-form and Structured design.

Additionally, we offer AI adds auto layout:

With the capability for AI to automatically create Auto Layout, it seems feasible to introduce a feature that allows for quick generation of Auto Layout. This would facilitate a smoother transition for users from draft versions to reusable designs.

Now, you just need to select the Frame you want to add Auto Layout to and click “AI adds auto layout”. AI will then assist in generating the Auto Layout and create new Frames within it.

We're thrilled to offer this gift to Auto Layout users and hope you will enjoy it.

Typical Usage Scenarios:

Through testing with different users, we gradually discovered the best practices:

• Users who do not use Auto Layout, when you need to quickly adjust spacing and padding, reorder and duplicate items, you can use "AI builds temporary structure". With the help of AI, you can complete these tasks, replacing the previous step-by-step adjustments.
• For Auto Layout users, we hope AI Layout can help you move beyond the default preference of using Auto Layout for all designs.
  • For components, repetitive content, or reusable modules that require an Auto Layout, you can now seamlessly accomplish this by using the “AI adds auto layout”.
  • For content that may change later, you can retain its Free-form mode. If there's a need for structured adjustments later, you can quickly accomplish this through "AI builds temporary structure".

In the Unseen Aspects: What We've Accomplished

The Challenge of AI

Creating Auto Layout with AI may seem simple, but it heavily relies on AI's understanding of the structure of designs. Designers often do not meticulously organize layers during the design process, making it initially challenging for AI to structure the designs.

To tackle this problem, we utilized two main kinds of information to assist AI:

• The overlay, containment, and alignment relationships between layers, groups, and frames in the designs.
• The frames relationships, similarities between frames and groups, and layers within frames in the designs.

With this auxiliary information, the accuracy of AI in identifying design structures significantly improved.

However, when dealing with designs with Auto Layout, how the related layers should respond to changes in certain content parts and how to maintain alignment relationships became new challenges for AI.

We collected a large amount of layout data to train and evaluate the model, gradually achieving the results we anticipated.

Fine Tunings

A core principle of AI builds temporary structure is that the designs before and after using AI Layout should look identical. That is, although layers are added to Auto Layout, the process does not alter the original spacing, alignment relationships, and visual effects of the layers.

Therefore, the first issue we needed to address was for non-frame users more accustomed to using "rectangle + group" to build designs. We needed AI Layout to have the ability to convert "rectangle + group" into frame. The rules here are mostly similar to those during the creation of Auto Layout, as shown below.

However, the world of design is not so simple, and in some cases, the above rule can lead to dramatic changes in the designs. For example, when you want to transform a "rectangle + group" that has effects applied to the rectangle, if you convert it into frame, the original effects of the rectangle will apply to all layers within the new frame. Similarly, if the rectangle layer's opacity is not 100%, the opacity would also affect the layers within the new frame after the conversion.

Therefore, we chose a different method to create Auto Layout—setting the rectangle to absolute positions and applying its constraints to scale, scale, so it can always change in size along with the frame's size changes. Then, by creating an Auto Layout frame for the other layers, we ensure the design looks identical to the previous designs.

Upon completing the temporary structural adjustments and exiting this mode, all designs will revert back to their pre-mode state. This ensures that you can proceed with your design work smoothly.

Furthermore, we encountered a problem: in AI Layout, if designers deliberately created 1px or 2px misalignments in some design draft content, they wouldn’t want these deliberately misaligned layers to be forcibly aligned due to Auto Layout's alignment rules.

We added many temporary frames, allowing the original misaligned layers to maintain alignment with Auto Layout rules within the new temporary frames, and changing size with content changes.

After refining these aspects, we are delighted to have adhered to the principle that "the content before and after using AI Layout should look identical," and we're eager to see it in production.

The Future of AI Layout

As you explore the feature set of design tools, you'll notice that for every feature helping you to structure design content, there's a free-form counterpart. For example, you can manage colors with styles, but if you want to leave things unstructured, you can use selection colors to make edits. You can use auto layout to manage designs, but if you prefer not to structure these layers ahead of time, you can use smart selection to adjust spacing and reorder items.

However, structuring designs requires effort and preparation. It undoubtedly makes our repetitive work more efficient, but it's undeniable that our preliminary preparations also involve a lot of sweat invisible to outsiders.

I often encounter two perspectives: one celebrates Auto Layout for significantly improving work efficiency. The other laments how auto layout confines their work within numerous flexible boxes, forever constrained by horizontal and vertical rules.

We don't wish to conform to the latter. We aim to preserve designers' space for inspiration and experiments, rather than confining creativity and imagination within a series of structured "flex box". Through AI, we hope you can enjoy the convenience of structured designs while maintaining a free-form state.

AI Layout represents the beginning of a more flexible dream between freedom design and structure design, and we plan to continue exploring more.