Rastermind:
Controlled AI design paradigm
This workshop dives deep into AI-assisted design particularly in the field of architecture by using advanced AI image-generating methods. It explores the power of controlled image generation as a tool for architectural conceptual iterations that could act as an idea catalyst especially during early design phases.
The combination of Stable Diffusion and ControlNet opens up endless possibilities for architects to harvest the incredible capabilities that Artificial Intelligence has to offer while still having applied solutions that could be used in real-world projects. Those tools could be the stepping stone towards a new design paradigm.
Course overview
Get to utilise the power of AI tools in the design workflow
This workshop will focus on generative AI tools for architecture design using Stable Diffusion and ControlNet. Specifically, we will dive deep into controlled image generation using prompts, sketches, and 3D massing. It will also cover advanced techniques for having more control over the generated images combining different AI models like ControlNet’s scribble, depth, and image segmentation models.
This opens up a whole world of possibilities during the form-finding and idea-initiation phases. We will also cover the basics of training our own models on a specific set of images to have a custom-stylized AI model (LoRA) that can be directly used within the prompt.
And finally, we will touch upon some image keyframe interpolation techniques that can create smooth videos out of AI-generated images. This workshop will provide the skillset that makes the user of those advanced design tools comfortable enough to explore design ideas and directly implement them into real-world design workflows.
What you will learn
- Prompting in Stable Diffusion
- Using textual inversions for negative prompts in Stable Diffusion
- Revit mass to image generation using Stable Diffusion and ControlNet’s MLSD model
- Advanced controlled AI image generation combining different ContolNet models
- Using LoRAs inside prompts for customised image styling
- Iterating consistently using ControlNet’s reference models
- Automatic1111 webUI for Stable Diffusion
- Sketch to image generation using Stable Diffusion and ControlNet’s scribble model
- Rhino mass to image generation using Stable Diffusion and ControlNet’s Depth model
- Inpainting inside a context image with a control sketch using ControlNet’s inpainting and scribble model
- Training custom LoRAs on a custom image set
- Fine-tuning final generated images using upscaling and inpainting techniques
Why it is important
1. Get to utilise the power of AI tools in the design workflow
Staying on top of cutting-edge technology and harvesting the power of artificial intelligence and machine learning to design better and with more freedom.
2. Get intuitive and creative while brainstorming ideas
Using AI image-generating tools as design assistants shifts back the focus of design to things that matter like intuition, composition, lighting, materiality, experimentation, exploration, sustainability, and proportion to name a few.
3. Instantly visualise feasible ideas within real-world context
The speed at which those tools are able to reason and generate is incredibly fast. This gives more space to visualize feasibility studies with different options on the fly without the need to pass through and tedious modeling and rendering phase.
Mentors
Ismail Seleit
Ismail Seleit is a design architect at the industry-leading Foster and Partners, specialising in design technology. With extensive experience in architecture and computational design, Ismail has contributed to diverse projects, ranging from design competitions to the realisation of buildings on various scales. Actively supporting design teams, Ismail leverages his expertise in BIM and Computational Design to navigate complex challenges.
His primary focus is on enabling project teams to achieve efficient, informed, and collaborative design outcomes. Collaborating across departments, Ismail works to implement innovative methodologies to manage the design process from urban planning to product scale.
Simultaneously, Ismail engages in applied research, exploring new opportunities for implementing cutting-edge design workflows. His research specifically delves into the integration of generative AI image-generating tools, with a keen interest in the combination of Stable Diffusion and ControlNet.
This unique approach offers various techniques for generating controlled image compositions based on diverse user-defined inputs. Beyond his architectural pursuits, Ismail is an ambient-electronic music producer, with a focus on film scores.
He has composed and produced soundtracks for architectural short films at Foster and Partners and contributed to the music and sound design of various independent films. Ismail’s passion for creative thinking, coupled with his dedication to design problem-solving, continues to drive his multifaceted and innovative endeavours.
Programme
DAY 1
Getting comfortable with Stable Diffusion + ControlNet using Automatic1111 webUI
- Short intro about myself and my work
- Introduction about what Stable Diffusion is and how it works
- Introduction about what ControlNet is, its models, and why it is a breakthrough
- Dive into Automatic1111 user interface and all its functionality – practical
Prompting in Stable Diffusion - Textual inversion for negative prompts
- Sketch to image using CN scribble model
- Revit mass to image using CN MLSD model
- Rhino mass to image using CN Depth model
Day 2
Advanced Stable Diffusion + ControlNet techniques and
applied architecture design workflows
- Combining CN models for more controlled image generation
- Deeper dive into CN image segmentation model
- CN inpaint with sketch for design iteration within context images
- Image upscaling and inpainting techniques for finalizing image output
- Using LoRAs inside prompts for more stylised image output – practical
- Training LoRAs on custom image set using Dreambooth (KohyaSS)
- Consistent image iteration using reference images to generate the same design from different angles
- Sketch + text to image to video workflows using Runway ML
- AI Walkthrough videos frame interpolation AI using Flowframes
- Final thoughts and further work
Knowledge, Software and Hardware
- Automatic1111 installed locally with minimum 8GB GPU VRAM (preferably NVIDIA): https://github.com/AUTOMATIC1111/stable-diffusion-webui
- If not locally then through Rundiffusion or Google Colab: https://rundiffusion.com/
- ControlNet 1.1 extension installed with at least Scribble, MLSD, Depth, Segmentation, Inpaint and Reference models downloaded and installed: https://github.com/lllyasviel/ControlNet-v1-1-nightly
- Realistic Vision V5.0 (or later) checkpoint downloaded and installed:
https://civitai.com/models/4201/realistic-vision-v50 - Dreambooth KohyaSS for LoRA training: https://github.com/bmaltais/kohya_ss
- Rhino + Revit for mass to image testing
- Photoshop for image segmentation (optional)
- Runway ML free account: https://runwayml.com/
- Flowframes (optional): https://github.com/n00mkrad/flowframes
Important Info:
- It is advisable to have active plans for the following tools: Midjourney, Prome AI, Krea AI and Runway. This tools covered have free versions, except for Midjourney.
- Basic knowledge of interior design and architecture is recommended.