Flux in ComfyUI
Integrating ControlNets & LoRAs
Flux in ComfyUI: Integrating ControlNets and LoRAs is a comprehensive 4-hour workshop designed for architects and designers seeking to integrate cutting-edge AI tools into their professional practice. This session focuses on the synergistic application of ControlNets and custom-trained LoRAs within Flux, a suite of advanced text-to-image generative AI models developed by Black Forest Labs. These models transform textual descriptions into high-quality images, offering significant improvements in visual detail, prompt adherence, and output diversity.
Participants will examine the theoretical foundations of Stable Diffusion and its applications in conceptual design phases, while acquiring proficiency in utilizing ControlNets for precise structural manipulation and LoRAs for style adaptation. The workshop curriculum encompasses dataset curation, training parameter optimization, and best practices for fine-tuning models to address specific architectural requirements.
Through a series of practical exercises and empirical case studies, attendees will construct bespoke workflows that harness the complementary strengths of ControlNets and LoRAs, facilitating rapid iteration and refinement of design concepts. The session concludes with an exploration of strategies for expanding creative parameters and fostering adaptive design methodologies within the evolving paradigm of AI-assisted architecture.
Workshop Overview:
Theoretical Foundations of Stable Diffusion in Architectural Design (30 minutes)
ControlNet Principles and Architectural Applications (45 minutes)
Custom LoRA Training Methodologies for Architecture (45 minutes)
Intermission (15 minutes)
Integration of ControlNets and LoRAs in ComfyUI Environments (60 minutes)
Empirical Applications and Case Study Analysis (30 minutes)
Advanced Methodologies and Creative Exploration (25 minutes)
Why it is important:
The integration of AI technologies such as ControlNets and LoRAs is fundamentally transforming architectural design processes, offering unprecedented efficiency and creative potential. This workshop is essential for professionals aiming to remain at the forefront of technological advancements in architecture. By mastering these tools, designers can:
- Expedite the generation and iteration of design concepts
- Exercise precise control over structural elements while exploring stylistic variations
- Develop bespoke models tailored to specific architectural idioms or project parameters
- Enhance interdisciplinary collaboration and communication within design teams
- Adapt to the evolving landscape of AI-assisted design in the architectural profession
What will you learn:
- Theoretical underpinnings of Stable Diffusion and its architectural applications
- Advanced methodologies utilizing ControlNet for structural guidance in iterative processes
- Custom LoRA training techniques for architecture-specific style adaptation
- Dataset curation and annotation methodologies for effective model training
- Construction and optimization of bespoke workflows in ComfyUI environments
- Strategies for synthesizing ControlNets and LoRAs to generate innovative design solutions
- Best practices for empirical testing, refinement, and implementation of AI-generated designs
- Analysis of case studies demonstrating practical applications of these technologies
- Methodologies for expanding creative parameters and fostering adaptive design approaches
Mentors
Ismail Seleit
Ismail Seleit is a design architect at the industry-leading Foster and Partners, specialising in design technology. With extensive experience in architecture and computational design, Ismail has contributed to diverse projects, ranging from design competitions to the realisation of buildings on various scales. Actively supporting design teams, Ismail leverages his expertise in BIM and Computational Design to navigate complex challenges.
His primary focus is on enabling project teams to achieve efficient, informed, and collaborative design outcomes. Collaborating across departments, Ismail works to implement innovative methodologies to manage the design process from urban planning to product scale.
Simultaneously, Ismail engages in applied research, exploring new opportunities for implementing cutting-edge design workflows. His research specifically delves into the integration of generative AI image-generating tools, with a keen interest in the combination of Stable Diffusion and ControlNet.
This unique approach offers various techniques for generating controlled image compositions based on diverse user-defined inputs. Beyond his architectural pursuits, Ismail is an ambient-electronic music producer, with a focus on film scores.
He has composed and produced soundtracks for architectural short films at Foster and Partners and contributed to the music and sound design of various independent films. Ismail’s passion for creative thinking, coupled with his dedication to design problem-solving, continues to drive his multifaceted and innovative endeavours.
Programme
Important Information for the Workshop
Thank you for registering for the "Flux in ComfyUI" workshop! Below are the essential details and resources you’ll need to prepare:
1. Downloading ComfyUI
To participate in the workshop, you must have ComfyUI installed. If you don’t already have it, download the portable version using the link below:
Download ComfyUI Windows Portable: https://github.com/comfyanonymous/ComfyUI/releases/latest
2. Required Models and Files
Please download the following models and place them in the specified folders:
Main Model Checkpoints:
- realflux10b_10bTransformerDev.safetensors
https://civitai.com/models/788550/realflux-10b - flux1-schnell.safetensors
https://huggingface.co/black-forest-labs/FLUX.1-schnell/tree/main - flux1-dev.safetensors
https://huggingface.co/black-forest-labs/FLUX.1-dev/tree/main - ae.safetensors (VAE)
https://huggingface.co/black-forest-labs/FLUX.1-schnell/blob/main/ae.safetensors - clip_l.safetensors (CLIP model)
https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/clip_l.safetensors - t5xxl_fp8_e4m3fn.safetensors (T5 encoder)
https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp8_e4m3fn.safetensors
Optional LoRA Files:
- InteriorDesigner-Flux_v4.safetensors
https://form-finder.squarespace.com/download-models/p/interiordesigner-flux - FormFinder-Flux_v2.safetensors
https://form-finder.squarespace.com/download-models/p/formfinder-flux
ControlNet Models:
- flux-jasper-normal-controlnet.safetensors
https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Surface-Normals/tree/main - flux-canny-controlnet-v3.safetensors
https://huggingface.co/XLabs-AI/flux-controlnet-canny-v3/tree/main
3. File Placement in ComfyUI
After downloading, place the files in the following directories within your ComfyUI folder:
- Transformer (Unet): Place in
ComfyUI/models/unet/ - LoRA Models: Place in
ComfyUI/models/loras/ - VAE: Place in
ComfyUI/models/vae/ - CLIP Models: Place in
ComfyUI/models/clip/ - ControlNet Models: Place in
ComfyUI/models/controlnet/
4. Required Custom Nodes
Ensure the following custom nodes are installed in your ComfyUI setup:
Lora Loader Stack (rgthree)DualCLIPLoaderJWImageResizeByLongerSideRandomNoiseBasicGuiderFluxGuidanceSamplerCustomAdvanced
You can download the latest versions of these nodes from the official ComfyUI GitHub repository or its community plugins:
GitHub Repository: https://github.com/comfyanonymous/ComfyUI
5. Additional Notes
- While we may have a limited number of pre-configured ComfyUI instances available during the workshop, we highly encourage participants to have everything set up on their system beforehand for the best experience.
- The workshop will explore workflows that integrate “Flux” models and methodologies with Stable Diffusion for enhanced design and visualization capabilities.
If you have any questions or encounter issues while setting up, please feel free to contact us. We look forward to seeing you at the workshop!
Important Info:
- It is advisable to have active plans for the following tools: Midjourney, Prome AI, Krea AI and Runway. This tools covered have free versions, except for Midjourney.
- Basic knowledge of interior design and architecture is recommended.