K2Lab: Standalone(ish) Krea2 bbox style prompting and lora containment

I've been digging into Krea2 to see if there's any way to condition inference in specific regions of pixel -> latent space to implement bbox type prompting in order to apply multiple simultaneous character loras to different subjects without leakage. Originally I tried to implement this as a node pack in comfyUI but that turned out to be a gigantic hassle so instead I opted for a pyside implementation using a bit of comfy backend. It's still a work in progress but it's far enough along that I figured others might find it useful if for no other reason than as a research direction to get better control of Krea2. My local GPU is an RX9060XT so I built this with that hardware in mind, but it is tunable down to 8GB VRAM/24GB RAM, or up to whatever. I have a runpod API implementation under construction at the moment to spare myself the pain of VRAM poverty... There isn't anything special about this UI, it was just convenient and I don't like comfyUI. With the info contained in the repo it should be totally doable for someone to use the same approach to make some functional nodes that accomplish the same thing... it's just not gonna be me. The main idea is this: A single diffusion pass placing prompt-specified subjects and/or background scenes/objects in regions defined by drawn rectangular boundaries with the option of applying any number of loras either globally or to specific regions without leakage. The same lora can be loaded multiple times to be applied to multiple characters at different strengths if desired. Phrases within the prompts can be emphasized, and there are a variety of parameters that can be tweaked to encourage obedience and tune spatial boundary adherence/blending. Ultimately prompting via regional gating is not absolutely deterministic because it only boosts or restricts crossmodal attention access between image/text tokens in/without a specified region, so Krea still makes "its own choices" to some degree about how it will render whatever you describe in a given region -- this is where good prompt design comes in. How it works: -Boxes are drawn on the canvas and each has a content prompt (describe what should appear) and an optional character/identity prompt (describe the character's appearance), and there is a single global prompt (describe style, lighting, etc) which will attend globally. -Loras are assigned as global or regional, and regional loras can be standard (ie concept, style, etc) or character/identity. For the latter, you provide the trigger word and the subject in the specified box will be auto triggered. -The prompts for all regions are combined into a unified prompt which is ordered based on scene prominence (front to back, accounting for overlap), pixel coordinates are translated into latent cell image tokens, and the regional prompt token spans and regionally owned image tokens are saved for attention routing, token emphasis, and lora gating. -Regional prompting is enforced by cross-modal attention permission management: image tokens in a box can access text tokens associated with that region, but image tokens outside cannot. Likewise text tokens outside the box are prevented from accessing image tokens within, etc. This prevents prompt leakage through later attention steps. Image tokens are not restricted in the same way to avoid rectangular artifacts at the boundaries (i.e. image tokens can attend across boundaries so that there is scene continuity). -The degree of restriction of attention at the boundary is controllable through a few different tunable parameters (e.g. spatial falloff, step-based easing, etc) to allow for strong conditioning in the first set of denoising steps and then allow global attention to stitch everything together seamlessly. -Lora leakage is contained by loading them as unfused adapters and enforcing five rule layers: lora delta is explicitly set to zero outside of the specified region, lora modified regional text is not accessible to image tokens outside the region, text tokens outside the region cannot access image tokens within the region, regional loras with globally scoped K/V projections are skipped so that only Q deltas are applied, and character loras use explicit region-specific trigger words auto-placed into the regional prompt. Lora deltas that remain after these are applied are combined and sent forward. Current features: -Regional prompt design, non-leaky lora, fine(r) control over spatial conditioning, OOM safeguards for low VRAM, optional Lanczos upscaling and native Krea projector vector value editing with a few included presets from the various comfy nodes available. -Image editing using the same approach but starting with VAE encoding then applying the same process to selected regions and/or globally with tunable denoising and blending parameters. -Face refinement using optional ONNX face detection (an early experiment that I left in but is pretty useless tbh), or selection by drawing a lasso. This is meant for improving character facial identity and allows you to apply the character lora at different strengths which can be useful for doing manual faceswap type stuff, or what I used it for initially, was refining poor facial quality on some character loras I did a bad job of training (since then I've made actual quality ones and don't really need it). Caveat emptor: I really own develop software for my own use, so I have no idea if this will work for anyone else, but if you want to give it a try or scrape the method for your own purposes, have at it: https://github.com/soomrenald/k2lab Attached are some demo images of the GUI showing regional prompting, non-leaky style lora, image editing, and non-leaky character loras. submitted by /u/coyoteka [link] [comments]

Related

• Best anime model for multiple characters or Lora?
• Mixing Style LoRa with Character LoRa in ComfyUI - how do you avoid conflicts?
• Question about training a lora for character style consistency
• Several Character Loras

Source: r/StableDiffusion | 2026-07-28