DORS: Dynamic Attention Routing for Diffusion-based Object Removal in Dense Scenes

Haitong Tang Haipeng Liu✉ Yang Wang✉

Hefei University of Technology

✉ Corresponding authors

ACM MM 2026

DORS compared with object removal baselines in dense scenes
Fig. 1. Object removal in dense scenes containing multiple similar instances remains challenging for state-of-the-art methods, which often fail to completely erase the target instance and leave residual artifacts, as highlighted by the green bounding boxes. In contrast, our training-free, plug-and-play DORS leverages a dynamic attention routing mechanism to fully remove the target instance while producing cleaner and more coherent results.

The idea

Abstract

Object removal aims to eliminate target objects specified by a mask while preserving visual consistency with the surrounding regions. Existing methods typically rely on contextual information from surrounding regions. However, in dense scenes where the surrounding regions contain instances visually similar to the removal target, such reliance often leads to semantic interference, resulting in incomplete removal. This problem arises from erroneous information propagation in the attention space, where masked queries tend to align with such instances due to global similarity matching in self-attention. To address this challenge, we propose a Diffusion-based Object Removal framework for dense Scenes, dubbed DORS, built upon a Dynamic Attention Routing mechanism comprising two complementary components: Instance-Filtered Attention (IFA), which suppresses misleading semantic information from similar instances through dynamically constructed mask-guided attention constraints, and Context-Guided Routing (CGR), which dynamically routes complementary scene information to maintain visual consistency. We further introduce DOR-Bench, a benchmark tailored for object removal in dense scenes. Extensive experiments demonstrate that DORS outperforms state-of-the-art methods, particularly in reducing incomplete removal and duplicate artifacts.

Shield misleading semantics. Bridge valid context.

How DORS Works

The complete DORS dynamic attention routing framework
Fig. 2. Overview of DORS. It introduces a plug-and-play Dynamic Attention Routing (DAR) mechanism into the denoising process, comprising Instance-Filtered Attention (IFA) and Context-Guided Routing (CGR). The former dynamically constructs mask-guided constraints to suppress similar-instance interference, while the latter dynamically routes information between the filtered and full-context pathways based on global density and local geometry. For clarity, mask resizing is simplified.

Motivation. Most diffusion-based object removal methods reconstruct masked regions through unconstrained global context aggregation. In dense scenes, self-attention can cause masked queries to match visually similar instances outside the mask, propagating target semantics back into the removal region. We call this failure mode instance interference; it leads to incomplete removal, duplicated structures, and visual artifacts.

Overview. DORS reframes dense-scene object removal as controlled information routing in self-attention. Its plug-and-play Dynamic Attention Routing (DAR) mechanism combines IFA, which shields the masked region from misleading target-similar semantics, with CGR, which restores useful structural context through density- and geometry-aware fusion. DAR is applied during early denoising to establish global structure and relaxed in later steps for detail refinement.

Semantic shield

Instance-Filtered Attention (IFA)

IFA uses the target removal region as a visual prompt for SAM3 to identify target-similar instances and align their masks with the attention resolution. It adds a large negative bias that prevents masked queries from attending to both the removal mask and the detected similar-instance regions, so they aggregate information only from non-similar context. This input-adaptive pruning is limited to masked queries, preserving information flow within the unmasked background.

Context bridge

Context-Guided Routing (CGR)

Because IFA can suppress useful structural cues together with misleading semantics, CGR maintains two complementary attention pathways: full-context and instance-filtered. At each masked location, a spatially adaptive weight derived from global similar-instance density and local distance favors full context in nearby regions for structural continuity, while progressively favoring filtered information farther away to reduce semantic interference. Their weighted fusion balances coherent reconstruction with instance-aware suppression.

Qualitative results

Object Removal on DOR-Bench

Qualitative comparison of DORS with state-of-the-art object removal methods on DOR-Bench
Fig. 3. Qualitative comparison of object removal results in DOR-Bench, our method (DORS) achieves more accurate target removal while better preserving structural and semantic consistency, resulting in cleaner and more realistic outputs.

Quantitative results

Comparison on DOR-Bench

Table 1: quantitative comparison with state-of-the-art methods on DOR-Bench

Reference

BibTeX

@article{tang2026dors,
  title     = {{DORS}: Dynamic Attention Routing for Diffusion-based
               Object Removal in Dense Scenes},
  author    = {Tang, Haitong and Liu, Haipeng and Wang, Yang},
  journal   = {arXiv preprint arXiv:2607.16656},
  year      = {2026}
}
↑