Agentic Planning with Reasoning for Image Styling via Offline RL

Direct prompt-based editing often fails on complex transformations because vague and subjective prompts often require nuanced understanding of what should be changed in the image. Our core intuition is that leveraging compositional image editing tools rather than direct prompting profits from structured agent-level planning with explicit reasoning, leading to better results. This structured planning framework enables efficient offline RL post-training on quality-scored trajectories to improve performance. We present a tool-based agentic RL post-training framework that addresses this through structured planning with chain-of-thought reasoning. Our key contributions include: (1) A tool-based agentic planning methodology that combines a compositional library of orthogonal primitive transformations, structured context representation, and explicit per-step reasoning to decompose complex styling into interpretable tool sequences. (2) A synthetic data generation pipeline producing three large-scale datasets (each ~10K trajectories) with reasoning chains, plans, and quality scores—necessary because no existing datasets provide explicit tool-based styling supervision. Our datasets are publicly available on Hugging Face. (3) An empirical study of offline RL training methods for vision-language tool-use planning, adapting Reward-Weighted (RW) and Standardized Reward-Weighted (SW) methods—previously validated only in text-only settings—to this domain for the first time, consistently improving over the Edit-Only baseline in visual quality and instruction following. (4) Comprehensive evaluation across 4B and 8B parameter Qwen3-VL models showing that our methods outperform other baselines in the majority of the compositional tasks, backed up by human evaluations on both the synthetic data and final results.

Our work demonstrates that structured planning with reward-aware training enables models to produce higher-quality images that better follow user instructions compared to direct editing approaches. Comprehensive evaluation on Qwen3-VL 4B/8B shows our compact open-source planners outperform GPT-4o and Gemini 2.5 zero-shot baseline planners on the majority of tasks, confirmed by human evaluation.