Abstract: With the development and iteration of image generation models, models  like Stable Diffusion based on the  diffusion model have become the mainstream image generation models, providing a new way for clothing design and rendering. The diffusion model usually uses the text prompt word as the image generation condition and the generated picture has randomness. It is difficult to accurately generate the virtual fitting effect of a specific style. The application of ControlNet neural networks makes the generation of images more controllable. The trained Controlnet network can use the image information such as Canny edge map, depth map, and Openpose map as additional generation conditions of the diffusion model to control the human body posture, edge features, front and rear position relationship of the generated image. This paper briefly describes the development history and principle of the diffusion model, and explores its feasibility for generating virtual fitting renderings. To achieve the purpose of visualizing the clothing style diagram as the garment effect and realize the rapid generation of virtual fitting effect, This paper attempts to use ControlNet neural network to control the diffusion model to generate virtual fitting effect of virtual models wearing specified clothing styles.
  The virtual fitting of three dresses was taken as an example for experimentation. Firstly, the images of real clothing models with expected posture were sampled, and the key human body images and pose depth maps of real models were extracted as the generation conditions. Then, the Controlnet control Stable Diffusion model was used to generate a virtual clothing model image that matches the intended pose. Subsequently, the edge image of the virtual model was generated by the Canny algorithm, and the edge image was edited and modified in combination with the dress style diagram. The edge image of the virtual model wearing the specified style dress was drawn, and it was used as the edge generation condition. The virtual fitting effect of the dress conforming to the specific style, color and fabric was generated by the text prompt-controlled diffusion model, and the style of the dress with the virtual fitting effect was changed in real time by modifying the edge image, so as to provide an intuitive reference for fashion designers to modify and adjust designs. In addition, the detailed feature control experiment of the virtual model was also carried out during the experiment; experiment on the control effect of text prompt word weight on clothing fabric and color was carried out. Finally, the generation effect of the proposed method was compared and evaluated with the effect of 3D modeling virtual fitting clothing.
The results show that the diffusion model combined with the ControlNet network can control the pose characteristics of the virtual model, allowing the virtual fitting effect of the expected clothing style to be generated by editing the Canny edge image control. Compared with 3D modeling, the virtual fitting effect is more expressive, the operation is more intuitive and faster, and it is more suitable for providing designers with intuitive clothing display in the style design stage, assisting designers to adjust the design style, color, fabric and process, and improving the efficiency of clothing design.

Key words: virtual fitting, diffusion model, ControlNet, virtual models, human keypoint detection, clothing design

摘要： 为快速生成特定服装款式的成衣效果图，采用扩散模型，应用ControlNet网络实现虚拟试衣。首先将人体的关键点检测图与深度图作为扩散模型的控制条件，生成姿态可控的虚拟模特；再通过Canny边缘图生成虚拟试衣效果图。以三款连衣裙为例进行虚拟试衣实验，并优化扩散模型控制条件的参数设置；最后将生成结果与三维建模虚拟试衣结果进行对比和评价。结果表明：结合ControlNet网络的扩散模型能够控制虚拟模特的姿态特征，通过服装Canny边缘图可以生成特定服装款式的虚拟试衣效果。该方法生成的虚拟试衣相较三维建模技术实现的虚拟试衣方法更具表现力，操作更加直观快捷，能够为设计师提供款式图的成衣效果可视化参考，从而提高服装设计效率。

关键词: 虚拟试衣, 扩散模型, ControlNet网络, 虚拟模特, 人体关键点检测, 服装设计