关键词: 虚拟试衣, 扩散模型, ControlNet网络, 虚拟模特, 人体关键点检测, 服装设计

Abstract: With the development and iteration of image generation models, models  like Stable Diffusion based on the  diffusion model have become the mainstream image generation models, providing a new way for clothing design and rendering. The diffusion model usually uses the text prompt word as the image generation condition and the generated picture has randomness. It is difficult to accurately generate the virtual fitting effect of a specific style. The application of ControlNet neural networks makes the generation of images more controllable. The trained Controlnet network can use the image information such as Canny edge map, depth map, and Openpose map as additional generation conditions of the diffusion model to control the human body posture, edge features, front and rear position relationship of the generated image. This paper briefly describes the development history and principle of the diffusion model, and explores its feasibility for generating virtual fitting renderings. To achieve the purpose of visualizing the clothing style diagram as the garment effect and realize the rapid generation of virtual fitting effect, This paper attempts to use ControlNet neural network to control the diffusion model to generate virtual fitting effect of virtual models wearing specified clothing styles.
  The virtual fitting of three dresses was taken as an example for experimentation. Firstly, the images of real clothing models with expected posture were sampled, and the key human body images and pose depth maps of real models were extracted as the generation conditions. Then, the Controlnet control Stable Diffusion model was used to generate a virtual clothing model image that matches the intended pose. Subsequently, the edge image of the virtual model was generated by the Canny algorithm, and the edge image was edited and modified in combination with the dress style diagram. The edge image of the virtual model wearing the specified style dress was drawn, and it was used as the edge generation condition. The virtual fitting effect of the dress conforming to the specific style, color and fabric was generated by the text prompt-controlled diffusion model, and the style of the dress with the virtual fitting effect was changed in real time by modifying the edge image, so as to provide an intuitive reference for fashion designers to modify and adjust designs. In addition, the detailed feature control experiment of the virtual model was also carried out during the experiment; experiment on the control effect of text prompt word weight on clothing fabric and color was carried out. Finally, the generation effect of the proposed method was compared and evaluated with the effect of 3D modeling virtual fitting clothing.
The results show that the diffusion model combined with the ControlNet network can control the pose characteristics of the virtual model, allowing the virtual fitting effect of the expected clothing style to be generated by editing the Canny edge image control. Compared with 3D modeling, the virtual fitting effect is more expressive, the operation is more intuitive and faster, and it is more suitable for providing designers with intuitive clothing display in the style design stage, assisting designers to adjust the design style, color, fabric and process, and improving the efficiency of clothing design.

Key words: virtual fitting, diffusion model, ControlNet, virtual models, human keypoint detection, clothing design