Img2vid with FramePack on ComfyUI (now with first and last frame support)
Introduction
This workflow implementation is a quality-of-life update to a FramePack/Hunyuan workflow published so far on Civitai.
Huge thank you to METAFILM_Ai for publishing the original workflow
Primary concerns addressed by this workflow:
Automatic image resize and crop to match original image orientation (based on relationship between image width and height, not composition)
Start and end frame support
Tuning node settings to run on low VRAM GPUs (tested on RTX 3080 10GB VRAM)
Original image is cropped and resized to 432 x 768 (9:16) or 768 x 432 (16:9) because it seems to provide best performance / quality on generation and upscale (2.5x to 1080p)
VAE decode is set to 192 32 32 8. Larger tiles caused out-of-memory errors.
Addition of control box to make following functions optional:
Set last frame
Upscale
Saving animated WEBP file
Saving video
Clean-up of unnecessary nodes
Additional settings to consider:
Video Combine:
frame_rate - by default set to 30 but you can consider other values like 24
loop_count - by default 0, will repeat frames as many times as you set up, useful for infinitely looping videos
pingpong - by default set to false, when set to true will cause frames added to the end of the video in reverse, useful for infinitely looping videos
Sample
Here is a sample based on the cover image
https://civitai.com/posts/15967069
Installation
Download models used by workflow:
ComfyUI\models\upscale_models
https://civitai.com/models/147759/remacri
To ComfyUI\models\text_encoders
To ComfyUI\models\diffusion_models
https://huggingface.co/Kijai/HunyuanVideo_comfy/blob/main/FramePackI2V_HY_fp8_e4m3fn.safetensors
To ComfyUI\models\clip_vision
https://huggingface.co/Comfy-Org/sigclip_vision_384/blob/main/sigclip_vision_patch14_384.safetensors
To ComfyUI\models\vae\hunyuan
Download workflow attached to this article and drag it into ComfyUI
Use ComfyUI Manager to install missing nodes or install custom nodes from git. They are listed below in workflow dependencies.
If you are just testing prompts, I would recommend turning off Upscale and Save Animated Image groups. Upscale in particular takes considerable time.
I would not recommend enabling Upscale and Save Animated Image together unless you increase the compression rate because 1080p WEBP files are huge.
Workflow dependencies:
ComfyUI Wrapper for FramePack by lllyasviel
https://github.com/kijai/ComfyUI-FramePackWrapper
Core functionality for video generation, might need to install from git
rgthree-comfy
https://github.com/rgthree/rgthree-comfy
Control box to bypass or enable optional functionality, available from ComfyUI Manager
KJNodes for ComfyUI
https://github.com/kijai/ComfyUI-KJNodes
Only used to get image width and height, but has lots of other useful nodes, available from ComfyUI Manager
ComfyUI-Impact-Pack
https://github.com/ltdrdata/ComfyUI-Impact-Pack
Only logic nodes used to decide on portrait or landscape, but has lots of other useful nodes, available from ComfyUI Manager
Comfyroll Studio
https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes
Used for a nice upscaler, available from ComfyUI Manager
ComfyUI-VideoHelperSuite
https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite
Combines image frames into video, available from ComfyUI Manager
Credits
Version History
v2 - better organization, optional end frame support, bookmarks for control box, input and results
v1 - the original workflow, although I did update the FramePack and updated some settings, like switching pingpong to false.