Native Long Video Understanding Models locally?

I've been building a personal project and wanted to check with the community on multi-modal inputs since I can't find a lot of material around this online. Ultimately I'm trying to build something that can ingest massive length (almost like a full stream - 6-10 hours) and accurately do multimodal analysis. How are you guys working with long (atleast 2+ hour) videos? I understand local LLMs with ViT designs can help do this but they usually suffer in quality (diffusion patches can rack up context really quickly) or require you to do some sort of frame sampling (which defeats the native multimodal aspects). I saw some work around vllm-omni which uses qwen3-omni to video input stuff, but ofcourse the context is severely limited so it's not very usable OOTB. So far what's worked for me: - get mp3 audio file -> transcribe with qwen3-asr -> get a full timestamped vtt file - summarize this vtt file with verbatim timestamped cliff notes (important for next steps) - calculate the max dynamic frame rate using context window length and the video length - sample at this rate then perform absdiff on the frames to eliminate frames where there's not a lot of change happening; downscale every frame to 720p max resolution or 540p - calculate the number of chunks you need to split into to fit into 64k context per chunk; each chunk is basically the relevant image frames for this part of the video + the transcript data for this part of the video - use transcript summary (which has verbatim timestamped stuff) + each chunk -> summarize keeping the verbatim aspects and global summary information + local transcript information. So essentially when I ingest a video I end up with [transcript summary] + [summary of chunk 1/N + summary of chunk 2/N + ... + summary of chunk N/N ] I'm experimenting with using the qwen3-asr output text + mp3 file directly to gemma4 12B to do appropriate corrections on the audio like speaker diarization, adding cues about music/noises/sounds/spell corrections etc. It's still a WiP. Apart from this I'm not sure if it's worth the headache of having a multi docker multi service setup to ingest video data if a model can do it natively. Anyone else working on similar stuff? Would love to see if this is being solved in a different way. Is there anything else that can be run on 128GB RAM that is better than my patchwork pipeline for long video ingestion/indexing/analysis? submitted by /u/dash_bro [link] [comments]

Source: r/LocalLLaMA | 2026-08-10

Related