====== General Discussion ====== I would appreciate to have an open_movie_monitor application at hand that gets the max out of standard hardware, with little effort. To what extend could applications like lives, xmovie, mplayer, gstreamer, pd+gem,... fill in? blender might be an option for some. I was thinking about coding a prototype for XV + X11extensions in the next year, basically some successor of [[oss:xjadeo:xjadeo]] in one way or another. Is anyone intending to do sth like this with openGL? - I need to do some reading before getting more specific, but gstreamer might be an option, too. ====== Brainstorm on jack video sync ====== These are Ideas gathered from development notes that I made while coding xjadeo. I might come back to do proper research on some of those.. some can might be **misleading** or simply wrong so **beware**. comments are welcome anyway. ===== timecode fps + jack transport + MTC ===== From the deprecated jack department: jack_transport_info_t had ''->smpte_frame_rate'' and ''->smpte_offset''; but ''jack_get_transport_info()'' is only avail from a process callback and passing this information along with the current-audio-frame requires clients to implement drop-frame timecode, SMPTE parsing, etc. The experimental ''(double)->audio_frames_per_video_frame'' suffers the same drawback. Why not export the SMPTE alike the bar,beat,tick in ''jack_position_t''? it might break binary compat,... ''jack_position_t->frame_time'' and ''->next_time'' would provide an alternative interface if they were rational (''int'') instead of (''double''). The denominator would depend of the time-masters's fps. ((personally I liked this solution best - it was marked with a box on my scribbled notes.)) A completely different approach would be to share the session-configuration (fps,..) fi. via lash. this would yield less redundant data in the jack_position struct; but more redundant code everywhere else.. (unless there's a shared-lib or so: can midi++2 fill in?) ===== jack video sync ===== I have not gone thru all the posts in the jackit-devel archive yet... but I am missing some specs. for synchronizing video and jack-audio frames. Well maybe it's not exactly jack's job to do that but providing a similar API as jack-MIDI for video would be great! ie: register a video frame to be displayed from/until a certain jack_audio_frame. ===== open_video_monitor ===== Here's some pseudo code of how I can imagine a real-time video jack display to work: This is described from the viewpoint of the //displaying app//. It's a user of the '''' and '''' API **data structures and definitions** /* this struct will be cast/serialized to conform to * proposed JACK_VIDEO_TYPE_H data types.. */ typedef struct { enum colorspace; int width,height; int stream_id; atom flags; /*< binary buffer flags: in-use/used/mapped/type * some implementation variants might require * an additional usage-counter... */ uint8 **vplane_buffer; /*BLOB(s) */ } vBUFFER; atomic int64_t Vcnt; /*< counter is incremented each time the screen's * Vsync wraps around. * discuss: need for an array? * interlaced displays, xinerama etc. */ typedef struct vQUEUE { vQUEUE *next; /*< a Vcnt tagged [static] cache * performs better than this linear list * - but that depends on application and architecture. */ int64_t Vcnt_start, Vcnt_end; /*< timestamp in units * of Vsync counts of this vQueue display(s) * Vcnt_end is optional (see: flags) */ int flags; /*< extra info for this frame (eg. * "display key-color after Vcnt_end") */ jack_nframes_t start, end; /*< redundant debug information */ vBUFFER *vframe; /* pointer to the actual video data or (NULL) */ } vQUEUE; vQUEUE *start,*end; /*< some might prefer to use static instead or * mlock(2)ed buffers ...? */ **function callbacks as implemented in the player** /* this function is linked to the video-output vertical sync */ (callback) vsync-raster-irq () { - increment Vcnt; - if (VQUEUE[head?].start >= Vcnt > QUEUE[head?].end) flip buffers() - flag the buffer in use and mark prev. buffer } /* this function is registered with * jack_set_video_process_callback(NEW) */ (callback) jack_av_transport_advance () { - query jack transport position (or recv as argument) - query or cache "transport-advance-ticks" (currently: audio-buffer-size, later: video-ticks ?) - calculate(*) Vcnt timestamps and video pts for frames until the next callback. - check if those video-frames are available by get_jack_video_buffer(%) - en-queue video frames. - flush old entries from the queue (maybe do this in a separate thread..) } ** annotations** The ''calc(*)'' function can be anything from simple maths to fancy lookup tables or sequencing code... a hot-swappable plugin system comes to mind to assign video events to audio frames! ''get_jack_video_buffer(%)'' could * source binary data from jack (alike audio) -> ''jack video buffer'' * and/or be a function that gets the data from a file (un*x: pipe, socket, shared-mem..) an initialization routine would look like this: { - open jack + video-display - async reset Vcnt from jack thread. - measure frequencies have a Vsync-irq callback store an array of monotonic jack usecs and vice versa: store some Vcnt in jack_callbacks. - if video_clk is faster than audio-clk or having a funny ratio { sync reset Vcnt.} - initialize calc(*) routines with Vcnt offset set known parameters (eg. video-source-framerate if the jack-transport-master provides this info.) } there'll be some '''', not to mention '''', but ''cat /dev/null > "NTSC_PAL-magic.h"'' assuming that a jack_av_transport_advance() can be implemented as a ''jack_audio_callback(TM)'' it will be executed at: 0.1\mathrm{Hz} < (f_a := \frac{\mathrm{AudioSampleRate}} {\mathrm{AudioBufferSize}}) < 10\mathrm{kHz} the Vsync callback function is triggered f_v times per second((some implementations might only be able to report multiples of Vsync wrap counts, others might be more accurate.)). 14\mathrm{Hz} < ( n \cdot f_v ) < 256\mathrm{Hz} \;\;;\;\; n\in\field{N}^+ I suggest that audio provides the fundamental clock, but video frame time-stamping should be possible up to the smallest video time granularity := GCF (vertical refresh rate of the display, orig-movie-frame-rate). For variable framerate video streams 90\mathrm{kHz} might be an appropriate value for f_m := \mathrm{VideoTimeAtom}^{-1} Does f_a > f_v need to satisfied? This opens a lot of A/V sync issues. I'd propose different sync algorithms depending on ranges or specific values of (f_a / f_v)_{f_m} implemented as //frame[rate] conversion Plugin//s((some of them include the NTSC/PAL/MTC/SMPTE magic)). Anyone to discuss impact of introducing "leap-frames" (a concept similar to latency_compensation) that would allow the video thread to offset/re-sync the audio. (..fast audio resampling, featured x-runs,.. :grin: just a thought) ===== future of jack ===== How good is or will net-jack-jack-transport synchronization get.. IMO it is a a nice extension to the jack suite. I'd guess that network packet timestamps, CPU clocks and PCI latencies can cause quite some headaches... Is a jitter > 10kHz is reasonable ? (do engineers measure jitter in the frequency domain ? oh well. < 0.1 ms then - that's ~0.1% RMS for those with an arbitrary unit calculator at hand :-P or it should not be worse than +-1.8 video-frames per 10 mins. some say that's bearable to watch. 8-) Some related thoughts that should probably be commented on in the "future of jack" thread on jack-devel: * multiple transports (derived from the same oscillator) - two of those could be use for gap-less audio-looping. - many of those could provide an interface for video sequencing.