Synchronization

Synchronisation in multimedia involves two concepts which are often confused: clock (or speed) and time (location in time).

A clock is the mechanism by which two systems tick simultaneously. In the audio world this is generally referred to as Word_clock. It does not carry any absolute reference to a point in time: A clock is used to keep a systems sample rate constant, regular and accurate. Word clock is usually at the frequency of the sample-rate - ie at 48KHz, its period is about 20μs. Word Clock is the most common 'sample rate' based clock but other clocks do exist such as Black and Burst, Tri-Level and DARS. Sample rates can also be derived from these clocks as well.

Time – or timecode – on the other hand specifies an absolute relationship or position on a timeline e.g. 01:02:03:04 (expressed as Hours:Mins:Secs:Frames). It is actual data and not a clock-signal per se. The granularity of timecode is Video Frames and is an order of magnitude lower than, say, Word Clock which is counted in samples. A typical frame-rate is 25 fps with a period of 40ms. In the case of 48kHz and 25fps, there are 1920 samples per video frame.

An interesting point to note is that LTC (Linear Time Code) is a Manchester Encoded, Frequency Modulated signal that carries both 'Clock' and 'Time'. It is possible to extract absolute position data and speed from it.

The concept of clock and timecode is reflected in JACK and Ardour:

JACK provides clock-synchronization and is not concerned with time-code (this is not entirely true, more on jack-transport later). Within software, jackd provides sample-accurate synchronization between all JACK applications. On the harware side JACK uses the clock of the audio-interface. Synchronization of multiple interfaces requires hardware support to sync the clocks. If two interfaces run at different clocks the only way to align the signals is via re-sampling (SRC - Sample Rate Conversion) - which decreases fidelity.

Timecode is used to align systems already synchronized by a clock to a common point in time, this is application specific and various standards and methods exist to do this.

NB. to make things confusing, there are possibilities to synchronize clocks using timecode. e.g. using mechanism called jam-sync and a Phase-Locked-Loop.

When speaking about synchronization, there is no way around also mentioning Latency: Latency is how you call the reaction time of a system to a certain stimulus. There are many factors that contribute to the total latency of a given system. In order to achieve exact time synchronization all sources of latency need to be take into account and compensated for.

• Sound propagation through the air: since it is a mechanical perturbation in a fluid, sound travels at comparatively slow speed of about 340 m/s. Some interesting consequences:
  • Your acoustic guitar or piano has a latency of about 1-2 ms, due to the propagation of the sound between your instrument and your ear .
  • At a large concert venue if you are far away from the stage the sound will travel faster through the path “singer → mic → nearest loudspeaker → your ear” than through the “singer → air → your ear” one, so you'll hear the real sound as an echo of the amplified one.
• Digital-to-Analog and Analog-to-Digital conversion: electric signals travel quite fast, so their propagation time is negligible in this context, but the conversions between the analog and digital domain take a comparatively long time to perform, so their contribution to the total latency may be considerable. Fast converters are, for instance, one of the factors that distinguishes a quality audio interface from a cheap one, along with other features like low noise, low distortion, etc.
• Digital Signal Processing: digital processors tend to process audio in chunks, and the size of that chunk depends on the needs of the algorithm and performance/cost considerations. This is usually the main cause of latency when you use a computer and one you can try to predict and optimize.
• Computer I/O Architecture: a computer is a general purpose processor, not a digital audio processor. This means our audio data has to jump a lot of fences in his path from the outside to the CPU and back, contending in the process with some other parts of the system vying for the same resources (CPU time, bus bandwidth, etc.) Thanks to the combined efforts of kernel, audio driver and jackd developers, you are in position to tune your system a bit more towards the digital audio processing task, but don't expect miracles. Remember you can use your computer also to write documents, surf the net, save some lemmings… Polyvalence comes at a cost.

Figure 1: Latency chain. The numbers are an example for a typical PC. With professional gear and an optimized system the total roundtrip latency is usually lower. The important point is that latency is always additive and a sum of many independent factors.

There is not much that can done about the first two other than using headphones or sitting near the loudspeaker and buying quality gear.

Processing latency is usually divided into capture latency and playback latency:

• Capture latency: the time necessary for the digitized audio to be available for digital processing. Usually it is one audio period.

• Playback latency: the time necessary for the digitized audio to be processed and delivered out of the processing chain. At best it is one audio period.

But this division is an implementation detail of no great interest. What really matters is the combination of both. It is called processing roundtrip latency: the time necessary for a certain audio event to be captured, processed and played back.

It is important to note that processing latency in a jackd is a matter of choice: It can be lowered within the limits imposed only by the hardware and audio driver. But the lower it is, the more likely the system will fail to meet its processing deadline and the dreaded xrun will make its appearance more often, leaving its merry trail of clicks, pops and crackles.

The digital I/O latency is usually negligible for integrated or PCI audio devices but for USB or FireWire interfaces the bus clocking and buffering can add some milliseconds.

The JACK Audio Connection Kit has a few parameters to configure the latency. However the settings are constrained by hardware (audio-device, CPU and bus-speed). Lower latencies increase the load on the computer-system (it needs to process the audio in smaller chunks which arrive much more frequently). If the system can not keep up: an x-run (short for buffer over-run and buffer under-run) occurs which usually results in audible clicks or dropouts.

Low-latency is not always a feature you want to have. It comes with a couple of drawbacks: the most prominent is increased power-consumption because the CPU needs to process many small chunks of audio-data, it is constantly active and can not enter power-saving mode. Furthermore, if more than one application (sound-processor) is involved in processing the sound, the operating system performs a context-switch to run each of these for each audio-cycle which results in a much higher system-load and an increased chance of x-runs.

Reliable low-latency (≤10ms) on GNU/Linux can usually only be achieved by running realtime-kernel.

Yet there are only few situations where a very low-latency is really important, because they require very quick response from the computer. Some examples that come quickly to mind are:

• Playing virtual instruments: a large delay between the pressing of the keys and the sound the instrument produces will throw-off the timing of most instrumentalists (save if they are church organists, whom we believe are awesome latency-compensation organic systems.)
• Software audio monitoring: if a singer is hearing her own voice through two different paths, her head bones and headphones, large latencies can be disturbing.
• Live-effects: This case is similar to playing virtual instruments: instead of virtual-instruments/sythensizers it is about real-instruments and and effects processing. Low latency is important when using the computer as effect-rack (e.g. guitar effects) - also precise synchronization may be important if you manually trigger sound effects like delays.
• Live-mixing: Some sound engineers use a computer for mixing live performances. Basically that is a combination of the above: monitoring on stage, effect-processing and EQ. It is actually more tricky since one not only wants low latency (audio should not lag too much behind the performance) but exact low-latency (minimal jitter) for delay-lines between speaker in front and back.

In many other cases - such as playback, recording, overdubbing, mixing, mastering, etc. latency is not important, It can be relatively large and easily be compensated for.

To explain the last statement: during mixing or mastering you don't care if it take 10 or 100ms between the instant you press the play button and sound coming from the speaker. The same is true when recording.

During tracking, it is however important that the sound that is currently played back is internally aligned with the sound that is being recorded.

This is where latency-compensation comes into play: There are two possibilities to compensate for latency in a DAW: read-ahead the DAW actually starts playing a bit early. So that the sound hits the speakers a short time later, it is exactly aligned with the timecode of the material that is being recorded. and write-behind since we know that the sound that is being played back has latency, the incoming audio can be delayed by the same amount to line things up again.

As you may see the second approach has various issues implementation issues regarding timecode and transport synchronization. Ardour uses internal read-ahead to compensate for latency. The time displayed in the Ardour clock corresponds to the audio-signal that you hear on the speakers (and is not where ardour reads files from disk).

NB. this is also one of the reasons why many projects start at timecode 01:00:00:00. When compensating for output-latency the DAW will need to read data from before the start of the session so that the audio arrives in time at the output when the timecode hits 01:00:00:00. Ardour3 does handle the case of 00:00:00:00 properly but not all systems/software/hardware that you may inter-operate with may behave the same.

To achieve sample accurate timecode synchronization, the latency introduced by the audio-setup needs to be known and compensated for.

In order to compensate for Latency, JACK or JACK applications need to know exactly how long a certain signal needs to be read-ahead or delayed:

Figure 2: Jack Latency Compensation. This figure outlines the jack latency API. -- excerpt from http://jackaudio.org/files/jack-latency.png

In Figure 2, clients A and B need to be able to answer the following two questions:

• how long has it been since te data read from port Ai or Bi arrived at the edge of the JACK graph (capture)?
• how long will it be until teh data writen to port Ao or Bo arrives at the edge of the JACK graph (playback)?

JACK includes an API that allows applications to determine the answers to above questions. However JACK can not know about the additional latency that is introduced by the computer architecture, operating system and soundcard. These values indicated by -I and -O in Figure 2 and vary from system to system but are generally constant values. On a general purpose computer system the only way to accurately learn about the total latency is to measure it.

Linux DSP guru Fons Adriaensen wrote a tool called jack_delay to accurately measure the roundtrip latency of a closed loop audio chain, with sub-sample accuracy. JACK itself includes a variant of this called jack_iodelay.

Jack_iodelay allows you to measure the total latency of the system, subtracts the known latency of JACK itself and suggests parameters for jackd's audio-backend -I and -O options.

jack_[io]delay works by emitting some rather annoying tones, capturing them again after a round trip through the whole chain, and measuring the difference in phase so it can estimate with great accuracy the time taken. This is not a theoretical estimation, jack_delay is a measuring tool that will provide very accurate answers.

You can close the loop in a number of ways:

• Putting a speaker close to a microphone. This is rarely done, as air propagation latency is well known so there is no need to measure it.
• Connecting the output of your audio interface to its input using a patch cable. This can be an analog or a digital loop, depending on the nature of the input/output you use. A digital loop won't factor in the converters latency.

Once you have closed the loop you have to:

1. Launch jackd with the configuration you want to test.
2. Launch jack_delay.
3. Make the appropriate connections between your jack ports so the loop is closed.
4. Adjust the playback and capture levels in your mixer.

There are three common timecode formats:

• LTC – Linear/Longitudinal Time Code
• MTC – MIDI Time Code
• MIDI-Clock – tempo based time

As well as a JACK specific timecode implementation:

• JACK-transport

Ardour supports all of these standards. It can generate timecode and thus act as timecode master providing timecode information to other applications. Ardour can also be slaved to some external source in which case the playhead follows the incoming timecode.

Combining the timecode slave and generator modes, Ardour can also translate timecode. e.g create LTC timecode from incoming MTC.

Each Ardour session has a specific timecode frames-per-second setting which is configured in session→properties→timecode.

This is pretty straight forward: simply turn it on. The MTC and MIDI-Clock generator do not have any options. For the LTC generator the volume of the generated LTC can be configured. JACK-transport can not be generated jack itself is always sample-sync to the jack-cycle and does not slave to anything.

The relevant settings for timecode generator can be found in the Preferences dialog: “MIDI Preferences” (for MTC, MClk) and “Transport Preferences” respectively.

The timecode is sent to jack-ports ardour:MTC out, ardour:MIDI clock out and ardour:LTC-out. Multiple generators can be active simultaneously.

Note that -as of writing- only the LTC generator supports latency compensation. This is due to the fact the ardour MIDI ports are not yet latency compensated.

In session→properties it is possible to define an offset between Ardour's internal time and the timecode sent. Currently only the LTC generator honors this offset.

There are no options. Ardour sends full MTC frames whenever the transport is relocated or changes state (start/stop). MTC quarter frames are sent when the transport is rolling and the transport speed is within 93% and 107%.

The volume of the LTC signal can be conigured in in the Preferences→Transport dialog. By default it is set to -18dBFS which corresponds to 0dBu in an EBU calibrated system.

The LTC generator has an additional option to keep sending timecode even when the transport is stopped. This mode is intended to drive analog tape machines which unspool the tape if no LTC timecode is received.

Switching the timecode-source can be done via the button just right of Ardour's main clock. By default it is set to Internal in which case ardour will ignore any external timecode. The button allows to toggle between Internal and the configured timecode source which is chosen in Edit→Preferences→Transport.

When ardour is chasing an external timecode source the following cases need to be distinguished:

1. the timecode source shares the clock
2. the timecode source is independent (no wordclock sync)

and

1. the timecode source uses the same FPS setting as ardour
2. the timecode source runs at different frames-per-second

In both cases the first option is preferred: clock sync + same FPS setting.

If the frames-per-second don't match, ardour can either re-calculate (map) the frames or the configured FPS (session→properties) can be changed automatically while the Slave is active. The behavior is configured with the checkbox in Edit→Preferences→Transport labeled Match session video frame rate to external timecode: When enabled the session video frame rate will be changed to match that of the selected external timecode source. When disabled the session video frame rate will not be changed to match that of the selected external timecode source. Instead the frame rate indication in the main clock will flash red and Ardour will convert between the external timecode standard and the session standard.

An edge case can also occur with 29.97 drop-frame timecode. While the SMPTE 12M-1999 specifies 29.97df as 30000/1001 frames per second, not all hardware devices follow that standard. The checkbox Lock to 29.9700 fps instead of 30000/1001 allows to use a compatibility mode for those devices:

When enabled the external timecode source is assumed to use 29.970000 fps instead of 30000/1001. SMPTE 12M-1999 specifies 29.97df as 30000/1001. The spec further mentions that drop-frame timecode has an accumulated error of -86ms over a 24-hour period. Drop-frame timecode would compensate exactly for a NTSC color frame rate of 30 * 0.9990 (ie 29.970000). That is not the actual rate. However, some vendors use that rate - despite it being against the specs - because the variant of using exactly 29.97 fps yields zero timecode drift.

As described in the introduction, timecode and clock are independent. If the external timecode-source is not sample-sync with the audio-hardware (and jack), ardour needs to vari-speed to adjust for the discrepancy.

The checkbox External timecode is sync locked allows to select the behavior according to your setup. When enabled indicates that the selected external timecode source shares sync (Black & Burst, Wordclock, etc) with the audio interface.

In other words: if enabled, ardour will only use perform initial synchronization and keep playing at speed 1.0 instead of vari-speed adjusting to compensate for drift.

Note that vari-speed is unavailable when recording in ardour and all tracking happens at speed 1.0. So if you want to record in sync with external timecode it must be sample-locked or it will drift over time.

MIDI Clock is not a timecode format but tempo-based time. The absolute reference point is expressed as beats-per-minute and Bar, Beat and Tick. There is no concept of sample-locking for Midi clock signals. Ardour will vari-speed if necessary to chase the incoming signal.

Note that the MIDI Clock source must be connected to ardour:MIDI clock in port.

The LTC slave decodes an incoming LTC signal on a jack-audio port. It will auto-detect the frame-rate and start locking to the signal once two consecutive LTC frames have been received.

The incoming timecode signal needs to arrive at the ardour:LTC-in port. Port-connections are restored for each session and the preference dialog offers an option to select it for all sessions.

Ardour's transport is aligned to LTC-frame start/end positions according to the SMPTE 12M-1999 spec which means that the first bit of an LTC-Frame is aligned to different Lines of a Video-Frame, depending on the TV standard used. Only for Film (24fps) does the LTC-Frame directly match the video Frame boundaries.

Figure 3: LTC frame alignment for the 525/60 TV standard

Ardour supports vari-speed and backwards playback but will only follow speed changes if the sync locked configuration option is disabled.

While Ardour is chasing LTC, the main transport clock will display the received Timecode as well as the delta between the incoming signal and Ardour's transport position.

A global offset between incoming timecode and ardour's transport can be configured in Session→Properties.

The user-bits in the received LTC frame are ignored.

Ardour's MTC slave parses full timecode (sysex messages) as well as MTC quarter-frames arriving on the ardour:MTC in port. The transport will only start rolling once a complete sequence of 8 quarter frames has been received.

Ardour supports vari-speed and backwards playback but will only follow MTC speed changes if the sync locked configuration option is disabled.

While Ardour is chasing MTC, the main transport clock will display the received Timecode as well as the delta between the incoming signal and Ardour's transport position.

A global offset between incoming timecode and ardour's transport can be configured in Session→Properties.

When slaved to jack, Ardour's transport will be identical to JACK-transport. As opposed to other slaves, Ardour can be used to control the JACK transport states (stopped/rolling). No port-connections need to be made for jack-transport to work.

JACK-transport does not support vari-speed, nor offsets. Ardour does not chase the timecode but is always in perfect sample-sync with it.

JACK-transport also includes temp-based-time information ie. Bar:Beats:Ticks and beats-per-minute. However, only one JACK application can provide this information at a given time. The checkbox JACK Time Master in the Session→Properties dialog allows to configure ardour to act as translator from timecode to BBT information.

Timecode related settings are accessed from the menus

Session→Properties→Timecode

Edit→Preferences→Transport

Edit→Preferences→MIDI

• “Timecode frames-per-second” – configure Timecode frames-per-second (23.976, 24, 24.975, 25, 29.97, 29.97 drop, 30, 30 drop, 59.94, 60). Note that all fractional framerates are actually fps*(1000.0/1001.0).
• “Pull up/down” – video-pullup modes change the effective samplerate of Ardour to allows for changing a film soundtrack from one frame rate to another. see Telecine
• “Slave Timecode offset” – The specified offset is added to the received timecode (MTC or LTC).
• “Timecode Generator offset” – Specify an offset which is added to the generated timecode (so far only LTC).
• “JACK Time Master” – provide Bar|Beat|Tick and other information to JACK

• “External timecode source” – select timecode source: JACK, LTC, MTC, MClk
• “Match session video frame rate to external timecode” – This option controls the value of the video frame rate while chasing an external timecode source. When enabled the session video frame rate will be changed to match that of the selected external timecode source. When disabled the session video frame rate will not be changed to match that of the selected external timecode source. Instead the frame rate indication in the main clock will flash red and Ardour will convert between the external timecode standard and the session standard.
• “External timecode is sync locked” – When enabled indicates that the selected external timecode source shares sync (Black & Burst, Wordclock, etc) with the audio interface.
• “Lock to 29.9700 fps instead of 30000/1001” – When enabled the external timecode source is assumed to use 29.97 fps instead of 30000/1001. SMPTE 12M-1999 specifies 29.97df as 30000/1001. The spec further mentions that drop-frame timecode has an accumulated error of -86ms over a 24-hour period. Drop-frame timecode would compensate exactly for a NTSC color frame rate of 30 * 0.9990 (ie 29.970000). That is not the actual rate. However, some vendors use that rate - despite it being against the specs - because the variant of using exactly 29.97 fps has zero timecode drift.

• “Send MIDI Timecode” – enable MTC generator
• “Send MIDI Clock” – enable MIDI Clock generator