Re: RFC - Work on FreeBSD's Audio Stack

Re-sending the text because the original contained a few errors, the worst being broken URLs from a PDF to text conversion.

Hello,

The FreeBSD Foundation will be contracting out work on FreeBSD's audio stack.  If you have any comments about the work, please share them before next Wednesday, December 20.

Joe

===

* Executive Summary

The FreeBSD audio stack is one of those fields that does not attract the same attention and development as others do,
since it has been left largely unmaintained, and, although high in quality, there is still room for improvement - from lack
of audio development frameworks, to missing userland utilities and kernel driver-related bugs. This project is meant to
touch on all those areas, and as such, will be more of a general improvement project, than an implementation of a specific
feature.

* Project Description

The end goal of the project is to provide FreeBSD audio developers with useful tools and frameworks to make sound
development on FreeBSD easier, without having to resort to custom solutions for everything.

On the user side, FreeBSD will ship with a more stable audio stack and a larger collection of userland programs, making
for a more coherent ecosystem in general. OSS generally does not come with many native tools except mixer(8), and users
have to resort to using tools from multiple different audio systems (ALSA, PulseAudio, sndio) to compensate. Additionally,
I think the introduction of new development frameworks will encourage more native audio development, which is going to
be especially beneficial for people - me included - who want to use FreeBSD for music production.

* Deliverables

Note: By nature of the project, it is possible that the exact details of some deliverables may change. The deliverable list
mentioned in the proposal is likely to grow if time constraints allow, so consider it a non-exhaustive list.

snd hda(4) pin-patching

Regarding the stability of the audio stack, the project will address the pin-patching issue present in the snd hda(4) (Intel
High Definition Audio) driver. Essentially, some laptop models have non-standardized mappings for the headphone and
speaker jack pins, which results in absence of audio from the headphones, until a patch is manually applied in snd hda(4)
to correctly map the headphone and speaker pins for that model so that the same audio stream is output from both the
speakers and headphones when they are plugged in.

The initial strategy to address the issue will be to see if it is possible to do the patching automatically by figuring out
what the Speaker pin’s as number is (see dev.hdac.<n>.pindump=1) and "forcibly" assign the same value to the headphone
jack pin as well. The following solutions follow the same logic:

   - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273809
   - https://i-bsd.com/freebsd-wireless-8265/

OpenBSD’s azalia(4) (their version of the HDA driver) will also serve as a point of reference, even though it too contains
a kind of manual patching mechanism.

snd uaudio(4) fixes

The project will also address bugs in the USB audio driver, snd uaudio(4), which I have been able to reproduce using my
Focusrite Scarlett USB sound card, with the most prominent and consistent one being noise produced during the first 12
seconds of playback and when moving along a track/video. If the user tries to move forward multiple times in a short time
window, the audio device most of the time becomes unusable (i.e no audio) and it has to be replugged. Though this issue
is largely bypassed if audio is routed to the USB device through virtual oss, this is still a bug that needs to be addressed.

Related bug reports include:

   - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257082
   - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194527

Initially I am going to test whether the open() and read() routines cause this issue, as DTrace suggests that this happens
around the same time open(2) or the first read(2) is called. As mentioned in the previous paragraph, virtual oss partially
fixes this issue, so I would like to investigate and understand why, and maybe find the root cause that way. Another
source of information will be Linux’s Scarlett and USB audio drivers which, as far as I know, do not have this issue.

Other USB audio bugs include 1) those mentioned in the snd uaudio(4) man page, 2) snd uaudio(4) not passing enough
information (e.g device name, associated device in /dev, and more) to the OSS API, 3) no explicit list of supported sound
cards.

MIDI device cloning

Unlike DSP devices, sound(4) does not allow cloning of MIDI devices, meaning applications cannot open the same MIDI
device multiple times (e.g when recording more than one track with the same device). This can be verified by tracing
the dsp clone() routine found in sys/dev/sound/pcm/dsp.c, which is triggered during a dev clone EVENTHANDLER(9)
event. For example, given a /dev/dsp8.0 device, trying to cat(1) /dev/dsp8.1 would trigger dsp clone() and create it on
the fly (see FILES section in sound(4) man page). However, doing that with a MIDI device would trigger the handler,
but result in an error, as MIDI devices do not support cloning. To fix this, the MIDI code will be patched to use the
snd clone framework used for cloning DSP devices.

Other kernel improvements

Other improvements to the kernel code include 1) better-syncing with the 4Front OSSv4 API, 2) addressing open bug
reports, 3) making optimizations where possible.

oss(3)

Following the motivation behind mixer(3), which was to make OSS mixer development more pleasant and easier, I will
implement an oss(3) library, responsible for handling the audio device’s state. oss(3) will serve as a unified interface for OSS
audio and MIDI developers and allow for quicker application development, like Digital Audio Workstations, instrument
effects, audio servers, and general utilities.

Widely-used existing codebases that can benefit from oss(3) are virtual oss and Mozilla’s cubeb oss audio framework,
which are also great sources of inspiration for features included in oss(3).

audio(8)

FreeBSD lacks an easy-to-use audio device handling utility similar to mixer(8), but for the device-side of OSS, like
OpenBSD’s audioctl(8). Such a utility will make use of the new oss(3) library, and offer a cleaner and user-friendlier
interface, concentrating all useful - but scattered - information currently provided by /dev/sndstat, hw.snd, dev.pcm and
hw.usb.uaudio, the OSS API into the output of a single application.

Hot-swapping

Users of plain OSS, that is, without virtual oss, will have noticed that hot-swapping audio devices (i.e changing the default
unit, hw.snd.default unit) mid-track does not work and sound keeps coming out of the previous device until the track
is restarted. This is because applications open(2) the device at the start of playback and close(2) it when the track has
stopped. virtual oss(8) can create a virtual audio device responsible for routing audio to the appropriate physical audio
device. The device(s) sound is routed to can be changed at run-time using virtual oss cmd(8).

This method has the advantage that sound can be re-routed to different devices while applications keep the same /dev/dsp
open; virtual oss(8) will make any necessary conversions on the fly and redirect audio to the new device without requiring
a restart of the track/application.

This functionality will be embedded in either mixer(8) or the new audio(8) program.

Bluetooth device management utility

Although not strictly audio-related, I plan to write a (possibly using bsddialog) bluetooth device management utility.
Setting up bluetooth devices on FreeBSD is still rather complicated and confusing, and involves setting up multiple

different programs to even pair with a bluetooth device. This tool will detect nearby bluetooth devices, prompt the user
to choose which one(s) they want to (un-)pair with, and handle configuration automatically.

Other userland improvements

Smaller improvements include 1) revisiting some parts of mixer(3) and mixer(8) with backwards compatibility in mind,
2) writing a MIDI testing utility similar to sndio’s midicat(1) and ALSA’s aseqdump(1).

* Development Process

Development will be done on a local Git branch of main, which will also be available on GitHub. Patches will be submitted
for review on FreeBSD’s Phabricator and eventually committed to upstream by me once reviewed and accepted.

Testing

Testing will be done on both VMs and actual hardware, with each patch build-tested locally and on the FreeBSD cluster.
The project will require testing some of the kernel patches with multiple different audio devices. Additionally, there will
be new tests written for the FreeBSD test suite, for both the kernel and userland parts.

The audio driver will be tested by writing a test program to go through most of the IOCTLs provided to by the driver,
to both confirm that the information returned is correct, and also to make sure that users cannot pass values that would
break the driver. Exact cases will be considered further down the project.

For the userland parts, I will write scripts similar to the old mixer(8)’s test script.

Documentation

The documentation part of the project includes updating the Wiki, Handbook and Foundation pages to reflect the most
recent changes and additions. New Handbook sections will be added to showcase the use of oss(3) and mixer(3). with
additional examples added to /usr/share/examples/sound. Man pages with missing, incomplete or outdated information
will be addressed.