Re: RFC - Work on FreeBSD's Audio Stack

On Thursday, December 14, 2023 3:59:13 AM CET Joseph Mingrone wrote:
> Re-sending the text because the original contained a few errors, the worst
> being broken URLs from a PDF to text conversion.
> 
> Hello,
> 
> The FreeBSD Foundation will be contracting out work on FreeBSD's audio
> stack.  If you have any comments about the work, please share them before
> next Wednesday, December 20.
> 
> Joe

Any effort in this department is very welcome indeed. I seem to be in the 
target group of this project - I'm the maintainer of audio/ardour and audio/
jack, and I wrote both the current and the upcoming (low latency) OSS backend 
for Jack. Currently I am working on an improved driver for snd_hdspe(4).

> 
> * Project Description
> 
> The end goal of the project is to provide FreeBSD audio developers with
> useful tools and frameworks to make sound development on FreeBSD easier,
> without having to resort to custom solutions for everything.
> 
> On the user side, FreeBSD will ship with a more stable audio stack and a
> larger collection of userland programs, making for a more coherent
> ecosystem in general. OSS generally does not come with many native tools
> except mixer(8), and users have to resort to using tools from multiple
> different audio systems (ALSA, PulseAudio, sndio) to compensate.
> Additionally, I think the introduction of new development frameworks will
> encourage more native audio development, which is going to be especially
> beneficial for people - me included - who want to use FreeBSD for music
> production.

Let's be realistic here. The OSS API is a dead end in the middle to long term. 
It's neither widely used nor particularly well suited for modern hardware. And 
it's conceptually broken on many levels, including parameter negotiation, 
consistency of parameters, buffer and latency management, error handling.

Fixes and usability improvements are a good thing. But we have to think about 
what to replace the OSS API with and how much time we still want to invest 
into the OSS ecosystem.

Besides that, for most users the OSS API is just a sink for their primary 
sound server like PulseAudio, sndio, pipewire. These will not go away. I'd 
focus on complementary features and coexistence here.

Music production is a very different beast altogether. It'll always need some 
more knowledge and custom solutions.

> 
> snd uaudio(4) fixes
> 
> The project will also address bugs in the USB audio driver, snd uaudio(4),
> which I have been able to reproduce using my Focusrite Scarlett USB sound
> card, with the most prominent and consistent one being noise produced
> during the first 12 seconds of playback and when moving along a
> track/video. If the user tries to move forward multiple times in a short
> time window, the audio device most of the time becomes unusable (i.e no
> audio) and it has to be replugged. Though this issue is largely bypassed if
> audio is routed to the USB device through virtual oss, this is still a bug
> that needs to be addressed.

From the description here this sounds more like an issue with the player or 
the vchan feeders, given that virtual_oss doesn't produce noise. Did you file 
a bug report?

> 
> Related bug reports include:
> 
>    - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257082
>    - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194527

I'm not convinced that these are related. There's probably better examples for 
snd_uaudio problems, if you skim the bug reports.

> 
> Initially I am going to test whether the open() and read() routines cause
> this issue, as DTrace suggests that this happens around the same time
> open(2) or the first read(2) is called. As mentioned in the previous
> paragraph, virtual oss partially fixes this issue, so I would like to
> investigate and understand why, and maybe find the root cause that way.
> Another source of information will be Linux’s Scarlett and USB audio
> drivers which, as far as I know, do not have this issue.

Most problems I've seen are about bad vchan constellations. A good 
troubleshooting checklist would be of great help for bug reports, something 
along the lines of

# sysctl hw.snd.verbose=2
# cat /dev/sndstat
# ...

and some more user friendly output from /dev/sndstat etc.

My Scarlett18i20 works fine, BTW. Maybe a different hardware generation 
though.

> 
> Other USB audio bugs include 1) those mentioned in the snd uaudio(4) man
> page, 2) snd uaudio(4) not passing enough information (e.g device name,
> associated device in /dev, and more) to the OSS API, 3) no explicit list of
> supported sound cards.
> 
> MIDI device cloning
> 
> Unlike DSP devices, sound(4) does not allow cloning of MIDI devices, meaning
> applications cannot open the same MIDI device multiple times (e.g when
> recording more than one track with the same device). This can be verified
> by tracing the dsp clone() routine found in sys/dev/sound/pcm/dsp.c, which
> is triggered during a dev clone EVENTHANDLER(9) event. For example, given a
> /dev/dsp8.0 device, trying to cat(1) /dev/dsp8.1 would trigger dsp clone()
> and create it on the fly (see FILES section in sound(4) man page). However,
> doing that with a MIDI device would trigger the handler, but result in an
> error, as MIDI devices do not support cloning. To fix this, the MIDI code
> will be patched to use the snd clone framework used for cloning DSP
> devices.

Is there really demand for this? Multiplexing MIDI is usually done in the 
sound server or DAW. And ALSA MIDI ports can be cloned, AFAIK, so audio 
software ported from Linux will be fine without it.

> 
> Other kernel improvements
> 
> Other improvements to the kernel code include 1) better-syncing with the
> 4Front OSSv4 API, 2) addressing open bug reports, 3) making optimizations
> where possible.

Again, OSSv4 compatibility won't help much, it's basically abandoned by audio 
software developers. Apart from open bug reports there's also missing kevent 
support (instead of poll() being the only option), or the buffer / blocksize 
based latency setting which is conceptually broken. I can give more details on 
these topics if there's interest.

> 
> oss(3)
> 
> Following the motivation behind mixer(3), which was to make OSS mixer
> development more pleasant and easier, I will implement an oss(3) library,
> responsible for handling the audio device’s state. oss(3) will serve as a
> unified interface for OSS audio and MIDI developers and allow for quicker
> application development, like Digital Audio Workstations, instrument
> effects, audio servers, and general utilities.
> 
> Widely-used existing codebases that can benefit from oss(3) are virtual oss
> and Mozilla’s cubeb oss audio framework, which are also great sources of
> inspiration for features included in oss(3).

What's the scope of this library? Main PCM devices, virtual PCM devices, 
hardware devices? Settings and state only, or also playback and recording 
operation?

Beware that there's a great variety of operation strategies for different use 
cases. With a simple poll() based read() / write() at one end of the spectrum, 
and something like my new Jack OSS backend on the other end. It's timer based, 
preferably mmap()ed, uses its own buffer management and strives to keep 
latency reproducible within +/- 1ms. The backend code is here:

https://github.com/0EVSG/sosso

Due to different requirements, it will be difficult to simplify much of the 
OSS device setup while still catering to all operation strategies.

> 
> audio(8)
> 
> FreeBSD lacks an easy-to-use audio device handling utility similar to
> mixer(8), but for the device-side of OSS, like OpenBSD’s audioctl(8). Such
> a utility will make use of the new oss(3) library, and offer a cleaner and
> user-friendlier interface, concentrating all useful - but scattered -
> information currently provided by /dev/sndstat, hw.snd, dev.pcm and
> hw.usb.uaudio, the OSS API into the output of a single application.

Many of the sysctls mentioned here are not dynamic and only apply on playback 
/ recording restart or device attach. That could limit this tool's usefulness.

For reference, this part of my Jack on FreeBSD notes should give you an 
overview of knobs that are beneficial for music production:

https://github.com/0EVSG/freebsd_jack_notes#system-settings

> 
> Hot-swapping
> 
> Users of plain OSS, that is, without virtual oss, will have noticed that
> hot-swapping audio devices (i.e changing the default unit, hw.snd.default
> unit) mid-track does not work and sound keeps coming out of the previous
> device until the track is restarted. This is because applications open(2)
> the device at the start of playback and close(2) it when the track has
> stopped. virtual oss(8) can create a virtual audio device responsible for
> routing audio to the appropriate physical audio device. The device(s) sound
> is routed to can be changed at run-time using virtual oss cmd(8).
> 
> This method has the advantage that sound can be re-routed to different
> devices while applications keep the same /dev/dsp open; virtual oss(8) will
> make any necessary conversions on the fly and redirect audio to the new
> device without requiring a restart of the track/application.
> 
> This functionality will be embedded in either mixer(8) or the new audio(8)
> program.

What you outline here is a complete sound server, but it doesn't tell how you 
want to implement that. Extend the vchan infrastructure, autostart virtual_oss 
on all PCM devices, or something else?

In many use cases this just duplicates the sound server that sits on top of 
the dsp device. At worst it compromises quality and latency because of 
conversions and routing buffers. As a provocative question, wouldn't the 
average user be served better if we just implemented full pipewire support, 
going forward?

> 
> Testing
> 
> Testing will be done on both VMs and actual hardware, with each patch
> build-tested locally and on the FreeBSD cluster. The project will require
> testing some of the kernel patches with multiple different audio devices.
> Additionally, there will be new tests written for the FreeBSD test suite,
> for both the kernel and userland parts.
> 
> The audio driver will be tested by writing a test program to go through most
> of the IOCTLs provided to by the driver, to both confirm that the
> information returned is correct, and also to make sure that users cannot
> pass values that would break the driver. Exact cases will be considered
> further down the project.

You mean IOCTLs provided by the dsp devices? On a dummy driver? Because the 
hardware drivers are usually well separated in kernel modules, which means 
they can be tested separately.

Sorry to sound a bit negative here, but some of the objectives look rather 
vague to me and I think another iteration of problem analysis and scope 
definition would increase chances of success.

Anyway, I'm glad to help if you have questions or want to discuss something.

Regards,

Florian