PCI PF memory decode disable when sizing VF BARs

[Eric Badger]

Hi Ryan and -current,

During IOV config, when setting up VF bars, several calls are made to 'pci_read_bar' (in sys/dev/pci/pci.c) in order to size VF BARs, which causes memory decoding to be turned off temporarily for the PF associated with those VFs. I'm finding that this can interfere with an already running PF. I've several thoughts about how this might be handled, but I'm not convinced I understand all of the consequences each of them entails, so any thoughts from others would be appreciated. Here are ideas I've considered:

1. Check the value of the 'reg' arg to 'pci_read_bar' and, if it is outside a standard BAR range, don't disable memory decoding. This is simple, but feels a little hackish and may have consequences I'm missing.
2. Pass some flag/context through such that pci_read_bar knows it is configuring VF BARs (we might instead disable VF MSE in this case, if it is enabled). It would be necessary to carry this flag/context through several function calls before reaching pci_read_bar, which might end up being ugly.
3. Rearrange the calls so that VF BARs are sized when the PF is not yet running, and that info saved until VFs are created. Probably it would be done when the PF BARs are sized for any device supporting IOV, even if that device never creates VFs.

Thanks,
Eric