NIC partitioning, SR-IOV and PF-IOV modes on SFN7XXX/8XXX adapters
------------------------------------------------------------------
This release of the Solarflare Linux Net driver supports NIC partitioning
(multiple PFs) and SR-IOV (VFs).

See "Solarflare Server Adapter User's Guide" SF-103837-CD Issue >=12
available from https://support.solarflare.com for more detailed information
on setting up NIC partitioning, SR-IOV or PF-IOV modes.

Some additional details as well as known issues are documented below:
General:
  ~ NIC partitioning/SRIOV/PF-IOV features require sfutils >=v4.3.0.1010 which
    contains adapter firmware v4.2.1.1014 and firmware-variant=full-feature
    (setup with sfboot)
  ~ Please note that Hybrid SR-IOV support for SFN5XXX|6XXX drivers is now
    deprecated. Please use an older driver to continue using this feature.
  ~ Link control (e.g. setting pause frames with ethtool and controlling MTU) is
    only available on the first PF on each port; (this is referred to as "PF0"
    or the "primary PF")
  ~ Only link control functions have the full MAC statistics available to them
    (via ethtool -S). All statistics that come from the MAC are now prefixed
    "port_" as they refer to a single physical network port.
    All other functions have statistics which show the number of packets and
    bytes both received and transmitted by that function, classified into
    unicast, multicast and broadcast traffic.
  ~ MTD partitions are only exposed from PF0 on each port, hence the utilities
    sfupdate, sfboot and sfkey need to be run where they have access to PF0.
  ~ It is now possible to identify which physical port an interface is bound to.
    See /sys/class/net/<interface>/device/physical_port (note this is zero
    based whereas the identifiers printed on the adapter brackets are one based)
  ~ The driver supports the use of shared RSS indirection tables, after the
    16 unique hardware RSS indirection tables entries have been consumed.
  ~ In order for active-backup bonding to work, the slave interface normally
    takes the MAC address of the master interface. Only PF0 is allowed to
    change its MAC address. Therefore for this to work with other functions it
    is necessary to set "sfboot mac-spoofing=enabled". An alternative is to
    set the bonding module option "fail_over_mac" and allow the gratuitous ARP
    to inform the other link partners of the change in MAC address.
  ~ At this time PTP will only work with the PF0 on each port and will
    not work with a VF.

NIC partitioning support on SFN7XXX/8XXX adapters:
  ~ The sfc.ko driver will bind to multiple PFs on the same physical adapter
    port and create a network interface for each PF.
  ~ Configure with "sfboot switch-mode=partitioning pf-count=X" and cold reboot
    the server if warned to do so.
  ~ If using NIC partitioning in the same layer2 broadcast domain you may
    need the kernel parameter "arp_ignore=2" as linux will reply to ARP
    on any interface. This might cause traffic to go over unexpected interfaces.
  ~ If using NIC partitioning with VLANs all PFs must be on a unique VLAN
    (which can include VLAN 0). Traffic that is sent/received from this PF will
    have VLAN tags transparently inserted/stripped by the adapter. A single PF
    on each port can be on VLAN0 where it will both receive and transmit
    untagged traffic.
  ~ Configure NIC partitioning with VLANs using
    "sfboot pf-vlans=<VLAN>,<VLAN>..."
  ~ Do not use LACP bonding modes when using NIC partitioning as the LACP
    partner (switch) will not be aware of the partitioning.
  ~ The driver parameter "num_vis" is now more important. This controls how many
    extra VIs (RX/TX/event rings) are allocated for the use of onload on each
    PF. If not using onload you may wish to set "num_vis=0" so that there
    are no false resource limitations when using multiple PFs.
  ~ When PF0 is put into promiscuous mode it will receive traffic for its own
    MAC address as well as any other incoming traffic that does not match
    another PF's MAC address. For other PFs there is no change in received
    traffic if the interface is put into promiscuous mode.

SR-IOV support on SFN7XXX/8XXX adapters:
  ~ The sfc.ko driver will bind to all VFs and create network interfaces.
  ~ Configure with "sfboot switch-mode=sriov pf-count=1 vf-count=X" and cold
    reboot the server if warned to do so.
  ~ RHEL6.5 and RHEL7 are supported for the host/hypervisor OS.
  ~ It is possible to consume the VFs using these KVM/libirt modes:
     - network hostdev (VF passed into the VM; migration is only possible if
                        the VF is hot-unplugged)
     - direct-passthrough  (macvtap interface used over the VF and virtio-net
                            driver used in the VM guest)
  ~ Note that VFs will initially be assigned random MAC addresses, but libvirt
    (and ip link) can override the MAC address.
  ~ A VF in a VM will be unable to set its own MAC address. See the item on
    active-backup bonding.
  ~ To enable VFs (for RHEL6.5 and RHEL 7)
      echo <num vfs> > /sys/class/ethX/device/sriov_numvfs
  ~ To enable VFs on older operating systems set the sfc.ko module parameter
    "max_vfs". If set to a single integer this is the number of VFs to enable
    and will apply to all PFs. If you wish to configure this to be a different
    value for each PF set this to a comma separated list.
  ~ Please ensure that you enable all BIOS settings that refer to virtualization
    and SR-IOV (note there might be several e.g. SRIOV, IOMMU, VT, VT-x, VT-d,
    IO virtualization)
  ~ Solarflare recommends setting "pci=realloc" as a kernel parameter
    (e.g. in /boot/grub/grub.conf). This allows the kernel to reprogram the
    initial BIOS PCI apertures in case PCI aperture space for the VFs is not
    assigned by the BIOS.
  ~ To view which VFs are associated with which PFs and the VF MAC addresses
    please use "ip link show"
  ~ A PF put into promiscuous mode will not see the traffic to/from a VF.
  ~ A VF put into promiscuous mode will not see any additional traffic.
  ~ Setting VF MAC/VLANs/link state is possible via the ip command
    (or libvirt configuration). From "ip link help"
      "ip link set { dev DEVICE}
         [ vf NUM [ mac LLADDR ]
                  [ vlan VLANID ]
                  [ state { auto | enable | disable} ] ]"
  ~ Setting VF spoofcheck/rate/qos (e.g. via the ip command) is not supported
    at this time.

SR-IOV with NIC partitioning mode support on SFN7XXX/8XXX adapters:
  ~ This mode offers partitioning of the adapter. Each partition is on a unique
    VLAN and within this layer2 broadcast domain there is switching on the
    adapter between the PF and VFs.
  ~ This support is currently considered EXPERIMENTAL i.e. all features are
    present, but there has only been limited testing. Solarflare will support
    the use of this feature in a test environment, but does not recommend use
    in a production environment until this feature becomes GA.
    Note that both SR-IOV and NIC partitioning as individual features are both
    considered GA quality.
  ~ Ensure you are using adapter firmware >= v4.4.2.1011
  ~ It is necessary to configure NIC partitioning with unique VLANs in this mode
  ~ Configure with:
    "sfboot switch-mode=partitioning-with-sriov \
       pf-count=X  pf-vlans=<VLAN>[,<VLAN>]... \
       vf-count=Y"
    and cold reboot the server if warned to do so.

PF-IOV mode support on SFN7XXX/8XXX adapters:
  ~ This mode offers switching between PFs. Use this mode where a server does
    not support SR-IOV (otherwise please use SR-IOV mode as this provides up-to
    240VFs versus only 16PFs)
  ~ Configure with "sfboot switch-mode=pfiov pf-count=X" and cold reboot the
    server if warned to do so.
  ~ In RHEL7 passing through a PCI function uses the new VFIO mechanism.
    Due to a missing capability flag on SFN7XXX/8XXX hardware, VFIO will only
    allow all PFs to be passed into a guest together. This will be addressed
    with a kernel quirk.
