Notes on libvirt configuration:

• libvirt debug logging
• cgroup configuration – not much here. now handled by systemd. still some libvirt config items, but I don't tend to touch these anymore.
• security driver configuration
• …

As I get to it…

Libvirtd supports multiple levels of logging, similiar to kernel console logging. Libvirtd supports 2 logging configuration parameters [OK, really 3 if you count log_buffer_size] that reside in:/etc/libvirt/libvirtd.conf:

# Logging filters:
# A filter allows to select a different logging level for a given category
# of logs
# The format for a filter is one of:
#    x:name
#    x:+name
#      where name is a string which is matched against source file name,
#      e.g., "remote", "qemu", or "util/json", the optional "+" prefix
#      tells libvirt to log stack trace for each message matching name,
#      and x is the minimal level where matching messages should be logged:
#    1: DEBUG
#    2: INFO
#    3: WARNING
#    4: ERROR
#
# Multiple filters can be defined in a single @filters, they just need to be
# separated by spaces.
#
# e.g. to only get warning or errors from the remote layer and only errors
# from the event layer:
#log_filters="3:remote 4:event"

Each libvirt source file that uses the logging facility defines and initializes a static virLogSource virLogSelf structure using the VIR_LOG_SOURCE() macro, specifying the “log source name”. virLogSelf is passed as the first argument to internal logging functions by the top level logging macros indicating the “source” of the log message. [Earlier versions of libvirt log filtering used to use the source file path name as the “source”.] The log source names use a period or “dot” to separate filter name components. Libvirt Log Filter Names contains a list of the log sources defined for the indicated libvirt version.

Filter comparisons are made using a simple strstr(3) search for the filter string [the “needle”] in the log source's name string [the “haystack”]. The first [leftmost] filter to match applies even when filters later in the list would also match. So, the filter qemu will match all of the dozen or so qemu.* sources listed in Libvirt Log Filter Names.

A filter that I have found useful to monitor libvirt API calls, external commands, interactions with the VMs' qemu monitor and cpu/cgroup functions:

log_filters="1:libvirt 1:qemu 1:cpu 1:cgroup 1:command"

This filter will include debug level [1] log messages from all source files in the source directories src/{libvirt,qemu,cpu,cgroup,command}. To eliminate debug messages from some particularly chatty but less interesting source files in a directory from which you want most debug message, assign warning level [3] to those source files. For example, to exclude info and debug messages from src/qemu/qemu_capabilities.c while including debug messages from all other qemu driver source files, one could use:

log_filters="1:libvirt 3:qemu_capabilities 1:qemu 1:cpu 1:cgroup 1:command"

With this filter libvirt will still log warning and error messages from qemu_capabilities.c.

# Logging outputs:
# An output is one of the places to save logging information
# The format for an output can be:
#    x:stderr
#      output goes to stderr
#    x:syslog:name
#      use syslog for the output and use the given name as the ident
#    x:file:file_path
#      output to a file, with the given filepath
# In all case the x prefix is the minimal level, acting as a filter
#    1: DEBUG
#    2: INFO
#    3: WARNING
#    4: ERROR
#
# Multiple outputs can be defined, they just need to be separated by spaces.
# e.g. to log all warnings and errors to syslog under the libvirtd ident:
#log_outputs="3:syslog:libvirtd"
#

The commented out definition above is apparently the default. It will log warnings and errors to the syslog [/var/log/syslog] using tag 'libvirtd'. By default libvirtd also apparently logs errors to /var/log/libvirt/libvirtd.log.

I usually configure the following output:

log_outputs="3:syslog:libvirtd 1:file:/var/log/libvirt/libvirt_debug.log"

That is, additionally log ALL message levels in the filtered categories to the libvirt_debug.log file.

12:56:38       kashyap ▒ blizzow: Speaking of verbosity, a non-straighforward way is to enable log filters in libvirtd.conf that shows
                       ▒ what commands libvirt is sending to QEMU via QMP
12:57:17       kashyap ▒ blizzow: I normally use these log filters if I want to debug migration related bugs:
12:57:19       kashyap ▒ log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util 1:qemu_monitor"
12:57:30       kashyap ▒ log_outputs="1:file:/var/log/libvirt/libvirtd.log"
12:57:54       kashyap ▒ After that, restart the libvirt daemon, and you should see all the the interactions b/n libvirt & QEMU in the
                       ▒ said log.

The libvirt-bin package installs a default, NAT'd virtual network definition under /etc/libvirt/qemu/networks/default.xml. You can see this network with:

root@bl460gen8-15(root):virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              inactive   no            yes

From the directory path, it appears that the qemu hypervisor is the only one for which libvirt manages virtual networks.

As with libvirt domains, you need the '–all' to list inactive networks. The 'default' virtual network is not automatically started when libvirt-bin is first installed. On upgrade, the package scripts will attempt to preserve the pre-upgrade configuration.

The initial, installed default network configuration can be viewed with “virsh net-edit default” or “virsh net-dumpxml default”.

<network>
  <name>default</name>
  <uuid>554137cf-b00e-43a3-a0c2-60bfc94ee45d</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

The uuid is generated by libvirtd on first definition of the network. Libvirt will also generate a mac address for the bridge when first started.

Note that the default network uses the subnet 192.168.122.0/24. You can create additional virtual networks by passing a similar xml definition with a different <name>, no <uuid> [let libvirt generate it] and a different subnet, such as 192.168.123.0, … using the “virsh net-define <xml-file>” command.

You can also specify the host names and IP address to associate with specific mac addresses. Establishing conventions for contiguous ranges of mac addresses and corresponding IP addresses can simplify automating multi-VM tests with, e.g., pdsh. Here's a snippet from a default network definition used with multiple automatically cloned hLinux test VMs:

<network>
  <name>default</name>
  <uuid>d172515e-535e-451c-8ad6-b7bd2cadcdad</uuid>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:cb:09:d3'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <host mac='52:54:00:3e:4c:60' name='fedora-guest' ip='192.168.122.60'/>
      <host mac='52:54:00:3e:4c:80' name='hlinux-guest' ip='192.168.122.80'/>
      <host mac='52:54:00:5e:6c:71' name='precise-guest' ip='192.168.122.71'/>
      <host mac='52:54:00:5e:6c:72' name='trusty-guest' ip='192.168.122.72'/>
      <host mac='52:54:00:5e:6c:73' name='guest-73' ip='192.168.122.73'/>
      <host mac='52:54:00:5e:6c:74' name='guest-74' ip='192.168.122.74'/>
      <host mac='52:54:00:5e:6c:80' name='hltest-base' ip='192.168.122.100'/>
      <host mac='52:54:00:5e:6c:81' name='hltest-01' ip='192.168.122.101'/>
      <host mac='52:54:00:5e:6c:82' name='hltest-02' ip='192.168.122.102'/>
      <host mac='52:54:00:5e:6c:83' name='hltest-03' ip='192.168.122.103'/>
      <host mac='52:54:00:5e:6c:84' name='hltest-04' ip='192.168.122.104'/>
      <host mac='52:54:00:5e:6c:85' name='hltest-05' ip='192.168.122.105'/>
      <host mac='52:54:00:5e:6c:86' name='hltest-06' ip='192.168.122.106'/>
      <host mac='52:54:00:5e:6c:87' name='hltest-07' ip='192.168.122.107'/>
      <host mac='52:54:00:5e:6c:88' name='hltest-08' ip='192.168.122.108'/>
      ...

To start the default virtual network use “virsh net-start <network-name>”.

root@bl460gen8-15(root):virsh net-start default
Network default started

root@bl460gen8-15(root):virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     no            yes

Once started, the network shows up in “virsh net-list” output as active, without the '–all' option.

If you want a libvirt virtual network started automatically at boot, enable autostart for that network:

root@bl460gen8-15(root):virsh net-autostart default
Network default marked as autostarted

root@bl460gen8-15(root):virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes

To stop or shutdown a libvirt virtual network, use “virsh net-destroy <network>”. You'll need to destroy and re-start a network to pick up additional hosts, e.g., after editing the xml with “virsh net-edit”. All running VMs will lose any connectivity via the destroyed network, so it's a good idea to add a few spare hosts if you want subsequently defined VMs to have well know static IP addresses.