Using bhyve PCI passthrough on OmniOS

drscream@cyber-tec.org (Thomas Merkel) — Wed, 29 May 2019 15:36:00 +0200

Some hardware is not supported in illumos yet, but luckily there is bhyve which supports pci passthrough to any guest operating system. To continue with my OmniOS desktop on “modern” hardware I would love wifi support, so why not using a bhyve guest as router zone which provide the required drivers?

First of all it requires an up-to-date OmniOS version which provide bhyve support. Including the following IPS packages:

$ pkg install system/bhyve
$ pkg install system/zones/brand/bhyve

You can also found additional system requirements at the OmniOS bhyve hypervisor page.

Simple overview

                                   __   _
                                 _(  )_( )_
                                (_  inet  _)
                                  (_) (__)
                                     |
                                     |
                                     |
           o-- global zone ----------|--------------------------o
           |        o-- bhyve zone --|----o                     |
           |        |                |    |                     |
pci-wifi --|- passthrough -> wifi-device  |                     |
           |        |         |           |                     |
           |        |       {NAT}         |                     |
           |        |         |           |                     |
           |        |        vnic <--> [ etherstub ] <---> vnic |
           |        o---------------------o                     |
           |                                                    |
           o----------------------------------------------------o

Global zone network configuration

An etherstub is required which provides a virtual switch between the virtual nic in the global zone and the one in the bhyve zone.

$ dladm create-etherstub switch0

Now it’s required to assign to virtual nics on this etherstub.

$ dladm create-vnic -l switch0 uplink0
$ dladm create-vnic -l switch0 bhyve0

You can verify your configuration with dladm show-vnic. It’s also required to assign an IP address to the uplink0 vnic which allow the communication into the bhyve zone.

$ ipadm create-addr -T static -a 10.1.1.2/30 uplink0/v4

Choose whatever network you like, but because of the communication from only two nics I choose a /30 network.

Configure passthrough device via ppt

Lookup the PCI information for the device you like to passthrough to your bhyve zone. On this example I’m looking for the wireless network card.

$ prtconf -dD
...
   pci8086,1311 (pciex8086,85) [Intel Corporation Centrino Adv....] (...)

Look for the correct pciex entry, I didn’t got it working with the pci entry only. The prtpicl -v tool might also provide you additional information about your PCI devices.

The pciex information need to be stored in the following two files:

The /etc/ppt_aliases should look similar to the /etc/driver_aliases file, which will overwrite or assign the required ppt driver on the device:

ppt "pciex8086,85"

The /etc/ppt_matches file contains only the PCI information:

pciex8086,85

An reboot is recommended to attach the driver on the device. A verification can be done with the following commands:

Look for the attached driver on the device:

$ prtconf -dD | grep ppt

Use pptadm to list dev and path:

$ pptadm list -a
DEV        VENDOR DEVICE PATH
/dev/ppt0  8086   85     /pci@0,0/pci8086,1e12@1c,1/pci8086,1311@0

The /dev/ppt0 will be used later to passthrough it to the bhyve zone.

Use zonecfg to configure guest zone

For the bhyve zone it’s required to create a dataset for the zonepath and a zvol which contains the guest itself. My zone will be called gw because it’s the gateway to the internet.

# Create the zvol
$ zfs create -V 10G rpool/bhyve0

# Create the zonepath
$ zfs create rpool/zones/gw

# Create some dataset for isos
$ zfs create rpool/iso

Download the latest FreeBSD (or any operating system you like) ISO and store it in /rpool/iso. For me it looks like:

$ cd /rpool/iso
$ wget -O freebsd.iso https://download.freebsd.org/...-bootonly.iso

To create a zone via zonecfg, I always recommend create one file which contains the relevant data for zonecfg to work correctly, for example: /tmp/gw.zone:

create -b
set brand=bhyve
set zonepath=/zones/gw
add net
  set physical=bhyve0
  set global-nic=switch0
end
add device
  set match=/dev/zvol/rdsk/rpool/bhyve0
end
add device
  set match=/dev/ppt0
end

add fs
  set dir=/rpool/iso/freebsd.iso
  set special=/rpool/iso/freebsd.iso
  set type=lofs
  add options ro
  add options nodevices
end
add attr
  set name=cdrom
  set type=string
  set value=/rpool/iso/freebsd.iso
end

add attr
  set name=bootrom
  set type=string
  set value=BHYVE_RELEASE
end
add attr
  set name=bootdisk
  set type=string
  set value=rpool/bhyve0
end
add attr
  set name=extra
  set type=string
  set value="-S -s 8:0,passthru,/dev/ppt0"
end

Everything related to the freebsd.iso can be removed later if you do not need the install medium anymore. The BHYVE_RELEASE bootrom is required for FreeBSD to work correctly, it might depends on your operating system which one you need to use.

The following two settings are required to get passthrough working correctly:

This will add the ppt0 device into the zone and allow the required access.

add device
  set match=/dev/ppt0
end

Will add extra parameters to the bhyve command to passthrough /dev/ppt0 correctly to the PCI ID 8:0 in the guest. The PCI ID can be chosen by yourself as long as it’s not used as an parameter already.

add attr
  set name=extra
  set type=string
  set value="-S -s 8:0,passthru,/dev/ppt0"
end

The -S is required to disallow the guest memory to be swapped. Otherwise passthrough isn’t supported!

Finish the configuration and creating the zone

$ zonecfg -z gw -f /tmp/gw.zone

Use zoneadm to install and boot the zone:

$ zoneadm -z gw install
$ zoneadm -z gw boot

If something is going wrong and the zone doesn’t boot correctly check the bhyve log file in zonepath/root/tmp/init.log (example: /zones/gw/root/tmp/init.log).

Configure FreeBSD bhyve guest

You should be able to log into the zone via zlogin:

$ zlogin -C gw

Follow the operating system installation based on you needs. For me it’s an awesome FreeBSD installation which is straightforward anyway.

After the installation some network configuration is required to support the wireless card and setup NAT.

Modify the /boot/loader.conf to load the correct firmware:

if_iwn_load="YES"
iwn6000fw_load="YES"
iwn6000g2afw_load="YES"

wlan_wep="YES"
wlan_ccmp_load="YES"
wlan_tkip_load="YES"

Setup the network interfaces via /etc/rc.conf:

# Check for the guest "ethernet" interface
ifconfig_vtnet0="inet 10.1.1.1 netmask 255.255.255.252"

# Wireless network interface
wlans_iwn0="wlan0"
ifconfig_wlan0="WPA up"
create_args_wlan0="country DE"

# Enable PF and IP forwarding
gateway_enable="YES"
pf_enable="YES"

Add the NAT entry for /etc/pf.conf:

nat on wlan0 from {10.1.1.0/30} to any -> (wlan0)

To configure the wireless network (as required) I use the wpa_supplicant and store the information in /etc/wpa_supplicant.conf:

wpa_passphrase "SSID" "PWD" >> /etc/wpa_supplicant.conf

A reboot might be required to load the required firmware and configure the network as described.

Test network configuration

The global zone should be able to ping the bhyve guest via:

$ ping 10.1.1.1

If you haven’t configured a default gateway yet, it might be good time to do that:

$ route add default 10.1.1.1

After that you should be able to reach other networks and the always loved internet via your awesome bhyve router :-)

Run Docker images on SmartOS

drscream@cyber-tec.org (Thomas Merkel) — Sun, 11 Feb 2018 12:46:00 +0100

This feature is available on the SkyLime SmartOS Version because we merged the changes from an existing issue into our branch to support the Docker Registry Version 2. This has been done because most of the existing Docker images only using version 2, which result in less usable images if you only support version 1. With this change no docker version 1 is supported any more, which is the biggest drawback if you’ve already version 1 images.

Usage

Configure imgadm to add docker hub sources:

$ imgadm sources --add-docker-hub

imgadm avail doesn’t work against the Hub, so you’ll have to search the Hub manually. But you could import images simple via the imgadm import command:

$ imgadm import busybox

Show installed docker images:

$ imgadm list --docker

UUID                                  REPOSITORY                             TAG  IMAGE_ID      CREATED
6357e9ab-0e79-5a0d-697b-b528d925026a  konradkleine/docker-registry-frontend  -    sha256:9976b  2017-10-11T23:50:25Z
5de66518-05f1-1ca2-34ee-6c8750a7a4bb  busybox                                -    sha256:0ffad  2017-11-03T22:39:17Z
1a99421d-7df8-23ec-1758-0b46b730aa1f  registry                               -    sha256:f792f  2017-12-01T22:15:41Z

Configure personal docker registry

Import the official image for the docker registry:

$ imgadm import registry

Install and activate docker registry with the vmadm command. For that store the following file on your SmartOS machine for example in /opt/docker-registry.json:

{
  "alias": "docker-registry",
  "hostname": "docker-registry.dev.example.com",
  "image_uuid": "1a99421d-7df8-23ec-1758-0b46b730aa1f",
  "nics": [
    {
      "nic_tag": "admin",
      "primary": true,
      "ips": [ "172.22.175.100/25" ],
      "gateways": [ "172.22.175.1" ]
    }
  ],
  "brand": "lx",
  "docker": "true",
  "kernel_version": "3.13.0",
  "max_physical_memory": 1024,
  "maintain_resolvers": true,
  "resolvers": [
    "8.8.8.8"
  ],
  "quota": 10,
  "internal_metadata": {
    "docker:cmd": "[\"/bin/sh\", \"/entrypoint.sh\", \"/etc/docker/registry/config.yml\"]"
  }
}

Please modify the ips, gateways and resolvers field in the JSON manifest.

The docker:cmd is based on the Dockerfile from the repository. The image_uuid need to be set to the latest version you’ve downloaded via imgadm. If you need to verify it run:

$ imgadm list --docker

Create and run the container:

$ vmadm create -f /opt/docker-registry.json

At the moment the configuration file described in the docker container show us that it will listen on port 5000.

Provide web interface for personal docker registry

This could be easily done with an image provided on Docker Hub.

$ imgadm import konradkleine/docker-registry-frontend:v2

Save the following manifest which describe the setup of the zone, for example in /opt/registry-web.json:

{
  "alias": "docker-registry-web",
  "hostname": "docker-registry-web.dev.example.com",
  "image_uuid": "6357e9ab-0e79-5a0d-697b-b528d925026a",
  "nics": [
    {
      "nic_tag": "admin",
      "primary": true,
      "ips": [ "172.22.175.101/25" ],
      "gateways": [ "172.22.175.1" ]
    }
  ],
  "brand": "lx",
  "docker": "true",
  "kernel_version": "3.13.0",
  "max_physical_memory": 1024,
  "maintain_resolvers": true,
  "resolvers": [
    "8.8.8.8"
  ],
  "quota": 10,
  "internal_metadata": {
    "docker:cmd": "[\"/root/start-apache.sh\"]",
    "docker:tty": true,
    "docker:attach_stdin": true,
    "docker:attach_stdout": true,
    "docker:attach_stderr": true,
    "docker:open_stdin": true,
    "docker:env": "[ \"ENV_DOCKER_REGISTRY_HOST=172.22.175.100\",\"ENV_DOCKER_REGISTRY_PORT=5000\"]",
    "docker:noipmgmtd": true
  }
}

You should be able to access the web service via http://172.22.175.101, or whatever IP you’re using.

Tipps

Logfiles for docker images are stored in the zone so you need to look there:

$ cat /zones/${UUID}/logs/stdio.log

You may like to login with a shell for some debugging:

$ zlogin -i ${UUID} /native/usr/vm/sbin/dockerexec /bin/sh

VMware vCenter 6.5 SSL ICA not send

drscream@cyber-tec.org (Thomas Merkel) — Sat, 20 Jan 2018 15:08:00 +0100

It looks like there is still a bug in vCenter 6.5 to handle the machine SSL certificates correctly. Somehow the vCenter webserver does not send the intermediate certificates also if they are selected with the certificate manager /usr/lib/vmware-vmca/bin/certificate-manager.

This result in a broken trust path if you connect from any application or from your web browser to the vCenter web interface. You could verify it also with a simple openssl command:

echo | openssl s_client -showcerts -connect vcenter.example.com:443

Mostly you could see it in these lines first lines

depth=0 OU = Domain Control Validated, OU = PositiveSSL, CN = vcenter.example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL, CN = vcenter.example.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=vcenter.example.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA

I have tried to select the correct intermediate certificate chain in the machine SSL certificate and also in the root certificate or in VMware terms in the “signing certificate of the Machine SSL certificate”. Both without any success.

The only thing that worked for me is installing the certificate with a valid chain again via the vecs-cli tool.

Check if the certificate chain are still missing with the following command. If it result in only one certificate I’m sure the chain isn’t send from the web server”

$ /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine_ssl_cert --alias __MACHINE_CERT

Save the machine certificate (server certificate) and the full chain (including all intermediate certificates) in one PEM file. The file content should be in the following order:

machine certificate (server certificate) #1
intermediate certificate #2 which signed the machine certificate #1
intermediate certificate #3 which signed the intermediate certificate #2
…

Also save the machine certificate key on the vCenter server for example both files are saved in /tmp. Then remove the existing certificate:

$ /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CERT

If you’ve saved both files on the server you could install the certificate and key again. Because /tmp/cert_with_chain.crt includes the full chain it will also send from the vCenter web server correctly:

$ /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert /tmp/cert_with_chain.crt --key /tmp/cert.key

Restart the virtual machine or restart the services of the vCenter!

Install ESXi 6.5 on MacMini 5,2

drscream@cyber-tec.org (Thomas Merkel) — Thu, 18 Jan 2018 17:48:00 +0100

There are lot’s of websites around which describe how to install ESXi (VMware vSphere Free) on an MacMini. So this short article only cover some issues.

The free version has some limitations which is mostly no problem for a personal small lab environment. You could download the iso image from my.vmware.com after you’ve created an account.

Installation of version 6.0

I followed the article from derflounder.wordpress.com to install ESXi version 6 first. In an VMware forum somebody told that the version 6.5 isn’t installable on the MacMini because of some hardware detection issues.

Especially if you run MacOS as your desktop system you like to create an bootable USB stick. Do not dd the iso image to the USB stick, this will not work. There is an existing Makefile on GitHub which provide you with most information to generate one. Mostly it’s a simple copy iso content to USB stick and add an additional syslinux.cfg file.

Upgrade to version 6.5

Most people recommended an update with the offline bundle but if you have only a free license the offline bundle couldn’t be downloaded. So I tried to update online with an remote shell on the ESXi, which works perfect.

Enable SSH access to the ESXi by using the troubleshooting menu or web interface:

Connect with your user and password via ssh to the server:

$ ssh root@hostname

The upgrade will be performed via http(s) so we need to allow this outgoing connection:

$ esxcli network firewall ruleset set -e true -r httpClient

Check for the latest version available to run the upgrade:

$ esxcli software sources profile list \
  -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml |\
  grep -i ESXi-6.5

Upgrade to specified version:

$ esxcli software profile update \
  -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml \
  -p ESXi-6.5.0-4564106-standard

Reboot the hypervisor and check the new installed version:

$ vmware -lv

Install new bash version on MacOS without homebrew

drscream@cyber-tec.org (Thomas Merkel) — Thu, 11 Jan 2018 14:12:00 +0100

Also the latest MacOS (High Sierra) provide an outdated bash version (from 2007) I would like to install the latest version without homebrew or similar tools!

I would keep the system as minimal as possible because it will be used to build software. There should be no big difference from a regular MacOS installation.

Download the latest bash source from GNU.org, compile and install it!

$ cd /tmp/
$ curl -O http://ftp.gnu.org/gnu/bash/bash-4.4.12.tar.gz

Extract it, your version may be newer!

$ tar xzf bash-4.4.12.tar.gz
$ cd bash-4.4

All not system defaults are installed in /usr/local in my case.

$ ./configure --prefix=/usr/local
$ make
$ sudo make install

Add the new shell to the list of legit shells:

$ sudo "echo /usr/local/bin/bash >> /private/etc/shells"

Change the default shell for the user:

$ chsh -s /usr/local/bin/bash

Update FreeBSD-current

drscream@cyber-tec.org (Thomas Merkel) — Fri, 07 Jul 2017 09:41:00 +0200

It may sound a little bit strange but mostly I couldn’t remember all the commands to update my FreeBSD-CURRENT installation. For that I share the commands I use in a blog post.

First of all get the newest base and kernel source via git or subversion:

$ cd /usr/src
$ svn update

We’re staying in the /usr/src folder. And building the new world and kernel:

$ make -j `sysctl -n hw.ncpu` buildworld

# Add "KERNCONF=yourkernelname" for a custom kernel
$ make -j `sysctl -n hw.ncpu` buildkernel

Install your kernel and reboot your system. It’s recommended to go to single user mode, but not required.

$ make installkernel
$ reboot

After the reboot we should merge the new and old configuration, for that reason we create a merge environment.

$ cd /usr/src
$ mergemaster -p                          

Install the new world and merge configuration files.

$ make installworld
$ mergemaster -iUF                         

Than we should remove old files and libs which are no longer required.

$ yes | make delete-old
$ yes | make delete-old-libs
$ cd /usr/obj && chflags -R noschg * && rm -rf *

That’s it and reboot. Maybe you need to update and check your userland files.

$ reboot

Sign pkgsrc packages manually

drscream@cyber-tec.org (Thomas Merkel) — Mon, 02 Jan 2017 11:00:00 +0100

Starting a post with “this is only a workaround and you should only use it if you sure what you’re doing” is maybe not the best start but anyway: This is only a workaround if you missed to sign your package or would like todo it for your home environment!

Please have a look at pkgbuild from Jonathan which provides most features to have a nice build environment for pkgsrc.

To sign a package manually and not during the build process you could use the pkg_admin tool. The tool provides an argument gpg-sign-package which allow you to sign packages with GPG based on your mk.conf file.

It’s required to have your GPG environment ready and working. So you need an public/private key pair in your trust store which you could use for signing the package. It’s not required to have gpg-agent or anything running but if not you need so insert your password on each package you like to sign.

The following variables need to be configured in your mk.conf:

# Set your gpg key ID to the following variable
GPG_SIGN_AS=your_gpg_id
# Set the path to the GPG binary
GPG=/opt/pkg/bin/gpg

After this is done you could use the pkg_admin command to sign you package:

$ pkg_admin gpg-sign-package unsigned/vim-nox-8.0.0086.tgz signed/vim-nox-8.0.0086.tgz

Because I’m a little bit lazy and only need to variables in the mk.conf I’ve created the following script which use a temporary mk.conf file for signing:

#!/usr/bin/env bash

GPG_SIGN_AS=your_gpg_id
MK_CONF=$(mktemp -q /tmp/mk-conf.XXXXXXXX)

cat <<EOF >> ${MK_CONF}
GPG_SIGN_AS=${GPG_SIGN_AS}
GPG=$(which gpg)
EOF

pkg_admin -C ${MK_CONF} gpg-sign-package $@
rm "${MK_CONF}"

SmartOS add delegate dataset later to zone

drscream@cyber-tec.org (Thomas Merkel) — Wed, 21 Dec 2016 11:46:00 +0100

Maybe you missed to add a delegate dataset to one of your SmartOS or LX-branded zone and would like to add it later. This is possible with zonecfg, but you’re be warned this is only a workaround and not based on the official SmartOS documentation.

First find out your zone UUID and your zfs_filesystem:

$ vmadm list
# If your zone UUID is df779413-9943-481d-beb8-ba1a376afa2e
$ vmadm get df779413-9943-481d-beb8-ba1a376afa2e | json zfs_filesystem

I would recommend stop your zone before you create the dataset and modify the zone metadata:

$ vmadm stop df779413-9943-481d-beb8-ba1a376afa2e

Create your dataset / file system in the global zone. It should be called data and based on the already existing dataset:

$ zfs create zones/df779413-9943-481d-beb8-ba1a376afa2e/data

Modify the zone metadata with zonecfg:

$ zonecfg -z df779413-9943-481d-beb8-ba1a376afa2e

The commands will add the dataset which you’ve created earlier:

zonecfg:df779413-...> add dataset
zonecfg:df779413-...:dataset> set name=zones/df779413-9943-481d-beb8-ba1a376afa2e/data
zonecfg:df779413-...:dataset> end
zonecfg:df779413-...> verify
zonecfg:df779413-...> exit

After that you could start your zone again and should be able to see the new delegate dataset. You could check this also via vmadm:

$ vmadm get df779413-9943-481d-beb8-ba1a376afa2e | json datasets
[
  "zones/df779413-9943-481d-beb8-ba1a376afa2e/data"
]

Memory and CPU usage tools on Illumos

drscream@cyber-tec.org (Thomas Merkel) — Thu, 03 Mar 2016 11:26:00 +0100

This is only a small overview of memory and CPU usage tools on Illumos. Mostly I forgot the commands so it’s like a personal reference page :-)

Total physical memory

Show DIMM and hardware overview via prtdiag:

prtdiag -v

Show max physical memory via prtconf:

prtconf | grep Memory

Create an overview 10 times every 5 seconds:

vmstat 5 10

Memory usage by process

Show percentage of memory usage by process, you could use sort to sorting the output:

ps -A -o pmem -o pcpu -o args

Overall CPU usage

Two processor reports with five seconds interval:

mpstat 5 2

Important fields are:

usr: user time in percent
sys: system time in percent
idl: idle time in percent

CPU usage by process

Use prstat which is more or less similar to top on Linux. There are different sorting and command line options available.

Sort by cpu usage (default):

prstat

Sort by memory usage:

prstat -s rss

Sort by virtual memory usage:

prstat -s size

Sort by process execution time:

prstat -s time

Find top five processes:

prstat -n 5

Also print process numbers for each user:

prstat -a

Follow a particular process id:

prstat -p <pid>

Follow threads of a particular process id:

prstat -L -p <pid>

Create ISO image from folder in MacOSX

drscream@cyber-tec.org (Thomas Merkel) — Tue, 01 Mar 2016 21:28:00 +0100

You maybe run into an problem that your remote server has no network connection because of some drivers are missing? And the only thing that works is to use IPMI and remote virtual devices which contains your driver?

For that I created an folder which contains the drivers, packages and create an ISO image with hdiutil.

hdiutil makehybrid -o ~/image.iso ~/your/folder -iso -joliet

The tool creates an ISO file in your home, image.iso. You could burn that to an CD or you could use it with your IPMI tool and mount it remotely.