Metadata-Version: 2.4
Name: grifter
Version: 0.2.12
Author: Brad Searle
Author-email: bradleysearle@gmail.com
License: GNU GENERAL PUBLIC LICENSE Version 3
License-File: LICENSE
Requires-Dist: jinja2
Requires-Dist: pyyaml
Requires-Dist: click
Requires-Dist: cerberus
Dynamic: author
Dynamic: author-email
Dynamic: description
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist

# Grifter
Python library to build large scale Vagrant topologies for the networking 
space. Can also be used the build small scale labs for networking/compute 
devices.

[![Build Status](https://travis-ci.org/bobthebutcher/grifter.svg?branch=master)](https://travis-ci.org/bobthebutcher/grifter.svg?branch=master)
[![Coverage Status](https://coveralls.io/repos/github/bobthebutcher/grifter/badge.svg?branch=master)](https://coveralls.io/github/bobthebutcher/grifter?branch=master)

NOTE: Python 3.6+ is required to make use of this library.

```
*****************************************************************
This project is currently in beta and stability is not currently 
guaranteed. Breaking API changes can be expected.
*****************************************************************

```
## Vagrant
What is Vagrant? From the Vagrant [website](https://www.vagrantup.com/docs/index.html)
```
A command line utility for managing the lifecycle of virtual machines
```

## Vagrant Libvirt
What is Vagrant Libvirt? From the `vagrant-libvirt` github [page](https://github.com/vagrant-libvirt/vagrant-libvirt) 
```
A Vagrant plugin that adds a Libvirt provider to Vagrant, allowing Vagrant to control and provision machines via Libvirt toolkit.
```

## Why
When simulating large topologies Vagrantfiles can become thousands 
of lines long. Getting all the configuration correct is often a 
frustrating, error riddled process especially for those not familiar 
with Vagrant. Grifter aims to help simplify that process.

##### Additional project goals
- Generate topology.dot files for use with PTM :heavy_check_mark:
- Generate Inventory files for tools such as Ansible, Nornir

NOTE: Only a `vagrant-libvirt` compatible `Vagrantfile` for 
Vagrant version `>= 2.1.0` will be generated. 

Support for Virtualbox or any other provider type is not supported or
on the road map.

## Dependencies
Grifter requires the help of the following awesome projects from the Python 
community.
- [Cerberus](http://docs.python-cerberus.org/en/stable/) - Schema validation
- [Click](https://click.palletsprojects.com/) - CLI utility
- [Jinja2](http://jinja.pocoo.org/docs) - Template engine
- [PyYAML](https://pyyaml.org/) - YAML all the things

## Installation
There is currently no PyPI release for this project. Grifter can be 
installed directly from source using PIP. 

Create and activate virtualenv.
```
mkdir ~/test && cd ~/test
python3 -m venv .venv
source .venv/bin/activate
```

Install `grifter` with `pip`
```
# Install the master branch.
pip install https://github.com/bobthebutcher/grifter/archive/master.zip
```

Releases are distributed via Github Releases.
```
# Install the latest release.
pip install https://github.com/bobthebutcher/grifter/archive/v0.2.11.zip
```

## Quick Start
Create a `guests.yml` file.
``` 
tee guests.yml > /dev/null << "EOF"
srv01:
  vagrant_box: 
    name: "centos/7"
EOF
```

Generate a Vagrantfile
``` 
grifter create guests.yml
```

Let Vagrant do its magic
``` 
vagrant up
```



## Config File
A file named `config.yml` is required to define the base settings of 
each box managed within the grifter environment. The default `config.yml` 
file can be found [here](grifter/config.yml)

### Box Naming
Grifter expects Vagrant boxes to be named according to the following list.

##### Custom Boxes
- arista/veos
- cisco/csr1000v
- cisco/iosv
- cisco/xrv
- juniper/vmx-vcp
- juniper/vmx-vfp
- juniper/vqfx-pfe
- juniper/vqfx-re
- juniper/vsrx
- juniper/vsrx-packetmode

##### Vagrant Cloud Boxes
- CumulusCommunity/cumulus-vx
- centos/7
- generic/ubuntu1804
- opensuse/openSUSE-15.0-x86_64

#### guest_config
The `guest_config` section defines characteristics about the Vagrant boxes 
used with grifter.
#### Required Parameters.
- data_interface_base
- data_interface_offset
- max_data_interfaces
- management_interface

Note: `data_interface_base` cannot be an empty string. If the box does not 
have any data interfaces the suggested value is "NA". This field will be 
ignored so it can be anything as long as it is not empty.

```yaml
guest_config:
  example/box:
    data_interface_base: "eth" # String pattern for data interfaces.
    data_interface_offset: 0 # Number of first data interface ie: 0, 1, 2, etc..
    internal_interfaces: 0 # Used for inter-box connections for multi-vm boxes.
    max_data_interfaces: 8 
    management_interface: "ma1"
    reserved_interfaces: 0 # Interfaces that are required but cannot be used.

  arista/veos:
    data_interface_base: "eth"
    data_interface_offset: 1
    internal_interfaces: 0
    max_data_interfaces: 24
    management_interface: "ma1"
    reserved_interfaces: 0

  juniper/vsrx-packetmode:
    data_interface_base: "ge-0/0/"
    data_interface_offset: 0
    internal_interfaces: 0
    max_data_interfaces: 16
    management_interface: "fxp0.0"
    reserved_interfaces: 0
```

#### guest_pairs
The `guest_pairs` section is used the define boxes that need two VMs to 
be fully functional. Some examples are the Juniper vMX and vQFX where 
one box is used for the control-plane and another for the forwarding-plane.

NOTE: This functionality will be added in a future release.

#### Custom config files
A default config file ships with the grifter python package.
This file can be customized with your required parameters by creating a 
`config.yml` file in the following locations.
 - `/opt/grifter/`
 - `~/.grifter/`
 - `./` 
 
 Parameters in a users `config.yml` file will be merged with the default 
 `config.yml` file with the user-defined parameters taking preference.

## Usage

#### CLI Utility
Grifter ships with a CLI utility. Execute `grifter -h` to 
discover all the CLI options. 

```
grifter -h
Usage: grifter [OPTIONS] COMMAND [ARGS]...

  Create a Vagrantfile from a YAML data input file.

Options:
  --version   Show the version and exit.
  -h, --help  Show this message and exit.

Commands:
  create   Create a Vagrantfile.
  example  Print example file declaration.
```

#### Create Vagrantfile
```
grifter create guests.yml
```

### Guests Datafile
Guest VMs characteristics and interface connections are defined in a YAML file. 
This file can be named anything, but the recommended naming convention is 
`guests.yml`.

#### Guest Schema
Jinja2 is used a the templating engine to generate the Vagrantfiles.
Guests definition within a guests file must use the following 
schema as it is required to ensure templates render correctly and 
without errors. The guest data will be validated against the schema 
using the Cerberus project.

```yaml
some-guest: # guest name
  vagrant_box: # vagrant_box parameters
    name: # string - required
    version: # string - optional | default: ""
    url: # string - optional | default: ""
    provider: # string - optional | default: "libvirt"
    guest_type: # string - optional | default: ""
    boot_timeout: # integer - optional | default: 0
    throttle_cpu: # integer - optional | default: 0

  ssh: # dict - optional
    username: # string - optional | default: ""
    password: # string - optional | default: ""
    insert_key: # boolean - optional | default: False

  synced_folder: # dict - optional
    enabled: # boolean - default: False
    id: # string - default: "vagrant-root"
    src: # string - default: "."
    dst: # string - default: "/vagrant"

  provider_config: # dict - optional
    random_hostname: # boolean - optional | default: False
    nic_adapter_count: # integer - optional | default: 0
    disk_bus: # string - optional | default: ""
    cpus: # integer - optional | default: 1
    memory: # integer - optional | default: 512
    huge_pages: # boolean - optional | default: False
    storage_pool: # string - optional | default: ""
    additional_storage_volumes: # list - optional
      # For each list element the following is required.
      - location: # string
        type: # string
        bus: # string
        device: # string
    nic_model_type: # string - optional | default: ""
    management_network_mac: # string - optional | default: ""

  internal_interfaces: # list - optional
    # For each list element the following is required.
    - local_port: # integer
      remote_guest: # string
      remote_port: # integer

  data_interfaces: # list - optional
    # For each list element the following is required.
    - local_port: # integer
      remote_guest: # string
      remote_port: # integer
```

#### Example Datafile
The following example datafile defines two `arista/veos` switches connected 
together on ports 1 and 2.
```yaml
sw01:
  vagrant_box:
    name: "arista/veos"
    version: "4.20.1F"
    guest_type: "tinycore"
    provider: "libvirt"
  ssh:
    insert_key: False
  synced_folder:
    enabled: False
  provider_config:
    nic_adapter_count: 2
    disk_bus: "ide"
    cpus: 2
    memory: 2048
  data_interfaces:
    - local_port: 1
      remote_guest: "sw02"
      remote_port: 1
    - local_port: 2
      remote_guest: "sw02"
      remote_port: 2

sw02:
  vagrant_box:
    name: "arista/veos"
    version: "4.20.1F"
    guest_type: "tinycore"
    provider: "libvirt"
  ssh:
    insert_key: False
  synced_folder:
    enabled: False
  provider_config:
    nic_adapter_count: 2
    disk_bus: "ide"
    cpus: 2
    memory: 2048
  data_interfaces:
    - local_port: 1
      remote_guest: "sw01"
      remote_port: 1
    - local_port: 2
      remote_guest: "sw01"
      remote_port: 2
```
#### Generated Vagrantfile
```ruby
# -*- mode: ruby -*-
# vi: set ft=ruby :

def get_mac(oui="28:b7:ad")
  "Generate a MAC address"
  nic = (1..3).map{"%0.2x"%rand(256)}.join(":")
  return "#{oui}:#{nic}"
end

cwd = Dir.pwd.split("/").last
username = ENV['USER']
domain_prefix = "#{username}_#{cwd}"
domain_uuid = "1f22b55d-2d7e-5a24-b4fa-3a8878df5cc5"

Vagrant.require_version ">= 2.1.0"
Vagrant.configure("2") do |config|

  config.vm.define "sw01" do |node|
    guest_name = "sw01"
    node.vm.box = "arista/veos"
    node.vm.box_version = "4.20.1F"
    node.vm.guest = :tinycore
    node.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true

    node.ssh.insert_key = false

    node.vm.provider :libvirt do |domain|
      domain.default_prefix = "#{domain_prefix}"
      domain.cpus = 2
      domain.memory = 2048
      domain.disk_bus = "ide"
      domain.nic_adapter_count = 2
    end

    node.vm.network :private_network,
      # sw01-eth1 <--> sw02-eth1
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.146.53.1",
      :libvirt__tunnel_local_port => 10001,
      :libvirt__tunnel_ip => "127.146.53.2",
      :libvirt__tunnel_port => 10001,
      :libvirt__iface_name => "sw01-eth1-#{domain_uuid}",
      auto_config: false

    node.vm.network :private_network,
      # sw01-eth2 <--> sw02-eth2
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.146.53.1",
      :libvirt__tunnel_local_port => 10002,
      :libvirt__tunnel_ip => "127.146.53.2",
      :libvirt__tunnel_port => 10002,
      :libvirt__iface_name => "sw01-eth2-#{domain_uuid}",
      auto_config: false

  end
  config.vm.define "sw02" do |node|
    guest_name = "sw02"
    node.vm.box = "arista/veos"
    node.vm.box_version = "4.20.1F"
    node.vm.guest = :tinycore
    node.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true

    node.ssh.insert_key = false

    node.vm.provider :libvirt do |domain|
      domain.default_prefix = "#{domain_prefix}"
      domain.cpus = 2
      domain.memory = 2048
      domain.storage_pool_name = "disk1"
      domain.disk_bus = "ide"
      domain.nic_adapter_count = 2
    end

    node.vm.network :private_network,
      # sw02-eth1 <--> sw01-eth1
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.146.53.2",
      :libvirt__tunnel_local_port => 10001,
      :libvirt__tunnel_ip => "127.146.53.1",
      :libvirt__tunnel_port => 10001,
      :libvirt__iface_name => "sw02-eth1-#{domain_uuid}",
      auto_config: false

    node.vm.network :private_network,
      # sw02-eth2 <--> sw01-eth2
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.146.53.2",
      :libvirt__tunnel_local_port => 10002,
      :libvirt__tunnel_ip => "127.146.53.1",
      :libvirt__tunnel_port => 10002,
      :libvirt__iface_name => "sw02-eth2-#{domain_uuid}",
      auto_config: false

  end

end
```

### Defaults Per-Guest Type
It is possible to define default values per guest group type. Grifter will 
look for a file named `guest-defaults.yml` in the following locations from 
the least to most preferred:

 - `/opt/grifter/`
 - `~/.grifter/`
 - `./` 

```yaml
arista/veos:
  vagrant_box:
    version: "4.20.1F"
    guest_type: "tinycore"
  ssh:
    insert_key: False
  synced_folder:
    enabled: False
  provider_config:
    nic_adapter_count: 24
    cpus: 2
    memory: 2048
    disk_bus: "ide"

juniper/vsrx-packetmode:
  vagrant_box:
    version: "18.3R1-S1.4"
    provider: "libvirt"
    guest_type: "tinycore"
  ssh:
    insert_key: False
  synced_folder:
    enabled: False
  provider_config:
    nic_adapter_count: 2
    disk_bus: "ide"
    cpus: 2
    memory: 4096
```

Group variables can be over-written by variables at the guest variable level. 
The values of the group and guest variables will be merged prior to building 
a `Vagrantfile` with the guest variables taking precedence over the group 
variables.

This means you can have a much more succinct guests file by reducing 
a lot of duplication. Here is an example of a simplified guest file. The 
values from the `arista/veos` guest type in the `guest-defaults.yml` file 
will be used to fill in the parameters for the guests.

```yaml
sw01:
  vagrant_box:
    name: "arista/veos"
  provider_config:
    nic_adapter_count: 2
  data_interfaces:
    - local_port: 1
      remote_guest: "sw02"
      remote_port: 1
    - local_port: 2
      remote_guest: "sw02"
      remote_port: 2

sw02:
  vagrant_box:
    name: "arista/veos"
  provider_config:
    nic_adapter_count: 2
  data_interfaces:
    - local_port: 1
      remote_guest: "sw01"
      remote_port: 1
    - local_port: 2
      remote_guest: "sw01"
      remote_port: 2
```

The generated `Vagrantfile` below is the same as the one above, but with a 
much cleaner guest definition file.

```ruby
# -*- mode: ruby -*-
# vi: set ft=ruby :

def get_mac(oui="28:b7:ad")
  "Generate a MAC address"
  nic = (1..3).map{"%0.2x"%rand(256)}.join(":")
  return "#{oui}:#{nic}"
end

cwd = Dir.pwd.split("/").last
username = ENV['USER']
domain_prefix = "#{username}_#{cwd}"
domain_uuid = "d35fb1b6-ecdc-5412-be22-185446af92d6"

Vagrant.require_version ">= 2.1.0"
Vagrant.configure("2") do |config|

  config.vm.define "sw01" do |node|
    guest_name = "sw01"
    node.vm.box = "arista/veos"
    node.vm.box_version = "4.20.1F"
    node.vm.guest = :tinycore
    node.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true

    node.ssh.insert_key = false

    node.vm.provider :libvirt do |domain|
      domain.default_prefix = "#{domain_prefix}"
      domain.cpus = 2
      domain.memory = 2048
      domain.disk_bus = "ide"
      domain.nic_adapter_count = 2
    end

    node.vm.network :private_network,
      # sw01-eth1 <--> sw02-eth1
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.127.145.1",
      :libvirt__tunnel_local_port => 10001,
      :libvirt__tunnel_ip => "127.127.145.2",
      :libvirt__tunnel_port => 10001,
      :libvirt__iface_name => "sw01-eth1-#{domain_uuid}",
      auto_config: false

    node.vm.network :private_network,
      # sw01-eth2 <--> sw02-eth2
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.127.145.1",
      :libvirt__tunnel_local_port => 10002,
      :libvirt__tunnel_ip => "127.127.145.2",
      :libvirt__tunnel_port => 10002,
      :libvirt__iface_name => "sw01-eth2-#{domain_uuid}",
      auto_config: false

  end
  config.vm.define "sw02" do |node|
    guest_name = "sw02"
    node.vm.box = "arista/veos"
    node.vm.box_version = "4.20.1F"
    node.vm.guest = :tinycore
    node.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true

    node.ssh.insert_key = false

    node.vm.provider :libvirt do |domain|
      domain.default_prefix = "#{domain_prefix}"
      domain.cpus = 2
      domain.memory = 2048
      domain.storage_pool_name = "disk1"
      domain.disk_bus = "ide"
      domain.nic_adapter_count = 2
    end

    node.vm.network :private_network,
      # sw02-eth1 <--> sw01-eth1
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.127.145.2",
      :libvirt__tunnel_local_port => 10001,
      :libvirt__tunnel_ip => "127.127.145.1",
      :libvirt__tunnel_port => 10001,
      :libvirt__iface_name => "sw02-eth1-#{domain_uuid}",
      auto_config: false

    node.vm.network :private_network,
      # sw02-eth2 <--> sw01-eth2
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.127.145.2",
      :libvirt__tunnel_local_port => 10002,
      :libvirt__tunnel_ip => "127.127.145.1",
      :libvirt__tunnel_port => 10002,
      :libvirt__iface_name => "sw02-eth2-#{domain_uuid}",
      auto_config: false

  end

end
```

## Example Files
Examples of the `config.yml`, `guests-defaults.yml` and `guests.yml` files 
can be found [here](grifter/examples)


## Interfaces
There are 3 types of interfaces that can be defined.

- internal_interfaces
- data_interfaces
- reserved_interfaces

#### Internal Interfaces
Config location: `guests.yml`  
Used for an inter-vm communication channel for multi-vm boxes.  
Known examples are the vMX and vQFX.

#### data_interfaces
Config location: `guests.yml`  
Revenue ports that are used to pass data traffic.

#### reserved_interfaces
Config location: `config.yml`  
Interfaces that need to be defined because 'reasons' but cannot be 
used. The only known example is the `juniper/vqfx-re`. The number of 
reserved_interfaces is defined per-box type in the `config.yml` file. 
Grifter builds out the interface definitions automatically as a 
blackhole interfaces.

#### Blackhole Interfaces
Interfaces defined in the Vagratfile relate to interfaces 
on the guest vm on a first to last basis. This can be undesirable when 
trying to accurately simulate a production environment when devices 
can have 48+ ports.  

Grifter will automatically create `blackhole interfaces` to fill out 
undefined `data_interfaces` ports up to the box types 
`max_data_interfaces` parameter in the `config.yml` file. 

#### Vagrantfile Interface Order
Interfaces are added to the Vagrantfile in the following order.
- internal_interfaces
- reserved_interfaces
- data_interfaces

Interfaces are configured using the udp tunneling type. This 
will create a 'pseudo' layer 1 connection between VM ports.

##### Example interface definition
```yaml
  data_interfaces:
    - local_port: 1
      remote_guest: "sw02"
      remote_port: 1
```
##### Rendered Vagrantfile interface
```ruby
    node.vm.network :private_network,
      # sw01-eth1 <--> sw02-eth1
      :mac => "#{get_mac()}",
      :libvirt__tunnel_type => "udp",
      :libvirt__tunnel_local_ip => "127.255.255.1",
      :libvirt__tunnel_local_port => 10001,
      :libvirt__tunnel_ip => "127.255.255.2",
      :libvirt__tunnel_port => 10001,
      :libvirt__iface_name => "sw01-eth1-#{domain_uuid}",
      auto_config: false
```

#### NIC Adapter Count
Config location: `guests.yml`  
Defines the total number of `data_interfaces` to create on the VM. 
Any undefined `data_interfaces` will be added as a blackhole interface.

The total is calculated against the sum of the `internal_interfaces`, `
reserved_interfaces` and `data_interfaces` parameters after blackhole 
interfaces have been added automatically by the template system.
