Bare metal deployment and introspection in RDO manager

 

Dmitry Tantsur

Imre Farkas

Agenda



  1. Bare metal services Overview
    1. Ironic
    2. Ironic-inspector
  2. Introspection and deployment workflow in RDO-Manager
  3. Debugging bare metal

Ironic

OpenStack Bare metal service

 

Rationale

  • deploying a cloud infrastructure
  • HPC
  • Hadoop cluster
  • Database server
  • multi-tenancy

 

Ironic

OpenStack Bare metal service

 

Architecture

  • RESTful API service
  • Conductor service
    • pluggable drivers
  • Database
  • Deployment ramdisk/agent

 

Ironic

OpenStack Bare metal service

 

Driver interface

  1. Core interfaces
    1. power
    2. deploy
    3. management
  2. Standard interfaces
    1. console
    2. inspect
    3. raid
  3. Vendor interface

 

Ironic

OpenStack Bare metal service

Interface example


class PXEAndIPMIToolDriver(base.BaseDriver):

    def __init__(self):
        self.power = ipmitool.IPMIPower()
        self.deploy = pxe.PXEDeploy()

        pxe_vendor = pxe.VendorPassthru()
        ipmi_vendor = ipmitool.VendorPassthru()
        mapping = {'pass_deploy_info': pxe_vendor,
                   'set_boot_device': ipmi_vendor}
        self.vendor = utils.MixinVendorInterface(mapping)

class IPMIPower(base.PowerInterface):

    def get_power_state(self, task, node):
        ...

    def set_power_state(self, task, node, pstate):
        ...

    def reboot(self, task, node):
        ...

Ironic

Deployment workflow

 

  1. Ironic
    1. fetches deploy ramdisk and overcloud image
    2. prepares PXE environment, updates Neutron
    3. set boot device and reboots the node
  2. Node
    1. Loads deploy ramdisk + kernel
    2. Expose the local disk to ironic via iSCSI
  3. Ironic
    1. Writes the overcloud image to disk and changes PXE environment
    2. reboots the node

ironic-python-agent

Ramdisk on steriods

 

  1. In-band vs out-of-band
  2. Features
    • Erase disks
    • Partition disks
    • Install bootloaders
    • Install an OS image
    • Update firmware
    • Configure RAID
    • and more...

Ironic-inspector

Hardware Introspection for Ironic

 

Rationale

  • hardware properties for scheduling

 

Features

  • Setting new IPMI credentials
  • Plugins
    • ramdisk_error
    • scheduler
    • validate_interfaces
    • raid_device (optional)
    • extra_hardware (optional)
  • Introspection rules

Introspection rules

 

Profiles

  • Compute
  • Controller
  • Storage

Ironic-inspector

Hardware Introspection for Ironic

 

Architecture

  • ironic-inspector service
    • API for administrator
    • callback for ramdisk
  • Dnsmasq service
  • Inspector ramdisk/agent

Ironic-Inspector

Node Introspection

 

  1. ironic-inspector receives introspection request
    1. validates node
    2. prepares PXE environment
    3. reboots the node
  2. ramdisk collects hardware properties and posts back to ironic-inspector, which

    1. validates received data
    2. compares received data to ironic
    3. updates ports
    4. updates state ready for deployment

Deployment workflow

 

  1. configuration files
  2. enroll nodes
  3. introspect nodes
  4. deploy the cloud

Deployment workflow

 

  1. configuration files
  2. enroll nodes
  3. introspect nodes
    1. automated profile tagging
  4. deploy the cloud

Deployment workflow

Configuration files

instackenv.json

 

{
    "nodes": [
        {
            "pm_type":"pxe_ipmitool",
            "mac":[
                "fa:16:3e:2a:0e:36"
            ],
            "pm_user":"admin",
            "pm_password":"password",
            "pm_addr":"10.0.0.8"
        },
        {
            "pm_type":"pxe_ipmitool",
            "mac":[
                "fa:16:3e:da:39:c9"
            ],
            "pm_user":"admin",
            "pm_password":"password",
            "pm_addr":"10.0.0.15"
        },
        ...
    ]
}

Deployment workflow

Enroll Nodes

 

 

  1. nodes get created in ironic
  2. ironic checks the power state
$ openstack baremetal import --json instackenv.json

Deployment workflow

automated profile tagging (1)

rules.json

 

 

[
    {
        "description": "Fail introspection for unexpected nodes",
        "conditions": [
            {"op": "lt", "field": "memory_mb", "value": 4096}
        ],
        "actions": [
            {"action": "fail", "message": "Memory too low, expected at least 4 GiB"}
        ]
    },
    {
        "description": "Assign possible profiles for compute and controller",
        "conditions": [
            {"op": "lt", "field": "local_gb", "value": 1024},
            {"op": "ge", "field": "local_gb", "value": 40}
        ],
        "actions": [
            {"action": "set-capability", "name": "compute_profile", "value": "1"},
            {"action": "set-capability", "name": "control_profile", "value": "1"}
        ]
    }
]

Deployment workflow

automated profile tagging (2)

 

 

$ openstack baremetal introspection rule import /path/to/rules.json

Deployment workflow

Introspect nodes (Inspector)

 

 

$ openstack baremetal introspection bulk start

Deployment workflow

automated profile tagging (3)

 

 

$ openstack overcloud profiles list

Deployment workflow

Deploy the cloud (Ironic)

 

 

$ openstack overcloud deploy --templates