'Openstack: Message: No valid host was found. There are not enough hosts available., Code: 500"
I am installing openstack in all-in-one mode and when try to create instances the operation halts with mentioned error. My nova-compute.log file shows following exception.
2022-03-31 11:45:50.435 7 ERROR oslo_service.service libvirt.libvirtError: unable to connect to server at '192.168.211.133:16509': Connection refused
2022-03-31 11:45:50.435 7 ERROR oslo_service.service
2022-03-31 11:45:50.435 7 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2022-03-31 11:45:50.435 7 ERROR oslo_service.service
2022-03-31 11:45:50.435 7 ERROR oslo_service.service Traceback (most recent call last):
2022-03-31 11:45:50.435 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_service/service.py", line 806, in run_service
2022-03-31 11:45:50.435 7 ERROR oslo_service.service service.start()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/service.py", line 159, in start
2022-03-31 11:45:50.435 7 ERROR oslo_service.service self.manager.init_host()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 1416, in init_host
2022-03-31 11:45:50.435 7 ERROR oslo_service.service self.driver.init_host(host=self.host)
2022-03-31 11:45:50.435 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 709, in init_host
2022-03-31 11:45:50.435 7 ERROR oslo_service.service self._update_host_specific_capabilities()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 813, in _update_host_specific_capabilities
2022-03-31 11:45:50.435 7 ERROR oslo_service.service 'supports_secure_boot': self._host.supports_secure_boot,
2022-03-31 11:45:50.435 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/host.py", line 1550, in supports_secure_boot
2022-03-31 11:45:50.435 7 ERROR oslo_service.service arch = self.get_capabilities().host.cpu.arch
2022-03-31 11:45:50.435 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/host.py", line 781, in get_capabilities
2022-03-31 11:45:50.435 7 ERROR oslo_service.service xmlstr = self.get_connection().getCapabilities()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/host.py", line 596, in get_connection
2022-03-31 11:45:50.435 7 ERROR oslo_service.service raise exception.HypervisorUnavailable()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service nova.exception.HypervisorUnavailable: Connection to the hypervisor is broken on host
2022-03-31 11:45:50.435 7 ERROR oslo_service.service
2022-03-31 11:45:50.497 7 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: unable to connect to server at '192.168.211.133:16509': Connection refused'
2022-03-31 11:46:05.487 7 INFO os_vif [-] Loaded VIF plugins: linux_bridge, noop, ovs
2022-03-31 11:46:11.219 7 INFO nova.virt.driver [req-85c2604c-936a-4e8c-a168-de17800dbbbb - - - - -] Loading compute driver 'libvirt.LibvirtDriver'
I am using ubuntu 20.04 VM on vmware with 4 VCPUs and 16 GB RAM. KVM is also installed with version EMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.21)
What is this URL nova shows connection to in log file and why HypervisorUnavailable exception being raised?
Any help will be appreciated as its been days I am facing this.
Edited: Following commands may give more information:
openstack hypervisor list
+----+----------------------+-----------------+-----------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+----------------------+-----------------+-----------------+-------+
| 1 | kypo-virtual-machine | QEMU | 192.168.211.133 | down |
+----+----------------------+-----------------+-----------------+-------+
nova service-list
+--------------------------------------+----------------+--------------------- -+----------+----------+-------+----------------------------+----------------- ------------------------------------------------------------------------------ -----------------+-------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | Forced down |
+--------------------------------------+----------------+----------------------+----------+----------+-------+----------------------------+----------------------------------------------------------------------------------------------------------------+-------------+
| 5aad1db3-7d9d-4991-811a-1a1ba0621725 | nova-scheduler | kypo-virtual-machine | internal | enabled | up | 2022-04-04T03:53:34.000000 | - | False |
| 8ae129c9-c1e8-4eb8-9626-176a856d1217 | nova-conductor | kypo-virtual-machine | internal | enabled | up | 2022-04-04T03:53:36.000000 | - | False |
| e935f63b-a55b-4ac9-a9e2-38e7c431e19a | nova-compute | kypo-virtual-machine | nova | disabled | down | 2022-03-31T11:47:57.000000 | AUTO: Failed to connect to libvirt: unable to connect to server at '192.168.211.133:16509': Connection refused | False |
+--------------------------------------+----------------+----------------------+----------+----------+-------+----------------------------+----------------------------------------------------------------------------------------------------------------+-------------+
Status of libvirtd
systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-04-04 03:40:41 GMT; 17min ago
TriggeredBy: ● libvirtd-admin.socket
● libvirtd.socket
● libvirtd-ro.socket
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 995 (libvirtd)
Tasks: 19 (limit: 32768)
Memory: 31.2M
CGroup: /system.slice/libvirtd.service
├─ 995 /usr/sbin/libvirtd
├─1256 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq /default.conf --leasefile-ro >
└─1257 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro >
Apr 04 03:45:11 kypo-virtual-machine libvirtd[995]: ethtool ioctl error: No such device
Apr 04 03:45:11 kypo-virtual-machine libvirtd[995]: ethtool ioctl error: No such device
Apr 04 03:45:11 kypo-virtual-machine libvirtd[995]: ethtool ioctl error: No such device
Apr 04 03:45:11 kypo-virtual-machine libvirtd[995]: ethtool ioctl error: No such device
I am creating two instances with following commands:
openstack stack create --wait -e kypo-base-params.yml -t heat/kypo-head.yml kypo-head-stack
openstack stack create --wait -e kypo-base-params.yml -t heat/kypo-proxy-jump.yml kypo-proxy-jump-stack
cat kypo-head.yml
heat_template_version: "2018-08-31"
description: "KYPO Head server."
parameters:
kypo: {type: json}
resources:
kypo-head:
type: "OS::Nova::Server"
properties:
flavor: { get_param: [kypo, kypo_base_head_flavor] }
image: { get_param: [kypo, kypo_base_head_image] }
name: "kypo-head"
networks:
- port: { get_resource: kypo-head-port }
key_name: { get_param: [kypo, kypo_base_head_keypair_name] }
depends_on:
- kypo-head-port
kypo-head-port:
type: "OS::Neutron::Port"
properties:
admin_state_up: true
allowed_address_pairs:
- ip_address: "0.0.0.0/0"
network_id: kypo-base-net
security_groups:
- kypo-base-head-sg
kypo-head-floating-ip:
type: "OS::Neutron::FloatingIPAssociation"
properties:
floatingip_id: { get_param: [kypo, kypo_base_head_floating_ip_id] }
port_id: { get_resource: kypo-head-port }
depends_on:
- kypo-head-port
(kypo-crp-openstack-base) root@kypo-virtual-machine:~/kypo-crp-openstack-base# cat kypo-base-params.yml
parameters:
kypo:
kypo_base_network_cidr: 192.168.64.0/18
kypo_base_public_router_ip: 192.168.64.1
kypo_base_private_router_ip: 192.168.64.2
kypo_base_network_allocation_pools: [{start: 192.168.64.10, end: 192.168.127.254}]
kypo_base_external_public_network_name: public
kypo_base_dns_nameservers: [1.1.1.1,1.0.0.1]
kypo_base_head_keypair_name: admin_kypo-base-key
kypo_base_head_flavor: standard.small
kypo_base_head_image: ubuntu-focal-x86_64
kypo_base_head_floating_ip_id: 65569d91-5d76-4b72-95cf-eaaeaab65740
kypo_base_proxy_keypair_name: admin_kypo-base-key
kypo_base_proxy_flavor: standard.small
kypo_base_proxy_image: ubuntu-focal-x86_64
kypo_base_proxy_floating_ip_id: 25601de5-6c84-4ec4-9821-8c02ba5548ed
Following are images and flavors I have for those instances:
(kypo-crp-openstack-base) root@kypo-virtual-machine:~/kypo-crp-openstack-base/heat# openstack image list
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 6e7467cf-44e6-4fed-9846-3cd4b59bd7c7 | debian-9-x86_64 | active |
| 0e4c0582-9d46-4e60-995b-323884976578 | ubuntu-focal-x86_64 | active |
+--------------------------------------+---------------------+--------+
(kypo-crp-openstack-base) root@kypo-virtual-machine:~/kypo-crp-openstack-base/heat# openstack flavor list
+----+-----------------+------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------------+------+------+-----------+-------+-----------+
| 1 | csirtmu.tiny1x2 | 2048 | 10 | 0 | 1 | True |
| 2 | standard.small | 2048 | 10 | 0 | 1 | True |
| 3 | standard.medium | 2048 | 10 | 0 | 1 | True |
| 4 | standard.large | 2048 | 10 | 0 | 1 | True |
+----+-----------------+------+------+-----------+-------+-----------+
Solution 1:[1]
The root cause was mentioned in libvirtd.log file which said: Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable
I found a libvirtd.pid file already existed in /var/run folder. I renamed it with command mv libvird.pid libvird.pid.old and created instances again. Every thing went fine.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | aneela |