OpenStack, Ceph RBD and QoS
The Havana cycle introduced a QoS feature on both Cinder and Nova. Quick tour of this excellent implementation.
Originally both QEMU and KVM support rate limitation.
This is obviously implemented through libvirt and available as an extra xml flag within the
<disk> section called
QoS options are:
total_bytes_sec: the total allowed bandwidth for the guest per second
read_bytes_sec: sequential read limitation
write_bytes_sec: sequential write limitation
total_iops_sec: the total allowed IOPS for the guest per second
read_iops_sec: random read limitation
write_iops_sec: random write limitation
This is wonderful that OpenStack implemented such (easy?) feature in both Nova and Cinder. It is also a sign that OpenStack is getting more featured and complete in the existing core projects. Having such facility is extremely useful for several reasons. First of all, not all the storage backends support QoS. For instance, Ceph doesn’t have any built-in QoS feature whatsoever. Moreover, the limitation is directly at the hypervisor layer and your storage solution doesn’t even need to have such feature. Another good point is that from an operator side it is quite nice to be able to offer different levels of service. Operators can now offer different types of volumes based on a certain QoS, customers then, will be charged accordingly.
First create the QoS in Cinder:
$ cinder qos-create high-iops consumer="front-end" read_iops_sec=2000 write_iops_sec=1000
Create a new volume type:
$ cinder type-create high-iops
Then associate the volume type with the QoS:
$ cinder qos-associate c38d72f8-f4a4-4999-8acd-a17f34b040cb 9c746ca5-eff8-40fe-9a96-1cdef7173bd0
Eventually attach the volume to an instance:
$ nova volume-attach cirrOS 743549c1-c7a3-4e86-8e99-b51df4cf7cdc /dev/vdc
While attaching the device you should see the following xml creation from the nova-volume debug log. Dumping the virsh xml works as well.
2013-12-11 14:12:05.874 DEBUG nova.virt.libvirt.config [req-232cf5eb-a79b-42d5-a183-2f4758e8d8eb admin admin] Generated XML <disk type="network" device="disk"> <driver name="qemu" type="raw" cache="none"/> <source protocol="rbd" name="volumes/volume-743549c1-c7a3-4e86-8e99-b51df4cf7cdc"> <host name="192.168.251.100" port="6790"/> </source> <auth username="volumes"> <secret type="ceph" uuid="95c98032-ad65-5db8-f5d3-5bd09cd563ef"/> </auth> <target bus="virtio" dev="vdc"/> <serial>2e589abc-a008-4433-89ae-1bb142b139e3</serial> <iotune> <read_iops_sec>2000</read_iops_sec> <write_iops_sec>1000</write_iops_sec> </iotune> </disk>
W Important note: rate-limiting is currently broken in Havana, however the bug has already been reported and a fix submitted/accepted. This same patch has also already been proposed as a potential backport for Havana.