Shared LVM in PVM
On a PVM cluster, several hosts read and write the same storage at the same time. A normal filesystem cannot survive that, because two hosts would allocate the same blocks. Shared LVM solves this by putting a distributed lock manager between LVM and the storage, so every metadata change is serialized across the cluster.
This document explains the LVM architecture behind PVM storage, then walks through a complete Shared LVM setup: Multipath, host identifiers, pvcreate, vgcreate --share, vgchange --lock-start, and finally registering the storage in the PVM Storage Manager.
If you are new to LVM itself, start with What is LVM?. If you want the full provisioning chain from RAID down to a PVM storage entry, read Storage Architecture and Configuration.
What LVM Is and Why We Need It
Imagine three physical disks with capacities of 500 GB, 1 TB, and 2 TB. The operating system sees each of them as a separate space, so storing a single 3 TB file is impossible. LVM (Logical Volume Manager) is a layer that sits between the physical disks and the operating system. It lets you merge all of them into one storage pool, then take as much space from that pool as you need.
In PVM, every virtual machine you create gets its disk stored as a Logical Volume inside this pool. Understanding the LVM structure is therefore a prerequisite for working with PVM.
The Three Layers of LVM: From Physical Disk to Virtual Disk
LVM is built from three nested layers. Follow a concrete scenario: a server with two disks, /dev/sda and /dev/sdb, and you want to use both for virtual machines.
Physical Volume (PV) is the bottom layer. When you have a disk such as /dev/sda and want to bring it into LVM, you must first convert it into a PV. In this process LVM writes its own metadata at the start of the disk or partition, and from that moment on it can identify and manage the disk. In our scenario, both /dev/sda and /dev/sdb become PVs.
Volume Group (VG) is the middle layer, and it is the storage pool itself. A VG is a set of one or more PVs that LVM presents as a single space. If you put both PVs into a VG named MyPool, you now have one pool with the combined capacity of both disks. Which data sits on which physical disk stops mattering, because the VG presents everything as one unit.
Logical Volume (LV) is the top layer, and it is what the operating system sees as an ordinary disk or partition. From the space in the VG you take a 100 GB LV for the first virtual machine and a 200 GB LV for the second. The operating system treats each LV exactly like a regular disk and builds a filesystem on it.
One detail matters in a PVM environment: Qemu creates a separate LV for every disk or ISO file it builds for a virtual machine.
Why a Regular Filesystem Does Not Work Here
To see the problem, start with what happens when a file is written. Suppose you save a file named report.txt. The filesystem first creates an inode for that file in its own metadata. An inode holds file information such as permissions, creation time, and the list of disk blocks assigned to the file. The filesystem then writes the actual file data into free blocks on the disk.
Now imagine that the same disk is connected to two servers at the same time and both want to write. Server one wants to write report.txt and server two wants to write backup.tar. Both look at the table of free blocks, and both see that block number 5000 is free. Server one assigns block 5000 to report.txt, and server two, with no idea what server one just decided, assigns the same block 5000 to backup.tar. The data of both files ends up written on top of each other, and both files are destroyed.
The situation gets worse when two servers assign the same inode to two different files. The inode is the backbone of filesystem metadata, so this collision makes the entire filesystem structure inconsistent and possibly unusable.
The Cluster Filesystem and Its Limits
The classic answer to the problem above is a cluster filesystem such as GFS (Global File System). A cluster filesystem is designed so that several servers can access one shared storage space simultaneously and safely.
The mechanism is simple: when the first server wants to write data, GFS takes a lock on the relevant metadata. While that lock is held, the other servers cannot modify the same metadata or allocate the same resources, so they have to wait. Once the write and metadata update finish, the lock is released and the next server can proceed.
Cluster filesystems solved concurrent access, but they brought their own operational cost: network stability, quorum maintenance, and a whole second storage model to learn and troubleshoot.
The GFS-based approach carried problems of its own: dependence on a stable network, the need to keep at least N/2 + 1 nodes active, dependence on iLO, and a high degree of implementation and maintenance complexity.
Shared LVM: PVM's Answer
To get past the GFS limits, PVM uses Shared LVM. Shared LVM extends the classic LVM architecture with extra services, locking mechanisms, and specific configuration, so that several hosts can safely reach one shared Volume Group without a cluster filesystem. The rest of this document walks through that architecture from the bottom up.
Multipath: Detecting the Shared Storage
When shared storage such as a SAN LUN is handed to a cluster of servers, the first step is configuring Multipath on every server. Multipath guarantees that if one of the connection paths to the storage drops, the alternative path takes over automatically and data access is never interrupted.
After Multipath is configured, you can confirm the shared storage was detected correctly with fdisk -l or lsblk.
fdisk -l
lsblk
The Lock Management Services in Shared LVM
When a Volume Group is used by a single server only, LVM uses its own internal locking and no extra service is needed. In a shared environment where several servers are attached to one VG, the picture changes. If two servers run lvcreate and lvremove on the same VG at the same time, both try to change the same metadata, and without coordination this ends in metadata corruption and loss of access to the Logical Volumes.
Shared LVM prevents this with Distributed Locking, managed by several main services: lvmlockd, sanlock, wdmd, plvreaper.timer, and pvghamgr.timer.
To use this capability you must run el9 with version 7.0 or newer, so that all of these services are available.
The Locking Architecture: How lvmlockd and sanlock Work Together
Every time an LVM management command runs, the request goes first to lvmlockd. lvmlockd, short for LVM Lock Daemon, is the service that manages lock requests in Shared LVM. Before any operation that changes metadata runs, it checks whether the required lock is held by another host. If the lock is free, execution is allowed. If another host holds it, the operation waits or stops with an error.
The key point is that lvmlockd does not store or maintain the locks itself. It only manages the requests. Creating and maintaining the actual locks is the job of the Lock Manager, which is sanlock or DLM.
In this chain, LVM performs the management operation, lvmlockd decides which type of lock is needed, and sanlock is responsible for creating, holding, and releasing the lock.
Every command that changes LVM metadata has to take the right lock first: pvcreate, pvremove, vgcreate, vgextend, vgreduce, vgremove, lvcreate, lvremove, lvextend, lvreduce, lvrename, and lvconvert.
Lock Types in lvmlockd
| Lock Type | Scope | Typical operations |
|---|---|---|
| Global Lock | The largest lock in LVM, guarding global operations. Only one host can hold it at a time. | Creating or removing a Volume Group, changing the overall metadata structure, PV-related operations |
| VG Lock | The metadata of one specific Volume Group | lvcreate, lvremove, vgextend, vgreduce on that VG |
| LV Lock | One specific Logical Volume, controlling access to it | Restricting activation to a single host, for example for a database LV |
Whenever an operation such as lvcreate, lvremove, vgextend, or vgreduce runs on a VG, the lock for that VG is taken first, so only one host at a time can change its metadata.
sanlock: Storage-Based Locking
sanlock is a Distributed Lock Manager based on storage. Unlike many Lock Managers that exchange lock information between servers over the network, sanlock writes the lock information directly onto the shared storage. The shared storage acts as the authoritative record of lock state, and every host reads the state from that same place.
Consider three hosts attached to one SAN. When an LVM command is issued, LVM sends the lock request to lvmlockd. lvmlockd asks sanlock for the lock. sanlock records the lock state in the Lock Area on the storage. If the lock is free, execution is allowed, and once the operation finishes, the lock is released.
The Lease Concept in sanlock
The core concept of sanlock rests on the lease, a temporary ownership of a resource. While a host holds the lease, no other host is allowed to change the metadata. The host that holds the lock renews its lease periodically to show that it is still alive.
Now consider a host failure. Assume Host1 holds the VG lock and suddenly goes down. Its lease is no longer renewed. Once the configured Lease Timeout expires, sanlock notices that Host1 is no longer active, releases the lock, and another host can take it over. This mechanism prevents locks from being left in limbo.
wdmd: Guardian of System Health
The wdmd service manages the hardware or software watchdog of the system, and it counts as one of sanlock's protection mechanisms.
Consider this scenario: two hosts, where Host1 holds the VG lock and suddenly suffers a kernel hang. The lock still belongs to Host1 and sanlock can no longer renew the lease. Host2 waits for the lock to be released, but if Host1 never powers off, the lock may stay in limbo forever.
This is where wdmd steps in. It regularly checks whether sanlock is healthy. If sanlock cannot renew the lease, wdmd concludes that the system state is dangerous and tells the watchdog to reset the machine. After Host1 reboots, the second host can take the lock, and metadata corruption is avoided.
plvreaper.timer: Cleaning Up Inactive Disks
This service was written by the PVM development team. Its job is to deactivate LVs that are still active on a node but have had no input or output activity for more than one minute. This situation normally occurs when a virtual machine has been shut down but its disk remains active, so no other virtual machine can use it. plvreaper.timer identifies and deactivates these abandoned disks so the resources are freed.
Implementing Shared LVM
The steps below are ordered. Note which node each command runs on, because running a cluster-wide command on every node, or a one-time command on all of them, breaks the setup.
| # | Step | Runs on |
|---|---|---|
| 1 | Set the host identifier | All nodes |
| 2 | Create the Physical Volume | One node only |
| 3 | Enable lvmlockd | All nodes |
| 4 | Create the shared Volume Group | One node only |
| 5 | Teach the other nodes about the VG | Remaining nodes |
| 6 | Start the lock on the storage | All nodes |
| 7 | View the storage | One node only |
| 8 | Add the storage to PVM | Storage Manager |
1. Set the host identifier
Once every server has detected the shared storage, the first configuration step on each node is setting its host identifier. Open /etc/lvm/lvmlocal.conf on each node and, in the host_id section, enter the node number you defined in sbone. This has to be done on all nodes, and every node needs its own unique identifier.
# /etc/lvm/lvmlocal.conf
host_id = 1
2. Create the Physical Volume
Next, create a PV on each LUN added to the cluster. Keep in mind that every LV (disk) or VG created consumes roughly 20 KB of metadata space. For that reason we set the PV metadata area to 1 GB by default, which leaves room for about 52,428 LVs. This command must be run on only one of the cluster servers:
pvcreate --metadatasize 1G /dev/mapper/test
Here, /dev/mapper/test is the name of your storage.
3. Enable lvmlockd
Open /etc/lvm/lvm.conf, uncomment the use_lvmlockd option, and set its value to 1. Then restart the service:
vi /etc/lvm/lvm.conf
systemctl restart lvmlockd.service
This step must be done on all cluster nodes.
4. Create the shared Volume Group
On each PV you created for a storage LUN, build a VG:
vgcreate --share --locktype sanlock S1 /dev/mapper/test
The --share flag marks the VG as shared. --locktype sanlock sets sanlock as the lock type. S1 is the Volume Group name, and /dev/mapper/test is the storage name. This command must also be run on only one of the cluster servers.
5. Teach the other nodes about the Volume Group
After the VG is created on one node, the other cluster nodes may not see it. To fix that, run the following command on the other cluster servers so their LVM detects the storage:
lvmdevices --adddev /dev/mapper/test
6. Start the lock on the storage
To activate the lock on the storage you created, run:
vgchange --lock-start S1
This command must be run on all cluster nodes.
Never run vgchange --lock-start on storage that is already configured and under load, because it can disrupt service.
7. View the storage
At this point the LVM configuration is complete. To view the storage spaces you created, run:
pvgs
This command shows the created storage name, total and free capacity, the hosts (nodes) that use it, and the state of each node.
8. Add the storage to PVM
In Storage Manager, add a new storage and set its type to volume group. Then enter the name of the LVM storage you created in the previous steps.
On each LVM storage you can only create one data area and one ISO area in the manager. If you have several storages, create the ISO area next to the data area on just one of them, and avoid creating multiple separate ISO areas.
Summary
| Concern | Answer |
|---|---|
| Why not a normal filesystem | Two hosts would allocate the same free blocks and corrupt both files |
| Why not GFS | Network and quorum dependency, iLO dependency, and high operational complexity |
| What Shared LVM adds | lvmlockd for request management, sanlock for distributed locking on the storage, wdmd for watchdog protection |
| What holds the lock state | The shared storage itself, through the sanlock Lock Area |
| What releases a dead host's lock | The sanlock lease, once the Lease Timeout expires |
| What one-time commands run on | One cluster server only (pvcreate, vgcreate) |
| What cluster-wide commands run on | All nodes (host_id, lvmlockd, vgchange --lock-start) |