Home › Blogs › dale's blog

Recursive Logical Volume Management for Xen dom0

dale's picture
Tue, 07/29/2008 - 04:34 — dale

I had some trouble removing an dom0 LV used as a domU VG after mounting it to look inside that VG (for background on this setup, see this post). I wasn't sure what steps I had taken, so I decided to reboot the dom0 to clear the decks and did the following experiment.

The goal was to create an LV and stack upon that a VG containing an LV and then remove these devices to get back where we started. And maybe to learn something along the way.

I'll create a logical volume LVOuter on VG VGRAID, add a partition to it, create a physical volume from this partition, create a VG VGInner on this PV, then a logical volume LVInner on this VG.

The Environment

The following was performed on a dom0 with a internal RAID5 array hosting a volume group named VGRAID. There is also a VG VGSanLun10 hanging off an fiberchannel HBA which can be ignored for the purposes at hand.

[root@dom0 ~]# vgs
  VG         #PV #LV #SN Attr   VSize   VFree
  VGRAID       1   8   0 wz--n- 200.59G 104.59G
  VGSanLun10   1   3   0 wz--n-  74.99G   4.99G
[root@dom0 ~]# lvs
  LV                 VG         Attr   LSize  Origin Snap%  Move Log Copy%
  LVHome             VGRAID     -wi-ao  1.00G
  LVOpt              VGRAID     -wi-ao  1.00G
  LVRoot             VGRAID     -wi-ao  2.00G
  LVUsr              VGRAID     -wi-ao  4.00G
  LVVar              VGRAID     -wi-ao  6.00G
  LVXen              VGRAID     -wi-ao 20.00G
  LVXenImgarc        VGRAID     -wi-a- 12.00G
  LVDomUAA00485AE58A VGSanLun10 -wi-a- 10.00G
  LVDomUAA0048640833 VGSanLun10 -wi-a- 10.00G
  LVDomUTest         VGSanLun10 -wi-a- 50.00G

Device Creation

Create a normal LV inside the LVM of our physical dom0.

[root@dom0 ~]# lvcreate --size 1G --name LVOuter VGRAID
  Logical volume "LVOuter" created

Create a single partition on this LV filling all the space.

[root@dom0 ~]# echo "0 + L" | sfdisk  /dev/VGRAID/LVOuter

Check the partition type and see it is type "83 Linux", and change it to type "8e Linux LVM".

[root@dom0 ~]#  sfdisk --print-id /dev/VGRAID/LVOuter 1
83
[root@dom0 ~]#  sfdisk --change-id /dev/VGRAID/LVOuter 1 8e
Done

Take a look at the partition table just to be sure.

[root@dom0 ~]# sfdisk -d  /dev/VGRAID/LVOuter
# partition table of /dev/VGRAID/LVOuter
unit: sectors

/dev/VGRAID/LVOuter1 : start=        1, size=  2088449, Id=8e
/dev/VGRAID/LVOuter2 : start=        0, size=        0, Id= 0
/dev/VGRAID/LVOuter3 : start=        0, size=        0, Id= 0
/dev/VGRAID/LVOuter4 : start=        0, size=        0, Id= 0

Now create a device for this partition so we can manipulate it. This will create /dev/mapper/LVOuter1.

[root@dom0 ~]# kpartx -av /dev/VGRAID/LVOuter
add map LVOuter1 : 0 2088449 linear /dev/VGRAID/LVOuter 1

Create a physical volume to hold our upcoming volume group.

[root@dom0 ~]# pvcreate /dev/mapper/LVOuter1
  Physical volume "/dev/mapper/LVOuter1" successfully created

Create a VG on this PV to hold our new LVM microcosm.

[root@dom0 ~]# vgcreate VGInner /dev/mapper/LVOuter1
  Volume group "VGInner" successfully created

Now let's stop and do another a status check to see what we've accomplished.

[root@dom0 ~]# vgs
  VG         #PV #LV #SN Attr   VSize    VFree
  VGInner      1   0   0 wz--n- 1016.00M 1016.00M
  VGRAID       1   8   0 wz--n-  200.59G  153.59G
  VGSanLun10   1   3   0 wz--n-   74.99G    4.99G
[root@dom0 ~]# lvs
  LV                 VG         Attr   LSize  Origin Snap%  Move Log Copy%
  LVHome             VGRAID     -wi-ao  1.00G
  LVOpt              VGRAID     -wi-ao  1.00G
  LVOuter            VGRAID     -wi-ao  1.00G
  LVRoot             VGRAID     -wi-ao  2.00G
  LVUsr              VGRAID     -wi-ao  4.00G
  LVVar              VGRAID     -wi-ao  6.00G
  LVXen              VGRAID     -wi-ao 20.00G
  LVXenImgarc        VGRAID     -wi-ao 12.00G
  LVDomUAA00485AE58A VGSanLun10 -wi-a- 10.00G
  LVDomUAA0048640833 VGSanLun10 -wi-a- 10.00G
  LVDomUTest         VGSanLun10 -wi-a- 50.00G
[root@dom0 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdl
  VG Name               VGSanLun10
  PV Size               75.00 GB / not usable 4.00 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              19198
  Free PE               1278
  Allocated PE          17920
  PV UUID               FQyIBW-k4a0-5Eps-uZqp-9RDi-VIC8-97Ii7i

  --- Physical volume ---
  PV Name               /dev/dm-11
  VG Name               VGInner
  PV Size               1019.75 MB / not usable 3.75 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              254
  Free PE               254
  Allocated PE          0
  PV UUID               xtBxwx-RTUf-Utw0-6WBN-hiOz-0BOV-NhzbLn

  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               VGRAID
  PV Size               200.60 GB / not usable 8.75 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              6419
  Free PE               4915
  Allocated PE          1504
  PV UUID               J5hJcm-BmN2-Cjnl-l4Du-ipun-jSvV-ckFz6J

Now let's create a 500M LV inside this 1G VG.

[root@dom0 ~]# lvcreate --size 500M --name LVInner VGInner
  Logical volume "LVInner" created
[root@dom0 ~]# lvs
  LV                 VG         Attr   LSize   Origin Snap%  Move Log Copy%
  LVInner            VGInner    -wi-a- 500.00M
...

Let's make sure we can put a filesystem on it and mount it.

[root@dom0 ~]# mke2fs -j /dev/VGInner/LVInner
[root@dom0 ~]# mkdir /tmp/LVInner
[root@dom0 ~]# mount /dev/VGInner/LVInner /tmp/LVInner
[root@dom0 ~]# df -h /tmp/LVInner
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VGInner-LVInner
                      485M   11M  449M   3% /tmp/LVInner

Device Removal

Now that we have a filesystem on a LV (LVInner) inside a VG (VGInner) inside a LV (LVOuter) inside a VG (VGRAID), let's see if we can delete the outermost LV (LVOuter).

Unmount the filesystem, remove the inner LV, remove the inner VG, remove the outer LV, and admire our proof of concept exercise.

[root@dom0 ~]# umount /tmp/LVInner
[root@dom0 ~]# lvremove /dev/VGInner/LVInner
Do you really want to remove active logical volume "LVInner"? [y/n]: y
  Logical volume "LVInner" successfully removed
[root@dom0 ~]# vgremove VGInner
  Volume group "VGInner" successfully removed
[root@dom0 ~]# vgs
  VG         #PV #LV #SN Attr   VSize   VFree
  VGRAID       1   8   0 wz--n- 200.59G 153.59G
  VGSanLun10   1   3   0 wz--n-  74.99G   4.99G
[root@dom0 ~]# kpartx -dv /dev/VGRAID/LVOuter
del devmap : LVOuter1
[root@dom0 ~]# lvchange -a n /dev/VGRAID/LVOuter
[root@dom0 ~]# lvs
  LV                 VG         Attr   LSize  Origin Snap%  Move Log Copy%
  LVHome             VGRAID     -wi-ao  1.00G
  LVOpt              VGRAID     -wi-ao  1.00G
  LVOuter            VGRAID     -wi---  1.00G
...
[root@dom0 ~]# lvremove /dev/VGRAID/LVOuter
  Logical volume "LVOuter" successfully removed

See Also