EugenyHo: HA Storage cluster setup process

Here is step-by-step instruction to do HA storage cluster showed New Big Picture.

Setup Ubuntu - both nodes :)
In my case it was Ubuntu 11.10 x86-64 at software raid-1 (using /dev/sda and /dev/sdb)
Lets first node will nas-u1110node01 and second - nas-u1110node02. Dont forget correct /etc/hosts file, and check it by ping nodes each other by name.
Setup proxy config, interfaces, and dont forget append jumbo frames (mtu 9000) at heartbeat interface at both nodes. Proxy settings depends on your network infrastructure, but /etc/network/interfaces first node look like this:

  
# INTERNAL heartbeat

auto eth0
  iface eth0 inet static
  address 172.16.0.1 (172.16.0.2 at second node)
  netmask 255.255.255.0
  mtu 9000  

# EXTERNAL iSCSI

auto eth1

 iface eth1 inet static

 address 172.16.70.205 (172.16.70.206 at second node)

 netmask 255.255.255.0

 network 172.16.70.0

 broadcast 172.16.70.255

 gateway 172.16.70.112

Do apt-get Update,upgrade at both nodes
Synchonize clocks!
Install: (mc,) lvm2, drbd8-utils, iscsitarget, heartbeat, (snmpd, sysstate) at both nodes:
```
apt-get install mc,lvm2,drbd8-utils,iscsitarget,heartbeat,snmpd,sysstate 
```
Packages mc, snmpd and sysstate is not strongly necessary, but mc is useful file manager for me and the others will need for administrative purpouse.
Set permissions for DRBD:

chgrp haclient /sbin/drbdsetup
chmod o-x /sbin/drbdsetup
chmod u+s /sbin/drbdsetup
chgrp haclient /sbin/drbdmeta
chmod o-x /sbin/drbdmeta
chmod u+s /sbin/drbdmeta

create PV - both nodes
create VG - both nodes
create LV: iscsi-config, and others for LUNs - both nodes
configure lvm.conf (filter and option) - both nodes
... filter = ["a|sd.*|","r|.*|"] ... cache_file_prefix = "" ....
and delete file .cache at /etc/lvm/cache if exists.
Make directory /srv/iscsi-config (for future mount drbd0)

Configure drdb-global.conf and drdb.conf - at first node and copy to another
Basic config will store drdb-global.conf:


common {
        protocol C;

        handlers {
        pri-on-incon-degr "echo 'DRBD: primary requested but inconsistent!' | wall; /etc/init.d/heartbeat stop";
        pri-lost-after-sb "echo 'DRBD: primary requested but lost!' | wall; /etc/init.d/heartbeat stop";
        local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
        outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
        }

        startup {
        degr-wfc-timeout 120;
        }

        disk {
        on-io-error call-local-io-error;
        }

        net {
        cram-hmac-alg sha1;
        shared-secret "password";

        after-sb-0pri discard-younger-primary;
        after-sb-1pri call-pri-lost-after-sb;
        after-sb-2pri call-pri-lost-after-sb;
        rr-conflict call-pri-lost;
        }

        syncer {
        rate 110M;
        verify-alg sha1;
        al-extents 257;
        }
}

And the DRBD devices at drdb.conf:

include "drbd.d/global_common.conf";
include "drbd.d/*.res";

resource iscsi.config {
        on nas-u1110node01 {
        device  /dev/drbd0;
        disk    /dev/PVELUNSVG0/iscsi-metadata;
        address 172.16.0.1:7788; # Use DRBD dedicated network
        flexible-meta-disk internal;
        }
        on nas-u1110node02 {
        device  /dev/drbd0;
        disk    /dev/PVELUNSVG0/iscsi-metadata;
        address 172.16.0.2:7788; # Use DRBD dedicated network
        flexible-meta-disk internal;
        }
}

resource iscsi.lv.lun0 {
        on nas-u1110node01 {
        device  /dev/drbd1;
        disk    /dev/PVELUNSVG0/lv.lun0;
        address 172.16.0.1:7789; # Use private inter-node address
        flexible-meta-disk internal;
        }
        on nas-u1110node02 {
        device  /dev/drbd1;
        disk    /dev/PVELUNSVG0/lv.lun0;
        address 172.16.0.2:7789; # Use private inter-node address
        flexible-meta-disk internal;
        }

#The rest resources must be configured at the same maner
#resource iscsi.lv.lun(x) {
#        on nas-u1110node01 {
#        device  /dev/drbd(x);
#        disk    /dev/PVELUNSVG0/lv.lun(x);
#        address 172.16.0.1:7789; # Use private inter-node address
#        flexible-meta-disk internal;
#        }
#        on nas-u1110node02 {
#        device  /dev/drbd(x);
#        disk    /dev/PVELUNSVG0/lv.lun(x);
#        address 172.16.0.2:7789; # Use private inter-node address
#        flexible-meta-disk internal;
#        }

}

Init all drbd metadata (drbdadm create-md all) - both nodes
Service drbd start - both nodes

DRBD state at both nodes:


drbd driver loaded OK; device status:
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
m:res            cs         ro                   ds                 p  mounted  fstype
0:iscsi_config   Connected  Secondary/Secondary  Inconsistent/Inconsistent  C
1:iscsi_lv_lun0  Connected  Secondary/Secondary  Inconsistent/Inconsistent  C

If all ok, set primary status for DRBD blocks:


drbdsetup /dev/drbd0 primary -o
drbdsetup /dev/drbd1 primary -o

Wait for sync - both nodes


drbd driver loaded OK; device status:
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
m:res            cs         ro                   ds                 p  mounted  fstype
0:iscsi_config   Connected  Primary/Secondary  UpToDate/UpToDate  C
1:iscsi_lv_lun0  Connected  Primary/Secondary  UpToDate/UpToDate  C

Make ext4 at /dev/drbd0 (will use as common iscsi target config storage) - at primary node
Mount /dev/drbd0 /srv/iscsi-config
Make links.
At primary:
mv /etc/iet/ietd.conf /srv/iscsi-config rm /etc/iet/ietd.conf ln -s /srv/iscsi-config/ietd.conf /etc/iet/ietd.conf mv /etc/iet/initiators.allow /srv/iscsi-config rm /etc/iet/initiators.allow ln -s /srv/iscsi-config/initiators.allow /etc/iet/initiators.allow mv /etc/iet/targets.allow /srv/iscsi-config rm /etc/iet/targets.allow ln -s /srv/iscsi-config/targets.allow /etc/iet/targets.allow
At secondary:
rm /etc/iet/ietd.conf ln -s /srv/iscsi-config/ietd.conf /etc/iet/ietd.conf rm /etc/iet/initiators.allow ln -s /srv/iscsi-config/initiators.allow /etc/iet/initiators.allow rm /etc/iet/targets.allow ln -s /srv/iscsi-config/targets.allow /etc/iet/targets.allow

At primary node edit /srv/iscsi-config/ietd.conf


Target iqn.2012-03.nas01:iscsi.PVELUNS
       Lun 0 Path=/dev/drbd1,Type=blockio,ScsiSN=NAS01PVE-001

Configure heartbeat: ha.cf, authkeys and haresource at primary node then copy it at secondary node

/etc/ha.d/ha.cf:

logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
bcast eth0
bcast eth1
node nas-u1110node01
node nas-u1110node02
auto_failback no

/etc/ha.d/authkeys:

auth 2
2 crc

/etc/ha.d/haresource:

nas-u1110node01 drbddisk::iscsi.config Filesystem::/dev/drbd0::/srv/iscsi-config::ext4
nas-u1110node01 IPaddr::172.16.70.2/24/eth1 drbddisk::iscsi.lv.lun0 portblock::tcp::3260::block iscsitarget portblock::tcp::3260::unblock

Reboot both nodes

Check services: heartbeat, iscisitarget, drdb at both nodes

First node:


root@nas-u1110node01:~# service heartbeat status
heartbeat OK [pid 1547 et al] is running on nas-u1110node01 [nas-u1110node01]...

root@nas-u1110node01:~# service iscsitarget status
 * iSCSI enterprise target is running

root@nas-u1110node01:~# service drbd status
drbd driver loaded OK; device status:
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
m:res            cs         ro                   ds                 p  mounted  fstype
0:iscsi_config   Connected  Primary/Secondary  UpToDate/UpToDate  C
1:iscsi_lv_lun0  Connected  Primary/Secondary  UpToDate/UpToDate  C

And second node:


root@nas-u1110node02:~# service heartbeat status
heartbeat OK [pid 1628 et al] is running on nas-u1110node02 [nas-u1110node02]...

root@nas-u1110node02:~# service iscsitarget status
 * could not access PID file for iSCSI enterprise target 

root@nas-u1110node02:~# service drbd status
drbd driver loaded OK; device status:
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
m:res            cs         ro                   ds                 p  mounted  fstype
0:iscsi_config   Connected  Secondary/Primary  UpToDate/UpToDate  C
1:iscsi_lv_lun0  Connected  Secondary/Primary  UpToDate/UpToDate  C

1 comment:

RubenMay 15, 2012 at 1:50 PM
Hi Eugeny,

How is the performance of Proxmox2 OpenVZ containers using iSCSI running from your HA cluster?

I've tried to use a GlusterFS (www.gluster.org) filecluster, but I've experienced it as slow in I/O and was unable to run a container on it....

Best Regards,
Ruben

EugenyHo

Pages

Saturday, April 21, 2012

HA Storage cluster setup process

1 comment: