Network Services Disaster Recovery

• Servers
• Server Incantations
• Upgrading SLE
• Mapped Drives Not Available in Windows Save Dialogs
• Creating Library Student Worker Accounts
• Superfluous eDirectory Accounts
• DRBD Recovery
• Tegile Array Information
• CWDB
• CWDB Archive
• CWDB Backup
• Backup Process
• WordPress Customizations
• DMZ Hosts & IP Addresses
• SSL Certificates
• Orbeon Setup
• Daily Ops Duties
• XenServer Cluster Documentation
• XenServer Recovery and Other Things
• Xen Appliance Conversion
• CWDB Dev Server Refresh Scripts
• FreePBX
• Comcast Documentation and Information
• Updating the Call List on Call Day
• Moodle
• Student Worker Admin Accounts
• Network Services Admin Accounts
• Trane Cloud VPN

Servers

Physical

Internal XenServer Cluster

Hosts

Virtual Machines

Access Virtual Machines

External XenServer Cluster

Hosts

Virtual Machines

Other

Server Incantations

SLES

Upgrading SLE

From SLE 11 SP3 to SLE 11 SP4

Taken from https://www.suse.com/support/kb/doc.php?id=7016711.

From SLE 12 to SLE 12 SP1

Taken from https://www.suse.com/documentation/sles-12/book_sle_deployment/data/sec_update_migr_zypper_onlinemigr.html.

Slow Boot Issues after Service Pack Migration

Check the boot loader in YaST for incorrect drive names both for the boot device and the kernel parameters.

Mapped Drives Not Available in Windows Save Dialogs

HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\ EnableLinkedConnections =1

New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System" -PropertyType DWORD -Name 'EnableLinkedConnections' -Value 1

Creating Library Student Worker Accounts

Current Active Worker Accounts

Superfluous eDirectory Accounts

These are current accounts which are not in the database as of 2017-01-16.

['wilsonbk', 'wagneras', 'penterwl', 'malkowjt', 'henselrh', 'buchhomd', 'townewm', 'schmitan', 'schlotkr', 'rynohg', 'retberan', 'nusharsm', 'millerrh', 'lochharc', 'lindemmr', 'has', 'everslj', 'bramstar', 'boveeke', 'andersre', 'walkerlm', 'miskotc', 'barretse', 'wileyca', 'weinstae', 'wallaj', 'viethsnj', 'tenyerjl', 'swansose', 'stuevecb', 'stanosta', 'schumass', 'schliemd', 'richardj', 'pretzear', 'polferrj', 'lindowkc', 'lincejm', 'kinneyee', 'kietahm', 'hollinca', 'hartmacj', 'greenwmp', 'franckag', 'douglarw', 'davisec', 'boylansm', 'bowlesmr', 'borreeka', 'krauseba', 'danelljm']

DRBD Recovery

This is documentation to bring back the old (Ubuntu 12.04) storage servers from a cold start to being able to connect with the XenServer cluster over NFS.

Current Configuration

Internal

External

The Steps

You can now have the XenServer cluster go ahead and fix the NFS SR issues. Things should now be working.

Tegile Array Information

Networking Information

T3100 - Jacob

Switch Ports

NS01

iSCSI VLAN

Management VLAN

SS2100 - Esau (Offline)

Switch Ports

CC01

iSCSI VLAN

Management VLAN

HA2100 - Isaac (Temp)

Switch Ports

iSCSI VLAN

Academic VLAN

CWDB

Installation

SLE Modules

Installed Packages

Users

Useful Incantations

Managing PostgreSQL Process

rcpostgresql start|stop|restart|reload

Load Firewall Rules

SuSEfirewall2

Cron Jobs

Root

Copies custom firewall rules into area where normal backups can grab a copy and changes the ownership so that it can be copied over easily.

0 0 * * * cp bin/SuSEfirewall2-custom /var/lib/pgsql/data/ | chown postgres:postgres /var/lib/pgsql/data/SuSEfirewall2-custom

Postgres

Runs the backup script that copies the /data directory via rsync.

15 3 * * * /var/lib/pgsql/bin/pg_binary_backup.sh >/dev/null 2>&1

Firewall

There is a need for custom rules for the firewall to handle PostgreSQL and SSH connections. They are stored in /root/bin/SuSEfirewall2-custom. You can find a copy of this file within the binary backup of the /data directory for cwdb stored on archive.

Custom Rules File

Add the rules within the fw_custom_before_masq() area

Backup

WAL archives and /data directory backups are housed on the archive server.

CWDB Archive

Installation

SLE Modules

Users

Cron Jobs

Archive

Runs the cleanup script for old backups. Currently only keeping a weeks worth of backups (including WAL archives).

15 4 * * * /home/archive/bin/clean_old_backups.sh >/dev/null 2>&1

CWDB Backups

Locations

Backup Pruning

Currently we keep only a week of backups. This script it run every night and delete the oldest backup.

CWDB Backup

Backup Overview

The backups for the CWDB are some of the most complex we do on campus. The effect is to allow us to both restore from nothing while losing as few database transactions as possible, and to be able to use PITR (point-in-time recovery) to recover from smaller issues than a complete loss. This is accomplished in three ways:

That is the 10,000 foot view of what is going on with CWDB backups.

WAL Archiving

Binary Backups

Disaster Recovery Backups

Backup Process

This document lays out how backups are handled.

Cadence

Weekly

Monthly

Restore Testing

Keep track of when restores and tested, how, and the outcome.

WordPress Customizations

Left Subnavigation Menu

.sidebar_left .widget_nav_menu {
  text-align: left;
}
#top .sidebar_left .widget_nav_menu ul ul li:before {
  left: 1px;
}

DMZ Hosts & IP Addresses

External Hosts

Internal Hosts

SSL Certificates

Orbeon Setup

Steps to Create an Orbeon App

CWDB

-- create user for Orbeon to use
CREATE ROLE orbeon_XXX LOGIN
  NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION;
-- create group for department users
CREATE ROLE XXX_forms
  NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION;
-- create the schema for forms to live in
CREATE SCHEMA orbeon_XXX AUTHORIZATION orbeon_XXX;
-- set the search path for the user Orbeon will be using
ALTER ROLE orbeon_XXX
  SET search_path = orbeon_XXX;
-- grant admin user for campus DB admin access to forms
GRANT USAGE ON SCHEMA orbeon_XXX TO admin_general;
-- grant department users access to forms
GRANT USAGE ON SCHEMA orbeon_XXX TO XXX_forms;
-- grant access to campus DB admin for any additional tables created by admin user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
    GRANT SELECT ON TABLES
    TO admin_general;
-- grant access to department users for any additional tables created by admin user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
    GRANT SELECT ON TABLES
    TO XXX_forms;

-- grant access to campus DB admin for any additional tables created by orbeon_XXX user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
    GRANT SELECT ON TABLES
    TO admin_general;
-- grant access to department users for any additional tables created by orbeon_XXX user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
    GRANT SELECT ON TABLES
    TO XXX_forms;

OES

Orbeon Server

cd /root/orbeon
bin/deploy.sh war/current_link.war
service tomcat restart

Daily Ops Duties

This lists the daily tasks done by operations personnel on campus.

Backups

Internal Backups

Weekdays

Weekends

DMZ Backups

Weekdays

Weekends

Support Tickets

XenServer Cluster Documentation

Internal Cluster

General Network Info

External Cluster

General Network Info

Storage Network

General Network Info

XenServer Recovery and Other Things

Error: "VDI Not Available"

When a host box dies, often it will die without first notifying the rest of the hosts about the issue. In those cases VMs can get stuck and when you try and restart them you'll end up with the following error: VDI Not Available.

This sucks. Follow the steps on this page to correct it:

Force VMs Down When Stuck

When a host box dies, often it will die without first notifying the rest of the hosts about the issue. In those cases, VMs can get stuck and are “missing” when viewed in XenCenter. You'll need to force them down so they show up again:

Xen Appliance Conversion

From Novell Cool Solutions.

Troubleshooting

CWDB Dev Server Refresh Scripts

The instructions below have been turned into two scripts. The refresh calls sync.

cwdb-sync.sh
cwdb-refresh.sh

CWDB Dev Server Refresh Instructions

# on the dev server
# ssh root@cwdb-dev

# sync
rsync -avz archive@cwdb-archive.mlc-wels.edu:cwdb/data/`date +"%y-%m-%d"`/ /var/lib/pgsql/data_new
rsync -avz archive@cwdb-archive.mlc-wels.edu:cwdb/wal/ /var/lib/pgsql/archive

# refresh
rcpostgresql stop

rm -r /var/lib/pgsql/data/pg_xlog
rsync -av /var/lib/pgsql/data_new/ /var/lib/pgsql/data

mkdir -m 700 /var/lib/pgsql/data/pg_xlog
mv /var/lib/pgsql/data/postgresql.conf /var/lib/pgsql/data/postgresql.conf.prod
mv /var/lib/pgsql/data/postgresql.conf.dev /var/lib/pgsql/data/postgresql.conf
mv /var/lib/pgsql/data/recovery.conf.dev /var/lib/pgsql/data/recovery.conf

cp /var/lib/pgsql/data/SuSEfirewall2-custom /root/bin/SuSEfirewall2-custom
SuSEfirewall2

chown -R postgres:postgres /var/lib/pgsql/archive
chown -R postgres:postgres /var/lib/pgsql/data

rcpostgresql start

rm /var/lib/pgsql/data/recovery.done

Old Instructions

FreePBX

SSH

GUI

E911

Any time an extension is moved to a different location, or if a new extension is created, the e911 information for that phone extension needs to be checked. Background: The campus has been divided into zones for the purpose of locating where a 911 call originated. Each zone is associated with an “Emergency Caller ID” that is assigned to each phone located in that zone. That Emergency CID needs to be entered into the configuration for each extension. The Emergency CID is a Direct Inward Dial (DID) of an assigned phone in that zone. Each room on campus is assigned a zone number in the public.rooms table of the Campuswide Database (CWDB). The public.valEmergencyZones table has the EmergencyZone_Name, EmergencyZone_Location, EmergencyZone_Comments, and the DirectDial_ID for each zone.

Comcast Documentation and Information

Here is information about current Comcast/XFINITY setup on campus related to connectivity.

Metro-E Service

XFINITY on Campus Circuit

Updating the Call List on Call Day

Update the Calls & Assignments page on the website:

Move Call Day List to Proper Location

Moodle

• MyLab & Mastering Tools
• Automatic, based on tool URL
• https://tpi.bb.pearsoncmg.com/highlander/api/o/lti/tools

• martinluther.moodleblti.com
• KsHKyCKe
• https://moodle.mlc-wels.edu/moodle/blocks/mylabmastering/pix/icon.jpg

Student Worker Admin Accounts

Network Services Admin Accounts

Account	Person	Assigned
arwen	Laura Stelljes
eowyn	Jill Roux
gaban	AVAILABLE
galadrie	AVAILABLE
gandalf	AVAILABLE
laker	James Rathje
legolas	Bob Martens
merlin	AVAILABLE
modred	Ken Jones
sauron	Aaron Spike

Trane Cloud VPN

Branch Office Gateway

Branch Office Tunnel

More Information

Use the wizard to setup the default BOVPN rules (using an All set) and then modify them for only the Trane VLAN and turn on logging for all rules. You may need to re-key the VPN if you make any changes.