# Network Services Disaster Recovery
This is the documentation for the infrastructure at Martin Luther College. The idea is to document everything well-enough that a person would be able to know what is where and what it does to get things back up-and-running.
We will also hold information for some known issues when bringing servers up from being powered-off.
# Servers
-
`chkconfig –add [service]` - starts the service on startup
-
`rpm -i [path to installation rpm]` - installs the rpm (useful for installing xs-tools on a host not included in the `install.sh` file)
-
`zypper up` - upgrade server to latest package revisions
-
`zypper search` - search for packages containing the term you want
-
`zipper dup –no-allow-vendor-change` - safer way to upgrade servers with additional repos
-
`rc[process name] start|stop|restart|reload` - manage processes (tab will show you the available processes
-
`SuSEfirewall2` - load and apply any custom firewall rules you have setup within YaST
# Upgrading SLE
## From SLE 11 SP3 to SLE 11 SP4
Taken from [https://www.suse.com/support/kb/doc.php?id=7016711](https://www.suse.com/support/kb/doc.php?id=7016711 "https://www.suse.com/support/kb/doc.php?id=7016711").
-
`zypper ref -s`
-
`zypper update -t patch`
-
`zypper update -t patch` (again)
-
`zypper se -t product | grep -h – “-migration” | cut -d\| -f2`
-
A sample output could be as follows: `SUSE_SLES-SP4-migration`
-
`zypper in -t product sle-sdk-SP4-migration SUSE_SLES-SP4-migration` (modify from what is shown in above command)
-
`suse_register -d 2 -L /root/.suse_register.log`
-
`zypper ref -s`
-
`zypper lr`
-
`zypper mr –disable ` any repos that are not needed
-
`zypper dup –from SLES11-SP4-Pool –from SLES11-SP4-Updates` plus other repos as needed
-
`suse_register -d 2 -L /root/.suse_register.log`
-
Reboot the machine
## From SLE 12 to SLE 12 SP1
Taken from [https://www.suse.com/documentation/sles-12/book\_sle\_deployment/data/sec\_update\_migr\_zypper\_onlinemigr.html](https://www.suse.com/documentation/sles-12/book_sle_deployment/data/sec_update_migr_zypper_onlinemigr.html "https://www.suse.com/documentation/sles-12/book_sle_deployment/data/sec_update_migr_zypper_onlinemigr.html").
-
Install the latest updates.
-
Install the packages `zypper-migration-plugin` and their dependencies.
-
Run the zypper migration: `zypper migration`.
-
Review all the changes, especially the packages that are going to be removed. Proceed by typing y.
-
After successful migration restart your system.
## Slow Boot Issues after Service Pack Migration
Check the boot loader in YaST for incorrect drive names both for the boot device and the kernel parameters.
# Mapped Drives Not Available in Windows Save Dialogs
-
[https://www.novell.com/support/kb/doc.php?id=7009906](https://www.novell.com/support/kb/doc.php?id=7009906 "https://www.novell.com/support/kb/doc.php?id=7009906")
-
[https://social.technet.microsoft.com/Forums/en-US/62456d84-95a1-4d43-9745-d8c4e8e600fb/since-kb3194798-enablelinkedconnections-is-not-working-anymore-mapping-shares-mmc-on-network?forum=win10itprogeneral](https://social.technet.microsoft.com/Forums/en-US/62456d84-95a1-4d43-9745-d8c4e8e600fb/since-kb3194798-enablelinkedconnections-is-not-working-anymore-mapping-shares-mmc-on-network?forum=win10itprogeneral "https://social.technet.microsoft.com/Forums/en-US/62456d84-95a1-4d43-9745-d8c4e8e600fb/since-kb3194798-enablelinkedconnections-is-not-working-anymore-mapping-shares-mmc-on-network?forum=win10itprogeneral")
`HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\ EnableLinkedConnections =1`
```
New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System" -PropertyType DWORD -Name 'EnableLinkedConnections' -Value 1
```
# Creating Library Student Worker Accounts
1.
Select an unassigned WorkerXX.wrk.lib.ac.mlc account to assign
2.
Configure WorkerXX with appropriate group memberships
3.
Configure WorkerXX with additional permissions as appropriate
4.
Be sure to configure station access restrictions as necessary
5.
Create an alias object in staff.lib.ac.mlc with the student’s login name
6.
Set a temporary password on WorkerXX
7.
Student logs in using the distinguished name of the alias object (e.g. *spikeac.staff.lib.ac.mlc*) and the temporary password assigned for WorkerXX
## Current Active Worker Accounts
-
bring servers back from the dead, you can have them both up before starting anything
-
`modprobe drbd` - checks and enables the proper kernel module
-
`drbd-overview` - check `drbd` status
-
On Primary
-
`drbdadm connect [i]nfs[1/2]` - connect to the `drbd` shares
-
On Secondary
-
`drbdadm – –discard-my-data connect [i]nfs[1/2]` - connect to the `drbd` shares
-
On Primary
-
`drbdadm primary [i]nfs[1/2]` - set the primary server as the primary device within `drbd`
-
`mount -o noatime /dev/drbd0 /srv/[i]nfs[1/2]` - mount the `drbd` block device to the proper mount point
-
`service nfs-kernel-service start` - start the `nfs` service
You can now have the XenServer cluster go ahead and fix the NFS SR issues. Things should now be working.
# Tegile Array Information
## Networking Information
-
SMTP: `mailhost.mlc-wels.edu`
-
Email: `servers@mlc-wels.edu`
-
NTP: `0.pool.ntp.org`
-
DNS: `192.168.95.100 192.168.95.101`
-
DNS Suffix: `mlc-wels.edu`
### T3100 - Jacob
-
Location: WCC Primary Server Room
#### Switch Ports
##### NS01
-
192.168.91.10 - Floating IP
-
192.168.91.11 - Jacob-A
-
192.168.91.12 - Jacob-B
-
192.168.91.13 - -Floating IP
#### Management VLAN
-
172.16.0.200 - Array Floating IP
-
172.16.0.201 - Jacob-A IP
-
172.16.0.202 - Jacob-B IP
-
172.16.0.203 - Jacob-A IPMI
-
172.16.0.204 - Jacob-B IPMI
### SS2100 - Esau (Offline)
-
Location: Chapel of the Christ Secondary Server Room
#### Switch Ports
##### CC01
-
172.16.0.205 - Controller IP
-
172.16.0.206 - IMPI
### HA2100 - Isaac (Temp)
-
Location: Chapel of the Christ Secondary Server Room
#### Switch Ports
-
172.16.0.210 - Controller Management
-
172.16.0.211
-
172.16.0.212
-
172.16.0.213
-
172.16.0.214
# CWDB
-
`postgres` (created when installing the `postgresql` package)
## Useful Incantations
### Managing PostgreSQL Process
```
rcpostgresql start|stop|restart|reload
```
### Load Firewall Rules
```
SuSEfirewall2
```
## Cron Jobs
### Root
Copies custom firewall rules into area where normal backups can grab a copy and changes the ownership so that it can be copied over easily.
```
0 0 * * * cp bin/SuSEfirewall2-custom /var/lib/pgsql/data/ | chown postgres:postgres /var/lib/pgsql/data/SuSEfirewall2-custom
```
### Postgres
Runs the backup script that copies the `/data` directory via `rsync`.
```
15 3 * * * /var/lib/pgsql/bin/pg_binary_backup.sh >/dev/null 2>&1
```
## Firewall
There is a need for custom rules for the firewall to handle PostgreSQL and SSH connections. They are stored in `/root/bin/SuSEfirewall2-custom`. You can find a copy of this file within the binary backup of the `/data` directory for cwdb stored on archive.
-
You will need to tell SUSE to load these custom rules by going to `YaST > System > /etc/sysconfig Editor > Network > Firewall > SuSEfirewall2 > FW_CUSTOMRULES` and then adding `/root/bin/SuSEfirewall2-custom` into the settings
-
When you make changes to the custom rules, you will need to run the `SuSEfirewall2` command as `root` (pay attention to any error messages)
### Custom Rules File
Add the rules within the `fw_custom_before_masq()` area
-
`/home/archive/cwdb` is the main directory
-
`/home/archive/cwdb/wal` directory holds the WAL archives
-
`/home/archive/cwdb/data` has a dated directory for each date a full binary backup has been done
### Backup Pruning
Currently we keep only a week of backups. This script it run every night and delete the oldest backup.
1.
**WAL Archiving** ships the PostgreSQL write-ahead logs to the archive server where they can be “played back” in the future to a certain point-in-time.
2.
**Binary Backups** use `rsync` to take complete backups of the entire database `data` directory which allows us to grab not just the data (most important) but also the configuration files for PostgreSQL.
3.
Peridocally, snapshots of both the binary backup and the wal archives will be committed to tape (or some other off-campus backup solution) for ultimate data recovery options. ***This is not yet automated***.
That is the 10,000 foot view of what is going on with CWDB backups.
## WAL Archiving
-
Each Thursday replace the prior longterm archival tape with a different tape for the coming week's archive operation
-
Label tape with the date of the archival process (the coming Wednesday)
-
IF A USED TAPE clear it before labeling in Bareos with `mt -f /dev/st0 rewind && mt -f /dev/st0 weof && mt -f /dev/st0 rewind` command
-
Label the tape using the name `Longterm-YYYY-MM-DD` which matches the label on the outside
-
Make sure the naming and mounting processes are successful
-
Take the prior archival tape and get it to director for storage offsite
### Monthly
-
Keep the prior month's latest archival tape for future restores
-
Put other tapes into the rotation to be reused for future jobs
## Restore Testing
Keep track of when restores and tested, how, and the outcome.
-
Create needed schema and roles for new Orebon app.
```
-- create user for Orbeon to use
CREATE ROLE orbeon_XXX LOGIN
NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION;
-- create group for department users
CREATE ROLE XXX_forms
NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION;
-- create the schema for forms to live in
CREATE SCHEMA orbeon_XXX AUTHORIZATION orbeon_XXX;
-- set the search path for the user Orbeon will be using
ALTER ROLE orbeon_XXX
SET search_path = orbeon_XXX;
-- grant admin user for campus DB admin access to forms
GRANT USAGE ON SCHEMA orbeon_XXX TO admin_general;
-- grant department users access to forms
GRANT USAGE ON SCHEMA orbeon_XXX TO XXX_forms;
-- grant access to campus DB admin for any additional tables created by admin user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
GRANT SELECT ON TABLES
TO admin_general;
-- grant access to department users for any additional tables created by admin user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
GRANT SELECT ON TABLES
TO XXX_forms;
```
-
Set password for `orbeon_XXX` user in PGAdmin.
-
Login: `psql -U orbeon_XXX -h database.mlc-wels.edu cwdb`
-
Check search path with: `show search_path;`
-
Grant additional permissions by pasting in SQL statement below as `orbeon_XXX` user
```
-- grant access to campus DB admin for any additional tables created by orbeon_XXX user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
GRANT SELECT ON TABLES
TO admin_general;
-- grant access to department users for any additional tables created by orbeon_XXX user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
GRANT SELECT ON TABLES
TO XXX_forms;
```
-
Paste edited schema definition from [https://github.com/orbeon/orbeon-forms/blob/master/src/resources/apps/fr/persistence/relational/ddl/postgresql-4\_8.sql](https://github.com/orbeon/orbeon-forms/blob/master/src/resources/apps/fr/persistence/relational/ddl/postgresql-4_8.sql "https://github.com/orbeon/orbeon-forms/blob/master/src/resources/apps/fr/persistence/relational/ddl/postgresql-4_8.sql") (edited copy in `/root/orbeon/conf`)
-
Add `, pk serial primary key` to each table def
-
Add access rules to `pg_hba.conf` on CWDB and reload postgresql service configuration
### OES
-
Create group `OrbeonXXX.groups.ac.mlc` in iManager
### Orbeon Server
-
Alter Orbeon config files in `/root/orbeon/config`
-
Create database resource in orbeon `context.xml`
-
Add role assignment in Orbeon `form-builder-permissions.xml`
-
Add orbeon persistence connection in `properties-local.xml`
-
Add role to `oxf.fr.authentication.container.roles` in `properties-local.xml`
-
Add role name to `auth-constraint` in `web.xml`
-
Add role name to `security-role` in `web.xml`
-
Check for active orbeon user sessions: [http://orbeon.mlc-wels.edu:8080/manager/](http://orbeon.mlc-wels.edu:8080/manager/ "http://orbeon.mlc-wels.edu:8080/manager/")
-
Re-deploy Orbeon
```
cd /root/orbeon
bin/deploy.sh war/current_link.war
service tomcat restart
```
# Daily Ops Duties
This lists the daily tasks done by operations personnel on campus.
## Backups
### Internal Backups
#### Weekdays
-
Verify that the prior backup was successful
-
Swap the backup tape with the tape labeled for the **NEXT DAY**
-
Log into `Portal` and `CWDB` and copy backups via SFTP to `ADMIN/Vol1/ServerBackups`
#### Weekends
-
Label tape with date for the next Saturday
-
Swap the backup tape with the tape you just labeled
-
**ON SUNDAY**, swap the backup tape with the tape labeled for **MONDAY**
### DMZ Backups
#### Weekdays
-
Verify there are no errors from the prior backup
-
Swap the backup tape with the tape labeled for the **NEXT DAY**
#### Weekends
-
Use the `bctapelist` script to find which tape should be used next
-
Swap the backup tape with the next tape from the `bctapelist` script
-
Enjoy your weekend because you will not need to swap out a tape for this system until Monday
## Support Tickets
1.
Log into support.mlc-wels.edu
2.
Look for new tickets that have not been assigned
3.
Triage the tickets you can, assign tickets to those people who need them
-
**Password reset** requests are usually assigned to **Jill**
-
**Phone** issues and **signage** issues are assigned to **Jim**
-
**Database** issues start at **Laura**
-
**Portal** requests are assigned to **Aaron**
-
**Network**, **Server**, and **File Sharing** requests go to **Bob**
-
**Printer** issues start with **Ken**
-
**Notebook** and **desktop** issues start with **Ken**
-
**Paper** requests go to a **student worker**
-
**Website** issues start with **Bob**
-
**Website content** request go to **Sallie**
4.
Just use your best judgement for others
# XenServer Cluster Documentation
## Internal Cluster
-
**Subnet**: 255.255.255.0
-
**Gateway**: 192.168.95.2
-
**
DNS**: 192.168.95.100, 192.168.95.101
-
**NTP**: oes.mlc-wels.edu, archive.mlc-wels.edu
## Storage Network
-
[http://support.citrix.com/article/CTX138234](http://support.citrix.com/article/CTX138234 "http://support.citrix.com/article/CTX138234")
## Force VMs Down When Stuck
When a host box dies, often it will die without first notifying the rest of the hosts about the issue. In those cases, VMs can get stuck and are “missing” when viewed in XenCenter. You'll need to force them down so they show up again:
-
[http://support.citrix.com/article/CTX126382](http://support.citrix.com/article/CTX126382 "http://support.citrix.com/article/CTX126382")
# Xen Appliance Conversion
From [Novell Cool Solutions](https://www.novell.com/communities/coolsolutions/converting-xen-appliances-xenserver-import/ "https://www.novell.com/communities/coolsolutions/converting-xen-appliances-xenserver-import/").
1.
Download the wanted Xen appliance from the Novell site. I chose iPrint 2 as my test appliance because I want to test iPrint.
2.
Unarchive the download. You should have a folder with a raw disk image and a xenconfig file. My Filr disk image is 21+
GB in size once it is expanded. The xenconfig file is only 179 bytes.
3.
Open your terminal application of choice and move into that newly created appliance folder.
4.
Grab xva.py and drop it into the folder above the unarchived appliance folder. I used `curl
http://www-archive.xenproject.org/files/xva/xva.py > xva.py` but you better just [grab it from here](https://kb.mlc-wels.edu/_media/disaster/xenserver/xva.py "disaster:xenserver:xva.py (80.5 KB)").
5.
Now is the fun part. Make sure you have enough free disk space to handle making a copy of the disk image. Also, make sure that xva.py is within that appliance folder. It will just make things easier.
6.
Next I ran the following: `python xva.py iPrintAppliance-2.0.0.529/iPrintAppliance.x86_64-2.0.0.529.xenconfig -d iPrintAppliance-2.0.0.529/iPrintAppliance.x86_64-2.0.0.529.raw -f iPrintAppliance-2.0.0.2529.xva` which will inspect the image and then output the whole thing as an XVA for import into XenServer. The xenconfig file contains the name of the disk image and other parameters needed, but there is the possibility you will need to include the disk anyway.
## Troubleshooting
-
You might need to use the `-d` flag to specify where to find the raw disk
# CWDB Dev Server Refresh Scripts
The instructions below have been turned into two scripts. The refresh calls sync.
```
cwdb-sync.sh
cwdb-refresh.sh
```
# CWDB Dev Server Refresh Instructions
```
# on the dev server
# ssh root@cwdb-dev
# sync
rsync -avz archive@cwdb-archive.mlc-wels.edu:cwdb/data/`date +"%y-%m-%d"`/ /var/lib/pgsql/data_new
rsync -avz archive@cwdb-archive.mlc-wels.edu:cwdb/wal/ /var/lib/pgsql/archive
# refresh
rcpostgresql stop
rm -r /var/lib/pgsql/data/pg_xlog
rsync -av /var/lib/pgsql/data_new/ /var/lib/pgsql/data
mkdir -m 700 /var/lib/pgsql/data/pg_xlog
mv /var/lib/pgsql/data/postgresql.conf /var/lib/pgsql/data/postgresql.conf.prod
mv /var/lib/pgsql/data/postgresql.conf.dev /var/lib/pgsql/data/postgresql.conf
mv /var/lib/pgsql/data/recovery.conf.dev /var/lib/pgsql/data/recovery.conf
cp /var/lib/pgsql/data/SuSEfirewall2-custom /root/bin/SuSEfirewall2-custom
SuSEfirewall2
chown -R postgres:postgres /var/lib/pgsql/archive
chown -R postgres:postgres /var/lib/pgsql/data
rcpostgresql start
rm /var/lib/pgsql/data/recovery.done
```
## Old Instructions
-
install PostgreSQL server packages for your
OS
-
`zypper in postgresql-server postgresql-contrib`
-
start up PostgreSQL on
OS (to create default directories)
-
`rcpostgresql start`
-
you'll need to move the full data backup from `cwdb-archive` to `cwdb-dev` and replace all of the contents of the `/var/lib/pgsql/data` directory (we keep a number of days back)
-
copy over wal directory from cwdb-archive to cwdb-dev and place it in the `/var/lib/pgsql/data` directory
-
create `pg_xlog` directory
-
`mkdir /var/lib/pgsql/data/pg_xlog`
-
make sure that everything in the data directory is owned by `postgres:postgres` with `700` permissions
-
chown postgres:postgres
-
make certain to open the PostgreSQL Server ports in the firewall
# FreePBX
## SSH
-
**Phone #:** (800) 741-4141
-
**MLC Account #:** 930-000-194
-
**MLC Phone #:** (507) 354-8221
-
**MLC Address:** 1995 Luther Ct, New Ulm, MN 56073
## XFINITY on Campus Circuit
1.
Log into [https://mlc-wels.edu/login](https://mlc-wels.edu/login "https://mlc-wels.edu/login") with your MLC WordPress Account
2.
Navigate to [https://mlc-wels.edu/assignments/](https://mlc-wels.edu/assignments/ "https://mlc-wels.edu/assignments/") and click `Edit Page` in the top toolbar
3.
Change the link for *May* under *2017* to [https://mlc-wels.edu/static/may-2017.pdf](https://mlc-wels.edu/static/may-2017.pdf "https://mlc-wels.edu/static/may-2017.pdf") (this link will not be live yet)
4.
Click on *Update* to save the changes
# Move Call Day List to Proper Location
1.
Log into `mlc-wels.edu`
2.
Copy PDF from `root` to `static` directory: `cp /root/may-2017.pdf /srv/www/htdocs/mlc-wels.edu/static/`
# Moodle
- [https://tpi.bb.pearsoncmg.com/highlander/api/o/lti/tools](https://tpi.bb.pearsoncmg.com/highlander/api/o/lti/tools "https://tpi.bb.pearsoncmg.com/highlander/api/o/lti/tools")
- [https://moodle.mlc-wels.edu/moodle/blocks/mylabmastering/pix/icon.jpg](https://moodle.mlc-wels.edu/moodle/blocks/mylabmastering/pix/icon.jpg "https://moodle.mlc-wels.edu/moodle/blocks/mylabmastering/pix/icon.jpg")
# Student Worker Admin Accounts
-
**Local Network:** 10.11.150.0/24
-
**Local Gateway:** 10.11.150.2
-
**Remote IP:** 52.43.55.153
-
**Remote ID:** 10.242.202.66
-
**Pre-Shared Key:** SEE PASSWORD SAFE
-
**Version:** IKEv1
-
**Phase 1 Transform:** SHA1-AES (256-bit)
-
**Phase 1 Key Group:** DH Group2
## Branch Office Tunnel
-
**Tunnel Local Addresses:** See Local Network
-
**Tunnel Remote Address:** 10.242.202.101/32
-
**Phase 2 PFS:** DH Group2
-
**Phase 2 IPSec Proposal:** ESP-AES256-SHA256
## More Information
Use the wizard to setup the default BOVPN rules (using an All set) and then modify them for only the Trane VLAN and turn on logging for all rules. You may need to re-key the VPN if you make any changes.