Network Services Disaster Recovery

• Servers
• Server Incantations
• Upgrading SLE
• Mapped Drives Not Available in Windows Save Dialogs
• Creating Library Student Worker Accounts
• Superfluous eDirectory Accounts
• DRBD Recovery
• Tegile Array Information
• CWDB
• CWDB Archive
• CWDB Backup
• Backup Process
• WordPress Customizations
• DMZ Hosts & IP Addresses
• SSL Certificates
• Orbeon Setup
• Daily Ops Duties
• XenServer Cluster Documentation
• XenServer Recovery and Other Things
• Xen Appliance Conversion
• CWDB Dev Server Refresh Scripts
• FreePBX
• Comcast Documentation and Information
• Updating the Call List on Call Day
• Moodle
• Student Worker Admin Accounts
• Network Services Admin Accounts
• Trane Cloud VPN
• Goats

Servers

Physical

Internal XenServer Cluster

Hosts

Virtual Machines

Access Virtual Machines

External XenServer Cluster

Hosts

Virtual Machines

Other

Server Incantations

SLES

Upgrading SLE

From SLE 11 SP3 to SLE 11 SP4

Taken from https://www.suse.com/support/kb/doc.php?id=7016711.

From SLE 12 to SLE 12 SP1

Taken from https://www.suse.com/documentation/sles-12/book_sle_deployment/data/sec_update_migr_zypper_onlinemigr.html.

Slow Boot Issues after Service Pack Migration

Check the boot loader in YaST for incorrect drive names both for the boot device and the kernel parameters.

Mapped Drives Not Available in Windows Save Dialogs

HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\ EnableLinkedConnections =1

New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System" -PropertyType DWORD -Name 'EnableLinkedConnections' -Value 1

Creating Library Student Worker Accounts

Current Active Worker Accounts

Superfluous eDirectory Accounts

These are current accounts which are not in the database as of 2017-01-16.

['wilsonbk', 'wagneras', 'penterwl', 'malkowjt', 'henselrh', 'buchhomd', 'townewm', 'schmitan', 'schlotkr', 'rynohg', 'retberan', 'nusharsm', 'millerrh', 'lochharc', 'lindemmr', 'has', 'everslj', 'bramstar', 'boveeke', 'andersre', 'walkerlm', 'miskotc', 'barretse', 'wileyca', 'weinstae', 'wallaj', 'viethsnj', 'tenyerjl', 'swansose', 'stuevecb', 'stanosta', 'schumass', 'schliemd', 'richardj', 'pretzear', 'polferrj', 'lindowkc', 'lincejm', 'kinneyee', 'kietahm', 'hollinca', 'hartmacj', 'greenwmp', 'franckag', 'douglarw', 'davisec', 'boylansm', 'bowlesmr', 'borreeka', 'krauseba', 'danelljm']

DRBD Recovery

This is documentation to bring back the old (Ubuntu 12.04) storage servers from a cold start to being able to connect with the XenServer cluster over NFS.

Current Configuration

Internal

External

The Steps

You can now have the XenServer cluster go ahead and fix the NFS SR issues. Things should now be working.

Tegile Array Information

Networking Information

T3100 - Jacob

Switch Ports

NS01

iSCSI VLAN

Management VLAN

SS2100 - Esau (Offline)

Switch Ports

CC01

iSCSI VLAN

Management VLAN

HA2100 - Isaac (Temp)

Switch Ports

iSCSI VLAN

Academic VLAN

CWDB

Installation

SLE Modules

Installed Packages

Users

Useful Incantations

Managing PostgreSQL Process

rcpostgresql start|stop|restart|reload

Load Firewall Rules

SuSEfirewall2

Cron Jobs

Root

Copies custom firewall rules into area where normal backups can grab a copy and changes the ownership so that it can be copied over easily.

0 0 * * * cp bin/SuSEfirewall2-custom /var/lib/pgsql/data/ | chown postgres:postgres /var/lib/pgsql/data/SuSEfirewall2-custom

Postgres

Runs the backup script that copies the /data directory via rsync.

15 3 * * * /var/lib/pgsql/bin/pg_binary_backup.sh >/dev/null 2>&1

Firewall

There is a need for custom rules for the firewall to handle PostgreSQL and SSH connections. They are stored in /root/bin/SuSEfirewall2-custom. You can find a copy of this file within the binary backup of the /data directory for cwdb stored on archive.

Custom Rules File

Add the rules within the fw_custom_before_masq() area

Backup

WAL archives and /data directory backups are housed on the archive server.

CWDB Archive

Installation

SLE Modules

Users

Cron Jobs

Archive

Runs the cleanup script for old backups. Currently only keeping a weeks worth of backups (including WAL archives).

15 4 * * * /home/archive/bin/clean_old_backups.sh >/dev/null 2>&1

CWDB Backups

Locations

Backup Pruning

Currently we keep only a week of backups. This script it run every night and delete the oldest backup.

CWDB Backup

Backup Overview

The backups for the CWDB are some of the most complex we do on campus. The effect is to allow us to both restore from nothing while losing as few database transactions as possible, and to be able to use PITR (point-in-time recovery) to recover from smaller issues than a complete loss. This is accomplished in three ways:

That is the 10,000 foot view of what is going on with CWDB backups.

WAL Archiving

Binary Backups

Disaster Recovery Backups

Backup Process

This document lays out how backups are handled.

Cadence

Weekly

Monthly

Restore Testing

Keep track of when restores and tested, how, and the outcome.

WordPress Customizations

Left Subnavigation Menu

.sidebar_left .widget_nav_menu {
  text-align: left;
}
#top .sidebar_left .widget_nav_menu ul ul li:before {
  left: 1px;
}

DMZ Hosts & IP Addresses

External Hosts

Internal Hosts

SSL Certificates

Orbeon Setup

Steps to Create an Orbeon App

CWDB

-- create user for Orbeon to use
CREATE ROLE orbeon_XXX LOGIN
  NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION;
-- create group for department users
CREATE ROLE XXX_forms
  NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION;
-- create the schema for forms to live in
CREATE SCHEMA orbeon_XXX AUTHORIZATION orbeon_XXX;
-- set the search path for the user Orbeon will be using
ALTER ROLE orbeon_XXX
  SET search_path = orbeon_XXX;
-- grant admin user for campus DB admin access to forms
GRANT USAGE ON SCHEMA orbeon_XXX TO admin_general;
-- grant department users access to forms
GRANT USAGE ON SCHEMA orbeon_XXX TO XXX_forms;
-- grant access to campus DB admin for any additional tables created by admin user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
    GRANT SELECT ON TABLES
    TO admin_general;
-- grant access to department users for any additional tables created by admin user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
    GRANT SELECT ON TABLES
    TO XXX_forms;

-- grant access to campus DB admin for any additional tables created by orbeon_XXX user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
    GRANT SELECT ON TABLES
    TO admin_general;
-- grant access to department users for any additional tables created by orbeon_XXX user
ALTER DEFAULT PRIVILEGES IN SCHEMA orbeon_XXX
    GRANT SELECT ON TABLES
    TO XXX_forms;

OES

Orbeon Server

cd /root/orbeon
bin/deploy.sh war/current_link.war
service tomcat restart

Daily Ops Duties

This lists the daily tasks done by operations personnel on campus.

Backups

Internal Backups

Weekdays

Weekends

DMZ Backups

Weekdays

Weekends

Support Tickets

XenServer Cluster Documentation

Internal Cluster

General Network Info

External Cluster

General Network Info

Storage Network

General Network Info

XenServer Recovery and Other Things

Error: "VDI Not Available"

When a host box dies, often it will die without first notifying the rest of the hosts about the issue. In those cases VMs can get stuck and when you try and restart them you'll end up with the following error: VDI Not Available.

This sucks. Follow the steps on this page to correct it:

Force VMs Down When Stuck

When a host box dies, often it will die without first notifying the rest of the hosts about the issue. In those cases, VMs can get stuck and are “missing” when viewed in XenCenter. You'll need to force them down so they show up again:

Xen Appliance Conversion

From Novell Cool Solutions.

Troubleshooting

CWDB Dev Server Refresh Scripts

The instructions below have been turned into two scripts. The refresh calls sync.

cwdb-sync.sh
cwdb-refresh.sh

CWDB Dev Server Refresh Instructions

# on the dev server
# ssh root@cwdb-dev

# sync
rsync -avz archive@cwdb-archive.mlc-wels.edu:cwdb/data/`date +"%y-%m-%d"`/ /var/lib/pgsql/data_new
rsync -avz archive@cwdb-archive.mlc-wels.edu:cwdb/wal/ /var/lib/pgsql/archive

# refresh
rcpostgresql stop

rm -r /var/lib/pgsql/data/pg_xlog
rsync -av /var/lib/pgsql/data_new/ /var/lib/pgsql/data

mkdir -m 700 /var/lib/pgsql/data/pg_xlog
mv /var/lib/pgsql/data/postgresql.conf /var/lib/pgsql/data/postgresql.conf.prod
mv /var/lib/pgsql/data/postgresql.conf.dev /var/lib/pgsql/data/postgresql.conf
mv /var/lib/pgsql/data/recovery.conf.dev /var/lib/pgsql/data/recovery.conf

cp /var/lib/pgsql/data/SuSEfirewall2-custom /root/bin/SuSEfirewall2-custom
SuSEfirewall2

chown -R postgres:postgres /var/lib/pgsql/archive
chown -R postgres:postgres /var/lib/pgsql/data

rcpostgresql start

rm /var/lib/pgsql/data/recovery.done

Old Instructions

FreePBX

SSH

GUI

E911

Any time an extension is moved to a different location, or if a new extension is created, the e911 information for that phone extension needs to be checked. Background: The campus has been divided into zones for the purpose of locating where a 911 call originated. Each zone is associated with an “Emergency Caller ID” that is assigned to each phone located in that zone. That Emergency CID needs to be entered into the configuration for each extension. The Emergency CID is a Direct Inward Dial (DID) of an assigned phone in that zone. Each room on campus is assigned a zone number in the public.rooms table of the Campuswide Database (CWDB). The public.valEmergencyZones table has the EmergencyZone_Name, EmergencyZone_Location, EmergencyZone_Comments, and the DirectDial_ID for each zone.

Comcast Documentation and Information

Here is information about current Comcast/XFINITY setup on campus related to connectivity.

Metro-E Service

XFINITY on Campus Circuit

Updating the Call List on Call Day

Update the Calls & Assignments page on the website:

Move Call Day List to Proper Location

Moodle

• MyLab & Mastering Tools
• Automatic, based on tool URL
• https://tpi.bb.pearsoncmg.com/highlander/api/o/lti/tools

• martinluther.moodleblti.com
• KsHKyCKe
• https://moodle.mlc-wels.edu/moodle/blocks/mylabmastering/pix/icon.jpg

Student Worker Admin Accounts

Network Services Admin Accounts

Account	Person	Assigned
arwen	Laura Stelljes
eowyn	Jill Roux
gaban	AVAILABLE
galadrie	AVAILABLE
gandalf	AVAILABLE
laker	James Rathje
legolas	Bob Martens
merlin	AVAILABLE
modred	Ken Jones
sauron	Aaron Spike

Trane Cloud VPN

Branch Office Gateway

Branch Office Tunnel

More Information

Use the wizard to setup the default BOVPN rules (using an All set) and then modify them for only the Trane VLAN and turn on logging for all rules. You may need to re-key the VPN if you make any changes.

Goats

Stolen from https://www.reddit.com/r/sysadmin/comments/4l7kjd/found_a_text_file_at_work_titled_why_should_i/.

• Found a text file at work titled “Why should I quit my job and become a goat farmer? (written during my “on-call” week)”
• You don't have to monitor the utilization on a goat.
• Milk a goat and the goat stays milked for a while.
• There are no 32-bit goats.
• You don't have to do a demo on a goat. And if you ever do, the goat will do what it's supposed to do and there's not a lot that can keep it from doing it.
• When a goat goes “down”, you just bury it and buy a new goat.
• Left alone, Billy goats and Nanny goats do what they're supposed to do. You don't need to format them, monitor them, be on-call for them, step, trace or inspect registers.
• Nobody cares if you're not a Certified Goat Engineer yet.
• Kill a goat to make a goat steak, and the goat stays dead.
• Most people will take advice from a goat farmer on how to paint a fence, cook a steak, fix a tractor, etc. but most people somehow just don't want to hear it from a computer weenie.
• Nobody can lie in a job interview about their goat experience.
• Goats don't page you.
• When it comes to “software” (food), EVERYTHING is compatible with a goat.
• You don't need to buy a “goat 98” to fix all the bugs in your goat 95
• You can tell whether a goat has been “debugged” by looking at it.
• Goats don't become obsolete. If they do, as long as you didn't neuter them, they make the necessary upgrades themselves.
• No commute.
• Goats are kind of cute. Computers aren't cute unless they're Macintoshes, and those are just plain annoying.
• No dress code. Of any kind. EVER.
• You always have the right “file permissions” to milk a goat.
• If a goat gives too many timeout errors, or does not avail you resources for your session, or if performance is generally slow for your applications on your goat, it just means you're having goat steak for dinner.
• You don't need to visit “shareware dot com” to get some tools to milk a goat. You either have your bucket or you don't.
• The bucket leaks, or it doesn't. You do not need to ask a network if you're still the owner of the bucket. You do not need to run a bucket compare against a copy you made of the bucket previously You couldn't care less about the checksum of the bucket.
• You don't need to “free up some megs” before you milk a goat.
• You get callouses on your hands - the way God intended!
• You don't need to call a staff meeting to make sure everyone's milking goats the same way.
• Nanny goats, with no TCP/IP stack loaded, and no DLC, still give milk.
• Just about any barnyard animal is fault tolerant (except some cows).
• You don't need to sign in with the front desk if you need to milk a goat on a weekend. You don't need to use a badge to open a front gate. If you find an empty coffee pot burning on the machine on a Saturday, you just yell at your wife.
• You don't need to worry if you've been spending a lot of time milking what you will later find out to have been an improperly labelled “development goat”.
• There is no such thing as a “preferred goat,” and your “goat context” is always correct. Passwords do not exist and your milking/slaughtering account will never be disabled because of intruder detection.
• Carpal tunnel is guaranteed. Don't worry about it.
• A goat has all the “patches” it will ever need. If it doesn't it just means you're having goat steak for dinner.
• Goats that become full do an automatic “core dump” but they take care of getting themselves reset and on-line. You just have to clean up. You do not need to worry about defragmenting or compressing the goat. The goat does not have to be zipped, archived or converted to Goat-32.
• As long as the stable hasn't caught fire, a goat couldn't care less about a power surge.
• Goats don't have to be backed up at night.
• Each and every one of the parts of a goat use the same interrupt, and the goat works just fine anyway.
• A goat is a goat is a goat.
• You don't EVER restart a goat. You do shut them down sometimes and it's the first step in many of your recipes.
• Nobody ever needed to draft up a goat-milking requirements document.
• You deliver applications to goats. Goats do not deliver applications to you.
• A goat will do practically anything do get more comfortable. Computers have been known to display the same error message over and over again, all day, without regard to how frequently or how hard the monitor has been hit, slapped, punched or kicked.
• You don't have to log off of a goat and listen to some silly “Exit Goat” sound effect for the next several minutes.
• You won't find out from your next phone bill that you milked your goat too much for your budget.
• On a goat, the SYS$ERR.LOG file is ALWAYS EMPTY.
• Operating systems come & go, but goats will probably never be “orphaned” as they are expected to be produced by their manufacturer for quite some time to come.
• There are no workstation licensing issues with goats.
• You don't get in trouble for milking a goat during business hours, and nobody cares if you reformat it.
• If it's late and you have a lot of goat-milking to do, at least you can see your kids before they have to go to bed. You can probably even make them help you milk your goats.
• You don't need 32 megs of RAM to get started milking your goat.
• Goat security is applied completely, thoroughly, and with all the features you'll ever need, using a stake and a rope.
• Nobody ever got a general protection fault milking a goat.
• You don't need to worry about your whole goat herd locking up if you put an ethernet goat and a token-ring goat together in the same stable.
• You don't name goats. If you do name goats, you can give two or more goats the same name and this will not interfere with your ability to access any of the goats.
• Your kids will not meet some pervert who wants to buy them a bus ticket when they play with a goat.
• There is no closely-watched dispute between Microsoft and any competitor, over who will dominate the goat-milking product industry. You will probably never be asked to check-mark a box that says, Make this my default goat-milking bucket.
• You do not want, need, or desire in any way for goats to run at a higher clock speed. And they don't.
• You do not need to use a wrist strap to ground yourself before milking, and there's never a need to put your goat in a little plastic baggie. Unless making goat steak
• There really aren't too many ways to improperly shut down a goat.
• Surrounded by a room full of younger goat farmers, you don't need to worry about dating yourself talking about 300-baud or 4.7-Mhz goats.
• y2k.
• You do not need to buy anything to “uninstall” a goat. Maybe a gun or a knife.
• Once you've filled a bucket with goat milk, the goat can crash and it doesn't matter whether you've “saved” or not. Just don't spill.
• When you buy a new goat, the goat does not need to re-write registry keys on the farm that could have unforeseen effects on the other animals already residing there.
• There are no easter eggs in a goat.
• Your wife will never yell at you for removing all of the RAM from her goat.
• You never need to learn Goat 2000, Goat Perfect 8, or Goat 123
• You don't need an Internal IPX Address to boot a Goat.
• Goats don't need a per-bucket license.
• You will never spend 4 hours upgrading a goat over the wire.
• There is no Goat Ops.
• Goats follow upgrade procedures.
• Goats eat org charts.
• If a goat gets an uncleanable virus, you shoot it.
• If a goat has a non-terminal virus it just does the poo-poo.
• Goats don't need pagers and never get a 'please advise'.
• Goats don't have to worry about whether or not it's Calcomp.
• A goat farmer doesn't care if people can't remotely access his herd.
• No MHN Goat herd.
• No one gives a rat's ass if the goats aren't talking to each other.
• Ever heard of a proprietary goat?
• No goat analysis meetings.
• No goat control meetings.
• No meetings.
• Goats will never need service pack 4.
• No DS problems at GOATADRIVE.
• You fuck the goat, he doesn't fuck you and the whole department.
• A goat might bite you in the ass, but he won't fuck you.
• Fuck Y2K.
• Goats don't ever ask stupid questions.
• Goats don't drive technology dollars away from your automobile lusts.
• If a goat takes a “dump” in the middle of the night, you take care of it when you damn well feel like it.
• Nobody will fire you for connecting “diskless goats” into a “goat server” when they think you should have purchased a massive mainframe goat to connect to a multitude of inexpensive “dumb goats”.
• ISO is not publishing any standards about how you should be farming your goats.
• Counting from zero instead of one, doesn't apply to anything goat farmers do and looks stupid. Hexadecimal is unheard of.
• When you sell a goat, you don't need to export it to a format that will be understood by the buyer's ancient goat-reading software.
• All your stuff will still work when you buy your 100th goat, and your 256th goat, and your 65,536th goat…
• People don't walk up to goat farmers at parties and whine about how they just got a French Alpine and don't know how to milk it.
• Nobody can go through your goat and get you in trouble for what they find in there.
• You don't have to administer a “user acceptance test” when you deliver goat cheese.
• You don't need any special utilities to delete a goat that is not empty.
• You don't need or want goats on your desktop, or shortcuts to goats on your desktop. Most goat farmers don't have desktops.
• Nothing a goat farmer does requires a mouse. If you have mice you get a cat.
• Goat farmer error messages: Goat not found; Goat dead; Goat not awake; Too soon after last milking; Billy goat detected. That's about all. You don't need silly numbers for these, and you don't need to look them up anywhere or check them out at goat.com.
• There are no read-only oats. There are no hidden or system goats.
• You don't need to mail anyone a core dump from a goat to fix a problem. The only time you would do this is to CAUSE a problem.
• A goat that doesn't know what time it is will work just fine.
• A goat that is not Y2K compliant will simply think it's not Y2K. This is doesn't even require documentation.
• If your spouse doesn't authorize the purchase of a new goat, you simply encourage your goats to make one from existing parts.
• A goat doesn't have enough fingers to press <shift><Shift><Ctrl><Alt><Esc>
• Goats don't argue about it being another goats problem. They just kick each others ass.
• If a goat had to document every time it took a shit, we would be out of forests.
• Goats don't give a shit about email.
• The only way a goat can deliver an 'application' is through it's ass.
• Goats can't get there benefits revoked they are just made into goat steaks for dinner.
• A goat farmer doesn't have to provide documentation on his goat's ablility to produce milk after the year 2000.
• GoatEng.
• Macintosh goat users will not make fun of you because your goat is more problematic & complicated than the goat they just bought.
• Goat farmers who voted for Perot have pretty much the same type of goat as everyone else, so they can go back to arguing about politics like they were doing before 1984.