Unix Script Security

I have received a lot of questions about this area recently. Here are a couple of tips and tricks I use to provide security within shell scripts.

Note: The syntax I will use is for Bourne type shells. The same concepts apply to csh, but the syntax changes.

The first scripting security issue for DBAs that we need to address is the problem of passwords in scripts. Many scripts we write as DBAs, especially applications DBAs, need to connect to the Oracle database. Unfortunately, these scripts frequently contain hardcoded passwords. This has two major problems, security and maintenance. Fortunately, the solution can be simple. In a secure location, have the system administrators create a file belonging to root in the dba (or oinstall) group with permissions 660 (owner read/write, group read/write, everyone else no access). For example, /etc/sysconfig/SID is a good choice in Linux. In this file, put your passwords.

/etc/sysconfig/SID:

SYSTEM=manager
APPS=apps

Your scripts can source this file as part of the configuration at the start of the script. Then anywhere in the script that you need to use the password, you use the variable. In other words instead of sqlplus apps/apps, you use sqlplus apps/$APPS. Of course, that would still show the password with a ps command. An even better solution is to use i/o redirection inside the script:

#!/bin/bash
. /u01/app/oracle/db/tech_st/10.2.0/VIS_vis12.env
if [ -f /etc/sysconfig/$ORACLE_SID ]; then
  . /etc/sysconfig/$ORACLE_SID
else
  APPS=apps
fi
sqlplus /nologin <<EOF
connect apps/$APPS
select * from dual;
EOF

After making this change,  no one who sees the script will learn the apps password,  no one who happens to issue a ps command while the script is running will learn the password, and the act of changing the password in /etc/sysconfig/SID will update all of your scripts.

Now that we have solved the major security issue within our own scripts, let’s address scripts that need to be run as root. In a segregated environment, that is one where the DBAs do not have root, sudo can be a useful tool to allow this functionality.

For example to allow the oracle user to run root.sh from $ORACLE_HOME, you could make the following entry in /etc/sudoers (use visudo to edit this file):

oracle ALL = NOPASSWD: /u01/app/oracle/db/tech_st/10.2.0/root.sh

Of course, this is is now a major security violation.  Since this script is in a directory owned by oracle, anyone who can connect to the system as oracle, can run any command they want as root simply by editing root.sh.  Obviously, the DBA team will continue to have to call the system administrators to have root.sh run when needed.  However, there are other commands that it may be more convenient to have the dba team able to run.  Examples of this may include viewing the system log, mounting and unmounting a backup or stage disk, or if cloning is done using some form of snapshot or mirroring on a SAN or NAS device, merging and splitting the mirrors.  One way to allow this is to have the system administration team write scripts to accomplish these functions and place them in a directory owned by root.  You can take the security a step further and make the directory only accessible to root.

For example (as root):

mkdir /usr/local/superuser_scripts
chown 700 /usr/local/superuser_scripts

Only root can access this directory.  If we want to allow the DBA team access to view the last 100 entries in /var/log/messages, we create a script tail100msg in /usr/local/superuser_scripts:

#!/bin/bash
tail -100 /var/log/messages

In /etc/sudoers, the following line is added.

oracle ALL = NOPASSWD: /usr/local/superuser_scripts/tail100msg

If the oracle user, gives the command: sudo /usr/local/superuser_scripts/tail100msg, they will see the last 100 lines of /var/log/messages.  However, they have to know the script exists, because oracle cannot do an ls in /usr/local/superuser_scripts to see what is there.

You have to be extremely careful if you allow parameters to any scripts written for this purpose.  You have to follow all the same protections against attacks through substitution variables that are followed in sql scripts.

Do you have any other techniques you use to improve the security of the scripts you use?

Too Many Open Files

I had an interesting problem happen with a customer late last week. A monitor we maintain on the alert log suddenly paged us with ORA-1116, ORA-1110 and ORA-27041 (Can’t open a database file, too many open files). Nothing had changed on this system in months.

After running: lsof | awk ‘{print $9}’ | sort | uniq -c | sort -n | tail -5 (as with all code blocks, your milage may vary, this was in the bash shell on Redhat Linux), the problem turned out to be with $ORACLE_HOME/nls/data/9idata/lx40030.nlb. A little poking aroud on Metalink led us to bug 5257698. This is a Generic bug for 10.2. The patch replaces lx40030.nlb and lx40003.nlb in $ORACLE_HOME/nls/data/old. Rerunning cr9idata.pl appears to have fixed the open files issue.

The worrisome part of this issue is that they had been live on 10.2 with E-Business Suite 11.5.10.2 for months and no patches or other changes had been made which would have triggered this that we could find. In spite of this, we had to reboot Thursday and were right on the edge of having to shutdown production again on Friday. Luckily, we were able to apply the patch this weekend.

If you are using 10.2 especially with the E-Business Suite, please make sure that you have some form of simple open file monitor running (lsof | wc -l if nothing else).

We discovered that the number of files was flucuating somewhat, but it was basically increasing (this was after a reboot–the problem all started when we got a page from the alert.log that some datafiles apppeared to be missing).

Here is a simple script that can be used to do the monitoring (just put it in cron):

#!/bin/bash
LSOF=/usr/sbin/lsof
MAIL=/bin/mail
WC=/usr/bin/wc
ALERT="root pager1@whatever.com"
THRESHOLD=25000
SUBJ="SID NAME"
COUNT=$($LSOF | $WC -l)
echo "There are $COUNT open files"
if [ "$COUNT" -gt "$THRESHOLD" ]; then
   $MAIL -s "$SUBJ $COUNT open files" $ALERT <<EOF

On $(date), the number of
open files on $(hostname)
was $COUNT.  This is
greater than the alerting threshold
of $THRESHOLD
.
EOF
fi