R12.2, Service Manager Will Not Start

Recently, I was working with a client when the managers in their 12.2 environment quit working. We had shutdown the applications tier to reboot the server, and when it came back up, only the Internal Manager (ICM) would run. After working through all the usual steps, relinking, autoconfig, cmclean, answering the same questions from Oracle repeatedly, etc., we still could not get the managers to start. The Administer Concurrent Managers form showed the Internal Manager running with a matching target (1), and the Service Manager running with a target of 0. This was the state every time we started the managers even after rebooting both the database and applications servers. To make things even stranger, if you viewed the processes under the Service Manager from the Administer Concurrent Managers form, all the processes were Terminated or Deactivated and no new entries were being made in spite of the parent form showing that one was running (If the managers were shutdown, the running count went to 0). Even starting the concurrent managers with DIAG=Y did not add any useful information to the logs for this issue.

I reached out to one of my contacts through OAUG (one of the major reasons to be active in user groups is to develop quality contacts), Michael Barone, to see if he could think of something I had not tried. One of his suggestions was to run FND_CONC_CLONE.SETUP_CLEAN and then rerun AutoConfig. This did not solve the problem, but it did cause a new message in the ICM log, we now had a message that the apps tier could not ping. We tried ping from the command line and got the error, ping: imp open socket: operation not permitted. As a result of this, we discovered that /bin/ping had permissions and ownership:
-rwxrwxr-x root root
instead of the expected:
-rwsrwxr-x root root

As soon as we issued chmod u+s /bin/ping, the service manager (and therefore all the other managers) started. This occurred on a Redhat 6.4 server, and we do not know why the permissions changed on ping between the previous time we started the managers and the reboot. I also am not clear why the ping error did not show up every time we attempted to start the managers (we did find it in the first ICM log after the reboot, but it was not in any subsequent logs until after running doing the setup clean and autoconfig).

If you have an issue with the concurrent managers not starting, check the permissions, verify that you can run ping as the applications owner.

RSS to e-mail and Oracle Linux Public Yum

I found a very useful service to help keep up with information that is released on the various Oracle blogs. Blogtrottr.com allows you to subscribe to feeds and have them sent to your e-mail. This can be set up realtime on in digest mode (at 2 hour, 4 hour, 6 hour, 8 hour, 12 hour, or daily intervals).

I stumbled upon blogtrottr.com when I realized that I had missed a blog announcement that Oracle Linux patches were now available on Oracle’s public yum site. This is a major announcement for companies considering what flavor of Linux to use. Many companies debate which flavor or Linux to use between Oracle and Redhat (and others, but those seem to be the big two for companies that are using Oracle products). The fact that you can now set up development systems without any cost for getting the linux patches and still run the same version of linux that you are paying for support on in production is another big plus for Oracle Enterprise Linux. There are a couple of downsides to using the public yum vs. paying the $119/year for network support that companies still need to consider. For more information, check out the blog entry on the Oracle Linux Blog.

For an excellent overview of the value proposition is using Oracle Linux, check out this entry on Wim Coekaerts Blog.

Unix Script Security

I have received a lot of questions about this area recently. Here are a couple of tips and tricks I use to provide security within shell scripts.

Note: The syntax I will use is for Bourne type shells. The same concepts apply to csh, but the syntax changes.

The first scripting security issue for DBAs that we need to address is the problem of passwords in scripts. Many scripts we write as DBAs, especially applications DBAs, need to connect to the Oracle database. Unfortunately, these scripts frequently contain hardcoded passwords. This has two major problems, security and maintenance. Fortunately, the solution can be simple. In a secure location, have the system administrators create a file belonging to root in the dba (or oinstall) group with permissions 660 (owner read/write, group read/write, everyone else no access). For example, /etc/sysconfig/SID is a good choice in Linux. In this file, put your passwords.

/etc/sysconfig/SID:

SYSTEM=manager
APPS=apps

Your scripts can source this file as part of the configuration at the start of the script. Then anywhere in the script that you need to use the password, you use the variable. In other words instead of sqlplus apps/apps, you use sqlplus apps/$APPS. Of course, that would still show the password with a ps command. An even better solution is to use i/o redirection inside the script:

#!/bin/bash
. /u01/app/oracle/db/tech_st/10.2.0/VIS_vis12.env
if [ -f /etc/sysconfig/$ORACLE_SID ]; then
  . /etc/sysconfig/$ORACLE_SID
else
  APPS=apps
fi
sqlplus /nologin <<EOF
connect apps/$APPS
select * from dual;
EOF

After making this change,  no one who sees the script will learn the apps password,  no one who happens to issue a ps command while the script is running will learn the password, and the act of changing the password in /etc/sysconfig/SID will update all of your scripts.

Now that we have solved the major security issue within our own scripts, let’s address scripts that need to be run as root. In a segregated environment, that is one where the DBAs do not have root, sudo can be a useful tool to allow this functionality.

For example to allow the oracle user to run root.sh from $ORACLE_HOME, you could make the following entry in /etc/sudoers (use visudo to edit this file):

oracle ALL = NOPASSWD: /u01/app/oracle/db/tech_st/10.2.0/root.sh

Of course, this is is now a major security violation.  Since this script is in a directory owned by oracle, anyone who can connect to the system as oracle, can run any command they want as root simply by editing root.sh.  Obviously, the DBA team will continue to have to call the system administrators to have root.sh run when needed.  However, there are other commands that it may be more convenient to have the dba team able to run.  Examples of this may include viewing the system log, mounting and unmounting a backup or stage disk, or if cloning is done using some form of snapshot or mirroring on a SAN or NAS device, merging and splitting the mirrors.  One way to allow this is to have the system administration team write scripts to accomplish these functions and place them in a directory owned by root.  You can take the security a step further and make the directory only accessible to root.

For example (as root):

mkdir /usr/local/superuser_scripts
chown 700 /usr/local/superuser_scripts

Only root can access this directory.  If we want to allow the DBA team access to view the last 100 entries in /var/log/messages, we create a script tail100msg in /usr/local/superuser_scripts:

#!/bin/bash
tail -100 /var/log/messages

In /etc/sudoers, the following line is added.

oracle ALL = NOPASSWD: /usr/local/superuser_scripts/tail100msg

If the oracle user, gives the command: sudo /usr/local/superuser_scripts/tail100msg, they will see the last 100 lines of /var/log/messages.  However, they have to know the script exists, because oracle cannot do an ls in /usr/local/superuser_scripts to see what is there.

You have to be extremely careful if you allow parameters to any scripts written for this purpose.  You have to follow all the same protections against attacks through substitution variables that are followed in sql scripts.

Do you have any other techniques you use to improve the security of the scripts you use?

Printer setup for the E-Business Suite in Oracle Enterprise Linux Part 2 Migrating

Now that we have the printers all added, the last thing we want to do is have to rekey them all on every server.  Every time you do a manual migration, you increase the chance for errors.

We can use system-config-printer to migrate our queues at the linux level and FNDLOAD to migrate the queues in the E-Business Suite.  However, there is one manual step you must go through if you added any PPD files.

The first step in our process is to download the printers.

For linux (as root), enter the command:

system-config-printer-tui –Xexport > printers.xml

For the E-Business Suite (as oracle with the environment set for your apps tier), enter the command:

FNDLOAD apps/$APPS 0 Y DOWNLOAD $FND_TOP/patch/115/import/afcppinf.lct source_printer_def.ldt FND_PRINTER

FNDLOAD apps/apps_password 0 Y DOWNLOAD $FND_TOP/patch/115/import/afcppinf.lct source_printer_def.ldt FND_PRINTER

If you read the first part of this post on adding custom printers, you will recall that I said to keep track of the order that you added the PPD files.  You need to repeat the load of the PPD files in the same order on every system to which you are migrating the printers.  If you did not record the order, look in printers.xml for the printer_id tags, the custom ppd entries will be ppd#, e.g. ppd2.  The surrounding tag will be the name of the queue, so you should be able to reconstruct the order to add the PPD files.
On each system to which you wish to migrate these queues,

copy printers.xml and source_printer_def.ldt

For linux (as root):

system-config-printer-tui ==Xexport > backup-printers.xml
system-config-printer-tui –Ximport < printers.xml
service cups restart

Note: You are interrupting print services, make sure that you are doing it an appropriate time.
After you have completed this step, the queues will be the same on both systems.

For the E-Business Suite (as oracle with the environment set for the apps tier):

FNDLOAD apps/apps_password 0 Y DOWNLOAD $FND_TOP/patch/115/import/afcppinf.lct ${TWO_TASK}_printer_def.ldt FND_PRINTER

FNDLOAD apps/apps_password 0 Y UPLOAD $FND_TOP/patch/115/import/afcppinf.lct source_printer_def.ldt FND_PRINTER

This will merge the printers from the source to the new system.  Any printers that already exist on the new system should still be there, but you may have wiped out the linux queue in the previous step.