R12 logs--454178.1

OPMN errors at startup

Possible symptoms:

• A failure is noted in "$ADMIN_SCRIPTS_HOME/adstrtal.sh",
• Attempts to start the opmn using "$ADMIN_SCRIPTS_HOME/adopmnctl.sh" fail
• "$ADMIN_SCRIPTS_HOME/adopmnctl.sh status" fails

Useful log files:

Examine opmn.log, opmn.out

HTTP Server fails at startup

Possible symptoms:

• "$ADMIN_SCRIPTS_HOME/adopmnctl.sh status" indicates the HTTP Server is down
• "$ADMIN_SCRIPTS_HOME/adapcctl.sh start" fails

Useful log files:

Examine HTTP_Server~<#>, error_log

OC4J Applications Module (oacore, forms, oafm) fails at startup

Possible symptoms:

• Executing "$ADMIN_SCRIPTS_HOME/adopmnctl.sh status" indicates the module is down

Note:The forms module will show as down, if forms are configured to run in socket mode (this is not an error)

• Script adctl.sh start fails
• Example: "$ADMIN_SCRIPTS_HOME/adoacorectl.sh start"

Useful log files:

Examine: OC4J~~default_group~<#>, std.out, std.err

HTML requests complete, Java requests do not

• "$ADMIN_SCRIPTS_HOME/adopmnctl.sh status" shows the oacore module(s) alive
• http:///OA_HTML/jsp/fnd/aoljtest.jsp returns no page

Useful log files:

Examine: OC4J~~default_group~<#>, std.out, std.err, application.log

Additional debugging steps:

• set s_jsp_dbg_mode in $CONTEXT_FILE to true
• Run Autoconfig $ADMIN_SCRIPTS_HOME/adautocfg.sh
• Stop then Start $ADMIN_SCRIPTS_HOME/adoacorectl.sh service
• Examine application.log for missing jsp files (see Known Issues)

HTML requests complete, Java requests complete, Applications Login page fails

Useful log files:

Examine: application.log , $APPLRGF/javacache.log (see Known Issues)

Additional debugging steps:

If the Applications Login page is returned or an error message such as: "You have encountered an unexpected error. Please contact the System Administrator for assistance" then the problem is with Oracle Applications setup rather than Oracle Application Server.

• Examine the output from aoljtest.jsp
• http:///OA_HTML/jsp/fnd/aoljtest.jsp
• Enable AFLogging
• Open the file $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/oc4j.properties in a text editor
• Add:the following values:
• AFLOG_ENABLED=true
• AFLOG_LEVEL=statement
• AFLOG_MODULE=% (% captures all - preferable to use products such as fnd, ies, por)
• AFLOG_FILENAME=/tmp/1.log
• Stop then restart $ADMIN_SCRIPTS_HOME/adoacorectl.sh service, reproduce the issue, and examine the log file denoted by AFLOG_FILENAME

Applications Login Page is displayed, Login Fails

Debugging steps:

• Examine the output from aoljtest.jsp

• http:///OA_HTML/jsp/fnd/aoljtest.jsp
• Enable AFLogging
• Edit $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/oc4j.properties
• Add: values
• AFLOG_ENABLED=true
• AFLOG_LEVEL=statement
• AFLOG_MODULE=% (% captures all, better to use products i.e fnd,ies,por,etc..)
• AFLOG_FILENAME=/tmp/1.log
• Stop then Start $ADMIN_SCRIPTS_HOME/adoacorectl.sh service, recreate issue, examine log file denoted in AFLOG_FILENAME..

Return to Top

Oracle Application Server Diagnostic Tools

The tools, techniques, and examples shown below should be used in conjunction with the supporting documentation, which gives more detail of the options available. If in doubt how to proceed, post your questions in an OracleMetalink Forum or open a service request with Oracle Support.

Recommendation: Do not enable debug settings indiscriminately. Standard logging should be used first, to determine which of the tools below is most suited to help uncover the source of a particular problem.

OPMN

The OPMN server provides a means to enhance debug logging during runtime if needed. Higher logging levels may be set for the ons (Oracle Notification Server) or the pm (Process Manager).
Example: Enabling debug values for OPMN in a UNIX environment at runtime. The log values will be disabled upon restart.

prompt> . $INST_TOP/ora/10.1.3/.env
prompt> opmnctl query target=log
prompt> internal;ons;pm -------------(default Applications response)
prompt> opmnctl set target=debug comp="ons[all];pm[all]"

A full list of query options is available in Oracle® Process Manager and Notification Server Administrator's Guide 10g Release 3 (10.1.3), Chapter 6
Log file: opmn.dbg

Recommendation: Enable usage of the opmn.dbg file only after checking with Oracle Support. The opmn.dbg file is used by Support to debug and diagnose OPMN issues. Messages that are contained in the opmn.dbg file are typically not readily comprehensible to the user.

Return to Top

Java Object Cache Monitor

In distributed mode (the Applications Release 12 default mode), the Java Object Cache can share objects and communicate with other caches running either locally on the same system or remotely across the network. Object updates and invalidations are propagated between communicating caches. Distributed mode supports object persistence across system shutdowns and program failures. It may become necessary to review information in regards to the controller and objects cached. Oracle Application Server provides a cache monitoring utility available at the operating system level for Applications Release 12.
Example: Initializing CacheWatchUtil in a UNIX environment

prompt >. ./$APPL_TOP/APPS(sid).env
prompt >java -jar $ORACLE_HOME/../10.1.3/javacache/lib/cache.jar watch \
> java -jar $ORACLE_HOME/../10.1.3/javacache/lib/cache.jar watch -config=$ORA_CONFIG_HOME/10.1.3/javacache/admin/javacache.xml
cache >

An explanation of values and options may be found in Oracle® Containers for J2EE Services Guide 10g Release 3 (10.1.3):Chapter 7 Java Object Cache:Monitoring and Debugging
Return to Top

Applications Server Forms Servlet Debug

Applications Release 12 provide Forms Servlet architecture by default through Rapid Install. This information is specific to the servlet itself: issues with Forms and forms architecture should be addressed using Forms Tracing and JVM Client Tracing

• To test the j2ee Forms Module configuration: http://yourserver/forms
• To debug runtime issues within the Forms Servlet:
• Edit the profile option ICX:Forms Launcher (user level)
• Values available: [reference]
• http://yourserver/forms/frmservlet?serverURL=/forms/lservlet/session --- Log messages are written whenever a Forms session starts or ends. These give the host name and IP address of the client (the computer on which the user's web browser is running), the runtime process id, and a unique internal session id number.
• http://yourserver/forms/frmservlet?serverURL=/forms/lservlet/sessionperf --- Performance summary statistics are included with the session end message.
• http://yourserver/forms/frmservlet?serverURL=/forms/lservlet/perf --- A performance message is written for every request from the client.
• http://yourserver/forms/frmservlet/debug?serverURL=/forms/lservlet/debug --- Full debug messages. Other debug messages are written in addition to the messages mentioned above. This logging level is very verbose and is intended mainly for debugging and support purposes.

Log file: application.log
Forms Servlet Documentation: Oracle® Application Server Forms Services Deployment Guide 10g Release 2 (10.1.2) Chapter 8
Return to Top

Class Loaders Debugging

To provide a generic summary of Class Loading as it applies to Oracle Applications Release 12, the following paragraph has been extracted from Oracle® Containers for J2EE Developer's Guide 10g Release 3, and paraphrased appropriately.
Standard classloaders are linked together in a parent-child hierarchy, with each classloader having an associated parent classloader. This hierarchy represents a tree structure, in which a child classloader imports a set of classloaders from its parent loader. In the OC4J context, all J2EE Applications Modules running within an OC4J instance are children of the system application. These Web Modules use multiple class loaders in place of a single system loader. Each deployed application (OACore, OAFM, Forms) has at least one loader, separate from all other applications; this means that each also has a distinct class-path. For each class lookup, the search begins at a specific loader level, and can normally only visit loaders "above" it. This means that a search starting at the Applications Web Module loader level would have a very different class-path than one starting at the system loader level.
Detailed information on on Class Loading within the J2EE Module may be found in Chapter 3 of Oracle® Containers for J2EE Developer's Guide 10g Release 3 (10.1.3).
Debugging Class Loaders in Applications Release 12 requires manual editing of the relevant files. The following example illustrates how to enable debug values for the Applications Module oacore in opmn.xml:

 ---------- excerpt ----------





data id="java-options" value="-server -verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=160M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Dclass.load.log.level=finest -Dclass.load.trace=loader-create -Dclass.load.log.file=/oracle/test.log -Doracle.security.jazn.config=/oracle/inst/apps/VIS_treebeard/ora/10.1.3/j2ee/oacore/config/jazn.xml"/>
---------- end excerpt ----------

-Dclass.load.log.level=

• For debugging information the level should be set at "all" or below default
• Available values
• all - Output all log messages.
• severe - Output messages at the SEVERE level only.
• warning - Output messages at the WARNING level or above.
• info - Output messages at the INFO level or above. Note that if the log level is set below "info" - for example, to "finer", then trace output will be filtered from the logger output.
• config - Output messages at the CONFIG level or above. This is the default log level.
• fine - Output messages at the FINE level or above.
• finer - Output messages at the FINER level or above.
• finest - Output messages at the FINEST level or above.
• off - Suppress all logging messages.

-Dclass.load.trace=[[:[,]] | [~]]
There are many variations of values that may be used, depending upon the circumstances and issue. Further details on available events, may be found at: in Oracle® Containers for J2EE Developer's Guide 10g Release 3 (10.1.3): Chapter 3 Utilizing the OC4J Class Loading Framework: Table 3-13 class.load.trace System Property Values
Return to Top

Log Files

Log files provide an insight into the configuration and health of the Oracle Application Server. The tables below list the log file locations and provide a short description of the type of information recorded in each file.
Oracle Applications Release 12 allows application tiers to be shared. As part of this strategy, log files are in a central location for the specific Applications instance. Oracle Application Server log files referred to in this article are located on the application tier under $LOG_HOME/ora/10.1.3. The $LOG_HOME environment variable is defined in the Applications context file (which is pointed to by the environment variable $CONTEXT_FILE).

Note: It is not advisable to access files and alter logging levels that are not defined in the $CONTEXT_FILE (and therefore controlled through AutoConfig), unless requested by Oracle Support or Development for debugging purposes. If direct access is required to a file variable that is not editable, an enhancement request should be logged outlining the requirement and business need.

OPMN

Log File	Location	Description	Associated File and values	$CONTEXT_FILE Variable
opmn.log	opmn	Contains messages useful for monitoring the operations of the OPMN server. Output written to the opmn.log file contains the exit status of a child OPMN process. A status code of 4 indicates a normal reload of OPMN.	opmn.xml	none
opmn.dbg	opmn	Capture debug messages for Notification Server and Process Manager.	opmn.xml	none
opmn.out	opmn	Contains the standard output (stdout) and standard error (stderr) logs of OPMN. Also referred to as the OPMN "console log". After a certain point in OPMN initialization, nothing else will be written to this file.	none	none
OC4J~~default_group~<#> example: OC4J~forms~default_group~1	opmn	Process Control log file Standard out/err for j2ee Components (shows GC size and timing)	none	none
HTTP_Server~<#>	opmn	Process Control log file Standard out/err for Component - http	none	none
std.err example: oacorestd.err	opmn/ _default_group_<#> example: opmn/oacore_default_group_1	Records standard out information from the OC4J runtime process designated System.out	opmn.xml	none
std.out example: oacorestd.out	opmn/ _default_group_<#r> example: opmn/oacore_default_group_1	records standard error output from the OC4J runtime process designated System.err	opmn.xml	none

Return to Top

HTTP Server

Log File	Location	Description	Associated File and Values	$CONTEXT_FILE variable
access_log.xxxxx	Apache/logs	Records all requests processed by the server.	No debug levels available	N/A
error_log.xxx		Contains diagnostic information, record any errors encountered when processing requests. It is the first place to look when a problem occurs with starting the Application server or with operation of the Application server, since it will often contain details of what went wrong.	httpd.conf Directive: Variable: LogLevel Values: debug, info, notice, warn, error, crit, alert, emerg.	s_apache_loglevel
mod_rewrite.log		Records server rewrite actions.	httpd.conf Directive: mod_rewrite Variable:RewriteLogLevel Values: 0-9	None: default value 0
sec_audit.log		Used for auditing connections/requests.	security.conf Directive: mod_sec Variable:SecAuditEnginel Values: on, off, relevant	None: default value off
sec_debug.log		Debug security filter issues.	security.conf Directive:mod_sec Variable:SecFilterDebugLevel values: 0-9	None: default value 0
ssl_engine.log		SSL connection requests/negotiation.	ssl.conf Directive: mod_ssl Variable: SSLLogLevel values: none, error, warn, info, trace, debug	s_apache_loglevel

Return to Top

J2EE Applications Modules

Two types of log files can be generated by OC4J:

• Plain text log files are the default log files used for OC4J components, and are ideal for use in a development environment. The messages logged in these text files can be read with any editor, including the Oracle Enterprise Manager 10g Application Server Control Console
• Oracle Diagnostic Logging (ODL) log files use an XML format that is accessible via Oracle Applications Manager (requires OracleAS 10.1.3.1)

Note: ODL logging is not currently enabled or configurable using AutoConfig.

Log File	Location	Description	Associated File and Values	$CONTEXT_FILE variable
application.log	j2ee//_default_group_# example: j2ee/oacore/oacore_default_group_1	Text log file -- records actions specific to the application examples: servlet and jsp initializations.	system-application.xml	none
		When true adds stack trace for certain runtime exceptions, shows full path from missing jsp files.	orion-web.xml	s_jsp_dbg_mode default: false
global-application.log	j2ee//_default_group_# example: j2ee/oacore/oacore_default_group_1	Text Log File - records information related to all deployed J2EE Application Components start/stop	application.xml	none
system-application.log		Text Log File - records internal system J2EE application initialization.	system-application.xml	none
server.log		Text Log File - records Notification server (ons) actions.	server.xml	none
jms.log		Text Log File - records JMS (Java Messaging Service) actions for distributed environments.	jms.xml	none
rmi.log		Text Log File - records RMI (Remote Method Invocation) actions for distributed environments.	rmi.xml	none
log.xml		ODL Logging	j2ee-logging.xml	By default, currently commented out in configuration files.
			system-application.xml application.xml server.xml jms.xml rmi.xml

Return to Top

Known Issues

JOC fails to initialize

Errors:
Browser:
500 Internal ServerError
java.lang.NoClassDefFoundError at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:120) at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:161)
Application.log:
html: Error initializing servlet
java.lang.NoClassDefFoundError
at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:120)
at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:161)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2231)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4617)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4541)
The issue above has been observed both intermittently and at startup. In both cases the issue is with the port value for the java object cache. Either a port conflict at startup or multiple instances with JOC in distributed mode attempting to use the same port resulting in intermittent errors.
Solution:
Choose another value for $CONTEXT_FILE variable: s_java_object_cache_port

Compiled JSP files are removed or corrupted

In Applications Release 12, class files for .jsp's are not compiled at runtime for performance reasons. If the files are removed, directories may be changed or files may become corrupted
Example Error in Debug Mode:
Missing class: _OA
Missing class: _AppsLocalLogin
Missing class: _OAInfo
Missing class: _jsp._fnd._aoljtest
javax.servlet.ServletException: oracle.classloader.util.AnnotatedClassNotFoundException:
Solution:
Compile associated class files using ojspCompile.pl:
$FND_TOP/patch/115/bin/ojspCompile.pl ojspCompile.conf --compile --flush -p 20
OracleMetaLink Note 215268.1 contains complete syntax and definition

JOC fails in distributed mode when registering group members on AIX

This problem may be noted after a restart of the oacore process.
Error noted in $APPLRGF/javacache.log:
Cache initialization failed: reason
java.lang.ClassCastException: java.lang.Long incompatible with
oracle.ias.cache.group.Address
at oracle.ias.cache.group.Address.equals(Unknown Source)
at java.util.Hashtable.contains(Hashtable.java:347)
Error noted in application.log:
IAS Cache initialization failed. The Distributed Caching System failed to initialize on port: xxxxxx.The list of hosts in the distributed caching system is: Yourmachine.domain.com. The port xxxxx should be free on each host running the JVM's. The default port xxxxx can be overridden using -Doracle.apps.jtf.cache.IASCacheProvidercacheProvider.port=
Error noted in browser:
500 Internal Server Error

java.lang.NoClassDefFoundError: oracle.apps.fnd.profiles.Profiles (initialization failure)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:123)
at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:120)
at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:161)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2231)
Solution:
Apply patches 5261515 and 5946958
Bug 5968938 has been filed to include 5946958 in OracleAS 10.1.3.4

OPMN-controlled service will not start (Apache/OAcore/Any)

Error in std.err file:
Error initializing server: Error initializing ejb-modules: Resource exception(OracleASjms) for MessageDrivenBean event during endpoint activation: failure looking up ConnectionFactoryJndiName:jms/XAQueueConnectionFactory: javax.naming.NameNotFoundException: jms/XAQueueConnectionFactory not found
Solution:
Remove any *.lock files in $ORA_CONFIG_HOME/10.1.3/j2ee//persistence//
This issue is commonly found when the operating system is shut down without shutting down the Oracle Application Server, or where the instance is a clone made from a source where OracleAS was running.