How to troubleshoot Concurrent Manager issue ? (Oracle Apps )

How to troubleshoot Concurrent Manager issue ? 

Check Concurrent Manager log $SID_[MMDD] where MMDD is Concurrent Manager start Month & Date
 1. $APPLCSF/$APPLLOG/$SID_[MMDD].mgr
Could NOT contact Service Manager FNDSM_INNOWAVE03_visr12.

The TNS alias could NOT be located, the listener process onINNOWAVE03 could NOT be contacted,

OR the listener failed TO spawn the Service Manager process.

  Process monitor session started : 24-SEP-2009 20:31:07Could NOT contact Service Manager FNDSM_INNOWAVE03_visr12.

The TNS alias could NOT be located, the listener process onINNOWAVE03 could NOT be contacted,

OR the listener failed TO spawn the Service Manager process.
2. Error message is pointing towards “FNDSM_INNOWAVE03_visr12″ so run tnsping to check status of TNS alias like
[avisr12@innowave03 ~]$ tnsping FNDSM_INNOWAVE03_visr12
Output like
TNS Ping Utility for Linux: Version 10.1.0.5.0 -

Production on 24-SEP-2009 20:36:12Copyright (c) 1997, 2003, Oracle. 

All rights reserved.Used parameter files:Used TNSNAMES adapter to resolve the alias

Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=innowave03.com)(PORT=1638)) (CONNECT_DATA=(SID=FNDSM)))

TNS-12541: TNS:no listener
3. Check Apps Listener status like
[avisr12@innowave03 ~]$ lsnrctl status APPS_visr12
LSNRCTL for Linux: Version 10.1.0.5.0 -

Production on 24-SEP-2009 20:37:36Copyright (c) 1991, 2004, Oracle. 

All rights reserved.Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=innowave03.com)(Port=1638))

TNS-12541: TNS:no listener 

TNS-12560: TNS:protocol adapter error 

TNS-00511: No listener  

Linux Error: 111: Connection refused
4. Start Apps Listener
[avisr12@innowave03 scripts]$ ./adalnctl.sh start
output like
adalnctl.sh version 120.3

Checking for FNDFS executable.

Starting listener process APPS_visr12.

adalnctl.sh: exiting with status 0adalnctl.sh: check the logfile /oracle/ apps/ r12/ visr12/ inst/ apps/ visr12_innowave03/ logs/ appl/ admin/ log/ adalnctl.txt formore information ...
5. Try again tnsping again
[avisr12@innowave03 scripts]$ tnsping FNDSM_INNOWAVE03_visr12
output like
TNS Ping Utility for Linux: Version 10.1.0.5.0 -

Production on 24-SEP-2009 20:39:09Copyright (c) 1997, 2003, Oracle. 

All rights reserved.Used parameter files:Used TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=innowave03.com)(PORT=1638)) (CONNECT_DATA=(SID=FNDSM)))

OK (0 msec)
6. Restart CM again

Issues related to startup/shutdown of CM
If you try to start CM using adcmctl.sh or adstrtall.sh script & you encounter problem in starting CM then check for log files.

All information related to why Concurrent Manager not coming up is recorder in log file at $APPLCSF/$APPLLOG with default name as $SID_$DD ( where DD is date on which you started Concurrent Manager ). This Concurrent Manager naming convention will be listed in file adcmctl.sh file (Concurrent Manager startup file). If $APPLCSF is not set then you can look for this file in $FND_TOP$APPLLOG
ICM is Up but some managers are still down

As you know adcmctl.sh script starts ICM (Internal Concurrent Manager) and its duty of ICM to start other Managers. If you see ICM is up with some other managers & Issue is with only Particular Managers then look for log files of these managers .
Concurrent -> Manager -> Administer -> Select manager which is not starting -> Click on “Processes” -> Click on “Manager Log” to find out error message.

These individual manager log file, you can access under $APPLCSF/$APPLLOG directory as well.
If your error message is not enough to find out cause of problem then you can enable debug on CM log files . You can enable diagnostics by changing DIAG_PARAM=”" to DIAG_PARAM=”Y” in adcmctl.sh (CM Startup script file)
Other things you can try for CM startup Issues :
Try to bounce CM & see all FNDLIBR processes are cleared before restrarting CM.
You can run cmclean.sql command to clear any Invalid State against any request or manager. Check for any messages in log file.
Check status of Manager from administer Concurrent Manager Window.
Concurrent -> Manager -> Administer (Login as System Administrator Responsibility)