This page describes various techniques for troubleshooting and for collecting the necessary data for Customer Support.
Component set up verification
The following sections describe how to identify set up issues.
Verify the CICS code module group and resources are installed correctly
From the CICS Region, look for these messages to validate the CICS resources have been defined:
CICSAPPL Install for group XXXX has completed successfully.
CICSAPPL OWNER CSSY Resource definition for ZDTAGT72 has been added.
CICSAPPL OWNER CSSY Resource definition for ZDTDC2 has been added.
CICSAPPL OWNER CSSY Resource definition for ZDTDC2A has been added.
CICSAPPL OWNER CSSY Resource definition for ZDTPLT has been added.
CICSAPPL OWNER CSSY Resource definition for ZDTPLTSD has been added.
CICSAPPL OWNER CSSY Resource definition for ZDTSOAPH has been added.
CICSAPPL OWNER CSSY TRANSACTION definition entry for DTAX has been added.
Verify the CICS code module PLT program is invoked
The CICS code module doesn't log its messages to the JESMSGLG DD.
Messages are logged to the zRemote log, accessible from within DESK, in the same manner as all the other code module log files. Look for log entries similar to the following:
2019-05-09 20:19:11.789 UTC [d37f9842] info [native] Registering a pgi for the job: HVBAC021, host=10.30.220.41, groupId= f39f4801966aa7c7, pgir.groupInstanceID= fad6dee63cfd1522, hostID= 95c0bb0371704b8c, nodeID= fad6dee63cfd1522, groupName=HVBAC021, hostGroup=, processGroupType= 28
2019-05-09 20:19:11.789 UTC [d37f9842] info [native] Registered SubAgent[C021,51,32aa8d038887d1c9] with zDC[Z021,52], rc=true
2019-05-09 20:19:11.789 UTC [d37f9842] info [native] ASID[51], smfID[S0W1], sysid[C021], jobName[HVBAC021], subagentid[32aa8d038887d1c9] snaId[NETD .HVBAC021], CICS release 54 was successfully registered with zdc[52] using protocol version=7.2.0, allocator=pooled.
2019-05-09 20:19:13.789 UTC [d37f9842] info [native] ASID[52], smfID[S0W1], sysid[Z021], jobName[AFVBZ021] - ZDC955I - DESK connection being processed ZDC-Job/ID:AFVBZ021/Z021.
2019-05-09 20:19:13.790 UTC [d37f9842] info [native] ASID[51], smfID[S0W1], sysid[C021], jobName[HVBAC021] - ZDTP008I - ZDTP008I ZDTAGT71.
2019-05-09 20:19:13.790 UTC [d37f9842] info [native] ASID[51], smfID[S0W1], sysid[C021], jobName[HVBAC021] - ZDTP020I - ZDTP020I Active Sensors: MQ DB2 DB2R SOAP CTG DB2Fetch DLI DLIR HTTP .
Verify the zDC started correctly
Verify that the zDC has started the correct version, initialized successfully and has started/attached to the zLocal (messages in zDC SYSPRINT DD).
ZDC000I INITIALIZATION STARTED FOR zDC VER 07.02.00
ZDC052I zDC IS RUNNING ON Z/OS RELEASE 02.02.00
ZDC053I LPAR NAME IBMSYS1 CVTSNAME S0W1
.
.
.
.
ZDC955L DESK connection being processed ZDC-Job/ID:AFVBZ021/Z021
ZDC958L DESK INIT completed, ZDC AgentId received ZDC-Job/ID:AFVBZ021/Z021
ZDC993I Opn1RFD:0008 /u/labuser/adcdk/ci/7.2build/log/dt_Z021_Z021_33620108.0.log
ZDC955I DESK connection being processed ZDC-Job/ID:AFVBZ021/Z021
If the following error message appears in the zRemote log file:
JJJJ-MM-DD HH:MM:SS [37ab2e00] severe [native] Exception while
connecting to Collector, info:<Could not register Agent with
Collector as the Collector declined service>
Make the appropriate adjustments to the affected components.
If the following message appears in the zDC job log (your subsystem name may be different than MEPC):
11:55:30.419083 ZDC006E SUBSYSTEM MEPC ALREADY EXISTS AND IS ACTIVE
Follow the steps outlined in the Emergency ZDC Termination section.
Insufficient access authority message when starting the zDC
If the following messages (or something similar) appear in the zDC job log:
H408I USER(xxxxxxxx) GROUP(xxxx) NAME(STARTED TASK )
BPX.FILEATTR.PROGCTL CL(FACILITY)
INSUFFICIENT ACCESS AUTHORITY
ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
And the associated messages appear in the zLocal logfile:
JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Server requests us to use Agent
dTMajor.dTMinorVersion.0.dTBuild with a hash of 0317af199c1ab1a03dda2cee90c2ea61
JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Requesting Agent library from Server
JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Error setting Agent library program
controlled: EDC5139I Operation not permitted.
JJJJ-MM-DD HH:MM:SS 3f8bbe02¨ info native¨ Loading Agent
/dt/desk-<dTMajor.dTMinorVersion.0>/agent/downloads/<dTMajor.dTMinorVersion.0.dTBuild>
/native/zos-s390-64/libdtzagent.so
These can be safely ignored.
After the initial download of the OneAgent, it tries to set a flag for the downloaded library that is needed in certain circumstances. It doesn't try to set this flag during the second run because it doesn't re-download the library if it's already there.
Verify the zLocal is running and connected to the zDC and DESK ActiveGate (in zDC SYSPRINT)
The standard code module logfile contains the standard code module messages relating to startup, logging, versioning, and connectivity to the Server/Collector. The zDC SYSPRINT contains the stdout
for the zLocal, and this contains the location of the zLocal log file.
If the SYSPRINT does not contain this information, instruct the zDC to display it using the following command:
Modify zdc,dt1 stdo
info (native) Log level console ........... info
info (native) Log level file .............. info
info (native) Log file .................... <dtInstallFolder>/log
/dt_<agent_name>\_<process_id>.log
Once that log file is located, it is accessible by the standard techniques. Use OMVS, ssh/telnet, or view it in the DESK UI like a regular OneAgent log file.
To verify that all the channels are connected properly, look for the following items in the OneAgent log file:
info [native] dynaTrace z Remote Agent data channel connected successfully, performing handshake.
info [native] dynaTrace z Remote Agent client handshake performed.
info [native] dynaTrace z Remote Agent data channel handshake successful, version[rr.rr.rr.bbbb].
info [native] dynaTrace z Remote Agent control channel connected successfully, performing handshake
info [native] dynaTrace z Remote Agent handshakes are complete, all channels are fully operational.
Also, look for the following records in the corresponding zRemote log file (the values inside the bracket should reflect your live data).
info [native] dynaTrace z Local Agent handshakes are complete, all channels are fully operational, version[rr.rr.rr.bbbb].
info [native] Data client socket listener thread started
info [native] ASID[48], smfID[S0W1], sysid[Z208], jobName[AF71Z208], subagentid[da57ff16] smfID.JobID[S0W1 .JOB92014], zDC release 65 was successfully initialized with protocol version=6.5.0
info [native] zDC[Z208] SMO is initialized with size=10M.
info [native] Registering the zdc[48]
Verify the connectivity loop by issuing a test ping
- Log in to CICS.
- Execute the
DTAX
transaction with a parameter of 'PING'.
- Verify that the PING traveled from CICS to zDC to zRemote and back again from zRemote to zDC.
The output of the DTAX Transaction looks like this:
The log message in the OneAgent log file looks like this:
info [native] ASID[228], smfID[CW06], sysid[C208], jobName[H06AC208] Ping data=CA72106A
zLocal could not be started
Check the job log of the zDC for any of the following error messages.
Invalid permission on the zLocal folder or libraries
ZDC996E Atx5!DTAGTCMD attach failed
ZDC997D RetRcRsn FFFFFFFF 0000006F 5B400002 EACCES: Permission is denied
ZDC997D Notice: unknown modid, reason text may be incorrect
ZDC992I IniA!z/OS Unix agent attach failed
ZDC996E @Abrtz/OS Unix agent attach failed,Abort
Correct the permission on the zLocal folder and binaries, so they are at least readable and executable.
TCP/IP access denied
ZDC996E Atx5!DTAGTCMD attach failed
ZDC997D RetRcRsn FFFFFFFF 0000006F EF076015 EACCES: Permission is denied
ZDC997D Description: SAF CKACC returned error.
ZDC996E @Abrtz/OS Unix agent attach failed,Abort
If your security system, such as Top Secret or ACF2, denies access to TCP/IP by default, add a rule to permit the zDC user ID to access TCP/IP.
Invalid location specified for the zLocal
ZDC996E Atx5!DTAGTCMD attach failed
ZDC997D RetRcRsn FFFFFFFF 00000081 0516006C ENOENT: No such file, directory, or IPC
member exists
ZDC997D JRFileNotThere: The requested file does not exist
ZDC992I IniA!z/OS Unix agent attach failed
ZDC996E @Abrtz/OS Unix agent attach failed,Abort
ZDC975E z/OS Unix agent attach failed,Abort
Correct the DTAGTCMD parameter in the SYSIN DD of the zDC to point to the correct zLocal binary location.
System under diagnosis (CICS region) could not connect to the zDC
Check the job log of the affected CICS regions for the following message, where yyyy
is the subsystem ID of the zDC that the CICS region is trying to connect to. It may be blank if the CICS region is trying to connect to the default subsystem that is configured with the DEFAULT(YES) parameter. We recommend to simply search for the error message code.
ZDTP002W zDC yyyy unavailable or CICS exits disabled
Verify that the zDC with that subsystem ID is started. If so, then try to issue a DTAX Enable command to re-enable the connections.
Expected queues and/or transactions (including SOAP and CTG transactions) are not being monitored
Verify that the CICS code module is configured correctly, and that the monitored CICS region's configuration matches that of the CICS Sensor in the system profile.
In the DESK UI select Settings then select Server-side service monitoring. Now under Server-side service monitoring select Deep monitoring. Under Deep Monitoring select New OneAgent features. Hit the right tab arrow (>) until you see the feature(s) you want.
Disabled IMS code module
If an ABEND occurs in the IMS code module code, the recovery process produces ABEND diagnostics if possible, then disables the IMS code module. The IMS system continue to function. When this occurs a series of WTO messages are written to the system log for the IMS Control Region and/or IMS dependent region. A sample normal message set follows:
ZDTI032W Recovery routine entered.
ZDTI036W ZDTIII14 BDCSBK00 20170713 10.15 VER 07.02.00 ABEND at offset 001DB6.
ZDTI033W Successful ABEND recovery, agent disabled.
Different or additional messages may be issued if abnormal conditions are encountered by the recovery process, such as when dynamic storage cannot be obtained, retry is not permitted, or no SDWA was passed. All of the messages related to the ABEND recovery process are documented in the z/OS code module messages section.
A Software (SFT) Error Record further describing the ABEND is usually written to the z/OS system SYS1.LOGREC data set. You should run the z/OS EREP utility program to print the Software (SFT) Error Record associated with the ABEND.
Optionally, an SVC dump may be taken during recovery, depending on the ABEND recovery option specified or defaulted to when the IMS code module was injected. The default action is to not capture an SVC dump when ABEND recovery is driven. This option can be specified as a parameter when the IMS code module is initially injected, or specified in conjunction with the Modify or Enable function parameters to toggle dump capture during recovery on or off for a previously injected IMS code module. See Deploy OneAgent on IMS for more details.
When the IMS code module becomes disabled as a result of the ABEND recovery process, it remains disabled until explicitly re-enabled using the IMS code module injection program contact support for a resolution.
Migration troubleshooting
To ensure proper migration:
- Do not connect newer CICS and IMS code modules of OneAgent to older z/DCs.
DESK ActiveGate always supports OneAgents of earlier versions. with older other components should normally be fine no matter if code modules are bootstrapped or not.
Agents started by the bootstrap part update on application restart.