So you come to a task in your self-preparation leading up to the real lab that explains that some form of error has so kindly been introduced for you in your UCM configuration, and you need to figure out how the UCM is “seeing” the problem. In such a case, we of course turn to our beloved friend, the traces.
To enable traces we need to navigate to the Unified Servicability UI, and go to Trace >>> Configuration >>> choose a Server (makes no difference which if we later check “Apply to All Nodes”) >>> choose CM Services >>> choose Cisco CallManager >>> click Apply to All Nodes >>> choose the Debug Trace Level of Detailed >>> and finally uncheck all but what you are interested in troubleshooting on this go-round with the traces – say for this example – All IP Phone Devices.

Now this is where the more difficult part comes in. It used to be with the old windows version of CCM (4.x and prior) that you would simply RDP or map a network drive into the Windows machine CCM was running on, navigate your way in Windows Explorer through the proper directory structure of C:\Program Files\Cisco\Trace\CCM\ and there you would find a list of the files that had been created as traces were accumulated – and you would simply open the highest numbered file in MS Notepad and scroll to the bottom of it. Of course you weren’t able to view that file in realtime (without viewing it through some part of third party tool such as Cygwin and tail) – but it was certainly helpful.
If we want to view trace files in the new version of UCM – we are told by documentation that the way to do this is to open up the RTMT (Real Time Monitoring Tool – which can be downloaded from the UCM WebUI Plugins page) and to access the trace logs and files that way. Well we could do that – but we have to first download the tool, install it, connect with it, and oh by the way, we have to have Windows to run it. Well many of us don’t have Windows – and we don’t want to have to open a virtualization program to run it just for that. Plus, even if we are on a native Windows platform, possibly we don’t want the overhead of another program running – especially a Java program that could potentially have a memory issue. Is there not another, possibly easier way to access them? Rest assured there is.
We have the ability to SSH into any of our servers that run on the common UCOS. To be clear we aren’t able to gain root level access into the RedHat Linux OS underneath, but Cisco was kind enough to build us a sandbox of a shell and we have a common set of CLI tools in this platform that (mostly) pervade all upper level servers on this UCOS. One of those tools is the “file” tool.
Before we SSH into a CUCM server, we must bear in mind, which Sub/Pub server is being used as the primary CPE for the device we are looking for insight into. This is because this will be the server where data where primarily be logged to in regards to traces. So in our case the CUCM-Sub is the server where our phones are trying to register and being rejected – so we will SSH into that server.
Marks-MacBook-Pro-17:~ mxs$ ssh -l admin 10.10.210.11
admin@10.10.210.11's password:
Last login: Fri Mar 13 13:17:40 2009 from 10.10.0.7
admin:
Now that we are in, we need to figure out where the trace files are kept, and which one we need to view. If we issue the ‘file’ command, we see that we have a number of arguments to issue against a particular file. Here we will describe usage of the sub-tools of ‘file list’, ‘file view’, and ‘file tail’, although I am sure you can also see how quickly a tool such as ‘file search’ would be ever so useful.
admin:file ?
file check
file delete*
file dump*
file fragmentation*
file get*
file list*
file search*
file tail*
file view*
We can quickly see what the filesystem structure looks like by issuing the ‘file list activelog’ command for every directory we find.
admin:file list activelog /
<dir> car_db
<dir> ccm_db
<dir> cm
<dir> core
<dir> mgetty
<dir> mohprep
<dir> patches
<dir> platform
<dir> sa
<dir> syslog
<dir> tomcat
dir count = 11, file count = 0
admin:file list activelog /cm/
<dir> bin
<dir> cdr
<dir> cdr_repository
<dir> log
<dir> report
<dir> tftpdata
<dir> trace
dir count = 7, file count = 0
admin:file list activelog /cm/trace/
<dir> ac
<dir> amc
<dir> bps
<dir> capf
<dir> carsch
<dir> ccm
<dir> ccmmib
<dir> ccmservice
<dir> cdp
<dir> cdpmib
<dir> cdragent
<dir> cdrrep
<dir> cef
<dir> cfrt
<dir> cmi
<dir> cms
<dir> ctftp
<dir> cti
<dir> ctlprovider
<dir> dbl
<dir> dhcpmon
<dir> dirsync
<dir> licensing
<dir> lpm
<dir> ris
<dir> rtmtreporter
<dir> syslogmib
<dir> taps
<dir> tct
dir count = 29, file count = 0
admin:file list activelog /cm/trace/ccm/
<dir> Proglogs
<dir> sdi
<dir> sdl
dir count = 3, file count = 0
admin:file list activelog /cm/trace/ccm/sdi/
ccm00000001.txt ccm00000002.txt
ccm00000003.txt ccm00000004.txt
ccm~num.bin
dir count = 0, file count = 4
admin:
Now we will use the ‘file view’ command to peer into file 4. We can see at the bottom there is a legend describing the ability to use the ‘e’ key to move us to the end of the file, and then the ‘p’ or ‘n’ key to move us to a previous page or next page, respectively. Give it time – it has to load the whole file into memory before it can display it to the VTY terminal screen.
admin:file view activelog /cm/trace/ccm/sdi/ccm00000004.txt
12/29/2008 22:31:31.979 CCM|<--RISCMAccess::DeviceTransientConnection(...)|<CLID::StandAloneCluster><NID::10.10.210.11>
<LVL::Entry_exit><MASK::10000>
12/29/2008 22:31:31.980 CCM|EnvProcessUdpPort - EnvProcessUdpHandler::fireSignal() varId = 1|<CLID::StandAloneCluster>
<NID::10.10.210.11><CT::2,100,186,1.174760><IP::10.10.200.253><DEV::SEP001BD4C6C2F6><
LVL::Arbitrary><MASK::0800>
12/29/2008 22:31:31.980 CCM|EnvProcessUdpPort - EnvProcessUdpHandler::send(buff, 394, 10.10.200.253:5060)|<CLID::StandAloneCluster>
<NID::10.10.210.11><LVL::Arbitrary><MASK::0800>
12/29/2008 22:31:33.064 CCM|//SIP/SIPHandler/ccbId=0/scbId=0/wait_SIPTimer: TimerExpired type=SIP_TIMER_REMOVE_TRANSACTION value=32000
retries=0|<CLID::StandAloneCluster><NID::10.10.210.11><CT::2,100,186,1.174672>
<IP::10.10.201.254><DEV::SEP001BD4C6C26E><LVL::Detailed><MASK::20000>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 67258) :
One more thing though before we go – what if we want to be able to see that info in real-time as it is logged into that file? This is where the ‘file tail’ command comes in. Tail is a *NIX command that looks at the last few lines of a file, but also to watch that file live. Basically as data is written into the file, it is also echoed out to our local VTY. No need to show you the output again from this command – it is the same trace output as above – only it scrolls by in real-time – something hard to show here. We also see that we have options at the end to be more specific as to what we want to see. Control-C is how we exit from the tail.
admin:file tail activelog /cm/trace/ccm/sdi/ccm00000004.txt ?
Syntax:
file tail activelog file-spec [options]
file-spec mandatory file to tail
options optional hex,[num lines],regexp "expression"
recent : to tail the most recently changed file in the directory.
So I hope that has helped you a lot in making viewing traces in UCM (and other UCOS based servers) much easier. It may not make you particularly want to view them more often – but then hey – you’re going to have to overcome that hurdle on your own.
After all – you did set out to become a Cisco Certifiied Internetwork EXPERT, didn’t you? ;-)
Cheers,
Mark
Tags: ccie voice 3.0, Cisco, Traces, Troubleshooting, UCM