How to get the IP of the connection responsible for something in the errlog.log?

[dejan]

I am still trying to figure out why we are getting a weird SIGSEGV errors from time to time on our server, so I have a new query...

An example from errlog.log:

server.example.com::[44515        IIGCC, 28981     , 0000000000000007]: Mon Mar 15 17:33:00 2010 E_GC2205_RMT_ABORT     Session failure: ABORT received from remote partner.
server.example.com::[44515        IIGCC, 28981     , 0000000000000007]: Mon Mar 15 17:33:00 2010 E_GC0001_ASSOC_FAIL    Association failure: partner abruptly released association

Question is - how can I get the IP of the connection associated with this session? Is it possible at all? This would at least help me find the client who is causing this error.

Thanks in advance.

[rajus01]

The error E_GC2205_RMT_ABORT followed by E_GC0001_ASSOC_FAIL error logged by Ingres/Net server ( IIGCC) in the errlog.log means that the remote DBMS server running on some other host that you are accessing from server.example.com had died for some fatal reason. A detailed tracing of the DBMS server will show the complete session information at the time of the error.. The GCC server itself does not log this information in the errlog.log....

Regards,
Usha

[paulfwhite]

I see these types of messages regularly when OpenROAD client disconnects from DBMS server without a formal DBSessionObject.Disconnect instruction. Generally I ignore the messages because it could indicate the user killed the process, the PC crashed, or network dropped. most likely it is just sloppy programming.

In answer to your question, while the session is connected you can determine the IP address of the client and also the process ID running on that machine.

# iinamu
Ingres 2006 Release 2 NAME SERVICE MANAGEMENT UTILITY --
-- Copyright (c) 2004 Ingres Corporation
IINAMU> show *
IINAMU> show iidbdb
IINAMU> show
INGRES * 56874 (sole server)
IINAMU> quit

# iimonitor 56874
IIMONITOR> show sessions
Session 0018DF80:36875152 (ingres ) cs_state: CS_EVENT_WAIT (BIOR) cs_mask: CS_INTERRUPT_MASK,CS_NOXACT_MASK
...
Session 04BAB9A0:125655952 (bramen ) cs_state: CS_EVENT_WAIT (BIOR) cs_mask: CS_INTERRUPT_MASK,CS_NOXACT_MASK
...
IIMONITOR> format 04BAB9A0
>>>>>Session 04BAB9A0:125655952<<<<<
DB Name: testreet (Owned by: ingres )
User: bramen (bramen )
User Name at Session Startup: bramen
Terminal: batch
Group Id:
Role Id:
Application Code: 00000000 Current Facility: CLF (00000001)
Client user: ingres
Client host: ADIOS
Client tty: ADIOS
Client pid: 6656
Client connection target: pit::testreet
Client information: user='ingres',host='ADIOS',tty='ADIOS',pid=6656,co nn='pit::testreet'
Description: Visual DBA
Query:
IIMONITOR>quit

# nslookup ADIOS
Server: peercoresbs2003.peercore.local
Address: 10.10.10.10

Name: ADIOS.peercore.local
Addresses: 10.10.10.111, 10.10.10.126


Here you can see the user name is Bramen, running VDBA. his PC name is ADIOS and the process ID on that pc is 6656. nslookup command tells me he has two network cards. (maybe wireless?)

On Bramen's PC I can use windows taskmgr to see processID 6656 is indeed vdba.

I have a unix script to capture and summarise connection info from iimonitor. Let me know if you want a copy.

Paul

[dejan]

Thanks Paul, I appreciate your help.

I suspect it is a legacy ABF application running on OpenVMS that causes this. I have talked to the authors of that application yesterday, and they confirmed that they do not explicitly close connections at all (*sigh*). Same for their new .NET-based (web) applications...

I definitely want a copy of that script - and will greatly appreciate it if you send it to dejan.lekic (AT) gmail.com . Kind regards.

[dejan]

OK, I think I have traced down what is causing these problems, and what might be the the cause of nasty SIGSEGV error that crashes IIDBMS... Seems that this error appears when ABF (legacy applications) users abruptly close their terminals (they use terminal emulator to connect to our OpenVMS server on which we have our legacy ABF apps).

I fear there is no way to fix this problem, isn't it? Because they can close their terminals (and by that kill the ABF application running inside) at any particular time...

[harda20]

Hi

Have you tried trace point sc906?

This logs message E_SC0233_SESSION_START at startup of a session and E_SC0234_SESSION_END when the session ends.

Using this in conjunction with iimonitor and a network lookup, as Paul suggests, should help to get the information you want.

The advantage of using the trace point is that if you monitor the errlog.log you can pick the user up just after they connect. This means you shouldn't miss anyone which might be possible if you just run iimonitor scans every now and then. You can have a current list of who is connected, where from etc. You can also remove them from the list when they disconnect.

David

[nick.bolton]

I do think some of the comms server errors should put a little more detail in errlog.log.
e.g. I saw this the other day:

xxxxxx ::[3256 IIGCC, 0000000000000029]: Mon Mar 8 10:02:58 2010
E_GC220D_AL_PASSWD_FAIL GCC AL received an invalid (null) password from a remote client.

Our users are hidden behind an application; so why would anybody get the password wrong!
If I had an IP address logged, I could start to trace who is trying to access the database.

[dejan]

Nick, that is exactly what I need. Or some trace point which, if enabled, give me IP of that connection...

[nick.bolton]

Whilst I am suggesting new features, it would be nice to have something in config.dat that allowed me to move all of the comms server messages to another logfile. The E_GC messages tend to add a lot of noise into errlog.log, often for negligible benefit.

I suspect that error logging is centralised and more-or-less hardcoded to that filename though, so this may not be the quick and simple change it sounds like to me ;-)

One day I will understand the source code enough to stick these features in myself (you can expect that feature in Ingres 27 then).

[kschendel]

Originally Posted by nick.bolton@barclays.com

Whilst I am suggesting new features, it would be nice to have something in config.dat that allowed me to move all of the comms server messages to another logfile. The E_GC messages tend to add a lot of noise into errlog.log, often for negligible benefit.

I suspect that error logging is centralised and more-or-less hardcoded to that filename though, so this may not be the quick and simple change it sounds like to me ;-)

You are right. All the server messages, except for DBMS log messages, go through the ule_format interface, which is a front-ender to the ER facility in the CL. It all ends up at ERlog. I'm sure that it would be possible to expand ER to include some sort of target indicator, but the effects would ripple throughout the code base. (This is the point at which one normally looks for something easier to do!)