Firmware 2.52.d3 snmp not working

[frogfoot]

I have been using the latest firmware 2.52.d3 for a couple of weeks now and I have noticed that the SNMP service keeps crashing. I use Cacti to record various stats, which has been working fine for several months on the previous firmware. However with the latest firmware it requires me to toggle the SNMP service on and off every few days

[frogfoot]

Debug level log for the time when SNMP crashes

Code: Select all

Nov  7 16:22:06 kern warn kernel: snmpd/26567: potentially unexpected fatal signal 11.
Nov  7 16:22:06 kern warn kernel: Cpu 0
Nov  7 16:22:06 kern warn kernel: $ 0   : 00000000 7fe53a98
Nov  7 16:22:06 kern info kernel:  01020201
Nov  7 16:22:06 kern info kernel:  000005f9
Nov  7 16:22:06 kern warn kernel: $ 4   : 7fe53ac4 00000000
Nov  7 16:22:06 kern info kernel:  00000000
Nov  7 16:22:06 kern info kernel:  7fe53ac4
Nov  7 16:22:06 kern warn kernel: $ 8   :
Nov  7 16:22:06 kern info kernel:  00000000 00000000
Nov  7 16:22:06 kern info kernel:  00000006
Nov  7 16:22:06 kern info kernel:  00000001
Nov  7 16:22:06 kern warn kernel: $12   : 00000002 00000001
Nov  7 16:22:06 kern info kernel:  00000002
Nov  7 16:22:06 kern info kernel:  00000002
Nov  7 16:22:06 kern warn kernel: $16   : 00411e58 00411e60
Nov  7 16:22:06 kern info kernel:  00000001
Nov  7 16:22:06 kern info kernel:  00425724
Nov  7 16:22:06 kern warn kernel: $20   : 00411d10 00411d28
Nov  7 16:22:06 kern info kernel:  7fe54a54
Nov  7 16:22:06 kern info kernel:  00000003
Nov  7 16:22:06 kern warn kernel: $24   : 00000000 7735fba0         
Nov  7 16:22:06 kern info kernel:          
Nov  7 16:22:06 kern warn kernel: $28   : 0042d300 7fe53ae8
Nov  7 16:22:06 kern info kernel:  7fe53ae8
Nov  7 16:22:06 kern info kernel:  00408d14
Nov  7 16:22:06 kern warn kernel: Hi    : 00000000
Nov  7 16:22:06 kern warn kernel: Lo    : 00000000
Nov  7 16:22:06 kern warn kernel: epc   : 00408d94 0x408d94
Nov  7 16:22:06 kern warn kernel:     Tainted: P            
Nov  7 16:22:06 kern warn kernel: ra    : 00408d14 0x408d14
Nov  7 16:22:06 kern warn kernel: Status: 00008d13    
Nov  7 16:22:06 kern info kernel: USER 
Nov  7 16:22:06 kern info kernel: EXL 
Nov  7 16:22:06 kern info kernel: IE 
Nov  7 16:22:06 kern warn kernel: Cause : 00000010
Nov  7 16:22:06 kern warn kernel: BadVA : 01020209
Nov  7 16:22:06 kern warn kernel: PrId  : 0002a080 (Broadcom BMIPS4350)

[billion_fan]

Debug level log for the time when SNMP crashes

Code: Select all

Nov  7 16:22:06 kern warn kernel: snmpd/26567: potentially unexpected fatal signal 11.
Nov  7 16:22:06 kern warn kernel: Cpu 0
Nov  7 16:22:06 kern warn kernel: $ 0   : 00000000 7fe53a98
Nov  7 16:22:06 kern info kernel:  01020201
Nov  7 16:22:06 kern info kernel:  000005f9
Nov  7 16:22:06 kern warn kernel: $ 4   : 7fe53ac4 00000000
Nov  7 16:22:06 kern info kernel:  00000000
Nov  7 16:22:06 kern info kernel:  7fe53ac4
Nov  7 16:22:06 kern warn kernel: $ 8   :
Nov  7 16:22:06 kern info kernel:  00000000 00000000
Nov  7 16:22:06 kern info kernel:  00000006
Nov  7 16:22:06 kern info kernel:  00000001
Nov  7 16:22:06 kern warn kernel: $12   : 00000002 00000001
Nov  7 16:22:06 kern info kernel:  00000002
Nov  7 16:22:06 kern info kernel:  00000002
Nov  7 16:22:06 kern warn kernel: $16   : 00411e58 00411e60
Nov  7 16:22:06 kern info kernel:  00000001
Nov  7 16:22:06 kern info kernel:  00425724
Nov  7 16:22:06 kern warn kernel: $20   : 00411d10 00411d28
Nov  7 16:22:06 kern info kernel:  7fe54a54
Nov  7 16:22:06 kern info kernel:  00000003
Nov  7 16:22:06 kern warn kernel: $24   : 00000000 7735fba0         
Nov  7 16:22:06 kern info kernel:          
Nov  7 16:22:06 kern warn kernel: $28   : 0042d300 7fe53ae8
Nov  7 16:22:06 kern info kernel:  7fe53ae8
Nov  7 16:22:06 kern info kernel:  00408d14
Nov  7 16:22:06 kern warn kernel: Hi    : 00000000
Nov  7 16:22:06 kern warn kernel: Lo    : 00000000
Nov  7 16:22:06 kern warn kernel: epc   : 00408d94 0x408d94
Nov  7 16:22:06 kern warn kernel:     Tainted: P            
Nov  7 16:22:06 kern warn kernel: ra    : 00408d14 0x408d14
Nov  7 16:22:06 kern warn kernel: Status: 00008d13    
Nov  7 16:22:06 kern info kernel: USER 
Nov  7 16:22:06 kern info kernel: EXL 
Nov  7 16:22:06 kern info kernel: IE 
Nov  7 16:22:06 kern warn kernel: Cause : 00000010
Nov  7 16:22:06 kern warn kernel: BadVA : 01020209
Nov  7 16:22:06 kern warn kernel: PrId  : 0002a080 (Broadcom BMIPS4350)

I have forwarded your logs over to our engineers, once I get a reply I will drop you a email.

[frogfoot]

Just an update on this, I have had reason to reboot the router via the web interface several times over the last few weeks due to a persistent line fault where several 'engineers' have attended. In each case the SNMP service crashed when rebooting, requiring a toggle of the SNMP on/off menu option to get it restarted.
Tom

[billion_fan]

Pm sent

[frogfoot]

Unfortunately no change with the beta firmware

Code: Select all

Dec  6 12:43:25 kern warn kernel: snmpd/28710: potentially unexpected fatal signal 11.
Dec  6 12:43:25 kern warn kernel: Cpu 1
Dec  6 12:43:25 kern warn kernel: $ 0   : 00000000 10008d00
Dec  6 12:43:25 kern info kernel:  02010130 000005f3
Dec  6 12:43:25 kern warn kernel: $ 4   : 7f955384
Dec  6 12:43:25 kern info kernel:  00000000
Dec  6 12:43:25 kern info kernel:  00000000
Dec  6 12:43:25 kern info kernel:  7f955384
Dec  6 12:43:25 kern warn kernel: $ 8   :
Dec  6 12:43:25 kern info kernel:  00000000
Dec  6 12:43:25 kern info kernel:  00000000
Dec  6 12:43:25 kern info kernel:  00000006
Dec  6 12:43:25 kern info kernel:  00000001
Dec  6 12:43:25 kern warn kernel: $12   :
Dec  6 12:43:25 kern info kernel:  00000002
Dec  6 12:43:25 kern info kernel:  00000001
Dec  6 12:43:25 kern info kernel:  00000002
Dec  6 12:43:25 kern info kernel:  00000002
Dec  6 12:43:25 kern warn kernel: $16   :
Dec  6 12:43:25 kern info kernel:  00413b38
Dec  6 12:43:25 kern info kernel:  00413b40
Dec  6 12:43:25 kern info kernel:  00000001
Dec  6 12:43:25 kern info kernel:  00426b14
Dec  6 12:43:25 kern warn kernel: $20   :
Dec  6 12:43:25 kern info kernel:  004139f0
Dec  6 12:43:25 kern info kernel:  00413a08
Dec  6 12:43:25 kern info kernel:  7f956314
Dec  6 12:43:25 kern info kernel:  00000003
Dec  6 12:43:25 kern warn kernel: $24   :
Dec  6 12:43:25 kern info kernel:  00000000
Dec  6 12:43:25 kern info kernel:  7722cba0
Dec  6 12:43:25 kern info kernel:          
Dec  6 12:43:25 kern info kernel:          
Dec  6 12:43:25 kern warn kernel: $28   :
Dec  6 12:43:25 kern info kernel:  0042e690
Dec  6 12:43:25 kern info kernel:  7f9553a8
Dec  6 12:43:25 kern info kernel:  7f9553a8
Dec  6 12:43:25 kern info kernel:  00409054
Dec  6 12:43:25 kern warn kernel: Hi    : 07dfc548
Dec  6 12:43:25 kern warn kernel: Lo    : b387be83
Dec  6 12:43:25 kern warn kernel: epc   : 004090d4 0x4090d4
Dec  6 12:43:25 kern warn kernel:     Tainted: P            
Dec  6 12:43:25 kern warn kernel: ra    : 00409054 0x409054
Dec  6 12:43:25 kern warn kernel: Status: 00008d13    USER 
Dec  6 12:43:25 kern info kernel: EXL IE 
Dec  6 12:43:25 kern warn kernel: Cause : 00000008
Dec  6 12:43:25 kern warn kernel: BadVA : 02010138
Dec  6 12:43:25 kern warn kernel: PrId  : 0002a080 (Broadcom BMIPS4350)

Seems to happen when Cacti queries the router for a list of monitorable interfaces, the same action works reliably on the 8800AXL

Code: Select all

Executing SNMP get for num of indexes @ '.1.3.6.1.2.1.2.1.0' Index Count: 11
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.2.1.2.2.1.1' Index Count: 0
+ No SNMP data returned

[billion_fan]

Thanks for that, I have reported it to our engineers

[frogfoot]

Update...
It seems to work if I revert to SNMP v1, The walk on .1.3.6.1.2.1.2.2.1.1 returns data without crashing the SNMP daemon.

Code: Select all

+ Running data query [13].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/volume1/web/cacti/resource/snmp_queries/interface.xml'
+ XML file parsed ok.
+ Executing SNMP get for num of indexes @ '.1.3.6.1.2.1.2.1.0' Index Count: 11
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.2.1.2.2.1.1' Index Count: 11
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.1' value: '1'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.2' value: '2'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.3' value: '3'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.4' value: '4'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.5' value: '5'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.6' value: '6'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.7' value: '7'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.8' value: '8'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.9' value: '9'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.10' value: '10'
+ Index found at OID: '.1.3.6.1.2.1.2.2.1.1.11' value: '11'
+ Located input field 'ifIndex' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.1'
+ Found item [ifIndex='1'] index: 1 [from value]
+ Found item [ifIndex='2'] index: 2 [from value]
+ Found item [ifIndex='3'] index: 3 [from value]
+ Found item [ifIndex='4'] index: 4 [from value]
+ Found item [ifIndex='5'] index: 5 [from value]
+ Found item [ifIndex='6'] index: 6 [from value]
+ Found item [ifIndex='7'] index: 7 [from value]
+ Found item [ifIndex='8'] index: 8 [from value]
+ Found item [ifIndex='9'] index: 9 [from value]
+ Found item [ifIndex='10'] index: 10 [from value]
+ Found item [ifIndex='11'] index: 11 [from value]
+ Located input field 'ifOperStatus' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.8'
+ Found item [ifOperStatus='Up'] index: 1 [from value]
+ Found item [ifOperStatus='Up'] index: 2 [from value]
+ Found item [ifOperStatus='Up'] index: 3 [from value]
+ Found item [ifOperStatus='Up'] index: 4 [from value]
+ Found item [ifOperStatus='Up'] index: 5 [from value]
+ Found item [ifOperStatus='Up'] index: 6 [from value]
+ Found item [ifOperStatus='Up'] index: 7 [from value]
+ Found item [ifOperStatus='Up'] index: 8 [from value]
+ Found item [ifOperStatus='Up'] index: 9 [from value]
+ Found item [ifOperStatus='Up'] index: 10 [from value]
+ Found item [ifOperStatus='Up'] index: 11 [from value]
+ Located input field 'ifDescr' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.2'
+ Found item [ifDescr='eth0'] index: 1 [from value]
+ Found item [ifDescr='eth1'] index: 2 [from value]
+ Found item [ifDescr='eth2'] index: 3 [from value]
+ Found item [ifDescr='eth3'] index: 4 [from value]
+ Found item [ifDescr='eth4'] index: 5 [from value]
+ Found item [ifDescr='eth5'] index: 6 [from value]
+ Found item [ifDescr='wl0'] index: 7 [from value]
+ Found item [ifDescr='ppp1.1'] index: 8 [from value]
+ Found item [ifDescr='dsl0'] index: 9 [from value]
+ Found item [ifDescr='atm0'] index: 10 [from value]
+ Found item [ifDescr='cpcs0'] index: 11 [from value]
+ Located input field 'ifName' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.31.1.1.1.1'
+ Located input field 'ifAlias' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.31.1.1.1.18'
+ Located input field 'ifType' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.3'
+ Found item [ifType='ethernetCsmacd'] index: 1 [from value]
+ Found item [ifType='ethernetCsmacd'] index: 2 [from value]
+ Found item [ifType='ethernetCsmacd'] index: 3 [from value]
+ Found item [ifType='ethernetCsmacd'] index: 4 [from value]
+ Found item [ifType='ethernetCsmacd'] index: 5 [from value]
+ Found item [ifType='ethernetCsmacd'] index: 6 [from value]
+ Found item [ifType='ethernetCsmacd'] index: 7 [from value]
+ Found item [ifType='ppp'] index: 8 [from value]
+ Found item [ifType='adsl'] index: 9 [from value]
+ Found item [ifType='atm'] index: 10 [from value]
+ Found item [ifType='aal5'] index: 11 [from value]
+ Located input field 'ifSpeed' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.5'
+ Found item [ifSpeed='1000000000'] index: 1 [from value]
+ Found item [ifSpeed='1000000000'] index: 2 [from value]
+ Found item [ifSpeed='1000000000'] index: 3 [from value]
+ Found item [ifSpeed='1000000000'] index: 4 [from value]
+ Found item [ifSpeed='1000000000'] index: 5 [from value]
+ Found item [ifSpeed='100000000'] index: 6 [from value]
+ Found item [ifSpeed='2144487264'] index: 7 [from value]
+ Found item [ifSpeed='0'] index: 8 [from value]
+ Found item [ifSpeed='79999000'] index: 9 [from value]
+ Found item [ifSpeed='79999000'] index: 10 [from value]
+ Found item [ifSpeed='79999000'] index: 11 [from value]
+ Located input field 'ifHighSpeed' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.31.1.1.1.15'
+ Located input field 'ifHwAddr' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.6'
+ Found item [ifHwAddr='60:03:47:27:9F:5A'] index: 1 [from value]
+ Found item [ifHwAddr='60:03:47:27:9F:5A'] index: 2 [from value]
+ Found item [ifHwAddr='60:03:47:27:9F:5A'] index: 3 [from value]
+ Found item [ifHwAddr='60:03:47:27:9F:5A'] index: 4 [from value]
+ Found item [ifHwAddr='60:03:47:27:9F:5A'] index: 5 [from value]
+ Found item [ifHwAddr='60:03:47:27:9F:5A'] index: 6 [from value]
+ Found item [ifHwAddr='60:03:47:27:9F:5B'] index: 7 [from value]
+ Found item [ifHwAddr=''] index: 8 [from value]
+ Found item [ifHwAddr=''] index: 9 [from value]
+ Found item [ifHwAddr=''] index: 10 [from value]
+ Found item [ifHwAddr=''] index: 11 [from value]
+ Located input field 'ifIP' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.4.20.1.2'

However I noticed a strange thing, not all of the ifDescr values tie up with the data. (For example eth4/5 is reports data from 5Ghz WiFi )

The reported values are:-

eth0 - ethernet port 1
eth1 - ethernet port 2
eth2 - ethernet port 3
eth3 - ethernet port 4
eth4 - 5 GZ WiFi
eth5 - WAN port used as 5th LAN port)
wl0 - 2.4Ghz WiFI
ppp1.1 - PPPoE
dsl0 - no data with my router config
atm0 - no data with my router config
cpcs0 - no data with my router config

[billion_fan]

Thanks for the update, we will investigate, I'll drop you a pm once I get a reply from our engineers