Distributed Network Management BOF (DISMAN) Reported by Donna McMaster/Bay Networks and Glenn Waters/Bell-Northern Research The DISMAN BOF, chaired by Bob Stewart, met on Tuesday, 4 April. The purpose of the BOF was to discuss the Manager-to-Manager (M2M) MIB and other distributed management work that the IETF Network Management Area might consider tackling. Bob Stewart presented the agenda, which was accepted as proposed: o Agenda Bashing o Introduction, Bob Stewart, Cisco o Steve Waldbusser, CMU o Brian O'Keefe, Hewlett-Packard o David Levy, SNMP Research o Karl Auerbach, CaveBear o Yechiam Yemini, Columbia University o Open Discussion and Summary Bob Stewart's Presentation Bob discussed why the SNMPv2 Working Group decided to separate M2M MIB from other SNMPv2 work. Bob expects the group will end up with at least a charter to clean up, fix, and finish the M2M MIB. Dave Perkins suggested that M2M work could be split, leaving some of the event-handling framework in the SNMPv2 group and creating a new group to handle the rest of the current draft. Bob also gave an overview of distributed management and suggested where we might go from here. (See slides.) Bob's final slide, ``Where We Might Go,'' summarized options. Bob expects that building ``something completely different'' will not be a popular option. He expects that we must at least fix M2M problems. Defining MIBs for expressions (e.g., computing utilization), scripts (e.g., ``if x then y''), and programs (more complexity) fall in between and will need more discussion. Steve Waldbusser's Presentation Steve pointed out that the fundamental concepts in the M2M MIB have been implemented and used in the RMON MIB for several years. He presented a slide to illustrate a split between the framework and applications pieces of the MIB. Manager-to-Manager o ``Framework'' - Event forwarding - Delegation? o Applications - Threshold polling (i.e., alarmTable) - History - Scripting - Topology discovery - ... With regard to the framework, Steve pointed out that event processing is a key feature of the management that we do, and that forwarding events to a hierarchy of consoles is a natural extension. He would be interested in getting together with others interested in exploring these ideas. Steve deferred the discussion of delegation to the speakers that follow. Brian O'Keefe's Presentation Brian discussed the Manager-to-Manager (M2M) MIB status and presented an overview of the MIB. He also suggested some evolutionary changes that could be made to M2M. Brian made the disclaimer that these slides represent his perceptions as he was not an author of the M2M MIB. When asked for implementation experience, two or three hands were raised, and only one indicated that the MIB is deployed. However, it was pointed out that there is a lot of implementation experience with similar concepts in the RMON MIB. Following Brian's presentation, someone asked the difference between M2M and RMON. Brian responded that the functions are similar, but RMON looks at what to monitor and M2M looks at how to monitor (the mechanism). The RMONMIB Working Group Chair, Andy Bierman, noted that the RMONMIB Working Group has deferred work on the Alarms and Events groups to the M2M effort. A text version of Brian's slides follows. Note that the two ``Proposal'' sections were not actually presented during the BOF, but are included here for anyone interested. Brian asks readers to keep in mind they are only preliminary/straw proposals to address the issues identified in the earlier portion of the presentation. o Purpose: (initial version) - Remote configuration of sampling/threshold monitoring engines for alarm generation - Also provides some capability for hierarchical-based polling/monitoring (i.e., last sampled value) o Current status: - Proposed Standard (RFC 1451, April 1993) - Originally part of the SNMPv2 document set - The SNMPv2 Working Group deferred work to new working group per {57} (so as to focus on ``core'' SNMPv2 framework) - Implementation experience: * Actual M2M MIB implementations? Deployed/in-use? * Many SNMP sampling/threshold monitoring engines exist (but not using standard configuration methods) o Overview: There are two object groups and a total of three tables (plus support_scalars): - Event Group snmpEventTable Description and statistics for configured notifications snmpEventNotifyTable Configures transmission of notifications specified in Event Table - Alarm Group snmpAlarmTable Configures sampling, threshold monitoring and alarm generation for a specified managed entity/variable o Issues: (for debate) 1. Specifying destinations in snmpEventNotifyTable - Currently done via contextId only as input to ACL search - Need QOS too? Replace outright? Not a problem? --> This is a black-hole discussion. 2. Fixes and improvements to snmpAlarmTable - Unconstrain snmpAlarmStatus rules {58} --> enable modification via single set-operation - Add rateOfChange as well as (instead of) deltaValue {105} --> avoids spurious alarms caused by delays in sample-taking - Add consecutiveSamples metric to threshold configuration {60} --> enables distinction between sustained and spurious alarms - Add equalTo threshold comparison {61} --> necessary for monitoring state variables, not just counters and gauges - Add MIB expression thresholds {62} --> necessary for monitoring derived statistics (i.e., percent errors) o Possible evolution of M2M-MIB: 1. Support wild-card object instance specification in snmpAlarmVariable --> simplify/reduce configuration transactions and storage 2. Maintain history of sampled data --> enable standard configuration of distributed data collection and trend services 3. Steps towards more automated actions/control operations, scripting --> greater management by delegation 4. Etc. o Proposal: Consolidated Alarm Table fixes/improvements - Textual conventions: SnmpAlarmType ::= ENUM { absValue, deltaValue, rateValue } SnmpAlarmComparison ::= ENUM { none, eq, ne, ge, gt, lt, le } - SNMP Alarm Table { contextIdentity, snmpAlarmIndex } snmpAlarmVariable r/c InstancePointer snmpAlarmInterval r/c Unsigned32 snmpAlarmSampleType r/c SnmpAlarmType snmpAlarmSampledInterval r/o Unsigned32 snmpAlarmSampledValue r/o Integer32 snmpAlarmExceptCompare r/c SnmpAlarmComparison snmpAlarmExceptThreshold r/c Integer32 snmpAlarmExceptConsec r/c Unsigned32 (1..65535) snmpAlarmExceptEventInde r/c Unsigned32 snmpAlarmClearedCompare r/c SnmpAlarmComparison snmpAlarmClearedThreshold r/c Integer32 snmpAlarmClearedConsec r/c Unsigned32 snmpAlarmClearedEventIndex r/c Unsigned32 o Proposal: M2M extension to support MIB Expressions - Textual Conventions: SnmpExprVarType ::= ENUM { absValue, deltaValue, rateValue, constant } SnmpExprVarOp::= ENUM { add, sub, mult, div, mod, sqrt, sq, x2y } - SNMP Expression Table { snmpExprIndex } snmpExprIndex n/a Unsigned32 snmpExprDescr r/c DisplayString snmpExprPrecision r/c Integer32 snmpExprStatus r/c RowStatus - SNMP Expression Variable Table { snmpExprIndex, snmpExprVarIndex g snmpExprVarIndex n/a Unsigned32 snmpExprVarType r/c SnmpExprVarType snmpExprVarVariable r/c InstancePointer snmpExprVarConstant r/c Integer32 snmpExprVarOperator r/c SnmpExprVarOp snmpExprVarStatus r/c RowStatus - Expression defined in post-fix notation, result truncated to Integer32 - Used when snmpAlarmVariable points to entry in snmpExprTable David Levi's Presentation Mid-Level Manager MIB and SNMP Script Language o History - Language designed for XNETMON o/a late 1992 - Customers requested language without GUI o/a mid 1993 Created Mid-Level Manager (MLM) - Customers requested a GUI to help configure MLM o/a early 1994 Created Mid-Level Manager Config Tool o Goals - Ease burden of management stations - Reduce/localize network traffic - Expand domain of manageable devices - Automatic corrective behaviors o Architecture - Showed BRASS server (a thing that consolidates SNMP traffic) - MLM implemented as Emanate subagent - Can run on same processor or on agent or other - Can have multiple MLMs talk with each other o MLM MIB (see Internet-Draft, draft-levi-snmp-mid-level-mgr) Three tables: - mlmScriptTable * mlmScriptTable used to upload/download scripts between the MLM and manager, e.g., the MLM Config Tool. Scripts are stored as Octet Strings - mlmCompileTable * mlmCompileTable used for configuring and running scripts +can contain filename of script or pointer to mlmScriptTable +can specify arguments to pass to a script +can specify frequency to run script periodically +can command script to be run once - mlmResultTable used to make result of running script available in MIB variables * scripts return varbind lists * result table contains encoding of varbind lists: mlmResultOID mlmResultType mlmIntegerValue mlmOctetStringValue etc. o Script Language - All variables are varbind lists - Basic control structures (if, while) - Various logical/mathematical operators - Language can be extended by registering C functions with script library - Scripts run asynchronously o Basic Script Capabilities - SNMP operations (get, set, etc.), result assigned to script variable - Send trap or M2M inform request - Log data to a file - Fork, call, or jump to another script - Launch another application o Experience Have written many MLM applications, for example: - Intruder detection script * uses RMON to capture source address of packet * script checks if packets are from a ``trusted'' device * if not, script issues a warning (could be a set request, send e-mail, etc.) - Audio counter-attack script * monitors audio MIB for noise level * if level is above threshold, turn on UPS with set * if level stays above threshold, do set to play back the noise - M2M-like scripts * retrieves several MIB variables, possibly from several agents * combines the variables using a mathematical expression * if a threshold is crossed, send an inform request o Future Directions - MLM MIB separate mlmCompileTable into an ``available scripts'' table and a ``script execution'' table - Explore alternative means for NMS applications to get results from MLM - Define useful extensions to script language - Harmonization between script language and snmp-tcl. David was asked if he had any feel for resource consumption. He said that scripts are stored in tokenized format, compiled once received, and can use as much resources as you want. Jeff Case commented that it has been implemented in ROM-based systems. Most of the time they tend to be blocked waiting for an event or timer, so they are not resource-consumptive at that point. Someone asked how specific is MIB to language? Could MIB be used for snmp-tcl? There are some detail problems, some MIB pieces are specific, but overall it would be possible with a few changes. Steve Waldbusser commented that it could have a script stored so that you push a button and it runs. For example, this could be done as a side-effect of a get request. Karl Auerbach's Presentation Karl thinks the previous approach is reasonable, but not useful for building real management. The premise is area managers are needed close to what we want to manage. He sees RMON scripts as impractical for area managers. Karl suggests a totally different approach. How do we put control out in the network efficiently and securely? A box runs in a multi-threaded environment with scheme manipulators, e.g., for SNMP, ping, etc. S-exprs return information to area manager. o Primitives - Bit-twiddling too low-level - Association protocol ``transport-agile'' could be used for UDP, dial-up, etc. - Self-delimiting, ASCII text - Program sends back response programs - Need way for manager to find out what is running and cancel - Manager trusts agent: checks periodically for reachability, does traceroute if loses connection - Script trust -- formal prof too expensive o Useful primitives and operators - List of addresses (who is out there) - Misconfigured IP subnet masks - MTU, traceroute, ICMP path discovery - Novellish stuff Jeff Case wanted to know what the difference is between this and Levi's proposal? Primitive? Security? Parallelism? Flexibility? Marshall Rose believes both speakers agree in value of remote execution environment, programmable entity. Differences revolve around functionality (primitives). Also Levi proposes SNMP as the transport mechanism; Karl did not want to be limited by SNMP as transport. Yechiam Yemini's Presentation Distributed Management by Delegation (MBD) o Current Management: centralized and labor intensive - Plugging in a network device is not as simple as plugging in a stereo - Must always go from centralized administrator to each end device - Trained people are expensive and hard to find - Expects vendor that created MIB to also build applications to manage it but vendor may not have the resources to do this, especially on multiple platforms - Want to eliminate labor intensive function from management - Replace people with programs that run in the network - As elements become more complex, boundaries between control of device and management blur (e.g., ATM virtual circuit) o Goal: automated embedded management - Localize control loops, automate functions - Developed at Columbia University in 1988 and licensed by several sites o Operation - ``Software backplane'' inserted into devices in network - Can plug functions into backplane - Download, execute under local or remote control - Move programs to data, focus on manager function deployment - Distinguishing feature: language independent, delegate compiled C, C++, interpreted TCL - OS independent (has a portable thread package) o Applications - Delegate * policy scripts to automate domain management * new protocol interface (CORBA access to MIB data) * config language interpreter and then config. scripts - End-user can leverage a small team of expert staff to manage a large network. - Defined a MIB view language, allows you to define views on a MIB, creates virtual table, SNMP + (expressive power > CMIP) o Notes/Status - not a product but an infrastructure - code is available to play with Steve Waldbusser asked what would be standardized? Yechiam Yemini responded: not new language or transport protocol; URL to access program; protocol for remote control; inter-process communications, threads. Jeff Case noted that 67k is needed just for delegation services. He asked where is C compiler? -- not in VxWorks! Dave Perkins asked how do you get results back from programs, scripts? Occam's razor -- build only what is there: SNMP MIB, RPC, CORBA, etc. Discussion o Marshall Rose warned to watch out for stereo analogy -- most folks cannot set their VCR clocks. - Observes general agreement concerning remote execution/delegation - Questions are * transport to remote * protocols, security, etc. * response, etc. - Back to reality: this is a standards activity * originally network management used UDP datagram -- fetch value/set value -- but hardware specific * interoperability requires specific semantics, which is where we get MIBs * if this group undertakes standardization, it must define: +transport mechanism +language +threads, IPCs, etc. +data structures +extensions, e.g., ping, traceroute o Karl Auerbach asked where can we start -- do something simple but useful? Number one is expressions that combine existing MIB variables. o Greg Bruell commented that Bay Networks has a homegrown scripting language that has been used for about six months. It can download programs; get results in octet strings. o Maria Greene has used TCL scripts and feels it would be good to standardize. - But in addition can we define scripts? - Could develop/define RMON using this capability (e.g., monitor threshold/traps) - Should working group charter include repository of scripted applications? o Someone suggested collecting information in a file -- use FTP or something to get the file. o Bob Stewart agreed and commented that history table is also a good idea. o Jeff Case suggested that the group: 1. Fix/modestly enhance M2M in short time; stay safe, get it done as soon as possible 2. If you want something more aggressive, then agree on a model, define execution environment, data structures, and primitives Later, consider scripts. The easiest part is probably the transport. Summary Bob asked for consensus: o M2M fix -- a big yes o Adding programmability to SNMP -- also strongly supported Bob will talk to Dierdre to set up a working group, get a charter under way, and set up a mailing list.