IP Provider Metrics BOF (IPPM) Reported by Stan Barber/Academ Consulting Services Administrative Items The IPPM BOF was chaired by Matt Mathis. The minutes were recorded and reported by Stan Barber with a final review by Matt Mathis. The session was broadcast live on the MBONE. The group's mailing list will be ready soon. Subscribe by sending mail to ippm-request@psc.edu. Attendees of this meeting will not automatically be added to the list. Most of the slides presented at the session were text only and have been imbedded in the minutes. Consider a Possible Working Group o Is the working group needed? o Does it conflict/interact with any other current or past groups? o What do we want to cover? o Chair issues and creating an editor/secretary. o What should the charter say? The primary topic for this BOF was the formation of a working group, however, this was not covered until the end of the session. The group first discussed several of the potential technical work items. General Requirements for Metrics What makes a good metric? How do we design a metric that works for all the users and providers? o It must be fair for uniform technology. It should be as fair as possible for non-uniform technology. Note that properly accounting for intrinsic differences in technology is difficult. o It should be non-pejorative in the sense that is should not imply quality. It should support providers who are selling services that are ``inexpensive yet good enough'' as well as ``the best possible.'' o It should be a true metric, providing concrete, repeatable measurements without subjective interpretation (e.g., a tape measure). o It should be useful for publication in data sheets. This means that it is repeatable by anyone. It should have a capability to measure one provider independent of all others. This is also hard to do. o It should be useful for diagnostics. It should be able to sectionalize a multi-provider path. o It should be useful for procurement, contracts and RFPs. o It must be owned ``by the community,'' or more specifically, not encountered by some company or individual. What Metrics Are Needed? (A Wish List) o Path Performance - Bulk data transfer performance (FTP, etc.) This is most important to the SuperComputing Centers (see treno below for a possible diagnostics tool) - Interactive performance (Telnet, X11, etc.) - Real-time and multicast performance (delay and loss statistics) - Packet delay and loss should be considered - Reliability -- should be considered on its own and not as a factor that is a second order effect of something else o Routing stability and robustness - End-to-end availability - Time to recover (e.g., switch to backup paths) - Route stability (spurious route transitions, etc.) - Coverage (Is the entire Internet reachable?) - Failure isolation (Do failures cascade, say due to insufficient routing resources?) - Are routes aggregated appropriately? A variety of routing metrics were suggested. This should include route stability, some notion of reachability/coverage as well as the degree of route aggregation in the routing system. Routing diameter should also be considered. (There is more on this later in these minutes.) o DNS performance and robustness o Infrastructure security (protection from sniffing and various spoofing attacks) o NOC responsiveness - Trouble reports - Security incidents - DNS problems How long does it take to fix problems that involve DNS caching? How long does it take to resolve trouble tickets? How long does it take to resolve security problems? There is also some concern that some of these are product or business related. It was suggested that there be a core and secondary sets of metrics. Matt believes that the core should only be whatever is really necessary to deliver packets, including IP and the associated routing system. Others believe that customer perceptions have to be considered and this includes the NOC and other difficult-to-measure properties. It has been suggested that the broader list is really a reflection of user expectations and requirements. Different user expectations exist and need to be addressed in some way. Matt thinks that some of this discussion reflects the differences between the roles of an IP wholesaler and retailer. It makes sense to start working on the core metrics and work towards a broader set. Therefore, The IPPM charter should include the broader metrics, but since they are ``fuzzy'' and subject to non-consensus associated with subjective measures, they should be ``tabled'' until after the core metrics are assured convergence. Existing Performance Tools o traceroute and its derivatives o ping and its derivatives o ttcp o tcpdump o Routing logs and analysis tools o rtpqual -- multicast sequence number counter (holes in the sequence space) o off-net from MERIT o mrt -- the multithreaded routing toolkit from Merit o Protocol analyzers, hardware tools and other ``test gear'' o Routing registry and analysis tools related to it o SATAN port scanning o Many, many SNMP based tools Existing tools were listed to give the group confidence that they had a complete metrics wish list. A New Tool: Treno, A Traceroute/Reno TCP Derivative It is an example of a bulk transfer metric. It provides a lot of useful information. It provides a reasonable mimicry of TCP, but does make use if ICMP just like regular traceroute. Christian Huitema said that ICMP was not intended for performance measurement, so as long as this is the case, ICMP does not provide a good mechanism for doing measurement. There is considerable variety among routing vendors in their ICMP processing. Since some vendors do respond to ICMP as fast as they forward packets and some do not, this may cause vendors' equipment to be at a disadvantage because of choices the ICMP responses architecture. A routing requirements working group does exists where this can be discussed. Some folks suggested that the RA machines might be used as landmark machines. Bill Manning said that should not be done. Other landmark machines could be identified. What is the future of treno? Does it really emulate idealized TCP? If not, vendors may optimize their software to make it work well with the tool and not really help make the network work better. Sally Floyd volunteered to work with Matt to see if treno does emulate an appropriate TCP. How do we use it for testing? There are issues in creating a mechanism to do testing (a standard procedure). John Boone volunteered to work on the development of a standard testing procedure to measure bulk transfer using treno.] Routing Metrics -- Bill Manning With the new Internet Architecture, different sites will use different paths to get to the same places. The RA mission is to insure a consistent set of routes for all users. By using a RIPE 181-based database and through an analysis, a consistent routing configuration can be presented. There are a number of things that might be measured: o The level of aggregation o Route persistence (or route flaps) [It was suggested that there is a rate of 30 transitions per second in the current Internet.] o Scalability -- will the current routing architecture scale? Does this relate to the first two items? What about reporting? Since this information can only really be collected at exchange points, what would be done with the data collected? Defining a scope is really critical to doing this. Concerns should be sent to the RA at ra-team@merit.edu or ra-team@isi.edu. The RA will be all NAPs and the two MAEs and other interconnects. The NAP operators are not willing to do these type measurements since it is not related to the Level 2 service they are provisioning. General information about the RA and current North American routing architecture is available under ``papers'' at: http://www.isi.edu/div7/ra Other Issues A number of other issues were also discussed during this meeting. Once this work is complete, specific ownership issues should be considered. Some standards groups assert ownership of the metrics and the tools that produce them. This may be appropriate here. There was also a concern that the meaning of the results should be as immutable over time (some folks called this tool drift). This may be too hard to do and is definitely hard to consider when the metrics and the tools are not yet articulated. Is it possible to create a metric that could be used by an independent party that would produce a meaningful result? Can the metrics be defined in an unambiguous way? Can specific issues such as the interactive performance of a Telnet situation from the user perspective be quantified? There was also considerable concern about providers testing themselves and other providers versus consumers doing the testing and issues involving the need for disclosure of components involved (Black-box versus Clear-box testing). One attendee asserted that information in the current NSFNET has been available publicly for years and folks have not exploited that to unreasonably bash on anyone. Matt would like there to be some way for this to be done using RA-type infrastructure. This data should be sanitized in such a way that any bias is effectively removed. If this issue is not resolved by the IETF and the providers, it is believed that the consumer will assert their right-to-know. There was also an open issue about how any of this work might be related to IPv6. Revisit Working Group Issues There was strong consensus that IPPM work is important and should be done within IETF. There was consensus that this work will initially focus on the delivery and routing datagrams (i.e., not related services, such as NOC characteristics). However, IPPM appears to be within the charter of an existing working group, the Benchmarking Methodology Working Group (BMWG). Their charter clearly meant to include measurements on real operational networks. It may be easier to take the IPPM Effort and move it. On the other hand, the effective orientation of BMWG has clearly been towards the laboratory environment. There was some concern that the BMWG is a little too open ended, and that a focused charter for IPPM would be better. Christian Huitema was particularly encouraging on this issue. Matt called for volunteers to chair or co-chair the working group. Nobody volunteered publicly. [But there have since been several private responses. -- Matt Mathis] The group decided that a subcommittee would convene to draft a charter. The following people volunteered: Paul Love, Teunis J. Ott, John Boone and Matt. Matt hopes that the charter and the relationship to the other working groups will be finalized by the NANOG meeting on 21 May 1995.