Benchmarking Methodology WG Minutes WG Chair: Kevin Dubray Minutes reported by Kevin Dubray. The BMWG met at the 49th IETF in San Diego, CA., on Monday, December 11, 2000. The proposed agenda: 1. Administration/Agenda tweaking 2. Terminology for Forwarding Information Base (FIB) based Router Performance Benchmarking. 3. Benchmarking DiffServ Mechanisms was amended to include an update on the status of the Individually Submitted I-Ds on Resource Reservation benchmarking. 1. Administration. Dubray announced that I-Ds and came back from IESG review and that they were being modified with minor edits. I-D was in WG Last Call. It was announced that the BMWG would undertake the new work item, DiffServ Mechanisms Benchmarking. The Multicast Benchmarking Methodology I-D had a few outstanding issues - it was asked that wording for contentious areas be offered so that the draft could be moved to last call. 2. Terminology for Forwarding Information Base (FIB) based Router Performance Benchmarking. Guy Trotter led a discussion on . It was thought that the notion of "fully meshed" was a methodological concern and should be addressed in the subsequent methodology document, as was specification of an aggregation heuristic. It was noted that this draft will target IPv4 exclusively; IPv6 could be addressed in a supplemental work. There was discussion regarding the desire to possibly include mechanisms to characterize the effect of prefix length on forwarding performance. A table of contents was suggested as a good thing towards improving the I-D; explicit identification of metrics from the supporting terminology was thought to be desirable, too. There was discussion on the attributes to consider in defining the term "FIB". A question was asked whether the author considered employing related MIBs in the benchmarks. It was thought that the question was interesting, but may be a more invasive (i.e., less than a black box) examination of the DUT. It was asked the meeting attendees pose their thoughts/comments/questions to the BMWG mailing list. Guy's presentation is appended to the Proceedings. 3. Benchmarking DiffServ Mechanisms Jerry Perser presented some thoughts on issues and metrics in benchmarking DiffServ mechanisms. While the presentation addressed measurements of congested vs. uncongested DUTs, the bulk of the discussion revolved around the notion of measuring DUT queue size. Black box or white box test? One queue or N queues? The discussion wasn't conclusive; it was suggested that further discussion be taken to the mailing list. A question was asked regarding the effort's intent to derive metrics that would give insight into a device's ability to enforce contracts or SLAs. It was thought that this is a more end-to-end thing - however a DUT's characterization might be a component that provides insight into the "bigger picture." Jerry's presentation can be found in the Proceedings. 4. Resource Reservation Benchmarking Krisztian Nemeth gave the group a short overview of the effort, its related models, and some preliminary findings. He then updated the group as to the history and status of the relevant individual I-Ds. Initially, was issued and presented to the BMWG in Pittsburgh; there was some preliminary interest and the authors followed the Pittsburgh group's recommendation to break the draft into separate terminology and methodology documents. The I-Ds, and were sent to the I-D secretariat, but only the methodology I-D was issued. Krisztian noted that responses to the work on the mailing list was light, but he again asked the BMWG to undertake this work. The chairperson asked Krisztian to check with the I-D secretariat and have the terminology draft issued. Once the I-D is issued, a WG Last Call on the New Work Proposal would be announced. Krisztian's presentation can be found in the Proceedings, too. 5. Goals for the Next Period. 5.1 Complete WG Last Call on the ATM methodology I-D; 5.2 Bring the multicast benchmarking terminology I-D to WG Last Call; 5.3 Advance the FIB terminology benchmarking draft. 5.4 Advance the Firewall methodology benchmarking draft. 5.5 Revisit the Router Benchmarking Framework document. 5.6 Issue the first WG I-D on DiffServ Mechanism Benchmarking Terminology in February. 5.7 Issue a WG Last Call on a New Work Proposal regarding Resource Reservation Benchmarking.