25 Aug Flawed Routers Flood UW Server
Low-cost Internet routers are the source of problem
Madison, WI- Over 2,200 computers on the University of Wisconsin-Madison campus were infected with the latest e-mail virus last week. At the same time, it was revealed that beginning in May 2003, UW-Madison discovered that it was the recipient of a continuous large scale flood of inbound Internet traffic destined for one of the campus’ public Network Time Protocol (NTP) servers. NTP servers are used to synchronize computer clocks on the Internet. The flood traffic rate was hundreds-of-thousands of packets-per-second, and hundreds of megabits-per-second. The problems are far from being resolved.
The university has determined the sources of this flooding are literally hundreds of thousands of real Internet hosts throughout the world. What was thought to be a malicious distributed denial-of-service (DDoS) attack, turned out to be a serious flaw in the design of hundreds of thousands of NetGear platinum products, including the RP614 and MR814. These are low-cost Internet routers targeted for residential use. At first the NetGear product support team was very unresponsive, according to the report. The unexpected flaw found in NetGear routers will cause significant IT problems for UW-Madison for years to come.
This details were revealed by David Plonka, a systems programmer with the University of Wisconsin, on August 21 at a meeting of the Madison Area Systems Administrators Guild (Mad- SAGE) as well as on a posting on the UW’s Computer Science web site at http://www.cs.wisc.edu/%7Eplonka/netgear-sntp The document includes the public disclosure of these products’ serious design flaws and how the UW, NetGear and Internet standards groups are attempting to address and solve this issue. A number of actions items have been called for:
1) Fixing the SNTP client
2) Proposals for new network operational options
3) A campaign to notify the Internet community
4) Clarification of Internet best practices and protocol standards
The problem, according to the document, is that there’s a flawed NetGear SNTP client implementation. The author, Dave Plonka, claims that 500,000 unique NetGear sources queried the Wisconsin time server in just one day, while NetGear has reported that 707,147 of its products might be affected by the problem.
Response to Plonka’s Internet posting has been strong. “The Community of users are applauding the efforts of the perpetrator and the victim that worked together on the solution,” added Plonka. “The big question is how do you notify the customer base?” Plonka suggested that a product recall would not be practical. “Both NetGear and other members of the review team felt that it was unlikely that all but a very small subset of the owners would return the affected device since they appear to be working fine. Also, very few customers have registered these products with the manufacturer, so it is impractical to contact them,” Plonka said.
Annie Stunden, CIO for the University of Wisconsin Information Systems Group said, “As soon as the issue was identified, NetGear worked with us to develop remedies for the problem. NetGear made changes to their newly manufactured routers as soon as they became aware of the issue. NetGear is supplying both technical support and money to help find a remedy for the routers that are already installed. The problem not only affects the University of Wisconsin, but the entire Internet community as it relates to standards for Internet Time Servers. Dave Plonka has done some great research and come up with some great solutions,” Stunden said
Doug Hagan, a spokesman for NetGear said, “We are fully cooperating with the university to find solutions for the problem including improving our products and how they interface with public access servers. We want to take a leadership role and do what is right for our customers and the Internet community as a whole,” Hagan said.
According to Plonka, the exposure of this issue at the UW serves a larger purpose. “This is a serious issue for the Internet in general and more specifically to vendors and the international internet community,” he said.
Plonka also points a finger at the IT press which he says have provided awards and favorable reviews for these products and yet there is no testing for these types of issues and the problem has not been revealed to their readers.
The impact of this product flaw is compounded by the fact that hundreds of thousands of home and small business users own these routers and are unaware of the flaw and the problem it is causing the University of Wisconsin- Madison. “To most users there is no problem, but in Europe where broadband users pay for data usage and not a flat monthly fee, the problem is costing users considerable dollars,” said Plonka. “ We have not been able to fully calculate the financial impact of this flaw yet.”
As of August 2003, the University is making its best efforts to service NetGear time requests. Users of affected products should not normally notice any problems due to this flaw.
A NetGear support page for their RP614 router, points out that some products use public NTP sources that can cause “spikes,” and gives a firmware fix for a series of products.