Email Delivery

Receive new posts as email.

Email address

Syndicate this site

RSS | Atom

Contact

About This Site
Contact Us
Privacy Policy

Search


November 2010
Sun Mon Tues Wed Thurs Fri Sat
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30        

Stories by Category

Basics :: Basics
Casting :: Casting Listen In Podcasts Videocasts
Culture :: Culture Hacking
Deals :: Deals
FAQ :: FAQ
Future :: Future
Hardware :: Hardware Adapters Appliances Chips Consumer Electronics Gaming Home Entertainment Music Photography Video Gadgets Mesh Monitoring and Testing PDAs Phones Smartphones
Industry :: Industry Conferences Financial Free Health Legal Research Vendor analysis
International :: International
Media :: Media Locally cached Streaming
Metro-Scale Networks :: Metro-Scale Networks Community Networking Municipal
Network Types :: Network Types Broadband Wireless Cellular 2.5G and 3G 4G Power Line Satellite
News :: News Mainstream Media
Politics :: Politics Regulation Sock Puppets
Schedules :: Schedules
Security :: Security 802.1X
Site Specific :: Site Specific Administrative Detail April Fool's Blogging Book review Cluelessness Guest Commentary History Humor Self-Promotion Unique Wee-Fi Who's Hot Today?
Software :: Software Open Source
Spectrum :: Spectrum 60 GHz
Standards :: Standards 802.11a 802.11ac 802.11ad 802.11e 802.11g 802.11n 802.20 Bluetooth MIMO UWB WiGig WiMAX ZigBee
Transportation and Lodging :: Transportation and Lodging Air Travel Aquatic Commuting Hotels Rails
Unclassified :: Unclassified
Vertical Markets :: Vertical Markets Academia Enterprise WLAN Switches Home Hot Spot Aggregators Hot Spot Advertising Road Warrior Roaming Libraries Location Medical Public Safety Residential Rural SOHO Small-Medium Sized Business Universities Utilities wISP
Voice :: Voice

Archives

November 2010 | October 2010 | September 2010 | August 2010 | July 2010 | June 2010 | May 2010 | April 2010 | March 2010 | February 2010 | January 2010 | December 2009 | November 2009 | October 2009 | September 2009 | August 2009 | July 2009 | June 2009 | May 2009 | April 2009 | March 2009 | February 2009 | January 2009 | December 2008 | November 2008 | October 2008 | September 2008 | August 2008 | July 2008 | June 2008 | May 2008 | April 2008 | March 2008 | February 2008 | January 2008 | December 2007 | November 2007 | October 2007 | September 2007 | August 2007 | July 2007 | June 2007 | May 2007 | April 2007 | March 2007 | February 2007 | January 2007 | December 2006 | November 2006 | October 2006 | September 2006 | August 2006 | July 2006 | June 2006 | May 2006 | April 2006 | March 2006 | February 2006 | January 2006 | December 2005 | November 2005 | October 2005 | September 2005 | August 2005 | July 2005 | June 2005 | May 2005 | April 2005 | March 2005 | February 2005 | January 2005 | December 2004 | November 2004 | October 2004 | September 2004 | August 2004 | July 2004 | June 2004 | May 2004 | April 2004 | March 2004 | February 2004 | January 2004 | December 2003 | November 2003 | October 2003 | September 2003 | August 2003 | July 2003 | June 2003 | May 2003 | April 2003 | March 2003 | February 2003 | January 2003 | December 2002 | November 2002 | October 2002 | September 2002 | August 2002 | July 2002 | June 2002 | May 2002 | April 2002 | March 2002 | February 2002 | January 2002 | December 2001 | November 2001 | October 2001 | September 2001 | August 2001 | July 2001 | June 2001 | May 2001 | April 2001 |

Recent Entries

In-Flight Wi-Fi and In-Flight Bombs
Can WPA Protect against Firesheep on Same Network?
Southwest Sets In-Flight Wi-Fi at $5
Eye-Fi Adds a View for Web Access
Firesheep Makes Sidejacking Easy
Wi-Fi Direct Certification Starts
Decaf on the Starbucks Digital Network
Google Did Snag Passwords
WiMax and LTE Not Technically 4G by ITU Standards
AT&T Wi-Fi Connections Keep High Growth with Free Service

Site Philosophy

This site operates as an independent editorial operation. Advertising, sponsorships, and other non-editorial materials represent the opinions and messages of their respective origins, and not of the site operator. Part of the FM Tech advertising network.

Copyright

Entire site and all contents except otherwise noted © Copyright 2001-2010 by Glenn Fleishman. Some images ©2006 Jupiterimages Corporation. All rights reserved. Please contact us for reprint rights. Linking is, of course, free and encouraged.

Powered by
Movable Type

« Cisco Offers Soup to Nuts Citywide Networks for Providers | Main | MetroFi Partners with Microsoft for Content, Local Ads in Portland »

November 14, 2006

Trust But Verify: Measurement for Metro-Scale Networks

New measurement firm joins two others that aim to audit performance of metropolitan-scale Wi-Fi networks: Wi-Fi veterans Phil Belanger and Ken Biba today launch Novarum, a firm that will produce 10 reports per quarter on how Wi-Fi networks that span cities and counties measure up. Novarum joins a field that's not yet crowded, but has at least two competitors I spoke with that make measurement part of larger businesses. Each of the three firms has a distinct approach to taking stock of these new networks.

For some time, I've been banging the drum of network performance audits for muni-Fi, because it makes little sense for a city or civic group to ask the same group that they paid or allowed to design, build, and operate the network to also provide guidelines for evaluating that network's performance. A disinterested third party with no financial stake in the outcome of a deployment should look at tests, pilot projects, and production rollouts to determine whether coverage and performance meets the contracted specifications.

I spoke with Belanger of Novarum, and the heads of Unplugged Cities and Uptown Services to learn about their methodology in measuring Wi-Fi networks of a scale that only came into being in the last year.

(Follow the link below for the rest of the article.)

Novarum

Belanger helped bring Wi-Fi into being. His name is on the 802.11b specification, and he was an early head of the Wi-Fi Alliance, partly responsible for the Wi-Fi name. Belanger has worked for several notable firms, including Wayport, Vivato, and, most recently, BelAir. His partner in this enterprise, Ken Biba, was an early engineer of basic Internet protocols in the 1970s, and guided a number of pioneering wireless firms, most recently as the head of Vivato.

(While Vivato got a black eye from its ultimate inability to deliver a product that conformed to its early promise, both Belanger and Biba left the firm years before its collapse, at the point at which, according to private and public sources, the firm picked the wrong future course.)

Belanger said he and Biba set out to determine whether Wi-Fi networks could deliver on the metro-scale promise. His high-level marketing work at BelAir, a maker of equipment for that market, gave him insight into the overall product picture, but with many vendors and many service providers building networks, Belanger wanted to see what real networks looked like.

"If you compare it to broadband that you have in your house, if you compare to broadband in public places, how does it stack up?" he said, given that many networks are being advertised as broadband alternatives, typically as dial-up replacements, but sometimes as wireline DSL or cable replacements or backups.

Novarum's methodology involves significant wardriving across a city to determine the heat map of Wi-Fi. Belanger said they chose to measure both third-generation (3G) cellular performance and Wi-Fi performance, looking at 3G within the defined area that Wi-Fi was supposed to operate in cities they tested.

What Novarum discovered is that despite Wi-Fi's much higher potential capacity--a raw data rate of 54 Mbps and a net rate of perhaps mid-20s--3G networks and Wi-Fi networks delivered about the same throughput of several hundred kilobits per second downstream, with some large variability in coverage areas.

Novarum tested four cities initially: Anaheim, Galt, Palo Alto, and Santa Clara, Calif. Belanger said he and Biba, the hands-on operators in these early tests, chose 20 locations in a city after a wardrive to do more extensive measurement using IxChariot, a well-regarded tool for delivering bandwidth over a network and measuring results. One end point for the test was in the gear they traveled with. The other, outside of the provider's network, elsewhere on the Internet. This allowed them to measure as-delivered Internet throughput, but not to test, say, video quality across the local network or other local services.

The firm tested networks using average client equipment, including laptops and smartphones. They'll increase the suite of devices they test with, and add authentication testing in the future, too. Belanger noted that smartphones might perform better over a 3G network with less bandwidth than a laptop over a higher-speed Wi-Fi networks because the phones' applications are tuned for less throughput and less interaction.

The results surprised Belanger, he said. First, the consistent availability of true 3G cellular was highly variable across the areas they tested, with frequent dropdowns to 2G and 2.5G speeds. In three cities, only one provider in each location achieved 100-percent 3G coverage in the Wi-Fi area; in Galt, Verizon hit 100 percent, but none of it was 3G coverage. In a more urban city, they found just 34% of tested locations had 3G coverage from the best provider. "We thought that 3G was more ubiquitous; it's not."

And while Cingular had the least coverage in the cities they tested, their HSDPA technology "seems to be the one that really works the best. It's just not very available. But when it is available, it works very well," Belanger said.

On the Wi-Fi side, Belanger said that the numbers of nodes they're seeing per square mile is typically 30 or higher, with the exception of Galt, which had an outdoor coverage mandate--"paint the outside of the building"--and has about 12 nodes per square mile. He said that about two years ago when Tropos, as the dominant vendor, was suggesting as few as 20 nodes per square mile, their competitors also suggested that number. However, the industry now recognizes and recommends 30 or more; EarthLink now typically talks about 35 nodes per square mile.

To Belanger's surprise, he found that regardless of vendor or service provider, Wi-Fi networks of this scale tended to work reasonably to quite well, even at this early stage in rollouts. Belanger used to be skeptical about eqiupment from vendors other than his former employer, BelAir. No longer.

"One of the things that was validated by this first phase of the market is that -- yes, Wi-Fi can be used as a technology to do this," he said. "I am fairly enthusiastic and positive about" Wi-Fi's use for large-scale networks. "It's not about its technical superiority for" that purpose, he said. "The fact that we're using [Wi-Fi] is driven by the economics" of cost for clients and infrastructure. That is, faster networks could be offered if it made financial sense to do so.

"The networks that we've tested, they're not running at the limits of the Wi-Fi performance; they're running at the limits of the economic justification for doing them," Belanger said. Belanger said there was no way to tell the difference in testing between networks that used throttling and those that performed at lower levels.

One interesting "network" that Novarum tested was Palo Alto, which Belanger said has no official citywide network. Rather, he and Biba tested the lily-pad notion, in which enough public and private open access points could provide relatively contiguous access, although not with handoffs. (Not all metro-scale networks offer seamless handoffs, either, they found.) The service availability measure Novarum uses--not just seeing a signal, but being able to accomplish a task over the Internet--in Palo Alto was only slightly below the next lowest performing network they looked at.

Unintentionally open access points are becoming a thing of the past, Belanger said, something that's been confirmed by a number of other sources that scan large numbers of APs in metropolitan areas. Belanger said that they found that 60 to 65 percent of APs they saw were secured, with about half of the remaining networks clearly designed for public access.

Belanger said that service availability was clearly something that should concern service providers. MetroFi's Santa Clara network had just 55 percent service availability in Novarum's testing, while EarthLink scored the highest with 72 percent in Anaheim tests. Santa Clara's network is one of the largest older networks, and was established without municipal involvement. Anaheim is one of the newest networks, and was bid out by the city to EarthLink; they tested the network after its first phase was deployed and tested only the stated working area.

Novarum plans to offer paid subscriptions for its blind studies to analysts and others that want the 10 reports that they produce each quarter, but they will also be offering marketing, troubleshooting, and other consulting to municipalities and service providers, as well as custom audits.

Belanger said the firm would move into testing VoIP as service providers start promoting voice over Wi-Fi as an offering. He said that as they are contracted directly by cities or service providers, they would almost certainly test from Wi-Fi adapters through to end points within the network to better identify bottlenecks.

Unplugged Cities

Based in the Twin Cities region of Minnesota, Unplugged Cities' most recent press has to do with their design of a network for St. Louis Park, an area suburb; the city just gave its go-ahead for the network, which will be municipally funded and owned. The design and operation of Wi-Fi networks is part of this firm's purview, although they have developed a measurement practice because of the obvious need for that sort of service. (In St. Louis Park, they are the local contractor for ARINC, a multinational firm that had the winning bid.)

Chad Peterson, the head of the firm, said that Unplugged Cities works closely with the network's owner, which has included cities for which they have built networks. They ask, "What do you want to measure and what are you looking to get out of the network?" said Henry Camacho, director of technology for the company. This can include tasks like delivering streaming video from crime scenes to a headquarters or pushing voice applications over Wi-Fi, especially in cities that combine public safety and public access on the same networks.

Camacho said that the company develops a list of the most challenging locations on a network to be tested, as well as some places that the service provider believes it has its best performance. They place hardware at the egress point, where traffic from a mesh cluster or node leaves the public access spectrum to provide testing across the local link. "I always do end up going to some access points that the vendor doesn't want me to" in order to get the best picture of the network's performance.

The company doesn't release results, Peterson said, except to their customer and the service provider as they don't want to single out vendors or providers; they also prefer to avoid publicizing faults of metro-scale equipment makers. "That's one reason why we try not to divulge the access point providers that we've tested on or the cities or networks we've tested on," Peterson said. We're not trying to give a head's up to one company or not. We're trying to test the network as a whole. We're trying to provide objective results."

Unplugged Cities doesn't do the kind of blind testing that Novarum expects to release in great quantities. "When you test that, going in blind, it's not really effective," Camacho said. "I prefer to create the relationship with the vendor, get the testing end points into the right spot in the network--then have a collaborative process to deploy the testing."

The company hasn't found any great surprises in the several cities' networks they tested so far, but note that they believe the number of nodes necessary to achieve complete coverage is far higher than typically cited. In St. Louis Park, they believe they may require 50 nodes per square mile due to the dense canopy of trees in that city.

But Peterson notes, "We shy away from just saying, or generalizing the number of access points per square mile. There really is no average deployment. Every city is different, and inside of every city, they have all their own individual areas that react differently."

Uptown Services

Neil Shaw started working with municipalities on broadband back in 1998, when fiber optic and coaxial were the only reasonable choices. His company has consulted on several fiber to the home (FTTH) projects in smaller cities and towns, where the scale and nature of FTTH makes sense for moving a city from mid-20th to early 21st century networking. In the last three years, they've helped four FTTP networks in Oklahoma and Tennessee come into being.

Shaw found that the rise of municipal wireless carried with it the need by cities he'd worked with for an independent evaluation of pilot projects and deployed performance. Shaw's firm doesn't design or build wireless networks. This year, he worked with a developer to create a measurement package that would allow him to validate networks. The service is called, cleverly, WiVeriFi.

For sampling, Shaw drove every street in three cities in California: Lompoc, Mountain View, and Sunnyvale. Shaw said that his firm would provide every-100-foot metrics of throughput, latency, and coverage and turn that into a geographical color-coded heatmap for visualization.

His testing methodology involves tracking metrics by retrieving the gateway page or authentication page on the local network. Shaw might test beyond that point in the future, but he thought this was the only reliable measure of the local link without access to the network. "I'm guessing the bottlenecks are going to be on the wireless side, not on the Internet side," he said. While his firm might carry out blind testing, he expects the majority of business to involve cooperation with the service provider.

As both other firms interviewed for this article noted, measuring indoor performance will be a bear, because it involves gaining access to typical locations, often in residential neighborhoods to perform testing. Shaw was involved in third-party validation of Metricom's Ricochet network where they had to test 500 indoor points, so he's well familiar with the drawbacks and complications. "We are going to be developing an indoor testing scheme, both for cooperative and for blind," he said, because of the utility of that information.

While Shaw remained close to the vest about particular results, he did allow that Lompoc's network performed well. Across three cities, he found performance ranging from 273 Kbps to 1.1 Mbps and coverage from 69 percent to 90 percent. In the 90-percent covered city, coverage was consistent in areas, but in the lowest-covered town, consistency of coverage was "all over the place."

"The results of our testing of the three systems that we tested validated the need for this testing," Shaw said. Meaning, I have to conclude, that he felt the level of service provided wasn't up to what he would recommend as acceptable were he evaluating those networks for his clients.

2 Comments

Hey Glen! Thanks for the post, very insightful.

You report: "In a more urban city, they found just 34% of tested locations had 3G coverage from the best provider. "We thought that 3G was more ubiquitous; it's not".

Wow, I am amazed. Things seem to be different in Europe. In virtually all cities >100.000 people, 3G coverage from several operators is close to being everywhere. Especially in Germany and the U.K., 3G indoor coverage is also found from several operators in towns with less than 10.000 inhabitants. But then, 3G UMTS/HSDPA is used in Europe for plain old voice calls as well.

Thanks for the post!
Martin

It would be nice to know which service providers gear was used in these municipalities and possibly compare density requirements with the manufacturers technical design of their equipment. Some of the equipment we have looked at vary drastically in antenna design (sectorized vs omnidirectional) which affects the number of users per radios and effective throughput when the nodes are saturated.

There is a lot of information that is hard to drag out of people because everyone wants to protect their "edge" in design. I think the picture will be clearer as time goes on and more users and municipalities post information to the web.

Great article.

[Editor's note: Novarum has all this information, but they're a firm that will make their money from selling information, which is perfectly reasonable. So they'll have to figure out precisely what they want to sell from these reports for the kind of needs you and others have! -gf]