dailywireless.org » Portland MetroFi Gets Okay

Portland MetroFi Gets Okay

MetroFi of Portland got good marks today from Uptown Services, the contractor hired by the city to perform a Proof of Performance test.

Logan Kleier, the city’s Project Manager, for Oregon’s Unwire Portland (left) is now recommending acceptance of Phase I of the network (with a couple of minor fixes). That should allow the build out to continue from its current state of 70 access points, covering some 2.5 miles, to several hundred more access points covering a much larger area.

Portland’s Proof of Concept tests (Summary and Analysis and Certificate of Acceptance Recommendation) were generally favorable.

The initial tests were performed on the network during February and March 2007 with retesting of some items taking place in April, 2007.

Throughput:
Uptown Services found that 99% (69 of 70) of access points (APs) provided at least 1 Mbps of downstream connectivity and 100% (all 70) of APs provided at least 256 Kbps of upstream connectivity. We also found that MetroFi’s POC network supports two 1 Mbps downstream connections and one 256Kbps upstream connection simultaneously on a single 802.11 access point.

Availability:
For the final results of the Availability Testing based on 711 hours of continuous availability testing, Uptown Services found that the network provides a connection (of at least 64 Kbps upstream and downstream) 91% of the time (38773 of 42587 minutes tested).
This testing also shows that the network has the ability to deliver packets between Uptown Services’ client device and the MetroFi Point of Presence (POP) in less than or equal to 100 milliseconds (i.e., a round trip time less than equal to 200ms) 89% of the time (37846 of 42610 minutes tested). The average downstream throughput for these tests was 1336 Kbps and the average upstream throughput was 669 Kbps. The downstream throughput exceeded 1 Mbps in 63% of the tests and exceeded 256 Kbps in 92% of the tests. The upstream throughput exceeded 1 Mbps in 28% of the tests and exceeded 256 Kbps in 74% of the tests.
Based on the completed drive testing, Uptown Services found that 92% of the (58 of 63) APs where at least two downstream and two upstream tests were performed within 250 feet of the AP delivered at least one full speed connection (1Mbps downstream/256 upstream). Of 730 downstream throughput tests performed within 250 feet of an AP, 539 (74%) met the downstream throughput requirement.

Of 614 upstream throughput tests performed within 250 feet of an AP, 517 (84%) met the upstream throughput requirement. Also based on the drive testing, Uptown Services has found that the network provided an average signal strength meeting the City’s minimum requirement of –79 dBm in 95% of the (2032 of 2147) locations in the POC area where an outdoor test for signal level was done within 500 ft of an AP.

Security:
MetroFi’s network supports 802.11i security features via WPA2. Uptown Services tests of the MetroFi-Premium SSID show that the MetroFi network supported 802.11i and 802.1x authentication and allowed us to connect at 12 different APs. MetroFi’s POC network supports at least 2 SSIDs.

Logan Kleier, the city’s Project Manager, for Unwire Portland, then wrote the city’s Chief Technical Officer, Matt Lampe, recommending a Certificate of Acceptance:

Based on Uptown Services, City of Portland, and MetroFi data, I recommend that the Bureau of Technology Services grant MetroFi its Certificate of Acceptance for its proof of concept network. This recommendation is based on standards listed in the RFP 104112: A Citywide Broadband Wireless System as well as RFP 106636: Unwire Portland: Proof of Concept Network Testing which include network performance criteria relating to:

Throughput: All access points achieve at least one 1 Mbps downstream connection and at least one 256 Kbps upstream connection.

Availability: 90% of the outdoor areas of the proof of concept territory receive a -79dBm signal or greater 99% uptime of the MetroFi network. Latency of less than 100 milliseconds between client device and MetroFi’s Portland POP (Point of Presence)

Security: 802.11i support, Multiple SSID support, Non-broadcast SSID support

MetroFi’s proof of concept network is in near total compliance with the performance standards listed above.
MetroFi’s ability to correct the substandard throughput performance of one access point at 24th and Couch is reliant on two factors beyond their control and (2) the activation of their Bandwidth Aggregation Point (BAP) at 4400 NE Broadway.
Given these factors and the contract’s language regarding proof of concept performance, I recommend that the City grant MetroFi a Certificate of Acceptance and re-test the throughput on the 24th and Couch access point and the 4400 NE Broadway BAP. . .

The results from Uptown Services seem in stark contrast to the findings of two members of Personal Telco, some two weeks ago. The non-profit organization, which provides free WiFi hotspot service by helping hundreds of individuals and dozens of small businesses to create “free” public access points, could only connect to MetroFi about half the time.

Personal Telco’s Russell Senior and Caleb Phillips describe their independent performance evaluation of MetroFi Portland at their website; UnwirePdx-watch.org. They were not the only ones to be skeptical of MetroFi’s coverage — just last week The Oregonian ran a snarky review.

A reaction from PTP’s Michael Weinberg follows:

Dear Logan,
I’m writing to express my concern over the City’s recent sign-off on
the MetroFi Proof of Concept deployment. Based on the only publicly
released document from Uptown, there is little reason for the public
to have faith in the City’s decision to issue the Certificate of
Acceptance.
As someone who has worked with outdoor wireless network design and
deployment for many years, initially as a volunteer with Personal
Telco and later as the administrator for that group’s first grant, a
wireless network along Mississippi Avenue; as a private contractor
deploying networks between buildings in downtown Portland and most
recently at Stephouse Networks, an ISP operating a number of wireless
services in downtown Portland, St. Johns and the N. Marine Dr. area,
I believe I can speak with some authority about the issues facing
outdoor, urban wireless deployment, particularly those delivering
service over the 2.4GHz band.
Throughput:
The Uptown Services Executive Summary, the only document available to
the public from Uptown, begins by stating that 69 of 70 tested access
points were shown to be capable of passing downstream traffic at
1Mbps and 70 of 70 were able to pass upstream traffic at 256 kbps.
The opening figure for bandwidth capability is not indicative of
performance over the 802.11g tier, where users experience it, but
rather is an assessment of the performance of the 802.11a backhaul.
Any client capable of connecting to an 802.11g AP ought to see these
speeds between the client and AP.
Uptown goes on to state that their testing indicated that this level
of performance was supported for 2 downstream clients, but only 1
upstream client, suggesting that the advertised performance is only
available when a single user is on each AP. In short, the throughput
assessment, as described in the Executive Summary, appears to ignore
the presumable average load on the network, and instead presents a
standard for success that virtually any functioning network should be
able to meet.
Availability:
As many formal and informal usage tests have shown, when a reliable
connection to the MetroFi network is found, it tends to remain
reliable, as such, the 91% figure, as presented, is neither
surprising nor impressive. It is entirely conceivable that a single
device, placed in a location that had reliable coverage and connected
continuously to the network, would have service availability of 91%
or more. However, a single device connected in a location chosen for
network reliability, is neither a rigorous nor a scientific means of
testing average network performance.
The Uptown “drive testing”, like the throughput testing before it,
presents success criteria that seem to ignore real-world use to the
point of being meaningless. The ability of a single user to achieve
the advertised performance is of little value if performance degrades
with the second user on any given AP.
Of all the availability results, the most unsettling is the -79dBm
figure, upon which network coverage has been judged. As anyone
familiar with wireless networking can attest, a single signal
strength reading is absolutely meaningless. All that this assessment
means is that the MetroFi network can be “heard” at a certain level
in 95% of the coverage area; it provides no insight whatsoever into
connection availability. This is especially troubling, since MetroFi
has repeatedly blamed end-users (and their hardware) for the
inability to connect to the network. The inclusion of this statistic,
is irresponsible and misleading without additional information about
the strength or other technical assessment of the receiving equipment.
I can only presume that prior to issuing the Certificate of
Acceptance, you received a more in-depth report from Uptown Services
that clarifies some or all of my concerns regarding the testing
methodology. It would behoove the City to release all or part of that
report to demonstrate that the decision to sign-off on the MetroFi
PoC was based on appropriate data and strong testing methodologies.
The Executive Summary raises serious questions about this, especially
in light of many assessments from frustrated users and the excellent
research by Russell Senior and Caleb Phillips which suggest that
usable coverage and performance are highly variable and MetroFi’s own
statements regarding the probable need for additional hardware in
order to connect.
Thank you for your time,
Michael Weinberg

Uptown Services of Colorado was paid $22,000 to test for minimum speeds, but also stated its results were not a guarantee of users to be able to sign on. Uptown Services has not finished their report, according Kleier. It should have a more detailed report in the next few weeks — but that report will not include the raw data. That belongs to Uptown Services, said Kleier, in an email response to a DailyWireless query.

How does he like his job as the director of Unwired Portland?

“The job is fascinating. It’s something new every week and it’s never just about RF. Some weeks I spend all my time on pole attachment issues, other weeks on zoning and permitting. There’s a surprising variety to the work involved in making this project happen”.