Slashdot Mirror
Testing and Mapping a Cellular Data Network?
bgsneeze writes "In order to resolve an ongoing issue with a vendor, I have been trying to find a way to test different 3G data devices empirically. I would like to be able to chart signal strength, latency, and bandwidth. I would also like to create a map of the coverage area. I have a test 3G card from three different providers. I would like to be able to travel with the setup to several different locations and run tests. What software or techniques would Slashdotters use to test the different devices? Are there any free or open source software packages that will do this?"
Do you mean a 3G SIM for use in mobile devices?
If you mean the USB/PC card 3G adapters for computers, I doubt you'll be able to run them on Linux without using WINE or a VM or whatever, which may interfere with latency readings. Those adapters require you to install software(ATT Connection manager in their case) that's only supported on Windows and Mac, and that software is required to "authenticate" your computer and use the network.
Anecdote: I had ATT 3G in the metro San Diego area around 2007. Speeds and coverage were decent enough to browse the web, but too spotty for anything hardcore.
I've also said this a million times, but ATT connection manager takes 10 minutes to install and spins the hard drive like a defrag...waaaay overkill given the reported footprint.
Note: too lazy to dig up hacks and testamonials from Google.
Obviously this can only be used for terrorism.
http://www.cellularmaps.com/coverage_finder.shtml
http://www.mytruecoverage.com/ is a good site to show coverage. You can install an app on your phone then manually run tests. The results usually take 12-36 hours to post to the website.
I assume that you are not working with a network operator? They have plenty of tools to do drives and plot out signal. Also, chip makers have these tools. I used to work for a large mobile phone chip maker and we had internal test setups that can be driven around in a car to plot signal, etc. Also, we developed a small device FPGA based device that could be tuned to various channels and recover the "sync" and "paging" channels of a CDMA system. You could do the same for UMTS, EVDO, etc. If you just wanted raw signal strength, all you really need is dBm for whatever provider and tower you are looking at. You could then compare strength between providers. However, this isn't really apples to apples. CDMA based systems can literally pick up signal out of horrid conditions and low signal powers... much than, say, a GSM based system. So if you were comparing purely 3G to 3G, that would be valid, but not in the 2G realm. Just get a phone for each provider, turn on the engineering mode to get the phone's reported dBm reading, and then plot the values as you drive around....
Why do it yourself when you can just wait for Google to announce they did it accidentally ;-)
Empirical testing with wireless CELLULAR networks can be very tricky; A few things that you will need to keep in mind is that for testing, you want to make sure your transceiver setup is perfectly reproducable; Same card, same ANTENNA, same position of antenna. When performing your testing, your signal strength will depend on several factors: Distance from the site, antenna type/gain, and specifically what sector/node on the site you happen to be on. while driving in a straight line, you may find that you approach steep nulls near the border of a cells sector boundary. Alot depends on the ability of your particular card to handle the handoff between sectors/sites.
Your latency measurements will also vary according to the individual usage of the sector/site that you are currently on, and additionally vary with time and also variable bandwidth allocation to the sites from the main switch.
There are quite a few test sets and software suited that are commercially available and are tailored for this use, and are used heavily by the mobile data/cellular industry in their drive testing and coverage verification methodologies.
Good luck to you on your testing.
--ToO
What software or techniques would Slashdotters use to test the different devices? Are there any free or open source software packages that will do this?
I would suggest the "grad student technique".
Find a nearby university and convince a professor to lend some (under)grad students to your cause.
It helps if you have friends that can refer you to professors/students who'd be interested in the challenge.
P.S. The only thing better than the "grad student technique" is the "summer intern technique"
[Fuck Beta]
o0t!
Many cellphones have test/debug modes that will give you all kinds of data regarding signal quality and strength. Can't say for bandwidth / latency though. You could check some of the phone hacking forums for a debug mode / debug drivers for your card(s), and see what kind of data you could get to.
There's plenty of network monitoring tools around.
Signal strenght might be harder if the manufacturer of the device doesn't give easy access to that data (many cellphones can be switched into diagnostic mode; or some smartphone OSes have diagnostic software available). Maybe you can find some electrosensitive person?
One that hath name thou can not otter
There is the Placelab.org project, but their mailing list has died down over the years.
Zhrodague.net - I do projects and stuff too.
Can you ping me yet?
How about now?
At least, this is what I see working with SNMP data coming off dumps from the cellular base transceiver stations attached to our towers.
Every 15 minutes, every GSM handset measures the perceived strength of the tower signal it is using, this is reported to the tower, and we record them all. Also every 15 minutes, the strength of the handset signal, from the perspective of the tower is sampled and recorded.
These readings go into buckets, for example if a reading showed a -78 dbm signal, it goes in the -78dbm bucket. Before long, a histogram can be generated with the datain the buckets, and we can see typical distribution for receive performance - for both handsets and the tower. Different towers have very different "signatures" in this data.
You may go around sampling receive performance, (which would be interesting) but I don't think you'll be able to map how well the cellular system is receiving from you.
Most providers use modelled data to estimate signal strength as it's too expensive to actually measure every point within their coverage. You might be able to obtain and use modelled coverage (which is mostly based on putting virtual towers into a digital terrain model) and undertake some validation. This would provide a meaure of the optimism usually present in modelled coverage. Actual coverage is usually going to be less due to buildings, weather, trees and other rubbish you rarely see in a DTM. Validation of published coverage maps would also require far fewer observations ("field work") than creating your own map from scratch.
Suck the whole lot into Quantum GIS (FOSS) and generate a text dump of obesrved/modelled coverage to analyse statistically in R (also FOSS).
Xix.
"Everything is adjustable, provided you have the right tools"
After what happened to wardriving be sure you legally have your butt covered. Especially if you prove the network sucks >.>
I would suggest learning that it's pointless to test for something you can't fix. You'll waste much less time and energy.
Your provider won't care even a Tiny Iota about your results. If you show that their network is fine, they'll ignore you. Even if you prove their network is not fine and all the packets are routed though a Sinclair ZX80 in Clive's basement they'll still ignore you. If you're really lucky, you might get a nice, polite "Fuck you very much, We're not fixing it."
Knowledge is knowing that they have 3000ms latency between nodes. Wisdom is knowing that the only thing you can do is vote with your wallet.
I wouldn't worry about charting the signal strength for 3G. You can be in a densely populated area showing five bars of 3G and your speed and latency can still be dog shit depending on how many people are hitting the tower, similar to your cable modem. It might be worth it to record whether or not you have 3G just to help map out your general coverage, but that doesn't mean you'll have great speed. Although, you can find something like that here.
As for speed I like to use a util called iperf for measuring speed from one device to another across a network. You may have to open ports on your firewall or setup a VPN, which will add unwanted overhead, but you will get a good idea of which carriers have the best speeds. You can also run the simple tests using other websites like here or here.
Loading...
Industry standard?
http://www.qacafe.com/cdrouter
I've done similar things using Linux. Basically, when you hook up the 3G card, it presents itself as an old-school modem (although it's over USB). You can then use a program such as wvDial or Minicom to send AT commands (if you want to do a connect, you need to find out the number to dial. You can find this on your carrier's website).
A lot of wireless modems have AT commands for signal strength and other info, you can do a Google search for documentation about AT commands for your specific modem. A warning: writing a robust, industrial quality parser for AT commands is one of the most difficult things I've ever programmed. A lot of people take the easy way out, and if a command doesn't work, they just send it a couple more times until it does (which is good enough for many tasks).
If that doesn't provide enough info for you, and you have money, then you should contact the manufacturer. Some modems, like Qualcomm modems, have a separate diagnostic port in a proprietary protocol that provides way more information than you ever dreamed existed, but you will have to pay to find out the protocol.
Qxe4
If you're on Linux you can use gpsd for interacting with your GPS. If you're on Windows using the new Location API in Win7 would be nice, but, more realistically, you'll be listening on the serial port your GPS uses and parsing NMEA packets. No worries, this isn't that hard to do. After a either a certain interval or time or distance traveled, send a few pings to a server and download a reference file. Dump your data into a CSV or use OGR to make a shapefile, KML, PostGIS layer, whatever.
http://www.sensorly.com/ Run it on your phone manually (or have it triggered intermittently). The 3G, EDGE and wifi coverage that your phone detects is uploaded to the central server and within a minute your phone receive the updated maps. You can only contribute to the maps for your phone's carrier, but you can view the maps for all carriers.
I'm a bit rusty, but since your network is probably TCP/IP, can't you do this with standard TCP/IP tools?
The only thing that could be a problem is measuring "signal strength". However, but a lot of mobile chipsets work under Linux. Even tools running under Windows, there may be access to this information (netstumbler)
For latency, you could just use ping:
Also, take a look at ping -f (use with caution) and traceroute.
For bandwidth measurement, setup an FTP server and script automated downloads from your clients using a .netrc file
http://www.stratigery.com/scripting.ftp.html
Or just use a command line HTTP client like wcget -- depending on your requirement.
You can use Gnuplot to chart data such as signal strength or error rates by location. For example, if you time sync GPS data and your statistics you can emit surface and contour maps (such as the second glass.dat example near the bottom).
_most_, if not ALL operators use TEMS at some point. All the equipment vendors I've dealt with won't even respond to a radio related trouble ticket unless there's a TEMS log.
In the end, the project was cancelled before we got a chance to get to test 3G coverage. But we did get to think about it. Our customer was a fleet of cargo ships, going through a fixed path on the Rio de la Plata (River Plate). Basically, we were going to install our CCTV system in there, and have it push images and other information to our servers whenever it had signal. We wanted to know approximately in what areas of the river we would have signal. We were going to base our system on the Vodafone mobile connect driver. It's a set of Python scripts. Of course, it communicates with the modem using simple AT commands. It's released under the GPL. It is capable of measuring signal, sending and receiving text messages, and other nice stuff (like, well, actually dialing and calling PPP to stablish the connection). We had it working with several Huawei devices, but I know it works with other brands too.
Our idea was to modify this scripts so that they would try to maintain a connection, auto-dial every time it disconnected, and log the signal at certain intervals to a MySQL DB. We were also going to run download tests all the time automatically. Since there was no chance we would go on the ships with the devices (the ships were cargo ships that transported and extracted sand, and there weren't very comfortable, not to mention their average trip was at least ~72 hs.), so we wanted to do all of this automatically. The devices would also inform their IP to a web service every time their IP changed, so we could SSH in the machine running this tests in case we needed to change something.
We were going to add a GPS to this system, that would also log its position at certain intervals, so that we could then generate a color-coded signal map.
I hope this helps. It's really fairly simple. I would be happy to provide you with source code, but we didn't get that far into the project as to produce actual source code, since the customer changed his mind due to budget restrictions real early. Feel free to contact me if you have other questions {almafuerte (at) gmail (dot) com}
WTF am I doing replying to an AC at 5 A.M on a Friday night?
http://www.cs.wisc.edu/~suman/pubs/citywide.pdf
ABSTRACT
We describe our experiences in building a city-wide infrastructure for wide-area wireless experimentation. Our infrastructure has two components — (i) a vehicular testbed consisting of wireless nodes, each equipped with both cellular (EV-DO) and WiFi interfaces, and mounted on city buses plying in Madison, Wisconsin, and (ii) a software platform to utilize these testbed nodes to continuously monitor and characterize performance of large scale wireless networks, such as city-wide mesh networks, unplanned deployments of WiFi hotspots, and cellular networks. Beyond our initial eorts in building and deploying this infrastructure, we have also utilized it to gain some initial understanding of the diversity of user experience in large-scale wireless networks, especially under various mobility scenarios. Since our vehicle-mounted testbed nodes have fairly deterministic mobility patterns, they provide us with much needed performance data on parameters such as RF coverage and available bandwidth, as well as quantify the impact of mobility on performance. We use our initial measurements from this testbed to showcase its ability to provide an ecient, low-cost, and robust method to monitor our target wireless networks. These initial measurements also highlight the challenges we face as we continue to expand this infrastructure. We discuss what these challenges are and how we intend to address them.
one possible solution:
Download the android SDK; write an app; run it in the emulator that comes with the SDK.
I'm not sure how much work it'd be to tie your 3G card(s) into the emulator (that comes with the SDK), but it's possible.
Linux would be my first choice, but the SDK also runs on windows or mac os.
Bonus for getting a useful app included in the app store.
While the goal is an admirable one, there are just too many variables to even begin to consider really doing this.
First off, you aren't working with a stable environment. There are changes being made to the cellular infrastructure all the time. Changes to settings in the towers, changes to the tower equipment and addition of new towers. You will not be informed about any of this making your information out of date almost immediately.
Next there is the problem that most throughput problems are going to be caused by overcommitment of resources, not lack of radio coverage. You will find that there are patterns to this, but again this sort of thing changes over time.
Essentially, what is wrong with the carrier coverage maps is they aren't detailed enough and do not account for throughput problems. You are going to have the same problem, only perhaps worse. In short, I would expect any data gathered in this manner to be extremely unreliable because you don't even know what variables you are missing.
...or "Spirent UTS"
I used to do this kind of programming all the time for Qualcomm and Kyocera Wireless. Cell phones have a User Terminal Diagnostic Monitor running on them constantly that can be used to get almost all data in realtime out of the phone through it's data port. There are a couple of areas of memory that are write only for things like passwords, but you have access to pretty much everything else.
A bigger number on the same device probably means you have a better signal. A 'two' on one device could be equivalent to a 'five' on another or a 'one' on a third. There is no standard that is used across the industry, or even across all devices from a given manufacturer.
When a device manufacturer gets customer reviews that say "I only get one bar with your phone but two from company X" the device manufacturer can either try and explain repeatedly that their one bar is better than X's two bars and that unlike X you can still make phone calls on our device on one bar. Or they can just double the number of bars reported on the next model so they don't look worse than X.
Which do you think they do?
To do a real test you need to use a constant antenna and location, attenuating the signal gradually until each device stops functioning. The amount of attenuation it can take is a crude indicator of the quality of the radio.
1) Ping Test. A simple ping to the nearest visible IP address, usually the GGSN, will give you the round trip time for a session in progress. You are looking for the variability. A simple ping -t command for a few minutes and plot the results in excel. You can vary the packet size, but the two tests below give a more accurate test for throughput. Protip: if you want to measure radio channel activation time too then you need to know a few network parameters, specifically the radio TBF timers. You would only do one ping at an interval greater than the timer so that each new ping needs a new radio resource allocation request. The round trip times for the first ping will always be longer, and recording variability in the allocation requests is useful if you are the network, not so much for users.
2) TCP Test. Use an FTP session download a file of known size to get the actual time taken. Then you need to upload that file back to a separate directory and compare the two (or just see if it opens). You actually have to do the math to get the bits per second, neither the windows or Linux native clients show it correctly. bps = bytes_recieved * 8 / seconds. Protip: you need to test several file sizes to get an accurate picture. I'd suggest one run would up/download several 300k and 1Mb files and one 10Mb file. To complete the test in the field you do need to check the upload is successful using a terminal session, but remember not to have the session open using the test equipment as it will bias the test.
3) UDP Test. For assessing the actual performance of the radio network itself the only test that has any relevance is a unidirectional UDP stream because the radio resources are allocated unidirectionally and have asymmetric speeds. TCP is not a test of the radio network itself as the uplink channel causes throttling on the downlink due to the slower uplink. You need a utility that sends UDP packets of a certain size, and a utility to receive them. Ideally the packets have sequentially numbered content because what you are looking for is out-of-sequence delivery. You measure the bandwidth by sending a known number of packets of a known size and simply subtract the timestamps from the client to get the duration. This has to be tested separately on the uplink and the downlink. To instigate the downlink stream you again need a remote session, ideally not using the test equipment.
I was a 3G data engineer for Sprint PCS during the launch of 2.5G and 3G data. There are a few problems with the type of testing it sounds like you want to do.
First, I would suggest reading the specs from the IEEE on CDMA 2000, aka 3G. CDMA2000 allows the ability to allocate and de-allocate bandwidth on demand and based upon quality of service configurations. At night, with no one on the cell tower, you're going to get the full pipe for data. You'll see bursts up in speed but then several things can happen. First, voice takes priority. So at midnight in this scenario, I pick up my phone and make a voice call and I take priority. You're pipe just got smaller. The next variable is overall tower usage. Cellular towers shrink and expand RF power with regards to usage. As more users get on, the cell tower will reduce it's footprint. So even if hardly anyone is on the phone, but there are a ton of subsribers on a cell, it can drop it's power. So your bandwidth and RF are variables which change by the second.
So if you're just doing a "Hey lets just see what we see," type of test, then expect a huge array of data with not too many descernable results. If you're looking for data to be compared with something else (carrier vs carrier, region vs region), then it also won't be terribly useful. As a geek, it'll be cool to know you can set the test up, but as a quantitative analysis tool it won't be repeatable or statistically useful. You also can't really compare it to desktop loading times either. The images you "download" via a wireless carrier are not the same. Go to Yahoo and download the GIF for their logo using an aircard or wireless carrier device. Now, download the same with a desktop PC over a wired (or other non-wireless carrier method). Not the same file size, huh? :)
Just as an aside, I used to test it using Stick Figure Death Theater ( www.SFDT.com ) since you can start a really long download and watch the stats. Also, it was entertaining to watch at the same time.
Try the application CellMapper from the Android market. It creates maps of user-submitted cell signal strength. Some of the areas you are interested in may be covered already, and any new data you submit will help to benefit the community as a whole.
I found it extermely easy to write a program myself for my windows mobile handset. It took me about an hour to write a program which a) collects network information (available techniques such as GPRS, EDGE or HSDPA) and signal strength along with tcp ping statistics (for some reason some Finnish wireless base stations block icmp ping) and b) gps data (time, speed, location). This gave me a possibility to gather quite interesting data about both coverages and the effect of speed as I ran the mapping program a couple times while traveling the same route by train. If you happen to have a wimo 6 -phone and have _any_ programming experience you should be able to hack something like that together (using free tools) very fast.
Shameless self-promotion ahead...
This is more a suggestion to help with mapping received signal strength (RSSI), rather than data network latency and bandwidth (you can argue that those data network metrics rely heavily upon your RSSI!):
Grab an old Nokia, use gammu to enable Network Monitor mode, fire up a GPS and display the combined information streams on a map. I did exactly that as an experiment using a Nokia 3310 and a Navman GPS receiver. Interesting to then correlate the signal peaks to the actual base station locations.
The main caveat is that the old Nokias in question only do (I believe) dual-band 2G GSM at best, so you won't actually be able to measure 3G W-CDMA statistics as if you were connected with a 3G data device. I would offer, however, that 3G RSSI might be related to that of 2G as many of the base stations handle both services (ignoring differences in signal propagation characteristics). A starting point at least...
Warning: shameless plug dead ahead...
Look at http://www.p3-group.com/communications/en/home.html, it's their core business and they just opened an office in the US.
If your end users will need to run 24 hours a day you will need to test 24 hours a day.
We have number of little (Linux of course) boxes using 3G USB modems here in Sweden, and have discovered that Telia does all sorts of maintenance at night and at weekends.
Furthermore the latency (as measured by ping) varies quite a bit depending on time of day, i.e. when the local farmers are downloading their porn.
Another thing we've come across is that the versions of the USB modems changes with great frequency! Which of course means when they break you have to be very careful who changes them.
Good luck!
Sorry, nothing profound to put here! (http://www.abacus4.com/
There are plenty of expensive software solutions for this that are used at a professional level (ROMES, NEMO TEMS etc).
If you are after a free solution, the AT commands of most units Huewai, Option etc will give you network information,
For example: Signal as RSSi. (look under 3GPP TS 27.007) AT+CSQ? gives a number 0 (-113dBm or Less) to 31 (-51dBm or greater)
Failing that a lot of the dashboards have open APIs (in the UK Vodafones dashboard gives you access to lots of information)
Most of the Samsung mobiles have a diagnostic screen giving levels Scrambe Code and Cell IDs
For more info Siroda.co.uk or pm me
Si
Back in 2001/2002 I was doing drive testing for US Cellular. One day we were cruising through some random suburb of northern Illinois. My driver that day was an Iranian American. Nice guy, but not the usual bleached white skin tones folks are used to seeing in such a location. Someone called us in to the police as possible terror suspects, driving the neighborhood around with "computer equipment".
So we got pulled over by a fleet of police. Took about 5 seconds for them to realise what we were doing and we went on our way.
As to the original question, we ran two phones, one that would continuously make short duration calls to test the ability to the network to identify and connect new call, and another that would try to stay online for as long as possible, testing the network's ability to hand off calls from one tower to the next.
If you are planning on testing a decent sized area, I would recommend using a similar system. If you're just going to go to specific hot spots and run a couple of tests at each location, I wouldn't go to such lengths though.
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
"In order to resolve an ongoing issue with a vendor..." Good luck with that. Doing the testing itself is very technically interesting and worth discussion. Trying to resolve an issue with a vendor by "proving" they have a certain level of service is a different matter entirely. That type of approach to resolving business disputes just isn't the right approach. It's a good example of trying to resolve business/legal issues with proving something in technology, and that is very difficult.
Just because I can hook a shark from a boat, I do no offer to wrestle it in the water.
I'd mod you Insightful, but if this telco has any competitors, a nice heaping pile of public shame could motivate them to take care of the issue...Post the results of the "cellular wardrive" online and send links to the competitors, ideally to someone in marketing if you can find the contact info. Again if there are competitors, maybe you can get a SIM from them and record data from that device simultaneously so you can compare the signal quality. The company with the best signal quality in the area would be very interested in the results.
As for capturing the data, it shouldn't be too hard on any Linux device as long as you can access the signal quality of the cellular modem. I know you could do it with a Python script or maybe even a shell script on an N900. The hard part will be using the data to produce a map visualizing the signal quality in different areas.
"When information is power, privacy is freedom" - Jah-Wren Ryel
I have used SMSlib for a similar purpose. It is an open source Java library for sending and receiving SMS with a wireless modem. I used it as a framework to communicate with the modem and extract its signal quality measurements. The basic algorithm is: -- Force the modem into 3G mode (that's what I'm interested in) -- Loop every minute: -- Read and record RSSI, BER, SNR You can read about the details on: http://knol.google.com/k/novak-petrovic/modem-based-3g-signal-analysis/1j9pv1wo3nq4h/20# There are also my measurements at two locations: "good" and "bad". With a bit more work this approach can be extended to correlate RF parameters with some other (IP) measurements you mention. It can also be extended to make it a mobile measurement that you require. I am currently working on a more accurate interpretation of some of the RF measurements. Best regards, Novak.