Slashdot Mirror
Accessing One's Own Metadata
skegg writes: Frustrated journalist Ben Grubb has documented his attempts at gaining access to his own metadata from his carrier. "After more than a year of phone calls and emails and a private mediation session, it still hasn't released the information or answered my one key question satisfactorily: the government can access my Telstra metadata, so why can't I?" Later, he says, "Telstra's one and only valid argument to date has been that identifying who calls me would be in breach of that person's privacy if they called from an unlisted number. I've agreed and said that in providing me with my metadata they should remove unlisted numbers. They argue this would be too difficult to do, which I think is baloney."
Don't you realize they'd have to re-lubricate the DB2 indexes with heavier oil to fulfill your request? Do you have any idea how hard this is? I just love it when normal people think data like this can be magically retrieved.
But their is no way they "can't figure it out"
excitingthingstodo.blogspot.com
If the government already has your meta data, request the government to provide you a copy. At no time should a government have any information about you that you cannot fully review.
If only we could fall into a woman's arms without falling into her hands
I have to conclude from the supposed difficulty that they store the metadata without noting which numbers are unlisted. Or more correctly, were unlisted at the time, since that status may change.
They argue this would be too difficult to do, which I think is baloney.
I think what they probably mean is, it'd be difficult for them to be able to provide this kind of metadata without risking legal/PR trouble. To make sure that they could provide your metadata without revealing information that could possibly open themselves to criminal prosecution or civil suits would require that they pay lawyers to review the whole process. And then they'd need to spend a lot of time internally figuring out whether they want to spin the whole thing for PR purposes, or if seeing your metadata is too scary the be released at all without a PR nightmare.
And that's a bunch of work to satisfy one reporter. Doing that opens to floodgates for them to have everyone request it. So now, they have to review their entire data collection policy and create policies for who can get access to what. That's a lot of work.
I'm not saying they're right to provide access to customer data to the government while denying customers access to their own data. I'm just suggesting that they're probably not lying when they say it's difficult. You just have to know what they mean by "difficult".
The reason, and I think they should just flat out say it because I think it's valid:
If they allow this guy to get it, then hundreds of thousands of other people will request it as well. They will need to build departments, processes, training, security procedures and create for themselves and very expensive endless quagmire of bureaucracy. Even if he offers to pay for it, someone will eventually sue, somewhere in the world and get it legally defined as a "Right" so then no-one will have to pay. It's Pandoras box, they know it, he knows it, and they are certainly not going to hand him the key.
Corporations are their own worst enemies at times. Just explain this and explain "We don't want to give it to the government either!! But they're making us!" If they're ordered by a court to release the information, they the court has to deal with most of the legal pitfalls. If the wrong information gets into the wrong hands, that's the courts fault. There's no way they are going to volunteer this.
HIPAA requires them to provide it to you, and sets limits on the fees for doing so. The fees are limited to cover copying the records.
http://www.gpo.gov/fdsys/pkg/C...
Yeah, in Australia we have a new centralised medical records system that patients absolutely CAN access. Heck, all my doctors have my folder open as they chat with me most times. I have even asked to see things and have never been refused.
...
2) This is Australia, not the US.
But there is an Australian law that requires they provide the information requested. It is in fact referenced in the original article.
excitingthingstodo.blogspot.com
They have the data, but there's a spider the size of a pig blocking access to the drive.
"Telstra's one and only valid argument to date has been that identifying who calls me would be in breach of that person's privacy if they called from an unlisted number.
Are anonymous phones calls really protected by law?
I mean is there a law that specifically protects the anonymity of people calling from unlisted numbers?
After all, the person holding the unlisted number placed the call.
Do people coming into your house from the street have a legal expectation of anonymity? Does someone getting into your car have a legal expectation of anonymity?
Why would someone calling your phone have a legal expectation of anonymity?
I suspect it has more to do with corporations that robo-call wanting to hide. It's profitable for the phone companies.
When you become a senior citizen, you will begin to receive endless solicitations for medical alert bracelets, "free product" scams, health insurance and so on. I suppose everyone gets some version of this crap. None of these are allowed under the "Do Not Call" act, but the callers always have unlisted numbers and do not reveal their companies' actual names in the calls.
I wanted my data ever since I've heard the first time about the Data Retention Directive (now longer in force since earlier this year, GOOD).
Mind you, they don't keep only the metadata for you calls but also a lot of "control plane"/out of band communication mobile-network. Apart from this being extremely interesting for law enforcement it's interesting for me too! That is the location part of the data.
Yes, I know I could keep a diary or keep a GPS logger with me but that needs a lot of extra effort - even for the most automated solutions (charging, downloading, etc - mind you this was well before smartphones, probably today you could do this much easily, especially if you are plugging your phone into a charger each time you step into a room...).
Anyway the point is that I've never got the data. Even if I would be willing to pay for it, every 6-24 months (that's the retention interval that was in the law).
And just exactly how do you propose to _fix_ invalid data if you can't access it??
You first need _access_ to it.
Second, he already has that right.
But I guess it is easy to just knee-jerk react focusing on the symptom instead of actually thinking about the problem.
The unlisted aspect only comes through the SS7(PTSN) or SIP(VOIP/IMS) protocol headers with a flag indicating whether the account is private, in addition to phone number paying for call, phone number to display, phone number originating, etc... -- AND -- this meta-data can change during a call if it was rerouted mid stream, delayed headers, etc. This gets even more complicated for reverse billed numbers (800) where the originating number is XXX, the billing number is YYY, the display number is ZZZ, and sometimes an interlink number ends up in there. (and as we found out last month with our call logs, some numbers have yet another header that contains virtualized/multi-ring which need to be taken into account; lest the "wrong" number be displayed)
Now, legally, we are required to keep the originating number, time stamp, and length of call;
And for billing and interconnect agreements, the billing number as well.
As we internally always have full access to the raw protocol data on the Enigeering side; the legal siphon (done at the switch level) just skims off all the legally required data and stores it in long-term storage (not DB); to handle the GBs of data a day of the minimally required data.
We then have a separate process which takes each session and generates a [display-phone number, timestamp] DB for 90 days of call logs for users to look up (or legal requirement on bills for chargable calls made depending on juristdiction).
Under no circumstances have we ever kept the "is unlisted" status of the call; as it's never been a datum required for any business logic, ever.
And when handling millions of calls daily, and relying on switches to read/dump data for secondary systems to process RT is a space and time sensitive process; and thus, only the absolute minimum required is kept to prevent buffer overruns in the data processing phase;
But, as the process is semi-manual to retrieve data for a given time-range I can understand their request to honor "all my metadata" as well.
Limited time-ranges as required by law enforcement is easier to obtain:
- fetch the raw hourly dump files for the time range requested
- run the script that goes through the files and formats a CSV output for any matches of the search phone number
- this process takes hours to run for a weeks worth of data as it churns through TBs of text files if it's outside the 90-day "fresh" window that is stored in a more processed state (but not kept as it's a lot of data to store for no company benefit); most requests from law enforcement only request the last 30days of calls; and this particular process is more streamlined.
- it would be entirely unrealistic to do for the lifetime of a given customer.
One point to take away from this, is that many telecom companies have no interest to keep your data. It's expensive, each item of data adds substantial more costs, overhead, and resource to manage it's storage. It also adds significant more liability as now more people have access to it internally; and safeguards and resources must be used to manage it. Which is why the legal information is done automatically at the switching level, and dumped in a non-processed state; processed and stored, and intentionally kept difficult to access. Because we do not want the liability that comes with storing it, or making it easily available to even a subset of internal employees. Each person that has access adds more risk.
Storing users meta data at least in the telecom world -- is not wanted in the slightest, and we only do the absolute minimum to meet government regulations. Sadly, this also implies that with the current state of laws; that the data is not easily accessible, nor is the data in a state that can be released to a private indiviual without substantial legal risk.
Oh, the person with the unlisted number has called me. If they did it purposefully, I see no reason they have any standing to hide behind an unlisted number. My privacy is as valuable as theirs. If they've pocket-dialed, tough luck. I'm still at the receiving end of the call.
Moreover, unlisted numbers aren't 128 bit hashes that noone has time to enumerate. It's not as if I can't call an unlisted number. Heck, it's easy to corral the unlisted numbers, since they are disjoint from the listed numbers. Start with a set that spans the range of valid 7-digit phone numbers in a given area code. Then remove the listed numbers. Then remove the numbers that get connection errors. You are left with unlisted numbers. Such scans, in the day and age of VOIP, are rather easy to do.
And, finally, many digital connection providers pass an unlisted number along with merely a bit indicating the the number is not to be presented at the terminal. So the information is there, and it doesn't take but an asterisk setup to leverage that.
So yeah, they telco is just stalling here, nothing new... :(
A successful API design takes a mixture of software design and pedagogy.
gurps_npc: The post I replied to was talking about medical information in the US.
There is an easy solution for this problem. Corporations could not store metadata for individuals. Then they wouldn't have to produce anything. They wouldn't need " to build departments, processes, training, security procedures and create for themselves and very expensive endless quagmire of bureaucracy."
If they want to keep that data, then they need to share it with the people creating such data. The other option would be to share it with everyone. Nobody would like that though. Or, when you login online to check your account, they share it there. That shouldn't be too hard.
Ninjas don't carry tic tacs
On T-Mobile, it is as simple as logging into your account on the web site, and looking at the reports. For a family plan, it lists the sender and receiver phone number of EVERY call AND text messages for everyone on the plan. These are accompanied with their time stamps, too, of course. There is also an option to download a PDF file with the "detailed" report on your bill, which contains all this information.
No idea why other carriers are claiming it is hard to deal with this sort of data.
We need to evolve to adapt to this new threat to the species, and instead of seriously *resisting* its effects on our being, we - the true power - direct the feature to our favour. If, out of the NSA catastrophe, we gain a "New Internet" wherein *everything, everywhere* for 15 years, was available to everyone, then we'd have indeed a new era in the human species. A truly evolutionary step, made by mistake - perhaps.
; -- the corruption of government starts with its secrets. a truly free people keep no secrets. --
CLI is what you want and you'll see the ID of every incoming call. That's your metadata. There you go, collect your own metadata you lazy bastard.
There was an unknown error in the submission.
Which results in very big collections of facebook data sent to you.
Maybe it's a good thing. It raises at least the -possibilty- that it might be hard for other people to get his data, as well.