Slashdot Mirror
Falsehoods Programmers Believe About Names
Jamie points out this interesting article about how hard it is for programmers to get names right. Since software ultimately is used by and for humans, and we humans are pretty tightly linked to our names (whatever the language, spelling, or orthography), this is a big deal. This piece notes some of the ways that names get mishandled, and suggests rules of thumb (in the form of anti-suggestions) to encourage programmers to handle names more gracefully.
I found the piece very interesting.
Though my inability to post this comment appears to have outlived the slashdotting of the site.
Who the hell has numbers in there name?
and let god sort them out...
I am fortunate enough to be the child of a professional smart-ass who intentionally gave all his children two middle names so that we would not fit into the computer systems of the era.
When I grew up my parents used my first middle name as a "given nickname" (it's actually in quotation marks on my birth certificate). So most of the time when I give my name for something I use my "given nickname" as my first name. Unless I feel like using my legal first name as my first name in which case I use that. There are probably four or five different versions of my name attached to my SSN in various different databases.
I've also got a sufffix: III. I don't have two ancestors with the exact same name as me, but since the various parts come from two different relatives my parents settled on III.
After just 15 minutes of the story being posted?
Wow, that's gotta be a personal best for /. (or, the site is a wee bit underpowered... ;)
Here's the Google cache in the meanwhile: http://webcache.googleusercontent.com/search?q=cache:http://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/
Even the cache needs tweaking to load.
Text only version.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
John Graham-Cumming wrote an article today complaining about how a computer system he was working with described his last name as having invalid characters. It of course does not, because anything someone tells you is their name is--by definition--an appropriate identifier for them. John was understandably vexed about this situation, and he has every right to be, because names are central to our identities, virtually by definition.
I have lived in Japan for several years, programming in a professional capacity, and I have broken many systems by the simple expedient of being introduced into them. (Most people call me Patrick McKenzie, but I'll acknowledge as correct any of six different "full" names, any many systems I deal with will accept precisely none of them.) Similarly, I've worked with Big Freaking Enterprises which, by dint of doing business globally, have theoretically designed their systems to allow all names to work in them. I have never seen a computer system which handles names properly and doubt one exists, anywhere.
So, as a public service, I'm going to list assumptions your systems probably make about names. All of these assumptions are wrong. Try to make less of them next time you write a system which touches names.
What's in a name?
http://www.youtube.com/watch?v=kFmJsTKWEBI
http://www.youtube.com/watch?v=cuUE7oKIkVI
Names of what?!
Fuck systemd. Fuck Redhat. Fuck Soylent, too. Wait, scratch the last one.
Software is NOT designed to be perfect and cover every case. Have a numeral in your name? Too bad. Need some names to be case sensitive, and others case insensitive? Sucks to be you. Have a 200 character name that doesn't fit in the 100 characters the designers thought no crazy person would ever have? Tough.
I started reading through the list, and it's just ridiculous. There's a few good points, like names don't change, or names are unique. But they're so obvious that the vast majority of the times it's not a big problem. More often it's just a matter of training the data edit/entry folks how to change someones name, or how to not assume a name is a sole identifier.
But assuming the worst and trying to design a system that'll allow people's names to be Chinese characters when you don't do business in China, have presence in China, or ever ever plan to? That's ridiculous. Software doesn't have to be perfect out of the shoot. It should be adaptable though if some unforeseen shortcoming becomes a larger problem. Gee, I guess if you ever chose to do business in China and need Chinese character names you might have to re-write part of the damn software. Oh well, that's what software developers are FOR!
If you don't even HAVE a name, then I submit you're crazier than the artist formerly known as the artist formerly known as Prince. At least HE had a name, though it was an unpronounceable symbol. The world can't accommodate every possibility, and software is no exception.
AccountKiller
He's essentially arguing that, because names vary a lot and are complex, your software should never do anything useful with them. Sorry, but that's a stupid answer. In a lot of systems, being able to sort by surname may well be more important than being able to handle people who claim they have no surname.
Of course, you shouldn't gratuitously do stupid things, and interfaces should aim to be relatively clear. But most people can figure out how to enter their names into relatively standardized forms, and those that don't should probably figure out how.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Thanks, Prince
A database MUST treat all of these names the same: McClean, MacClean, MCLean, Mc Clean, Mac Clean. McCleen, ...
Are you sure? What if "Mac Clean" is actually somebody's first and last names?
I know plenty of people whose legal name is a single word, such as "Alex", "Max" or "Virgil." Would your system put that in the first_name, middle_name or surname column? Storing names and using them sensibly is hard, as TFA acknowledges.
You'd think that e-mail addresses by comparison would be simpler, but I have a hard time trying to register my e-mail address with sites that won't allow even simple things like "+", "-" or "." characters in the local part.
A database MUST treat all of these names the same: McClean, MacClean, MCLean, Mc Clean, Mac Clean. McCleen, ...
I assume you left out a "not" in that sentence? I think there are quite a few people that will kindly (or maybe not-so-kindly) explain why "Mc" and "Mac" are not the same.
My last name is O'Leary and over the past 5 years web sites have not gotten any better, and arguably have gotten worse, at handling the apostrophe in my last name
Help me Slashdot, you're my only hope.
Considering how many entry forms still don't allow '+' in an e-mail address (or, worse, allow it in the sign-up box but not in the unsubscribe box), and considering how many banks still restrict you to an 8-character password, does it come as any surprise that they have difficulty with something that isn't defined in an RFC?
Scratched Emulsion
First thing I learned back in 1993 when I got started.
1) George Foreman has five boys named George Foreman. Your database better be able to handle that. :-)
2) Your database better be able to handle Cher (no last name).
3) People are not required to have Social Security numbers. (it's an optional program - you don't have to partipate).
4) Not everyone's last name starts with a capital letter.
5) Mexican people's names break ASCII (the tilda n).
6) People named O'Grady have a hard time getting their name in a database sometimes and have a hard time getting their name passed via a URL sometimes and generally mess stuff up.
7) People from Sri Lanka will break your name length limits.
8) Some people's name is only a single letter.
9) Some people go by their middle name god damn it!
My first name: "where 1=1 "
My last name: "'; drop table users; --"
I code to spec. The product and marketing departments write the spec (what little there is); the QA department amends the spec with overly specific test cases. I suggest that the spec is incomplete and won't handle...but I'm told, just code it to spec. I recommend changed, but we don't have time for edge cases. I point out potential problems, but we're unlikely to get any of those. I warn of potential compatibility problems but we don't care. Are you just trying to be difficult? If there's a problem QA will catch it. The project is overdue already, and by the way here are some new requirements that need to make it in, and we can't change the release date because we already promised the stockholders. Why is your code so complicated, my twelve-year-old kid could write this.
It's not my fault. I code to spec.
Software shouldn't have to satisfy every whim and excentricity. If you don't have a well-defined first name and last name that consists of extended alphanumeric characters in Unicode and starts with a letter, well, then get one, OK? And while you're at it, come up with decent Romanized and ASCII (traditional Latin) versions of your name, conformant with one of the common Romanization systems of your language; you will need that too if you want to travel internationally. Single letter names are also a potential problem because they are confusable with abbreviations, so consider using a variant spelling ("O" -> "Oh").
This isn't because programmers have some sort of hangups about names, it's because people themselves need to be able to refer to individuals in some reasonable and standardized way, they need to be able to write your name, alphabetize it, and correct errors.
It seems to me that most misconceptions about names can be fixed by the following:
Allow a single, Unicode-enabled field of "unlimited" length (let's say 4 kilobytes) which represents "name". Several would be defined by different roles -- "Real name", "Nickname", "login", where only login (sometimes simply an email address) is required to be globally unique.
Now let's look at what that breaks:
First, #1, 2, 4, and 5. How am I supposed to avoid assuming these? People should be allowed to enter an arbitrary number of names for themselves? I suppose that's possible, but it immediately kills most of the potential uses of this data. If I want to set a nickname that goes with my forum posts, say, what good is it for me to have five nicknames? Seems like the only potential use would be making people easy to find by real name -- so, a social network.
#6 -- surely 4k is enough, but this is also not a terribly difficult assumption to change later. Annoying, but not devastating, not even as hard as changing from the first name / last name combination into one "real name" field.
#7, 8 -- most systems would make it trivial for people to change their names.
#9, 10 -- UTF8 is easy.
#11 -- very, very curious to see an example. And wouldn't that be a bug in Unicode? And this is again one where I have to ask -- how do you change this? Allow arbitrary images?
#12, 13 -- obvious solution is to make the name system case-preserving, thus allowing both case-sensitive and case-insensitive searches.
#14 -- again, avoid by simply allowing the name to be a single opaque field.
#15, 16, 17 -- if your name supports random unicode, no idea why these would be a problem.
#18 -- not sure why it matters.
#19, 20 -- again, if it's just arbitrary text, it just works.
#21, 22, 23 -- not sure how I'd make that assumption.
#24, 25, 26, 27 -- again, the name is just an opaque bunch of characters.
#28 -- what?
#29 -- opaque characters.
#30 -- keep the original text as-is. If you want to try to split people out by naming scheme, do it later, but keep the original. This should be a "duh" concept -- always preserve the original user input. Cache transformations for speed, if you like, but they're a cache -- keep the original. Your algorithm might change.
#31 -- bad idea to assume bad words won't cause problems in general. I currently play an MMO in which I physically can't talk about Emily Dickinson, and have occasion to more frequently than you might suspect.
#32-36 -- why would it matter? Unless...
#37 -- Fine, but how would I otherwise connect the same person?
#38 -- How about unicode-equivalent? And of course, they might not -- one might make a mistake, or the name might be represented differently. But you'd have to deal with typos anyway, so this isn't exactly shocking.
#39 -- I'm going to have to agree with the assumption, though. If I develop a system which works well for people who only follow the US standard, and I suddenly have a ton of people from China wanting to use my service -- enough that this is actually a problem for me -- that's a nice problem to have.
#40 -- People can make up names. I guess this explains #32-36, though.
The sense I get is that half the list is stuff you'd almost have to be stupid to run into (seriously, who doesn't use Unicode?), and the other half involves some seriously weird names and cultures that are going to have to meet me halfway, if they expect me to do anything interesting with their name. As I understand it, the only way to get this right would be to allow people to have zero or more names, each of which is either an unlimited amount of text in any encoding, or an image (raster or vector) of unlimited size. To query such a system requires insane amounts of logic just to deal with the text, and throw in some OCR for good measure.
I think this is a case where I would much rather see people evolve to match the technology, rather than the other way
Don't thank God, thank a doctor!
You'd think that e-mail addresses by comparison would be simpler, but I have a hard time trying to register my e-mail address with sites that won't allow even simple things like "+", "-" or "." characters in the local part.
Proper email validation is not trivial
Check out the huge regex at the bottom of the RFC 5322 compliant validator from CPAN:
http://cpansearch.perl.org/src/RJBS/Email-Valid-0.184/lib/Email/Valid.pm
Yes. It's programmer's fault that they write applications that make poor assumptions about names -- not the people who design software requirements who are neither programmers nor usually very worldly.
Perhaps we should have a list of "assumptions people make about developers"!
* Developers get to design their own software.
* Developers get to have some say in how their software is designed.
* Developers at least can prevent really stupid things from being put in the software they write.
* Developers aren't smart enough to know that outliers are inevitable.
* Developers aren't smart enough to know that of course there are people with punctuation, extra words and spaces, even letters that no one has seen before.
* Developers wouldn't rather code just one column to hold an identifier rather than two.
Terrorists can attack freedom, but only Congress can destroy it.
A database MUST treat all of these names the same: McClean, MacClean, MCLean, Mc Clean, Mac Clean. McCleen, ...
I assume you left out a "not" in that sentence? I think there are quite a few people that will kindly (or maybe not-so-kindly) explain why "Mc" and "Mac" are not the same.
Read between the lines a bit. Treat them the same means: treat them as all potentially valid, not that all the names would match in a string comparison.
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
This issue is pretty much universal. Even outside the binary world people still silly assumption about people's names.
For example, numerous people have raised objections about my signature. They always give me bullshit complaints like "Sir, that is not legible." or "um... that's not your name." or even "Did you just draw a penis on the dotted line?".
My signature does not have to be legible.
My signature does not have to be my name.
My signature does not have to contain my name.
My signature does not have to contain any name.
My signature does not have to be in the English language.
My signature does not have to be in any human language.
My signature does not have to consist of meaningful symbols.
I swear if I hear one more complaint about my signature I will carry around a portable photo printer to render goatse as my signature:
"Yes, my signature is an 600 ppi out-stretched anus. Deal with it. The law says that any mark that I make is a legally valid signature and you have to recognize it as such. You either sign the mortgage or I'm going to the next bank."
A database MUST treat all of these names the same: McClean, MacClean, MCLean, Mc Clean, Mac Clean. McCleen, ...
I assume you left out a "noot" in that sentence? I think there are quite a few people that will kindly (or maybe not-so-kindly) explain why "Mc" and "Mac" are noot the same.
fixed that
A lot of forums and banking sites written by noob programmers for starters. Notice I said "local part", I haven't found one that cares about -'s and .'s in the domain part.
A database MUST treat all of these names the same: McClean, MacClean, MCLean, Mc Clean, Mac Clean. McCleen, ...
I assume you left out a "not" in that sentence? I think there are quite a few people that will kindly (or maybe not-so-kindly) explain why "Mc" and "Mac" are not the same.
Yeah, one goes in front of 'Donald's' and the other goes in front of 'beth'.
"I don't care about the Constitution!" --Bill O'Reilly, November 17, 2009
But in practice, shouldn't you just be able to copy the regex from somewhere (open source, with attribution of course) and check if it matches?
(+1, Disagree)
True. I run into email validation problems constantly. I have a two-part first name that has "-" in the middle, so my firstname.lastname email addresses (usually work addresses) always have a "-". In addition at the moment I'm a consultant in a large company, where they put "ext-" in front of everyone who is not employed by them but works for them and has an email account from them. I also often run into problems with length, because my name is 19 characters and the last place I worked for had a 15 character company name and when you add TLD to that, you sum to an email address that is 39 characters long, which for some seems to be too much. I really don't get why you would use only 32 characters to store an email address..
This problem very often bites in name fields, too, that don't accept "-" and two capital letters in my first name.
And I used to live near a border of two cities, where my postal address was from one city while my real city of residence was the other one. I have had a lot of problems with that, when the guys who made the systems were trying to deduce my city of residence from my postal address. Which is also impossible in my country, because the national post office also permits addresses that have postalnumber + company (instead of city) for large companies who take their mail in one place and deliver it themselves the rest of the way.
The regular expression, if one must be used, doesn't need to be any more complex than:
^[^@]+@[^@]+$
Sending out response emails to an improperly validated address just turned you into an open relay. Spammers can use your server to send spam by embedding their entire message as the email address, trailed by '\x004@.'
Validate your inputs. Always.
Wow, if you consider McLean and MacLean the same, I suggest you never visit Scotland.
The Mc's and the Mac's consider the correct usage as a matter of extreme pride. You could end up with one or more bruises if you get it wrong and then insist that "well, they're the same anyway".
The author must have missed his history lesson explaining that family names only became popular in Western European culture when governments started tabulating people. In a rural village everyone knows that Jack the butcher is different from Jack the baker.
Hence Butcher, Baker, Smith, Brewer, Tanner, Farmer, etc became "family names".
*Even if the system did a conversion to a latin representation of an asian name most people can't pronounce them because they are based on different sound primitives.
Such a "translation" can easily be one to many, dependent on various factors.
Which is why Asians tend to adopt westernised versions of their real names.
Or they adopt a regular English, German, French, Spanish, etc name to be known by.
I don't think that's what it meant at all. I think the author is trying to be too smart by suggesting that someone looking for MacDonald might have heard it wrong, and so might type in McDonald instead. It's probably a valid point for fuzzy searches, but to say that they should all be treated the same is wrong.
That said, his other points, especially about the fact that not all names are properly mapped in unicode, is a good one. I just wish he'd posted citations and solutions, rather than simply pointing out the issues. But the first step in fixing a problem is acknowledging the problem.
I though the article was about the inability of programmer to remember names and recognise people, Maybe I should have read the article.
It's a real problem though - is it just me? I often know things about people (ah yes, plays squash, good at making cakes, father of that kid who rides a unicycle), but their actual name - no. It's a miracle if I recognise them at all.
Mind you, it means if anyone says "Hello" to me, I am obliged to be polite to them as I might actually know them quite well, but haven't recognised them yet - and certainly don't know their name.
It's a right pain. Anybody else suffer from this - and what the heck do they do about it? (I'd like a camera attachment what would whisper in my ear "that's Mrs Jones, her daughter, Kira is in the same class at school as your daughter. Likes chess and is obsessed with kayaking" - something tiny that could clip on my glasses, maybe).
"Cats like plain crisps"
Why would you be doing the validation in the database?
If he'd meant "should treat them all as valid", then he should have written that.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
You could end up with one or more bruises if you get it wrong and then insist that "well, they're the same anyway".
Isn't that a general risk when dealing with scotsmen? :-)
But where does the difference with the missing 'a' come from?
bickerdyke
Just looked it up. I'm Scottish, live in Scotland and always hear people say that the difference in Mac/Mc is important because of the Scots/Irish thing, but according to this article, that's bollocks:
http://www.scottishhistory.com/articles/misc/macvsmc.html
which is totally what she said
Sometimes I despair when I read or hear somebody referring to eg. Djengis Khan as "Mr Khan" ("Khan" is a title, not a name) or even call Hu Jintao, "Mr Jintao"; you would have thought people would, by now, have caught on to the idea that something like half the world's population has the family name first.
Oh, come now - are you seriously saying you expect every single person to understand every subtle nuance of every other culture's use of titles and names? Here are some non-English equivalents to Mr., are you seriously telling us you know all of these? Here are the various forms of address in the UK alone, do you know all of these and every other culture's equivalent? How many of these should I learn before I go from being someone you despair of to someone you feel is welcome in your titular elite?
If half the world's population has the family name first, which half do I choose to offend when I don't know the exact rule for the home country of the person I'm speaking to? That's even assuming I know which country they're from. There's no reason to assume in this shrinking planet that someone who looks like they're from country A wasn't in fact born in country B to parents from countries A and C - a person born in Japan but with lineage in China might take great offence if I use Chinese honorifics to address him, surely it's better to be polite within the confines of my own known culture than to make such crass assumptions about his? The key thing I take from someone saying "Mr Khan" or "Mr Jintao" is that they're at least making the effort to communicate in a civil manner, which certainly causes me no despair.
Because no one ever automated the process of filling out web-forms right?
I was born with a complicated Spanish name.
One first name.
Two second names.
One hyphenated, accented surname from my father.
One simpler surname from my mother.
One of the first things I've done since reaching majority is to give a precise, simple, standard name to everyone who asks for it:
Xxxxx Xxxxxxx
where X is in A-Z and x is in a-z. Xxxxx is my first name, and Xxxxxxx is a shortened, accent-and-hyphen-free version of my father's surname.
You know why?
Because, in life, there are lots of things one must be "unreasonable" about in order to effect progress, but accommodation of one's name is not of them. It's a tedious, selfish expression of nothing more than ego which ultimately will land you in more trouble than others: some day you will be denied access to something thanks to some computer system not being designed to handle your name, and "computer says no" gets priority over the angry demands to the immigration officer of "Joe\0\rBlogg$ 3'); DROP TABLE citizens; -- [insert spinning cube here] Jr."
If you and your friends/colleagues have some other name by which they call you, sure why not? But, as any cat will tell you, the world is best when you have three names:
(i) one for communicating formally;
(ii) one for more intimate discourse (there's no reason why this can't be the same as (i), though many people end up with peculiar nicknames); and
(iii) one personal identification which you can keep to yourself and you can't express in words.
If you want the sum of all your history, culture and personality as expressed in (iii) to be embodied in (i), you're both expecting others to be burdened with your ego and bad at understanding human communication. All I asked for was a couple of words I can use in a reasonably uniform way to easily pick/call you out from a small crowd - that's what (i) is for, after all.
tl;dr The naming of cats is both a delightful poem and an insightful account of the multiple namespaces for kitty/human names and their different purposes. Don't confuse them.
.. is Hfuhruhur-Uumellmahaye you insensitive clod !
To make things worse, it's not necessarily the family name you use to address someone politely.
If you have to speak to Paul McCartney (of Beatles' fame), you have to formally address him as "Sir Paul". No, "Sir McCartney" is impolite, you shouldn't use it.
If you have to speak to Vladimir Putin, you won't address him as "Mr. Putin". It's "Vladimir Vladimirovich", please!
Proper email validation is not trivial
The regular expression, if one must be used, doesn't need to be any more complex than:
^[^@]+@[^@]+$
Actually, the local part of an e-mail address can be a quoted string, containing pretty much any character, so "user@host"@example.org is a perfectly valid e-mail address, and doesn't match your regex. Most systems won't accept it, but it's valid...
You know, attitudes like yours are IMHO the root of all that's wrong with computers today. And I'm saying that as a programmer, not as Jane Grandma. The whole idiotic OCD idea that you _must_ make up rules about everything, and that your rules are more important than what people are actually trying to do. The idea that if even someone's name doesn't fit "your" database, then you can just brush them off and have a beer.
Here's some free clue: yes, you can't handle every edge case in the universe, but you'll find it's easier if you don't create such edge cases in the first place. If your database (actually more likely the program in front of it) can't handle last names with more than one capital letter, or with a dash in the middle, or which are more than 32 bytes long (which with UTF-8 might mean less than you'd think), then guess what? _You_ created an artificial edge case that had no reason to be there in the first place. Instead of handling every edge case in the universe, how about not creating them in the first place?
I find that about 90% of the problems boil down to the above: some idiot put some artificial limits or rules, that really aren't needed anywhere else. Just because he has the delusion that he's some kind of Moses on the mountain and just _has_ to come down with some rules.
E.g., he just had to define a byte limit, because he's prematurely optimizing a non-problem he doesn't understand. God forbid wasting space in the database by allowing 256 or 2000 byte strings... never mind that if he actually understood that underlying database, he'd know that a VARCHAR is not padded to that max length. If someone just entered "Alex", the same 4 bytes will be actually used in the database, regardless if the field is a defined as maximum 4, 32, 256 or 2000 characters. But nah, he has to put some restrictive number there, 'cause it looks more like he's doing some smart job.
There is hardly any reason to even use a user name for anything other than display purposes. (You do have a primary key for that record for everything else, right?) As such there is no reason to make any assumptions about it, or enforce any particular format, or anything. There's no reason to even disallow SQL keywords (just effing quote it before using it in SQL) or angular brackets (just quote it before using it in HTML.)
There is no reason to create any edge cases in the first place.
And really it's not even just about names. Names are just one case where people make up BS rules just to feel more like they did the great design job. One could make the same case for the gazillion other pointless rules imposed upon the user or his work-flow or data, not because they're actually needed anywhere, but just because some OCD idiot feels like he _must_ impose some rigid structure upon things that really have none and don't need any. But he'd just feel naked without defining that kind of rigid structure, or without imposing upon humans some data structures theory that was intended only for use by programs.
A polar bear is a cartesian bear after a coordinate transform.
Honestly, this sort of thing is an example of overusing regex when it's the only parsing tool they know. Regex becomes unwieldy when you put too much of it in one place -- but this is because regex is unwieldy, not because the problem of parsing email addresses is fundamentally hard. Parsing email addresses is a case for a modular parser such as Parsec (or any of its ports and imitators) ... which will give you the added advantage of useful error messages on invalid input, instead of just a match failure.
Moreover, isn't it kind of silly to point at an example of someone already having written the code to do something as a way of saying that doing it is difficult? In code, once it's already been done once, correctly, it doesn't need to be done again. If you think CPAN's huge regex (or any other implementation) is correct, and you've tested it to your satisfaction, you don't need to reimplement it; just use it.
Because no one ever automated the process of filling out web-forms right?
Pffft, the idea is absurd! You'd need a computer to do that, and what spammer has a computer?
Oh, right. Yeah, all of them. My bad.
Security is mostly a superstition... Avoiding danger is no safer in the long run than outright exposure. - Helen Keller
So is mine!