Microsoft Releases 'Caller-ID For Email' Specs

gfilion writes "Microsoft has released a draft specification for Caller-ID for email, 'to address the widespread problem of domain spoofing' - the concept is similar to SPF, but is using XML. There's already an Caller-ID to SPF converter in the works. A few weeks ago, Microsoft discussed compatibility between the projects with Meng Weng Wong (SPF's project leader), but most SPF users are against using XML, so nothing has come of it thus far." We recently covered a brief article mentioning Microsoft's anti-spam work, though this is a clearer indication of their intentions. Update: 02/26 21:36 GMT by T : NewsForge is carrying a brief article with FSF counsel Eben Moglen's take on the draft; Moglen says it is "encumbered with unclear and unnecessary patent license claims."

Comments

XML... in its place.

[SoTuA]

While I acknowledge that XML is great for some things, why is it that it gets used for almost everything nowadays? Damn buzzword-dominated market...

Ok, I'll be quiet now :)

Re:XML... in its place.

[trix_e]

because its become what it was intended to become. A 'data format' that everyone (thereabouts) understands. More than just everyone, but most everything understands how to parse it (everything from a 'modern day systems' standpoint, not a 'my toaster' standpoint... though wait a few years...).

While I agree that there are no absolutes, why not go with the path of least resistance when it doesn't really matter? XML has become the path of least resistance *at a macro level*. it's universally accepted these days, so unless there's a compelling reason *not* to use it... use it.

The reason I say at a macro level, is that yes, on an individual project using XML may be a bit harder -- though most development platforms these days have trivialized the difficulty of implementation.

Re:XML... in its place.

[Karamchand]

XML is not a data format. XML is an idea. You still need the DTD.

Re:XML... in its place.

[trix_e]

that's why I used 'data format' in quotes...

agreed, if you want to be picky it's not a format by itself, but XML as a framework for structuring data (to include DTDs, XSLTs, etc.).

The term 'XML' is used generically these days as reference to a particular way of structuring data as contrasted to other ways.

Re:XML... in its place.

[Hard_Code]

Sort of. You don't REALLY need a DTD - you only need one if you are validating the XML. XML can still be used as a generic ad-hoc hierarchical data format... of course you'd only want to do so because by now XML parsers are pretty ubiquitous and it makes it as good a choice as P-lists, or any other ad-hoc format.

Re:XML... in its place.

[blowdart]

though most development platforms these days have trivialized the difficulty of implementation.

And there's the rub. It's so damned easy to parse XML these days, why reinvent the wheel having to parse a comma delimited file, a fixed width file, a bizzare internal format?

Re:XML... in its place.

[Tinidril]

Sorry, I don't care what tools are available, parsing a comma delimited file when the records are reasonably simple in structure will always be easier. XML is really only usefull when the data resists structure.

Documents are really the only place where I can see XML adding any benifit. ( Unless more bits in the stream are considered benifit. )

Re:XML... in its place.

[wfberg]

Sort of. You don't REALLY need a DTD - you only need one if you are validating the XML. XML can still be used as a generic ad-hoc hierarchical data format... of course you'd only want to do so because by now XML parsers are pretty ubiquitous and it makes it as good a choice as P-lists, or any other ad-hoc format.

Assuming you don't have a DTD, you don't have a specification of what's in the files syntactically, let alone semantically. Maybe you can reverse engineer most of this (the tag "name" is likely to contain a name, etc.) but there will always be freakish exceptions and ambiguities that even DTDs and XML-Schemas don't address.

And the overhead of using XML is enormous.. All those possible encodings, character sets, namespaces, etc. S-expressions are really much, much nicer is you just want to parse without a formal syntax specification. And they've been around "forever".

Most irksome though, are so-called "XML databases".. Argh! I suppose the people who think that's a good idea also love "CSV databases" or "XLS databases"..

Re:XML... in its place.

[Short Circuit]

Your first point is only an issue if you intend to write a program that can handle arbitrary XML data.

For many (most?) applications, you already have a good idea what the data you're working with is going to be like.

I find XML mostly useful because I can edit data with vim, and not require the original application.

Re:XML... in its place.

[AndroidCat]

20 years ago, everyone used yacc/lex when they needed to parse something. They were handy tools and they were there. Now people need to parse a whole lot of stuff and the tools for XML are there.

XML is handy, and it's a lovely big hammer. Ooo, look at all the nails!

Re:XML... in its place.

[Hescomin]

too bad MS has patented XML(or attempted to before it gets shot down)..jsut another MS land-grab

Re:XML... in its place.

[ThaReetLad]

Did you read the licence file? It grants you "a fully paid, royalty-free, non-exclusive, worldwide license under Microsoft's Necessary Claims to make, use, sell, offer to sell, import, and otherwise distribute Licensed Implementations"

that is, it's free, so long as you grant a reciprocal licence to microsoft for your implementation.

Re:XML... in its place.

[Hard_Code]

True, XML is overkill for many many uses, but the matter of fact upside is ubiquity. I disagree in that a DTD gives you anything other than validation. Even if you have a DTD you can only validate the STRUCTURE of the XML...you still can't glean any MEANING from it. Which is why a lot of platforms simply choose to parse XML loosely with regular expressions and just treat it as a simple hierarchical format.

There are certain discrepencies between XML and S-expressions. It is true that any of these other formats "would do", and believe me, I am by no means an advocate of inappropriate, and over- use of XML, but the reality is that the proposed format is so tiny to begin with, and XML is so universally accepted, that it is practically moot whether this or that format would be "better". There are already a wealth of tools to index, mine, translate, etc. etc. XML.

Re:XML... in its place.

[pomakis]

Sorry, I don't care what tools are available, parsing a comma delimited file when the records are reasonably simple in structure will always be easier. XML is really only usefull when the data resists structure.

I have to slightly disagree with you there. Even if you start with a simple structure that can be handled okay by a simple comma-delimited set of values, things tend to evolve. What will happen to your encoding when a decision is made to add a bit more structure to the data in the future, such as allowing some parameterization on the items, or allowing tuples of items, etc.? More importantly, what will happen to the existing codebase that has been hardcoded to parse the simple comma-separated list? Using XML for even simple structures allows for more structure to be added in the future while allowing existing applications to read through it. That's been my experience, anyways.

Re:XML... in its place.

[wfberg]

True, XML is overkill for many many uses, but the matter of fact upside is ubiquity. I disagree in that a DTD gives you anything other than validation. Even if you have a DTD you can only validate the STRUCTURE of the XML...you still can't glean any MEANING from it. Which is why a lot of platforms simply choose to parse XML loosely with regular expressions and just treat it as a simple hierarchical format.

I did make the distinction between syntax ("STRUCTURE") and semantics ("MEANING") in my post, thank you very much :-P


There are certain discrepencies [prescod.net] between XML and S-expressions.

Particularly entertaining it say there that S-expressions don't have a notion of attributes. Which is a very astute observation, since attributes suck the big one.

It is true that any of these other formats "would do", and believe me, I am by no means an advocate of inappropriate, and over- use of XML, but the reality is that the proposed format is so tiny to begin with, and XML is so universally accepted, that it is practically moot whether this or that format would be "better". There are already a wealth of tools to index, mine, translate, etc. etc. XML.

I'm not claiming otherwise, just that XML is horribly overcomplicated and that it really ought not to be used as widely as it is. It would make no sense to say "it really ought not to be used as widely as it is", if it wasn't used widely.

XML is one of those historical errors that will bite us all in the ass from time to time for all time to come..

Re:XML... in its place.

[gnu-generation-one]

"It's so damned easy to parse XML these days, why reinvent the wheel having to parse a comma delimited file, a fixed width file, a bizzare internal format?"

because one requires a "while(<>)split(/\t/, $_);", and the other takes loading a library, DLL, or module, initialising it, setting up callbacks which may or may not be possible in the context of your function, loading data into the XML parser, then trying to keep track of your place in the data structure as the XML library floods your callback functions with crap and just praying that it's all thread-safe.

Plus text files are human-readable.
And smaller.
And take less memory to parse.
And are easier to debug ...or edit ...or validate

Re:XML... in its place.

[jonadab]

> parsing a comma delimited file when the records are reasonably simple
> in structure

Have you ever worked with real data? Real data has commas within some of the
records -- so then you switch to allowing records to be quoted, or commas to
be escaped. Then you get data with newlines embedded in the records, so you
have to allow those to be quoted or escaped too (unless you have a fixed number
of columns per row, which is the exception rather than the rule). So you
establish a rule that entries which start with ' or " are quoted, and you go
to the end quote mark even if there are embedded commas and newlines. Then
you have entries with embedded quote marks -- both kinds of them.

Bah. It's more complicated wheel than you think, and XML handles all this
stuff. There are other formats that handle it too, but XML is a perfectly
cromulent choice and is more human-readable than average.

Yes, it uses a few extra bytes, but the number of bytes can still be described
as O(n) -- i.e., the size grows only linearly with the amount of data that have
to be represented. Squabbling over whether the file is 10k or 100k is, this
century, misguided. There are more important things programmers should spend
their time on than saving a few bytes. Robustness, for example, is far more
important to the end user (and, in terms of maintenance, to the programmer)
than a few bytes one way or the other.

Re:XML... in its place.

[Tinidril]

Yes, I have worked with real data. Why is it that so many people on slashdot assume that if someone disagrees with them that they must be ignorant?

By moving from comma-delimited to XML you don't solve the problem, you just move it. What happens if someone includes text in a record that just happens to close your field? I know there are answers to that, but they are not very different from those with comma separated lists.

BTW: To my knowledge Microsoft is the only developer brain-dead enough to try and solve the comma-in-a-field problem with quotes around the entry. But then again they are the ones who are trying to use XML for everything now, so I guess it fits.

The correct way to do it is escape them with slashes, which is way less complicated than you make it sound.

',' becomes '/,'
'/' becomes '//'
NEWLINE becomes '/n'

Thats it! Any other escape sequences would just be for added human readability, and would be needed in XML for the same purpose.

Your comments really underscore my problem with XML. It claims to fix many problems, but in fact it just makes them more opaque. (Much like OOP, but thats another matter.)

At least you stayed away from the idiotic notion that I always hear about XML providing a standard format for structuring data. In reality it is no more standard than plain text. Which of these is correct?

<LUSER><UID>12<UID><NAME>Biff</NAME></LUSER>
<LUSER UID="12"><NAME>Biff</NAME></LUSER&g t;
<LUSER UID="12"><NOMBRE>Biff</NOMBRE></LUSER>

And Isn't this easier to read?

LUSER,12,Biff

IMHO: XML is excelent in a DocBook like implementation where the data will not fit into a clean record structure, but for all other implementations that I have seen it is snakeoil. It's more dificult for humans, more dificult for machines, and claims to fix a lot of problems that it just sweeps under the rug.

BTW: I manage a data retention system (not a relational database) that stores about 50GB/day and has to be kept on local storage for a full month. The data is replicated between two remote locations and backed up daily. If I had to move the data from comma-delimited to XML, our costs would more than double for bandwidth, storage, and labor (switching tapes). That doesn't even include the extra processing that would need to be done to reference the data. I'm not sure my boss would call that "a few bytes".

Re:XML... in its place.

[jonadab]

Wow, that was weird. It looked fine in preview. Let's try this again...

The difference is that XML-handling libraries all handle this automagically
(usually by encoding angle brackets within text data). Yes, it's possible
to have a library that does other escaping schemes automatically, but
there's still the issue of human-readability...

> <LUSER><UID>12<UID><NAME>Biff</NAME></LUSER>
> <LUSER UID="12"><NAME>Biff</NAME></LUSER&g t;

These will parse out to the same thing. And yes, if the records are all this
simple, and all *the same*, XML is unnecessary. But the minute the records
get even remotely complex, especially if some of the records have bits of
information that other records don't have, the human readability gets lost
in a sea of stuff like

LUSER 7125,Johnson,Biff,G.,,Jr.,,462-3203,44833
LUSER 6784,Johnston,Maria,,Taylor,,,468-1708,44833

Then you need a better structure than CSV. Is XML the only option? No.
But XML has the advantage of being fairly intuitive and strongly resembling
something (HTML) that everyone and his dog (thinks he sort of) knows.

<luser id="7125" zip="44833">
<name last="Johnson" first="Biff" middle="G." suffix="Jr.">
<phone>462-3203</phone></luser>
<luser id="6784" zip="44833">
<name last="Johnston" first="Maria" maiden="Taylor">
<phone>468-1708</phone></luser>

Yeah, it's longer. One order of magnitude longer than the CSV, not so much
longer than some of the other options. In many circumstances, the extra
length is a good tradeoff. I don't understand the desire to bash XML every
time it comes up, just because it's a buzzword. Sure, it's a buzzword, and
using XML doesn't really add inherent value, but it doesn't detract, either.
I still maintain, it's a perfectly valid choice.

> BTW: I manage [stuff]

That's nice. I'm TCG at a public library. We work daily (as does every
library) with a format called "MARC Records", and let me tell you, XML
looks mighty attractive.

Re:XML... in its place.

[Tinidril]

The difference is that XML-handling libraries all handle this automagically

How is that different? I could write a library to parse a CSV in about 10 minutes. Oh wait that is different. How long does it take to write a decent XML library? How many lines create how many bugs?

I think your points are fair, and not knowing anything about "MARC Records" I can't really comment on how XML would work for it.

I believe that there are good aplications for XML, but my reaction to it comes from the fact that people try to apply it in all sorts of places where it doesn't belong. (Like in a network protocol to validate emails) Bad programing bothers me because it makes bad programs that I may be forced to use at work. If this takes hold I will end up involved in tracking down email problems, and instead of being able to use a simple split command to break down the data I'll have to deal with mountains of useless tags.

My favorite mis-application of XML was made by Cisco for a network load-balancing device. They built an XML interface for bringing servers in and out of rotation, and it was the only way to automate the process. It never worked right, and even there own tools could never do the job reliably. I don't know how many hours I spent pulling my hair out on that one. A high-school kid should have been able to write that interface in 10 minutes, but using XML it was a nightmare.

We're probably closer in our thinking than our posts let on. I still don't see a single problem that XML solves for structured data, but for documents it has no equal. In the real world I'm sure there are all sorts of places where the line between structured data and document data is blury.

BTW: I love your "Lamejoke Generator".

At least

[pubjames]

At least this is one area where MS will have a real problem using their monopoly to enforce a closed standard. A solution that doesn't work for people that don't use MS software just isn't going to fly.

Having done work on (opt-in) HTML newsletters for clients, I know that email clients used are really varied - more varied than web browsers for instance.

Re:At least

[liquid-groove]

RTFA - Microsoft proposes a standard which any vendor can implement and provides a license for its use on the website describing the process. There sis nothing client specific about the implementation.

Parent is +5 interesting? Could anyone who moderated it up provide a reason other than they're bashing MS, that's +1 baby!

Re:At least

[pubjames]

RTFA - Microsoft proposes a standard which any vendor can implement and provides a license for its use on the website describing the process. There sis nothing client specific about the implementation.

I did read the article. But MS has a history of breaking standards to create customer "lock-in", and also trumpeting open standards when in fact what they finally implement isn't open at all (Office "XML" for example). What I'm saying is that, in this case it would be difficult for MS to do that because email client software is very varied.

Re:At least

[NightRain]

Could anyone who moderated it up provide a reason other than they're bashing MS, that's +1 baby!

Well no. They can't comment if they moderate now, can they?

Ray

Re:At least

[Gaijin42]

Have you ever used office XML? I have. Their namespace is of course proprietary, but EVERYONE's namespace is proprietary. There isn't a standard document schema out there. (And no, OpenOffice and StarOffice etc are not standards, they may be open, but they are not a standard.)

The XML is in plain english (well technical english maybe, but it isnt encrypted/encoded gibberish) , and very easy to use. I write applications all the time that output word, xl, and popwerpoint files from code.

I think you just like to bash MS.

Re:At least

[Hard_Code]

Not only would it be difficult, it would be POINTLESS because spam prevention only works if EVERYBODY DOES IT.

Re:At least

[evilad]

Have you ever tried to emit those types of compound documents without using any Microsoft controls? I.e., on another platform? A non-trivial task.

Re:At least

[mrogers]

And when you post, your previous moderations in that discussion are undone.

D'oh!

Re:At least

[Illissius]

Personally, I think in this case MS is actually, honestly trying to do the Right Thing. And it's easy to see why. What is one of the three biggest reasons the average user would even consider moving away from MS and Windows? Exactly. (The other two are spyware and virii. Popups don't get a seperate category, as they're just another form of spam.)
Microsoft realizes this, and are trying to fix it, in their own very good interest. See also: SP2 contains antivirus, an upgraded firewall, a popup blocker integrated into IE, buffer overflow protection for processors that support it (Athlon 64 and Opteron currently), and I assume there's more.
So you can safely expect for it to be That Much Harder convincing people to move to *nix, once SP2 is released. Do it while you still can. (Note again that I am not saying *nix will lose any advantage it has/had over Windows. Merely that in the eyes of the average user, it will.)

Re:At least

[cavemanf16]

Well, it's really called GnuPG, but you're right, it is the standard that basically states: "the sender's signing key validates against the original key you trusted by signing it with your own key." I've started signing all of my emails in Thunderbird using the help of the Enigmail plugin and encrypting any files I attach in my emails with the help of WinPT. I know this post looks like a giant plug for these "products," but since they're all free, open source software which I have no affiliation with, it's simply me trying to get the word out that there IS a manner in which to get your emails to your friends in a trusted, reliable manner, and hopefully convert a few of your friends and family to using the same method in the future. We wouldn't have to worry about address spoofing if email gpg signing was a defacto standard of every email client! Plus it would be a lot safer and difficult to circumvent (ultimately) than Yet Another Format for email.

two things

[WegianWarrior]

Whats to stop a spammer from signing up for a free email account with a false name, blast out a few thousand messages, drop the account (it'll be closed anyway by abuse), wipe hands and repeat?

True, I see how this may help stop some spam, but it also means (if I understood the article correctly) that everyone can find out where I mail from... and in some instances that could be a problem too.

Re:two things

[geminidomino]

True, I see how this may help stop some spam, but it also means (if I understood the article correctly) that everyone can find out where I mail from... and in some instances that could be a problem too.

It's the classic claim that "If you're not doing anything wrong, you've got nothing to hide" anti-privacy excuse.

Re:two things

[leerpm]

Whats to stop a spammer from signing up for a free email account with a false name, blast out a few thousand messages, drop the account (it'll be closed anyway by abuse), wipe hands and repeat?

I don't know about all free email services, but Hotmail does not allow this anymore. Accounts are limited in how many messages per day they can send out. This is why most spammers are still relying on open relays and zombie machines.

Re:two things

[zero_offset]

In addition to what that other guy posted (accounts having daily limits), sending mail through those types of systems is generally just too slow to be of interest to dedicated spammers.

A couple years ago I wrote a bunch of software for very large e-mail runs -- not spamming related, but the lists were in the high hundreds of thousands -- and to successfully blast out hundreds of thousands of e-mails in any reasonable amount of time requires quite a bit of planning, software built for that purpose (our evals showed even the well-known and venerable lsoft offerings perform abysmally for these purposes), not to mention having a fairly hefty chunk of bandwidth at your disposal.

Re:two things

[blowdart]

True, I see how this may help stop some spam, but it also means (if I understood the article correctly) that everyone can find out where I mail from... and in some instances that could be a problem too.

I don't think so. What people can find out is what IP addresses are valid when sending email from a domain. Nothing more. All they are doing is a lookup on the connecting IP against the FROM: domain. Hell, that information is in your headers anyway. (Well unless you're using a remailer)

Re:two things

[Hard_Code]

So don't comply and risk getting your mail dropped. You can have your privacy, but you can't FORCE others to read mail from suspicious and unknown sources. Your call. There are plenty of non-email alternatives to be anonymous. Post in a random newsgroup from a web cafe. Or use a secure IM protocol, or secure IRC.

Re:two things

[kinnell]

This is why most spammers are still relying on open relays and zombie machines.

Which begs the question, how does this solution deal with zombie machines, given that these are being used more and more to send spam? It shouldn't be too difficult to set up a trojan remailer which uses the user's email account to forward spam. Wouldn't this be declared as valid, and presumably laying the blame on the user.

Re:two things

[walt-sjc]

It doesn't even take a free account.

The major problem with ALL these systems is critical mass.

Corporations are not going to be blocking mail based on a lack of SPF, Caller-ID, or anything. Too many companies are going to be slow to implement, or apathetic about it. No larger business is going to block mail and potentially lose contact with potential customers, or existing clients.

90% of the current crop of spam would stop if all ISP's would block outbound port 25 from dynamic IP clients by default (unblock if the client agrees to keep their system patched and secure and face penalties if found spamming.)

For the most part, open relays have been closed due to RBL like activity, as enough sites use RBL's to make life very difficult for admins that leave their systems open. So spammers have moved to dynamic's, which there is a virtually unlimited supply due to the piss poor security of Windows and clueless users. RBL's are helping with that too, but it's hard to keep up. Again, many corporations won't use RBL's due to problems noted above.

While I have not read the detail on MS's solution, SPF has the "roving user", "mail forwading" problem that there is no solution for that has been discussed to death. Anyone know if MS's solution has the same problem?

Re:two things

[m00nun1t]

Maybe it's not absolutely perfect. But what protocol is? Here's a list of other protocols that have major problems:
TCP/IP
HTTP
SOAP
FTP
SMTP

If /. was in charge of releasing protocols, the internet would never have happened. There's always someone finding a problem. Well, guess what, there is always a problem.

Instead of complaining, contribute, find a good place to start with and improve it over time - that is what has happened to all the above protocols.

Re:two things

[Snowmit]

True, I see how this may help stop some spam, but it also means (if I understood the article correctly) that everyone can find out where I mail from... and in some instances that could be a problem too.

That's true in the real world too. They're called postmarks. You may have seen them stamped on your snail letters.

Don't like it? The don't send email that complies with the standard and hope that the people receiving are willing to read letters from people who aren't complying. Or use a messageboard. Or a webcafe.

Re:two things

[EJB]

I can see that this can cause problems as a consultant. You're connected to the network of customer A, and have to send an e-mail to customer B.

You don't necessarily want customer B to know that you also work for customer A.

- Erwin

Re:two things

[Alioth]

Wouldn't this be declared as valid, and presumably laying the blame on the user.

Yes - and then we'd know exactly who's machine has been trojaned with much less effort. The ISP can then disconnect them until they have patched their OS/removed the trojan.

Re:two things

[Alioth]

The SPF website gives the solution for the 'roving user' and 'mail forwarding' problems.

In summary, the 'roving user' problem can be solved by any of the following:
* SASL enabled SMTP on the SPFed SMTP server for the domain. Users then send their mail via that server instead of $RANDOM_ISP server. Port 25 blocking by the ISP isn't an issue since there's another port for SASL SMTP.
* Provide web mail access for roving users.
* Provide shell access for advanced roving users.
(Personally, I use the latter).

The forwarding problem can be fixed by rewriting the envelope. The solution is shown on the SPF website.

Re:two things

[mlefevre]

ISPs can already see exactly whose machine has been trojaned from the time and IP. Checking their logs to find that info is trivial - the tricky part is getting the user to patch/clean their computer. Knowing the email address of the person whose machine is trojaned doesn't really help the recipient.

Having correct sender addresses would be nice, and would force spammers and virus writers to adapt somewhat. The question is whether the effort of implementing it is worth it for the gains available.

Re:two things

[JerkBoB]

You're connected to the network of customer A, and have to send an e-mail to customer B.

... So you connect to your own firm's mail server and use SMTP AUTH to authenticate yourself and send mail through it. If customer A has network nazis working for it, you connect to your own firm's webmail service.

Problem solved.

Re:two things

[kaisyain]

So Customer A should allow 3rd parties to forge email through their servers to support business unrelated to Customer A? Why should they be doing the consultant that favor? And why do they care if the consultant is at their site (probably billing them) and can't waste time solving Customer B's problem?

Re:two things

[geoffspear]

Yes, but none of those protocols were developed and patented by Microsoft. What do you think the chances are that MS is going to allow the open source community, or, for that matter, anyone outside of MS to contribute to their "standard"?

And considering how they treat other standards, why should anyone trust them? Look at what they tried to do to Java... it was intended as a standardized programming language that would work exactly the same on any platform to allow the creation of truly portable applications, and they decided to use their browser dominance to get people to start using a non-complying implementation and writing code that wouldn't work except on Windows machines. This, more than IE's HTML and CSS extensions that make the vast majority of the pages on the web non-valid HTML, shows their complete disregard for the concept of "standards". They only like a standard insofar as it can help them control the market.

Re:two things

[jafiwam]

One could also use "pop-from" authentication for roving users.

I've got a whole mail server with 20k users on it that only half of whom connect to the same network.

"Pop-From" just makes the user check for mail first, when a successful authentication for POP3 has been completed, SMTP traffic is opened up for 30 or so seconds after the last download. The email software then can have SMTP accepted by the server to send out their mail.

It works great, no messing with settings for roving users at all. Just about any email program I have run across works with it without further messing around, though Outlook has to be told to no "send immediately" otherwise the mail gets stuck in it's outgoing queue.

Re:two things

[Eivind]

Well, the nice thing about SPF is that it works, and has benefits even if not everyone uses it.

For example, it allows me to tell SpamAssassin that IF a domain has SPF-records, and the email doesn't come from one of the ips that send mail for that domain, then in the spam-bucket it goes.

Thus, for example, all the spam that claims to be from hotmail is gone.

Secondly, I can, by publishing spf-records on my own domain eliminate the problem of spam bouncing back to me because it *claims* to be sent from me.

Third, once a sufficient part of the people I communicate with email from domains that *have* spf-records, I'm free to, for example, implement a challenge-response system for email coming from other domains. Yes, this will mean people using those domains gets some challenges based on spam that only *claimed* to be from their domain, but actually isn't. That migth serve as a good incentive to get them to also publish spf-records. It's not as if it's a huge deal to stick 2-3 extra records in your dns-info.

Re:two things

[dbc]

Corporations are not going to be blocking mail based on a lack of SPF, Caller-ID, or anything.

??

Why do you say that? It doesn't make sense to me. Corps large enough to have 1+ mail admins already are up to their armpits in deployed and operational spam and virus filtering tools. SPF doesn't have much downside for them, only upside. Maybe *tiny* companies where the mail server is a 1 hour a week of some programmer that has been saddled with playing net-admin during his lunch hours will be slow to get this rolling, but it seems to me that companies with actual IT staff will be pretty quick about it.

Roving user is not an issue for big companies, since the road-warriors need to VPN into the corp net to get to the mail server anyway, so viola, they are no longer "roving" as far as SPF is concerned.

Feel free to convince me that I'm wrong. Use data, actual experience, and facts. OK -- I realize that using any of those three is a risk to one's karma. Post AC, if you need to :-)

Imagine when Hotmail gets this

[ObviousGuy]

Microsoft is one big player in the email world through their Hotmail service. They probably serve more spam to more places than any other single mail service. As such it makes sense that they would want to be at the forefront of spam-elimination technologies. They ought to be applauded for their initiative here, as well as their cooperation with SPF and Sendmail.

However, it disconcerts me that they are also applying for a patent in this area instead of engaging the community through a consortium-like committee that could share the technology across the board unencumbered by licensing fees. The specter of Hotmail becoming a proprietary mail system requiring foreign mail servers to run Microsoft-licensed "Caller-ID" to interact with Hotmail is a very legitimate concern.

Re:Imagine when Hotmail gets this

[liquid-groove]

As part of an overall spam identification and scoring system, the MS standard and the Yahoo proposed standard are both interesting pieces of the puzzle. They are hardly solutions to the spam problem in and of themselves and unilateral implementation of either protocol as an absolute requirement for acceptance of incomining communication by either Hotmail or Yahoo would likely be met with a varacious subscriber backlash which would result in decision being revered within hours.

Re:Imagine when Hotmail gets this

[leerpm]

However, it disconcerts me that they are also applying for a patent in this area instead of engaging the community through a consortium-like committee that could share the technology across the board unencumbered by licensing fees.

It is called defensive patenting. There is nothing wrong with applying for a patent on this. We do not want another Eolas, where some other company that produces zero innovation gets a patent on it instead, and puts a strangehold on the industry. While not perfect, Microsoft has been pretty good about not going after other companies with frivolous lawsuits over patenting issues. Since the USPTO now seems to accept pretty much anything, companies have to apply for patents on whatever possible, so that they have something to use to defend themselves in the future.

Re:Imagine when Hotmail gets this

[leerpm]

Sorry for replying to my own post. But I missed another bit of information:

From the "terms of the patent license for implementing this specification":

"Microsoft and its Affiliates hereby grant you ("Licensee") a fully paid, royalty-free, non-exclusive, worldwide license under Microsoft's Necessary Claims to make, use, sell, offer to sell, import, and otherwise distribute Licensed Implementations, provided, Licensee, on behalf of itself and its Affiliates, hereby grants Microsoft and all other Specification Licensees, a reciprocal fully paid, royalty-free, non-exclusive, worldwide, nontransferable, non-sublicenseable, license under Necessary Claims of Licensee to make, use, sell, offer to sell, import, and otherwise distribute Licensed Implementations. "

Re:Imagine when Hotmail gets this

[PhotoBoy]

This is a bloody pain in the neck. SPF was just starting to look like it might be adopted on a large scale basis an MS have to stick their proprietary oar in.

I don't want to have to make my mail servers compliant with this AND SPF, I also do not like the idea of sending XML packets to/from Hotmail (and other MS mail system) for every email allegedly from them.

Also I'd rather not use an MS solution since there are always security holes. How long till the spammers find a way around this and start sending out spam via a flaw in Hotmail?

Re:Imagine when Hotmail gets this

[Anonymous Coward]

Reread it.

If you implement the patented technology, you must allow MS to use and distribute YOUR IMLPEMENTATION if they want to.

I.e. Give them your code.

Re:Imagine when Hotmail gets this

[bigsteve@dstc]

> You mean, open source? Yes, I think.

But I don't think you could prevent MS from distributing your software in a closed source product. In particular, it says this:

You are not licensed to distribute a Licensed Implementation under license terms and conditions that prohibit the terms and conditions of this license.

You are not licensed to sublicense or transfer your rights.

(I think that) one effect of those two sentences is that you cannot distribute a Licensed Implementation under the GPL. The GPL grants the right to sublicense (among other things) that the "Caller ID" license does not allow.

MSXML experience

[RobertB-DC]

I've had the unfortunate experience of attempting to generate XML using Microsoft's MSXML object. What a piece of crap! In an attempt to completely abstract the format, the objects are obfuscated beyond reason. Even the simplest things require ridiculous complexity: just to escape-out special characters requires instantiating a new "entity" element in the middle of the text string element.

And I still haven't figured out how to make the thing give me a CRLF at the end of each element. No, XML doesn't require the whitespace, but it would have sure made it easier for my clients to read the file!

But the worst part is that I *succeeded* in using MSXML. Now, if I wanted to go back to just writing a text file (which I do!), I can't -- my code is tangled up in the objects to the point that it would take a complete rewrite.

That's the simple reason why, every time I hear about Microsoft doing something with XML -- like this proposal to use XML as part of email identification -- I cringe in ph33r.

Re:MSXML experience

[chrisbtoo]

And I still haven't figured out how to make the thing give me a CRLF at the end of each element. No, XML doesn't require the whitespace, but it would have sure made it easier for my clients to read the file!

Tell me about it. My favourite part is when you try to load one of their MSXML-generated files into their Visual C++ 6.0 product and it bitches about lines being greater than 2048 characters long and how it's going to shove random line breaks in the middle of tags.

Thanks, MS!

Re:MSXML experience

[Cereal Box]

just to escape-out special characters requires instantiating a new "entity" element in the middle of the text string element.

Maybe that's the "right" way to do it, but I highly doubt that you cannot set the value of a text node to a string that contains an entity (i.e., "this is an ampersand: &amp;"). That would be the more direct approach.

And I still haven't figured out how to make the thing give me a CRLF at the end of each element. No, XML doesn't require the whitespace, but it would have sure made it easier for my clients to read the file!

First, you could have them read the file with Wordpad or just about any text editor other than notepad. And BTW, why are you complaining about MSXML not generating CRLF? You DO realize CRLF is a Microsoft-ism and not "standard", right? So you're complaining about MSXML generating text files in a manner more in line with the way every other system does it. Baffling...

But the worst part is that I *succeeded* in using MSXML. Now, if I wanted to go back to just writing a text file (which I do!), I can't -- my code is tangled up in the objects to the point that it would take a complete rewrite.

I've got news for you -- every decent XML parser library requires you to manipulate the XML tree in an object-oriented manner! It's called the Document Object Model for a reason -- you're not manipulating raw text! You can go ahead and do that if you like, and we'll see how much "easier" that is for any project requiring more than the most basic use of XML.

Mods, get a clue. The way the MSXML library handles XML is not unique in some "Microsoft always makes crap" kind of way. Every decent XML library handles XML the same way.

Re:MSXML experience

[the endless]

I've had the unfortunate experience of attempting to generate XML using Microsoft's MSXML object. What a piece of crap! In an attempt to completely abstract the format, the objects are obfuscated beyond reason. Even the simplest things require ridiculous complexity: just to escape-out special characters requires instantiating a new "entity" element in the middle of the text string element.

Er... in that respect, Microsoft are following the standards, because that's how it's done with the W3C's Document Object Model. If you have a problem with it, you have a problem with the DOM, not with Microsoft.

But the worst part is that I *succeeded* in using MSXML. Now, if I wanted to go back to just writing a text file (which I do!), I can't -- my code is tangled up in the objects to the point that it would take a complete rewrite.

Again, that's your fault, not Microsofts. Either live with it, or split out the XML-generation code into a separate module. The world and his dog has long since learned to separate out logic code and database-access code so that it's possible to change DBMS by just rewriting the database-access module rather than the entire application - exactly the same thing applies with XML.

Re:MSXML experience

[pohl]

I've got news for you -- every decent XML parser library requires you to manipulate the XML tree in an object-oriented manner! It's called the Document Object Model for a reason

This isn't true. The SAX API is event-oriented, and though it may be a little bit more difficult to wield than DOM it has the advantage of giving you complete control over memory allocation. That is, you can allocate as little as you need, and only when you need it, whereas DOM libraries allocate all that is required to completely represent the entire document in memory up-front.

Every decent XML library handles XML the same way.

Also not true; the same example suffices.

Re:MSXML experience

[Cereal Box]

What I meant was that every decent XML parser requires you to handle the XML tree in some manner other than messing with raw text, like the original poster seems to think the optimal way to do things. SAX or DOM -- either way you're going to have to deal with all sorts of objects representing things like nodes, text, etc.

Re:MSXML experience

[dasmegabyte]

And I still haven't figured out how to make the thing give me a CRLF at the end of each element.

Tabbing, spacing and linefeeds are not required in XML, and everybody wants to use them a little differently. No, MS' API doesn't do it automatically, but you can do it programmatically fairly easily by appending an XmlWhitespace object after each XmlNode. You can retreive one of these from an XmlDocument by calling the doc.CreateWhitespace(sting whitespace) method.

I do that at the beginning of the program, assing the whitespace to a variable called xCRLF (along with another called xTAB), then every time I do a Node.AppendChild(element), i call Node.AppendChild(xCRLF) as well. Result is perfect tabbing.

MS' xml API is pretty robust and fairly easy to use, even if it gets a little crazy. For the longest time they had a superior XSL processor (now it's about equal to XALAN), at least from a "I am an XML idiot trying to learn how to use the technology BEHIND the keynote speeches" standpoint.

Zombie Boxen hastens Trusted Computing?

[G4from128k]

Caller-ID for email will help prevent spoofing, but will only increase spammers use of zombies. I wonder if increased exploitation of Microsoft OS weaknesses (to create spammer platforms) will have a long-term detrimental effect on Windows or whether it will hasten adoption of Trusted Computing? I wonder if Microsoft wants ISPs to become so sick of zombie boxen that the ISPs will prohibit all but a few chosen OS options (read the lastest version of Windows) for connection to their networks.

For a very well-entrenched provider, making everyone sick of you old product is a good way to force them to buy your new product.

Re:Zombie Boxen hastens Trusted Computing?

[tiger99]

Sadly you are right. Almost all the trouble I get now is from zombies (not sure if I mean the PCs or their owners!). Of course most of it happens because the stupid morons are continuing to use Outlook, which is a singularly pathetic program apart from its major security holes.

As an aside, I set up a firewall, and the equivalent of Internet Connection Sharing (i.e. forwarding) on a Linux box the other day, IIRC it needed 4 lines of commands to iptables in one of the startup scripts, which being lazy I got out of a book. I went to grc.com for a test, and it was every bit as good as Zone Alarm, a product I use successfully on the inferior OS.

The point is that in an open OS, useful and essential things tend to be fully documented, visible, and easy to set up. I fear that in this case, Sir Bill's anti-spamming system will be obfuscated, needlessly difficult to configure, and will at the slightest provocation automatically default to doing it Sir Bill's way, even if that is not what you want. There is a precedent in every previous M$ application, the world's most unpopular Word processor being the prime example.....

It is of course another con trick to move us towards Longhorn, which on its own would get no acceptance whatsoever, because its drastically cut-down API set will break compatability with virtually everything. of course, if the Convicted Monopolist was competent, they would have had a much smaller, more manageable and properly documented API set in the first place, and we would not have nearly as many bugs, crashes or security holes.

It seems to me that someone needs start the RFC process right now, describing a properly working, non-proprietary system. Otherwise, the Convicted Monopolist will once again do as described in the Halloween Documents.....

Re:Zombie Boxen hastens Trusted Computing?

[cca93014]

Ah yes, because every day users can comprehend IPTables.

I mean, wtf?

thanks

[flaez]

if it will mean I have to pay fees to Microsoft to get my domain signed, I'd rather continue filtering out spoofed-bounces, thank you.

Interesting how instead of supporting a perfectly sound project that has been going for a year, everybody seems to have to come up with their own little *patented* scheme.

Re:thanks

[zerocool^]

One of the most effective ways I've ever seen to filter out mail is to just simply follow the RFC. When you get mail from a domain name, look up the ip address, when you get the ip address, reverse lookup the name. If forward and backward don't match, reject the mail.

Unfortunately, this rarely is implemented. Why? People can't seem to figure out how to set up their DNS zones. So whenever I've implemented it, we always get calls from people saying "my mail is getting bounced, error code 0-B". And we go and look, and it's some client trying to send mail from their in-house mail server legitimately, but they don't have it configured properly in DNS.

The volume that we get of people complaining about it is high enough that we can't leave it turned on, and I'm unwilling to do tech support on someone else's name server. So, even though it blocks about 1/3 of all the spam we get, it stays off.

~Will

Re:thanks

[rjw57]

SPF is already a IETF draft, the first stage towards RFC-style standardisation.

Re:thanks

[Masem]

For a lot of home residental (DSL) users, it's very hard to get the upstream ISP to implement reverse mapping on the DNS, since the ISP is the one in control of the IP number, not the end user. The end user can point domain names all they want to the IP, but reverse mapping will always come up with the ISP's naming scheme. This is a nice idea, but in practice, it's not going to work.

Mine you, you're talking about your block of residental DSL users that run their own mail server (commercial DSL users generally do get the reversing mapping through their ISP); they will most likely not be clients and may be a larger source of spam than other sources.

Re:thanks

[warrax_666]

There is another reason. You can't reverse map one IP to several domains -- so no sending mail from multiple domains from one box. That's why using reverse lookups is a bad idea. Note that SPF (and similar schemes) does not have this problem because the authorization is in the actual domain you're (claiming to be) sending from, not the reverse domain.

Re:thanks

[RedHat Rocky]

Why should I, as a DSL user, depend on my ISP's SMTP server?

I don't depend on their nameservers, either. I don't want anything from them except to have my packets passed up the line.

Why? Control. If I only use services I provide, I am responsible when they are broken and I CAN FIX THEM.

Labelling all SMTP servers on DSL lines as spam relays is really over the line.

Re:Why not?

[Marxist Commentary]

Then why not just call you in the first place, and do away with the email?

sucks / rocks

[jilbert]

I hate XML, and a quick google reveals:

XML sucks = about 215,000
XML rocks = about 174,000

I'm pleased to see I am in the majority - I thought its buzzword status would have rated it higher.

Re:sucks / rocks

[stanmann]

What you were looking for is:

XML Rules = about 2,580,000

Re:sucks / rocks

[Trigun]

Ooh, edifact! I dream of edifact! We're still using ANSI/X12 EDI.

For those of you born after most mainfraimes, ANSI EDI is Satan's preferred method of data exchange. It is based on the assumption that characters are expensive to transmit, so they minimize the file to as few characters as possible using codes that might have had meaning when they wrote the standard, but not anymore. Most times, the files don't even transmit eol characters. It's a mess!

Re:sucks / rocks

[jimi1283]

no no no, you've gotta do it with quotes, otherwise you just get a lot of .xml files with the other key words in them:

"XML rocks" = 79
"XML sucks" = 671
"XML rules" = 5630 (obviously they're actually talking about rules here, and not commenting on quality - perceived or actual)
"XML pwns j00" = 0

Obviously the poor kids using 1337 speak have obviously never picked up the standard...

Re:sucks / rocks

[ideonode]

I can assure you that when Satan sends messages, he sends them ASN.1-encoded. Especially the BER encoding, which doesn't even have one canonical means of encoding.

It's so much fun that it causes buffer overflows all over the place (Microsoft OSes, OpenSSL...)

PR Issue

[millahtime]

I do believe this is one area we have to really keep an on eye on M$ in. Do they really want to stop the spam or is it just PR. They have the browser that doesn't block pop ups and on a default install of windows Ad-Aware will find things it considers an issue right after the default install.

This may just be a PR issue to show people they are pushing for it. When they implement something like this will they put their own hooks in it to allow what they want???

M$ really needs to be kept an eye on if they do this.

If Microsoft cared about SPAM...

[Knertified]

They would have allowed a user to disable a the javascript popup function in the browser. Instead we have to rely on bandaids like googles toolbar to block popups from websites.

Re:If Microsoft cared about SPAM...

[jfengel]

It shouldn't have taken so long, but they claim that it's coming.

Re:If Microsoft cared about SPAM...

[no soup for you]

If Microsoft cared about SPAM......allowed a user to disable a the javascript popup function in the browser

I think that's a pretty expansive definition of SPAM. Does everything annoying become SPAM? I see popups as advertising (and something that mozilla effectively killed for me), and SPAM as fraud.

Danger! Read the fine print!

[Eponymous Cowboy]

Look what happens if you add support for "Caller ID for Email" to your software:

Microsoft and its Affiliates hereby grant you ("Licensee") a ... license ... to make, use, sell, offer to sell, import, and otherwise distribute Licensed Implementations, provided, Licensee ... grants Microsoft and all other Specification Licensees, a reciprocal fully paid, royalty-free, non-exclusive, worldwide, nontransferable, non-sublicenseable, license under Necessary Claims of Licensee to make, use, sell, offer to sell, import, and otherwise distribute Licensed Implementations.

(From Microsoft's license.)

So by building support for "Caller ID for Email" into your software, you suddenly give Microsoft an unlimited license to use and sell it. And, in fact, not only Microsoft, but everyone else who writes software that supports "Caller ID for Email."

There is a word for this: Insane.

No thanks. I'll stick with SPF--especially since the two are essentially identical, just a slightly different parsing format.

Re:Why not?

[kalidasa]

They already have systems that do this [challenge-response], you know. This doesn't require any changes to standards; but it does require that the sending user be clueful - and given how quickly Netsky.C spread, I think that's a hopeless cause.

In the US at least, caller-ID is not a challenge response system, it simply displays the originating phone number - and ONLY if you haven't requested that your number be hidden, and only if you live in an area that supports it.

So, what lessons can we carry from this fact to MS's suggestion of "caller ID" for email? 1. We'll still get emails that are unauthenticated, because it will take a long time for folks to upgrade MTAs to manage this - after all, there are still open relays - and 2. someone will figure out some way to sell a solution to get past the authentication system so blocked spam senders can still get through (can you say "sales@viagra.hotmail.com"???).

Re:Why not?

[leerpm]

Why not have *real* caller-ID for email authentication? Before you can get on my white-list, you have to call a phone number for some sort of challenge-response

So every person that wants to email you, now has the added burden of phoning some system and following the voice menu options? I think that most people will simply not bother and won't send the email at all.

Email is a great tool and easy to use. Even existing challenge-response systems have been found to have many problems. Let's not ruin email, by taking away the best parts of it. Any authentication needs to be seamless and the details should be hidden from end-users.

Re:Why not?

[geminidomino]

That still has the same problem as every other C/R system. In order to GET that phone number, presumably every email is responded to by a notice to call that phone number. It still bombards the poor shmuck whose email was forged with C/R requests.

Why we shouldn't use XML here...

[doofusclam]

... because the performance is crap. This is true on my pc (with any parser you care to name - i've tried it) so what it'd be like on a mail server handling x thousand messages a minute I have no idea.

XML is great, but only when the underlying data is sufficiently variable within a pre-defined schema and where throughput is not an issue. It's not necessary here.

sean.

Re:Why we shouldn't use XML here...

[viktor]

Oh, pleeeeeze!

Is there no end to the Microsoft-bashing in this forum?

If Microsoft had done this using a home-made format, then everybody would be screaming death to them for inventing their own standard "just like they did with Word documents".

And when they do use a public format like XML? Then we all scream death to them because XML is so bloated etc. etc.

It's time to grow up.

PS. I will NOT make the mandatory "I really don't like them, but in this case..." argument, which seems to be the only standardized way of saying anything positive at all about Microsoft here.

Re:Why we shouldn't use XML here...

[doofusclam]

Oh Pleeeeeze yourself.

I ain't bashing Microsoft and I don't spell it with a '$' either. I've spent the last 14 years programming using their tools and operating systems, so quit with thinking i'm an OSS zealot.

So read my comment again - i'm not bashing them, and at least they're doing something about spam. But for such a simple datastream, with the throughput needed, it seems unnecessary to bloat it (cpu and memory wise) by having to use an XML parser, regardless of which evil/non evil company designed it.

Would YOU like your mail to be delayed because some bright spark decided to go all trendy and use XML in the mail processing rather than something which just does the job?

Re:Why we shouldn't use XML here...

[DarkEdgeX]

XML parsing isn't that slow, I bet the time it takes to download each message will be the bigger bottleneck until latency and throughput for in-home internet connections is to the point where we need gigabit ethernet to get connected.

The point for XML is that it's a standard way of presenting data. No issues with using commas, tabs, or INI-style presentations (or issues presumably with the differing end of line format between *nix and Windows/DOS).

IOW, I sincerely doubt the overhead is going to kill anyone.

Port 25

[npcole]

On a first reading, I thought the ideas seemed quite sensible. One problem they did address in an interesting way was that of people with several email identities. One of their suggestions is that whoever is hosting the incoming email provides outgoing smtp services too, which would be a change from the (outdated?) idea that one should always use the "nearest" smtp server for all email. Though ISPs who currently block outgoing port 25 (such as my University!) would have to think again.

N.

Because it would not work...

[Matthias Wiesmann]

Why not have *real* caller-ID for email authentication? Before you can get on my white-list, you have to call a phone number for some sort of challenge-response. Caller-ID could be part of this.

I really don't see the point of including the phone in the system. Processing voice calls is complex and expensive and has no advantage over online processing. Either the thing is done manually, and would be damn expensive, or it is automated and would have no advantage over doing it over ip.

Did you consider that e-mail are used outside the US? I am certainly not going to pay a trans-atlantic call each time I want to send an e-mail to a new guy in the US. What about people that don't speak English? What about people who don't have a phone, or don't have a number on a system that supports caller id? With the advent of IP phones, this would become more and more common.

bad idea

[Anonymous Coward]

In General Ackbar's legendary words'It's a trap!'

Spoofing SPF?

[mmerlin]

I guess the Joe-Jobbers will be hard at work trying to find all the ways of spoofing SPF.

Zombie writers will be in even greater demand from the spam factories.

Apart from spammers using zombified users email accounts, are there any other possible ways around SPF?

Having read the executive summary and skimmed a few pages, the general precepts make sense.

At the very least, the transitional phase of mass implementation of SASL or similar (which IMO should be mandatory for mail servers anyway) is a Good_Thing_(tm)

Granted it will take a lot of time and effort for the second phase to be reached, but anything which cuts down on spam gets my vote!

microsoft.com already doing this

[ergonal]

Not sure if this is mentioned in the .doc, but _ep.microsoft.com already appears to be doing this:

_ep.microsoft.com. 1H IN TXT "<ep xmlns='http://ms.net/1' testing='true'><out><m>" "<mx/><a>213.199.128.160</a><a>213.199.128.145</a> <a>207.46.71.29</a><a>194.121.59.20</a><a>157.60.2 16.10</a><a>131.107.3.116</a><a>131.107.3.117</a>< a>131.107.3.100</a>" "</m></out></ep>"

Good idea

[broothal]

This is a good idea, and we (tinw) has discussed this many times before, and various implementations already exists (that is - verifying the sender domain, not the specific MS implementation).

Now, what bothers me is this line:

Microsoft believes that it has patent rights (patent(s) and/or pending applications(s))

Given the latest stories on how easy it is to patent everything "over there", I am pretty sure MS is granted this patent. Now I don't know about you, but this geek ain't licensing nothing from MS.

Re:not even MS can't produce readble word document

[chrisbtoo]

The SPF guys have them: http://spf.pobox.com/caller-id/

Damn advertising-like clause again

[rjw57]

In the license Microsoft grant implementers there is the following nasty clause:

If you distribute, license or sell a Licensed Implementation, this license is conditioned upon you requiring that the following notice be prominently displayed in all copies and derivative works of your source code and in copies of the documentation and licenses associated with your Licensed Implementation:
"This product may incorporate intellectual property owned by Microsoft Corporation. If you would like a license from Microsoft, you need to contact Microsoft directly."

Isn't this incompatible with the GPL?

SPF?

[TheTomcat]

I looked into SPF, briefly, and it doesn't seem to solve a problem I have...

I have various (virtual) users (~20-25) on my domains.

These users use both my SMTP server (when using squirrel mail, or (ssh-)tunnelling to the SMTP server, itself), as well as their local ISP's mail server (sympatico, videotron, etc)... My SMTP server doesn't relay from anywhere except localhost.

So, in order for SPF to work, I need to allow email from my domain, and these ISPs.

The ISPs are large, and when an email virus goes around, mail is undoubtedly sent "From" me (actually from/by outlook users with me in their address books), through these ISPs' SMTP servers, making SPF useless.

Am I just missing something?

S

Re:SPF?

[weave]

Remote users have to use your SMTP server and authenticate using SMTP AUTH. saslauthd is the necessary glue to make it work with pam, if that is what you use for authing other services.

XML Is Awesome When....

[Omega1045]

XML is awesome when you are looking for interoperability between different applications/systems. I would think that when the Internet community agrees upon whatever protocol, it should be a common standard and will not need the benefits of XML. Indeed, XML would actually be a bad choice, as the extra market will just use more bandwidth. Sounds like MS should just bow its head, say thanks to SPF, and adopt it. If they want XML on their side, then let them right an internal API/converter so those developing with Exchange or Outlook will have access to an XML version. By leave it off the pipe! And I say this as a guy who works with XML everyday and enjoys the benefits it provides my company.

This is rather unfortunate...

[qtp]

because the Sendmail sender verification proposal (mentioned here) relies only on already existing tech (Domain Keys, mx records, and smtp auth) thaty is already incorporated into the vast majority of MTAs, it does not really make much sense (from a users, or a non-microsoft, point of veiw) to create a seprate and more complicated solution (even if the license is rather innocuous).

I cannot help but think that continuing to allow senders that do not have a mx record for the sending machine to bypass smtp-auth for sending messages will fail to curb the spam problem, as it fails to tie the sent mails to an actual domain, and it allows (encourages) ISPs to restrict mailing through their email services only. With smtp-auth, it is still possible to send using an smtp server connected anywhere on the net, which allows accountability, but also makes it more possible to identify those providers who are allowing their users to send spam.

Pure FUD

[leerpm]

No, it is not insane. It is called cross-licensing. They are saying if you want to use this technology, then you agree that you are not going to come back and sue Microsoft (or any other licensee too!) for patent violations relating to this implementation. This is a good thing!! They are protecting themselves.

So by building support for "Caller ID for Email" into your software, you suddenly give Microsoft an unlimited license to use and sell it. And, in fact, not only Microsoft, but everyone else who writes software that supports "Caller ID for Email."

Absolutely not. There is something called copyright law. Microsoft or any other company cannot just go and resell your software on their own terms. The license just means you cannot sue them for patent violations when they choose to build software that implements technology similar to yours in this area (provided you had obtained additional patents relating to this 'Caller-ID').

Re:Pure FUD

[IGnatius T Foobar]

It is called cross-licensing. They are saying if you want to use this technology, then you agree that you are not going to come back and sue Microsoft

That's all well and good, and Microsoft will even "generously" give you a free license. But it's poison anyway. Here's why.

Their "free" license is only "free as in beer." It specifically prohibits you from sublicensing it to someone else. This means you can't use Microsoft's "freely licensed" technology in any project whose license meets the Open Source Definition.

Convenient for them, isn't it? They get to say "look, we're open, we're free, competitiors can use our technology at no charge" while they've quietly locked out the only real competition they have these days.

No thanks. I'll take SPF over "MS Caller ID" any day.

Re:Pure FUD

[swillden]

There is something called copyright law. Microsoft or any other company cannot just go and resell your software on their own terms.

Unless you grant them a license.

Which appears to be precisely what their license requires you to do. It's not clear to me precisely what you're licensing to them, maybe it's just any patents you hold on the techniques used, but it doesn't say that. What it says is that you grant them an unlimited license to "make, use, sell, offer to sell, import, and otherwise distribute Licensed Implementations", which certainly sounds like you're giving them permission to do what they like with your software.

I may be misreading this, but that's what the plain language seems to say. I'd want to get a legal opinion before I'd interpret it any other way.

New MS commercial

[pixelbend]

Can you spam me now? Good.

Re:Danger! Read the fine print!

[DHam]

Actually, it doesn't say that. The important phrase is "Necessary Claims" and the word "reciprocal" gives a good hint too. This is just a defensive patent licence. It says that Microsoft won't sue you for breach of patent for implimenting the standard or dealing in implimentations and you promise the same to Microsoft and everyone else.

It is NOT a copyright licence to Microsoft to use and sell YOUR implimentation. It only affects you if you hold patents which Microsoft or someone else infringes by implementing this standard. It effectively sets implimentations of this standard in a "patent free zone".

Do you Microsoft

[tobybuk]

I say ignore them.

Microsoft has never been interested in helping the community but rather wants only to further its own dominance of the market. When did they start being philanthropic?

What's to say in a few years time when everyone is relying on this that they don't pull some stunt and start charging people? Do you know enough about the law to say they couldn't?

Anyway their record on enhancing email is not good. I knew the first time I saw the ability to embed HTML and * SCRIPTS * into email that the virus writers would have a field day. I mean, what complete arseholes to allow code to be executed when someone just *reads* and email. It beggars belief!

If they are serious they could assign their patents over to the FSF and then we'll consider it. I bet they won't.

What is a PGP signature?

[stefaanh]

Shouldn't widespread adoption of PGP be the best solution? For me any implementation of PGP sig IS a Caller ID, only it is not XML, but it could easily be wrapped.

IMHO MS is reinventing a wheel, or trying to own it.

So, if everybody should become aware of the sense of a PGP sig, maybe with a service like "pgp://pgpserver.domain.tld" the problem is on its way to its solution... It shouldn't be part of SMTP sendmail or ... but is should be easy to hook it up anything.

Maybe the idea that mail could potentially be completely private (read:encrypted) is not that appealing to everyone.

So, tell them you read it here first. (Or point me to a similar idea.)

Re:What is a PGP signature?

[Greyfox]

That's what I'd do, but it still requires you to get and process an E-mail before you decide if it's rejected or not. Maybe combined with some other solution to reduce the network load it'd provide you a 100% effective filter.

I haven't seen a mail filter that will bounce E-Mails based on whether or not they're encrypted to your obnoxiously large PGP key that takes 30 seconds to encrypt to on a 2GhZ pentium or signed by someone on your whitelist. I suppose one could be written...

Re:MSXML

[pandrijeczko]

To be perfectly honest, if MS used their own proprietary XML extensions, I don't see how it would work anyway.

It's a fact of life that MS Exchange lives in corporate environments but ISPs and everyone use sendmail (or a sendmail derivative) for mail routing over the Internet.

It's actually in MS's interests to work with sendmail on an open protocol to do spam filtering properly (whatever that protocol is ultimately).

Remember that TCP/IP is an open standard and MS supports TCP/IP open protocols like FTP, HTTP, POP3, SMTP, etc. already in their products so this is no different.

We can stop Zombies too...

[aug24]

Take a look at the spf faq, section starting "What about the cracked, open-proxy DSL machines that are spam sources today?"

The skinny is: while spf on its own can't do prevent zombies from sending mail, if the upstream host routes port 25 through its own servers it can control this.

For example, my upstream hosts, Nildram, block all port 25 traffic outbound and inbound unless and until they have checked your (static) ip for open-relay-ness and then put you on a whitelist.

If all ISPs were like that, and spf were to become widely adopted, spam would be toast.

J.

Useless only for large documents

[wiredog]

Say, greater than 1 megabyte. I've been working with XML for a few years now and even DOM can handle simple messages in fractions of a second. How complex can this be? A tag defining a 'to' e-mail address, another for the 'from', a third for the relays. One for the signing authority. Tags for the subject, body, and attachments. No more than 10 tags, probably.

What XML REALLY is....

[jmlyle]

It's is not a data format.

It's not a framework.

XML is a badly-formed roman numeral.

It should probably be written "MXL".

But even that might be a problem. You might need to use the Unicode Standard symbols: 2169,216F,216C

MS is trying to pull a fast one

[Anonymous Coward]

Isn't this likely Microsofts attempt to get everyone using passport of something similar?
Once they authenticate everyone using their anti-spam system, they'll be able to authenticate for financial transactions, etc...

What about 'localhost' servers with dynamic IPs?

[davids-world.com]

I use a locally running postfix SMTP server on my laptop to send pretty much all of my email. Microsoft's proposal doesn't address this: of course, my laptop gets various IPs. I cannot use the SMTP server provided by my organization, as they firewalled it... With the MS proposal, I will have to go for VPN or talk to my sysadmins about smtp-auth -- and lose my independence...

Has caller ID worked on phones?

[PhiltheeG]

Like caller ID worked for the phone system. About 90 percent of my calls were either "Unknown" or "Private Line", and some action was still requried on my part to respond to the ringing phone.

I don't have facts readily available to back this up but I'll assume somebody made money off caller-ID, as will Microsoft will attempt to do with their new "standards".

Re:Danger! Read the fine print!

[LordKaT]

You work for SCO, don't you? ;)

--LordKaT

Summary

[dskoll]

Basically, it's a very poor re-implementation of SPF, with all of SPF's disadvantages and none of its advantages.

Under the MSFT scheme, the TXT records are verbose, likely requiring several records where SPF will probably fit in one. They have a hare-brained scheme to parse Received: headers to get around certain problems. Their scheme is absurdly complex.

And neither SPF nor MSFT's scheme do anything about spam coming from <>, cracked Windoze machines, or "valid" throwaway accounts. They also make forwarding more difficult than it should be.

Here's an idea,,,

[PHanT0]

Let me see the address of the people who send me e-mail... On hotmail, there is no way or option to see the e-mail address of the sender without opening the e-mail and we all know those nasty verify address e-mails by asking for a picture...

This is kinda unrelated yet not. But it's MS and SPAM in the same topic area, so I wanted to vent.

Re:Try OpenOffice.org

[NickFitz]

I downloaded the latest version of OO the other day, but haven't got round to dealing with the installation issues yet. Something to pass the time this afternoon :-)

(For any other Mac users with the same problem, TextEdit, as of Panther (10.3), can open Word docs.)

MS 1, SPF 0

[TA]

Wow. I looked at MS' proposal as well as SPF's, and darn if MS didn't do much better.
First: SPF's webpage is mostly slogans about how it makes the world better, but you have to dig around a lot to find out how their scheme works. Mostly you'll just find more of the same self-hugging and no real technical info.
Secondly: MS' scheme seems simple enough, just one addition to DNS (list those mailservers allowed to send mails from your domain), and a very nice, standard-compliant way of handling the mobile-user problem:
If you're away from home and you're sending from your name12@somefreemail.com account, and you want your From: line to be your standard Me.Myself@my-own-domain.cx, whatever actual account you're sending through, then just make sure that your Sender: is name12@somefreemail.com and you're set. This is a nice alternative if you can't list your freemail ISP's mailserver in your DNS (maybe you don't know its IP address, or it's changing all the time).
Maybe SPF's scheme is similar, but they sure didn't mention any Sender: header there. Seemed to be some home-cooked up non-standard header, and a lot of talking about forwarding not working etc.
The only thing I didn't like with MS' scheme is the XML thing, why would you want to put XML in your DNS records? Nothing else in DNS is XML. Oh well.

Re:MS 1, SPF 0

[Matts]

This is because you don't understand SMTP.

The Sender entry in the headers is often added by MTAs as the value in the SMTP envelope's MAIL field. This is the same value that SPF validates against.

Just because you don't understand SMTP and SPF is written in RFC language does not mean that Caller-ID is better. The XML in DNS TXT records is a big deal. The fact that with Caller-ID you have to validate after DATA is a big deal. But you won't understand these issues if you don't understand SMTP.

Poor Name

[BeBoxer]

Given the effectiveness of caller-id when it comes to the spammers of the phone world, I don't think it's the best model. Basically, caller-id allows anybody who has a PBX connected with digital trunks to the network to forge whatever caller-id information they want. Most telemarketers left it blank. Lots of legit companies send the id information for their main switchboard number, no matter what actual phone line the call is travelling down.

It's not the metalanguage that's important

[hexene]

I think "do we want XML" vs. "do we want a series of header fields" is asking the wrong question. It's the schema that's wrapped up in the XML or fields that's important.

XML is great for expressing tree-like data structures, where as the "field-name: field-body" approach is probably better for expressing linear data. If you look at a schema it is usually obvious if XML is being used just for the sake of it, and parsing SPF as it stands is trivial.

Companies with an "embrace, extend and extinguish" mentality towards standards can leverage XML by using it without any formal machine-processable schema (DTD, XSD or RNG), whilst all the while insisting it is "standard" because it uses XML. Look no further than WordML for an example of Microsoft doing this.

XML inside a 2k TXT Record???

[jbrayton]

Previous comments have been for or against XML being used to deliver this information. I don't have a strong opinion either way on that; it seems reasonable enough. What does seem silly is that this information is being stuffed into a TXT record, and limited to 2k. A goal of using XML should be to easily add information and to make the information hierarchical. But that goal will likely never be realized in a 2k string. The XML tags will eat away at the number of allowed characters pretty quickly. And the zone file examples in the document are pretty ugly.

SPF is better in that it keeps the information simpler. If XML is should be used, perhaps the TXT record should simply include an HTTP URL to the XML file. Alternatively, a simple URL standard could be used, such that one could reliably get Caller ID information regarding mydomain.com from http://mydomain.com/callerid or http://callerid.mydomain.com/.

Incompatible with GPL

[Peer]

If you distribute, license or sell a Licensed Implementation, this license is conditioned upon you requiring that the following notice be prominently displayed in all copies and derivative works of your source code and in copies of the documentation and licenses associated with your Licensed Implementation:

"This product may incorporate intellectual property owned by Microsoft Corporation. If you would like a license from Microsoft, you need to contact Microsoft directly."

By including the above notice in a Licensed Implementation, you will be deemed to have accepted the terms and conditions of this license. You are not licensed to distribute a Licensed Implementation under license terms and conditions that prohibit the terms and conditions of this license.

I guess this means no GPL apps, but I will now head to Groklaw and refresh the page until some legal info comes up ;-)

Re:How about text?

[dangermouse]

I wish you would learn something about existing mail standards before you say something so stupid. Email is primarily a simple text format, my HTML/word document/virus packed mailbox not withstanding. I am not surprised M$ would want to further polute the standards but why would you?

I wish you would learn something about existing mail standards-- like their colossal drawbacks. SMTP is entirely "a simple text format", and that's one of its biggest problems. We have all kinds of lame hacks for mailing binaries around and handling attachments. Nearly everyone who writes a mail client writes a mail parser and a composer. Not just a formatter, or presentation-level stuff-- basic goddamn parsing and composition.

You don't seriously believe that any format that is newline-dot-newline-delimited is a good one, do you? SMTP is a relic, all the way down to the message format. I hope to god someone eventually succeeds in dislodging it.

Microsoft *IS* critical mass

[JohnQPublic]

I wish this wasn't true, but if Microsoft implements some sort of spam-blocking in Exchange, that's all the critical mass you'll need. Especially if they turn it on by default.

And that is why

[mdfst13]

And that is why Microsoft is using it I'm sure. They have a bunch of nice GUI tools that parse XML, so anything they do now has to be XML.

It's the same as the way they do email. If I switch to source edit view, my simple text message (e.g. Got It.) balloons into ten lines of generated HTML gobbledygook. Yes, I really need to specify the font for *each* line...even the ones that are blank.

I really hope that the standard is not set by MS. Something very simple (this is who can transmit for this domain) could turn into something ugly. I can write SPF declarations by hand. Chances are that their XML declarations will be twenty times as long and will need tools to create them. Yes, the XML parsing tools are ubiquitous, but a simple format doesn't require a parsing interface to feed you info. I see no reason not to make a human readable interface.

Trademark squatting?

[Aidtopia]

A quick US PTO search reveals the "Caller ID for E-Mail" is a trademark held by an individual in Houston, Texas. He filed in March 2003 and claims to have used it in trade since December 2002.

There are several other similar trademarks, like "Web Caller ID" and "SBC Caller ID Internet."

I wonder if the MS lawyers cleared that term or not.

I am more concerned...

[mdfst13]

I am more concerned with the generation overhead. I can write an SPF specification by hand (plus they offer a nifty web tool to do it for you). It is human readable. An XML format can easily balloon into something that is not simply readable.

Email and DNS are both currently simple text formats. If they want to offer a new format for email and/or DNS that is XML based, that's fine (although I'm not really interested in adopting it). They can try to push the whole thing through and people can adopt it or not as they choose.

However, if they want to extend the existing formats with spam protection, it should still be a simple text format. SPF does this. It uses a standard +/- system to include/exclude certain entities from sending email. It works through DNS. No worries about commas, tabs, ends of lines, etc. DNS parsers already exist. This just adds an extra element to an existing standard.

Funny licence.

[rew]

from my understanding of the licence: If I want to implement a compliant implementation, I can go right ahead. (as long as I promise not to bother MS about patents that I might own on this technology).

If I then sell or distribute the software I wrote: Fine.

You however get to pay MicroSoft to use my software.

Oh, and they've included a GPL incompatible advertizing clause.

Email standard proposal

[amightywind]

Colossal drawbacks to text? LOL! It is a feature. You could say the same for most internet services. There are no standard client API's for FTP or Telnet or most other services either. Has that stopped their widespread adoption? Has it made them any less useful? No.

I am not concerned at all of people like you who make the internet groan under the weight of 20MB excel files wrapped in proprietary XML formats. MIME has done enough damage. Maybe the Standard should be a Microsoft (C, TM) paperclip icon that does a dance while he speaks your message in one of a hundred supported languages.

Re:Email standard proposal

[dangermouse]

For god's sake, go take a networking class.

MIME isn't damage, MIME is a hack to fix the crippled SMTP message format. Maybe you are only interested in sending ASCII text messages-- and that's very hardcore of you and all-- but the rest of the world is interested in sending pictures, documents, text in languages other than English (well over a hundred, and you're fucking well right the "Standard" should support them), etc., and your underdeveloped message format just can't properly deal on its own. Maybe you should read up on the subject.

Text itself isn't a drawback-- XML is generally represented as text-- but a message format that is defined only for transmitting text just doesn't cut it now that we're out of Green Terminal Land and into the World Where People Use Computers to Do Stuff.

And you're missing the point of my remark about XML libraries. The problem is not that parsing email is hard, but that there's no standard for an internal representation of an email message, and if there was it would probably be completely non-interoperable with the rest of the world. XML has the DOM and SAX, among others. This means a whole world of functionality, in the form of libraries and technologies that understand XML via DOM or SAX, is available to the program author. You can transform the message into another format using XSLT, access and modify the message content and headers with XPointer, find references to and merge in external resources with XInclude, extend the message format using namespaces (thereby allowing anyone who doesn't care about your extension to safely ignore it), transform the message (with XSLT) into XHTML and provide rich formatting with CSS (both of which can be found in reusable libraries), and so on and so forth.

You use XML, you get all of the above essentially for free. You go with some application-specific grammar, and you can either limit your email to plaintext or you can reinvent all of those wheels. But I know how much you reet haxorz hate usability and interoperability... maybe we can hook you all up with some nice teletypes.

Dogfood

[pjrc]

I'll believe Microsoft is serious about their Caller-ID when they actually implement it for their own domain name.

paul@preston ~ > host -t txt microsoft.com
paul@preston ~ > host -t txt hotmail.com

No responses! Compare to SPF:

paul@preston ~ > host -t txt aol.com
aol.com text "v=spf1 ip4:152.163.225.0/24 ip4:205.188.139.0/24 ip4:205.188.144.0/24 ip4:205.188.156.0/24 ip4:205.188.157.0/24 ip4:205.188.159.0/24 ip4:64.12.136.0/24
ip4:64.12.137.0/24 ip4:64.12.138.0/24 ptr:mx.aol.com ?all"
paul@preston ~ > host -t txt pobox.com
pobox.com text "v=spf1 mx mx:fallback-relay.pobox.com a:smtp.pobox.com a:emerald.pobox.com ?all"
paul@preston ~ > host -t txt livejournal.com
livejournal.com text "v=spf1 a mx ip4:66.150.15.140 ?all"

Here is the real reason Microsoft had to publish their Caller-ID spec now!

Before replying with "those 7500 domains are tiny", AOL is publishing a SPF record NOW. Microsoft is not publishing their own Caller-ID record yet.

Re:Dogfood

[belphegore]

Well, hotmail *has* published records. Just that Caller-ID is more complicated and hard to query than SPF. Compare the SPF examples you gave above to the ungodly:

[craig@belphegore craig]$ IDN_DISABLE=1 host -t txt _ep.hotmail.com
_ep.hotmail.com text "<ep xmlns='http://ms.net/1' testing='true'><out><m><indirect>list1._ep.hotmail .com</indirect><indirect>list2._ep.hotmail.com</in direct><indirect>list3._ep.hotmail.com</indirect>< /m></out></ep>"
[craig@belphegore craig]$ IDN_DISABLE=1 host -t txt _ep.list1._ep.hotmail.com
_ep.list1._ep.hotmail.c om text "<ep xmlns='http://ms.net/1' testing='true'><out><m><r>209.240.192.0/19</r><r>6 5.52.0.0/14</r><r>131.107.0.0/16</r><r>157.54.0.0/ 15</r><r>157.56.0.0/14</r><r>157.60.0.0/16</r><r>1 67.220.0.0/16</r><r>204.79.135.0/24</r><r>204.79.1 88.0/24</r><r>204.79.252.0/" "24</r><r>207.46.0.0/16</r><r>199.2.137.0/24</r><r >199.103.90.0/23</r></m></out></ep>"
[craig@belph egore craig]$ IDN_DISABLE=1 host -t txt _ep.list2._ep.hotmail.com
_ep.list2._ep.hotmail.c om text "<ep xmlns='http://ms.net/1' testing='true'><out><m><r>204.182.144.0/24</r><r>2 04.255.244.0/23</r><r>206.138.168.0/21</r><r>64.4. 0.0/18</r><r>65.54.128.0/17</r><r>207.68.128.0/18< /r><r>207.68.192.0/20</r><r>207.82.250.0/23</r><r> 207.82.252.0/23</r><r>209.1" ".112.0/23</r><r>209.185.128.0/23</r><r>209.185.13 0.0/23</r><r>209.185.240.0/22</r></m></out></ep>"
[craig@belphegore craig]$ IDN_DISABLE=1 host -t txt _ep.list3._ep.hotmail.com
_ep.list3._ep.hotmail.c om text "<ep xmlns='http://ms.net/1' testing='true'><out><m><r>216.32.180.0/22</r><r>21 6.32.240.0/22</r><r>216.33.148.0/22</r><r>216.33.1 51.0/24</r><r>216.33.236.0/22</r><r>216.33.240.0/2 2</r><r>216.200.206.0/24</r><r>204.95.96.0/20</r>< r>65.59.232.0/23</r><r>65.5" "9.234.0/24</r><r>209.1.15.0/24</r><r>64.41.193.0/ 24</r><r>216.34.51.0/24</r></m></out></ep>"

It' s not *just* that it's XML instead of more concise readable text, though that certainly is fucking idiotic.

Re:uh, yeah

[wfberg]

You said it! I'm sure we'll all regret using a standard format for hierarchically arranged tuples of name-value pairs. I only have to use this type of data in maybe 99% of my projects.

Nothing wrong with agreeing. Agreeing on a standard that's cruddy will bite you in the ass. There are many, many standards, and most of them are cruddy.

And "name-value pairs"? How do attributes figure into that? Well.. Cruddily, that's how!
Perhaps you're thinking of RDF (which has issues of it's own.. A lot..).

And the output files sure are difficult to understand if you've never seen any markup language before and don't have a file viewer that understands ASCII text.

XML allows for a lot more than ASCII.. Which is the reason a fully compliant XML parser is enormously bloated.

Instead why doesn't everyone just make up their own format that is uniquely tailored for the individual application? You can leave off the attribute names since the recipient of the data should just know what they are anyway. And you can use a binary encoding to really add efficiency to the process. And developers love the challenge of trying to figure out new data formats on top of interpreting the data itself.

Slippery slope? Or straw man? The latter. I never said no standard should be agreed upon. I would have preferred if it had not been something as complex and cruddy as XML. I even specifically gave S-expressions as an example that would be much simpler; you might note how that's not a binary format.

One day, ASN.1 was what XML is now (well, it still holds telecommunications and cryptography in its stranglehold). Do you propose we use ASN.1 because it's so well accepted and standardized and there are so many tools? Or do you recoil in shock at how bloated the featureset is, how convoluted the encoding, how shockingly incomprehensible the parsing process? XML is simpler than ASN.1, and XML is better than ASN.1 (except that ASN.1 has a cute way of compiling parsers from its syntax/schema language, which is a nice feature); but that does not mean XML is the best general purpose meta-syntactic language imaginable. It's not.

Re:How about text?

[gnu-generation-one]

"You don't seriously believe that any format that is newline-dot-newline-delimited is a good one, do you?"

Ask that again when you've got your x million messages-per-hour email gateway parsing an XML file each time...

Email is so simple you could probably parse it with a circuit board and a few NAND gates, and that's very good indeed when you want people to start using it.

Re:CRLF certainly not a Microsoft-ism

[spitzak]

CRLF is from CP/M, which was based mostly on the Dec operating systems such as RSTS/e and RSX-11M, and on the pdp8 systems (I forget what these were called).

The original reason for it is that mechanical teletypes did not have enough power in their motors to both advance the platen and return the carriage to the left border at the same time. So they split these into two steps and built "CR" and "LF" into the 5-bit baudot code. Believe me they did this only because they needed to, the most popular baud rate then was about 50 baud so the time wasted was considerable, and certainly there was no precedence for such a design in existing typewriters. ASCII teletypes were made with minimal changes and thus inherited the CR LF pair.

It seems until K&R thought it was acceptable to put some smarts into the terminal driver so it would translate a single character into a pair, everybody was forced to copy this standard so that text files would print correctly on a teletype. Why K&R chose LF instead of CR is confusing, but I think they wanted to keep CR for overprinting, while the old function of LF was pretty useless. They should have made input turn CR into LF, however.

In any case if anybody asks, MSDOS is based on 1940's technology. At least Unix is based on 1970's technology.