Is Dedicated Hosting for Critical DTDs Necessary?

pcause asks: "Recently there was a glitch, when someone at Netscape took down a page that had an important DTD (for RSS), used by many applications and services. This got me thinking that many or all of the important DTDs that software and commerce depend on are hosted at various commercial entities. Is this a sane way to build an XML based Internet infrastructure? Companies come and go all of the time; this means that the storage and availability of those DTDs is in constant jeopardy. It strikes me that we need an infrastructure akin to the root server structure to hold the key DTDs that are used throughout the industry. What organization would be the likely custodian of such data, and what would be the best way to insure such an infrastructure stays funded?"

Comments

I know!

[Colin Smith]

ICANN!

Mhahahahaha. Yeah. I know, I crack myself up.

Re:I know!

[commodoresloat]

ICANN!

Mhahahahaha. Yeah. I know, I crack myself up.

No you cann't!

Centralization

[ushering05401]

Nothing too insightful to write, but worth saying in today's volatile political climate. Centralization makes me nervous.

Regards.

Re:Centralization

[Bogtha]

There needs to be a way to refer to decentralized internet resources in a unique fashion. We need the equivalent of the URL for a file that is hosted simultaneously in many places.

This is known as a URN. URLs and URNs are together known as URIs.

Re:Centralization

[frisket]

The defects of the URN/URI/URL mechanism were well known at the time this was discussed in the working groups and SIGs while XML was gestating.

The correct solution would have been to fix the outstanding problems with FPIs and use a combination of local catalog and DNS-style resolution, but this was turned down. Perhaps it's time to wake it up.

In the 1990s I did try to devise a resolution server for FPIs, in the hope that someone like the (then) GCA (now IdeAlliance) -- who were the ISO 9070 Registration Authority and theoretically still are -- would pick up the idea.

I still have the large collection of SGML DTDs used at the time, now largely redundant, but replacing it with current XML is not the problem. This is something that should probably be discussed at the Markup conference in Montreal this summer.

w3c

[partenon]

w3c.org . There's no better place to keep the standards related to the web.

Re:w3c

[JordanL]

There's no better place to keep the standards related to the web.

Some say that wistfully, others begrudgingly.

DTD?

[mastershake_phd]

and DTD stands for? Distributed Technical Dependency?

Re:DTD?

[x_MeRLiN_x]

Document Type Definition

Re:DTD?

[Sporkinum]

It's the sound Carlos Mencia makes...

In case of death...

[Kjella]

...keep a copy, host it on your own site and reference that instead. There was no problem except that some were using that file to download the definitions. Or just expand the definition to include a checksum and a list of mirrors. Is this even a problem worth solving? I mean except for the slashdot post it seemed to me like this went by without anyone noticing.

Sane?

[DogDude]

Well, I wouldn't call it sane if anybody who is actively using XML and needs a DTD isn't hosting it right along with whatever web site they're using the XML for. Relying on somebody else to maintain a critical DTD that you use isn't sane. It's pretty dumb.

No

[Bogtha]

You shouldn't be using DTDs any more. Validation is better achieved with RelaxNG, and you shouldn't use them for entity references because then non-validating parsers won't be able to handle your code.

For those document types that already use DTDs, either you ship the DTDs with your application, or you cache them the first time you parse a document of that type.

The Netscape DTD issue was caused not by the DTD being unavailable, but by some client applications not being sufficiently robust. You shouldn't be looking at the hosting to solve the problem.

Re:No

[Anonymous Coward]

The Netscape DTD issue was caused not by the DTD being unavailable, but by some client applications not being sufficiently robust.

Not sufficiently robust is an understatement. ****ing stupid is what I would call it. If every browser had to hit the W3C site for the HTML DTDs every time they loaded a web page, the web would collapse.

Don't use them

[Anonymous Coward]

If the absence of these files will break your app or service, then you need to make your app or service more robust.

Sure, DTD files are necessary for development. If your app requires that they be used to validate something in real time each time it is comes in from a client or whatever, then use an internal copy of the version of the DTD file that you support. If the host makes a change to it (or drops it, or lets it get hacked), your app won't break, and you can decide when you will implement and support that change.

I really don't see what is gained by making the real time operation of your application dependent on the availability and pristinity of remotely and independently hosted files. It just makes you fragile, and you can get all the benefits you need from just checking the files during your maintenance and development cycles.

Re:Don't use them

[Skreems]

Exactly. The only point of having a URL associated with a DTD is to assure a unique identifier for each one. It wasn't worth starting a group specifically to regulate DTD identifiers, so they hooked it to a system that's already regulated. Yeah, it's nice to have the DTD live at that location, so if you get a file with a reference to an unfamiliar DTD you can pull it down on the spot, but it shouldn't be required.

Perhaps something like "pool.ntp.org"?

[Zocalo]

NTP.org" maintains a pool of public NTP servers that are accessible via the hostname "pool.ntp.org", so perhaps something similar would work for a global DTD repository. An industry organization with a vested interest, the W3C seems like the most logical, could maintain the DNS zone and organizations could volunteer some server space and bandwidth to host a mirror of the collected pool of DTDs. Volunteering organizations might come and go, but when that happens it's just a matter of updating the DNS zone to reflect the change and everyone using DTDs just needs to know a single generic hostname will always provide a copy of the required DTD.

Just a thought...

using non-local cached copy considered harmful

[tota]

Most tools provide a way to refer to a DTD on a public URL, yet use the local copy instead. (ie: taglib-location directive in java)

Doing anything else strikes me as fundamentally dangerous and insecure: it makes a remote dns vulnerability into an easy application DoS (or worse).

Call me crazy...

[Nimey]

but just have your DTD as a W3C standard, distribute copies with your software, and don't bother a remote server until a new version of the DTD is released. Then distribute it with a new version of your software.

Seriously, what the fuck were they thinking relying on a server to be always available?

Re:Call me crazy...

[Nimey]

It's not like someone would update it, and then give it the old version number.

Your trust in the world is cute. :-)

URI vs URL

[Sparr0]

A key mistake in your assumptions was brought up when the Netscape fiasco was news, and I will bring it up again...

"http://my.netscape.com/publish/formats/rss-0.91.d td" is a URI. It uniquely identifies a file. It *HAPPENS* to also be the URL for that same file, for now, but that is just a fortunate intentional coincidence. Your software should not rely on or require the file to be located at that URL. /var/dtd/rss-0.91.dtd is a perfectly valid location for the file identified by the URI "[whatever]/rss-0.91.dtd". What we need is for XML-using-software authors to support and embrace local DTD caches, AND package DTDs along with their applications (with the possibility of updating them from the web if neccessary).

It is silly that millions of RSS readers fetch a non-changing file from the same web site every day. It is only very slightly less silly that they fetch it from the web at all.

Not again

[dedazo]

This has been covered before here and elsewhere... anyone who is using a DTD as a URL rather than a URI needs to be taken out and shot. I say bring them all down and let all the apps that rely on them die or be fixed.

Supply local DTDs with your app

[Dragonshed]

I recently (within the last year) deployed an application that end users use for downloading and viewing custom content, and are intended to install the app onto laptops, tablets, and other portable devices allowing them view said content both on and off-line.

When prototyping our "offline mode", we ran into this exact same problem because the Xml APIs we used wanted to validate xml against online dtds. We ammended the validator's resolver to use locally embedded or cached dtds for all our doctypes, problem solved.

In in my app it was an obvious problem to solve because offline usage was a big scenario, but I could imagine that being "out of scope" for a less-than-robust website.

I have a server in my basement we could use.

[fyoder]

Linux box with an uptime of 153 days. It does have to go down now and again so I can clean the dust and cat fur out of it, but that doesn't take too long.

DTDs, XML entities and the non-breaking space

[Darkforge]

Unfortunately, DTDs aren't just for validation... they're also the only good way to define "entities" (e.g. "&foo;") in XML. This comes up a lot when trying to put HTML in XML feeds, because HTML has a lot of entities that aren't in the XML spec. Specifically, you may notice that you can't type " " in ordinary XML.

It's trivial to define "&nbsp;" yourself in a DTD, (<!ENTITY nbsp "&#a0;">) and many of the standard DTDs out there do define it, but by the XML 1.0 standard it's got to be defined somewhere or else the XML won't parse.

short answer: no

[coaxial]

Validation is overrated. Especially, when it comes to RSS. There's so many competing "compatable" standards, that really aren't. feedparser.org has a great write up about the state of RSS. It's pathetic.

If you're reading a doc, don't bother validating it. You're probably going to have handle "invalid" XML anyway. When you're constructing XML, you should write it according to the DTD, but if you're relying on a remote site, then you're asking for trouble. Just cache the version locally, but seriously, you're tool shouldn't really need it. You're engineers do, but not the tool.

Finally, it's trivial to reconstruct a dtd from sample documents.

EXACTLY

[wowbagger]

Exactly right, but it is even worse than that:

A DTD spec SHOULD have both a PUBLIC identifier and a SYSTEM identifier. The system identifier is strongly recommended to be a URL so that a validating parser can fetch the DTD if the DTD is not found in the system catalog.

The system catalog is supposed to map from the PUBLIC identifier to a local file, so that the parser needn't go to the network.

If you are running a recent vintage Linux, look in /etc/xml/ - there are all the catalog maps for all the various DTDs in use.

So:

1. The application writers SHOULD have added the DTDs to the local system's catalog.
2. Failing that, the application SHOULD have cached the DTD locally the first time it was fetched, and never fetched it again.

XML catalog files let your app use local copies...

[KarmaRundi]

You can map public and system identifiers to local resources. Use them for dtds, schemas, stylesheets, etc. Here's the spec. Google for more information.

Re:Catalog files?

[EsbenMoseHansen]

Or better yet why can't you just copy the blasted thing to your own site if your going to use it?

Is there some technical reason I'm not aware of that means it has to stay somewhere central?

There shouldn't be, yet I would be greatly surprised if some application didn't match on the entire DTD string, hostname and all.

I am equally baffled at what applications need the DTD for anyway. Except for generic XML applications, what use is a DTD? Most applications only handles a fixed few XML document types anyway.

Finally, if they really need that DTD... any distro have most major DTDs available. No reason why they couldn't carry a few extra. Should be easy to just search for them locally.