Building a TCP/ IP Network Over Dark Fiber?

1101z asks: "Well I work for a public access station in a city where a second cable/phone/internet company has moved just started operating. Part of there deal with the city was to let us have (for free) dark fiber links between several location in the city and our studio, so that we would be able to cablecast live from those locations. As the computer guy I would like to be able to interconnect computer networks that already exist at several of those locations, when we are not using the fiber for cablecast. The question is what is the cheapest way to build a TCP/IP network over this dark fiber." I wonder if the fiber being used is related to this story, from a month ago?

Comments

You can't do it.

[MarkusQ]

Goal: TCP/IP over dark fiber.

So far as I can see, you can't do it.

As soon as you try, the fiber won't be dark anymore, invalidating one of the conditions. You can have dark fiber, or fiber with TCP/IP (or just pretty lights for that matter) but not both at the same time.

-- MarkusQ

Nothing to do with dark fiber

[hab136]

"Dark fiber" is a nice buzzword, but has nothing to do with your question:

"I don't know how to design a network, can someone do it for me?"

Even if someone was willing to do this for you, the answer's no, since you've given absolutely no details about sites, number of users, applications.. really, anything useful to go on other than that you want to use TCP/IP.

There is no 'network in a box'; everyone's requirements are different. If you would post some of your requirements, we might be able to give you some ideas.

Re:Nothing to do with dark fiber

[nathanh]

"I don't know how to design a network, can someone do it for me?"

This should not have been moderated down as flamebait. The question was so open-ended as to defy imagination. This "flamebait" was simply the truth and sometimes the truth hurts.

Just to explain how damn useless the question is, there is no information about existing equipment, servers, desktops or applications. No explanation of the requirements for inter-site traffic (queuing only? interactive sessions? thin clients?). Does he want to use the fibre for data and voice? Does he want a single LAN or routing between sites? How many people per site? Is this a distributed or centralised server model? What's his budget?

He hasn't even explained what sort of fibre it is! Single mode? Multi mode? Frequency division? Can he afford a fibre ring? What sort of redundancy does he need? How long are the fibre runs?

There are plenty of solutions here - ATM between sites with LANE, GigE into some 3550s, 10baseF into tranceivers - but there's no way you can give him an answer without more information.

This guy obviously doesn't even know the extent to which he's in over his head. He should hire somebody with experience to do this job for him.

Re:Nothing to do with dark fiber

[CableModemSniper]

Right about now the querant's head just exploded. "You mean I can't just buy a Linksys Fibre-Optic router?"

Joking! Joking!

Re:Nothing to do with dark fiber

[Twirlip of the Mists]

He hasn't even explained what sort of fibre it is! Single mode? Multi mode? Frequency division?

It doesn't matter. When you buy dark fiber from a telco, you're given an MMF connection for each end. The link behaves just like it's a nice short run of multi-mode fiber. What actually happens in between-- DWDM, repeaters, microwave links, whatever-- is irrelevant. In fact, if everything is working properly, you'll never even be able to tell that there's anything going on in the middle at all.

I guess a lot of people are confused by the term "dark fiber." It's hardly ever literally true. When you buy "dark fiber" from a telco, what you're getting is an analog optical link to do with what you will. You can run anything over it that you can run over ordinary MMF. Is it ever actually, literally, "dark?" Hardly ever.

Can he afford a fibre ring?

Read the submission again. The telco is providing these links in a hub-and-spoke topology for free. "Can he afford a fiber ring" is a completely irrelevant question.

What sort of redundancy does he need?

None. They're going to use the links for IP traffic only when they're not being used for video. It'll be an ad-hoc network.

How long are the fibre runs?

It. Does. Not. Matter.

There are plenty of solutions here - ATM between sites with LANE, GigE into some 3550s, 10baseF into tranceivers

Fortunately, the submitter gave you a hint. He said "cheapest." Would ATM with LANE be "cheapest?" Of course not. Would Cisco gear be "cheapest?" Of course not. You're not even trying to be helpful, are you?

This guy obviously doesn't even know the extent to which he's in over his head.

You're just trying to make it sound more complicated than it is. Dark fiber is, far and away, the simplest form of long-range communication known to man. Well, maybe smoke signals or cups-and-string are simpler. Shine a blinky light down one end, and it'll come out the other. The question before the group is what's the cheapest way to turn Ethernet into blinky light and back again.

Re:Nothing to do with dark fiber

[narziss]

I'm not sure where you get your information from, but it's responses like this that really bother me. Under FCC guidelines, when a *LEC purchases "Dark-Fiber" they are purchasing an unused pair or unlit fiber between points A-Z; the only equipment between said locations are patch panels and splice huts. And it is SM fiber, 99.9% of the time (When's the last time you saw a bundle of MMF in a manwhole or fiber hut?

The cheapest way to put up a TCP/IP network via dark fiber would probably to be a cheap, used Cisco GE capable ethernet switch with a 5486 (SM/Long Range) GBIC.

If you're going to give advice about telecom, know what you're talking about.

Thanks

Re:Nothing to do with dark fiber

[Twirlip of the Mists]

Call your local telco, dude. When you buy "dark fiber" from the telco, they do not just sell you a piece of glass. They sell you a service, and that service includes repeaters and whatever else is necessary to get the light from point A to point B. When an old customer of mine bought dark fiber from Seattle to Chicago for an HD project they were working on, do you think it was one contiguous piece of glass? Hell, no. And here's another little tip, friend: it's never, ever single-mode, either. Customer equipment (as opposed to telco equipment) is nearly always multimode.

I don't know where you get your information, but you need to look again.

Re:Nothing to do with dark fiber

[nathanh]

I don't know how you got moderated up to Informative: 5 because you're spouting nonsense.

The run-length always matters.

The dark fiber is probably single-mode, not MMF, but there's no information to tell us either way.

I can't even imagine the confusion in your mind to lead you to think microwave might be involved! This is dark fiber, not a data service!

ATM with LANE might be cheaper if he can also share voice costs over the fiber. You can find some amazingly cheap second-hand ATM switches these days. Thank-you Dot-Com-Bust!

If he already has Cisco gear then he probably has a GBIC hole ready and waiting for a tranceiver. But without knowing how much bandwidth he needs how could you tell whether he even needs the capacity of GigE?

The question never even mentioned Ethernet, so I don't know where you got the idea that he wanted to know how to convert "blinky light into Ethernet" and back again. He said "TCP/IP network". TCP/IP is not dependent on Ethernet.

I stand by my first post and my defence of the person who got moderated to "flamebait" for saying the truth. This isn't a straightforward operation. If the person who submitted the Ask Slashdot doesn't think things through - or hire somebody to do the thinking for him - then he's going to waste money on hardware before finding out it doesn't do what he wants. Then he will have to waste money again, and again, and again, until he gets something that works to his satisfaction. This kind of irresponsible spending might have been par for the course during the Dot-Com-Boom but it's getting a little hard to bear these days.

Re:Nothing to do with dark fiber

[freebase]

Ok.. Here's there deal. Working for a telco, I have some first hand knowledge here.

When a customer is willing to pay the HUGE fees associated with a "dark fiber" cross country, that's typically what they get - dark fiber.

When your telco is selling you "dark fiber" for local use, metro ethernet, whatever, as long as it stays in the same LATA, and is handled by the same carrier, it's actually just a DWDM wavelength on an already lit ring.

A lot of the time, but not always, the lambda (DWDM wavelength) will be delivered to the Customer Premise on MMF from a shelf in the basement, attic, electrical room, telco room, telco hut down the street, or somewhere near by. If the customer is large enough, they might even rate their own shelf.

Sometimes, though, customers request SMF, which can normally be handled as well. Usually, with SMF, though, a customer is buying a service such as an OC-48.

Now... what's the diff? With the "dark fiber" metro connection above, you've normally got $#@% for redundancy, unless you buy enough of those "dark fiber" pairs to implement it yourself. The telco probably won't give you much of an SLA on it. If they do... you're definitely getting a lambda.

On the other hand, though, the OC-48 service probably has a good (decent) SLA that can be negotiated to an acceptable level.

It's all about price, performance, reliability and control.

Re:hehe

[Twirlip of the Mists]

Are you trying to be funny, or are you just simple? Dark fiber is just what it sounds like: fiber-optic cable in the ground that doesn't have any light passing through it. It's not in use, in other words.

you guys all suck

[Twirlip of the Mists]

I know it's a Friday night (in some places) but man, the responses on this Ask Slashdot really suck so far.

Short answer: you can set up a TCP/IP network over a dark fiber link for as little as a few hundred bucks, if you can find equipment for a good price. Here's how.

I'm going to make a couple of assumptions here; correct me if I'm wrong. I'm going to assume, first of all, that each link you've got access to is actually a pair of links; that's the way dark fiber is almost always sold. Second, I'm going to assume that you've actually got a dark fiber link, as opposed to buying a lambda. (Buying a lambda means that the telco is letting you use one frequency of a dense wave division multiplexed [DWDM] link. Not the same as dark fiber in the literal sense, but the same in most practical senses.) Finally, I'm going to assume that the telco has provided you with the necessary repeaters on the line so that you can actually push light from one end to the other without any additional hardware. If your telco has sold you (or given you, whatever) "dark fiber," chances are that all three of these assumptions are true.

If all of those things are true, then you're in a really good position. You can run anything across these fiber links that you could run across a shorter length of optical cable: FDDI, Ethernet (any speed), Fibre Channel, FireWire, HIPPI, whatever you want.

You said "cheapest," and what's cheap depends on what's available. If you can get your hands on a couple of old Ethernet switches with 10BASE-F or 100BASE-F (which are simply 10 Mbit and 100 Mbit Ethernet over fiber optic cable instead of copper cable) you're in business. Just plug the dark fiber into a switch at each location and poof! A single TCP/IP network running across the fiber to both sites, at 10 or 100 Mbps depending on what you can find.

My last company had, among other things, some Bay Networks (now Nortel, I think) stackable Ethernet switches with 24 100BASE-T ports and two 100BASE-F ports. I think they sold for about $2,500 when new (in 1998 or so), but should now be available for a lot less used. If you can find some of those used you'll be in good shape. Asante also makes switches like these; I've never used them, so I won't vouch for them, but you can buy them.

Another option would be to bridge Ethernet to FDDI; switches that do this should be available for really cheap, if you can find them, because FDDI fell completely out of favor in the mid-1990's. FDDI runs at 100 Mbps, just like 100BASE-F, but it has to be bridged, and sometimes this can cause problems with packet splitting and MTU sizes, especially on Cabletron switch gear. Unless you're looking at an absolutely killer deal, avoid the FDDI option.

If you want to go with something more up-to-date, you can run Gigabit Ethernet over the fiber links. It'll cost more, but you'll get better bandwidth. A good idea might be to buy a couple of cheap 100BASE-T switches with 1000BASE-T gigabit uplink ports (about $150 each), then equip each switch with a 1000BASE-T to 1000BASE-SX media converter (as little as $200 each).

Any of those solutions-- 10BASE-F, 100BASE-F, 1000BASE-SX, bridged FDDI-- would require nothing more than a switch with the right media type at each end; you wouldn't have to mess with routers or anything, and you wouldn't have to do anything fancy with your IP network. In fact, you wouldn't be limited to running just IP. You could run anything that can be carried over Ethernet: AppleTalk, NetBIOS, whatever.

If you get the gear for a reasonable price, you can run any of those networks for really, really cheap. When the links aren't being used for video, plug 'em in to the switches and go to town. When you're ready, just unplug 'em and go back to video. The link will be down, but neither the switches nor the computers will care.

Re:you guys all suck

[toast0]

good information, one thing i think you may have missed is that there are apparently multiple (more than 2) access points. If you're going to run the fiber at gigabit w/ the converters, then if all the links share a location for one end (which would likely be in the main tv building, but who knows) and you would need a gigabit switch capable of handling all that fiber there. Of course, if its actually more of a ring layout, then each switch would need two gigabit fiber connectors, and you'ld really want more of a router than a switch, since otherwise it'd get ugly. It'd probably be better to have a router in each building anyhow, but it wouldn't be strictly needed, with creative network design.

Re:you guys all suck

[Twirlip of the Mists]

Nah. We're talking about doing this on the cheap. For the central office-- assuming all roads lead to Rome, if you know what I mean-- you might choose to have one switch with the same number of optical ports as you have fiber links, or you could just as easily have one switch with one optical port each for every fiber link. There's zero need to do routing here. If it turns out to be cheaper to buy 6 (or whatever) switches with 12 100BASE-T ports and 1 100BASE-F port each than to buy a switch with 6 100BASE-F ports-- which will likely be the case if this guy is buying used gear-- then you can just cascade or stack all the switches together. It will work fine, because the "three hop rule" only applies to dumb repeaters, not to switches.

I repeat: there's zero need to do routing, even if it's not a hub-and-spoke network. If one building connects to the next which connects to the next, just set it up as a bus.

Remember, the most important word in the question was "cheapest."

Re:you guys all suck

[Twirlip of the Mists]

For someone claiming to be a guru...

Pardon me, sir or madam, but I believe you have me confused with somebody else.

Honestly, now, let's take a survey. Who here can keep NetBIOS/NetBEUI straight, huh? Who among us hasn't given up on the whole thing?

Re:you guys all suck

[Gothmolly]

I just did keep it straight. One is an API, one is a protocol. Duh.

Re:you guys all suck

[drinkypoo]

I repeat: there's zero need to do routing, even if it's not a hub-and-spoke network. If one building connects to the next which connects to the next, just set it up as a bus.

A bus is an inherently unreliable system, which is why using coax for ethernet sucked even when it was faster than your OS could push. Ah, those heady days of winsock. Yuck.

Seems to me that if you're forced into doing a bus the best thing you could do for yourself would be to do a ring. Sure you'd need to do routing, but you could always use cheap linux boxen with 100BASE-F and 100BASE-T in them.

While you are correct about what is cheapest, I think it would be best to mention what is cheapest without being inherently unreliable. Especially if this is pacific bell fiber, which they cut all the goddamn time. Even in downtown SF. Amateurs, after all these years...

The Cisco way to do it.

[Mordant]

http://www.cisco.com/en/US/products/hw/modules/ps8 72/ps4025/index.html

plus a Catalyst switch at each end:

* Cisco Catalyst 2948G

* Cisco Catalyst 2980G-A

* Cisco Catalyst 2950 Series

* Cisco Catalyst 3550 Series

* Cisco Catalyst 4000/4500 Series

* Cisco Catalyst 6500 Series

Re:The Cisco way to do it.

[Anonymous Coward]

The key word there was "cheapest," Sparky. Cisco gear is a lot of things, but it is never, ever cheapest. Ever.

Re:to get you started....

[GoRK]

Welcome to my friends list my drunk european friend!

Your sincere dedication to giving this guy such a completely incorrect answer is very admirable. It fills my heart with joy to know that there are still people out there who want to care.

~GoRK

NICs not switches

[MrResistor]

Everyone seems to be focusing on puting fibre switches at each location, which I think is an unnecessary expense. The way I read the question you already have existing network at each location, and you just want to hook them together.

It seems to me that you could just get a few PCI fibre NICs and use them to set up existing machines at each location as bridges. I don't remember how much they cost, but it would definately be cheaper than switches. You'd have to make sure you had the right plugs/jacks, obviously.

It seems to me that it would be a pretty simple thing to do.

Re:NICs not switches

[Gerry Gleason]

Machines with NICs are much more expensive and complex than network gear. The type of thing that is needed here is probably available used for not much money. So much network hardware was purchased in the dotcom boom, that there is probably still a backlog of really nice stuff on the resale market.

Get advice from a network consultant, because you need someone who knows what to buy and how to hook it up. With the right kind of hardware, you can probably share with the video link too (might be expensive, that's why you need some advice).

Re:NICs not switches

[Twirlip of the Mists]

Everyone seems to be focusing on puting fibre switches at each location, which I think is an unnecessary expense.

Yeah, but the thing is that each site is presumably going to need a switch anyway. It's much simpler to just link the Ethernet switches together to form a single network segment across all the links than to mess with routing and whatnot on dual-NIC PC's. Cheaper, too, since, like I said, they're going to need those switches anyway.

That said, dual-NIC PC's could work. You'd have to allocate a different subnet to each site, and set up each computer on the whole network with the correct routing tables, but it could be done. I just think switch-to-switch is simpler.

Re:NICs not switches

[Twirlip of the Mists]

Keep in mind that even HDTV only uses up to 19.2Mbps

That's only after the signal has been compressed for OTA transmission. That usually happens either at or right before the transmitter. Within a facility, you're pushing around full-bandwidth SMPTE 292M serial digital HD, at a bit rate of about 1.3 Gbps. Which is still no problem for dark fiber.

This is kind of a tangent, but the most impressive use of dark fiber I've ever heard of was a test that SGI did with the government. They leased a dark fiber link from one of the big nationwide telcos (for some reason I want to say Qwest, but I'm not sure that's right) for a couple of days and ran GSN over it. GSN runs at 6.4 Gbps. My friend was on the team, and he said they ran 790 MB/s (bytes, not bits) us OS bypass across the country for hours and hours. The network link was so fast that they could measure to an accuracy of about 2 microseconds the speed-of-light latency from Chantilly, VA, to Mt. View, CA. Pretty amazing stuff.

Re:NICs not switches

[drinkypoo]

Machines with NICs are much more expensive and complex than network gear.

I love to be the one to break this to you: network gear is just machines with NICs in it.

Oh sure, a 7500 series cisco router is much simplified from doing the same stuff on a PC, it has a bus designed for a routing architecture and individual expansion cards do more than their PC equivalents. But a 4000 series cisco router is basically a very slow computer with a minimal set of hardware and a mediocre bus which has a few cards plugged into it to do I/O.

Similarly, a mini-itx PC with a PCI tree, or a passive backplane PC (so as to get a huge load of PCI slots) is really no less reliable than that cisco gear. I have had really crappy no-name PCs which lasted for years, a couple of which I still have. Meanwhile, my only computer as slow as old Cisco routers (A Macintosh IIci; actually most cisco equipment from the same vague era was 68020-powered, not 68030) is still in great shape, though I wouldn't use it for a router unless I had a really slow link to handle. :)

Get advice from a network consultant

Did you fail to notice the 'Ask Slashdot' aspect of this? This place is full of network consultants. Hell, anyone who has ever answered someone's networking questions for money outside of a salaried position is a network consultant.

Re:NICs not switches

[MrResistor]

I would totally agree if he was building the whole thing from scratch, but he's not. Each site already has an existing network, presumably already set up with their own subnets, switches, etc. Throwing a fibre NIC into an existing PC at each site is still going to be the cheapest and least disruptive way of acheiving his stated goals.

I agree that switches would be easier, but that doesn't make it the best solution for this particular situation.

Re:NICs not switches

[Twirlip of the Mists]

I would totally agree if he was building the whole thing from scratch, but he's not. Each site already has an existing network, presumably already set up with their own subnets, switches, etc.

Is that what it sounded like to you? I didn't get that at all. He referred to himself as "the computer guy." It sounds to me like this is obviously a one-horse operation.

Re:NICs not switches

[Gerry Gleason]

I love to be the one to break this to you: network gear is just machines with NICs in it.

I hate to break it to you, but this is wrong. The simplest switches will be nothing but a backplane and special purpose hardware to connect each link up to the backplane. No processor necessary. More complex and flexible gear that can do a lot of complex routing an filtering will probably have a processor, but it only gets involved in configuration. Packets flow in and out without a processor every touching them.

PCs have lots of things that aren't even a little necessary for this, in particular disks that have a very high failure rate compared with chips and such. Further, the biggest problem is the OS that you have to boot and configure, and a purpose built device will just turn on and go. It is just much more likely to just work, whether you are talking about cheap simple NetGear stuff or more complex Cisco routers and switches.

Did you fail to notice the 'Ask Slashdot' aspect of this? This place is full of network consultants. Hell, anyone who has ever answered someone's networking questions for money outside of a salaried position is a network consultant.

So what, my point was that there are a lot hardware choices, and as others have pointed out, he didn't specify enough to know for sure. Rather than spend money on devices that don't quite work for the job, get a little help from someone who can say for sure what will and won't work. I've actually done networking work both for salary and as a consultant, but I don't consider myself a network consultant because I don't do it enough to be able to definitively say what will and won't work. Expirimenting can be expensive.

Re:NICs not switches

[drinkypoo]

I hate to break it to you, but this is wrong. The simplest switches will be nothing but a backplane and special purpose hardware to connect each link up to the backplane. No processor necessary. More complex and flexible gear that can do a lot of complex routing an filtering will probably have a processor, but it only gets involved in configuration. Packets flow in and out without a processor every touching them.

Even a non-managed switch is most likely to contain some type of fairly reasonable CPU core to manage the arp tables... what are they called typically, CAM tables? A PC tends to have more stuff built onto it, but recently it has tended more and more to be on a single chip; in the case of this new linux on a chip seen on the front page not long ago, it really is all on a single chip, all you need to do (I assume) is interface it. That's a pretty damned simple system. While a PC may have a north and south bridge (or it may have a one-chip chipset for something embedded, which nonetheless provides the usual PC-like I/O) this still does not significantly increase the likelihood of failure. Buying cheap hardware will still get you the usual repayment.

Any expandable managed switch really is the equivalent of a PC. They have a CPU, frequently something in the motorola 68k range, and they have a bus of some sort. Cards do a fair amount of processing themselves, but they still take instructions from some sort of central control. They have serial I/O, and some kind of storage; a lot of them will netboot. A large number of them have PC Card slots for memory, I don't remember what type the catalysts with supervisor 3s were using but I remember some of them coming with over 40 MB of PC Card flash memory of some kind.

Anywho I know that routers are supposed to be built to higher specs than consumer-targeted PCs but most of them aren't really, and some of the more reasonably priced motherboard manufacturers are some of the better ones as well, companies like abit, asus, shuttle, and tyan (though some of these have increased in price, and everyone has made a lemon or two.) Cisco used to make some really advanced hardware, and then it got cheaper to make the good stuff and PCs got really incredibly good... PCI helped dramatically in that regard. Now PCs are very reliable and have a very good bus which tends to be very reliable in all applications, to the point at which Sun has felt comfortable putting it in workstations. Of course, they're on their backs in more ways than one these days...

Re:NICs not switches

[Tony-A]

The simplest switches will be nothing but a backplane and special purpose hardware to connect each link up to the backplane.
Impossible.
Hub, maybe. A switch is capable of running all links full duplex, which means that the switch must have the processing speed and storage to store and forward packets that would collide if it were just a hub.

Re:NICs not switches

[Gerry Gleason]

The only time you would need storage is if a packet was received for an interface that is already busy or backed up. A simple minded switch might even kill this packet by signalling a collision on the sending interface. Any decent switch would have enough bandwidth on the backplane to support all the interfaces it has. Perhaps "switching fabric" is more descriptive of most switch architectures than backplane, but in principle it could be a backplane running much faster than the links.

Perhaps someone with actual knowledge of specific switch designs could comment, but bottom line, only very poor switch designs would have a processor interacting directly with packet flow (for the typical case).

buy some gigabit transceivers

[Peartree]

buy some gigabit transceivers, plug em into a gigabit switch. BAM! you've got a gigabit MAN at your disposal

fiber?

[TREETOP]

Ahh, the dark fiber is strong in this one..... he will be a good network warrior.

Ask Slashdot: building a RFC 1149 TCP/IP network

[Anonymous Coward]

As part of a deal with a new pet store, the city has been provided several free "dark" carrier pigeons.

Can someone please help me discover what additional hardware I need to do RFC 1149/CPIP?

I would also try "goo-gling" but I'm a grown man and it sounds like something more approbate for a baby to do.

No they DON'T suck

[FreeLinux]

The previous posts, though unforgiving and potentially offensive are right on the money! Your recommendation fails to take into account the two most important details of such a network. Details that the poster failed to provide. They are fibre type and distance.

Without knowing the fiber type and the length of the runs it is not possible to answer the question, and anyone who has a clue about it would know that type and distance are imperative.

Your recommendations would be great, assuming that the fiber runs are multimode fibre at under 500 meters. But, the post was talking about a MAN which suggests far greater distances than your proposed campus network. But, what are the distances? Are the runs one kilometer or are they 30 kilometers? What type of fibre is it? How may connections are there in the fibre? Has the fibre been tested(characterization) to determine dispertion levels due to fibre quality, distance and connections.

Going further, what is cheap? It's rather subjective, don't you think? Cheap to some people means a couple hundred dollars. It's highly unlikely that he could build the network for that. Cheap to some other non-profits that need such a network could be in the millions, as could this project. What's cheap to him? Budget information is imperative with an optical network. You have to have a power budget for the optics(not dollars) and you have to have a financial budget.

The poster didn't come close to providing the required information with the question and got what he deserved. Your recommendations, though well intended, are not applicable to the situation, in other words, just some wild ass guess(SWAG).

Re:No they DON'T suck

[Twirlip of the Mists]

Without knowing the fiber type and the length of the runs it is not possible to answer the question

You're new to this whole "dark fiber" thing, aren't you? When a telco sells you "dark fiber" they're either literally selling you unused MMF, with repeaters in place, or they're selling you an unused lambda. In either case, the interface to customer equipment is multi-mode fiber, and you can run anything over it that you would run over a shorter piece of fiber.

But, what are the distances? Are the runs one kilometer or are they 30 kilometers? What type of fibre is it? How may connections are there in the fibre? Has the fibre been tested(characterization) to determine dispertion levels due to fibre quality, distance and connections.

If you were pulling your own glass, those would all be relevant questions. Since these connections come from a telco, the telco takes care of all of those things for you.

Look, buying dark fiber is like buying a dry pair from the telephone company. You have a pair of wires on one and and a pair of wires on the other, and you can use them as if they were opposite ends of the same piece of continuous copper. Are they really? No. The signals on your dry pair pass through switches and muxes from here to there. But the telco guarantees that the dry pair will act like a single piece of wire.

Dark fiber is the same way. No matter what is actually between you-- DWDM mux and demux, repeaters, microwave links, whatever-- the telco guarantees that the dark fiber link will act like one long piece of MMF.

Going further, what is cheap? It's rather subjective, don't you think?

Sure, it would be subjective if he'd said "cheap." If he'd said, "What's a cheap way to built a TCP/IP network over dark fiber?" there would have been hundreds of good answers. But he didn't say "cheap." He said "cheapest." And my answer, to my knowledge, comes down as the cheapest possible way to run TCP/IP over dark fiber.

Cheap to some people means a couple hundred dollars. It's highly unlikely that he could build the network for that.

It's highly likely that he could build the network for that, if he could get ahold of cheap used 10BASE-F or 100BASE-F gear. With the current business environment, the market is positively saturated with this kind of gear for pennies on the dollar.

The poster didn't come close to providing the required information with the question and got what he deserved.

Yup. As of the time that I wrote my post, the submitter had received about a dozen smart-assed, sarcastic responses and two helpful ones. Par for the course for an Ask Slashdot.

Re:No they DON'T suck

[FreeLinux]

You're new to this whole "dark fiber" thing, aren't you? When a telco sells you "dark fiber" they're either literally selling you unused MMF, with repeaters in place, or they're selling you an unused lambda. In either case, the interface to customer equipment is multi-mode fiber, and you can run anything over it that you would run over a shorter piece of fiber.

Again, you assume way too much. First of all, the poster says that the city was to let us have (for free) dark fiber links between several location in the city and our studio. He did not say a telco was providing him with anything. Further he says dark fibre links. He does not say that they are providing bandwidth or lamdas, meaning that it cannot be assumed that they have repeaters or muxes or DWDM equipment on these links. It is entirely possible, if not likely that they are getting strands of glass and nothing more.

You also talk about multi-mode fibre. While it is true that telcos often provide a multimode fibre connection to their customers, this is just a short link back to the telco's multiplexer on a single mode sonet ring. This case could easily be just glass strands, like I said before. If that is the case then they are much more likely to be single mode because multi mode cannot run the same distances that single mode can. If the run is longer than a kilometer it will almost certainly be single mode and require totally different equipment than the multimode scenario you propose. Again we don't know that from the post, as I stated earlier the poster did not provide nearly enough information to answer the question. But, maybe you are correct, provided that your other assumtion was correct when you said You're new to this whole "dark fiber" thing, aren't you?

Re:No they DON'T suck

[Meleschi]

Okay.

This is getting silly...

When's the last time you ever received dark fiber from a telco?

If the fiber run is going to be over 50 miles or so, it will usually go through repeaters of some type.

HOWEVER if the runs are shorter than that, it's really not the telco's responsibility to put in repeaters for you. As soon as repeaters come into play, you end up with a limitation on that fiber, regardless of what equipment you have at the ends.... OC3, OC12, OC48, whatever the repeaters are rated at is now your limitation.

For the short spans (<50 miles)that don't require repeaters, you're free to go as high as you can with whatever equipment you can afford. There are short range, medium range, long range, and extended long range cards (that I have experience with) that will do more than get the job done. And you're free to choose the speed of card you wish to purchase, as long as the fiber that is run is the correct type for the card you purchase.

Short range cards if I remember correctly, are usually Multi-Mode fiber. Medium to Extended Long Range cards are almost aloways single mode.

Stop running around and saying that having dark fiber means it's lit up by the Telco. It's not. The telco's responsibility is to provide fiber connectivity however is most appropriate considering the distances involved.

Re:No they DON'T suck

[adamsc]

Cheap to some people means a couple hundred dollars

Check eBay - people have been getting 100Mb switches w/fiber uplinks for as little as $10 recently. One big switch for the main office and a couple little ones for the other end would be doable for $200.

Re:hehe

[Twirlip of the Mists]

I think your over-simplifying your defination of "dark fiber" since it sounds like some telcos are willing to market unused portions of DWDM as "dark fiber."

It's not really my definition of "dark fiber" that counts. It's the telco's. When you buy "dark fiber" from a telco, you get a multimode fiber connection on each end. Plug each end into your equipment and you're up and running. What actually goes on in the middle is the telco's business. "Dark fiber" is a guarantee that the link will act like a continuous piece of MMF, only without all the bother associated with long MMF runs. The telco might actually sell you a piece of unused glass with repeaters already on it, or they might sell you a lambda of a DWDM fiber. In any case, you can treat it like it's just one long connection.

Have you done this before?

[DrZaius]

The short answer is buy some layer three switches with gbics (Extreme makes some nice stuff that I would recommend, but you can always stick with cisco too).

The long answer is that if you haven't done this before you better get some consulting help. Chances are you are talking about a ring topology and are going to be linking sites with different networks.

Perhaps you have telco supplied networks for each office hooked up over DSL or T1. If that's the case, have fun getting routing working without having the telco people disconnect you. Good luck reconfiguring the telco routers for that matter.

Routing complex networks is tough. Do you already have VPN's interconnecting these sites? Are you going to be introducing redundant routes? How are you going to manage these routes? IOS can suck if you've never seen how to configure routing processes. Routing software is also complex. Ripv2 is about as simple as it gets and it doesn't offer much control over which route you take -- the only metric is hop count. OSPF has design guides as big as phone books.

If you have a bunch of nats at your different locations, do the networks overlap? Are you going to have to renumber your networks?

If you are just playing in your spare time, you won't be able to do this for under a few grand. Fibre connections are generally not cheap. If you're lucky you could put a few fibre nics into a couple of linux boxes, but I don't foresee those nics being under $400 each.

If this is to be a business network, do it right from the start or you'll make yourself look stupid. People expect the stuff they don't understand to just work. There will be very little tolerance if services are going up and down and your fibre links are to blame.

Crazy hack: PPP over Digital Video...

[PaulBu]

Assuming that you would want to have your video equipment plugged in all the time and want to hijack its bandwidth only when it's transmitting blank stream, you can hook up (entirely in software) a weird setup which would dump data packets over digital video link (as if they were streams of pixels) and decode them on the other end, assuming that your video streams pass thru some kind of video caption cards inside more or less general-purpose computer... :)

Yeah, it might be slow, but as a next step you might consider some steganography: why not use couple LSBs in each pixel for data and the rest for video?

(If anyone cares to moderate this, please mod it as "funny", not "informative" ;-) )

Paul B.

Re:Crazy hack: PPP over Digital Video...

[deicide]

I remember seeing a PCI card that allowed hooking up a regular VCR and using VHS tapes for backups. Several gigabytes per tape - mind you, this was when 1 gigabyte hard drives were common.

If you "watched" the tape it would appear as series of white/black squares on the screen.

Re:Crazy hack: PPP over Digital Video...

[Da VinMan]

Ok, I'll bite. I actually had something similar for my Amiga 500. It was the only way I was able to back up my "massive" 80 MB hard drive. IIRC, it worked from the parallel port.

You know, using a VCR for this sort of thing wasn't too bad of an idea for a home user. Granted, there were limitations, but the cost factor was great for me at the time.

Ah, the good ole days. ;+)

What would you use for Video and Digital

[Gerry Gleason]

Yes, this is really the reason I suggested consultants. I know that all of this is possible, but not what gear would do it best. Would you use something line ATM based connections with channel adapters to hook up the raw video stream (more like 300MB according to the other reply, which seems about right)? If I understand the purpose of ATM protocols, it is designed for these situations (disperate services sharing a single high speed channel), but I would call on a good network consultant who has experience with similar configurations.

The TOS stuff isn't relavent to dark fiber, because in effect you aren't going outside of your private network, so you control everything. If you also run and own your own PBXs in both locations, you should be able to connect them over the fiber as well. This probably would not be voice over IP, but it could be.

WRT FDDI, in my experience, FDDI was already well on the way out for the last of the dotcom period. I'm not surprised that a lot of it didn't work well, and if I was sitting on a cache of FDDI hardware, I'd probably junk it and take the right off. I'd be interested in a more informed report, on what are/were the more popular technologies. I thought ATM was big with the telcos and big pipe providers. The telcos also have a set of standards are protocols that are more "circuit switched" and related to ISDN where 64K channels that support a single voice channel are agregated to T1 (24 channels at 1.5M in the US, 32 channels elsewhere), and then into higher bandwidth channels.

Yes, it has been three years since the boom, so a lot of the used equipment dumped on the market is probably in use someplace else (hopefully, if you are CISCO). I supose it is likely that whatever is still available (if anything) is likely to be stuff you don't want (e.g. broken FDDI stuff), but again, this is where the consultants come in. Besides, the bust isn't finished, so some stuff will become available. DirectTV DSL is being taken off line (a pain for me), and I doubt they have an use for all of the equipment that will be taken out of service.

Fiber (or ANY) Specifications

[eric2hill]

I worked with some underground fiber (owned by the city water department) a couple of years ago. It was running a low-speed serial link between a PLC and a computer. My task was to run ethernet over this link (at 10MB or 100MB speeds). The problem was the fiber was a little older and was a larger size. We BARELY got 10MB to run over it. 100MB was impossible without VERY expensive equipment.

If you'd post some specs on fiber size, termination, and distance, we could give you better answers than "just a couple of routers with fiber ports".

Re:hehe

[zaqattack911]

Your obviously the simple one.

You don't know a fucking joke when it's right in front of you?

Re:hehe

[Twirlip of the Mists]

Your obviously the simple one. You don't know a fucking joke when it's right in front of you?

Maybe I would have been better able to recognize it if you'd made it funny at all.

(Witty comeback plan B: "I may be simple, but at least I know the difference between 'your' and 'you're!'")

Dark Fiber Networking

[gybrwe8]

Hmmm...seems everyone wants to either be a smart*** or overanalyze this. First off: It appears that he is working with the local cable company, so, actually, he has already defined what sort of issues and designs he will have with this. Dark fiber from a cable company means a point to point link. If he has multiple offices, this will generally mean that there will be a hub and spoke design on the fiber from a central location to the edge sites. In addition, the local cable co will most certainly provide the distances and loss budget for the fiber, making selection of equipment an act of trivia. The only thing not defined was the type of network he really wants, but in this case, it is also probably completely irrelevant. If you can't put all your services over TCP/IP, then you shouldn't be maintaining a network. First off, gear selection. I work for a Cisco-centric company, so most of my experience is with Cisco, but I have worked with Foundry and Nortel and a few others. If you want reliable, Cisco is certainly acceptable, and you can always put a contract on it to meet your needs. Second, protocol selection. Well, he wants TCP/IP. This will certainly be easy enough. Engineer the backend properly (IP addressing 101, and get a router for the links, your done. If you need to convert Analog Video to H.323, there are plenty of vendors who can do this. If you need COS, again, there are plenty of solutions. Layer 2 Selection. My recommendation would be to go with GigE. This is certainly cheaper than going with outdated FX technology if you buy new, and more reliable. When you have the distances and loss budget of the fiber links, you can select the appropriate GBIC's for the link, and if you get light at both ends, you have a network. In Cisco terms, I'd go with a 3550-12G at the core if you have more than 2 locations. This will be a reasonable fiber concentrator ($9,999 list) and also does QOS, policing, and routing (both packet and protocols, such as EIGRP). At the edge, depending on how much intelligence you need, you can put in Cisco 3550-24's, either EMI or SMI flavor. The EMI boxes are routers (same code as the 3550-12G. The SMI boxes have port routing in the newest code, but won't run routing protocols. These boxes will allow you to converge your network (data, VOIP, and video) and really don't have a horrible price tag. The SMI lists at $2,999 and the EMI at $4,999. This doesn't include GBIC's but the max cost on those is long-haul (Cisco ZX) which lists at $5,995. The intermediate reach LX is only $995. Don't mess around with MMF (SX GBIC's) if you don't have to. Even if you do short haul, this will be a more stable solution and will allow you to repurpose gear in the future. The big gotcha with this is support of the fiber. Most of the time Cable dark fiber contracts mean that the customer is responsible for paying for repair of the fiber. Find out who is responsible for the fiber, and make sure you budget appropriately if the cable company is going to charge you for repairs on the lines. If they won't do it, find someone who can (check the contractors who already work with the cable company, they already know the systems and people) and get them on retainer or whatever. Many schools and governments have ended up shocked because they had to find someone in the middle of the night with a fusion splicer to fix a damaged link. Or the cable company sent them an outrageous bill for repair after the fact. This sucks, because it is hard to budget for an 18-wheeler gone awry in an ice storm. If you do it this way, you can treat the fiber just like Ethernet, and be done with it. You don't have to relearn protocols, and you will have a decent growth path for the future. My networks (that look just like this!) are WAN's that I treat like LAN's, and they are easy to support. Gybrwe

Let it GO

[Oculus Habent]

Look,

Without the additional details that are "required" by some of the posters, you can only speculate at an answer. Twirlip Of The Mists has done that, and had some very reasonable suggestions.

However, for a complete and assuredly valid answer, we simple need for information. As he said "let us have (for free) dark fiber links" that may mean that he really has nothing but fiber between buildings. Telcos aren't known for their sweeping generosity. He may have a lambda, which is absolutely reasonable in a metro area, which would give him something to work with.

If the fiber has sufficient bandwidth, he could split the fiber into data and video traffic (my high school used an OC3 in this manner for ITV classes and Internet access), but this would likely cost a good deal more.

The issue I have with the whole thing is "...when we are not using the fiber for cablecast." If you want cheap, as Twirlip Of The Mists suggests, that will mean (I could be wrong) physical disconnection of network cabling at both ends each time you broadcast, and then re-connection after broadcast. Would you have trained network people at each "public access" location, or would you end up driving around town before shows? Is it worth it to you?

Granted public access television isn't rolling in money, but to make your life easier, you probably want something you don't have to physically connect each time. Of course, you could get a fiber switch, and some X10 appliance switches and the Home Connect kit. Then you could just call up and switch off the data and switch on the feed.

I want it to be clear that I am not saying that would work.

Re:hehe

[Doco]

I think your definition is messed up.

From my work in the telecom industry for the last 11 years all of it with making fiber optic equipment, a piece of dark fiber is really just a piece of glass with nothing hooked to it. Newton's Telecom Dictionary also says that dark fiber is fiber with nothing on it.

Now - you may work with a telco company that calls their point-to-point service selling "dark fiber" but they are abusing the term. If the telco provides repeaters, then you can't run anything you want over the fiber you must run exactly what they provide for. If they have OC-3 repeaters - then all you can hook up in a 155Mbps OC-3. If they have 100Mbps Ethernet repeaters - then you can only use that. This means it is not really that flexible - they are selling OC-3 or 100Mbps Ethernet service - not dark fiber that can run anything that has enough power to reach from one point to another.

A gray area is when you have DWDM systems that can take a single wavelength of light at any bit rate and transport it through the system. Those systems will often have optical amplifiers which don't care about bit rate or frame format - but they are just what the name says - an amplifier. If noise gets into the signal they amplify that along with the signal so you eventually need a repeater which will convert the signal back to digital form and resend it. Repeaters are therefore bit rate dependant and therefore mean you can't run just any signal over that kind of a system.

Again - my experience with lots of telecom companies and fiber optic equipment is that dark fiber is just a piece of glass - almost always single mode fiber. Multimode fiber can't go more than a couple of hundred meters at any decent bit rate. Therefore the only place it makes sense to use is inside the same building. That would mean that your connection in your type of system would need to be in the telco central office, or they bring the single mode up to your building and then put a box there to do the single mode to multi mode conversion. If they are selling dark fiber they aren't adding equipment - so it will almost undoubtably be single mode fiber.

Just because you have state the same incorrect statements more than anyone else doesn't make the correct.

Of course - the submitter of this story didn't even start to give enough information and apparently doesn't want to show his face here to provide more. Without more information you can't even begin to know what to recommend.

Re:hehe

[Twirlip of the Mists]

Of course - the submitter of this story didn't even start to give enough information and apparently doesn't want to show his face here to provide more.

It seems more likely to me that the submitter got the answers that he needed and went on about his business.