I will ask my contacts at Pixar if they know anything about it. I imagine they are using something a bit more robust.
All the render farms I know use LSF, one I know of uses qube! and maybe a few have in house tools, so the market is open for a good product still. I don't know how many studios will jump on getting Apples after just switching to Linux a few years ago. Also, with the Apple/Pixar connection, some may not want to switch ever.
A good render pipeline has a ton of features that need to be added to basic queueing systems. It is much more than just batch processing and parallel computing.
Most server farms do one task with different data. Although rendering is a task, it is extremely dynamic. Different shots can be orders of magnatude different. Also, each shot can have different priorities with multiple productions on going. They also may use different resources, different directory structures... many, many variations. We need a queueing system that can take all of that into account and find the best system at the best time for any given shot. There is an art form just to setting that up. This is why an extremely robust solution is needed. Job dependency is also a must. In one shot there may be 5 layers being renedered, an FX layer done seperately, a composite and then a movie generated. Each of those would need to be strung along in order with the system knowing what to do if any failur occurs.
The less automation, the more man hours are needed, and watching frames is tedious work.
Few packages meet these high demands, and even then, some still need tweeks and a few of the features.
-Tim
Re:Dear Slashdot... do my work for me please...
on
Building a Render Farm?
·
· Score: 2, Interesting
Maya has a real cheesey remote render tool. Its not really worth the time.
I built a 500 node/1000 proc render farm. Maya is a beefy application. We had scene files that were easiy 500mb. You can not chop up a scene file like that, it all needs to be loaded so that lighting can be done correctly.
Rendering to ram doesn't make much sense. You need all of that memory for the OS and the application.
A local hard drive makes a world of difference. I have seen night and day differences between rendering off of scene files locally and off the network (connected to an SGI raid file server). Few NFS servers can keep up with many boxes. And slow reads and writes are even worse. It is much better to burst the read and writes by doing whole files at a time. The server will thank you.
I can't recomend Platform's LSF enough, it allowed a small shop to scale from 40 desktops to 500 render boxes. All it does is job scheduling, so you need to put some intelligence into it. With one or two developers, you can come up with a snazy system that is pretty reliable.
I am also impressed with Qube! from pipelinefx. I have only demoed the software. It has the logic in it that LSF does not. So it all depends on how out of the box you want to go.
I have found that most render solutions cater to studios with 3 artists and 5 render boxes and don't scale well outside of that. If one wants to work on a larger scale, spending the time looking into the slightly more pricey options is well worth it.
I don't know of any studios taking that aproach. Maybe for Shake, but not for rendering. Every studio I have worked with or talked to has had all inhouse software or Maya or some combination on linux/x86.
I would love to see more information about this and how it scales and compares to other commercial offerings.
You know, I don't know if I have ever seen memory cause a problem. But if I had, I don't know if I would have figured out that was what it was, and I have done some extremely heavy renders before.
But at the current cost structure of ECC vs non, it is a good idea to be safe.
I say spend the money on ram instead of more machines. Hitting swap kills a job.
I have maxed out the 2gb limit many many times. Had it been in swap it would have been much worse.
I also like to cache scenes local as well as render local, so a fast local drive can make a difference.
In some of the servers we had 2 20 gb scsi that were striped.
Another good trick is if you have 2 drives, put swap on both of them at equal priority. That can lower some bottle necks as well.
-Tim
Re:Dear Slashdot... do my work for me please...
on
Building a Render Farm?
·
· Score: 2, Informative
A few corrections...
Hard drives are important. You never ever want to render to NFS. Bad things happen. Trust me. Also, its best if the files are local that its rendering. So having a 20gb scsi drive speeds things up... cache the scene file local, render local, copy the result back.
And 256 mb ram... hmm.. try 4bg of ram. I have hit the 2gb limit for renders many times, and if you want to have two renders going at the same time, you need that extra memory.
Software wise, as for 3rd party tools, renderman is great, but dang expensive. Maya is a good product. SoftImage is good as well, but Lightwave, pov-ray and others are not really in the camp of commercial products for animation.
100mb to the rack switch, dual gb from rack to big switches is plenty. No need for 1000mb. Most servers can't serve up data that fast. Not for hundreds of boxes.
And trust me, nobody has "that whole render farm thing figured out". Not even the big guys... we learn stuff every day.
I am a render farmer... and this is what I have gathered.
If you are doing single frames, get good systems. If you are doing animation, get many systems.
You need a scheduler or batch queueing system.
I suggest building one off of LSF like I have in the past and it is what I use now -or- get a nice prebuilt one like what pipelinefx has. This is extremely important. Without a good job scheduler, your cpu utilization will be less than optimal and it will cause you to get more machines than needed.
I have worked both with inhouse and 3rd party renderers. Maya doesn't do that bad of job, and the license for rendering is free. Big plus. Inhouse is great too, but if you don't have a good sized company, you probably won't have one and probably wouldn't be asking this question.
As for the machine. Most products are only speced for RedHat and for only certain releases. This is important when suport becomes an issue, and it will, when things don't work for a shot. I suggest a dual proc Pentium 4. I do not know if AMD's chips have been blessed by the vendors or not. I know when last I looked it was being investigated. Go for a dual system, even if you don't have a mutli-threaded renderer. Also max out the system ram at 4gb.
As for hardware vendors, I have used VA-Linux, HP, Compaq and SGI linux boxes as well as white-boxed. Hardware support is a big deal when you are a small shop. Go with a name brand unless you want to buy a ton of hot spares.
Storage is a big deal too. You obviously need a place to put the images, but you also need a server beefy enough to serve up a couple hundred mb to several boxes at the same time. When you get 500+ boxes, it gets to be an even bigger deal.
Other administration software i found usefull... SystemImager and gsh ( a distributed shell ) they allowed me to manage the 500 boxes as well as develop code and debug renders... I can not imagine what it would have been like without it.
3d animators are as much artists as 2d animators. And some of the really good ones used to be 2d animators. Having worked with several 3d animators, I can honestly tell you, working with computers all day does not mean they are geeks. Many of them know 3 things. How to turn the computer on, how to use the application and how to call the tech support for problems.
With that being said, there are many good 3d animators that are computer savy. Some of the bigger 3d studios want you to know some scripting just because of the nature of the work.
At the end of the day, 3d gives some freedom to directors and artists that 2d did not provide. It also provides some limitations as well.
I will miss traditionaly animated Disney movies, but I do think that Disney needs its own in house 3d team (if only to keep friends employed).
you are right, made on linux does not change the cg look and feel because its on linux. But what it has provided is more, faster machines for the masses. Because CG always takes all the resources you have, regardless of how many machines or how many months you have to produce, faster, better, and cheaper machines allow for better cg to be created.
Render times have always been about the same for a feature film, regardless of how fast the proc is. That is because the artists keep getting more and more detailed, the shaders more complex... so on and so on.
The other nice thing about a linux solution has been the adoption by various 3rd party vendors (where other unix variants are not supported). My last render farm setup was running perl, LSF (from Platform) and Maya (from Alias). Two of these are not available on all unix flavors. Maya is only available on 3 unixes, so the choice gets limited fast. Now many larger studios have mostly in house software but they do use third party applications many times, so it is still something that is taken into consideration.
But you are right, the linux kernel does absolutely nothing different graphicly than any other kernel out there. In some cases the linux kernel is still a bit behind in comparison to Irix .
Many studios are using Linux boxes and several of those are HP boxes. Most large stuidos run this way because the bottom line is as important to them as it is to the smaller studios. (See previous SCO takes on Hollywood story for more on this).
I had never used the corperate service agreement, and I was running a farm of 500 boxes. We had some sort of agreement on the fileserver and the raid, but the renderfarm was just toasters. And that is really how they need to be treated.
It only took two of us to keep them running within the guaranteed uptime. I was doing more administration, queue management and odds and ends, the other was doing hardware swapping and debugging. And most of the problems we had was heat related. We had too many machines for the data center, so we made make shift render rooms. A few racks here with portable ac units, a few racks there, lots of fiber ran everywhere. It was shoestring and ducttape, but it worked.
If I could spend money on more machines or 4 hour onsite response vs RMA'ing boxes, I would always go with more machines. Most of the time, you will need the horsepower before you need a rack of systems fixed. Generally, if it is catastrophic failure on several boxes at one time, and you have a good relationship with the vendor, they will next day replacement machines to you.
Agreed. That is a direction I would push for at most studios. At my previous place I had looked at that for a composoting farm of XServes... it would have been a good solution and allowed us to make avide files for editing to import.
I don't know if that is a viable option for all studios though. Linux is such a great product for the desktop. It is to 3d studios now as what SoftImage/Microsoft was in the 90's. It has lowered the entry price for feature films. It allows larger farms which allows for better quality films. At the end of the day, as much as we want to believe that films are just works of art, they are investments. Either they make money or they don't. The ROI for films is closely watched and if it is not high enough, you will be looking for other work.
There is also the problem with in house development as well. I would hate to have to port and support my code on yet another unix variant. We know the core is the same, but everything that makes each flavor different... that is what you have to watch out for.
I have gone with white boxes, hp, valinux boxes... I found that unless corperate agreements are involved, generic white boxes do great. You can have active spares and sometimes get 2 for the price of 1 IBM/HP system. This gives you 2x the power, which is great for when deadlines hit, there can never be enough render power to get stuff done that was due yesterday.
The only problem with 1u servers I have found is that its much easier to max out the cooling and power, just because you don't realize how many machines you really have in a rack.
They may be able to do that, but Alias has been slow with adding newer versions of RedHat as supproted OS's. I have had them whole versions behind on official supprot. And when you pay thousands of dollars for the software and a good deal for support, you want to run on supported hardware and software.
I hope that they start doing QA on fedora now, so that when it comes time, it will be suported.
BSD would be great, but some software products support RH only (like Maya). And the last thing I would want to do is debug render problems running Linux apps on BSD.
Right when I am getting ready to start work in the entertaiment industry again (in a week) SCO has to pull this stunt. I sure hope this doesn't scare people into dropping linux on the desktop or the renderfarm, the industry has just really started embracing it. The cost of swtiching to linux wasn't cheap, the cost switching back, that would be way way too expensive. Compaines with 1000+ box render farms would probably fight SCO on this (I can only hope) because the non linux solution would be going back to Solaris or IRIX, neither of which are cheap OS's or cheap hardware.
This could be the last stupid move that SCO makes. Maybe they are wanting to be bought out. HP has to hate this too, because they are really, really heavy in the CG industry as a Linux solutions provider.
Seeing that this is something near and dear to me (having built and ran render farms) I would love to know more information on what they are using to manage the renders... what batch queueing software, what render software, what animation and modelling software? I would love to know how they approached the problem.
My experience in the past was with Maya covering all the 3d and LSF from Platform for the queue management. Wrote some perl scripts for the frontend and for the backend of the system, did some database calls so that people could resubmit jobs if they failed without having to look up all the settings agin, also forced some uniformity to how it was submitted...
I know that student projects aren't the same as feature films or half hour animations, and managing for 60 artits on 500 procs is not the same as keeping students rendering, but it is still the same basic task.
And a bit of advice from somebody who runs such systems for a living... Just because the horsepower is there and it seems like you will be the only one using the system, if you can spend a few more minutes optimizing the models and the textures, it is worth it. Also take advantage of using layers and simple A over B composites. It will save you time in the long run, and it is possible that others may hit crunch time the same time you do. Computer resources are finite. Anything you can do to use as little of it as possible makes it easier for everybody to make deadlines. And if you do make it into the industry, it will be even more valuable, because your stuff with go through with less problems, and be less costly, and people notice that.
There are companies that provide render resources. They are great if you are a small studio and only go to final render once ever so often. They are expensive if they are your only resourse.
For a past project, we priced out outsourcing our final renders and discovered that it would have cost us $6M to render on $1M worth of hardware for about 4 months. Prices may have changed since. We dropped the money and brought in 400 boxes because we could have reused the machines for future projects.
But studios of 10-20 people, its not a bad idea. When it hits crunch time, its always good to have a couple hundred extra machines to get stuff done.
I had a render farm that went south because of this. Crappy switches were running in one mode, ethernet on the boxes another... and they were both set to auto negotiate. I dumped the piles of crap for some real switches and they started to work again.
Slashdot Mirror
I will ask my contacts at Pixar if they know anything about it. I imagine they are using something a bit more robust.
All the render farms I know use LSF, one I know of uses qube! and maybe a few have in house tools, so the market is open for a good product still. I don't know how many studios will jump on getting Apples after just switching to Linux a few years ago.
Also, with the Apple/Pixar connection, some may not want to switch ever.
A good render pipeline has a ton of features that need to be added to basic queueing systems. It is much more than just batch processing and parallel computing.
Most server farms do one task with different data. Although rendering is a task, it is extremely dynamic. Different shots can be orders of magnatude different. Also, each shot can have different priorities with multiple productions on going. They also may use different resources, different directory structures... many, many variations. We need a queueing system that can take all of that into account and find the best system at the best time for any given shot. There is an art form just to setting that up. This is why an extremely robust solution is needed. Job dependency is also a must. In one shot there may be 5 layers being renedered, an FX layer done seperately, a composite and then a movie generated. Each of those would need to be strung along in order with the system knowing what to do if any failur occurs.
The less automation, the more man hours are needed, and watching frames is tedious work.
Few packages meet these high demands, and even then, some still need tweeks and a few of the features.
-Tim
Maya has a real cheesey remote render tool. Its not really worth the time.
I built a 500 node/1000 proc render farm. Maya is a beefy application. We had scene files that were easiy 500mb. You can not chop up a scene file like that, it all needs to be loaded so that lighting can be done correctly.
Rendering to ram doesn't make much sense. You need all of that memory for the OS and the application.
A local hard drive makes a world of difference. I have seen night and day differences between rendering off of scene files locally and off the network (connected to an SGI raid file server). Few NFS servers can keep up with many boxes. And slow reads and writes are even worse. It is much better to burst the read and writes by doing whole files at a time. The server will thank you.
I can't recomend Platform's LSF enough, it allowed
a small shop to scale from 40 desktops to 500 render boxes. All it does is job scheduling, so you need to put some intelligence into it. With one or two developers, you can come up with a snazy system that is pretty reliable.
I am also impressed with Qube! from pipelinefx. I have only demoed the software. It has the logic in it that LSF does not. So it all depends on how out of the box you want to go.
I have found that most render solutions cater to studios with 3 artists and 5 render boxes and don't scale well outside of that. If one wants to work on a larger scale, spending the time looking into the slightly more pricey options is well worth it.
-Tim
I don't know of any studios taking that aproach. Maybe for Shake, but not for rendering. Every studio I have worked with or talked to has had all inhouse software or Maya or some combination on linux/x86.
I would love to see more information about this and how it scales and compares to other commercial offerings.
-Tim
You know, I don't know if I have ever seen memory cause a problem. But if I had, I don't know if I would have figured out that was what it was, and I have done some extremely heavy renders before.
But at the current cost structure of ECC vs non, it is a good idea to be safe.
-Tim
When I looked at them, they seemed a tad pricey for me.
It has been over a year though, they may be more competitive. I also don't know if it scaled well.
-Tim
I say spend the money on ram instead of more machines. Hitting swap kills a job.
I have maxed out the 2gb limit many many times. Had it been in swap it would have been much worse.
I also like to cache scenes local as well as render local, so a fast local drive can make a difference.
In some of the servers we had 2 20 gb scsi that were striped.
Another good trick is if you have 2 drives, put swap on both of them at equal priority. That can lower some bottle necks as well.
-Tim
A few corrections...
Hard drives are important. You never ever want to render to NFS. Bad things happen. Trust me.
Also, its best if the files are local that its rendering. So having a 20gb scsi drive speeds things up... cache the scene file local, render local, copy the result back.
And 256 mb ram... hmm.. try 4bg of ram. I have hit the 2gb limit for renders many times, and if you want to have two renders going at the same time, you need that extra memory.
Software wise, as for 3rd party tools, renderman is great, but dang expensive. Maya is a good product. SoftImage is good as well, but Lightwave, pov-ray and others are not really in the camp of commercial products for animation.
100mb to the rack switch, dual gb from rack to big switches is plenty. No need for 1000mb. Most servers can't serve up data that fast. Not for hundreds of boxes.
And trust me, nobody has "that whole render farm thing figured out". Not even the big guys... we learn stuff every day.
-Tim
I am a render farmer... and this is what I have gathered.
If you are doing single frames, get good systems.
If you are doing animation, get many systems.
You need a scheduler or batch queueing system.
I suggest building one off of LSF like I have in the past and it is what I use now -or- get a nice prebuilt one like what pipelinefx has. This is extremely important. Without a good job scheduler, your cpu utilization will be less than optimal and it will cause you to get more machines than needed.
I have worked both with inhouse and 3rd party renderers. Maya doesn't do that bad of job, and the license for rendering is free. Big plus.
Inhouse is great too, but if you don't have a good sized company, you probably won't have one and probably wouldn't be asking this question.
As for the machine. Most products are only speced for RedHat and for only certain releases. This is important when suport becomes an issue, and it will, when things don't work for a shot. I suggest a dual proc Pentium 4. I do not know if AMD's chips have been blessed by the vendors or not. I know when last I looked it was being investigated. Go for a dual system, even if you don't have a mutli-threaded renderer. Also max out the system ram at 4gb.
As for hardware vendors, I have used VA-Linux, HP, Compaq and SGI linux boxes as well as white-boxed. Hardware support is a big deal when you are a small shop. Go with a name brand unless you want to buy a ton of hot spares.
Storage is a big deal too. You obviously need a place to put the images, but you also need a server beefy enough to serve up a couple hundred mb to several boxes at the same time. When you get 500+ boxes, it gets to be an even bigger deal.
Other administration software i found usefull...
SystemImager and gsh ( a distributed shell ) they allowed me to manage the 500 boxes as well as develop code and debug renders... I can not imagine what it would have been like without it.
-Tim
3d animators are as much artists as 2d animators. And some of the really good ones used to be 2d animators. Having worked with several 3d animators, I can honestly tell you, working with computers all day does not mean they are geeks. Many of them know 3 things. How to turn the computer on, how to use the application and how to call the tech support for problems.
With that being said, there are many good 3d animators that are computer savy. Some of the bigger 3d studios want you to know some scripting just because of the nature of the work.
At the end of the day, 3d gives some freedom to directors and artists that 2d did not provide. It also provides some limitations as well.
I will miss traditionaly animated Disney movies, but I do think that Disney needs its own in house 3d team (if only to keep friends employed).
-Tim
HAHA
They are seperate machines though.. (but I am sure you knew that)
-Tim
you are right, made on linux does not change the cg look and feel because its on linux. But what it has provided is more, faster machines for the masses. Because CG always takes all the resources you have, regardless of how many machines or how many months you have to produce, faster, better, and cheaper machines allow for better cg to be created.
Render times have always been about the same for a feature film, regardless of how fast the proc is. That is because the artists keep getting more and more detailed, the shaders more complex... so on and so on.
The other nice thing about a linux solution has been the adoption by various 3rd party vendors (where other unix variants are not supported). My last render farm setup was running perl, LSF (from Platform) and Maya (from Alias). Two of these are not available on all unix flavors. Maya is only available on 3 unixes, so the choice gets limited fast. Now many larger studios have mostly in house software but they do use third party applications many times, so it is still something that is taken into consideration.
But you are right, the linux kernel does absolutely nothing different graphicly than any other kernel out there. In some cases the linux kernel is still a bit behind in comparison to Irix .
Many studios are using Linux boxes and several of those are HP boxes. Most large stuidos run this way because the bottom line is as important to them as it is to the smaller studios. (See previous SCO takes on Hollywood story for more on this).
-Tim
I had never used the corperate service agreement, and I was running a farm of 500 boxes. We had some sort of agreement on the fileserver and the raid, but the renderfarm was just toasters. And that is really how they need to be treated.
It only took two of us to keep them running within the guaranteed uptime. I was doing more administration, queue management and odds and ends, the other was doing hardware swapping and debugging. And most of the problems we had was heat related. We had too many machines for the data center, so we made make shift render rooms. A few racks here with portable ac units, a few racks there, lots of fiber ran everywhere. It was shoestring and ducttape, but it worked.
If I could spend money on more machines or 4 hour onsite response vs RMA'ing boxes, I would always go with more machines. Most of the time, you will need the horsepower before you need a rack of systems fixed. Generally, if it is catastrophic failure on several boxes at one time, and you have a good relationship with the vendor, they will next day replacement machines to you.
Agreed. That is a direction I would push for at most studios. At my previous place I had looked at that for a composoting farm of XServes... it would have been a good solution and allowed us to make avide files for editing to import.
I don't know if that is a viable option for all studios though. Linux is such a great product for the desktop. It is to 3d studios now as what SoftImage/Microsoft was in the 90's. It has lowered the entry price for feature films. It allows larger farms which allows for better quality films. At the end of the day, as much as we want to believe that films are just works of art, they are investments. Either they make money or they don't. The ROI for films is closely watched and if it is not high enough, you will be looking for other work.
There is also the problem with in house development as well. I would hate to have to port and support my code on yet another unix variant. We know the core is the same, but everything that makes each flavor different... that is what you have to watch out for.
I have gone with white boxes, hp, valinux boxes... I found that unless corperate agreements are involved, generic white boxes do great. You can have active spares and sometimes get 2 for the price of 1 IBM/HP system. This gives you 2x the power, which is great for when deadlines hit, there can never be enough render power to get stuff done that was due yesterday.
The only problem with 1u servers I have found is that its much easier to max out the cooling and power, just because you don't realize how many machines you really have in a rack.
They may be able to do that, but Alias has been slow with adding newer versions of RedHat as supproted OS's. I have had them whole versions behind on official supprot. And when you pay thousands of dollars for the software and a good deal for support, you want to run on supported hardware and software.
I hope that they start doing QA on fedora now, so that when it comes time, it will be suported.
BSD would be great, but some software products support RH only (like Maya). And the last thing I would want to do is debug render problems running Linux apps on BSD.
-Tim
Right when I am getting ready to start work in the entertaiment industry again (in a week) SCO has to pull this stunt. I sure hope this doesn't scare people into dropping linux on the desktop or the renderfarm, the industry has just really started embracing it. The cost of swtiching to linux wasn't cheap, the cost switching back, that would be way way too expensive. Compaines with 1000+ box render farms would probably fight SCO on this (I can only hope) because the non linux solution would be going back to Solaris or IRIX, neither of which are cheap OS's or cheap hardware.
This could be the last stupid move that SCO makes. Maybe they are wanting to be bought out. HP has to hate this too, because they are really, really heavy in the CG industry as a Linux solutions provider.
-Tim
Hmm... not much details into how the queueing works. Or how the renders get split up. Anybody with more details, would love to "talk shop".
Looks like they have a few options for modeling and rendering.
-Tim
Seeing that this is something near and dear to me (having built and ran render farms) I would love to know more information on what they are using to manage the renders... what batch queueing software, what render software, what animation and modelling software? I would love to know how they approached the problem.
My experience in the past was with Maya covering all the 3d and LSF from Platform for the queue management. Wrote some perl scripts for the frontend and for the backend of the system, did some database calls so that people could resubmit jobs if they failed without having to look up all the settings agin, also forced some uniformity to how it was submitted...
I know that student projects aren't the same as feature films or half hour animations, and managing for 60 artits on 500 procs is not the same as keeping students rendering, but it is still the same basic task.
And a bit of advice from somebody who runs such systems for a living... Just because the horsepower is there and it seems like you will be the only one using the system, if you can spend a few more minutes optimizing the models and the textures, it is worth it. Also take advantage of using layers and simple A over B composites. It will save you time in the long run, and it is possible that others may hit crunch time the same time you do. Computer resources are finite. Anything you can do to use as little of it as possible makes it easier for everybody to make deadlines. And if you do make it into the industry, it will be even more valuable, because your stuff with go through with less problems, and be less costly, and people notice that.
-Tim
There are companies that provide render resources. They are great if you are a small studio and only go to final render once ever so often. They are expensive if they are your only resourse.
For a past project, we priced out outsourcing our final renders and discovered that it would have cost us $6M to render on $1M worth of hardware for about 4 months. Prices may have changed since. We dropped the money and brought in 400 boxes because we could have reused the machines for future projects.
But studios of 10-20 people, its not a bad idea. When it hits crunch time, its always good to have a couple hundred extra machines to get stuff done.
-Tim
I stand corrected, I should have known this. All the video work I had done previously had been to DVD or NTSC standards. Thanks for the correction.
-Tim
right. and 1280x1024 means 1280 lines, 20% wider... and its at 4x3 instead of 16x9, so it has more than 20% more pixels.
People do realize that the xbox output at NTSC or HD at best... max res is 1080
Most gamers won't play below 1280x1024 on the PC. Comparing the speed between the two is hardly fair unless it is done at 640x480 on both...
-Tim
I had a render farm that went south because of this. Crappy switches were running in one mode, ethernet on the boxes another... and they were both set to auto negotiate. I dumped the piles of crap for some real switches and they started to work again.
Never again will I use those switches.
-Tim