No Technical Difficulties

CRWBY: Behind the Episode: No Technical Difficulties

Join CRWBY as they open the studio doors for an in depth look on how each episode was made. Kerry, Miles and the team will walk you through the 3D pipeline and talk about how art, tech, post, and animation come together to create RWBY.

Binge Mode

More CRWBY: Behind the Episode

See All CRWBY: Behind the Episode Videos

Other Videos You'll Like

Comments (42)

  • Zabon95 FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold That One Tool

    3 hours ago

    tldr; Tech team are the support mains

  • LestatXero FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    5 days ago

    This is quite fascinating as I had no idea it took so many different people working in different departments to make these shows. It's great that THIS show, at the very least, lets us see behind the curtain a bit and puts faces with names that we may have read in the credits but had no idea what they actually did. 

    • DavidtheWavid FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      2 days ago

      Yeah, it's like DVD bonus features but seeing them as the episodes come out is pretty cool too.

  • dkphanson FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    Really interesting. Would love to hear more from the tech team!

    • TSalvaggio FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      6 days ago

      Thanks! We would dig hearing what you want to know.

  • paperflower FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    Amazing work, keep it up, CRWBY! I'm so excited for the episodes and it's also exciting to see what's going on behind the scenes. Looking forward to the future episodes.

  • gem_scheltema FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold RTBox reviews each month

    1 week ago

    Thank you tech team for your hard work. Someone's gotta build the stuff that the artists work on. You guys make a lot of stuff happen and I'm very appreciative (nobody likes technical difficulties).

  • Wadcroft FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Shameless Fiction Writer

    1 week ago

    The sysadmins! My people!

  • phubbe FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    it gets better and better every episode. I can just imagine the possibility.

  • ninjaassassin FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    I NEED MOAR! Also great job.

  • stopboorider FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    It's episodes like this that help put doing creative work on your own into perspective, as well as how a team can grow. I think before today I just assumed all artists dealt with their own technical issues on their own, it didn't occur to me there'd be another team to work on that. What a beautiful environment to work in. I don't feel so unaccomplished now when I have to learn this all on my own.

    • theVFXmex FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Lighting Artist

      1 week ago

      Believe me, we wouldn't be able to work as efficiently as we do without our tech team, so big thanks to them! Plus, they're a great group of people.

    • TSalvaggio FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      6 days ago

      Before I was part of this tech team, I was one of the artists that dealt with the technical issues on my own (which is what lead me here). With a team this size, it's cool to be able to help out my animation peeps and let them do what they do best. They should be animating, not figuring out why software has decided to revolt on a given day and how to get it to behave.

  • Jaune_Black_Smudge FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    @plumpo


    What type of server boxes do you guys use to package the GPUs, power suppliers, coolant, etc. for your Renderers? Any particular reason why you prefer it?


    Is there a reason why each GPU Renderer is limited to 4 GTX 1080/Ti's in each box? Could you fit more inside each of them or would there be some problems with that? 


    Is there certain parts of the production or show that require CPU rendering over GPU rendering? I'm asking this because it is a fact that GPUs render a lot faster than CPUs so I was wondering if there are certain VFX or other parts of the show that would require the use of CPU rendering over GPU rendering.




    • CallumGibson FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      1 week ago

      I clearly don't work there, but I imagine that they are limited to 4 GPU's per box because 1. Heat Dissipation, especially in that tight config, 2. Most motherboards only support 2 way multi GPU's so they will need slightly more specialised hardware, including PSU's. 


      I'd imagine CPU rendering is mabye for physics, but GPU's are also good at this. I'm sure CPU's will be used for actual video encoding and compression. The video provided here on the site is very well compressed considering H.264 and the profiles/levels. 


      I was baffled at the 1/2 a petaflop of computation they have in GPU's, but if these are made weekly, that a lot of people sharing the farm rig from Compositors, Special Effects etc.


      This is my outside take. Would love to know more! ;)

    • plumpo FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Technofixer

      1 week ago

      We are using Chenbro cases. They are 4U and are based around consumer hardware, so that keeps the price down a lot. Being able to run to a store locally and grab parts is nice fall back if stuff breaks.


      There are some Server level cases that can support 8-10 GPU's in 5U, but to power them we need more space on the rack for UPS's. I think we ended up getting like 8 more cards per rack when we specced that out, but it was a lot more expensive. 


      Using Blower style cards make cooling the cards really easy, so we could throw a bunch side by side and not have to worry about over heating (provided you have a good amount of intake and a temp controlled room).


      Currently we are using CPU rendering for Pencil+ (Eric mentions it in the video) and a lot of the Comp work is still CPU bound, but things are generally moving to GPU rendering whenever possible. Video encoding is mostly CPU bound as well. VFX is also pretty heavy on the CPU still.  2D shows (Camp Camp, RTAA) are all CPU rendered as well. 


      We won't know till later in the year what our Render needs will be, but I'm pretty confident we're going to be adding at minimum another 16 machines. Once you have the render muscle, the artists want to push what they can do and make things look better or more details, which in turn means we need more hardware. Its a vicious cycle that I very much enjoy. 


      Very excited to see what the GTX line for Volta is going to look like in a few months. Probably going to be swapping in those as soon as we can get a hold of them. 

    • theVFXmex FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Lighting Artist

      1 week ago

      "Once you have the render muscle, the artists want to push what they can do..." Yup! Guilty as charged!

    • jarble FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      6 days ago

      As @plumpo said you can many more gpu's per but cost starts to scale exponentially with super dense configurations and it is better to just add more clusters.  

    • Jaune_Black_Smudge FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      6 days ago

      @CallumGibson


      Thank you for your input. So they limit themselves to 4 GPU's per box for conservation of heat. That makes sense. Thank you.


      @plumpo


      That was pretty informative. Thank you.


      I got some info you might be interested in. In regards to rendering, I know that in maybe 1-3 years, Pixar will release Renderman XPU, a rendering software intended to use both the CPU and GPU concurrently, taking full advantage of workstation resources. That means your stuff gets rendered even faster. 


      Of RenderMan’s main rivals, Arnold is still purely CPU-based, its much-anticipated GPU support having failed to materialise in Arnold 5 this year; while Redshift is GPU-based, and takes little advantage of the CPU. 


      V-Ray does have a new hybrid CPU/GPU rendering system, rolled out in V-Ray 3.6 for 3ds Max earlier this year, but it’s only for the V-Ray RT interactive renderer, not the production renderer.


      So if Pixar gets it right, they'll be able to help practically every production company in the world use all their hardware resources more efficiently. Of course, whether you guys decide to switch from Redshift to Renderman XPU depends on whether you like the latter's rendering visual product. But I think it's something pretty cool to watch out for.


      --------


      What type of CPU do you think would be needed to complement the Nvidia Titan V?


      -------


      The other thing is about the Nvidia Titan V. I read through a review of the Titan V by Puget Systems:


      https://www.pugetsystems.com/blog/2017/12/12/A-quick-look-at-Titan-V-rendering-performance-1083/


      The Titan V is mainly intended for Deep Learning and Artificial Intelligence but it can also be used to render due to the sheer number of CUDA cores it has. So even if you don't have any real need for any of the Deep Learning/A.I. stuff, you got this. 


      Through a benchmark test that they compared to a GeForce GTX 1080Ti, using the available rendering software that currently work on the Titan V, which are V-Ray and Furryball RT, with Redshift and OctaneRender being available for it in the future.


      Anyways, the results showed that the Titan V was easily 60% faster to the 1080Ti on V-Ray and nearly twice as fast as the 1080Ti on Furryball RT. So 1 Titan V basically equals 2 GTX 1080Ti. 


      However, it's $3000 price tag, which is just a bit over 4 times the price of a GTX 1080Ti so the price is a bit of a problem. Either way, it's still a better deal than getting the workstation class Nvidia Quadro P6000 with its insane price of $5000. Anyways, take a look at the article if you want. 


      Although I'm kind of worried about something. Nvidia recently made an announcement that they were banning the mass buying of GTX 1080Ti's for use in data centers because they were cheaper than the Quadro P600. So would your Renderfarm count as a "data center"? Or at what point in expansion will it be considered one?




    • plumpo FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Technofixer

      2 days ago

      @Jaune_Black_Smudge 


      The idea of using both CPU/GPU sounds nice, but I feel like the cost of the CPU to really see a significant boost isn't worth it at our scale. Currently we are using 5930k and I wouldn't expect them to really offer much to the process. Something in the higher core count Xeon level could mix things up though. 


      As for the Titan V, If you're talking about GPU rendering, you're not going to need much. The CPU is just going to help set up and clean up after a render. So any 4+ Core 3ghz chip from the last 5 years or so shouldn't impact the Render times (Provided the mobo is using PCIe 3.0). 


      For rendering, yeah the Titan V is a beast compared to a single 1080Ti, but over all its not efficient from a cost to performance ratio. More cards are better than fewer faster cards. 


      Ugly math time with Pugent's numbers:


      Titan V = 42 seconds per frame

      1080TI = 67 seconds per frame


      If we run 4 Cards in Parallel, 4x 1080TI's could run 4 frames in 67 Seconds (1 Card Per Frame), while 1 Titan V would take 168 seconds to move through the same work load. So for the money more the TI's are king. For a long time the 1070 was actually the Sweet Spot for rendering. Cost/Performance was awesome. We went 1080 because the 10% boost in rendering speed over an entire season of rendering is pretty massive and worth the extra $200 per card. 


      Rendering is currently not defined as a Data Center, so we're good with their new use terms. Currently we aren't going through NVIDIA directly to make purchases, just through standard re sellers. Direct would mean larger bulk orders which we haven't been doing. Usually grabbing 12 cards here or there as we expand. 










    • Jaune_Black_Smudge FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      2 days ago

      @plumpo


      Thanks. 


      I was mainly just asking about the CPU/GPU rendering thing because I read in an article that most VFX companies don't use GPU but are instead CPU-only when it comes to the final render because of quality and especially memory. Is Rooster Teeth the same where they only CPU render the final render for the show?


      By the way, any idea on where big animation and VFX companies like Pixar, Disney, ILM, Weta, DreamWorks, etc, get their rendering hardware from and what type of hardware do they get for their HUGE renderfarms? 


      I got 3 hypothetical questions that I'd like your professional opinion on if you'd like.


      First hypothetical question. Between an 18 core Intel Xeon W processor versus the 18 core Intel i9 7980xe, which would be better, specially for rendering considering that the Xeon W is a bit over $500 more than the i9 7980xe? Would the both of them be suited for rendering?


      Second, hypothetically speaking, if Rooster Teeth ever expanded enough and you got the necessary resources, would a Dual Core Xeon Platinum 8180 (28 core 56 thread) processor be suited for rendering or would it be better off for analytics? How much faster would it be?


      Alright, cool to find that out about the Titan V. But my last hypothetical question. I've seen your picture of the guts of one of RT's render machines. So say you had one such render machine except instead of 4 1080Ti's, you had 4 Nvidia Titan V's in the same configuration (with the other hardware being upgraded to accommodate if necessary), how much faster would that 4 Titan V render machine be compared to the standard 4 1080Ti render machine? Would it just cost too much power? 



    • plumpo FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Technofixer

      2 days ago

      @Jaune_Black_Smudge


      I'm not sure how much longer the "CPU-Only" rendering solutions will be standard (or if it still is) for larger companies. GPU rendering is just significantly faster than CPU rendering. The visual look can be different on the finished product so that end "look" might also have played some role in why they are still CPU bound. RT is the only company I have worked for in this capacity so I can't speak for how other shops do things or their rational behind them, but my guess would be Cost/Infrastructure might be why they are still on CPU. 


      Another end of it might be that a lot of their tool sets are most likely proprietary, and getting a dev team to build/test that kind of stuff is going to take a while to get ready for a $600 Million production. I would be really surprised if they don't have all of this stuff working in the background and planned for an upcoming project.


      We have a lot of custom tools that are written in house, but we are using off the shelf products for a lot of things. This helps us to be a lot more agile with testing out and implementing new software/technologies. We swapped between ALL CPU rendering at the end of RWBY 3 to Mostly GPU rendering for RWBY 4, and that let us drastically change/improve the look of the show.  But as we grow, it does get harder and harder to change directions on some of them. 


      Technically Yes, we final render with a CPU (with GPU acceleration) in After Effects and Premier. But that is just the comp effects and then the final transcoding at the end. We add our ToonLines around the characters with a CPU rendering software called Pencil+, but the majority of the heavy lifting is all GPU. 


      Where they get their hardware? They are most likely getting things direct from Manufactures for anything in house. They have some very deep pockets they can probably just call up Steve Intel and ask for $10 Million Dollars of Hardware.


      We are specing out some beefier CPU rendering machines, so a high thread count box would definitely be helpful for anything we can't GPU render. We are also looking into some caching and playblast systems that would let us farm that work out so its not being done on an Animators machines, but I don't think we are going to go with the 8180's. When all is said and done that is probably a $30-40k machine (if running two of them). 


      Can't really speak to how much faster it would be for CPU rendering, but I would guess a lot. If a render can be multi threaded, then it would crush since you have so many more cores churning. If the render is locked to a single or a couple cores, it probably wouldn't go as well as something with faster core speeds, just a lower core count. Thankfully multi threading is pretty common in our tool sets. 


      Between the W-2195 and the i9-7980XE, its pretty neck and neck based on performance. The Xeon has some more cache and  higher bus speed and can support a ton more Ram. Clock speeds are much faster on the i9, so I would assume its going to move through frames better while the Xeon would handle the set up/finishing of the renders better. If you're using a really complex scene the extra ram on the Xeon would be necessary and that would take the i9 out of the running all together. So its hard to say which is better, its probably based purely on what is being rendered. 


      I could drop 4 Titan's into any of our Render nodes without having to change anything out. I would want to swap the 1200W PSU's to 1500W ones just to have some more overhead. I did this on all of our machines with 1080TI's. The Titan V and the TI both have a TDP of 250w, so in terms of power consumption/resources they are identical.


      How much faster would it be? Based on those Pugent numbers the Titans could do theoretically do 1 frame every 10.5 seconds (4 frames in 42 seconds) vs the 1 frame 16.75 seconds (4 frames in 67).  So that 60% faster still sticks around, but only if the cost of the hardware is taken out of the equation. 


      With that added in, a single Titan machine would run around $14.500 while a 1080TI machine is $5400. So without spending more, I can get 2 Machines (8 1080Ti's) for the cost of 1 Titan machine. 


      Titans = 1 frame every 10.5 seconds (4 frames in 42 seconds) 

      1080TI = 1 frame every 8.375 seconds (8 frames in 67) and $3700 left over for pizza. 


















    • Jaune_Black_Smudge FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      1 day ago

      @plumpo


      Thank you for your response. It's very informative.


      When it comes to the GPU rendering machine, do you need a strong enough CPU to complement the GPU (because GPU rendering still requires the CPU to function)?


      And one last hypothetical rendering rig question. If you got a Dual Xeon Platinum 8180 processor combined with 8 Nvidia GeForce GTX 1080Ti's along with all the other hardware which would cost an estimated $26,000, how fast do you think it would be at rendering? 





  • Favna FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold RWBY Fan for life

    1 week ago

    Can we just appreciate how the second chapter in RWBY GE was titled "Technical Difficulties" and this is "No Technical Difficulties"

  • CallumGibson FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    I'd love to see final rendering tech and process at some point. Do you use a CPU server, workstation GPU's? 

    • plumpo FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Technofixer

      1 week ago

      We're using Desktop GPUs. Our primary farm has 64 GPUS (32 1080s and 32 1080TI's). CPU Rendering is done mostly on a bunch of 4970k's and some 4770k's. 


      The Guts of one of our GPU Renderer (with 4x 1080s)

      aZQA0Pp.jpg

      First 7, we have 16 now. 

      hHqnotC.jpg

    • CallumGibson FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      1 week ago

      Interesting. Wasn't expecting so many GPU's. Any reason why you don't use Quadro's? Edit : 576 TFLOPS with GPU's, that's nuts.

    • plumpo FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Technofixer

      1 week ago

      Quadros/Tesla's are expensive and for the price don't offer the proportional speed for rendering. They give a lot more VRAM, so they can have does much more complex scenes (and we're not there yet).

  • Invader-Jim FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    Look at that poor bastard pretending to enjoy Kerry's jokes.  What a cruel work environment.

    • hypeathon FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      1 week ago

      That "poor bastard" is the lead storyboard/Camera & Layout artist and voice of Yatsuhashi Daichi, Joe MacDonald. He's been involved with the show since volume 2, if not earlier than that.

    • crimson45 FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Nope

      1 week ago

      lol

    • Invader-Jim FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      1 week ago

      @hypeathon

      You say that as though it's supposed to negate the fact he looked like a Best Korean "laughing" at one of their Dear Leaders jokes for a propaganda film.

  • blu_CH33Z FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    Fun episode this week! the animation style is gorgeous and the music goes great with the action sequences <3

    • 1SunStrider FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Sunstrider

      1 week ago

      The animation looks pretty but it makes the action much slower. They might want to return to motion capture after the poor quality of the fights in volumes 4 and 5.

  • thmsmcd6 FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    These are always interesting to watch. I appreciate all the work that goes into making RWBY.

    • crimson45 FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Nope

      1 week ago

      ya know, im 18 now, but when i saw the first trailers in like...what? 2012? that litterally inspired me to become an animator, im graduating soon and i was able to take classes in cinematography and animation thankfully from my district, Monty and RT are amazing

  • Zahlen FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold Danger Noodle Animator

    1 week ago

    As always, from an aspiring animator from a team of artists inspired by Monty Oum, RWBY, and Rooster Teeth, thank you for these videos.

  • gem_scheltema FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold RTBox reviews each month

    1 week ago

    NO

    rQc9Oy9.jpg

    • TSalvaggio FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      6 days ago

      Nice! This is also one of my tech-work soundtracks. :)


    • gem_scheltema FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold RTBox reviews each month

      4 days ago

      @TSalvaggio really cool song. The drums at the start immediately reminded me of 'Divide' from the RWBY V3 soundtrack.

    • TSalvaggio FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

      3 days ago

      In the spirit of things, I have to repost this for some reason :)

  • Hounds_of_War FIRST Member Star(s) Indication of membership status - One star is a FIRST member, two stars is Double Gold

    1 week ago

    Nora looks smug. Between this and the episode preview on RWBY Rewind, I’m excited.