Articles, Blog

2. Colorization with DeepRemaster, DeOldify and Photoshop Tutorial (How To)

January 19, 2020


Now if there’s a lot of movement in your video,
EBSynth is going to struggle with that. Like if it’s a running football player or
something, it’s going to smear and it’s going to tear. You’ll have to draw a lot of keyframes maybe. But there is another way to do this. I’ve been working on these experiments with
my friend Alex Douglas. And he wanted to use a program called DeepRemaster. This is an AI program which intends to automatically
restore old film. It looks for dirt and damage, and it’s also
a colorization program. It’s very different from EBSynth. EBSynth you control every step of the way. You type in your frame numbers, you give it
your keyframes. Deepremaster is a black box. You upload your video, and you upload a directory
of your keyframes. Deepremaster does the rest, and its ways are
mysterious. It goes through the video, and it tries to
make it look like your keyframes. It works best at its default resolution which
is I believe 400×320. And that is pretty low resolution but it’s
fine for colorization. And it does give a more vague, less saturated,
kind of smeary look, but it also handles motion really well so that’s going to save us a lot
of work. And what I’m going to do is, I’m going to
use my EBSynth renders, and I’m going to patch them with DeepRemaster. Anytime the EBSynth run doesn’t look good,
anytime it’s smeary and has the wrong color, I’m going to use Deepremaster to fill in the
blanks. It’s got something in common with DeOldify. Really, your computer needs an NVidia GPU
card in order to run it. I don’t have an NVidia GPU. Neither does Alex. DeepRemaster runs without it, but without
graphics acceleration it runs very slowly. But as with DeOldify you can use your web
browser and access a Google collab which can run Deepremaster. I wouldn’t recommend giving it too much to
do. Alex had a problem where Deepremaster kept
crashing. The collab had about 15 GB of memory, and
this AI process was using up all of it, and failing, especially when we tried to do something
at higher resolution than the default of 400×320. And we also found out that the default resolution
of 400 by 320 also gave us better results. So we split Metaphorically Speaking up into
a bunch of parts, uploading only one part at a time, and its associated keyframes, and
rendering the result at a smaller resolution, which gave us the best results. And it handles the motion very well. It looks a little worse than the EBSynth output. It’s a little blurrier, a little less saturated,
a little vague. But it’s also basically done at this point,
because it’s done such a good job of tracking the motion. EBSynth’s output is more accurate to what
I actually painted, but will require some additional work to make it perfect. With DeepRemaster we simply provided our keyframes
and it did the rest without further input from us. So these two programs actually work quite
well together. What EBSynth does well, Deepremaster doesn’t
and vice versa. Deepremaster tracked the motion very well
so I’m able to boost the saturation, we’re gonna boost the saturation of the Deepremaster
output in Premiere using the 3way color corrector, not the regular color corrector, not the Hue/Saturation
and Lightness but let’s use the 3way color corrector in Premiere and bring the Saturation
up to 175 or to 200. Wherever the EBSynth run looks a little weird
or where it’s obviously changing from one color keyframe to another we’re going to plug
in the DeepRemaster run at varying opacities depending on how good or how bad the EBSynth
run is at that point and we’re going to raise the saturation of the DeepRemaster to match
175 to 200 and that should make them look about the same. Just occurred to me the irony of using Python
for this because the Python programming language was named after Monty Python and technically
there is a member of Monty Python in some of these clips and there were four members
involved in this show. So okay we’re gonna go to colab.research.google.com/
and we’re gonna start a new Google collab which already has a lot of stuff installed
on it that we’re gonna need and we’re gonna start a new notebook in Python 3 and I’m gonna
click run time and hardware accelerator I’m gonna select GPU so that we have an NVIDIA
GPU and that’ll help us render this stuff. To create a new block of code to run there’s
a plus code button at the top left. So I’m going to type in exclamation point
!nvidia-smi and that’s loading in all of our NVIDIA GPU. Then I’m gonna clone the github I’m gonna
type in an exclamation point git clone and type in the github of Satoshii Izuka from
SIGGRAPH Asia 2019 remastering the DeepRemastering github I’m gonna create a directory %cd /content/siggraphasia2019_remastering
and then we’re gonna run a !bash download_model.sh on the left there should be a tool bar you
can open click the files tab right click the SIGGRAPH asia and click upload and choose
your video right click it again and create a new folder for your references right click
your references folder and click upload choose all of your reference frames like 10-15 frames
at once because if I load too many it’ll fail it actually screwed up when I had an underscore
in the file name of this video so I’m gonna take the underscore out I can then run the
remaster python Run (without brackets) !python remaster.py –input [Name Of Your Video] –reference_dir
[reference folder name] –gpu Uploaded files will be recycled when the session
ends but the code blocks will stay and will be saved to your Google account for easy reuse
you can put in a flag that says min dim that’s minimum dimension that specifies the actual
size of the video and we found that the smaller images actually had better color in a lot
of ways like the 600 width image or the 450-500 with image had these gray areas this is also
what we found would deal defy that if it was at its lowest setting the colors were much
more vivid like around 25 or 30 when he wanted to go up to 75 and that was just basically
a black and white image. But anyway we were still dealing with an error
an error on line 116 and it would say CV to resize error it would say open CV it’s saying
it can’t resize the last frame because there’s no more frames so it fails at the very end
of the video and that’s-that’s pretty stupid. We didn’t always have this problem but once
we started to have this problem we did not stop having this problem. We had this problem with everything we did
and it’s terrible. There are temp directories saving every frame
as a PNG file in a temp directory I could call up a zip file. So I put exclamation point !zip -r and then
I put in the name of a zip file I’d like to create and then the full /content/siggraphasia2019_remastering/tmp/output/
that’s the full path to the temp directory that DeepRemaster is outputting. And I’m zipping all that up into an output
file and then I save that file. Google has been blocking this download of
these zip files just over and over and over again. It’s like, we have blocked this site from
downloading multiple files. It never like asks me if I want to do this
it never says I can just allow this it just keeps doing this over and over and and screwing
with me and it takes a long time to download these files from Google colab and it’s Google
blocking Google. You’re blocking your own software! Every time I’ve had so many downloads fail
because it doesn’t go directly into Chrome’s downloads it’s doing its own download thing
and then when it’s done it goes to Chrome but then chrome blocks it as suspicious or
it’s downloading multiple files and there’s no way to shut this off and it’s driving me
nuts. I’ve been wasting quite a bit of time but
I did manage to download everything in the end it just took a little longer. So be on the lookout for that. Chrome may block your downloads. But anyway our final product, as you can see
here, is a mix. I’ve edited together the three different versions
that we have the EBSynth and then the DeepRemaster with saturation raised. I’ve brought that in at different opacities
up and down up and down. Whenever I didn’t like that section of the
EBSynth run whenever you could see a keyframe change or when it got smeary or was the wrong
color I brought up the DeepRemaster. And then over the top of that I changed opacity
blending mode luminance for the mini DV version of the original image from the DVD in full
quality so everything else was just the color and that looks pretty beautiful, if I do say
so myself.

1 Comment

  • Reply The Colorization Channel January 19, 2020 at 8:01 am

    That looks cool. I'm probably gonna colorize a Led Zeppelin concert video with this. Can it be done on Command prompt?

  • Leave a Reply