# transcriptions_fts_content

c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
So, it can be hard to tell what's easy and what's 
nearly impossible, and in 2015 or at the end of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
that's nearly impossible. We're now going to build 
exactly that system for free in about two minutes!


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
it just doesn't work that way. So I just decided 
to use forest - I thought okay pictures of forest  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
particularly big ones and it takes a surprisingly 
large amount of time just for a computer to open  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
by passing in that bird that we downloaded at the 
start. This is a bird. Probability it's a bird:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
a joke to so easy that I can run it on my laptop 
computer in (I don't know how long it was) about  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
generate new pictures. And I thought this was an 
amazing thing that this guy nick did where he took  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
his friends twitter bios and typed them into the 
DALLꞏEꞏ2 input and it generated these pictures. So  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
And so when actual artists, you know, this for 
example, this guy said he knows nothing about art,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
you know he's got no artistic talent, it's just 
something you know, he threw together. This guy is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
artists have gone on to bring deep learning and 
art together, and it's a very exciting direction.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
And it's not just images to be clear, you know one 
of another interesting thing that's popped up in  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
And so you can see, you know, again, deep learning 
models are doing things which I think very few,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
course, taught by my co-founder Dr Rachel Thomas, 
which goes into these issues in a lot more detail.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
if you're following along that doesn't really mean 
much obviously. This is really for people who are,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
email me to say that they've got their dream job. 
Yeah… If you're looking for inspiration on how to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
get into the field I couldn't recommend nothing… 
nothing would be better than checking out Radek's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
writers and researchers on education Paul Lockhart 
and David Perkins, and many others talk about how  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
effectively is from the way we teach sports, for 
example, where we show you a whole game of sports.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
technically detailed classes you'll find - later, 
right! But first you'll learn to be very very good  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
at actually building and deploying models. And you 
will learn why and how things work as you need, to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
get to the next level. For those of you that have 
spent a lot of time in technical education (like  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
if you've done a phd or something) will find this 
deeply uncomfortable because you'll be wanting to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
before - the guy who came up with the really cool 
cups things. There'll be a lot of tricks that have  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
“Deep Learning for Coders” and that book is what 
this course is quite heavily based on. We're not  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
but the reason actually is that the educational 
research literature shows that people learn things  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
and you'll also see the same information presented 
in a different way, in these videos. So on,e of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
loves the book. Hal Varian Emeritus Professor at 
Berkeley, Chief Economist, Google, likes the book.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
Jerome Pecente who is the head of AI at Facebook 
likes the book. A lot of people like the book, so  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
working in and around machine learning including 
building a number of companies that relied on it.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
medicine, and MIT voted it one of the 50 smartest 
companies in 2016, just above Facebook and Spacex.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
a big impact on the world already, including work 
we've done with our students, has been globally  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
was getting hbr's attention) a lot of people have 
been watching the course, and it's been, you know,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
by looking at their histopathology image slides. 
Now, so this is, like, what I would call a classic  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
experts at Stanford. So this project took years, 
and a lot of people, and a lot of code, and a lot  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
they found. And this set of weights, for example, 
finds diagonal edges. This set of weights finds  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
yellow to blue gradients. And this set of weights 
finds red to green gradients, and so forth, right.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
with a neural net which is the basic function used 
in deep learning, we don't have to hand code any  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
of these or come up with any of these ideas. You 
just start with actually a random neural network  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
And when you combine these features it creates a 
feature detector, for example, that finds kind of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
features it can find are. And so you can imagine 
that trying to code these things by hand would be,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
right! So what we're going to learn is how neural 
networks do this automatically, right, but this  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
is the key difference of why we can now do things 
that previously we just didn't even conceive of as  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
possible, because now we don't have to hand code 
the features we look for. They can all be learned.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
image based algorithms and image-based algorithms 
are not just for images and in fact this is going  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
There's something else I wanted to point out which 
is that as you saw when we trained a real working  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
bird recognizer image model we didn't need lots 
of math; there wasn't any. We didn't need lots of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
data. We had 200 pictures. We didn't need lots of 
expensive computers; we just used my laptop. This  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
mainly passed along by big companies that want to 
sell you computers to store lots of data and to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
process it. We find that most real world projects 
don't need extraordinary amounts of data at all  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
the majority of people that were doing Tensorflow 
in 2018 researchers, the majority have now shifted  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
to Pytorch and I mention this because what people 
use in research is a very strong leading indicator  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
I will say however that Pytorch requires a lot of 
hairy code for relatively simple things. This is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
This gray bit here is the code required to do the 
same thing with fast.ai. fast.ai is a library we  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
you shouldn't be writing all this code if you only 
need to write this much code, right? The problem  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
find you get better results. So... so fast.ai has 
been a really popular library and it's very widely  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
cloud platforms that I'm not controlling is, make 
sure that I'm using the most recent version of any  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
of a lot of libraries so fast.ai provides a lot of 
convenient things for you. Oh I forgot to import…


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
generally… you'll find that for computer vision 
algorithms you don't need particularly big images,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
Okay, so the main thing I wanted to tell you about 
was this data block command. So the data block  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
is the key thing that you're going to want to get 
familiar with as deep learning practitioners at  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
the start of your journey because the main thing 
you're going to be trying to figure out is how do  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
be spending all of our time talking about neural 
network architectures, and matrix multiplication  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
and gradients and stuff like that, the truth is 
very little of that comes up in practice, and the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
it almost never comes up. Kind of like if you ever 
did like a computer science course or something  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
get to the real world and you never use it again. 
So this course is called practical deep learning,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
Okay, so our images have finished downloading, and 
two of them were broken so we just deleted them.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
programs I find for kind of the kind of work I do 
that a functional style works very well if you're,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
more from other things. So yeah that's why you'll 
see me using stuff like map and stuff quite a lot.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
So the first thing that we tell fast.ai is “what 
kind of input do we have”?, and so then, so there  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
are lots of blocks in fast.ai for different kinds 
of inputs so we said “ah, the input is an image!;  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
that function. In this case we'll get a list of 
image files. Something we'll talk about shortly is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
called a “validation set.” It's so critical that 
fast.ai won't let you train a model without one,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
of a photo”, how do we know if it's a bird photo 
or a forest photo? and this is another function,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
which is something that can do thousands of things 
at the same time and that means it needs thousands  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
of things to do at the same time so a data loader 
will feed the training algorithm with a “bunch”  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
is going to be the actual neural network function 
that you want to pass in, and as I said there's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
is that we integrate a wonderful library by Ross 
Wightman called “timm” (the pytorch image models)  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
which is the largest collection of computer vision 
models in the world, and at this point fast.ai is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
the first and only framework to integrate this. 
So you can use any one of the pytorch image models  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
And if we click on here, you can get lots and lots 
of information about all the different models that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
What's that? well, the reason we can do this so 
fast is, because somebody else has already trained  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
What about other kinds of models? right? There's 
a lot more in the world than just computer vision.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
and classified every pixel of, every one of these 
images according to what that pixel is a pixel of,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
I trained in about 20 seconds using a tiny, tiny, 
tiny, amount of data. So you know, again, like,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
20 seconds of training to solve it, not amazingly 
well, but pretty well. If I trained it for another  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
two minutes it'd probably be close to perfect. So 
this is called “segmentation”. Now you'll see that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
are actually going to look quite familiar. In fact 
in this case we're using an even simpler approach.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
approach that you can take to handling almost any 
kind of data, but for the kinds of data that occur  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
you can see the basic information we're providing 
is very very similar regardless of whether we're  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
fast.ai for all the, kind of common data sets that 
you might want to use; you know all the ones that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
but this time it's tabular data loaders, but we 
provide pretty similar kind of information to what  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
for your data, regardless of what kind of data it 
is. So, if you call show_batch() on something you  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
So for a table it shows you on the information in 
that table. This particular data set is a data set  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
or which users used which products, and then we 
use that to guess what other products those users  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
or whatever, it'll ask you first, like, “what's 
a few pieces of music you like,” and you tell it,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
and then fine-tune. Now, again, you know we don't 
really need to fine-tune here because there's not  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
or “fit_one_cycle”, but actually fine_tune() works 
fine as well. So, after we train it for a while,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
that error goes down and down, and for any kind of 
fast.ai model you can always call show_results() ,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
and get something sensible. So in this case it's 
going to show a few examples of users and movies.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
Here's the actual rating that user gave that movie 
and here's the rating that the model predicted.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
and in your notebook it gives you an extra little 
thing on the side where you say which things are  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
fastbook repo, you can read the whole book, and 
because it's all in notebooks, every time we say:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
recommendation systems, playing games, robotics… 
I mean it's just… I've tried… I've tried elsewhere  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
Yes, deep learning is incredibly powerful now, 
but it's taken decades of work. This was the first  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
neural network. Remember, neural networks are the 
basis of deep learning. So this was back in 1957.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
So, let's start with kind of a graph of, like, 
well, what does a normal program look like? right?  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
so, in the pre deep learning, and machine learning 
days, well, you know, you'd have… you still have  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
is random – says “this isn't a bird,” and actually 
it IS a bird, we would say: “oh you're wrong!” So  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
puts them through a model and gives us results. It 
looks exactly like our original idea of a program  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
and that's why we can do what I described earlier, 
that is once we've got that learn.predict for our  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
bird recognizer we can insert it into any piece 
of computer code, right? Once we've got a trained  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
details but the basic idea in your code is that 
you're just going to have a line of code that says  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
for those of you that are already familiar with 
notebooks and Python, there's... you know... this  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
familiar with and some slightly new libraries. 
For those of you who are not familiar with Python,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
but we do have links to great python resources in 
the forum. So check out... check out that thread.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
run. You could try changing things a little bit. 
I'd really love you to try doing the... the bird  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
sure you get something finished before the next 
lesson. Most importantly, read chapter one of the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
book. It's got much the same stuff that we've seen 
today but presented in a slightly different way.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
It's been really cool to watch people's journeys 
and some of them are just plain fun, you know. So  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
using exactly this kind of work. We've already 
actually seen this example. Ethan Sutin the... the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
dataset's website and found that he beat the state 
of the art for that. Alena Harley did tumor-normal  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
different really interesting pieces of cancer work 
during that first course if I remember correctly.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
Anyway there's been lots and lots of examples. So 
all of this is to say, you know, have... have a go  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: 8SF_h3xF3cE
c3:
at you know, starting something. Create something 
you think would be fun or or interesting and share  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
you know… stuff that hasn't really been in courses 
like this before. So, I'm super super excited.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
similar things in different directions… but, like, 
read the book as well, and remember the book is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
entirely available for free as well. You can go 
to the fast.ai fastbook repo to see the notebooks,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
in the course like, you know, stuff I find pretty 
interesting about the history of neural networks,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
the book, and they're really nicely curated, and 
put together. So check out aiqquizzes.com as well.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
links to all the notebooks, and kaggle stuff, and 
all that stuff. You'll also find on forums.fast.ai  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
already, after just a few days since I recorded it 
— we haven't even launched the course so it's just  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
the people doing it live — there's already a lot 
of replies, and that can get pretty overwhelming.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
So one of the cool things about this week is I… 
as promised, put up the “show us what you've made”  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
ones to share because they're all so good, so I've 
actually decided to kind of, you know, go the easy  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
route, and I just picked the firsts. So I'm just 
going to show you the first ones that were posted  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
the creator, said about this, is that… you know, 
wow, it's a bit uncomfortable to run this code, I  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
don't really understand yet but I'm just doing it. 
And so I'm like, yeah, good on you Matt for just…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
mainly because it's very easy to get from bird to 
beard by just changing one letter to two, and this  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
a few years now, I believe. And so you know, one 
day you too might be able to create your very own  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
well… you've kinda done step one, right? Step one 
is… step one, two, three, four… is to figure out  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
data cleaning, and if you go to Chapter 2 of the 
book, which I'm going to go ahead and open up now… 


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
course.fast.ai will run you through exactly how to 
run these notebooks, and so you can see Chapter 2  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
is this one here called collapsible headings, and 
that's the one which gives me these nice little  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
website, and getting that sorted out. ddg doesn't! 
So it's basically exactly the same… (and you can…)  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
we can search for things, and so in the book we 
did a bear detector because at the time I wrote it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
is that if you put a double question mark next to 
any function name you'll actually get the whole  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
Now, I know that's going to sound really backwards 
to what you've probably heard a thousand times,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
model. But I'm going to show you something really 
amazing. First we're going to train a model and  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
you'll see why in a moment. So to train a model, 
just like before, we use a DataBlock to grab our  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
make sure we can see the whole picture. Same one 
here. This one was out of the picture, we squished  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
zeros, which is black, you can see here now we've 
got the whole image and the correct aspect ratio.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
is RandomResizedCrop. So instead of saying resize, 
we can say RandomResizedCrop. And if we do that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
you'd grab different bits of the image, and this 
is very nice because you could run lots and lots  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
this idea of getting different pictures each time 
from the same image is called “data augmentation.”


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
which you'll probably want to do most of the time 
unless you've got a super easy problem to solve,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and the answer is “no we're not copying the image. 
What happens is that image… so each epoch, every  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
image gets read. And what happens here is though… 
is… is kind of… in memory, in RAM, this… the image  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
it's a real time process that's happening during 
model training. So, there's no copies being stored  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
because that's what the model ends up seeing. So, 
I hope that makes sense Alex, and everybody else,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
transforms, so that we can get at DataLoaders from 
that and then we can go ahead and train our model.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
It takes about a minute. In this case we only did 
four epochs of fine tuning, and we'll talk about  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
probably didn't really need RandomResizedCrop and 
aug_transforms because there's so few epochs, but  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
And what it says is how/what category errors are 
you making. And so this is showing that the model  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and it thought it was a black bear, and there was 
two times when there was actually a black bear,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
well what are the hard ones? right? So for example 
if you use the pets data set that we quite often  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
of pets, you know, really shows you which ones are 
difficult to identify, and I've actually gone in,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
it tells us the places where the “loss” is the 
highest. Now if you remember from the last lesson,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
I don't know enough about bears to know whether 
the model made a mistake or whether this actually  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
right? Now, you'll notice a couple here... it's 
got grizzly/grizzly, teddy/teddy… they're actually  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
wasn't very confident. It was only 66% confident, 
right? So here's a teddy… it's only 72% confident  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
“learn”, right? And I mentioned that I don't know 
much about black bears and grizzly bears but I do  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
know a lot about teddy bears, so I'll pick teddy 
bears. And if I click teddy bears it's now showing  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
bears and here's what's really important: they're 
ordered by loss, so they're ordered by confidence,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
This one maybe is just a bit hard to see from the 
background, but these, otherwise, they look fine…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
I don't know why it had trouble with that one but 
so be it and we'll have a quick scroll through…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
on your computer your normal RAM doesn't really 
get filled up, because if you use up too much RAM,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
you know, as you go along. Or you can just stop, 
try, stop, try. The approach I really like — and a  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
without touching the keyboard, to get a sense of 
what the video is about, and then go back to the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
actually something that I think most of you are 
probably going to find a lot more useful nowadays,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
really everything we're going to cover today. So 
there's a link to that from the forum and from the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
you can do for free, that's the good news. Okay, 
so, give it a name, just create something minimal.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
As I say there's a few different products you can 
use with it, we're going to use Gradio, also free.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
very useful tool. I'm not going to talk about it 
in detail, but let's kind of quickly learn about  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
which is actually pretty great and even people 
who use Git through the console should probably be  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
considering using Github Desktop as well, because 
some things are just much faster and easier in it,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
in fact I was talking to my friend Hamil today and 
I was like, oh, help, I've accidentally committed  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
use Github Desktop and you can just click on it.” 
Oh! That's a great idea. So that's… that's useful.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
so in fact every time you see me using a terminal, 
I'm actually using that Windows Terminal, works  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
you've already got a shell set up. In Windows, you 
almost certainly want to use Ubuntu. So Windows,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
believe it or not, can actually run a full Linux 
environment and to do it, is right typing a single  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
line, which is this… So, if you go to (just google 
for WSL install) run PowerShell as administrator,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
data scientists may be less familiar with. So you 
know, don't be worried about the terminal, you're  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
And, I particularly say, like for me, I choose to 
use Windows and that's because I get, you know,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
complete Linux environment as well. And that Linux 
environment uses my GPU and everything, so for me,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
or in the terminal on Mac, or whatever? Well, all 
you do in Windows, if you just type “explorer.exe  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
Now, Kaggle already has fastai installed but I 
always put this first, just to make sure we've got  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
And so, I trained it, it took… you know, the GPUs 
on Kaggle are a bit slower than most modern GPUs  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
but they're still fast enough, I mean, it takes 
five minutes. And there's one bit at the end here,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
So you do that on Kaggle, or on Colab. So here's 
exactly the same thing on Colab… you can see “pip  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
install”, is_cat, our data, ImageDataLoaders, so 
I've got a show_batch here as well, just for fun.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and here it is: model.pickle, right? So now I can 
go ahead and download that and that will then be  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and I pin it to my Quick Access and that way I can 
always jump in Windows straight to my Linux files.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
Let's do… so, I created a Space called testing and 
I downloaded my “model.pkl” and I pasted it into  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
the exact same is_cat definition we had before. 
That's important, any external functions that you  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
training a learner, we use load_learner(). We pass 
in the file name that we saved, and that returns  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
here we are, here's Colab, right? We've just been 
training a learner, so at the end of that, there's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
If you — in Jupyter things that start with percent 
are called magics, they're special Jupyter things,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and then the probability that it's a dog, and the 
probability that it's a cat. So the probability of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
so here's our function. So we're going to call 
predict, and that returns as we said three things:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
is… I showed my six-year-old daughter, she's like: 
“what are you doing dad?”, like, I'm coding, “what  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
dog mixed with a cat called. Like, there's no such 
thing as a dog mixed with a cat. Anyway, she goes  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
to the images tab, and finds this picture, and 
she's like: “look there's a dog mixed with a cat”.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and then — so they're the steps, right? And then, 
at the very bottom here, I've imported something  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
How does it know to call it app.py? That's because 
there's a special thing at the top default export:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
just push it to Git. So… you can either do it with 
Visual Studio Code, or you can type “git commit”.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
“what's the difference between a pytorch model and 
a fastai learner?” Okay, that's fine, we'll get to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
then you could use more data augmentation, and you 
could train it for longer. If you train for long  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
enough, as we'll learn about soon, and then… maybe 
the next lesson. If you train for long enough,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
why. So basically yeah, you can train until it's 
good enough or until you've run out of patience,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
that's how you actually grab your model. So as you 
can see, the step where you actually need a GPU,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
talk about in future lessons. And then you can do 
everything else on your own computer, including  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
for that, unless you're doing thousands of them. 
Okay, here it goes, now it's asking me to save it.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and, as you'll see, it brings this up saying: okay 
I'm ready to save this for you, so I click clone,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
Now, if you've got Linux or Mac you've already got 
python on your machine. Don't use that python!!  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
And the reason is because that python, it's called 
the system python, it's used by your computer to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and all the libraries you want, and, by far, the 
best one for you is almost certainly going to be  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
Yeah, okay, we actually want to do “conda install 
-c fastchan fastai”. So let's just copy and paste.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
you can use pip otherwise. Still a little bit of 
red. Please let us know how we can help you again.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
mamba is just a much faster version and we'll keep 
some notes on the course website because at the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
moment they're actually working on including the 
speed ups from mamba into conda. So at some point  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
there's actually, you know, a reasonable amount 
of flexibility, around like how you can make your  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
that does the actual model predictions for us is 
going to be handled by HuggingFace Spaces Gradio,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
because I just write any code and I can do it and 
they'll be excited.” And a lot of data scientists  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
mean obviously one option here is just to kind of 
say, “hey I've got a model” and throw it over to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
the wall to your mate who does know Javascript and 
say please create a Javascript interface for me.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
a website. It's a start right, and up here I've 
got some examples, here we go, single file, click,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
Oh! And there it is! Now not very amazing right, 
but the fact is that this is now a Javascript App  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
read function says create a file reader, read the 
file and when you finished loading, call loaded,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
and then set the HTML. So as you can see it's like 
(okay) if you haven't used javascript before these  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
so we can now click a few… so we've got a newfie, 
a ragdoll, a basset hound, and some kind of cat.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
So again, like here you can kind of see how (oh) a 
basset hound’s ears must be cleaned inside and out  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
frequently. So this is like combining models.So 
you can see this is something that you couldn't do  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
create those, and how do you make a website out 
of them? Well watch this! Let's… here's…. here's….  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
And so once it's done you'll basically end up with 
this empty site which, again, you just go “code”  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
“open with github desktop” or “open with visual 
studio” whatever… so “open with github desktop”...  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
something else I did then, I was like, okay, well… 
that's all very well that fast.ai has created this  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
So once you fork, one thing you have to do, which 
normally FastPages does for you, is you do have to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
go to “Settings”, and click “Pages”, and actually 
enable “Github Pages”. So you basically have to,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
the config.yml file to point at a different theme. 
And so, if you look at now… the jph's Tiny Pets…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
So, anyway, look, it's a very brief taste of this 
kind of world of Javascript and websites and so  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: F4tvM4Vb3A0
c3:
forth, but I wanted to give you a sense of, like, 
you know, you don't need any money. You don't need  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
quick survey this week to see how people feel that 
the course is tracking and over half of you think  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
it's about the right pace, and of the rest, some 
of you think it's a bit slow and some of you think  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
it's a bit… sorry… think it's a bit slow and some 
of you think it's a bit fast. So, hopefully that's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
for anybody who's already familiar with the basic 
technology pieces, and then the later lessons get,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
you know, more into some of the foundations. Today 
we're going to be talking about things like matrix  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
as well. The last bit of Lesson Zero is about how 
to set up a Linux box from scratch — which you can  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
happily skip over unless that’s of interest — but 
the rest of it is full of juicy information that I  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
without really knowing where it's heading, if that 
makes sense. And the idea of writing the notebook  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
is, you know, there's a few notebooks you could go 
through, so obviously there's the book! So going  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
quite a stretch goal, particularly at the start of 
the course, because there's so many new concepts  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
to some extent, with others — for most people. You 
know, the research shows that you're more likely  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
to stick with things if you're doing it as kind of 
a bit of a social activity. There are… the forums  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
of fantastic activity. I can't show all of it so, 
what I did was, I used the summary functionality  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
more of him if that's what you need, or maybe less 
of him. I thought this one was very interesting.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
And I thought this redaction detector project was 
really… was really cool as well – and there's a  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
which is, I was playing a little bit with how do 
you make your neural network more accurate during  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Now, because I put this out on HuggingFace Spaces, 
you can download and look at my code, because,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
if you just click files and versions on the space 
— which you can find a link on the forum and the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
make it particularly good and they just… they just 
finished. So, I've been using it all week and it's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
situation — you can do really everything kind of 
graphically. There's a file browser, so here you  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
to muck around with saving things to Google Drive 
and stuff. On Kaggle there isn't really a way of,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
you to understand was the concept. There's really 
two pieces: there's the training piece, and at the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
you generally don't need a GPU once you've got 
that trained. And so then there's a separate step,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
be doing the inference in production; one is the 
one where I train the model. So this first bit I'm  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
going to skip over because you've seen it before, 
I create my ImageDataLoaders, check that my data  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
There are, I think, at the moment — in the Pytorch 
image models libraries — over 500 architectures  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Most of the time those details don't matter, what 
we care about is three things: How fast are they?  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
ones up here and find out some of these better 
models. The ones that seem to be the most accurate  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and fast are these LeVit models, so I tried them 
out on my pets and I found that they didn't work  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
down to 5.5 percent. So, you know, that's a pretty 
big relative difference... 7.2 divided by 5.5...  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
pretty hard to beat this by much. If any of you 
find you can do better, I'd love to hear about it.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
tells you… well… like so. So here's that list of 
categories and here's the probability of each one.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So from there, just like last week, we can go and 
create our interface and then… and then launch it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
basically what you'll find is: it contains lots 
of layers, because this is a deep learning model,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and then the TimmBody contains something called 
“model”, and it can then contain something called  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So what is all this stuff? Well, let's take a look 
at one of them. So to take a look at one of them,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
it's basically how we can create kind of typeset 
mathematical equations. Okay, so let's run that.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and at some particular point x, it's going to be a 
x squared plus b x plus c. And so let's test that.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and you pass in the function, and then you pass in 
the values that you want to fix. So for example,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
I get the exact same value I did before. Okay, so, 
we've now got an ability to create any quadratic  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
equation we want by passing in the parameters of 
the coefficients of the quadratic — and that gives  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
us a function that we can then just call as just 
like any normal function, so that only needs one  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So if we plot that function, we'll get exactly the 
same shape, because it's the same coefficients.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
it's going to have some noise. So here's a couple 
of functions to add some noise. So you can see,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
so — as I say — the details of that don't matter 
too much, the main thing to know is we've got some  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
one at a time. Try increasing a little bit, see if 
that improves it. Try decreasing it a little bit,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
And actually you can see that's not bad, because 
I know the answer's meant to be three, two, one.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So we need something better. Well the first step 
is, we need a better idea of like, when I move it,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
One… we need some… something we can measure, which 
is a number that tells us how good is our model.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
which is going to be… so it's going to get in our 
predictions, and it's got the actuals. And we're  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
the values that we predict, f(x); remember where 
“f” is the quadratic we created and the actuals…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
still manual, but at least we can, like, we don't 
have to rely on us to kind of recognize: does it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
b and c, and pass each of them to the function. So 
we've now got a quadratic with those coefficients  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
just work with numbers, it also works with lists 
or vectors of numbers… that's called a 1d-tensor.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
them all in a single 1d-tensor. Now, 1d-tensor is 
also known as a rank-1-tensor. So this is a rank  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So here is our tensor, it contains 1.5 three times 
and it also tells us it's… we flagged it to say,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and not surprisingly it's the same number we saw 
before: 11.46. Okay, not very exciting, but there  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
is one thing that's very exciting, which is added 
an extra thing to the end called grad function,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
create — calculate — the gradients for our inputs. 
And to tell Pytorch: yes please, go ahead and do  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
it nothing happens, or it does look like nothing 
happens… But what does happen is: it's just added  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
this tells me that if I increase a, the loss will 
go down. If I increase b, the loss will go down a  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
right? So that means we should increase a, b and 
c. Well, how much by? Well, given that a is… says,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
this parameter, the loss decreases. So in other 
words, we want to adjust our parameters a, b and c  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
bigger than before, because we subtracted negative 
numbers. And we can now calculate the loss again.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Here it's being used in a function, but we don't 
want the derivative of this, this is not our loss,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Pytorch loop, and every neural net deep learning 
machine… pretty much every machine learning model,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
our quadratic. Call backward and then subtract the 
gradient times the small number from the gradient.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Okay, so that's how those work. Now, why is this 
interesting? Well, it's not interesting of itself,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So there's some slope m1, b1 – some second slope 
m2, b2 – we're gonna calculate it at some point x.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and b1 is going to change its position, okay. And 
I'm sure you won't be surprised to hear that m2  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and if we add as many ReLUs together as we want, 
then we can have an arbitrarily squiggly function,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
deep learning, there's nothing between these two 
steps. When you have ReLUs getting added together,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and gradient descent to optimize the parameters, 
and samples of inputs and outputs that you want,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
all very important, but when you come down to like 
trying to understand something in deep learning,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
just try to keep coming back to remind yourself of 
what it's doing, which it's using gradient descent  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
for what I'm doing? Do I need it faster, for what 
I'm doing? Could I accept some trade-off to make  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
you've trained the model, but as you've seen, it 
only takes a few minutes to train a quick model.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
more data. But also what do you mean when you say 
“get more data.” Do you mean more labeled data?  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
For example, in medical imaging — where I spent 
a lot of time — it's generally super easy to jump  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
into the radiology archive and grab more CT scans. 
But it might be very difficult and expensive to,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
we're at 1.5; so if we increase from 1.5 to 2.5, 
what would happen to the loss? And the answer is:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
it's kind of like, in an infinitely small space, 
right? …because actually it's going to be curved,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
optimal value, all functions look like quadratics 
anyway, all right, so we can kind of safely draw  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Now we haven't picked any yet in any of the stuff 
we've done (should I remember) and that's because  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
intuition here. If you pick a learning rate that's 
too big, you'll jump too far and so you'll end up  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
way over here, and then you will try to then jump 
back again, and you'll jump too far the other way,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
finding the right learning rate is a compromise 
between the speed at which you find the answer and  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Okay, hope you had a good rest, have a good break, 
I should say. So I want to now show you, a really  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
m * x + b. And we want… don't just want… to do m 
* x + b. We're going to want to have, like, lots  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
of variables. So, for example, every single pixel 
of an image would be a separate variable, so we're  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
going to multiply every single one of those, times 
some coefficient, and then add them all together,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
at some point did matrix multiplication at high 
school. I suspect also a lot of you have forgotten  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
the third is “2 3 1” - so those are our three rows 
of data. These are our three sets of coefficients,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
set of coefficients “a b c” “2 6 1” and then our 
second set is “5 7 8”. So here's what happens when  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
add, multiply, add, multiply, add. So that's going 
to give you the first number because that is the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
one with the top row and the left hand one with 
the second row. Keep going down, keep going down,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and that's it!! That's what matrix multiplication 
is - it's multiplying things together and adding  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So the GPUs that we use… the thing that they are 
good at is this, matrix multiplication. They have  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
special cores called tensor-cores, which we can 
basically only do one thing, which is to multiply  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
ongoing competition, so 14 000 people have entered 
it, or 14000 teams have entered it so far. It's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
just a competition for a bit of fun, there's no 
end date, and the data for it, is the data about,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
clearly were not going to be important things, 
like the name of the passengers, the passenger id,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
just to try to make it a bit simpler. And so I've 
ended up with, each row of this is one passenger.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So you should always look for a data dictionary, 
right? To find out what's… what… number of parents  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Male and Female were very easy. I created a column 
called isMale, and as you can see there's just an  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
IF statement that says, if sex is male then it's 
one, otherwise it's zero. And we can do something  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
but it's not really… it's not really a continuous 
measurement of something. There isn't one or two  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
they second class? Okay, so that's all that. The 
other thing that I was thinking, well, you know,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
our fares, and ages, and so forth, what happened 
was that these numbers here, are way bigger than,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
you know, these numbers here. And so, in the end 
all that mattered was, what was their fare. That’s  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
from zero to one. These numbers were too big. So 
what I did up here is, I just grabbed the maximum  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
of this column, the maximum of all the fares is 
$512. And so then… actually I do age first… I did  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
that way, all of these are between zero and one. 
Just like all of these are between zero and one.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
seven dollars. Generally speaking when you have 
lots of really big numbers, and a few small ones…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
so generally speaking when you've got a few really 
big numbers and lots of really small numbers, this  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
is really common with… with… with money. You know, 
because money kind of follows this relationship  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
where a few people have lots of it, and they spend 
huge amounts of it, and most people don't have  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
how's that working? Well there's a nice trick that 
we pretty much always use in machine learning,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
then that parameter becomes your constant term. 
So you don't have to have a special constant term,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
right? Now if you've done regression before, you 
might have learned about it as something you kind  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
of solve with various matrix things. But in fact 
you can solve a regression using gradient descent.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
been a bit easier to think about, but SUM is going 
to give us the same result. So here's our loss,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
a few issues here, which is like sometimes it's 
predicting ~(less than one, sorry) less than zero,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
and that's an example of some of the things we're 
going to learn about, that make this stuff work  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
a little bit better, right? But you can see it's 
doing an okay job. So this is not deep learning,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
easier if we used matrix multiplication, so let's 
finally do that. So this next one is going to be  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
exactly the same as the last one, but with matrix 
multiplication. So all our data looks the same.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
the X and Y is, kind of, the opposite way around. 
The rows and columns are the opposite way around.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
operation. So i still find it helpful to kind of 
remind myself, it’s just doing these sum products,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Okay. So we're teaching it in a very different way 
in the course, to what's in the book, and you know  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
encode the input with the original column? Yes! 
That's exactly the reason. So, there's no need to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
So what we're going to be doing in our next lesson 
is looking at natural language processing. So far  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
prose, so like, plain English text. So, you know, 
English is the most common language used for NLP,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
you'll find that for many languages, there's less 
resources in non-English languages, and there's a  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
great opportunity to provide NLP resources in your 
language. This has actually been one of the things  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
word “document”, it sounds like that's a specific 
kind of length but it can be a very short thing  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
Something like figuring out what author wrote the 
document would be an example of a classification  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
fast.ai has a very layered architecture so you can 
do things at a very high level with very little  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
intentionally use a library which is a little bit 
less user-friendly in order to see, kind of, what  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
of fast.ai alumni, so they have very high quality 
people working on it. So, before the next lesson,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
whether two concepts are referring to the same 
thing or not, where those concepts are represented  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
as English text. And when you think about it, that 
is a classification task, because the document is,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
you know, basically text one blah, text two blah. 
And then the category is similar or not-similar.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: hBBOjCiFcuo
c3:
by step through this notebook. And we're going to 
take advantage of that, as an opportunity also,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: toUgBQv1BT8
c3:
for NLP at the time was developed called Transformers.
And Transformers were particularly built because


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: toUgBQv1BT8
c3:
And Kaggle competitions are interesting, particularly
the ones that are not playground competitions,


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: toUgBQv1BT8
c3:
dictionary which says: “oh, which model
uses which tokenizer?”. So when we say “AutoTokenizer.from_pretrained”


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: toUgBQv1BT8
c3:
Test and validation sets are all about identifying
and controlling for something called “overfitting”


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: toUgBQv1BT8
c3:
Another thing I'll mention, is that scikit-learn
conveniently offers something called train_test_split,


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: toUgBQv1BT8
c3:
to do classification as we've discussed, of
sequences, so if we call AutoModelForSequenceClassification,


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: toUgBQv1BT8
c3:
you pass in one label to AutoModelForSequenceClassification,
it turns it into a regression problem which


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
these networks actually work. Last week we saw how 
to use a slightly lower level library than fastai,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
being Hugging Face Transformers, to train a pretty 
nice NLP model. And today we're going to be going  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
which if you remember back a couple of weeks, is 
the data set that we looked at on Microsoft Excel.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Okay, so I've opened up the clean version of the 
linear model and neural net from scratch notebook.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
through the clean notebook and before you run each 
cell try to think about like oh what did jeremy  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
I find it really nice to be able to use the same 
notebook both on Kaggle and off Kaggle – so most  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
checks whether we're on Kaggle. So Kaggle sets an 
environment variable, so we can just check for it,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
otherwise if I haven't downloaded the data before, 
then I need to download it and unzip it. Okay so  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
you can use your console, pip install and whatever 
you want to install. Or, as we've seen before,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Most of the time we're going to be using at least 
Pytorch and Numpy, so we import those so they're  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
which is really annoying, because it always wraps 
things. So if you want to put these three lines as  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
value is nan. So ‘nan’ is ‘not a number’ and the 
most common reason you get that is because it was  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
just because in Excel it's a little bit harder to 
work with. In pandas it's very easy to work with.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
a reduction on that and it reduces each column. 
So now we've got the total and in Python, Pandas,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
rows, it's nearly always empty. ‘Age’ is empty a 
bit of the time. ‘Embarked’ is almost never empty.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
I’m just not going to mention it at all, because 
I'm assuming that you'll google it. So it's really  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
important to know, for example, if you don't know 
what “iloc” is: that's not a problem, it doesn't  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
and so if I run that. Now if I call .isna.sum() 
they’re all 0. So that's like the world's simplest  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Yeah, so I would certainly never throw them out, 
right? There's just no reason to throw away data  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
for everything that's got missing values, which is 
boolean, which is: did that column have a missing  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
nice to know like a few quick things you can do 
to get a picture of what's happening in your data  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
is describe(). And so describe; you can say: okay, 
describe all the numeric variables, and that gives  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
me a quick sense of what's going on here; so we 
can see ‘Survived’ clearly is just zeros and ones  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
each amount of ‘Fare’, how often does that ‘Fare’ 
appear and it shows me here that the vast majority  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Linear models is certainly one of them and neural 
nets are generally better behaved without them as  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
things that kind of can grow exponentially, you 
very often want to take the log of. So if you have  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
and we can just go get_dummies() a column that 
says, for example, is ‘Sex’ female, is ‘Sex’ male,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
ways we can do this: one is that for n… an n level 
categorical variable, we could use n-1 levels.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
I probably should have automated that, but never 
mind, I just copied and posted them. And so here  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
‘Name’, obviously, because we'd have 891 columns 
and all of them would be unique. So we'll ignore  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
we can use our domain expertise to recognize that 
things like “Mr” have meaning and that people with  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
But that's not something I'm going to do, I'll let 
you look at those notebooks if you're interested  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
In fact this idea originally in computer science 
came from a notation developed in the 50s called  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
APL, which was turned into a programming language 
in the 60s by a guy called Ken Iverson. And Ken  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Iverson actually came up with this idea from, he 
said, his time doing tensor analysis in Physics.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
and we'll call that tensor our dependent variable. 
That's the thing we're trying to predict. Okay,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
is called its rank. That's the rank of the tensor. 
It's the number of dimensions or axes that it has.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
to need one coefficient for each column. Now in 
Excel we also had a constant, but in our case now,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Instead they can calculate a sequence of numbers 
that behave in a random-like way. That's actually  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
I’ll mention in passing, a lot of people are very, 
very into reproducible results. They think it's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
part of understanding your data is understanding 
how much it varies from run to run. So if I'm not  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
teaching, and wanting to be able to write things 
about these pseudo-random numbers, I almost never  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
use a manual seed. Instead, I like to run things 
a few times, and get an intuitive sense of like…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
“oh, this is like… very very stable” or “oh, this 
is all over the place”. I'm getting an intuitive  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
that work? This is using the incredibly powerful 
technique of broadcasting, and broadcasting again  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
coefficients times row one, coefficients times row 
two, and so forth. Which is exactly what we want.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Also, that broadcasting all happened in optimized 
C code, and if, in fact, it's being done on a GPU,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
has a very large magnitude and all the other ones 
are small same with row two, same with row three,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
It's not the end of the world but it's not ideal, 
right?, because it means that a coefficient of,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
axes or dimensions, Pytorch calls some dimensions. 
So that's going to give us the maximum of each row  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
and that would give us predictions. Now obviously 
just like in Excel, when we started out, they're  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
not useful predictions, because they're random 
coefficients, but they are predictions nonetheless  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
And often, stuff like this, you'll see people will 
use pre-written, mean absolute error functions,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
create coefficients, run a linear model, and get 
its loss. So, what I like to do in my notebooks,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
like not just for teaching but all the time, is to 
like do everything step by step manually and then  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
do them in a different notebook, or they're like 
working in an IDE, they'll go and do it in some,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
you know, line oriented REPL or whatever but if, 
you know, think about the benefits of keeping it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
and how we got there, or if you're showing it to 
your boss or your colleague you can see, you know,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
exactly what's happening, what does each step look 
like, I think this is really very helpful indeed.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
from scratch that the one bit we don't want to do 
from scratch is calculating derivatives because  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
it's just manual and boring so, to get Pytorch to 
do it for us you have to say: well, what things do  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
right?, but it also changes them in place. So now 
we've got exactly the same numbers as before but  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
remember back to the gradient descent from scratch 
notebook– we need to subtract the coefficients  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
gradient times the learning rate; and so the loss 
has gone down, that's great, from 0.54 to 0.52.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
many years ago, that's basically what we did –if I 
remember correctly–. So we can split them randomly  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
to be seeing how to do more stuff in fast.ai so i 
want to make sure we have exactly the same split.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
so just copying and pasting stuff into cells here. 
And then the bit on the very top of the previous  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
by updating coefficients, so we can put that into, 
them together into something that trains the model  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
trained a linear model on a real data set –I mean 
it's a Kaggle data set– but it's important to like  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
data set so it's not like anybody actually cares 
about predicting who survived the titanic because  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
and missing values and normalization and so forth. 
So, you know, it's a good, it's a good playground.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
them along the way, the function itself in my, you 
know, for me, doesn't need extensive comments. And  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
I'd rather explain the thinking of how I got there 
and show examples of how to use it and so forth.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
didn't - zero. It would be nice if we had a way to 
automatically squish everything between zero and  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
This should be a much easier thing to optimize and 
ensures that all of our values are in a sensible  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
range. Now, here's another cool thing about using 
Jupyter plus Python. Python is a dynamic language.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
that train_model() will eventually call. So if I 
now call train_model(), that's actually going to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
chuck it through a sigmoid. Generally speaking, 
if you're wondering why is my model with a binary  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
dependent variable not training very well, this is 
the thing you want to check: “Oh are you chucking  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
target that you're trying to hit. So those are the 
things that you actually need to know about. So,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
for example, this thing of, like, well, you need 
to know about the sigmoid function and you need to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
know that you need to use it. fast.ai is very good 
at handling this for you. That's why we haven't  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
had to talk about it much until now. If you say, 
“Oh, it's a category block dependent variable, you  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
what happens while using test data. I have a new 
category - let's say male, female, and other - and  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
use fast.ai. So fast.ai always creates an extra 
category called other. And at test time… inference  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
time… if you have some level that didn't exist 
before, we put it into the other category for you.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
be matched to a column. So yeah. It's a good point 
and definitely worth checking. For categorical  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
my training dataframe, and stuck them here for 
the test data frame and the normalization as well.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
rows. And if you look at the Kaggle competition's 
data page, you'll see this is what the submission  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
use this character. As far as I know it's pretty 
arbitrary. It's one of the ones that wasn't used.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
our coeffs now to make a proper neural net. We 
need one set of coeffs here, and so here they are,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
getting everything working properly. But you don't 
have to worry about feeling right and creating  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
it's going to be much bigger than it was before. 
So it's all very fiddly. So then I want to take…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
of survival. So again, torch.rand(), and this time 
the n_hidden will be the number of coefficients by  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
one. And again, like trying to find something that 
actually trains properly required me some fiddling  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
around, to figure out how much to subtract and I 
found if I subtract 0.3, I could get it to train.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
So we're going to pass in all of our coefficients. 
So a nice thing in Python is, if you've got a list  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
going to be a list of three things. So we'll call 
them L1, layer 1, layer 2, and the constant term,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
with commas between them like this, it creates a 
tuple. A tuple is a list. It's an immutable list.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
through our second matrix-multiply. So our second 
layer, and add the constant term. And remember,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
worked. Like, it was super fiddly. But you know, 
we got there. We got there. It's a very small test  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
set. I don't know if this is necessarily better 
or worse than the linear model, but it's certainly  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
not just neural nets. So we better make this deep 
learning. So, this one only has one hidden layer.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
can just initialize as many layers as we want, of 
coefficients. So the first one… the first layer,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
working through these matrix multipliers on, like, 
a spreadsheet or a piece of paper or something,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Why not the last layer? Because remember, the last 
layer has Sigmoid(). So these things about, like…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
activation function torch.sigmoid(), and F.relu(). 
They're the activation functions for these layers.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
or kind of variants of architectures, is to mess 
up their final activation function, and that makes  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
and generally, like, the more features you have, 
and the more levels in your categorical features,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
But yeah, I definitely would say a… an insight 
here is that, you know, you want to include simple  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
and normalization stuff, and so forth. Because we 
can do it for you. And it's not like everything's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
that's why we're going to look at this - “Why you 
should use… use a framework” notebook. And again,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
I'm going to look at the clean version of it. And 
again, in the clean version of it, step one is to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
want to do any feature engineering, because every 
column I added was another thing I had to think  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
again, like, take this out of a function, put them 
into individual cells, run each one, look up the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
“Pclass” was a number, but I'm telling fastai to 
treat it as categorical. Here's all the continuous  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
That's it! That's all the pre-processing I need to 
do, even with all those extra engineered features.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
we did in our final example. What learning rate 
should I use? Make a suggestion for me, please. So  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
learning rate that's very very small, 10 to the 
negative seven, and it puts in one batch of data,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
higher learning rates. And it keeps track of the 
loss as it increases the learning rate. Just one  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
improves faster until you make the learning rate 
so big that it overshoots, and then it kills it.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
you get a pretty good learning rate. So here we've 
got about .01, and about 0.08, so I picked 0.03.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
models. That's just a coincidence, right, I mean 
there's nothing particularly about that accuracy.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
but all the pre-processing, we just have to use 
this one function called test_dl(). And that says  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
And so that will give us the average prediction of 
our five models. And again we can turn that into  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Yeah, okay. So that one, actually finally gets 
into the top 20… 25 in the competition. So I mean,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
actual probable… probability predictions, and take 
the average of those, and then threshold that.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
2000s. I was like, everybody kind of knew me as Mr 
Random Forests for… for years. I implemented them,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
of my early Kaggle results from random forests. 
I love them, and I think hopefully you'll see why  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
the entire thing falls apart. Random forests, I… 
it's very rare to… that I've seen somebody screw  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
dataframe… dataframe and the test dataframe. Split 
things into set… into categories and continuous.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Well that's interesting. It looks exactly the same 
as before. Male and Female. But now it's called  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
look at the histogram. How many of them are there? 
About twice as many males as females. Consider  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
That wouldn't be bad, would it? Because there's a 
big difference between the males and the females,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
If you're a woman, you probably survived or… not 
just a man or a boy, so male or female. That would  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
be a pretty good model, because it's done a good 
job of splitting it into two groups that have very  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
model. So if we created a model which just looked 
at Sex, how good would it be? So, to figure that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
independent variables, which we'll call the “xs”, 
and the dependent variable, which we'll call y.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
all the people that didn't survive, this is their 
median Fare here, and then this is their quartiles  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
Fare because it's continuous. We could bucket it 
into groups to create a histogram. So I guess we  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
can create a histogram. That wasn't true. What I 
should say is, we could create something better,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
of the two groups of the dependent variable. And 
then if there's a really small standard deviation  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
is right-hand side. And then we'll just take the 
average of that. So for example, if we split by 6,


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
For example, if we wanted to find what's the best 
split point for Age, then we just have to create…


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
we made a binary split on that level of Age. So 
here's a list of all of the possible binary split  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
which is, find a single binary split, in short, 
is a type of model. It has a name. It's called 1R.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: _rXzeWq4C6w
c3:
assume that you have to go complicated. It's not 
a bad idea to always start creating a baseline of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
machine learning algorithm from scratch, this one 
is about the world's simplest one, it's “OneR”,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
So, to do that we can repeat the exact same piece 
of code we just did but let's remove sex from it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
And so this is going to be like a “OneR” rule for 
how do we predict which males survive the Titanic.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
And let's have a look: 38, 37, 38, 38, 38. Okay, 
so it's ‘Age’, were they greater than or less than  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
And you can see here it's going to first of all 
split on sex, now, it looks a bit weird to say sex  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
is less than or equal to 0.5 but remember what our 
binary characteristics are coded as zero or one,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
So you can see you can kind of get a quick summary 
of what's going on and one of the reasons people  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
who survived or just people who didn't survive the 
probability would be one, you get the same time,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
So that's why we just, yeah, that's where this 
formula comes from in the binary case. And in fact  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
that in each of these, see how it says “samples”, 
which in this case is passengers on the Titanic,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
depth, right? So this group here, which is males 
who had cheaper fares and who were older than 20  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
And so yeah so I submitted that and I got to score 
a little bit worse than most of our linear models  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
you know... for… for “class” you know it's... it's 
one two or three, you can just split on one, two,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
or three, you know. Even for like, what was that 
thing... like the... the “embarkation city code”,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
like we just convert them kind of arbitrarily to 
numbers one, two, and three, and you can split on  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
yeah... you can generally get away with not doing 
stuff like dummy variables. In fact, even taking  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
splitting on log fare less than 2.7 is exactly the 
same as splitting on Fair is less than e to the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
2.7 you know or whatever log base we used, I can't 
remember. So all that a decision tree cares about  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
but we turned it into a categorical variable and 
so pandas for us creates this this vocab this list  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
one group and Q into the other we would need two 
binary splits first C on one side and Q and S on  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
predictions. So like you do have, like sometimes 
it can take a little bit more messing around but  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
most of the time I find categorical variables work 
fine as numeric in decision tree based approaches  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
I mean, we could, but, you know, there's only 50 
samples in these leaves, right? it's not really…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
great model, but it predicts the correct thing on 
average, it's not completely hopeless and again,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
some errors a bit high, some bit low, whatever. So 
what would happen if I averaged their predictions?  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
predictions of these multiple uncorrelated models 
—each of which is unbiased— will be the correct  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
prediction because they have an error of zero. And 
this is a mind-blowing insight, it says that if  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
let's just grab a different subset of the data 
each time, let's just grab at random half the rows  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
but it will be unbiased, it will be predicting the 
average on average, it will certainly be better  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
than nothing because it's using, you know, some 
real data to try and create a real decision tree.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
of data— so let's say we put 75% of the data in 
each time —or we could change it to 50%— whatever,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
than decision trees, this is just a small data set 
to show you what to do. One of my favorite things  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
about Random Forests is we can do something quite 
cool with it, what we can do is we can look at the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and we can see what columns did it find a split 
on and so, say here, okay well the first thing it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
by the number of samples as well. ‘LogFare’, how 
much does that improve Gini and we can keep track:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
the distribution of your data and they can handle 
categorical variables and stuff like that, that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and Random Forests, you know, for most data sets, 
only take a few seconds to train, you know, really  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
most of a minute or two. And so, if you've got a 
big data set and, you know, hundreds of columns:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
So I've done that, for example, I did some work in 
credit scoring, so we're trying to find out which  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
things would predict who's going to default on a 
loan and I was given something like seven thousand  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
columns from the database, now I put it straight 
into a Random Forest and found, I think, there was  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
about 30 columns that seemed kind of interesting. 
I did that like two hours after I started the job  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
how much does the accuracy improve. So I then did 
the same thing on our data set, so I actually just  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
hand wavy theory I gave you— that you'd expect the 
more trees, the lower the error, because the more  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
things you're averaging and that's exactly what we 
find: the accuracy improves as we have more trees.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
tree is the kind of thing you can really do fast, 
in fact, it's it's quite easy to like literally  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
spit out C++ code with a bunch of if statements 
and compile it and get extremely fast performance.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
so that means for each tree there was 25% of the 
data we didn't train on. Now this actually means  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
if you don't have much data, in some situations 
you can get away with not having a validation set,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and the reason why is because for each tree we 
can pick the 25% of rows that weren't in that tree  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and see how accurate that tree was on those rows 
and we can average for each row their accuracy on  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and the second part is: could we create a bagging 
model which includes fast.ai deep learning models?


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
a meta method, it's not a prediction, it's not a 
method of modeling itself, it's just a method of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
thought of me as the Random Forest guy. I used to 
go on about Random Forests all the time and one of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
the reasons I'm so enthused about them isn't just 
that they're very accurate or (~that they require,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
you know…) that they're very hard to mess up and 
require very little (~processing…) pre-processing,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
they'll repay but we're not confident of that we 
would probably want to give them less of a loan.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
for example, for credit… let's say you rejected 
that person's loan… “Why?...” And a Random Forest  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
will tell us “What what is the… what is the reason 
that we made a prediction.” And you'll see why,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
characteristics that you're not allowed to use in 
your model. So they might check things like that.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
deviation is high, that means all of them (all of 
the trees) are predicting something different!!  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
And that suggests that we don't really know what 
we're doing. And so that would happen if different  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
equipment like bulldozers - this is specifically 
the blue books for bulldozers Kaggle competition.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
Now you might think that you could easily create 
this plot by simply looking at your data at each  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
equipment more expensive to buy. And most things 
didn't have air conditioning back in the 60s and  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
we take every single row - and for the year made 
column we set it to 1950. And so then we predict  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
And that does exactly what I just said, remember 
I said the special words, “all else being equal…”  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
through the tree to get to the leaf node?” And we 
can say, oh well, first of all it looked at sex,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
then we could do the same in Tree #2, same in Tree 
#3, same in Tree #4… Does this sound familiar?  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
Reds says this made our prediction go down. Green 
made our prediction go up. And so overall you can  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
John: All right, thank you. Zakia followed up with 
the question about bagging and we're just going,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
you know, layers and layers here, you know: we 
could go on and create ensembles of bagged models?  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
a wider range of other kinds of models, you could 
have like Neural Nets trained on different subsets  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
Forest models sometimes has different results when 
you compare it to other explainability techniques  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
pretty good at, you know, doing, you know, if it's 
like a computer vision model Random Forest aren’t  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
certainly you know adding more trees won't make it 
worse. If you don't have enough trees and you let  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
more about these columns, and say like: “oh! well 
that one there we've actually got a better version  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
Normalization you don't have to worry about, you 
know you don't have to have normally distributed  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
another tree, which attempts to predict that —a 
very small tree— and then you create another very  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
small tree which tries to predict the error from 
that and so forth, right? Each one is predicting  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
the residual from all of the previous ones. And so 
then to calculate a prediction, rather than taking  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
go-to thing. Having said that there are ways to 
avoid overfitting, but yeah, (it's just, it's not,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
“oh, that's interesting”, like, because you never 
quite have a sense of how well these things work.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
And then I thought: “well there's all these other 
things we should be doing as well”, and I tried  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
leaderboard. So I thought it'd be cool to take you 
through the process. I'm going to do it reasonably  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
you can ask it to make sure that pip things are 
installed —that might not be up to date otherwise—  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
some work project or something, you might be able 
to convince yourself and everybody around you that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and doesn't overfit is important. And furthermore 
structuring your code and analysis in such a way  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
more of a tangled mess of impossible to understand 
code and having no idea what UntitledCopy13 was,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
should work, you know, in other settings as well. 
And so these are the two focuses that I recommend:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
Get a really good validation set together - we've 
talked about that before, right, and in a Kaggle  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
competition (that's like…) it's very rare to see 
people do well in a Kaggle competition who don't  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
because the the the test set seems to be a random 
sample, but most of the time it's not actually,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
I would say. And then how quickly can you iterate? 
How quickly can you try things and find out what  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
in machine learning in general, you actually have 
to do everything reasonably well. And doing just  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
one thing really well will still put you somewhere 
about last place. So I actually saw that a couple  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
You know, when I share a notebook I like to be 
able to say “as you can see this is 0.83 blah blah  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
blah right and know that when you see it, it'll be 
0.83 as well. But when I'm doing stuff otherwise,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
I would never set a random seed. I want to be able 
to run things multiple times and see how much it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
get_image_files()... so that gets us a list of 
the file names of all the images here recursively,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
but my fastcore library has a parallel sub module 
which can basically do anything that you can do  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
opposed to cropping randomly a section out. So if 
we call Resize() ‘squish’ it will squish it down,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
So I jump into creating models much more quickly 
than most people, because I find… model, you know,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
So that means that I'm going to need a model that 
can train quickly. So Thomas Capell and I recently  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
did this big project “The Best Vision Models For 
Fine Tuning” where we looked at nearly a hundred  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
we measured were how much memory did it use, how 
accurate was it, and how long did it take to fit.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
learning about exactly how a resnet26d is made and 
convolutions and resonant blocks and transformers  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
anything, and at some point the learning rate is 
high enough that the loss will start coming down,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
be able to iterate, rapidly, just a minute or so. 
So that's enough time for me to go and you know,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
So this is a dataloader which we can use for our 
test set. A test dataloader has a key difference  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
rice disease for every class, it'll tell you what 
is the index of the most probable rice disease,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
pandas user you've probably seen map used before 
being passed a function —which is really, really  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
mental energy, so even submitting to Kaggle needs 
to be fast. So I put it into a cell, so I can just  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
run this cell: api.competition_submit_cli(), this 
CSV file, give it a description, so just run the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
cell and it submits to Kaggle. And as you can see 
it says: here we go!, “successfully submitted”.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
you get for entering a competition, but this time 
rather than finding out, in no uncertain terms,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
you know, “first steps in the road to the top part 
one dash copy one”, you know, and that's okay.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
or something. But yeah, it's like, it's a very low 
tech approach that I find works really well: which  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
is just duplicating notebooks and editing them and 
naming them carefully and putting them in order  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
what it is… generally speaking from me, you know, 
my notebooks will only have one submission in and  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
for example, “does whether we squish or crop make 
a difference?” you know, “are some models better  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
But this fantastic researcher named Leslie Smith 
invented the learning rate finder a few years ago,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
computer vision Deep Learning model and which one 
I would use, well, if I'm transfer learning —which  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and then the same thing for planet is: “which ones 
are best for fine tuning for two data sets that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
are very different on what they're trained on”? 
And it happens in both case they're very similar,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
probably going to give me a slightly better result 
if I need it, and can be bothered fussing around


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
were ending up with only 128 pixel size bits for 
speed. So there's no point doing that every epoch  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and so, when I then, trained on that, on Kaggle, 
it went down to four times faster with no loss of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
Oh, by the way, this graph is very useful, so this 
is… this is speed versus error rate by family.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
Here it is convnext_tiny. So we were looking at 
resnet26d, which took this long on this data set,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
still spending all of our time waiting for the CPU 
anyway. So it turned out that when I switched my  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
So yeah… so I started out by running the resnet26d 
on the resized images and got similar error rate,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
but I ran a few more epochs, got 12% error rate. 
And so then I do exactly the same thing but with  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
you can tell them, we've actually… we've moved on, 
you know. Resnets is still probably the fastest  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and then like most things there's different sizes 
- there's a tiny, there's a small, there's a base,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
so if I remove the “method=squish” that will crop. 
So you see how I've tried to get everything into  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
a single function, right, the single function. I 
can tell it… (let's go and find the definition…)  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
Instead of cropping, we can pad. Now padding is 
interesting… do you see how these are all square,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
so padding is interesting because it's the only 
way of pre-processing images which doesn't distort  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
pointless - they contain zeros. So every way of 
getting this working has its compromises but this  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
quite well - in this case it was about as good as 
our best so far, but no, not huge differences yet.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
What else could we do? Well, what we could do is… 
see these pictures? This is all the same picture  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
And so this is the same model as before, which had 
a 4.5%, so if instead if we get TTA predictions  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and then get the error rate, wait why does it say 
4.8?... last time I did this it was way better.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
check that out, and I'll try and come back to you 
and find out why in this case this one was worse.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
now that we've got a pretty good way of resizing, 
we've got TTA, we've got a good training process,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
type of computer vision competition - they'd have 
computer vision data set almost. But you know, it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
and make it an array so that's going to be a list 
of 10 things and then we can index into that vocab  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
with our indices which is kind of weird this is a 
list of 10 things this is a list of… I don't know  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
vocab item for that thing? So this is another way 
of doing the same mapping, and I would spend time  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
playing with this code to understand what it does, 
because it's the kind of like very fast, what,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
the CPU very very well.This is the kind of coding 
you want to get used to, this kind of indexing.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
John: Um yeah, there's there's two I think that 
would be good if we could touch on quickly before  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
On Fast Ai – and as I say I don't think anybody 
else does this as far as I know – if you call TTA,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
it will use the exact same augmentation approach 
on whatever data set you pass it, and average out  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
so you don't have to do anything different, but if 
you didn't have any data augmentation in training  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
aspect ratios, so you know if there's just as many 
tall skinny ones as wide short ones you know you  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
doesn't make sense to create a rectangle because 
some of them… you're going to really destroy them,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
not 89, 99 percent of people, given a wide variety 
of aspect ratios, chuck everything into a square.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
you know so the issue with with padding as you say 
is that you're putting you know black pixels there  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: AdhG64NF76E
c3:
still by default uses a reflection padding which 
means if this is… I don't know, let's say this is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
What you'll discover if you've tried to use larger 
models… so you can replace the word small with the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
it runs out of memory. And it also doesn't do such 
a good job as your CPU at kind of shuffling things  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
around to try and find memory, it just allocates 
blocks of memory and it stays allocated until you  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
remove them. So if you try to scale up your models 
to bigger models, unless you have very expensive  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
GPUs, you will run out of space, and you'll get an 
error, something like “CUDA out of memory error.”  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
Almost it's, you know, basically you'll be able to 
use a x-large model on Kaggle. So, let me explain.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
your home computer or Paperspace or whatever, but 
sometimes you'll have — if you want to do Kaggle  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
they call “Code Competitions” which is where the 
only way to submit is from a notebook that you're  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
running on Kaggle, and then a second reason to run 
stuff on kaggle is that, you know, your notebooks  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
will appear, you know, with the leaderboard score 
on them, and so people can see which notebooks are  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
actually good. And I kind of like, even in things 
that aren’t code competitions, I love trying to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
all about fast iteration. So I wanted to really 
quickly find out how much memory will a model use,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
problem is that now, the dynamics of my training 
are different, right? The smaller your batch size,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
messed up. You don't want to be messing around 
with trying to, you know, find a different set of,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
How do we make it behave as if it was 64 images? 
Well, the solution to that problem is to consider  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
how many images are in this batch. So initially I 
start at zero, and this count is going to be 32,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
and does it in a mathematically slightly incorrect 
way. As a result of which, if you've got batch  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
thought about it that deeply. In practice I found 
adding gradient accumulation for Convnext has not  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
then it's gonna be… yeah yeah, probably yeah, you 
can probably tell I didn't actually run this code.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
different architectures have different amounts of 
memory, you know, which they which they take up,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
thats working really well for batch size 64 won't 
necessarily work really well for batch size 32.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
performance reasons I think it's generally a good 
idea to have it be a multiple of eight. Everybody  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
JEREMY: Yeah so generally speaking the rule of 
thumb is that if you divide the batch size by two,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
got something called a callback, and a callback is 
something which changes the way the model trains.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
vit_large, swinv2_large, swin_large, and on each 
of these I just tried running it with accum=1, and  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
actually every single time for all of these I got 
an “out of memory” error. And then I tried each of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
them independently with accum=2, and it turns out 
that all of these worked with accum=2, and it only  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
took me 12 seconds each time, so that was a very 
quick thing for me. Then okay, I now know how to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
which means that, well, most Transformers models, 
nearly all of them, have a fixed size – this one's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
There are… there is a way of working around this 
but I haven't experimented with it enough to know  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
So with this dictionary of architectures, and for 
each architecture, kind of pre-processing details,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
otherwise I find what happens is your GPU memory 
kind of, I don't know, I think it gets fragmented  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
So this way you can really do as much as you like 
without running out of memory. So they all train,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
my data loaders does not have the seed= parameter, 
so I'm using a different training set every time.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
each of these different runs they're using also 
different validation sets, so they're not directly  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
comparable, but you can kind of see they're all 
doing pretty well, 2.1%, 2.3%, 1.7%, and so forth.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
models. These are not intentionally weak models, 
they're meant to be good models, but they're all  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
different – they're using different architectures, 
and different pre-processing approaches, and so  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
So we can stack up that list of different… of all 
the different probabilities, and take their mean,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
good intuition of what's going to help, right. So 
partly I'm showing you this to say, it's not like  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
purely random as to whether things work or don't, 
once you've been doing this for a while, you know,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
you will generally be improving things most of the 
time. So as you can see from the descriptions my  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
training subsets. And then this is the same thing 
again, this is the thing we just saw, basically  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
So what I've done here is I've trained a bunch of 
models with different training sets, each one is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
but what it says is rather than picking, like say 
five samples out with different random subsets,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
of which have non-overlapping validation sets, 
and then you'll ensemble those. You know in theory  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
those five validation sets because there's no kind 
of overlap between them to get a cross validation.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
By definition you had to do it because your GPU 
is full so you're obviously giving a lot of data.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
So it's probably going to be using its processing 
speed pretty effectively, so yeah, no it's just…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
it's just a good technique that… we should all be 
buying cheaper graphics cards with less memory in  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
them, and using you know, have like… I don't know 
the prices, I suspect it's like you could probably  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
JEREMY: No, not really. I mean obviously at the 
moment Nvidia is the only game in town, you know.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
If you buy… if you trying to use a you know Apple 
M1 or M2 or an AMD card you're basically in for  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
enterprise cards, and the reason is that there's a 
licensing issue that Nvidia will not allow you to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
use an RTX consumer card in a data center – which 
is also why cloud computing is more expensive than  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
for kind of the same features. So yeah, if you do 
get serious about deep learning to the point that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
ImageDataLoader.from_folder(). So that's like the 
highest level API, the simplest API. If we go down  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
a level deeper into the DataBlock we have a lot 
more flexibility. So if you've been following the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
replicate the previous notebook but replace the 
ImageDataloader.from_folder() with a DataBlock to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
You can put in as many as you like, so if you put 
in three things we're going to generate one image,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
a single thing such as the parent_label function 
which looks at the name of the parent directory,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
So that's actually all we need to do to define a 
metric that's going to work with our new dataset…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
dependent variable was a single category, and it 
knows the best loss function that's probably going  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
to be the case for things with a single category, 
and it knows how big the category is. So it just  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
a... kind of a... mini imagenet thing where you're 
trying to predict whether something... an image,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
and so forth. Now if you think about it, since the 
denominator adds up all the e-to-the-power-ofs,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
then when we do each one divided by the sum, that 
means the sum of these will equal 1 by definition,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
really try to pick one thing” as having most of 
the probability, because we are trying to predict,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
let's say, is it a teddy bear or a grizzly bear or 
a black bear, and they feed it a picture of a cat,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
like, there's something else you could do which 
is you could actually have them not add up to one,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
It's actually the log of the softmax but don't 
worry about that too much, it's just a... slightly  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
we've got our five probabilities, and then we've 
got our actuals. Now the truth is the actual, you  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
turned out to be the number one, but what we tend 
to do is we think of it as being one-hot-encoded,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
a smaller loss if the softmax was high where the 
actual is high. And so here's how we calculate...  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
the five categories so M is 5, and for each one we 
multiply the actual target value, so that's zero…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
pytorch prefers to use log softmax because then it 
kind of skips over having to do this log at all.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
little bit tricky to kind of say, oh look this up 
in an array, which is basically what it's doing,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
fastaI to know what loss function to use, because 
we've got multiple targets, so you have to say:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
because again it… there's more than one target, 
so you have to say the number of outputs to create  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
And once we've done that we can train it, and we 
get, you know, basically the same kind of result  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
And the trick is actually very simple, and you 
might have almost got the idea of it when I talked  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
you kind of think through the manual neural nets 
we've created, this loss function will be reduced  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
the combined_loss – and as the metrics, our list 
of all the metrics and n_out=20 and now look what  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
happens when we train! As well as telling us the 
overall train and valid loss, it also tells us the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
that sounds weird, right, because we're trying to 
do more stuff, [and] that's the same size model.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
Well the reason is that quite often it'll turn out 
that the kinds of features that help you recognize  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
do better than a single target model at predicting 
disease, but I just want to let you know sometimes  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
where we tried to predict a second thing… (I can't 
even remember what it was, maybe it was a type of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
So there's at least, you know, there's two reasons 
to learn about multi-target models: one is that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
like actually look at the inputs and look at the 
outputs and look at the data loaders and so forth.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
“oh I can't think of any way to improve this.” So 
I thought I'll just leave it as is, but we have  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
this one's LA Confidential. And then this rating 
says how did Jeremy feel about Return Of The Jedi,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
So these are the first few rows. This is not a 
particularly great way to see it. I prefer to kind  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
why there's so few empty cells here, I actually 
grabbed the most watched movies and the most movie  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
three categories, being science fiction, action, 
or kind of classic old movies, would say The Last  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
eight (0.98), plus point eight (0.8) times point 
nine (0.9), plus negative point six (-0.6) times  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
and we're going to do the same thing for movie 49 
–pick another five random numbers– and the same  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
thing for movie 57 –pick another five numbers. And 
you might not be surprised to hear we're going to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
do the same thing for each user, so for user 14: 
we're going to pick five random numbers for them,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
feels not very strongly about the fact that for 
movie 27 has a value of 0.71. So therefore in here  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
you might notice that the matrix product of a row 
with a column is the same thing as a dot product  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
that gives us the dot product. So here's the dot 
product of that by that –or the matrix multiply,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
that is the mean squared error –and if we take the 
square root, that is the root mean squared error–  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
since they have the same feeling about these three 
movies, maybe they'll feel the same about these  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
of three of them so, you know, you kind of… this 
is the idea, right?, as if somebody says to you:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
“I like this movie, this movie, this movie” and 
you're like: “oh, they like those movies too” what  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
that you're going to like the same thing. That's 
the basis of collaborative filtering, okay, it's…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
and mathematically we call this matrix completion. 
So this matrix is missing values, we just want to…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
Pytorch has it. Okay. So to do that in Excel the 
first thing I'm going to do is I'm going to, see,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
one row, and so you can see here it is 0.19, 0.63, 
0.19, 0.63 et cetera, right? So here's the second  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
that's going to be: 0.75, 0.47 et cetera. And so 
same thing, right?, but now we're going to offset  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
from here, by 14, to get this row which is 0.75, 
0.47 et cetera. And so the prediction now is… the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
Embedding actually means: “look something up in 
an array”. So there's a lot of things that we use,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
then we've now got the user id and the actual name 
of the movie –we don't need that, obviously, for  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
user column and an item column. So the user column 
is what it sounds like: the person that is rating  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
this thing, and the item column is the product or 
service that they're rating. In our case the user  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
and the item column will be called “item”. Give it 
a batch size, and as usual we can call show batch,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
is appropriate for different size categories on a 
piece of paper in a table –well actually in Excel–  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
lots of papers will be like: “using fastai's rule 
of thumb for embedding sizes… here's the formula…”


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
Look, this has returned 0.87 –and particularly if 
I take the negative off here, if I add this up–  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
is called when you create an object of this class. 
So we could pass it a value, and so now we set the  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
attribute called “a equal to that value”, and so 
then later on we could call a method called “say”,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
this is really important to know, Pytorch is going 
to call a method called “forward” in your class.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
and it's going to be passed the object itself, 
and the thing you're calculating on. In this case,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
the user and movie for a batch. So this is your 
batch of data, each row will be one user and movie  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
do you remember when we learned about Sigmoid. 
The idea of squishing things between zero and one.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
Now if you think about it, if you take something 
and put it through a Sigmoid, and then multiply it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
through to 5.5. so it can't be smaller than zero, 
can't be bigger than 5.5. Why didn't I use five?  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
but some people do give things... movies a five so 
you want to make it a bit bigger than our highest.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
our kind of formulation of this model to say this 
user tends to give low scores and this user tends  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
embedding of n_movies also by 1 and so we can now 
look up the user embedding the movie embedding,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
the sum of the weights squared. This is something 
you should go back and add to your titanic model,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
if our weights are all zero, our model doesn't do 
anything, right? So we'd have crappy predictions.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
But if it increases the weights too much then it 
starts overfitting. So how is it going to actually  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
because remember the whole purpose of the loss is 
to take its gradient, right? And to print it out.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
like tabular and collaborative filtering, we don't 
really know enough about your data to know what  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
to use here. So you should just try a few things. 
Let’s... try a few multiples of 10. Start at 0.1  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
weights, the more they're moving the model around, 
right? So we want to keep the weights down but not  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
so far down that they don't make good predictions 
and so the value of this if it's higher, will keep  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
the weights down more, it will reduce overfitting 
but it will also reduce the capacity of your model  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
around so Zakia if you want to maybe clarify in 
the... in the chat as well but “can recommendation  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
systems be built based on average ratings of users 
experience rather than collaborative filtering?”


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: p4ZZq0736Po
c3:
like... they like this movie, this movie, and this 
movie therefore they might also like that movie  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
our coefficients, or parameters, or weights are - 
Pytorch does that for us. And the way it does that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
so here is a class we've created called “T” which 
is a subclass of module and I've created one thing  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
just contains three ones. And so the idea is, you 
know, maybe we're creating a module and this is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
thing almost doesn't really do anything. In fact 
last time I checked it really quite literally had  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
in this Parameter class. So, if we do exactly the 
same thing as before, which is to set an attribute  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
as we did here. And so how, yeah, what is… what is 
happening behind the scenes, you know, in Pytorch?  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
And there it is: it's training. So we'll be able 
to have a look when this is done, at for example…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
classic kind of film noir detective movie with the 
aussie Guy Pierce, even if you don't really like  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
classic kind of… I guess you'd say like, horror 
kind of… not horror is it, a suspense movie… even  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
the bias… oh and by the way, we could do the same 
thing with users and find out, like, which user  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
movie factors. Which has got the number of movies 
as one axis and the number of factors as the other  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and we passed in 50 – what's in that huge matrix? 
Well, pretty hard to visualize such a huge matrix  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
movies. And over here on the left, we've got more 
of the kind of critically acclaimed gritty kind  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
of movies. And then towards the top we've got very 
kind of action-oriented and sci-fi movies and then  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
like those kind of movies. And “Titanic” is right 
up there as well. Even if you don't really like  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
do now is we can look through all of the movies 
in our latent factors and calculate how far apart  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
movie is to the “Silence of the Lambs” on, based 
on these latent factors. And so then we can find  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
do that matrix completion exercise using all those 
dot products. There's a different way however,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
can… we could basically create our user and item 
embeddings as per usual and then we could create  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
a sequential model. So a sequential model is just 
layers of a Deep Learning Neural Network in order.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and then do a reLU. So this is… this is basically 
a single Hidden Layer Neural Network, and then a  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
linear layer at the end to create a single output. 
So this is the very, you know, world's most simple  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
In practice, nowadays, a lot of companies kind of 
combine…they kind of create a combined model that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
your users, like: when did they sign up; how old 
are they; what sex are they; you know, where are  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
top ones seem to be anime. And so you can imagine 
what's happening in the matrix completion exercise  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
a unique id, just arbitrarily, well actually in 
this case it's alphabetical order, but it doesn't  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
have to be. And so we kind of talked about that 
and that's what we do with categories in general,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
And so, this is what an embedding matrix looks 
like in an NLP model and it works exactly the same  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
way, as you can see. And then you can do exactly 
the same things in terms of interpretation of an  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
can actually show you this, right? So we're going 
to use “tabular_learner” to create the “learner”  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and so “tabular_learner” is one, two, three, four, 
five, six, seven, eight, nine lines of code. And  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
So a “TabularModel” is a module, we're going to be 
passing in how big is each embedding going to be,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
So then it's going to create a layer, for each of 
the layers we want, which is going to contain a  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
“SigmoidRange” we've talked about at the very end. 
And so the “forward()”, this is the entire thing!  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
layers being concatenated together and then going 
through a couple of layers of a neural network,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and it's showing here… it points out in the paper 
exactly what we learned in the last lesson, which  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
And, yeah, they found that their technique worked 
really well. One of the interesting things they  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
Random Forests and embeddings or Gradient Boosted 
Trees and embeddings, which is really interesting.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
Now, what I really wanted to show you though is, 
what they then did… so, as I said this was a thing  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
about the predicted amount that different products 
would sell for at different shops around Germany.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
for their German regions and when they create a 
chart of them, you can see that the locations that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
close together in the embedding matrix are the 
same locations that are close together in Germany.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
And we're going to look at it using an image from 
MNIST. So MNIST is kind of the world's most famous  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
it's a dataset of handwritten digits, each one is 
28 by 28 pixels. But it, yeah, you know, back in  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and as a result ended up with convnets being used 
in the american banking system for reading checks.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
of those ones with a stroke through it – and this 
is what it looks like. This is… this is the image.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
Now, how did I do this? I did this using something 
called a convolution and so, what we're going to  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
going to keep zooming in. So take a look.. keep 
an eye on this… on this image, and you'll see that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
once we zoom in enough, it's actually just made of 
numbers which, as we discussed in the very first…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
conditional formatting to basically make things: 
the higher numbers more red. So that's how I turn  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
And so let me show you how we went about creating 
this top edge detector. What we did was we created  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
you do a dot product of the corresponding elements 
of that 3 by 3 with the corresponding elements of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
this 3 by 3 matrix of coefficients. Now, why does 
that create something that finds, as you see, top  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
edges. Well, it's because of the particular way 
I constructed this 3 by 3 matrix. What I said was  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
The one underneath are exactly the same formulas, 
a 3 by 3 sliding thing here, but this time we've  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
done 5 by 5 and then we'd have a 5 by 5 matrix of 
coefficients –or whatever, whatever size you like.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
which is we're going to multiply each of these, by 
each of these, and sum them up. But then we do it  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
And so that gives us… and I think I just picked 
some random numbers here, right?. So this is going  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
to now be something which can combine… oh sorry, 
the set… the second one, the second set… so it's,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
back propagate through this using SGD in the usual 
way. And that is going to end up optimizing these  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and we would do something called Max Pooling. And 
Max Pooling is kind of like a convolution (it's  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
(oh that goes off the edge). And you can see what 
happens when this is called a 2 by 2 Max Pooling.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
of the number of activations we used to have. And 
that's actually a good thing because if we keep on  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
That's effectively what we used to do, but, the 
other thing I mentioned is we didn't normally keep  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
normal linear layer (and this is called the Dense 
Layer) and then we would add them all up. So we  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
So we would after doing column “I”, we would skip 
columns “J” and would go straight to column “K”  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
So that's basically, instead of doing Max Pooling. 
And then the other thing that we do differently is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
This is actually quite important to know because 
if you think about what that means, it means that  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
you know, if it's… if the bear is big and takes up 
most of the frame then most of those 7 by 7 bits  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
being a great bear detector. And so this is where, 
like, the details of how we construct our model  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
Although, you know, again like, the key thing here 
is that… this is happening right at the very end,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
in a certain way of thinking about it, it turns 
out that convolution is the same thing as a matrix  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
so forth, so we can say Q = alpha x B + beta x C 
etc. And so this is how we calculate a convolution  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
Okay, I think… well, let's look at one more thing. 
Because there was one other thing that we saw,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and I actually have this in my excel spreadsheet. 
If you go to the “conv-example (dropout)” page …  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
our filtered image. And what that means is we end 
up with exactly the same image we started with…  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
but it's data augmentation not for the inputs, but 
data augmentation for the activations. So this is  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
And you can decide how much you want to compromise 
between good generalization – so lack of… so good,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and it was rejected from the main neural networks 
conference, then called NIPS, now called NeurIPS.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
I think that's given us a good tour now, we've 
really seen quite a few ways of dealing with input  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
to a neural network, quite a few of the things 
that can happen in the middle of a neural network.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
multipliers and activation functions. The matrix 
multipliers can sometimes be special cases, such  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
as the convolutions or the embeddings. The output 
can go through some tweaking, such as the Softmax,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and then, of course, you've got the loss function 
such as cross-entropy loss, or mean squared error  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
Yeah, because, you know, you've got to this point, 
you may as well know how to get to the next point  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
and writing. So if you've kind of zipped through 
the videos on, you know, 2x and haven't done any  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
exercises, you know, go back and watch the videos 
again. You know a lot of the best students end up  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
watching them two or three times – probably more, 
like three times. And actually go through and code  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
as you watch, you know, and experiment. You know, 
write posts, blog posts, about what you're doing.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
things you could try. One of the most important 
things to do is to get together with other people,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
passionate about. Yes, it's really important to… 
to not just put the videos away, and go away and  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
which is Mish, which is now, as Tanishq has just 
reminded me on our forums, is now used in many of  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
how that's doing? Let's have a look… go to Google 
Scholar… nice, 486 citations. They're doing great.


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
Yeah, that's a very interesting question, I mean, 
I think Lucas… the important thing is to realize:  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
And yeah, from time to time, you know, take a dip 
into other areas that maybe you're not following  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
not changing that fast at all really, it can kind 
of look that way because people are always putting  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
activation functions and the stuff you do to the 
final layer, you know. Everything else is tweaks.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
I feel the field is getting more and more skewed 
towards bigger and more computationally expensive  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
models and huge amounts of data. I keep wondering 
if in some years from now I would still be able  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
to train reasonable models with a single GPU or if 
everything is going to require a compute cluster.”


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
engineers always want to push the envelope in, 
like, the… on the biggest computers they can find,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
going to get slightly better results than people 
doing exactly the same thing on smaller computers.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
the thing I try to do in my live-coding, or study 
groups, whatever, is yeah, definitely try to show  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
either a problem that you face day to day in your 
work, or in some hobby or passion that you have,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
doesn't work as well as it ought to. Particularly 
something where you think to yourself: “you know,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
the “minimum viable product”. You create something 
that solves that problem, it takes you as little  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
You know, even the bit in the middle where you're 
like: “oh there's going to be a neural net here…”  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
– which often drives my co-workers crazy because 
they're like, you know: “why aren't you focusing  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
building up this kind of exponentially improving 
base of expertise, in a wide range of areas. And  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
a whole bunch of tools and skills and ideas which 
(yeah/no) other people don't necessarily know. So  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
like I think that's one thing that's been helpful. 
And then another is, yeah, like trying to really  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
tenacity, you know, I've noticed a lot of people 
give up much earlier than I do. So yeah, if you,  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
if you just keep going until something's actually 
finished then that's going to put you in a small  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
minority, to be honest, most people don't do that. 
When I say finish, like finish something really  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
in my kind of chosen area I've spent quite a bit 
of time trying to make sure it's really easy for  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
the youtube recommendation system. And please do 
come and help other beginners on forums.fast.ai.  


c0: PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU
c1: htiNBPxcXgo
c3:
It's a great way to learn yourself… is to try to 
teach other people. And yeah, I hope you'll join  


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: _QUEXsHfsA0
c3:
of neurons, it was enough to allow any mathematical
model to be approximated to any level of accuracy,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: _QUEXsHfsA0
c3:
If you haven't used Python before - it's ridiculously
flexible and expressive and easy-to-use language.


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: _QUEXsHfsA0
c3:
So in this case, for example, to do the actual
building of the dataset, we called ImageDataLoaders.from_name_func.


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: 5L3Ao5KuCC4
c3:
this aug_transforms function. And what aug_transforms
does is it actually returns a list of different


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: 5L3Ao5KuCC4
c3:
the right part of the screen before, but this
is what I was describing in terms of the pre-formatted


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: 5L3Ao5KuCC4
c3:
it. So we need to get the average of the pixels,
so we're going to learn some nice Python programming


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: p50s63nPq9I
c3:
to list in Python, it returns all of the things
from the iterator. So here are my three mini-batches,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: p50s63nPq9I
c3:
section core data augmentation and so aug_transforms()
was the thing which can zoom in and zoom out,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: p50s63nPq9I
c3:
and might need, you know, fixing and the ‘ImageClassifierCleaner’
actually let us find things like, an image


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: krIVOb23EH8
c3:
Acts also involving accountancy and classification
by IBM machines and utilized in the concentration


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: krIVOb23EH8
c3:
of, principles that are often held cross culturally,
although, yeah, it's rare for something, probably,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: krIVOb23EH8
c3:
And he's done a ton to raise awareness about
this issue and founded the nonprofit ‘AlgoTransparency’


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: krIVOb23EH8
c3:
themes in her book, ‘Weapons of Mass Destruction’,
kind of, under the idea that the, ‘The privileged


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: krIVOb23EH8
c3:
But really disinformation is often about orchestrated
campaigns of manipulation and that it involves,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: krIVOb23EH8
c3:
of us think of ourselves as pretty independent-minded,
but discussion really does you know we evolved


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: krIVOb23EH8
c3:
And this would be (I think) virtually impossible
to catch today, if someone was using a sophisticated


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: krIVOb23EH8
c3:
Non-collapsible steering columns would frequently
impale drivers and then even after the collapsible


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: cX30jxMNBUw
c3:
have another look. Have a look at the questionnaire
particular, and make sure that you're comfortable


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: cX30jxMNBUw
c3:
thing we're going to do is look at model interpretation.
There's not much point having a model if you


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: cX30jxMNBUw
c3:
classification. So what's multi-label classification?
or maybe... so maybe let's look at an example.


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: cX30jxMNBUw
c3:
don't have to write this ourselves, of course,
because PyTorch has one and it's called �F.binary_cross_entropy�.


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: cX30jxMNBUw
c3:
the initial sigmoid, actually. If you want
to include this initial sigmoid you need F.binary_cross_entropy_with_logits


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: cX30jxMNBUw
c3:
the accuracy for all of those different thresholds,
and plot them. Ah, looks like we want a threshold


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
of big flat areas like accuracy does classification.
So what we want to do is to make it less likely


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
how they work. It's also a great way to understand
the foundations of what's going on is to actually


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
and then inside that there's going to be automatically
created for it... well sorry not automatically...


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
you can do exactly the same thing for 50 dimensions.
It doesn't just work for two dimensions. There's


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
for gradient-based methods and random forests
and decision tree growing are not really gradient-based


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
that's comma separated values -- and the training
set contains information such as unique identifier


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
in any interesting way, but if there's a particular
level which kind of stands out as being important,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
those errors is zero. So, therefore, the average
of the models should give us an accurate prediction


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
Breimen in 2001 showed a way to do this specifically
for decision trees where not only did he randomly


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
in other words this is like giving us a validation
set result without actually needing a validation.


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: VEG5xT5gAHc
c3:
lr_find and then we can go ahead and use fit_one_cycle
(there's no pre trained model obviously, because


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
sentiment, and we trained it with a very standard-looking
classifier trainer approach. But we haven't


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
pre-trained weights that when we said text_classifier_learner
were automatically loaded in. So, conceptually,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
the most commonly occurring substrings. We'll
see that in a moment. Or the simplest, character-based:


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
do it for us, rather than doing all that messing
around manually, so we can just create a DataBlock,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
here. So once we've run this we can call �show_batch�,
and so you can see here we've got for example


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
containing, say, 40 words. And so we can just
go through those two sentences and call learn.predict,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
we had an is_lm equals true when we said TextBlock.from_folder
to save as a language model. We don't have


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
only fastai we used was to create the DataLoaders(),
and it's just grabbing directly from the datasets,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
network architecture, which takes three numericalized
words at a time as input, and tries to predict


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
and out of that we're going to get some activations.
And so, circles represent computed activations,


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
arrow, and then output activations are a triangle.
So the output would be batch size by num classes.


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
embedding? They all should have the same representation,
because they alll have the same meaning. Same


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
pre-trained, right. This is not an application-specific
learner, so it wouldn't know what pre-trained


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
works shortly. Okay, so back propagation .. Oh
sorry, I was using the wrong jargon - �backpropagation


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
actually one quite popular approach to initializing
the hidden to hidden activations is to initialize


c0: PLfYUBJiXbdtRL3FMB3GoWHRI8ieU6FhfM
c1: WjnwWeGjZcM
c3:
If you call TextLearner(), instead of Learner(),
it will add the ModelReseter and the RNNRegularizer()


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
In particular, hopefully you have gone to notebook tutorial. It's at the top. That's right with zero zero here as this grows


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
This will feel a little bit more intimidating because the code that you're seeing will be unfamiliar to you. Yes, Rachel


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
That's I think this will be the only time like this in the in the lesson where we've assumed you've got this set up


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
you can go back after this and make sure that you can get this running using the information in course III go faster do


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Think works well to use these notebooks and to use this material and this is based on the kind of last three years of experience


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Don't try and follow along because it's not really designed to go to speed where you can follow along


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Take in the information you get a general sense of all of the pieces how it all fits together, right?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And also don't try and stop and understand everything the first time if you do understand everything the first time good for you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But most people don't particularly as the lessons go on they get faster and they get more difficult, okay


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Deep learning box yourself and that's something that you can do after you don't later on once you've kind of got going


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So why should you listen to me? Well, maybe you shouldn't but I'll try and justify why you should listen to me


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
I started out in management consulting where actually initially I was I think Mackenzie and company's first


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
The president of cattle but actually the thing I'm probably most proud of in my life is that I got to be the number one


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Fact to call why can you actually train a predictive model that predicts things pretty important aspect of data science?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Nowadays, I'm on the faculty at University of San Francisco and also co-founder with Rachel of fast AI


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Machine learning throughout that time and I guess I'm not really although I am at usf for the University


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Through fast AI we are trying to help people use deep learning to do useful things through creating software


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
To make deep learning easier to use at a very high level through education such as the thing you are watching now


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Through research which is where we spend a very large amount of our time, which is researching to figure out


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
easier to use at a very high level which ends up in as you'll see in the software and the education and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
By helping to build a community, which has made me through the forums. So that practitioners can find each other and work together


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Some folks whose do the two parts can spend a whole year doing it really intensively. I know some folks


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Ten hours a week or so approach for the seven weeks by the end. You will be able to build an image classification


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Text again using whatever datasets you're interested in you'll be able to make predictions of kind of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Very very high-level versions of these things. So that might surprise you that's slightly over the the prerequisite here is


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Less now than a couple of years ago than we started but a lot of naysayers telling you that you can't do it


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's not a black box as you'll see it's really great for interpretive interpreting. What's going on?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It can be used very widely for lots of different applications. Not just for vision, which is where it's most well-known


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
You don't need lots of hardware, you know that thirty-six cent and our server is more than enough to get world-class results for most problems


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's true that maybe this is not going to help you to build a sentient brain, but that's not our focus. Okay, so


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Well, this was an example from Nikhil who's actually in the audience now cuz he was in last year's course as well


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
people playing cricket and people playing baseball and around the coach will see you today and build a


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Examples like this or you can try stuff as we'll see in the workplace that could be of direct commercial value


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Many of the academic courses. So for those of you who haven't kind of an engineering or math or computer science background


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Postgraduate degree and you're finally at the point where you can build something useful. We're gonna learn to build the useful thing today


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Okay, there will be lots of aspects of what we do that you don't know why or how it works. That's okay


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
unfortunately, it's still a situation where people who are good practitioners have a really good feel for how to


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
so everybody will know that you're not a real deep learning practitioner because real deep learning practitioners know the keyboard shortcuts and the


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Going all the way up here finding your clicking at just shift enter. Ok, so type like type shift enter don't actually


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So we're going to go through this quickly. And then later on we're going to go back over it more carefully


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
These things starting with percent are special directives to Jupiter notebook itself. They're not Python code. They're called magics


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
but basically it says hey if somebody changes the underlying library code while I'm running this place reloaded automatically if


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Far far far more things that are far more easily than can with plate or alone. So it's a good combination


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Something where something is the application you want concurrently fast AI supports for applications computer vision


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Collaborative filtering and we're and we're going to see lots of examples of all of those during the seven weeks


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Feeling sick because you see me go import star which is something that you've all been told to never ever do


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Okay, and there's very good reasons to not use import star in standard production code with most libraries


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's kind of the opposite everything's there for you all the time. You don't even have to import things a lot of the time


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
You've got a scientific programming community that has one way and then you've got the software engineering community that has the other


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We actually support both approaches, you know, you put a note block where you want to be able to quickly


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
You don't want to be constantly going back up to the top and importing more stuff and trying to figure out where things are


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
When you see me doing stuff, which at your workplace is found upon. Okay, it's it's this is a different style of coding


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There are no rules in data science programming. It's that the rules are different right when you're training models


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
The most important thing is to be able to interactively experiment quickly. And so you'll see we use a lot of very different


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And you'll learn about them over time. You can choose to abuse a similar approach or not. It's entirely up to you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It designed in a very interesting modular way and you'll find over time that when you do use import star there's far less


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Clobbering of things and you might expect it's all explicitly designed to allow you to pull in things and use them quickly without having problems


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Curating and gathering a data set so that they can show how well different kinds of approaches work with that data


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So we're going to be starting with an academic data set called the pet data set the other kind of data set


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Both academic data sets and cadwal data sets are interesting for us particularly because they provide strong


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
You can actually submit your results to Carol and see how well would you have gone in that competition and if you can get in


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Models that get right up towards the top of capital competitions preferably actually in the top ten. What does the top 10%


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But if you're interested in learning more about it and why it was created and how it was created all the details


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
so in this case, this is a pretty difficult challenge the PEC datasets going to ask us to distinguish between


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We've used a different data set which is one where you just have to decide is something a dog or is it a cat?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There are lots of dog breeds and cat breeds look pretty much the same. So why have we changed that dataset?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We've got to the point now where deep wedding is so fast and so easy that the dogs versus cats problem which a few years


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
our models were basically getting everything right all the time without any tuning and so they want, you know, really a lot of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Opportunities for me to show you how to do more sophisticated stuff. So we've picked a harder problem this year


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So this is the first class where we're going to be learning how to do this difficult problem and this kind of thing


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Figuring out particular kind of pet and so the first thing we have to do is download and extract the data that we want


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We're going to be using this function called ant our data which will download it automatically and will interact automatically


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So they are download super quickly for you. And so the first question then would be how do I know what entire data?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Something you might not have seen before even if you're an experienced programmer is what exactly do you pass to it?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
These bits these bits are tight. And if you've used a tight programming language, you'll be used to seeing them


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It'll figure that out for us from the URL. So and for all the data sets, we'll be using in the course


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Since you've already downloaded it, it won't download it again since you've already untied it it won't untie or it again


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Python that are less convenient for interactive use and they should be for example when you do have a path object


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
If you're an experienced Python programmer you may not be familiar with this approach of using a splash like this now


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
This is a really convenient function. That's part of Python three its functionality from something called path Lib


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
These are path objects path objects are much better to use then strings that lets you basically create


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Alright, so if you're starting with a brand new data set trying to do some deep learning on it, what do you do?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Well, the first thing you would want to do is probably see what's in there. So we've found that these are the


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There's one called get image files that will just grab a array of all of the image files based on extension in a path


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
For image computer vision datasets to get passed around as that is just one folder with a whole bunch of files in it


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
in machine learning the labels refer to the thing we're trying to predict and if we just eyeball this


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Because that's all you need to build a deep learning model. You need see pictures so files containing the images and you need some labels


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
You need to build a model and there's basically some factory methods which try to make it really easy for you to create


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
You know re is the module in Python that does regular expressions things. That's really useful for extracting


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Super useful to be very useful to spend some time figuring out how and why that particular regular expression is going to extract


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
From this text. Okay. So with this factory method we can basically say, okay. I've got this path containing images


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
This is the regular expression pattern that is going to be used to extract the label from the file name


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Images square shapes in part two. We'll learn how to use rectangles as well. It turns out to be surprisingly


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Nuanced but pretty much everybody in pretty much all computer vision modeling nearly all of it uses this approach of square


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
The little bits of artists in the ship that I want to teach you folks, which is like what generally just works, okay


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And in fast AI everything you model with is going to be a data bunch object. We're going to learn all about them


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And what's in them and how do we look at them and so forth. They're basically a data bunch object contains


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We'll learn about this shortly. It'll contain your validation data, and optionally it contains your test data and for each of those it contains your


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Your images and your labels or your texts and your labels or your tabular data and your labels or so forth


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Normalization, but generally in all nearly all machine learning tasks. You have to make all of your data about the same size


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
This thing called transforms is is used to do a number of things and one of the things it does is to make something


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Zoomed and cropped in a reasonably nice way. So basically what it'll do is something called by default


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
so we'll talk more about the detail this because it turns out to actually be quite important but basically a combination of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So there's also all kinds of different ways and it depends on data augmentation, which we're going to learn about shortly


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Pixel values we're going to be learning more about pixel values the pixel values start out from naught to 255


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Really bright and some might tend to be really not bright at all and some might be area large and some might not very much


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Haven't studied or don't remember means and standard deviations. We'll get back to some of that later, but that's the basic idea


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
GPU man will be in power up to doesn't size 256 some more practical considering to be a little utilization


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So we actually want something where if you go seven times to a bunch of times then you end up with something


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Yeah, all of these details we are going to we are going to get to but the key thing is I wanted to get you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But you know, one of the most important things to be a really good practitioner is to be able to look at your data


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Weird black borders on earth or some of the things have text covering up some of it or some of its rotated in odd ways


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
and so here they are that's almost the possible labels that we found by using that regular expression on the file names and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
For things like regression problems and multi-label classification and stuff. That's not exactly accurate, but it will do for them


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Important piece of information that is something like or at least for classification problems. It is the number of classes


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There's one called con flora, which is something that will create a convolutional neural network for you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
What's your data and not surprisingly it takes a data bunch and the second thing you need to tell it is. What's your model?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Or what's your architecture? So as I learned there are lots of different ways of constructing a convolutional neural network


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But for now, the most important thing for you to know is that there's a particular kind of model called a res net


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
That's kind of it. That's as much as you need to know to be a pretty good practitioner about architectures for now


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
For a particular task and that particular task is that it was trained on looking at about one and a half million pictures


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
of all kinds of different things a thousand different categories of things using an image data set called image net and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So we can download those pre trained weights so that we start start with a model that knows nothing about anything


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But we actually start with a model that knows how to recognize there are thousand categories of things in image


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Or in image net but there was certainly some kinds of dog know certainly some kinds of cat. So this pre trained model


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And it certainly knows quite a lot about what animals look like and what photos look like. So the idea is that we don't start


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Automatically the first time we use it a pre trained model and then from now on it won't need to download it again


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
How to take a moral that already knows how to do something pretty well and make it so that it can do your thing


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Take a pre trained model and then we fit it so that instead of predicting Li a thousand categories of imagenet with image net data


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
1/100 or less of the data the regular model training. In fact potentially many thousands of times less


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
How do you know that it was going to actually that it can actually recognize pictures of people playing cricket versus baseball in general


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Maybe it's just cheating. Right and that's called overfitting. We'll be going talking a lot about that during this course, right?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But what a fitting is where you don't learn to recognize pictures of say cricket versus baseball, but just these particular


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
cricketers and these particular photos and these particular baseball players in these particular photos


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Like in this case error rate get printed out automatically using the validation set and sort of images that our model never got to see


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And it turned out to be dramatically better both more accurate and faster than any previous approach


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It means it could over fit but sees the same picture too many times. It'll just learn to recognize that picture not pets in general


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Starting out with four is a pretty good start just to see how it goes and you can actually see after four


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Just remember I said the nice thing about using academic papers or capital data sets is we can compare


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There'll be a section called experiments about 2/3 of the way through and if you find the section on experiments


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
You know how far we've come with deep learning and particularly with pay torch and fast AI how easy things are


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And just remember if you're in the audience and you see a question that you want asked please click them up heart next to it


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We don't exactly know what that involved or how it happened. But we do know that we're three or four lines of code


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
6% arrow certainly sounds like pretty impressive for something that can recognize different dog breeds and cat breeds


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Fell into the habit of watching the lectures too much and googling too much about concepts without running the code


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And at the very end, I started running the code and oh it turned out I learned a lot more. So please run the code


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
should have spent the majority of my time on the actual code and the notebooks running at seeing what goes in and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So your most important skills to practice our learning and we going to show you how to do this in a lot more detail


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
the reason we've been able to do this so quickly is heavily because of the fostered a library now if I stay a


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Library is pretty new but it's already getting an extraordinary amount of direction as you've seen all of the major cloud


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Software is something which is going to take you a long way and the best way to really understand the faster your software


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There's really only one major other piece of software like fast AI that is something that tries to make deep learning easy to use


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It was kind of the gold standard for making deep learning easy to use before but life is much easier with bostero


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Fast AI lets you get more much more accurate less than half the error on a validation set, of course


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Are more important than you might realize because those 31 lines of Karis code involved you making a lot of decisions


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Processing which people are calling the image net moment, which is basically we broke a new state of the art resolved in text classification


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Which open AI then built on top of our paper to do with more computing more data into different tasks to take it even further


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And like this is an example of something that we've done in the last six months in conjunction actually with my colleague Sebastian Reuter


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
An example of something that's being built in the fassio library and you're going to learn how to use this brand-new model in


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Who you'll come across on the forum plenty because he's a great guy very active built a new system for natural language


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
where you can actually type in English sentences and find snippets of codes that do the thing you asked for and again,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's being built with the FASTA a library using the techniques. You'll be learning in the next seven weeks


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
In production. Yeah. Well, I think this stage is a part of their experiments platform. So it's kind of pre-production I guess


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
where as well as categories for each part of the course, and there's also a general category for deep learning where people talk about


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Even though today, we're kind of got to focus on a small number of lines of code to a particular thing, which is image classification


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And we're not learning much math or theory or whatever over these seven weeks and then part two another seven weeks


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Economics didn't have a background in coding math computer science. I think she started learning to code two years before she took our costs


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
they helped build this amazing system where they attached old mobile phones to trees in the Kenyan rain forests and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
She's just been publishing some papers now. She is going to Africa to set up a Google brains first deep learning


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Research Center in Africa now, I'll say like she worked her ass off. You know, she really really invested in


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
and you can find her post and actually listen to her music samples of she actually built something to


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Automatically create chamber music compositions you can play and you can listen to online. And so again, it's her background


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
She's a festival pianist who also has a master's a medical researcher in Stanford and studied neuroscience


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But you know, I think it's really cool to see how I kind of a domain expert in this case the domain of playing piano


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
You know of the three top research institutes bugle playing or open a would be two of them probably along with Diamond


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
She found the most useful and we're going to be talking a lot about you doing projects and making them fantastic during this course


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
What I really want you to do is go back to your workplace or your passion project and apply these skills


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Released a deep learning course and they highlighted in their announcement for this deep learning course this medical imaging example


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
That this is what happened in your model. So alex is combining his knowledge of radiology and his knowledge of deep learning


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
practical aspects that you'll learn in this course and bring them together like Alex is doing here and so a lot of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There's a data science Institute at the ACR now and so forth and Alex is one of the people who's providing kind of a lot


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So another great example was this was Melissa fab bras who was a English literature PhD who studied like gendered language in?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Wrench over the previous job taught her to code I think and then she came into the first day a course and she helped


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Faces, why is that necessary? Well, we're going to be talking a lot about this. But because most a I


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
What is that like 30 or 40 times worse for black women versus white men? And this is really important because for chemo


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
after taking our course and again working in her ass off and being super intensive in her study and her work won this


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And I actually talked to a blind lady about these kinds of apps the other day and she confirmed to me


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Level that you can get to with with the content that you're going to get over these seven weeks and with this software


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Lots of people want to train on the imagenet dataset. We smashed the world record for how quickly you can train it


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Results and so I think this is super cool. She's not a professional artists. She is a professional software developer


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There's recently been some quite high-profile articles describing how she is creating a new form of art again. This is come out of the


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
course that she developed these skills or equally important bred counselor who figured out how to make a picture of Kanye out of pictures of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
This particular style this particular type of what's called style. Transfer was a really interesting tweak


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Which basically turned out its plant to be fantastically good at identifying fraud and we'll talk more about it shortly


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
If you've seen Silicon Valley the HBO series the the hot dog hot dog app. That's actually a real app


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
I'm like, yes, it wasn't very nominated. So I think we only have one any nominated fast day alumni at this stage


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Boring McKinsey consultant like me. It's a Francisco and I both have this shameful past that we were McKinsey consultants


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Basically as students have been coming up with new study that results across lots of different languages and this all is entirely being done


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Because remember and one of the people everybody you see on the forum the vast majority posting post all the damn time


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Intimidating because it can feel like you're the only new person there, but you're not right all of you people in the audience everybody


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Who's watching? Everybody? Who's listening? You're all new people, right? And so when you just get out there and say like


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Okay, just make sure you provide all the information. This is the you know, I'm using paper space. This was the particular instance


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Crop yield analysis and you're a farmer and you think you know, oh, I've got something to add so please


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's fine, you know just get involved and because remember everybody else in the forum's started out


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Or imagenet classification you'll see in first place in second place in third place in fourth place is faster. I


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Jeremy Hatton first a hydrometer plus the irony response from the Department of Defense innovation team


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
There are other architect is the main reason you might want a different architecture is if you want to do inch computing


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
most of the time I reckon the best way to get a model onto somebody's mobile phone is to run it on your server and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But if you really do need to run something on a low powered device, then there are some special architectures for them


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So the particular question was about inception that's a particular another architecture which tends to be pretty memory intensive


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's not terribly resilient. One of the things we try to show you is like stuff which just tends to always work


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Creating a set of weights. If you've ever done anything like a linear regression or logistic regression, you'll be familiar with coefficients


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So if we want to start doing some more playing around and come back later, we probably should save those weights


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
In the same place the data came from so if you save different models or different data bunches from different data sets


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So we've talked about how the most important things that add on learn what goes into your model what comes out


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We're going to use this factory method from learner. So we pass in a loan object. So remember a learn object from those two things


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's actually a trained model inside there and that's all the information we need to interpret that model


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But actually you were wrong then then that's going to have a high loss because you were very confident about the wrong answer


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But then tells you more information. So in this case, I saw the thing that tells me the title of eight shows


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
so the documentation always has working code and so in this case, it was trying things with handwritten digits and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Somewhat experienced Python programmer. You'll find the source code of faster. I'm really easy to read


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
How to use the faster I'm I agree because every line of code here nearly every line of code is calling stuff in the faster


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
I've got another really cool trick about the documentation that you're going to see a little bit later


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Okay, so that's how we can look at these top losses and these suppress the most important image classification


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Breeds that are actually very difficult to tell apart and you'd be able to look at these and say oh I can see why


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
another useful tool kind of is to use something called a confusion matrix, which basically shows you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Type of dog or cat. How many times was it predicted to be that dog? Okay, but unfortunately in this case because it's so accurate


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
This diagonal basically says how it's pretty much right all the time and you can see this in slightly darker ones like a five here


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So what I suggest you use is instead of if you've got lots of classes don't use a classification confusion matrix


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But this is my favorite named function in faster. I are very proud of this. You can call most confused and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Was what it should have predicted and instead it predicted an American Pitbull Terrier and so forth. It should have ridiculous


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
I mean actually predicted Burma that happened four times this particular combination happens six times


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So this is again a very useful thing because you can look and you can say like with my domain expertise


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Does it make sense that that would be something that was confused about so these are some of the kinds of tools?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
and the reason it ran pretty quickly is that there was a little trick we used these deep learning models these convolutional networks they have


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Lanes, they learned a lot about exactly what layers are but but now just know it goes through a computer


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It'll just fine tune these few extra layers add up to the end and it'll run very fast. It'll basically never over fit


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But to really get it good you have to call an crits and unfreeze is the thing that says please train


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Order to understand why we're actually going to have to learn more about exactly what's going on behind the scenes


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We're gonna start with this picture these pictures come from a fantastic paper by Nets, Iowa who nowadays is CEO of clarify


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So a convolutional neural network will learn mathematically about what the layers are shortly but the basic idea is that you're red green and blue


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
The first layer and something comes out of that and then the result of that goes into a second layer


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But that's not that layer one. There's this very simple computation. It's a convolution if you know what they are


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
What comes out of this first layer? Well, we can actually visualize these specific coefficients the specific parameters by drawing them as a picture


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So we won't draw all of them and let's just look at mine at random. So here are my examples of the actual


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So this first one basically finds groups of pixels that have a little horizontal diagonal line in this direction


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
This one finds greated to go from pink to green in this direction and so forth. That's a very very simple


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Takes the results of those filters and does a second layer of computation and it allows it to create


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
kind of a way of visualizing this one of the second layer features and you can see it's basically learned to


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We have things that can find just one line and lay it - we can find things that have two lines turned up


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
so this is the kind of stuff you've got to get a really good intuitive understanding for slightly the start of my


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Very simple gradients lines the second layer can find very simple shapes the third layer can find combinations of votes


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So now we can find repeating patterns of two-dimensional objects, or we can find kind of things that joins that join together


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Mainly they're bits of text although sometimes windows, so it's nice to be able to find kind of like four petered


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Can take all the stuff from layer three and combine them together by layer four. We put something that can find dog faces and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So you kind of get the idea. So by layer five, we've got something that can find the eyeballs of birds and wizards


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Or faces of particular breeds of dogs and so forth so you can see how by the time you get to layer 34


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And we just trained a few more layers on top of all of those sophisticated features that are already being created


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
The image net data that this is originally trained on so we don't really want to change layer one very much if at all


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
so you kind of want this intuition is understanding that the different layers of a neural network represents different levels of kind of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Just as much as it tries to update the things that represent the exact specifics of what a my ball looks like


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Learning rate finder we're learning about what that is next week, but for now just know this is the thing that figures out


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Then we can go and learn don't recorded a plot and that will plot the result of our LR finder and what this basically shows


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
you is this is T parameter that we're going to learn all about called the learning rate and the learning rate basically says


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It actually so happens. In fact, I can check this if I press shift tab here my learning rate defaults to


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But there's no point trading all the layers of that rate because we know that the latent layers work just fine


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
0.003. So what we can actually do is we can pass a range of learning rates to learn theater and we do it like this


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So let's use about what I knew for and then this one should be a value from your learning rate finder


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Maybe about here. So I picked something. That's at least ten times smaller than that. So if I do that then I get


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
But it'll in practice you'll get something that's you know about as good in practice as the vast majority of practitioners can do


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We can improve it by using more layers and we'll do this next week by basically doing a ResNet 50 instead of ResNet 34


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And you can try running this during the week if you want to you'll see it's exactly the same as before


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Therefore it has more parameters and therefore it uses more of your graphics card memory. Just totally separate to your normal computer Ram


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
and that's because it's just trying to do too much too many parameter updates for the amount of RAM you have and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Somewhere around a three percent error for something where you've got a fifty percent chance of being right and the two things work totally different


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
This case I unfroze it and fit it a little bit more than from 4.4 to 4.3 five. It's a tiny improvement


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's interesting because again, you can call the most confused here and you can see the kinds of things that it's


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
so quite often I find that rag doll and bir-men of things that it gets confused and I actually have never heard of either of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Apparently the main difference is like the particular Kennel Club guidelines as to how they are assessed


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So this is the kind of stuff. We're actually even if you're not a domain expert it helps you become one


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
He started the language to model thread and he's you know now helping to TA the costs. He's actually putting together a dye


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It will show you how to download data from Google Images so you can create your own data set to play with


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Because how to create labels in lots of different ways because your data set wherever you get it from won't necessarily


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So it was telling you how to do this. I'm going to use the feminist sample embolus is pictures of hand drawn numbers


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So basically the people that put together this data set have already decided what they want you to use as a validation set


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It's basically to say Oh everything that's a three. I'll put in a folder called three everything. That's a seven


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So if you have something in this honor where the labels just whatever the folders called you can say from folder


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And not the possibility and then you can coordinated up classes to see what them another possibility is as we've seen this you've got paths


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
That look like this and so in this case, this is the same thing. These are the folders that I could actually grab the


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So what if you it's something that's in the file name of a path, but it's not just a regular expression. It's more complex


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
That extracts a label from the file name or path and in that case you would say from name and function


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Even you need something even more flexible on there. And so you're going to write some code to create an array of labels


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So here as I've created an array of labels through my labels is from lists. Okay, and then I just pass in that break


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Every single line of code, I just showed you. I took it this morning and I copied and pasted it from the documentation


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
It turns out that all of our documentation is actually just stupid about books. So in this case I was looking at vision data


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Experiment and you'll see in in github. It doesn't quite render properly. This github doesn't quite know how to render notebooks properly


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Documentation has actual working examples in it with actual data sets that are already sitting in there in the repo for you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
We've probably what you're looking at maka GPU into your pot true. It's easy to do and you'll find it on the forum, but


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
So before I wrap up I'll just show you an example of the kind of interesting stuff that you can do by


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
I've got these fantastically good results for police overall analysis software. So it they're pleased to think


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
creatively so if you're wanting to study sounds a lot of people that study sounds do it by actually creating a spectrogram image and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Then sticking that into a confident. So there's a lot of cool stuff you can do with this. So during the week


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
Make sure that you can use lesson one and work through it and then see if you can repeat the process


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: XfoYk_Z5AkI
c3:
And if you're able to successfully build a model with a new data set let us know and I will see you next week


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
You're not going to be bugged by anybody else except stuff that we think you need to know for the course


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Everyday and that the folks involved in these platforms are testing every day and they definitely work


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
updates thread' for that lesson. So 'Lesson One Official Updates' and the same thing only Fast.ai people will be posting to that


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
You can you can watch it safely and we'll have all the things like the videos the notebooks and so forth


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
any of you that have missed some of the updates so far go back and have a look through because we're really trained to


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
One thing which can be more than a little overwhelming is even now after just one week the most popular thread has 1.1 thousand (1100) replies


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I've actually read every single one of them and I know Rachel has and I know ( ) has and I think Francisco has


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Ones will appear and then they'll be view 31 hidden replies or whatever in between. So that's how you navigate these


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
giant topics that also way it's important you click the like button because that's the thing that's going to cause people to


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So when you come back to work, hopefully you've realized by now that on the official course website, 'course.Fast.ai' v3


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
you read somewhere else - only use the commands that you read about here. And that will make everything very smooth


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
If things aren't working for you, if you get some into some kind of messy situation, which we all do,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Sharing their work and this is really cool because it's pretty intimidating to put yourself out there and say like I'm new to all this


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And so example four things. I thought was really interesting was figuring out who's talking. Is it Ben Affleck or


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I wanted to clean up while whatsapp downloaded images to get rid of memes. So I actually built a little neural network


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I mean, how cool is that to say like? Oh, yeah. I've got something that cleans up my whatsapp. It's a deep learning application


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Sounds data that was used in this paper. And in this paper, they were trying to figure out what kind of sound


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Ethan Sutan then tried using the Lesson 1 techniques and got 80.5 % accuracy. So I think this is pretty awesome


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Then take all of these with a slight grain of salt, but I've mentioned them on Twitter and lots of people on Twitter


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
follow me, so if everybody knew that there was a much better approach, I'm sure somebody would have said so.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I did' and this guy on Twitter was like, "oh I made that data set. Congratulations. You've got a new record."


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And she's got some nice pictures describing like what she wants to do with this and like how she can go about turning this into pictures


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Thought it looked to be a state of the art in this particular point mutation one as well. So that's pretty exciting


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
You know something like this one in particular is is using a lot of domain expertise like it's figuring out what picture


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Simon is one of these very annoying people like Christine Payne who talked about last week who seems to be good at everything he does


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
He created Django when it's the world's most popular web frameworks. He founded a very successful startup and bla bla bla bla bla


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So, you know, that's fine. You know Simon can go and win a hackathon on his first week of playing with deep learning


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
um, and I think like it's important to mention this because there was this really inspiring blog post this week from James Dellinger who talked


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
He said he he nearly didn't start on deep learning at all because he went through the scikit-learn website


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It's not something I understand' and then this kind of realization of like, oh I can do useful things


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
which was here saying I want to contribute to the library and I looked at the docs and I just started overwhelming and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
But I love the way it forces me to learn so much and then one day later, I just submitted my first pull request


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
you know, try and try and push a piece of code or a documentation update or create a classifier or whatever


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
dog and cat breeds thing from last week and actually doing some exploratory work to see what the main features were and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Somebody else in the forum took that and did the same thing for anime to find that they had accidentally discovered an anime haircolor classifier


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Panamanian buses correctly. Apparently these are the new ones I much prefer the old ones, but maybe that's just me


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
This was a really interesting Henri Pallacci discovered that he can recognize, with 85% accuracy, which of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Projects.  So don't worry, it's only been one week. It doesn't mean everybody has to have had a project out yet


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
A lot of the folks who already have a project out have done a previous course, so they've got a bit of a head start


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
We're then going to look at the same thing for text. We're then going to look at the same thing for tabular data


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So we're going to recommendation systems that's going to take us into a topic called embeddings, which is basically a key


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
NLP and computer vision a few weeks apart and that's going to force your brain to realize like 'oh I have to remember this'


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Hey, 'here's some code - type it in start running it approach' rather than here's lots of theory approach


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Confusing and surprising and odd at first and so for those of those of you I just wanted to remind you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Keep going.  You're not expected to remember everything. Yet.  You're not expected to understand everything. Yet.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Push on. Right?  Most of the people who have done the course and have gone on to be really successful watch the videos at least


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
the second time, then they go through it really slowly the third time and I consistently hear them say 'I get a lot more out


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
He describes this approach of the whole game, which is basically if you're teaching a kid to play soccer you don't you know


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
soccer ball with their bare hands and then teach them the mathematics of parabolas when you kick something in the air...


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Then you you know gradually over the following years learn more and more so that you can get better and better at it


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So this is kind of what we're trying to get you to do is to play soccer which in our case is to type code


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Our first notebook, which is called "lesson2-download" and what we're going to do is we're going to see how to create


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
But it'll detect whatever you like. So it'll be like those some of those examples we just saw. How would you create your own


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
inspired the approaches inspired by Adrian Rosebrock, who has a terrific website called "pyimagesearch" and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So that was definitely an inspiration for some of the techniques we use here. So, thank you to Adrian


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
She needs to know what she's dealing with.  In our house, you would be surprised at the number of monsters,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
paste in that code. So this is the JavaScript console, for those of you you haven't done any JavaScript before, I hit enter and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Black bears and for brown bears since that's a classifier I would want, and i would put each one in a file with an appropriate name.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Download those URLs to our server. Because remember, when we're using Jupyter notebooks, it's not running on our computer


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Let's start with black bears. I've already got my black bears URL, so I click on this cell for black bears and I'll run it


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
People with a more strict scientific background are horrified by.  This is not reproducible research.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I'll run this cell and so that way I'm just going backwards and forwards to download each of the classes that I want


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
My notebooks and see things where there's kind of like configuration cells doing the same thing in different places


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
experiment in my notebook - I treat it like a lab journal. I try things out and I see what happens...


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
"Wrong."  That you should only ever run things top to bottom.  Everything you do should be reproducible


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
For me, I don't think that's the best way of using human creativity. I think human creativity is best inspired


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
by trying things out, seeing what happens, and fiddling around.  So you can see how you go. See what works for you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
"max_workers = 0" and that'll do it without spitting up a bunch of processes and will tell you the errors better.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Images that aren't actually images at all. 
 And this happens all the time. There's always a few images in every batch that


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Other words I have the basic structure we need to create an image data bunch to start doing some deep learning


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
data sets.  In this case, we don't have a separate validation set because we just grab these images from Google search, right?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Otherwise, you don't know how well your model is going and we'll talk about more about this in a moment


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
the training set is in the 'current' folder because by default it looks in a folder called /train and I want you to set aside


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
20 percent of the data, please. So this is going to create a validation set for you automatically and randomly.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
The randomness is to me really important a really important part of planning out is your solution stable, you know


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Is it going to work like each time you run it but what is important is that you always have the same?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Validation set. But otherwise when you're trying to decide has this hyper parameter change improved my model


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
But you've got a different set of data you're testing it on then you don't know maybe that set of data


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So it knows that the classes or you know, so by classes we main all the possible labels black bear grizzly bear or teddy bear


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Some of them might be tricky like if you ended up with a black bear standing on top of a grizzly bear that might be


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
They are (remember '.c' is the attribute which the classifiers tells us how many possible labels there are?).


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So at that point we can go ahead you'll see all these commands are identical to the pet classifier from last week


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
and let's print out the error rate each time and run 'fit_one_cycle()' four times and see how we go and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I always like to save where I'm up to, to save me the 54 seconds of going back and doing it again and,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
As very usual we unfreeze() the rest of our model. We're going to be learning more about what that means during the course


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
it (it tells you exactly what to type) and we take a look.  Now, we're going to be learning about learning rate, today, actually.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
But for now, here's what you need to know: on the learning rate finder, what you're looking for is the strongest


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So it's kind of like it's something you're going to have to practice with and get a feel for, like what


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
bit works so like if you're not sure is it this bit or this bit, try both learning rates and see which one


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I've been doing this for a while and I'm pretty sure this looks like where it's really learning properly. So I would pick something


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
3x10^-5 so, you know somewhere around here. That sounds pretty good. So that's for my bottom learning rate


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
We can use the ClassificationInterpretation class to have a look at what's going on. And in this case, we made one mistake.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Most of the people watching this are domain experts not computer science experts. And so this is where you can use your knowledge of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
You know 'point mutations' in genomics or 'Panamanian buses' or whatever. So, let's see how that would work.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
either the most wrong about or the least confident about we're going to look at those and decide which of those are


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
If there is some mislabeled data that it's going to be predicted correctly and with high confidence.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
'it's not confident of' or it was 'confident of but it was wrong about'.  They are the things which might be mislabeled.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
top losses... there's not just .plot(top_losses), but there's also just .top_losses() and .top_losses() returns two things


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It's going to actually return the entire data set, but sorted, so the first things will be the highest losses.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
all of the file names in our data set and when I say 'in our data set' and this particular one is in our


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
to clean up your training set to get the noise out of that as well. So it's a good practice to do both


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
We'll talk about test sets later as well. If you also have a test set you would then repeat the same thing.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Basically the same thing as plot_top_losses. So in other words, these are the ones which is either wrong about


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Okay, and all the rest do look, indeed, like bears and then so I can click confirm and it'll bring up another five


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Not there. Well, that's a teddy bear I'll leave that.  That's not really, I'll get rid of that one. 
 Confirm.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Okay. So what I tend to do when I do this is I'll keep going Confirm until I get to a couple of screens all


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
The things that all look, okay, and that suggests to me that I've kind of got past the worst bits of the data


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I'll just note here that what our San Francisco study group did here was that they actually built a little app


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Jupyter notebook, which you might not have realized is possible, but not only is it possible, It's actually


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Surprisingly straightforward and just like everything else you can hit double question mark to find out their secrets


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
normal, you know there's there's basically callbacks for what happens when you click on a button where you just do standard Python things and to


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Applications inside notebooks is like it's really underused but it's super neat because it lets you create


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
You will probably find that there's a whole lot more buttons here because we've already got a long list of to do that


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
to write applications in your notebook, what are you going to write?  And if you google for 'ipywidgets'...


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
GUI programming environment you can play with and this will all work nicely with your models and so forth


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
that's normal. Okay, it's fine.  But it's still a good idea just to make sure that you don't have too much


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
distributed serving system they should use and how do they use a thousand GPUs at the same time and whatever else,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
queuing, and running it all together, all of your users have to wait until that batch has got filled up and run.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It's a whole lot of hassle, right? And then if you want to scale that there's another whole lot of hassle


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It's much easier if you just wrap one thing throw it at a CPU to get it done and it comes back again


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
But it's so easy to scale. All right, you can chuck it on any standard serving infrastructure. It's going to be cheap as hell


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So most people I know who are running apps that aren't kind of at Google scale based on deep learning are using CPUs


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So when you're running when you're not training a model, but you've got a trained model and you're getting to predict things,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
you've got your pre-trained model, you saved those weights, and how are you going to use them to create something like Simon relations?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
that you trained with?  Right?  You need to not know not just what are they, but what were the order?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we have a 'teddy bear detector' and my daughter Claire is about to decide whether to cuddle his friend


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Daddy's deep learning model and she gets a picture of this, and here's a picture that she's uploaded to the web app,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we're going to create a special kind of DataBunch, which is one that's going to grab one single image at a time


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
The only reason we pass it a path is so that it knows where to load our model from, right? That's just the path


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
But what we do need to do is that we need to pass it the same information that we trained with. So the same


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we'll learn more about, but just make sure it's the same stuff that you used before.  And so now you've got a DataBunch


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So you can now create a CNN with this kind of fake DataBunch. And again, you would use exactly the same


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
model that you trained with, you can now load in those saved weights, okay?  And so, this is the stuff that you do once,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
just once when your web app's starting up, okay, and it takes you know, 0.1 of a second to run this code.  And


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
learn.predict(img), and it's lucky we did that because it is not a teddy bear. This is actually a black bear so, thankfully,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
excellent deep learning model, my daughter will avoid having a very embarrassing black bear cuddle incident.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And but basically it's going to look something like this. So Simon used a really cool web app toolkit called Starlette, if you've ever used


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
without using up a process.  So for things like 'I want to get a prediction' or 'I want to load up some data' or whatever


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So Starlette would come highly recommended for creating your web app. And so yeah, you just create a


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
You open your image, you call .predict() and you return that response and then you can use, you know,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
whatever, JavaScript client or whatever to to show it.  And that's it. That's basically the the main


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
The one that Simon's used (we'll mention that on the forum) it's something you can basically package it up as a docker


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
classifiers that you're creating, you can turn them into web applications. So I'll be really interested to see


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
you'll find most of the time people are posting things saying "I downloaded these images, I tried this thing.."


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
it really helps to know a little bit about what's going on. So first of all, let's look at examples of some problems.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So we're going to learn about what those mean and why they matter but first of all, because we're experimentalists,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
let's try them.  All right?  So let's go with our teddy bear detector and let's make our learning rate


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So if you see your validation loss do that, right?  Before we even learn what validation loss is, just


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So this is just I've just copied and pasted what happened when we trained before with a default error


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we were down to a 2 or 3% error rate.  With this really low learning rate, our error rate does get better,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
validation and training loss and you can just see them just like gradually going down so slow, right?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
5%.  But look at this.  The difference between training loss and validation loss.  The training loss is much higher than the validation loss. So


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
100,000 million then put it back to where it was and try a few more epochs. That's the balance, right?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Okay, "Too many epochs" (we're going to be talking more about this) create something called "overfitting".


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I turned (we're going to learn all about these terms soon), I turned off all the data augmentation, I turned off dropout,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
You will see a lot of people, even people that claim to understand machine learning, tell you that if your


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
you will learn today in more detail, and during the rest of the course, that is absolutely not true.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Any model is trained correctly will always have training loss lower than validation loss.  That is not a sign of overfitting.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
The sign that you are overfitting is that your error starts getting worse, because that's what you care about, right?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So as long as your training and your model error is improving, you are not overfitting. How could you be?  Okay?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
There are some other details that we will learn about during the rest of this course, but honestly, if you stopped listening now,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
(please don't, that would be embarrassing) and you just, like "Okay. I'm going to go and download images..."


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It's a matrix of numbers.  For this grayscale one, it's a matrix of numbers, if it was a color image, it would be


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
have a third dimension.  So when you add an extra dimension, we call it a 'tensor' rather than a matrix.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
three numbers.  A number for the probability that it's a teddy, a probability that it's a grizzly, and the probability is a black bear


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
there's some hypothetical function that's taking the pixels representing a handwritten digit and returning ten


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
strengthen your understanding of the concepts and make sure that you know, you know what's going on and and really help you here.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Q: "Can we have a definition of the error rate being discussed and how it is calculated?  I assume it's cross validation error."


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we now know that means 'find out which particular thing it is' and then look at how often that equals the target.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
metrics (so these things that we pass in, we call them metrics) are always going to be applied to the validation set.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So anytime you put a metric here, it'll be applied to the validation set because that's your best practice, right?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
performance on data that your model hasn't seen and we'll be learning more about the validation set shortly.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
which it might not well be, you actually want the documentation.  That will both give you a summary of the types in and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Where, for example, you will find complete runs through and sample code and so forth showing you how to use all these things


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
also in the documentation both in the doc function and in the documentation, you'll see a source link. This is like ??,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So you can see exactly how that's implemented, and what else is around it so lots of good stuff there.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And then I tend to kind of just multiply from there. So I generally find then that the the next stage


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
for the second part of the slice and whatever the LR_finder() found for the first part of the slice.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
it's just a rule of thumb which is like 10 times less than your your first part which defaults to 3e-3,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
and then the first part of the slice is what comes out of the LR_finder() and we'll be learning a lot more about these


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
But yeah for now all you need to remember is that in your you know, your basic approach looked like this it was


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
here...and that's the number that I get from the learning_rate_finder a bit where it's got the strongest slope.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Kind of "don't have to think about it, don't really have to know what's going on" Rule of Thumb that works most of the time.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So we're going to create this mathematical function that takes the numbers that represent the pixels and spits out probabilities for each possible [?]


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And by the way, a lot of the stuff that we're using here, we are stealing from other people who are awesome,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So like, please check out their work because they've got great work that we are highlighting in our course.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So I, all the time, am trying to either remind myself about something or I never learnt something and so


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So there's another way of saying the same thing.  Another way of saying that would be if I could multiply a2


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So now at this point I'm actually going to say let's not put the number 1 there but let's put an x1 here


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
indexing, right?  But rather than things in square brackets with pytorch indexing,  we kind of put them


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
kind of in the subscript of our equation. Ok?  So this is now saying there's actually lots of these different y(i)s


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Because I want to do linear algebra. Why do I want to do in linear algebra? Well one reason is because


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So if you're interested check out 'Computational Linear Algebra for Coders', so it's a good opportunity for me to throw in a pitch for this


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
you see like these little "i" subscripts, that sounds like you're going to have to do loops and all kind of stuff,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
but, what you might remember, from school, is that when you've got like two things being multiplied together,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I don't know who created this.  So now I do, somebody called Andre Stouts,  created this fantastic thing called "matrixmultiplication.xyz" and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Okay.  So.  This is actually an excellent spot to have a little break and find out what questions we have coming through our students.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
(because if you make it higher, then it goes off into massive losses, if you make it lower it goes really slowly)...


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Okay?  So, you know that you've trained for long enough.  And you're still not happy with the accuracy.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
and we'll learn about some of them during, er, pretty much all of them, during this course, but one of the easiest ones is:


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
If you get more data, then you can train for longer, get a higher accuracy, lower error rate - without overfitting.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Unfortunately, there's no shortcut. I wish there was.  I wish there was some way to know ahead of time, how much data you need.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
nothing.  Try it.  It works.  A lot of people ask this question about how do I deal with unbalanced data?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
analysis with unbalanced data over the last couple of years and I just can't make it not work. It always works. So


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
like, I haven't found a situation in practice where I needed to do that. I've found it always just works fine, for me.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
(which will technically be more than one cycle) or do you redo everything with a longer epoch for the cycle?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
if you do another cycle, then it'll kind of maybe generalize a little bit better if you start again,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Depends how patient you are.  It won't make much difference, you know?  For me personally, I normally just train a few more cycles.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
functional form' - it doesn't take up any room, it doesn't contain anything - it's just a function.  resnett34 is just a function.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Or "is it a quadratic?" or "is it a reciprocal?" This is just a function, this is the "resnet34 function" -


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
faster.  PyTorch really doesn't like loops, right? It really wants you to send it a whole equation to do all at once,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So let's go and take a look because what we're going to try and do then is we're going to try and take


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
this, we're going to call it an 'architecture', it's like the tiniest world's tiniest neural network.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It's got two parameters, you know, a1 and a2, we're going to try and fit this architecture to some data.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And you guys all experimented and found really good stuff. So that's where the 'S' would be Student. That would be Student Gradient Descent.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
where we're going to have a computer try lots of things and try and come up with a really good function and that will be


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
so we're going to kind of go bottom up rather than top down. We're going to create the simplest possible model


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
some data.  The data we're going to generate looks like this. So this might represent temperature and this rate represent number of ice creams


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we sell or something like that, but we're just going to create some synthetic data that we know is following a line.  And


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It can be a vector-vector product, a matrix-vector product, a vector-matrix product or a matrix-matrix product.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And then actually in PyTorch, specifically, it can mean even more general things where we get into higher rank tensors,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
our computers are going to be basically doing this: multiplying numbers together and adding them up, which is a surprisingly useful thing to do.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
"tensor()". You might have heard this word 'tensor' before.  Who's heard the word tensor before?  About 2//3 of you. Okay, so


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So it's not an array where row 1 has two things and row 3 has three things and row 4 has one thing what you


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
you know, a shape where every element every row is the same length, and then every column is the same length


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
people come up with slightly different words or actually no; they come up with very different words for slightly different concepts.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Why is a one dimensional array a 'vector' and a two dimensional array's a 'matrix' and then a three dimensional array...


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Not really.  It doesn't have a name.  Like, it doesn't make any sense. We also you know with computers


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
instead, we're going to say there's always this second x value which is always 1 (you can see it here, always 1),


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
'x array' of data which is going to have we're going to put random numbers in the first column and a whole bunch of ones


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
anytime you've got a function that ends in an underscore, it means "don't return to me that uniform random number but replace


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
then I can do a scatter plot.  And I'm not really interested in my scatter plot in this column of ones,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It's certainly a library, you'll want to get familiar with because being able to plot things is really important.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Lots of them, the other packages, are better at certain things than matplotlib, but like matplotlib can do everything


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
because there's really nothing it can't do (even though some libraries can do other things a little bit better or a little bit prettier).


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So ,now what we're going to do is we're going to pretend we were given this data and we don't know that


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
the values of our coefficients are 3 and 2. So we're going to pretend that we never knew that we have to figure them out,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So we're actually going to use an almost identical approach.  So that's (this is the bit that I found in previous classes,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
but we're fitting a neural network".  So "I know, and we can't print the 50 million numbers anymore,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
identically, doing the same thing".  And the reason this is hard to digest is that the human brain has a lot of trouble


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
conceptualizing of what an equation with 50 million numbers looks like and can do. 
 So you just kind of, for now,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we're going to learn a little bit more in just a moment, about how to make them extra powerful, but for now,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
the thing we're going to learn to fit these two numbers is the same thing that we've just been using to fit 50 million numbers.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
what PyTorch calls 'parameters'.  Or in statistics, you'll often hear called 'coefficients'.  These values a1 and a2.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
this line and they would say, like, "oh, how far away is each of those points"?  And then eventually we come up with


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So you can see how in each case we can say how far away is the line at each spot away from its point and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And for regression the most common loss function is called 'mean squared error', which pretty much everybody calls


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
'MSE'.  You may also see RMSE just 'Root Mean Squared Error'. And so the mean squared error is a loss,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
people normally refer to the actual, they normally call it "y" and the prediction, they normally call it


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
there's no point writing ice cream here and temperature here because we wanted to apply it to anything.  So we tend to use these like


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
that is actually a 'vector' or what we now call it, a "rank 1 tensor" and that is actually a rank 1 tensor,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
(and we're going to be learning a lot more about this), but it does something called element-wise arithmetic in other words


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
then if we take the square of that, it squares everything in that vector.  And so then we can take the mean of that


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
you can use it you can send things into it and get stuff out of it and see how it works, alright?  So, for me,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
all the cases we'll look at, they're exactly the same, they're just different notations for the same thing. But one of the notations is executable,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
"What is the line that fits through here?"  Remember, we don't know (we're going to pretend we don't know)


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
not a floating point. So that's going to "spit the dummy" if you try to do calculations with that in neural nets, all right?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
then the whole thing is now floats, right?  So that's why you'll often see it written this way, particularly by lazy people like me.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
scatter plot of x against y and we can plot the scatter plot of x against y-hat (our predictions) and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
...sorry, the (-1,  1) line and here's actuals. So that's not great, not surprising, it's just a guess.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
engineering or probably computer science at school will have done plenty of this, like Newton's method what all the stuff that you did...


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
"What if we change those two numbers a little bit, what if we made the intercept a little bit higher...?"


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
for each of those four possibilities and see what see what worked.  Did lifting it up or down make it better?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I just told you everything you need to know about it, right?  Which is that it tells you how changing one thing


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we're going to create a loop, we're going to loop through 100 times and we're going to call a function called .update().


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It will then calculate the gradient and in PyTorch calculating the gradient is done by using a method called .backward().


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So if you do a mathematical operation on a tensor in PyTorch, you can call .backward() to calculate the derivative.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
this underscore here... Why?  Because that's going to do it in place.  So it's going to actually update those coefficients a


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
All right, and let's say your algorithm's job was to find where that quadratic was at its lowest point.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And so, well, how could it do this? Well, just like what we're doing now, the starting point would just be to pick


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Okay?  That's its starting point. And so then it can calculate the gradient and the gradient is simply the slope.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
moving in which direction is going to make you go down.  And so the gradient tells you you have to go this way.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Then that's actually not going to be very helpful because then, you see, well where does that take us to?  Oh!


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And so then we'll just do another little jump;  see what the gradient is and do another little jump.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we find our gradient to tell us kind of what direction to go and like,  do we have to go a long way or not


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
all the way past the right answer further than we started with and it got worse and worse and worse.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
so, eventually you're going to get there but you're doing lots and lots of calculations along the way.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we'll learn about some minor...not minor... We'll learn about some easy but important optimizations we can do to make this go faster.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
There's a couple of other little minor issues that we don't need to talk about now one involving zeroing out the


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
gradients and another involving making sure that you turn gradient calculation off when you do the SGD update.  If


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
This is one of the nice things that you can do with matplotlib is you can take any plot and turn it into an animation.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
this function.  And this function just called that update() that we created earlier and then updated the 'y'


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
but it's basically doing the same thing, okay?  And so studying these simple versions is actually a great way to get an intuition.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Maybe you can even try a 3d plot. I haven't tried that yet, but I'm sure it would work fine too.   So


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
You'll see what we did here was we calculated the value of the loss on the whole data set on every iteration.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
we update our weights.  And then we grab another 64 random images. We update the weights. So in other words, the loop


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
there, right, so some random indexes on our x and some random indexes on our y to do a minibatch at a time,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
those random few points accord your minibatch and that approach is called SGD, or Stochastic Gradient Descent.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
the 'learning rate' is a thing that we multiply our gradient by, to decide how much to update the weights by.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
then 'loss function' is the thing that's telling you how far away or how close you are to the correct answer.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
So, these models, these predictors, these Teddy Bear Classifiers, are functions that take pixel values and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Next week, we'll learn how to do the same thing for classification where we have multiple numbers.  But it's basically the same.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
research around this, there are not math people and non-math people. It turns out to be entirely a result of


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
where she will introduce you to some of that academic research.  And so if you think of yourself as not a math person


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
there are some cultures, like Romania and China, where the 'not a math person' concept never even appeared.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
It's almost unheard of in some cultures for somebody to say "I'm not a math person" because that just never entered


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
don't freak out if words like 'derivative' and 'gradient' and 'matrix product' are things that you're kind of scared of,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
That's the worst part about Simon. Not only is he really smart he's also really modest which I think it's just awful.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
I mean if you're going to be that smart, at least be a horrible human being and, you know, make it okay.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
to our data.  But imagine that our data wasn't actually line 'shaped', right? And so if we try to fit something which was, like


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
it's very unlikely we go and look at some other place to find out the temperature that it is and how much ice cream they're


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
doesn't have too many parameters in it.  It turns out that's actually completely not the right way to think about it.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
There are other ways to make sure that we don't overfit, and in general, this is called 'regularization'.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And so as we're going to learn a lot about next week, that's why we have this thing called a 'validation set'.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
minibatch SGD training loop with one set of data (with one set of teddy bears, grizzlies, 
 black bears) and


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
to see how good is it on a bunch of images which were not included in the training. And so, if we do that,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
it'll tell us.  "Oh, your loss function and your error is really bad", because on the bears that it hasn't been trained with,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Because that's the thing they can then use to figure out, you know, if somebody's selling them snake oil or not, you know,


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
And this is also one of the things we go into in a lot of detail in the 'Intro to Machine Learning' course.


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
for this course, but it is certainly something that's worth deeper study as well.  Any questions or comments before we wrap up?


c0: PLfYUBJiXbdtSIJb-Qd3pw0cqCbkGeS0xn
c1: ccMHJeQU4Qw
c3:
Okay, good.  All right, well, thanks everybody. I hope you have a great time building your web applications. See you next week.


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
tip number two for the video is because I can't edit them all I can do is add these things called cards and


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Unfortunately, they're not easy to notice so keep an eye out for that because that's going to be important updates to the video all right


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You can then turn off enable GPU and click start Jupyter, and you'll have a Jupyter notebook instantly that costs you some money. It's


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Three cents an hour, okay, so if you don't mind spending three cents an hour to learn machine learning


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Both the deep learning and for regular machine learning so check them out because there's lots of options


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You're ever looking at my screen and wondering "where are you?" Look up here, and you'll see that tells you the path fastAI / courses / ml1


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So there's a couple of different ways you can do this both here in person or on the video you can either


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Attempt to follow along as you watch or you can just watch and then follow along later with the video


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's quite hard to multi-task, and if you're working on something you might miss a key piece of information


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Which you're welcome to ask about okay, but if you follow along with the video afterwards then you can pause stop


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So the basic approach we're going to be taking here is to get straight into code start building models


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
But at the point where you deeply understand what it's for and at the point that you're able to be an effective practitioner


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
my suggestion or at least and all of my students have told me the ones who have gone away and spent time studying


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
The stuff that we're showing in this course a lot of it's never been shown before this is not a summary of


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Going to be shown for the first time and so that's kind of cool because if you want to write a blog post about something


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That you learn here, you might be building something, but a lot of people find super useful all right, so


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
There's a great opportunity to practice your technical writing, and here's some examples of good technical writing okay page by showing people stuff


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
it'll be "I just learned this thing and I'm going to tell you about it and other people haven't seen it"


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Built on top of the fastAI library so even just stuff in the library is going to be new to like everybody


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Something that's quite convenient as if you use these to auto reload commands at the top of your notebook


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You'll notice when I import the libraries that for anybody here who is a experienced Python programmer


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Generally speaking in software engineering we're taught to specifically figure out what we need and import those things


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
The more experienced you are as Python programmer the more extremely offensive practices you're going to see me use for example. I don't follow


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
"from library import *" means you don't have to figure out ahead of time what you're going to need from that library


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Functions as possible be like no more than about five lines of code. It's just going to be really easy


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Right so all the documentation plus the source code, so you can see like nothing has to be mysterious and


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
I'll just go "??" and the name of the scikit-learn function I'm typing and I'll just go ahead and read the source code


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
All right so today, we're going to be working on a kaggle competition called Blue Book for Bulldozers


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Solve it according to a specification that that actual person with that actual problem decided would be actually helpful to them


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Which was why did this company to start up the side that predicting the option sale price of bulldozers was important


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Where did they get the data from? how did they clean the data? and so forth okay, and that's all important stuff as well


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
But the focus of this course is really on what happens next which is like how do you actually build the model?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Whether they be running now or whether they be old ones is that you can submit yours to the leaderboard even old closed


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Your accuracy is really bad. Is it because this is just very hard like it's just not possible then the


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You can't do better? or is it actually that it's an easy data set and you've made a mistake? and like


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
This is going to be something you're going to find very hard and there isn't a simple solution to it which is


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Otherwise yeah, there's really no way to know there's no way other than creating a keggle competition


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Or getting you know a hundred top data scientists to work at your problem to really know what's possible


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And I suggest that you read this later, but basically we're to try and predict the sale price of heavy equipment


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And one of the nice things about this competition is that if you are like me you probably don't know very much about


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Heavy industrial equipment options right I actually know more than I used to because my toddler loves


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Building equipment, so we actually like watched youtube videos about front end loaders and forklifts


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Machine learning should help us understand a data set not just make predictions about it so by picking an area


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Intuition about the data can make it very difficult for you to be open-minded enough to see what does the data really say?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Right and so you can go ahead and do that if you're running on your own computer right now if you're running on AWS


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's a little bit harder right because unless you're familiar with textmode browsers like a links, or links. It's quite tricky to


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Get the data set to keggle so a couple of options one is you could download it to your computer and then?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That's probably what I want. That looks good, so what you can. Do is you can right-click on that and say copy


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That's going to create a command that has all of my cookies, headers everything in it necessary to download this


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
2.0. Thing to the command that doesn't seem to work with all versions of curl so something you might want to do


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That would be brew install unzip if brew doesn't work. You haven't got homebrew installed so make sure you install it and then unzip


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Can actually get a web-based terminal, and so you'll find on crestle There's a slash data sets folder


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Okay, so assuming that we don't have it already downloaded in paper actually paper space should have most of them as well then we would


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And if you're using what we using git right you'll find that that doesn't get added to git because it's in the git ignore


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So don't worry about creating the data folder. It's not going to screw anything up, so I generally make a folder called data


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You can now see I generally have like anything that would change that might change from person to person


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
I kind of put in a constant so here. I just defined something called path, but if you've used the same path


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Let's go ahead and keep moving along, so we've now got all of our libraries imported, and we've set the path to the data


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
shell commands from within Jupyter notebook by using an exclamation mark so if I want to check what's inside that path I can go LS


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Or you can even use Python variables if you use a Python variable inside a Jupyter show command you have to put it in { }


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Python command it's a bashed command and bash doesn't know about capital path because capital path is part of Python


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So this is a special Jupyter thing which says expand this Python thing please before you pass it to the shell


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
use the training set which contains data through the end of 2011 to predict the sale price of bulldozers and


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Right so one easy way to look at the data. Would be to use shell command head to look at the first two lines paired


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Right, so here's the first few five lines, okay, so there's like a bunch of column headers, and then there's a bunch of data


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So that's pretty hard to look at so what we want to do is take this and read it into a nice tabular format


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Does Terrance putting classes on mean I should make this bigger, or is it okay is this big enough font size, everybody? okay


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
this kind of data where you've got columns representing a wide range of different types of things such as an identifier of


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
I say I refer to this as structured data because like there have been many arguments in the machine learning community on Twitter about


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
When I refer to structured data, I mean columns of data that can have varying different types of data in them


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Pandas is so important that it's one of the few libraries that everybody uses the same abbreviation for it which is PD


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Pandas as PD and so everybody does this right, so you'll see lots of people using PD dot something. They're always talking about pandas, so


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And I always add this low-memory equals false. That's going to actually make it read more of the file to decide what the types are


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You've probably used lots of different ways in the past in Python of interpolating variables into your strings


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Python 3.6. Has a very simple way that you'll probably always want to use from now on and it's you to create a normal string


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
confusing these are not the same Curly's that we saw earlier on in that LS command right that LS command is specific to Jupyter and


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
These Curly's are Python 3.6. Format string Curly's they require an F at the start so if I get rid of the F


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It doesn't matter if it's an integer right normally if you like to do string concatenation with integers place and complains


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And then use various methods on it so for example df_raw.tail will show us the last few rows of the data frame


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
I've created one extra function here display. All normally if you just type DF Raw, or if it's too big to show


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
But this is just changing a couple of settings to say even if it's got a thousand rows in a thousand columns


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Okay, so this is finished. I can actually show you that so if I just type this is really cool in in Jupyter


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And then we got a whole bunch of things we could predict it with and when I start with a data set I tend


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
I actually kind of don't like I I want to find out at least enough to know that I've like managed to import it okay


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
The academic books I've read say that that's one of the biggest risks of everything, but the practical books say, let's do some EDA first


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Yeah, so that the truth is kind of somewhere in between and I generally I generally try to do machine learning driven


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So this means they're going to look at the difference between the log of our prediction of price and the log of the actual price


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Okay, so because they're going to be focusing on the difference of the logs that means that we should focus on the logs as well


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
But did I miss by ten percent right so if it was a million dollar thing and you're a hundred thousand dollars off


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Assuming that you have some familiarity with numpy if you don't we've got a video called deep learning workshop


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So okay now that we've replaced its sale price with its log. We can go ahead and try to create a random forest


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
In general it doesn't overfit it can and we'll learn to check whether it is, but it doesn't generally overfit too badly


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You don't need a separate validation set in general it can tell you how well it generalizes even if you only have one data set


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It doesn't assume that the relationships are linear. It doesn't assume that you've just specified the interactions


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Very few pieces of feature engineering for many different types of situation you don't have to take the log of the data


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's a great place to start right if your first random forest does very little useful, then that's a sign that


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Yeah great question, so there's this concept of curse of dimensionality. In fact there's two concepts


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
I'll touch upon curse of dimensionality and the no free lunch theorem these are two concepts you often hear a lot about


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
the more dimensions you have the more all of the points sit on the edge of that space alright, so if you've just got a single


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Dimension where things are like random then they're spread out all over right where else if it's a square


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Then the probability that they're in the middle means that they've kind of been on the edge of either dimension


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So it's a little bit less likely that they're not on the edge edge dimension. You add it becomes more addictive


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And what that means in theory is that the distance between points is much less meaningful, and so if we assume that


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Just use them without being very careful to remove the ones you don't care about that somehow things won't work that


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
One is that the points still do have different distances away from each other just because they're on the edge


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
More similar at this point that it is to that point so even things will learn about K nearest neighbors


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Machine learning and so particularly there was this concept of these things called support vector machines that were theoretically very well justified


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And it turns out that in practice building models on lots and lots of columns works really really well


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
By definition is random right so there isn't going to be some way of looking at every possible random data set that's in some way


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Mathematically, we would say it sits on some lower dimensional manifold. It was created by some kind of


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So the truth is that we're not using random datasets and so the truth is in the real world there are actually techniques that work


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
researchers who spend a lot of time studying this which is which techniques work a lot of the time and


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
came along there wasn't really any standard way to pre-process them properly and to properly set their parameters


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So when I try to create a ra... So random forest regressor what is that? Random forest regressor, okay? It's part of something called SK learn


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It is by far the most popular and important package for machine learning in python. It does nearly everything


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You might find in the next part of this course with your net you're going to look at a different kind of decision tree ensemble


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Where actually there's something called XG boost which is better than gradient boosting trees in scikit learn


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
we're trying to predict a continuous variable price so therefore we are doing regression and therefore we need a regressor


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Scikit-learn has the same form you first of all create an instance of an object for the machine learning model you want


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
independent variables the things you're gonna use to predict and the dependent variable the thing that you want to predict so in our case the


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That will bring up the you know a quick inspection of the parameters in this case it doesn't quite tell me what I want


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Yes, and that tells me. It's a single label or list like list like means like anything you can index in Python


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Question mark and double question mark for the docs and the source code like if you know nothing else about using Python libraries


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
stack trace like this so an error the trick is to go to the bottom because the bottom tells you what went wrong a


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Buffer it tells you all of the functions the court other function could cause other functions to get there


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
categorical variables which could either be numbers where the meaning is not continuous like zip code, or it could be a


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Get to a point where we have a data set where we can use all of these variables so they have to all be


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Numeric and they have to be usable in some way, so one issue is that we've got something called sale date


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
We told it that that's a date so it's been passed as a date, and so you can see here. It's Data type


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Right, and this is actually where we need to do our first piece of feature engineering right? Inside a date there's a lot of interesting stuff


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
All right, so since you've got the catch box. Can you tell me? What are some of the interesting bits of information inside a date?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That's true, I hadn't expressed very well. What are some columns that we could pull out of this? Yeah, month


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Week of year yeah, okay, I'll give you a few more like that you might want to think about would be like


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
soma, you would probably want to know was there a San Francisco Giants ball game on that day? right so like what's in a


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Less than half a page of code alright, so here is something. It's going to so rather than often rather than having Docs


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
DF square bracket field name is how we grab a column where that column name is stored in this variable okay?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Attribute with that name, so this is going to go through and you can again you can google for Python get attribute


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's a cool little advanced technique, but this is going to go through it's going to find for this field


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Let's go field equals. This is the kind of experiment. I want you to do right play around say all date


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Interesting for having any reason right and this is like the opposite of the curse of dimensionality


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That could be ever be interesting at all add that to your data set and every variation of it you can think of


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
We're going to go ahead and add all of these different attributes and so for every one I'm going to create a new field


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
or was it already saved as one in the original? Yeah, it's already a date type and the reason it was a date type


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Is there a way to do that so we just look through all the columns and say, like, if it looks like a date, make it a date?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Let's do that one on the same forum thread that savanah creates because I think it's a reasonably advanced question, but generally speaking the


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Just as strings and at the end there they all are right, so it's removed sale date, and it's added all those


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
But by default it doesn't turn anything into a category for you, so I've created something called Train cats


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That's actually a number right as an integer and it's going to store a mapping from the integers to the streets, okay?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Mappings to let you all make sure your test set of validation set uses the same mappings, okay, so when I go trained cats


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's actually not going to make the data frame look different at all, but behind the scenes it's going to turn them all into


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Let's see how we go I'll try to finish on time, so you'll see now remember. I mentioned there was this dot DT


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
There's a dot cat attribute that gives you access to things assuming something's a category all right


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And so usageband was a string and so now that I've run train cats. It's turned it into a category so I can go


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's that this list is in a bit of a weird order high low medium. The truth is it doesn't matter too much


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
But what's going to happen when we use the random forest is it's actually good that this is going to be 0 this is going


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And so we're going to have a decision tree that can split things at a single point so it either be high


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Medium versus high and low that would be kind of weird right it actually turns out not to work too badly


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And they map to the position in this array and as we're going to learn shortly a random forest consists of a bunch of trees


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Versus low and medium, which that semantically makes sense it's like is it big or we could split it into


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's also good to know that people when they talk about like different types of categorical variable


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Ordinal and an ordinal categorical variable is one that has some kind of order like high medium and low


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That's what I'm saying it. Helps a little bit right it means you can get there with one decision rather than two


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
I noticed there is a negative one in that list of categories is that like an NA yeah exactly so for free we get


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And what are the things we're going to do is we're going to actually add one can somebody pass the vector Paul is we're going


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Which we'll get to in a moment is going to create three separate columns ones and zeros for high once


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
There's a million ones in series for low where else this one creates a single column with an integer zero one or two


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
We're going to get to that one shortly, yep, did you have a question to Paul or just pointing out okay?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Fastest way to save something in the fastest way to read it back right so most of the folks you deal with


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
In spark in lots of like things for like communicating across computers because it's incredibly fast


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Exists okay equals true. If there are some sub directories. It'll create them for you, so this is a super handy little function


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
crastle for the first time it's complaining about that so if you get a message that something's not installed


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
You don't have to rerun all the stuff they love you could just say pd.read_feather and we've got our data frame back


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So quite a lot of the functions have a few additional parameters that you can provide and we'll talk about them later


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
But basically we're providing the data frame to process and the name of the dependent variable that the the Y field name, okay


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
We don't need it for categories because pandas had is handles categorical variables automatically by setting them to minus one


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's a categorical type we'll talk about the maximum number of categories later, but lets us units is always true


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Pandas uses minus one for missing so now zero will be missing and one two three four will be all the other


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Optionally you can say that if you already know about dummy values there are columns with a small number of


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Possible values you can put in two dummies instead you can America lysing them, but we're not going to do that for now okay?


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
replacing missing values with the median adding an additional column telling us which ones were replaced and removing the dependent variable


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Sale price is no longer here. Okay. We've now got a whole new color a whole new variable called Y the contain sale press


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
These boolean z' are treated as numbers. They're just considered contributed a zero or one that is displayed as false and true


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Which presumably is something like that could be a serial number? It could be like the model identifier


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
That's created by the factory or something. We've got like a data source ID like some of these are numbers


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
We'll talk about why and how and a lot about that in detail but for now all you need to know is no problem


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Okay, so as long as this is all numbers, which it now is we can now go ahead and create a random forest


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Trivially paralyse abour so what that means is that they if you've got more than one CPU which everybody will basically on their


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
It's a separate process basically for each CPU you have so that's pretty much what you want all the time


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
We pulled out and then get the score ok the score is going to be the r-squared. We'll define that next week


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Because maybe we had data that had points that look like this and we fitted a line that looks like this


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Whether we've actually done a good job is by having some other data set that we didn't use to train the model now


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Into twelve thousand rows, which we're going to put in a separate data. Set called the validation set


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
And so that means that the most recent twelve thousand rows are going to be our validation set again


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Like this is pretty cool right with with like with no thinking at all using the defaults of everything


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
So I'm using here's where I got it from his like the stack trace of where I got an error or here's like


c0: PLfYUBJiXbdtSyktd8A_x0JNd6lxDcZE96
c1: CzdWqFTmn0Y
c3:
Practicing data scientists that competing table competitions, okay great good luck, and I'll see you on the forums. Oh


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Practical deep learning for coders is all about getting you up and running with deep learning in practice getting world-class results


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Theory in details that are necessary to rebuild all of the world-class results. We're learning about from scratch


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
although they're exactly the same videos the important thing about watching them through our website is that you'll get all of the information


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Why something works the way it does or there may be some computer problem that you have or so forth?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
On forums.fast.at AI there are thousands of other learners talking about every lesson and lots of other topics besides


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
By doing some coding. This is an approach. We're going to be talking about in the moment called the top-down approach to study


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We'll be talking about them more shortly, but for now I'm going to show you how you can get access to a GPU


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Specifically you're going to need an NVIDIA GPU because only NVIDIA GPUs support something called CUDA


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Obviously, it's not ideal that we're stuck with one particular vendors cards and over time we hope to see more competition in this base


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Your laptop almost certainly doesn't have one unless you specifically went out of your way to buy like a gaming laptop


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Recent survey of tens of thousands of data scientists was rated as the third most important tool in the data scientist toolbox


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
It's really important that you get to learn it well and all of our courses will be run through Jupiter


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Yeah, He might have changed that recently to less hours, but you can check the FAQ or the pricing but you certainly get some free hours


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
The pricing varies because this is actually runs on top of Amazon Web Services so at the moment. It's 60 cents an hour


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
The nice thing is though that you can always turn it turn it on you know start your Jupiter without the CP without the GPU


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
can pick which one of the three data centers to use so pick that one closest to you, so I'll say a West Coast and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And then it says choose machine and you can see there's various different machines I can choose from


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That's actually going to be much faster than Crescent 67 our machine or for 65 cents an hour way way way faster


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You may find if you trailers do the 65 cents an hour one that it may require you to contact paper space to say


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So now we need to configure this further course and so the way you can figure it for the course is you type


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And the data we use for the first part of the course ok so that takes an hour or so and when it's finished running


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That's where your python is you've got a data directory which contains the data for the first part of this course


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
CD fast AI and from time to time you should go get Paul and that will just make sure that all of your


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
To be the paper space IP address so if you click on the little arrows to go smaller you can see the IP addresses here


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Where it used to say local host okay, so it's now HTTP and then my IP and then everything else a copied before


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So here we are ready to go depending whether you're using Kressel or paper space or something else if you check course


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
It's not fast today. I will keep putting additional videos and links to information about how to set up other


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
If you've got the tool bar showing you can just click on the little Run button so you'll notice that some cells contain


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Code and some contain text and some contain pictures and some contain videos so this environment basically has


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You'll see that cell turn into a star the one turn into a star for a moment, and then it finished running, okay


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So let's try the next one this time instead of using the toolbar. I'm going to hold down shift and press ENTER


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Crystal or the paper space script that we just used to set up that this will already be and made available for you


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That we really need and so there are a couple of extra steps if you're using cresol you have to run two more cells


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That just runs the stuff that the stuff that you only need for crystal. I'm using paperspace, so I'm not going to run it.


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
There's a training set and a validation set if you're not familiar with the idea of training sets and validation sets


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Actually a lot of students who would you know as they went through these as said? They'll they've liked doing them together?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So people have done both seen you know say they find it they they each support each other. I wouldn't say it's a prerequisite


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
I don't know what that means at least Google, but do a quick read you know because we're assuming that


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And as we just wanted to say in general with fasting our philosophy is to kind of learn things on an as-needed


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You'll never get around elite learning the stuff you actually want to learn exactly that shows up in deep learning, I think


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
There's a "cats" folder and a "dogs" folder and then inside the validation cats folder is a whole bunch of JPEGs


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
The reason that it's set up like this is that this is kind of the most common standard approach for how?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So we can have a look so if you go plot dot a.m.. Show we can see an example of the first of the cats


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So there's no cat, but we're going to mainly be interested in the underlying data that makes up that cat


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
It's an image whose shape that is the dimensions of the array is 198 by 1/7 9x3 is it's a three dimensional?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
and this is the red green and blue pixel values between naught and 255 so here's a little subset of what a


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
this this data set actually comes from a caracal competition the dogs versus cats caracal competition and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
This actually would have won the caracal competition of that time the best in the caracal competition was 98.9 and we're getting about


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And I think that's because like a lot of people assume that deep learning takes a huge amount of time


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
One of the ways, we've made it much simpler is that this code is written on top of a library we built


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Best practices approaches that we can find and so each time a paper comes out you know we that looks interesting we test it out


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
If it works well for a variety of data sets and we can figure out how to tune it we implement it in fast AI


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Most people are more familiar with tensorflow than pi torch because google markets that pretty heavily but


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Most of the top researchers I know nowadays at least the ones that are at Google have switched across to PI torch yes, Rachel


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Hopefully you'll really like about last AI is that it's really flexible that you can use all these kind of curated best


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Practices as much as little as you want and so really easy to hook in at any point and write your own


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Data augmentation write your own loss function write your own network architecture whatever and so we'll do all of those things


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So the zeros if we look at data dot classes the zeros represent cats the ones represent dogs. You'll see here


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So we're going to get predictions through our validation set and so to do that we can call learned predict


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So you can see here the first ten predictions and what it's giving you as a prediction for dog and a prediction for cat


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Now the way PI Torche generally works and therefore fast AI also works. Is that most models return the


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Of the predictions rather than the probabilities themselves we'll learn why that is later in the course


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And there's a few functions here that you can look at yourself if you're interested. That's just some flooding functions that we'll use


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Incorrect alright so you can see that some of these which it thinks are incorrect obviously are just


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
But clearly this one, which it called a a dog is not at all a dog. So there are some obvious mistakes


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Which cats is it the most confident are cats, which dogs are the most doglike the most confident dogs


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So yeah, so finally, I just mentioned the last one. We've got here is to see which ones have the probability closest to 0.5


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Always the first thing I do after I build a model is to try to find a way to like visualize what it's built


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Then I need to take advantage of the things that's doing well and fix the things that's doing badly and so in this case


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So this rather makes me suspicious that we're going to need to use something called data augmentation that we'll learn about


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Two or more different types of thing put them in different folders and run the same three lines of code


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
CT scans or something this won't work very well as well learn about later. There are some other things


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Downloaded for Google Images ten examples of pictures of people playing cricket ten examples of people playing baseball


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Train a neural network, but we didn't first of all tell you what a neural network is or what training means or?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
where you learn every little element piece by piece, and you don't actually get to put them all together and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Throughout the course you're going to see like new problems that we want to solve so for example in the next lesson


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And we'll see why it is that this approach that we're learning today doesn't quite work as well and what?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
things do we have to change and so we'll learn enough about the theory to understand why that happens and then we'll learn about the


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So during the course we are gradually going to learn to solve more and more problems as we do so we all need to learn


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Architecture from scratch and our own training loop from scratch and so were actually built everything ourselves


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And it would only be years later that you might learn about the physics of how curveball works for example


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Well with music we put a instrument in your hand and you start banging the drum or hitting the xylophone


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And it's not until years later that you learn about the circle of fifths and understand how to construct a cadence example


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We'll start to look at how to use neural nets to look at structured data, and then look at language data


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
What exactly did it do and how to do work and how do we change things around and use it in different situations for?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So what that kind of means is that most students find that they tend to watch the videos two or three times?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Like watch lesson one two or three times and listen to two or three times in verse and three three times


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
But like they do the whole thing into end lessons one through seven and then go back and start lesson one again


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That's an approach which a lot of people find when they want to go back and understand all the details


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Enough that can work pretty well, so I would say you know aim to get through to the end of lesson seven


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You know as as quickly as you can rather than aiming to fully understand every detail from this data


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And particularly we're going to look at images of from satellites, and we're going to say for a satellite image


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
What kinds of things might you be seeing in that image and there could be multiple things that we're looking at so a multi-label classification


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So we're going to specifically look at this data. Set of predicting sales the number of things that are sold at different stores


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Based on different holidays and and so on and so forth and so we're going to be doing its sales forecasting exercise


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Classifiers for any kind of image will learn to create in NLP classifiers to classify any kind of language in lots of different ways


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Then we'll look at something called collaborative filtering which is used mainly for recommendation systems


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Then for each one we can look up for each user how much they liked that movie and the goal will be of course to


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We haven't seen before are they likely to enjoy that movie or not and that's the really common approach used for like


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Actually, we're gonna look at the writings of Nietzsche the philosopher and learn how to create our own


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So this here perhaps that every life values a blood of intercourse when it senses. There is unscrupulous


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We're going to learn how not just to recognize cats from dogs how to actually find like where the cat is


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
With this kind of hate map, and we're also going to learn how to write our own architectures from scratch


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
one of the things that we've heard a lot of students say is that there's been too much time on theory and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And even after we tell people about this morning where they still come to the end of the course not and say


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
World-class models in a code first approach learning what you need as you go is very different to a lot of the advice


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Finally start doing ml so we would say if you want to become an effective practitioner do exactly the opposite of of this


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We think this is bad advice and this can be very discouraging for a lot of people to come across this yeah


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
yeah, it's it's it's it's you know we now have thousands and more tens of thousands of people that have done this course and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Google brain residence or you know have created patents based on deep learning and so forth who have done it by doing this course


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
World-class image classifier in 17 seconds I should mention by the way the first time you run that code


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Pre trained model from the internet, so you'll see the first time you run it. It'll say downloading model


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
The first time you run it it pre computes and caches some of the intermediate information that it needs


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
To download and pre compute stuff that's normal if you run it again. You should find it takes 20 seconds or so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Need to recognize cats versus dogs very often on a computer. You can probably do it yourself pretty well, but what's interestingly?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And at for each one there was a label saying whether that go board ended up being the winning or the losing


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And that's really the key most important step in playing Gowell is to know which which move is better


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So this is how you know some of the ways you can use deep learning specifically for image recognition and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Mainframe to play against itself lots of times and figure out which kinds of things led to victories and which kinds of things didn't


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Using this machine learning approach rather than written by hand by writing the loops and so forth by hand


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
I think one of the reasons it didn't happen for a long time is because traditional machine learning actually was very


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Knowledge and time intensive so for example. Here's something called the computational pathologist or C path


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Relationship between epithelial and stromal objects and so forth and so they came up with all of these ideas of features


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Features were passed into a logistic regression to predict survival, and it ended up working very well


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And it ended up that the survival predictions were more accurate than pathologists own survival predictions work


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We're going to try and create an infinitely flexible function a function that could solve any problem


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Now the neural network. We're going to learn all about it and implemented ourselves from scratch later on in the course


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You get something called the universal approximation theorem and the universal approximation theorem says that


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
okay, so now we need some way to fit the parameters so that this infinitely flexible neural network solves some specific problem and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Probably most of you will have come across before at some stage called gradient descent and with gradient descent


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Let's figure out a slightly better set of parameters and a slightly better set of parameters and j6v follow down


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
These things a court local minima now interestingly it turns out that for neural networks particularly in particular


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
There's basically just one right or think of it another way there are different parts of the space which are all equally good


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Which is I mentioned that these neural network so you can intersperse multiple sets of linear and then nonlinear layers?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Something that we learned in the last few years is that these kinds of neural networks, although they do


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
number of parameters to do so so they don't actually solve the fast and scalable for even reasonable size problems


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That is where it becomes called deep learning, so deep learning means a neural network with multiple hidden layers


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
They actually hired Geoffrey Hinton who's kind of the father of deep learning and his top student Alex Bogusky?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And so what you see now is around Google that deep learning is used in like every part of the business


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That I could send for you and so it's actually using deep learning here to read the original email and to generate


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Translate it in real time to Chinese or Spanish and then when they talk back to you in Chinese or Spanish Scott will in real-time


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So here's an example of low drawing something just sketching it out, and then using a program called neural doodle


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
This is from a couple of years ago - then say, please take that sketch and render it in the style of an artist


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Think learning and tried to use it to solve a really important problem and so the problem. I picked was


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So it's a really important problem to solve so I got together with three other people none of us had any medical background


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
and we went on to build this in a start-up in just into a company called analytic, which has really become pretty successful and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Of kind of things that people do in the world currently aren't using deep learning and then each time somebody says oh


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
They nearly always get fantastic results and then suddenly everybody in that industry starts using it as well


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You know are things, which people spend a lot of money on or have a lot of you know important business opportunities


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That these are some examples of things that maybe your company you could think about applying deep learning for


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
so as I briefly mentioned the thing we created is something called a convolutional neural network or CNN and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
kind of in fact this over here in the bottom left is a very zoomed in picture of somebody's face and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Now how is it doing that? well, if we look at each of these are all three by three areas. This is moving over


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Black through white pixel values and we multiply as you can see each of them by the corresponding value in the kernel


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So as you can see it is a linear operation and so based on that definition of a neural net I described before


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And we're going to look much more at convolutions later including building a little spreadsheet that implements them ourselves


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
turns it into some different value in a nonlinear way, and you can see this orange picture here is an example of a


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
nonlinear function is that it allows us to create arbitrarily complex shapes as you see in the bottom right and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
The reason why is that and this is all from Michael Nelson's neural networks and deep learning com really fantastic


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And this is actually the essence of the universal approximation theorem this idea that when you have a linear layer


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Set these parameters, so it's all very well knowing that we can move three a meters around manually to try to


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Let's pick here, and so you go up there, and you calculate the value of your quadratic at that point


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
To find out which direction is down, and what you'll find out is that moving a little bit to the left


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Decreases the value of the function so that looks good and so in other words, we're calculating the derivative


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
It's the gradient and so now that we know that going to the left is down. We can take a small step in that direction


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Which way is down now and we can now take another step and another step and another step another step another step, okay?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And we're going to be writing all this stuff out in code from scratch ourselves, but this number here is called the learning rate


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
This is an example of basically starting out with some random line and then using gradient descent to gradually make the line better and better


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
The non-linearity and gradient descent because they're all tiny small simple little things it doesn't sound that exciting


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So, this is a really interesting paper by Matt Siler and Rob Fergus and what they did a few years ago


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
was they figured out how to basically draw a picture of what each layer in a deep learning net Network learned and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Simple dat or grid patterns some of them learnt these simple gradients right and so for each of these filters


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
These learnt or ever these these are learnt using gradient descent these filters were not programmed


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Had you know this is like my in kind of attempts to draw one of the examples of the filters in layer two


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
What are examples of little bits of images that activated them, and you can see by layer two? We've got?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Basically something that's being activated nearly entirely by little bits of sunset some things that's being activated by


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Another filter has learnt to recognize the presence of human faces right so just three layers is enough to get some pretty


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We've got something that can recognize the eyeballs of insects and birds and something that can recognize


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Thanks to the universal approximation theorem and the use of multiple hidden layers in deep learning


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So we've learnt the idea of like we can look at the pictures that come out of the other end to see what the model


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
I just caught it L before the learning rate, and you might have noticed. There's a couple of numbers these kind of magic numbers


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Right so this number is how much do you want to multiply the gradient by when you're taking each step?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Right, but probably also. It's obvious to see why you wouldn't want it to be too low if you had it too low


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
too little step little step, and it would take lots and lots and lots of steps, and it would take too long so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Unfortunately almost nobody noticed so almost no deep learning researchers. I know about actually are aware of this approach


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And I'll show you the idea right it's built into the fast AI library as something called @lr find or the learning rate finder


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
cyclic or learning rates for training neural networks by a terrific researcher called Leslie Smith and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Basic idea that we've seen before which is if we're going to optimize something pick some random point


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Start doing almost nothing right and it's going to then suddenly the loss function is going to improve very quickly


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
I basically mean like how accurate is the model how close in this case the loss would be how far away is the?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And so if we plotted the learning rate against the loss we'd say like okay, initially it didn't do very much


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We're not saying like where was it the lowest because that could actually be the point where it's just jumped too far


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So if you create your learn object in the same way that we did before we'll be learning more about this these details shortly


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
That's the object which actually does this learning rate finding and that object has a plot learning rate function and so you can see here


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Over by iteration you can see the learning rate alright, so you can see each step the learning rate is getting bigger and bigger


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You can do it this way we can see it's increasing exponentially another way that Leslie Smith the researcher suggests is to do it linearly


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Recently I've been mainly using exponential, but I'm starting to look more using Linea at the moment


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
What you might find happen is if you run it for too long the accuracy you'll start getting worse all right


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
you know how many epochs you can run and the other thing that might happen is if you've got like a really big model or


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
a lot lots and lots of data, maybe it takes so long you don't have time and so you just run enough epochs that


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Fit into the time you have available so the number of epochs you run. You know that's a pretty easy thing to set


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
how does the number of the learning rate change the accuracy you get and so forth like really experiment and then?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You know we're going to be digging into a lot more detail, and so you'll really feel ready to do that now


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Numpy, the fastai library, and so forth. and so I want to show you some tricks in Jupiter notebook to make that much easier


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
So one trick to be aware of is if you can't quite remember how to spell something right so if you're not quite sure


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Then it actually pops up a whole little separate window with that information. Okay, so shift tab is super helpful!


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
We're going to be looking at the actual source code of fast AI itself and learning how it's built and why it's built that way


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Half a screen full of code, and they're often under six lines, so you can see this case it's calling predicted with tags


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
and then that's calling a function called predicted with tags so we could get that documentation for that in the same way and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
And then finally that's what it does it either rates through a data loader gets the predictions and then passes them back


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
You actually have to stop an increase or or in paper space and don't forget to do that or you'll end up being charged until


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: IPBSB1HLNLo
c3:
Right because that is going to tell you about like things that have changed. Okay, so if there's been some change -


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But that one oh, that's great, sorry, I don't know your renders. Oh, okay great. That's better isn't it me?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So just to remind you the way that we built this image classifier was we used a small amount of code


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
was that this path which was data dogs cats and had to have a particular structure, which is that it had a


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And if the cats on the docs folders was a bunch of images of cats and dogs, but this is like a pretty standard


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And generally that seems to be working pretty well from what I can see on the forums so to make it clear


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
What was it uncertain about? What was it wrong about and so forth and that's always a really good idea


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Is the one key number is 0.01 and this is called the learning rate, and so I wanted to go over this again


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
During the rest of the course in quite a lot of detail and for now I just wanted to talk about the practice


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We're going to talk about the other other ones shortly so the main one we're going to look at for now is is the last


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
First column as you can see is the epoch number so this tells us how many times has it been through the entire dataset


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Trying to learn a better classifier and in the next two columns is what's called the loss which we'll be learning about


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And the second is the loss of the validation set these are the images that we're not looking at and we're training


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But we're just sitting on the side to see how accurate we are so we'll learn about littering loss in accuracy


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So the basic idea of the learning rate is it's the thing that's going to decide how quickly do we zoom do we kind of


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Hone in on the solution, and so I find that a good way to think about this is to think about like well


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Looks something like this right. We're trying to say okay. Where's where abouts is the minimum point?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This is basically what we do when we do deep learning is we try to find the minimum point of a function


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Millions, or hundreds of millions of parameters, but it works the same basic way, and so when we look at it


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But how would you do that if you are a computer algorithm and what we do is we we start out at some point


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It tells us that down is going to be in that direction and it also tells us how fast is it going down?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Proportional to the gradient sort of unfortunately how steep it is the idea is if it's deeper, then we're probably further away


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which is how steep is it at this point, and we multiply it by some number and that number is called the learning rate?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Then we're guaranteed that we're going to go a little bit closer and a little bit closer and a little bit closer each time


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Right, but it's going to take us a very long time to eventually get to the bottom if we dig a number


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's very big. We could actually step too far could go in the right direction, but we could step all the way over to here


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Oscillate and get worse and worse so if you start training a neural net and you find that your accuracy or your loss is like


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Is a better problem to have because you're going to have to wait a long time, but wouldn't it be nice


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
What's the best learning rate something where you could kind of go quickly go like Bom Bom Bom right and so?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Mini-batch how many batches a few images that we look at each time so that we're using the parallel processing power of the GPU


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We started really really tiny learning rates to make sure that we don't start at something too high and we gradually


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Eventually the learning rate will be so big that the loss will start getting worse and so what we're going to do


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Python we'll generally write it like this rather than writing 10 to the negative 1 or 10 to the negative 2


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Here is the the final loss at the very at the end of it's not of any interest right so ignore this this is only


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's not interesting for the learning rate finder the thing that's interesting for the learning rate finder. Is this loan shed plot and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Specifically we're not looking for the point where it's the lowest back to the point where it's the lowest it's actually not getting better anymore


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So I generally look to see like where is it the lowest and then I go back like one for magnitude so?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You just adjust this number at nothing else most of the time you're going to be able to get pretty good results


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And this is like a very different message to what you would hear or see in any textbook or any video or any course


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
there's been like dozens and dozens of these they're called hyper parameters dozens and dozens of hyper parameters to set and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
During the course we're going to learn that there are some more we can quake to get slightly better results


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
They'll be like deep learning so difficult as someone to say it's a real art form and so that's why there's this


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
as is difference right and so that the truth is that the learning rate really is the key thing to set and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Who's not from a famous research labs so most people kind of ignored it and in fact even this particular technique was one


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Really nobody outside this classroom just about knows about it obviously the guy who wrote it Leslie Smith knows about it. Yeah


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So it's a good thing to tell your colleagues about is like here is actually a great way to set the learning rate and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
There's even been papers caught like one of the famous papers is called no more pesky learning rates


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And if that doesn't quite work you can always try you know going back another multiple ten, but this is always worked for me


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Or what's like the advantages a disadvantage with just learning rate rate like technique. We're just feels


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This is orthogonal in fact so one of the things the faster a library tries to do is figure out the right


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This is actually using something called Adam and so this technique is telling us. This is the best learning rate to use


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So it's not that there's some compromise between this and some other approaches who sits on top of those approaches


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
optimizers like Adam ditched Harvick adaptive learning rate so and he said this learning rate is Italy initial learning rate because it changes during the


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So there's still like a number called the learning rate there. There. Isn't a even these so called dynamic learning rate


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Do what's called overfitting and so overfitting means that they're going to start to see like the specific details of the images you're giving them?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Now obviously one way to do that would just be to collect more data from where you're getting it from or label more data


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And so it's built into the faster you library to make it very easy to do and so we're going to look at the details


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
To understand data augmentation, it's may. Be easiest to look at some pictures of data augmentation so what I've done here again


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Data class like multiple times. I'm going to do it six times and each time. I'm going to plot the same catch and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This cap here is further over to the left this one here is further over to the right and this one here is fit


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Different types of image you're going to want different types of data augmentation right so for example if you were trying to recognize


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Letters and digits you wouldn't want to flip horizontally because like it's actually has a different meaning


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Photos of cats and dogs you probably don't want to fit vertically because cats aren't generally upside down all right


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And slightly zoom in and out a little bit and move them around a little so each time it's a slightly different fight billionaires


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This works for every CNN for CNN every minute. There's a running right fine done. Yeah, exactly, yeah


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's less than the lowest point and so the reason why is to understand what's going on with this learning rate finder


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Like how do we figure out what learning rate to use right and so what we're going to do is we're going to take


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The learning rate so kind of double the amount by which we multiply the grander gradient so in other words would go


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So instead what we do is we go back to the point where the learning rates quickly are giving us a quick increase


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So here is so here is the actual learning rate increasing every single time. We look at a new mini batch


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Then here is learning rate versus loss, so here's that point at the bottom. Where is now already too high?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Shortly where we're going to see like in a sense you might want to go back to 1 enoch 3 where it's actually even steeper


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Mean I am learning rate so as we increase the iterations from the learning rate finder the learning rate is going up


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Because I asked the question because you were just indicating that you know even though the minimum was at 10 to the minus 1


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This graph also shows the same is that is this something that was observed, or is the logic theory behind it


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
After I unfreeze layers, I sometimes run it again if I do something to like change the thing I'm training or change the way


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Transform side on or transforms top down later on we'll learn about creating your own custom transform lists as well


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
but for now because we're taking pictures from the side cats and dogs will say transform side on and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Now each time we look at an image. It's going to be zoomed in or out a little bit moved around a little bit


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Possibly flipped okay, and so what this does is it's not exactly creating new data, but as far as the


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Convolutional neural net is concerned. It's a different way of looking at this thing and it actually therefore allows it to learn


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
From somewhat different angles right so when we do data orientation. We're basically trying to say based on our domain knowledge


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Here here are different ways that we can mess with this image that we know still make it the same image


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which we'll learn more about shortly we can now pass in this set of transforms, which actually have these augmentations in


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Augmentations actually don't do anything and the reason initially they don't do anything is because we've got here something that says


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But basically what this is doing is do you remember this picture we saw where we learn each different layer


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The the later layers of this convolutional neural network have these things called activations and activation literally


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We've got a pre trained network remember and a pre trained network is one where it's already learned to recognize


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Certain things in this case it's learnt to recognize the one and a half million images in the imagenet dataset


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
What kind of thing a thing is and we can save those activations so basically saving things saying you know?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We can just very quickly train when all the details there shortly we can very quickly train a simple linear model


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It uses a special that all had a scratch space that disappears each time you restart your press or instance


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Lizard's eyeball there and so forth that means that data augmentations don't work right in other words even though


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We're trying to show at a different version of the cat each time. We've pre computed the activations for a particular version of that cat


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Loss is much lower than the validation loss, and we're going to talk about that a lot during this course


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
recent fairly recent discovery and deep learning called stochastic gradient descent with restarts and the basic idea is this as


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Now getting closer and closer I may want to start to decrease my learning rate right because as I get closer


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I'm kind of like oh, I'm pretty close down, so let's let's slow down my steps to try to get executive the right spot


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Down right because as we go along we're getting closer and closer to where we want to be and we want to like get exactly


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Really horrendously hacky. It's basically that researchers like pick a learning rate that seems to work for a while


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And then they keep learning a bit more until it doesn't seem to be improving and they drop it down by another ten times


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's what most academic research papers and most people in industry do so this would be like stepwise annealing very manual


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You kind of quickly drop down and do a few iterations with a really low learning rate, and so this is called cosine annealing


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So to those of you who haven't done trigonometry for a while cosine basically looks something like this right, so we've picked


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
High dimensional space right near we're only able to show three dimensions right but in reality. We've got hundreds of millions of dimensions


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Fairly flat points there. No not the actual local minima, but they're fairly flat points all of which are pretty good, right?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which is to say if we use a different data set where things are just kind of slightly different in one of these directions


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But it rather suggests that if you move if you have slightly different data sets that are slightly moved in different directions


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It's still going to be good right so in other words. We would expect this solution. Here is probably going to generalize better


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Loading rate annealing approach will start of go down here or downhill downhill downhill downhill to one spot


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which is to say we do a cosign annealing and then suddenly jump up again into a cosign annealing and then jump up again?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Could you get the same effect by running multiple iterations through the different ground of my starting point so that


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's exactly what people used to do. They used to create these things called ensembles where they would basically


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Relearn a whole new model ten times in the hope that one of them's like but it ended up being better


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The cool thing about this decosta gradient descent with restarts is that the model once? We're in a reasonably good spot


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It doesn't restart it actually hangs out in this nice part part of the space and then keeps getting better


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Didn't pick the point where it's got the steepest slope. I actually trying to pick something kind of aggressively high


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So I have a few questions, but the first one is how many times do you change your learning rate you want to work?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We change the learning rate every single mini - all right, and then the number of times. We reset it is set by the


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Interestingly this this point that when we do the learning rate and kneeling that we actually change it every single batch


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Really critical to making this work, and it again is very different to what nearly everybody in industry in academia has done before


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Multiple times in this course so the way this course has been a work is we're going to like do a really high-level


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Version of each thing and then we're going to like come back to it in two or three lessons and then come back to it


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
at the end of the course and each time we're going to see like more of the math more of the code and get a


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Demas yeah, that's a it's a very short. Summary. This method are we keeping track off to minimize and averaging them? Ah?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Level of sophistication and indeed you can see there's something here called snapshot ensemble, so we're not doing it in the code right now


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You'll be straight back to where you came from okay, so it's a good idea to do that from time to time


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
When you go learn dot save when you create precomputed activations another thing we learn about soon when you create resized images


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So this is automatically created and all of my pre computed activations end up in here. I mention this because if


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
If things are if you're getting weird errors that might be because you've got some Oh pre computed activations like we're only half


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You'll also see there's a directory called models, and that's where all of these when you say dot save with a model


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I saw a tweet that was somebody was like Oh to make your deep learning work better turn it off and then on again


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So if I want to see I want to retrain my model fuselage again do I just do everything the 10 folder if you want


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
There's generally no reason to delete the pre computed activations because the pre computed activations are


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Yeah, I mean the only reason you want to delete the pre computed activations is that there was some error caused by like half


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
as you change the size of your import change different architectures and so forth they all create different sets of


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Has not changed any of these free trained filters right so we've used a pre trained model that already knows how to find


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Some new layers that we've added on top of these things. We've learned how to mix and match these pre-trained features


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We're going to have some minor differences, but we still may find. It's helpful to slightly tune some of these


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So to tell the learner that we now want to start actually changing the convolutional filters themselves


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We simply say unfreeze okay, so a frozen layer is a layer, which is not trained is not updated, okay?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Probably doesn't need to change by much if at all right from the 1 and a half million images on image net it probably already


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It probably already knows also like which kind of corners to look for and how to find which kinds of curves and so forth


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Where else these later ones are much more likely to need more learning, and this is universally true regardless of whether you're looking for


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Where we say okay? These are the learning rates to use for our additional layers that we've added on top


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Kind of sophisticated convolutional features, and these are the ones that are used for the features that we've added and went from stretch


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Then when we called up fit and pass an array of learning rates. It's now going to use those different learning rates for different


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But I'd also say it's like it's so not that common that it doesn't even have a name as far as I know so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
If it actually has a name or indeed if somebody's actually written a paper specifically talking about it. I don't know


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Great researcher called Jason your Sinskey who who did write a paper about the kind of the idea that you might want different learning rates?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The it's your answer is many many right and they're kind of in groups, and we're going to learn about the architecture


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Yeah, they're ones closest to the pixels that represent like corners and edges and gradients, but why


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Why do you well I thought those layers are frozen at first so yeah right so we just said unfreeze the streets


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Although lately we've trained. We've trained our added layers. Yes, now you are we training the Oh step exactly?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's right, because you just find a want to find food. Yeah. Yeah, we probably don't want to change them at all


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
There's no similarity to grid search so grid search is where we're trying to find the best hyper parameter


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Learning rate finder as a really sophisticated grid search, which is like trying lots and lots of learning rates to find, which one is best


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This is actually for the entire training from now on it's actually going to use a different learning rate for each layer


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
They used or we're going to be talking about sizes later, but the short answer is that with this library and the modern


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We can we're not doing it yet, but if you wanted to you can learn dot freeze underscore two and pass into layer number


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The the less layers you unfreeze the less memory it takes and the less time it takes so there's that kind of practical aspect


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You could probably delve inside the library in phase one phase one layer, but I don't know why you would


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
and we've kind of found these like tiny number of tricks and so once we do that we now go learn about fit right and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
There's one other trick you might see here that as well as using stochastic gradient descent with restarts a cycle length equals one


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Cycles of each of two epochs or do six because so here I've said two three cycles yet somehow. It's done seven epochs


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The length of the cycle after each cycle and so in the paper that introduced this stochastic gradient descent with restarts


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And then it pops out and it goes to a try and photographs button pops out it never actually gets to find a good spot


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's like smoother, but then later on you want it to fight do more exploring and then more exploring right so that's why this


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Suddenly we're introducing more and more hyper parameters having told you that there aren't that many?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
With no molt. Okay. There's kind of like two things that seem to work a lot, and there's not too much fiddling


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This intuitive explanation, I try to just kill the whole thing. I try to give back here, which is that?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Have a slightly different relationship between this parameter, and how caddy versus dog it is it may instead look a little bit like this


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Right then it's not going to do a good job on this slightly different data set for else if we end up on this point


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Okay, so that's what psychomotor equals doing okay, so we've got one last thing before we going to take a break


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
which is we're now going to take this model, which has 99.5 percent accuracy and we're going to try and make it better still and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
what we're going to do is we're not actually going to change the model at all right, but instead we're going to look back at


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Now what I've done is I've printed out the whole of these incorrect pictures, but the key thing to realize is that?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But basically the GPU doesn't go very quickly if you have like different dimensions for different images because it needs seems to be consistent


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's the state of the technology we have so our validation set when we actually say for this particular thing is it's a dog


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And then as much of the middle as we can and so you can see in this case we wouldn't actually see this dog's head


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I think the reason this was actually not correctly classified was because the validation set only got to see the body and the body


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Doesn't look particularly doglike or cat-like. It's not at all punctual what it is, so what we're going to do


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
when we calculate the predictions for our validation set is we're going to use something called test time augmentation and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Augmented sent a cropped image, and we're going to do a prediction for all of those, and then we're going to take the average


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Augmentation this term of like what a what do we call up when we're making predictions from up from a model?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We go learn TTA check the accuracy and lo and behold we're now at ninety nine point six five percent, which is kind of crazy


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You could and sometimes like I've written libraries where after a cheap up I run TTA to see how well it's going


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We were randomly changing it a little bit so each epoch each of these seven epochs. It was seen slightly different versions of the picture


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Predictions with different random orientations plus one on the organ under Augmented version average them all together


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Yeah, actually every data augmented for image is is unique because the rotation could be like point zero three four


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So like there's lots of different types of a better orientation you can do and so one of the things you can do is to


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Basically adding a border around it in my experiments doesn't doesn't help it doesn't make it any less


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Cat-like it's not the convolutional neural network doesn't seem to find it very interesting basically


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Something that I do do we'll see later is I do something called reflection padding, which is where I add some borders


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That are the outside just reflected. It's a way to kind of make some bigger images works well with satellite imagery in particular


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
you now have to use more GPU and if you're going to use more that multi figure you could have zoomed in and used a


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Asked some of my friends in the natural language processing community about this. We'll get to natural language processing in a couple of lessons you


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Example I carry very few number examples of people where papers would like try replacing synonyms for instance


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The question was could couldn't we just use a sliding window to generate on the images so in that dog


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
One degree off five you know five degrees off ten pixels up like lots of slightly different versions


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And so if you just have three standard ways then you're not giving it as many different ways of looking at the data


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And I just haven't gotten around to writing that yet. I have a version in an old library. I think having fixed cropped locations


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The reason I've got around to it yet is because in my testing it didn't seem to help him practice very much


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Yeah, that's a great question so the faster you go libraries open source and let's talk about it a bit more generally because


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
makes it very hard to learn deep learning when you have to write hundreds of lines of code to do anything so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
really needed PI torch or at least with PI torch it was far easier and but then PI thought itself just wasn't suitable as a


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which is what we taught last year and but then we realized we could actually make things much much much easier than care us


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So we ended up kind of building this this this library in order to make it easier to get into deep learning


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Then over the last year as we started developing on top of that we started discovering that by using this library


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It made us so much more productive that we actually started kind of developing you state-of-the-art results and new methods ourselves


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And we started realizing that there's a whole bunch of like papers that have kind of been ignored or lost which


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
When you use them, it could like automate or semi-automated stuff like learning read finder. That's not in any other library, so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It's like very early version and so through this course by the end of this course. I hope as a group


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Behind the scenes it's it's creating play torch models and so apply torch models can then be exported


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Having said that like a lot of folks like issue if you want to do something on a mobile phone for example


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
do in chaos and cancel flow so you can going to get a sense of what the different libraries look like and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
To learn to do it and care us in tensorflow versus fast AI and high torch and the more complex stuff often


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This won't be possible so that like if you needed to be intensive flow you're just kind of simplify it off in a little bit


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Every year the kind of the libraries that are available and which ones are the best totally changes so like the main thing


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I hope that you get out of this course is an understanding of the concepts like here's how you find a learning rate?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's built on top of paper so one of the things we'll learn about in this course is we'll see that PI torch is much


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
From scratch which we're actually going to do and pyro is a great example of what people are now doing with might watch


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This is not just deep learning, but in any kind of classifier machine learning where we say okay. What was the actual truth there were?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It turns out there were nine hundred ninety-eight cats that we actually predicted as cats and two that we got wrong okay


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And then for dogs there were nine hundred ninety-five that we predicted were dogs and then five that we got wrong and so often these


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Confusion matrices can be helpful particularly if you've got like four or five classes you're trying to predict to see like which


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's better to actually look back and see like okay, which ones in particular were incorrect and we can see here


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So they weren't they weren't wrong, okay, so it's actually these two were wrong cats and this one isn't obviously a cat at all


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This one is but it looks like it's got a lot of weird artifacts, and you can't see its eyeballs at all so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
How many dogs where they're all wrong there were five wrong dogs here are four of them? That's not obviously a dog


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That looks like a mistake that looks like a mistake that one. I guess doesn't have enough information that I guess it's a mistake so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I can tell you it's a state of the art classifier. It's it's right up there with the best in the world


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Classifier the steps that we just went through was that we started our week's term data augmentation on


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
By saying oil transforms equals, and you either say sidon or top-down depending on what you're doing


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Then we turn off pre compute, which allows us to use data augmentation to do another two or three epochs


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Either somewhere between a 3 times 2 10 times mobile learning rate in the previous so in this case I did


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
10 times right so it's like this was my learning rate that I found from the learning rate finer than I went 10 times smaller


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You know if you can see that the things that you're now trying to classify a pretty similar the kinds of things in imagenet ie


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You probably want about a 10x difference because you want those earlier layers like you think that the earlier layers are probably very good already


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which is not at all like image net then you probably want to be training those earlier layers a lot more so you might have


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But like once you've unfrozen all the layers you've turned on differential learning rates. You can then call a lot of fine


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And so you can then check like oh does it still look like the same point. I had last time is about right


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The thing that's actually going to print out is the learning rate of the last layers right because you've got three different learning rates


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
then I trained the full network with cycle more equals two and it'll either it starts with the fitting or I


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I noticed that some of you on the forums were playing around with this playground Kegel competition very similar


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
They're all dogs, but you have to decide what kind of dog. It is, but there are 120 different breeds of dogs


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Different kinds of icebergs and satellite images whatever right as long as you've got some kind of labeled images


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which is a github project you can search for and if you read the docs to basically run cagey download?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Provide the competition name and it will grab all the data for you to your crystal or Amazon or whatever instance


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This is the second main way that people kind of give you image labels one is to put different images into different folders


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
the second is generally to give you as some kind of file like a CSV file to tell you here's the image name and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Just to see how many of each breed there were and I sorted them, and so I saw okay, they've got like about a hundred


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Enable data augmentation so to enable data augmentation when we call this transforms from model you just pass in and all


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Transformers in this case I chose side on again. These are pictures of dots and stuff so this side on photos I


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
More detail later, but maximum basically says when you do the data augmentation we like zoom into it by up to


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
One point one times okay, so but randomly between one the original image size and one point one points


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So it's not always cropping out in the middle or an edge, but it could be cropping out a smaller part, okay, so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So previously we went from paths and that tells it that the the names of the folders are the names of the labels we go


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We need to also tell it where the test set is if you want to submit to cattle later talk more about that next week


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Rows in the CSV file which must be the number of images that we have and then this is a fast AI thing get cross-validation


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Indexes now. We'll talk about cross-validation later, but basically if you call this and pass in a number. It's going to return to you


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
by default a random twenty percent of the rows who uses your validation set and you can pass in parameters to


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We can pass in a parameter, which is talent, which indexes to treat us a validation set and so that's passed in those indexes


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Actually have I checked they actually have a dot jpg on the end and these obviously don't have a dot jpg


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It says that the labels don't actually contain the full file names. You need to add this to them, okay


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Inside that data object you can actually get access to the data set like what the training data set by same train yes


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So train desktop file names contains all of the file names of everything in the training set and so here's like one file name


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's the next thing I did was to try and understand what my file my dataset looks like and it found an adorable puppy


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's a key issue if they're huge, and then I have to think really carefully about how to deal with huge images. That's really challenging


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's that's really hopeful. You're probably not going to have to do too much different in this case the first image


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So what I did, then is I created a dictionary comprehension now if you don't know about list comprehensions and dictionary comprehensions in Python


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Is that are going through all of the files and then putting a dictionary that map's the name of the file to?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Again this is a handy little Python feature, which I'll let you think learn about during the week if you don't know about it


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
which is zip and using a special star notation is never to take this dictionary and turn it into the


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Then matplotlib is something you want to be very familiar with if you do any kind of data science or machine learning in python


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The histogram so I can actually go through and I can see here for four thousand five hundred of them are about 450 okay


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Using 20% is fine unless you kind of feeling like my data is my data. Sets really small. I'm not sure that's enough


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And you're getting very different validation set results and your validation sets kind of small but smaller than a thousand or so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And you've got like a thousand things in your validation set then you bring about like a single image changing class is changing


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Would say in general like at the point where you care about difference between like out of 0.01 and 0.02 like the second decimal place


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You know so those who are interested in this kind of thing we're going to look into it a lot more detail in our machine


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Results and checked in and again found they're kind of like 4 or 500 seem to be about the average size


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I kind of thought ok this looks like a pretty normal kind of image data set that I can probably use pretty normal kinds of


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That the dog like takes up most of the frame right, so I'm not too worried about like cropping problems. You know


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
If the if the dog was just like a tiny little piece of one little corner that I'd be thinking about doing different


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Like a medical imaging that happens a lot like often the tumor or the cell whatever is like one tiny piece and there's much more


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Went ahead and created a little function called get data that basically had my normal two lines of code in it


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
the reason for this is that when I start working with new data set I want everything to go super fast and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Architectures at which point I started running out of GPU memory, so I started getting these errors saying


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
CUDA out of memory error when you get a CUDA out of memory error the first thing you need to do is to go kernel


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
restart once you get a code an out of memory error on your GPU you can't really recover from it right doesn't matter what you


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
so that's where I created this to allow me to like start making my size as bigger as I looked into it more and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You know I went through this a couple of iterations, but I basically found everything was working fine


