# transcriptions_fts_content

c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And then it ran through this in about 4 or 5 seconds, and you can see I was getting eighty-three percent accuracy


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Now remember accuracy means, it's it's exactly right and so it's predicting out of a hundred and twenty categories


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Levels you know so when I saw like eighty-three percent accuracy with just a pre computed classify and OData augmentation though


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It often looks fine, and I found it kind of kept improving so I tried five epochs, and I found my accuracy getting better


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This actually is another way you can get state-of-the-art results and I've never seen this written in any paper


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Basically if I over fit before I'm definitely not going to over fit now might have changed the size of my images


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
They're still picked the same kinds of pictures are the same kinds of things so I found this trick of like starting training on small


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Images for a few a box and then switching to bigger images and continuing training is an amazingly effective way to avoid


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I don't understand why it's never been written about before maybe it's in some paper somewhere, and I haven't found it


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
and you'll see we don't use vgg in this course because it doesn't have this property but most of the


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Okay, so I call get data again remember get data is the just a little function that I created back up here right get data


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I've now got that data augmentation working, so I kind of run a few more a pox and what I notice here


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Is that the loss to my training set and the loss of my validation set my validation set loss is a lot lower than?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It means it's like finding something better popped with popping out and it's like never getting a chance to zoom in properly


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Okay, so that's kind of thinking. Yeah. This is this is about the right track and so then I tried using test time


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Augmentation to see if that gets any better still didn't actually help a hell of a lot just a tiny bit


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Actually, you haven't tried unfreezing the reason why I was going to try to unfreezing and training more


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
set is so similar the image net that the training that convolutional layers actually doesn't help in the slightest and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This is only a playground competition, so it's not like the best of here, but you know it's still interesting


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
These people up here. I know they actually posted that they cheated. They actually went you downloaded the original images and train to that so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And this is why this is a playground competition they call it. It's not it's not real right you know


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
and we haven't even used the whole data set you're going to use to eighty percent of it like to get a better result I


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Would go back and remove that validation set and just rerun the same steps and then submit that exact


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Between sixty and a hundred like it's it's it's it's not unbalanced enough that I would give it a second thought okay?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Yeah, let's get to that later in this course and don't let me forget right the short answer is that there was a recent list


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Right right so it's not only they frozen their pre computed so the data augmentation doesn't do anything at that point


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So we're going to learn more about the details as we look into the the math and stuff in coming lessons


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
with with everything frozen and indeed with pre compute equals true all we're learning is told is those couple of layers that we've added and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Calculate like how much does this image have something that looks like this a ball one looks like this face and so forth


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You know we're actually showing exactly the same activations each time we can then set pre compute equals false


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It's still frozen, but data augmentations now working because it's actually going through and recalculating all of the activations from scratch


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And then finally when we unfreeze that's actually saying okay, now. You can go ahead and change all of these earlier


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It's about you know ten or more times faster, so particularly if you're working with like quite a large data set


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Accuracy reason ever to use pre computed calls true. It's just a it's just a shortcut. It's also like quite handy. If you're like


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
don't worry about pre compute equals true because that's just saving a little bit of time you know so so I


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
By default everything is frozen from the start so then you can just go ahead and run two or three epochs or cyclic Nichols one


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
And then train the rest of the network with differential learning rates, so it's basically three steps learning rate finder


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
We're going to see like all this stuff about precomputing batch sizes we dig into the details of the algorithms


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which means it's less accurate which means like knowing which direction to go and how far to go in that direction


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Is less accurate so as you make the batch size smaller you're basically making it kind of more volatile


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I generally find only dividing with the batch size by like 2 or 4 it doesn't seem to change things very much


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
If you if you change the batch size by much you can rerun the learning rate finder to see if it's changed if I match


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Generally looking at like a power of 10 it probably is not going to change the it's not that you can't because plus back there


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Basic questions, we're going back to the previous night where you should put in the thought behind, sorry yeah


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
This is well one for conceptual so a basic question. We've actually really slide where you should what the different layers were doing?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So these these gray ones basically say this is kind of what the filter looks like so on the first layer


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Right so you can absolutely say and remember we looked at what a convolutional kernel was like was that three by three thing


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So this look like there's seven by seven kernels you can say this is actually what it looks like but later on


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Alright, so this is kind of what the photos look like and then here is specific examples of patches of image which?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Activated that filter highly so yet the pictures are the ones that I kind of find more useful because it tells you this


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Probably a part two actually because this paper this paper uses to create these things this paper uses something called a deconvolution


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Which I'm pretty sure we won't do in this part, but we will do it in part two so if you're interested


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
figure out roughly which parts of an image and most likely to have the interesting things in them and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I was using is smaller than the average size of the image that we've been given I can increase the size and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Architecture now an architect. We're going to talk a lot in this course about architectures, but basically


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Different ways of putting together like what size convolutional filters, and how are they connected to each other and so forth?


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Different architectures have different like numbers of layers and sizes of kernels and number of filters and so forth and so


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Starting point and often a good finishing point because it's like it's pretty it doesn't have too many parameters often


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You can kind of see what this minimum service desk looks like I didn't need to worry about learning rate finder


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I didn't do the cycle mop thing because I found like now that I'm using a bigger architecture. It's got more parameters


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Pretty quickly so rather than like cycle length equals one never finding the right spot it actually did find the right spot


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So what does that mean that means? I have one incorrect dog for incorrect cats and when we look at the pictures of them


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
My incorrect dog has a cat now this one is not a either this one is not either so I've actually got one


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Dog breed one. I did this morning. I was like it was it was getting the dog breeds much better than I ever could


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Vertically and they still make sense right and so you can see here. I'm doing this trick round back to size equals 64 and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Set this is true, but it's clearly I can use super high learning rate, so I use a lot here at a point, too


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I said like if the data sets very different to image net I probably want to train those middle layers a lot more


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
so I'm using divided by three rather than divided by ten all right the other than that is the same thing cycle Nauticals -


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Pops out gets better pops out if better pops out and each time it finds something better than the last time


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
so these basic steps work super well this this thing where I went all the way back to a size of 64 I


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Destroy those imagenet weights like 64 by 64 is so small but in this case the satellite imagery data


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
It's so different to imagenet. Um you know I really found that it worked pretty well start right back to these tiny images


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I was getting like much better cackled results than really everybody on the leader board and when I say 30th place


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The last year like a lot of people have got a lot better at computer vision and so the people in the top 50 in


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So now that we've got to this point right. We've got through two lessons if you're still here then hopefully you're thinking


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I mean, it's it's it's pretty handy. It's pretty cheap and something we haven't talked about much is paper


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Space is another great choice by the way paper. Space are short. They're going to be releasing kress or like instant Drupal. Notebooks unfortunately


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
AWS educate can get credits these our office students, so there's lots of places you can get started on AWS


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
But if you've got like a model where you've done all the steps before you're thinking. This is looking pretty good


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Um we didn't start with AWS because well. Hey it's like twice as expensive as Chris Hall for the cheapest GPU


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Amazon.com and it'll take you to the console right and so you can follow along on the video with this


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You need to do what's called launching an instance so launching an instance means you're basically creating a computer


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
template for how your computer's going to begin so if you've got a community a Mis and type in fast AI


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
then we need to say what kind of computer do you want and so I can say I want a GPU compute computer and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
If you don't have a terminal if you've got a Mac or Linux box you've definitely got one if you've got Windows


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So then what I do is I copy that key somewhere that I know where it is, so it'll be in the dot SSH folder


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So if you're in a macro and Linux it'll already be in an easy to find place it'll be in your SSH folder


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So from there back in AWS you have to tell it that you've created. This key, so you can go to key pairs and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You don't have to do those depths if you've used AWS before you've already imported it. You don't have to do that step


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
SSH to it so SSH into a computer means connecting to that computer so that it's like you're typing in that computer


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
I have to do which is I have to connect up the jupiter notebook on that instance to the jupiter notebook on my machine and


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
localhost 8 8 8 8 ok so like once you've done it once you can like save that as an alias and type in the


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
So you'll find that the very first time you log in it takes a few extra seconds because it just kind of is getting everything


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Let's make sure it's got all the most recent code the second thing you should do is type Condor and update


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You can just do this maybe once a month or so and that makes sure that the libraries there are all the most recent


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Libraries I'm not going to run that so it takes a couple of minutes okay, and then the last step is to type particular notebook


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
That's it, okay, so this notebook is now actually not running on my machine. It's actually running on AWS


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
Okay, using the AWS GPU it's got a lot of memory. It's not the fastest around, but it's not terrible


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
You can always fire up a p3 if you want something that's super fast. This is costing me ninety cents a minute


c0: PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
c1: JNxcznsrRb8
c3:
The second time I went through I didn't choose p2 by mistake, so just don't forget choose gpq compute P


