Why are people picking the layer shapes they are when designing neural networks? For example, convolutional NN ``VGG-16`` essentially moves from 224 -> 112 -> 56 -> 28 -> 14. How would the properties of the network be different were I to start from 14 (of course same input size) and fan out to 224? What would happen if we did more a u-shape, where we shrunk down then grew back out? Are there any general rules here?
NN designer for autonomous vehicles here, you are overthinking it. It's just a big stew of random numbers, sometimes if you aren't getting the results you want you have to stir it up a little bit.
>>10646780Surely there's some kind of effect?
>>10646787You're getting it all wrong. A NN is organic. Stop trying to think of it like some kind of equation. It's more of a living thing, like a pet monkey. When it does something good you give it a little treat and reinforce the good behavior. When it does something bad you slap it a bit and give it a mild sedative. But you can't just go in and discombobulate it's brain and give in 26 arms like you are suggesting. You're trying to circumvent artificial intelligence by using normal logic (human intelligence). The less you know about the inner workings of the NN the better it is.
>>10646812Then how do these people even come up with these kinds of schemes to begin with? A dartboard they don't mention in their paper? Why isn't it common practice to just determine these parameters with a pRNG?
>>10646812>just a black box lol who cares>computer sciencewow really makes you think!
>>10646842If you have a problem with ML, take it up with Bayes, asshole
>>10646827Everything about NNs makes more sense when you realize they are actually just regression curves in lots of dimensionsNeural network training is just curve fitting in lots of dimensions, which is significantly harder than curve fitting in very few dimensions, but still just curve fitting.
>>10646850the neural network is like a monkey that you feed treats. why do monkeys like treats? who knows. I sure don't.
There is no point or advantage to it. They just picked what felt convenient. The only researchers who have any clue what they're doing are neuroscientists but they don't know how to code. VGG-16 is ancient trash by the way.
>>10646890yeah, but it was used as the basis for the CNN in a 2018 CRNN model I'm looking at dealing with sheet music reading. I'm just thinking about how to tweak their model. What I might do is stick in some dropout layers or try and do some residual style networking.
>>10646812I think he's asking why hidden layers are constructed the way they are.
>>10646771What are you talking about, channels (i.e. the number of filters) or image dimensions (i.e. width x height)?Image dimensions go down because it's the only way neurons can capture a broader field of view. Neurons in the first layer of filters can only see pixels from an area of size equal to the filter kernel. That is, if you have a 3x3 kernel each neuron is only seeing 9 pixels. If you then do a convolution with another 3x3 kernel on that layer as well, each neuron is seeing 9 neurons from the previous layer, and is, in turn, seeing a total area of 5x5 pixels from the input image. You may want to speed up this process with pooling layers (I don't know VGG-16's architecture by memory, but from the numbers it looks like they're using pooling layers) because sometimes you don't need to compute more features of features, you just want to extend the field of view quickly.In terms of channels, it usually goes the other way (from less to more). You start with a small number of channels that can capture basic features (borders in different directions), and you then build increasingly more features on top of those.>>10646780>>10646812I would be ashamed if I were working with NNs and I wasn't able to answer OP's question.
>>10646890>The only researchers who have any clue what they're doing are neuroscientists but they don't know how to code.Yeah, that's definitely true. As a programmer most everything I do is a hacky workaround concerned more with getting a job done than with discerning the mathematically optimal / ideal approach for it. Scientists in contrast are the kinds of people who use functional languages like Haskell and employ formal techniques to come up with the genuine best solution for a problem but can't do anything practical with the results and end up storing it in Microsoft Excel in a way that saves their IDs as a date field thereby irreparably destroying years of research.
>>10646931>I would be ashamed if I were working with NNs and I wasn't able to answer OP's question.Most actual programmers don't care at all about the theory stuff.
>>10646890>>10646937This is only partially true. There are a ton of DL researchers that are clueless and just start randomly changing layers until they get a good accuracy result. Good researchers are making progress by actually innovating on NN architectures and such. There's a reason behind pooling layers, as I said in another comment. I don't know what to say about your comment on neuroscientists.>>10646941Which is stupid, because if you're just hiring someone to tune hyperparameters you're better off using a Bayesian optimizer, which doesn't ask for a salary. ML workers that can be replaced by ML. Ironic.
>>10646955>Which is stupid, because if you're just hiring someone to tune hyperparameters you're better off using a Bayesian optimizer, which doesn't ask for a salary. ML workers that can be replaced by ML. Ironic.Most programmers who work with neural networks don't ONLY work with neural networks. Which is another reason why programmers really don't care that much about the theory stuff here. It's almost never the entirety of their job.
>>10646979I see. I'm on the research side of things, so I'm not very knowledgeable about how things work in the industry.Even though you probably don't need much for stuff like market analysis, I would expect someone who's working on autonomous driving to know the theory.
>>10646991>I see. I'm on the research side of things, so I'm not very knowledgeable about how things work in the industry.Yeah, non-research programming is total anarchy. Your boss very likely won't be a programmer himself so you aren't having any incentives for doing anything theoretically interesting or impressive. The interest is 100% on results. Also most of the working programmers out there are self-taught. There are constant attempts to try to impose order on this scenario through software development paradigms like this:https://en.wikipedia.org/wiki/Scrum_(software_development)But they're all incredibly retarded, not at all helpful, and inevitably subverted and circumvented by all parties involved.>I would expect someone who's working on autonomous driving to know the theory.To be fair I definitely haven't personally done work on anything related to autonomous driving like that other anon said he does. But I imagine there's plenty of work that needs to be done for that topic which doesn't involve ML theory. You gotta have logistical concerns / tasks and mechanical details that need attending to. The ML might be the most interesting part that allows for the new sort of technology to work, but it's probably dwarfed by the number of non-revolutionary but still very important non-ML tasks that need to be accomplished to bring everything together in a practical real world environment kind of way.
>>10646812Incredibly bad answer>>10646931Good answert. doing my Master's in CV an AI
>>10646812Someone that gets paid to develop NNs >>10646931>>10647222People that pay to have someone tell them how they work (and apparently are being taught 15 year old info). If you're a student keep your mouth shut when it comes to real discussion, or stick to the "whose major is better" threads.
>>10647235>payNot in Europe friendo, also I am a programmer by heart since over 10 years now.I am literally training an AI right now, I have been literally coding for the last couple of hours.What other point are you going to make?
>>10647235Please be bait.
>>10646812what a shitty answer. ironically this guy probably has the most ML experience of anyone in the thread
Oops probably should've posted here>>10647840
>>10647839It's only a shitty answer because it's not the answer some people want. He's right though because it's the answer that works. If you're trying to create some perfectly idealised NN based on some higher level logic it's doomed to failure. In many optimizations you actually use one NN designed to caluclate the best the number of hidden layers and node sizings for the actual NN you are interested in. No human will ever come up with those final values.
>>10646812Basically this. Also read https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf (parts 3 and 4) that gives some heruistics on why they designed GoogLeNet this way
>>10646771The tensorflow playground taught me that such NN train very poorly.
>>10647885He's not right. The very fact that he's working in autonomous driving (and thus with CNNs) but can't tell you what the purpose of pooling layers is should tell you he has no fucking clue about what he's talking about. He's just doing monkey labor randomly changing parameters and annotating wether the accuracy result goes up or down. Designing big NNs certainly implies doing some of that, because you can't possibly trace all activations, but there is a general logic that you can follow as to what indicators to check and what things you can try to improve your model.
>>10646812>The less you know about the inner workings of the NN the better it is.so every NN model I haven't heard of has 100% accuracy?