In deep learning, optimizers are the secret heroes that guide models toward better performance. Whether you’re training a small neural network or a massive transformer, selecting the right optimizer is very crucial. This post explores four widely-used optimizers SGD, Momentum, RMSprop, and Adam and how they adapt parameter updates for optimal learning..
Author: Okay
I am software engineer and passionate about technology.
In particular, I like following newest software design trends and cutting-edge technological advancements.
I am perfectionist and believe that making software is an art.
Gradient descent is one of the main tools that is used in many machine learning and neural network algorithms. It acts as a guide to finding the minimum of a function. But what’s happening under the hood, and why do we need it?
Categories
Understanding Bias in Neural Networks
Neural networks are systems that mimic certain aspects of human cognition, aiming to identify and learn patterns within complex data. One crucial element that is hard to grasp in these networks is bias. Bias might seem like just another parameter, but it plays a significant role in the learning process.
I consider time as one of the most important assets of our lives, something we should use wisely. Sometimes, I’m surprised by how much time we waste when we could have used it better. One of the most interesting forms of wasted time is waiting for something or someone. In most cases, it’s up to us to speed things up or eliminate unnecessary waits altogether, yet we still don’t do much about it. Maybe it’s because we don’t care enough or don’t even realize it. I’m not saying we should rush everything, but I do believe we should question the need for certain waits.
Categories
Way of life: Time efficiency
To us humans, life does not come equally. Some are born rich, some are born poor. While some have great families and wonderful childhoods, for others it is not so. Even if we were born in the same family, but in a different country or in a different culture, we wouldn’t be the same person. These are the facts that we cannot change most of the time.
I think the only thing that treats people equally is time. Leaving the relativity theorem aside, time passes at the same speed for all of us. That is, a unit of time has the same value for everyone. What we do in this unit of time is also largely up to us. And I believe, how we use time determines the biggest difference between us.
Categories
Converting Types in Go
While programming, we often need type conversions. If you come from a language that usually does implicit conversions, converting types in Go can be a bit of a hassle.
First of all, let’s start with some basics. Go is a statically and strongly typed programming language. This means that, variable types must be known at compile time. Additionally, type conversions must be made explicitly.
Categories
Arrays and Slices in Go
Arrays and Slices are one of the fundamental and important concepts of Go programming language. They both contain sequence of types but with some differences.
Categories
Autoboxing and Unboxing in Java
In java, we often need to convert primitive types to their object(boxed wrappers) counterparts or vice versa. Unfortunately, doing these conversions all the time is tedious and make code verbose. For this reason, Java 1.5 introduced two concepts, autoboxing and unboxing, that automate these conversions.
Autoboxing: It is the process of converting primitives to their corresponding object type automatically.
Unboxing:It is the process of converting object types to their corresponding primitives automatically.
In Java, Object class has two identity methods called equals and hashcode. We usually override them in our classes to achieve equality. Understanding the underlying mechanism of equals and hashcode can be crucial in some cases. Let’s take a brief look at what they are used for.
Categories
Argument passing in Java
Argument passing in Java is simple but sometimes confusing. Let’s go over some concepts briefly before diving into it.
If you come from a C/C++ programming background, you’ve probably heard of the “Call By Value” and “Call By Reference” argument passing concepts.
Briefly,
- Call by Value : A copy of the object instance is passed to called method. Any modifications made on the object inside the method will not be reflected back to the original object.
- Call by Reference : A reference to the original object is passed to called function. This reference may be thought as an alias for the original object and any modification done on the alias object inside the method actually done on the original object.