I was actually looking to normalize the audio from a video and realized that this is not so easy. By going through internet I found a pretty nice solution : ffmpeg-normalize.
By quoting his github, This program normalizes media files to a certain loudness level using the EBU R128 loudness normalization procedure.
This is working perfectly and pretty happy from the result.
You will need python and install the script via pip (take care of the message from it, you will need to have the script path to your PATH on windows to be able to run the cmd
Example in my case:
ffmpeg-normalize input.mp4 -c:a aac -b:a 192k -o output.mp4