I have found wpa_supplicant tries to authenticate with my AP. However, at the association stage, it fails with:
Failed to set PTK to the driver
After some research, I found that I have missed some critical kernel crypto modules, and I believe this is the reason for the failure of connection. Here is the list of modules:
ccm ctr lib80211 (I am not sure about this module)
Adding those modules to my /etc/initramfs-tools/modules (as I am creating a custom initramfs) solves my problem.
Hope this helps if you encounter similar issue.