I figured out that it uses the amd64 compiler:
alex@MacBook-Pro-alex ~ % ls /Users/alex/Library/Caches/hardhat-nodejs/compilers/mac*
list.json
solc-macosx-amd64-v0.8.9+commit.e5eed63a


But I already have the solc compiled for arm64, installed from homebrew.
alex@MacBook-Pro-alex ~ % solc --version
solc, the solidity compiler commandline interface
Version: 0.8.9+commit.e5eed63a.Darwin.appleclang

When I removes the amd compiler it downloads it again. So, Is it possible to specify path to compiler? Anyway, could it resolve slow compilation or there is something else what could help me?