(Disclaimer: I'm not an expert, I just googled a little bit before posting that message. Fact-checking is welcome.)
"Is most of the rest silicates?" The question is ill-posed, because it mixes elements with compounds. Specifically, silicates contain oxygen so they can't be counted separately.
If we go by elements, we already said that oxygen is the most abundant. The rest of the crust is mostly silicon (28%), aluminium (8.2%), iron (5.6%) and more stuff. The rest of the mantle is mostly magnesium (22.8%) and silicon (21.5%).
But most elements don't remain isolated, they react and form compounds. Those made primarily by silicon and oxygen are known as silicates, and they are indeed the most common type of rock in the crust and mantle.
So yes, there is plenty of silicate material if one wants to use it for construction. I have no idea how good it is compared to current materials, however.
Me with android auto and Bluetooth loudspeaker: hold my beer