I think it's like this:
Due to their rarity, gold and silver are only available in limited quantities. This makes them suitable as a means of payment, as the limited availability of these materials allows them to have a relatively stable exchange value. They are also durable and easy to work with. So as a means of payment, gold and silver became representative of goods that can be purchased with them.
Coins made of gold or silver, which were used as currencies early on, were not very forgery-proof. However, as the materials themselves are rare, good forgeries were only possible if the material was available. And the rarer the material used, the more forgery-proof the coin became, which is why it made sense to mint coins with a high exchange value from rarer materials (bronze or copper for lower value coins, silver or even gold for coins with higher exchange value).
Early examples of inflation illustrate quite well how value and material are linked: In ancient Rome, for example, silver coins were used as a means of payment. Under Emperor Gallienus, there was massive inflation because he had the silver content of coins reduced to below 5% in order to finance military campaigns and other expenses (he basically created value out of thin air with this move). This ultimately led to a loss of confidence in the currency and thus to a decline in the value of the coins, as they were not inherently forgery-proof (it became quite easy to mint counterfeit coins with such a low silver content).
Still, the value of the material itself has always been variable and depended heavily on its rarity: When the Spanish imported large quantities of silver from their colonies in South America in the 16th century, the value of silver fell because the material became less scarce. This eventually led to massive inflation throughout Europe at the time.
However, the value is not so much determined by the material itself, but by the value that people ascribe to it at a certain time. There are also great historical examples of how speculative bubbles work: In the 17th century, tulip bulbs were at times traded at enormously high prices in the Netherlands because they were difficult to obtain and therefore rare - this made them coveted as a status symbol. However, this bubble finally burst in 1637 when traders could no longer find buyers due to the astronomical tulip bulb prices - demand was covered; the "Tulipmania" had ended. As a result, the merchants began to sell these tulip bulbs in "panic sales" at ever lower prices in order to recoup at least part of their investment. As a result, the price fell to near worthlessness.
Such speculative bubbles still exist today, of course. This is demonstrated very impressively time and again by cryptocurrencies, for example, which are not even material and have no central regulatory authority (such as a central bank) - their value results basically just from what someone is willing to pay at a certain point in time.
So gold and silver were and are valuable not only because of their utility value, but also because of their use as a means of payment, whereby the value - as with today's currencies - results from a (more or less stable) consensus on the exchange value in goods attributed to a material or currency.
As gold in particular has historically been used as a means of payment for a very long time, most countries still have gold reserves to back their currencies to a certain extent, even if their currencies are generally no longer directly linked to an equivalent value in gold as they used to be. Nevertheless, the US, for example, still holds large amounts of gold at Fort Knox and elsewhere to counteract the fact that paper money, and especially its virtual equivalent in some digital form, is materially worthless.
Unfortunately, only for their own benefit.