A TPM is a very slow and dumb chip: It can hash data somebody sends to it and it can encrypt and decrypt data slowly. That's basically it. There is no privacy concern there that I can see. That chip can not read or write memory nor talk to the network.
Together with early boot code in the firmware/bootloader/initrd and later user space that chip can do quite a few cool things.
That code will use the TPM to measure data (create a hash) it loads before transfering control over and then unlock secrets only if the measurements match expected values. There is no way to extract that key on any system with different measurements (like a different computer, or even a different OS on the same computer). I find that pretty interesting and would love to use that, but most distributions do not offer that functionality yet :-(
Using the TPM to unlock the disks is just as secure as leaving the booted computer somewhere. If you trust the machine to not let random people log in, then TPM-based unlocking is fine. If you do not: Stay away.
Extracting the keys locked to an TPM is supposed to be impossible, so you do not need to worry about somebody stealing your keys. That alone makes TPMs very interesting... your own little FIDO tocken build right intomyour machine:-)
Yes, I should not have said "impossible": nothing is ever impossible to breach. All you can donis to make a breach more expensive to accomplish.
Those separate tpm chips are getting rare... most of the time they are build into the CPU (or firmware) nowadays. That makes sniffing harder, but probably opens other attack vectors.
Anyway: Using a TPM chip makes it more expensive to extract your keys than not using such a chip. So yoj win by using one.