Should I touch a patient's hand without putting gloves on first?!

I am about to start volunteering at my local hospital. Being a ward helper, I will be involved in making direct contact with patients, such as reassuring them and providing a listening ear.
Why am I worried? This time last year I worked full-time for a week in the hospital, as part of my work experience programme. Sometimes when I brought a patient their dinner, I would stay in the room to talk to them a little - especially with the older, more lonely patients that had no families. Sometimes they would take my hand, and hold it while they spoke. I was worried that perhaps I was breaking the hygiene rules without knowing it.
Extra facts: It is not obligatory to wear gloves while doing the meal round. In every room there is a sink in which to wash hands with soap, as well as at least one full bottle of sanitising gel.
Should I wear gloves when talking to the patients in case they want me to hold their hand?
My gut instinct is yes, I should because it helps prevent possible spread of infection. However, doesn't it seem a bit cruel to the patients?
Thank you