The word, "that", is actually a determiner, a demonstrative pronoun, and a relative pronoun (based on Cambridge Dictionary at www.dictionary.cambridge.org). "We use [the word] that to point to a thing or a person. We use it with singular nouns. The thing or person is often distant from the speaker annd sometimes closer to the listener, or not visible to either the speaker or listener". For example:
Can you pass me that green bowl over there? (determiner)
That's Melissa in the white shirt, isn't it? (pointing to one of a selection of different pain colors)
If he gets that job in London, he'll be able to visit us more often. (referring back to something already mentioned)
She picked up the hairbrush that she had left on the bed. (relative pronoun)
I disagree the word "that" is used as an adverb because adverbs are meant to express a specific detail. Using the word "that" is an ambiguous word and not specific.