Even as things with Kaley Cuoco got more serious, Tom Pelphrey was oblivious to his partner's Big Bang Theory fame.

The Ozark star shared that before he introduced Cuoco to his family, he had "never" seen a single episode of her 12-season stint as Penny Hofstadter.

"Matter of fact, when I first brought Kaley to New Jersey to meet my family and friends, my mom's partner — who apparently was a Big Bang Theory fan — was there, and he kept calling her Penny," Pelphrey, 41, recalled in a new W Magazine profile.

"I had no clue what was going on," he admitted. "So I pulled Kaley aside, like, 'I'm sorry, I don't understand what's happening. Why does he keep calling you Penny?' She's like, 'That's my character in The Big Bang Theory.'

"I was completely unaware," he added.

The Love & Death actor said he is no longer oblivious to Cuoco's iconic character, and the couple has since enjoyed the CBS sitcom together.

"I've watched a few episodes with her since," he said. "Obviously, she's fantastic."

And Pelphrey wasn't just clueless about The Big Bang Theory.

According to the actor, he had never seen his 37-year-old girlfriend — who has been acting since she was a child — "in anything" before they started dating.

"Look, I live in a cave," he told the magazine. "Before I met Kaley, I was living in upstate New York on a dirt road in the middle of the woods, without much Wi-Fi."

"She's brought me into modern times," he added.

When Pelphrey (who made his small-screen debut with a cameo on The Burg in 2007) was a child, he crushed on other actresses — but everything changed when he met Cuoco.

"I definitely had a crush on Winona Ryder," he shared, adding that he thought she was "the most beautiful woman I'd ever seen when I was a kid."

"But now I have a crush on Kaley Cuoco," the actor said. "A deep and abiding crush, a love crush on Kaley."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

Pelphrey and his "love crush" have now been together for more than a year and share 4-month-old daughter Matilda, as well as six dogs.

Since welcoming his little girl in March, Pelphrey doesn't have to tune into CBS to get his Cuoco fix — he just has to look at her face.

"I see myself a little bit in her. She's got one dimple on her left cheek, like I do. Other than that, though, I see Kaley," he said proudly. "Kaley says she sees me, but when Matilda's kind of confused or frustrated about something, she makes these faces and I'm like, 'Oh my god, there's your mom.'"

Cuoco, however, will argue that little Matilda looks far more like her father.

In May, the Meet Cute star told PEOPLE that not only does Pelphrey "just love" Matilda — he resembles her, too.

"She looks exactly like him," Cuoco said. "And he has a very distinct voice, so when he talks to her, she's like 360. Her head turns around, and she's like, 'I know that voice.'"

"It's very sweet," she added.