Rachael Kirkconnell and Matt James.Photo: Rich Fury/Getty

Rachael Kirkconnell and Matt James

Matt James never actually thought he’d be the Bachelor.

“I thought it was a joke initially,” the entrepreneur, who has been dating his pick Rachael Kirkconnell for over a year, tells PEOPLE. “But I’ve learned a lot. And at the end of day, everything that we’ve been through has prepared us for the future.”

Indeed, James, 30, whose new memoir,FirstImpressions, is on stands now, has certainly had his own ups and downs following his stint on hisBachelorseason last year.

After giving Kirkconnell, 25, his final rose, James leftThe Bachelorbubble to discover racially insensitive photos of Kirkconnell at a 2018 antebellum plantation-themed fraternity formal resurfaced online.

Rachael Kirkconnell and Matt James.Matt James/Instagram

matt james

“It was crazy,” James recalls. “It felt like the world was against us. Some people felt like the situation was blown out of proportion and others felt like it hadn’t gotten enough attention. We were just trying to enjoy each other and grow our relationship without having to deal with people we didn’t even know weighing in.

Kirkconnellissued an apologybut the couple eventuallyannounced they had splitduringThe Bachelor’sAfter the Final Rosespecial. One month later, James told PEOPLEhe was “pursuing” a relationshipwith Kirkconnell again.

“The best thing for us was to remove ourselves from the show, from everything,” he now tells PEOPLE.

“We still just try to operate in our own lane. People are moving in, others are engaged… that stuff doesn’t really matter to us. What matters is having a solid foundation and being able to communicate and care for each other.

RELATED VIDEO: Bachelor’s Rachael Kirkconnell Urges People Defending Her Past Racist Actions

That’s not to say things aren’t moving forward as Kirkconnell is looking for places to live in James' Miami hometown.

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

“Rachael has all the qualities I want and it’s nice to find that,” says James. “She’s got a good sense of humor and we don’t take each other too seriously.”

Of the future, “We’re not worried about pleasing everyone,” says James. “It’s not about checking boxes. it’s about finding someone you’re compatible with and focusing on that first.”

source: people.com