The British comic relocated his wife Julia Carey and their family to Los Angeles five years ago to present the nightly programme when he succeeded Craig Ferguson as the host of the show in March 2015. But speaking to Britain's Daily Mail newspaper, James suggested family commitments may force him to step down from the coveted slot.
"I have a year left on my contract. It's hard. It's so much more than just, 'What do I want to do?'," the 40-year-old reflected. "We're a long way from home, my wife and I, and our children, and there are people at home that we miss deeply and we care about and they're getting older.
"Then there is the question of where do we educate our children and where is the best environment for them."
The star went on to insist the conflict wasn't about money, and claimed he doesn't want to get too comfortable in the role.
"I think you've got to be too careful not to get into money. It can be a dangerous thing if you are using it for new creative endeavours," he continued. "I might just slip away and be a footnote and a memory for someone.
"Doing the show itself was a punk move and I will always be drawn to the next punk move."
James and Julia share three children - son Max, eight, and daughters Carey, four, and 18-month-old Charlotte.
The star is returning to U.K. TV screens later this year for a one-off special of his hit sitcom "Gavin & Stacey", which will air on BBC One as part of the channel's Christmas Day schedule.