The 41-year-old actor was originally tapped to star alongside Kirsten Dunst as Drew Baylor in the 2005 romantic tragicomedy, but was let go in favour of Orlando part way into filming.
Opening up about the dismissal during a segment on First We Feast's YouTube series "Hot Ones", set to debut on Thursday, September 26, the star confessed he was "fired" from the role - and suggested director Cameron Crowe always wanted the "Carnival Row" actor as leading man.
"(Cameron) originally wanted Orlando Bloom for 'Elizabethtown' and Orlando Bloom was working on a Ridley Scott picture, and wasn't available," Ashton shared. "So I went (to) audition, he cast me and then we started working on it."
"I think he wanted to see the character rehearsals all the way, and I probably wasn't disciplined enough as an actor to get myself to a point where I was able to do that and show it to him in a way that he felt comfortable."
However, he went on to insist the decision to depart was mutual, adding: "At a certain point we just kind of agreed that it wasn't working out. More him than me."
"Also, I found out at the same time that Orlando Bloom had just become available right when he was gonna let me go."