Well, there's user experience, and then there is UX design. There are many components to UX design that help craft a good user experience—for example, user research can inform many levels of user experience, from messaging and content hierarchy on the website to specific services you offer your customers, or the way that your stores are laid out. Additionally, there are specific components to UX design for the web—what I do professionally—that don't necessarily touch the rest of the customer experience.
But "user experience," as I define it, is similar to brand experience and customer experience—it's inherent in every action your customer has with your business. If you're a retail store, user experience is not only the experience people have shopping your site online, it's the employees your customer deals with in the store, how long it takes to get your products, and how easy it is to return orders. If you're a software company, the user experience isn't just about what it's like to use the software; it's whether a user can get the help they need when something goes wrong. It's how upgrades are handled, how pricing is dealt with, and how quickly the user can get their tasks done so they can move on to the next thing in their to-do list.
In one of my first jobs, a manager told me, "90% of unhappy customers won't rant and rave and yell at you; they'll just leave and never come back." On the web, you can look at bounce rates—where folks are leaving your site. Are they clicking around to a few pages and leaving? Just coming in to read a blog post and never looking further? Are they putting things in a cart, but leaving the site without finishing the purchase? These can give you some idea about where the sticking points are in your design. If you have the luxury of seeing customers face to face, take a look at their faces while they're working with your product; do they seem pensive, like they're looking around? Do they seem frustrated with something, or exasperated? If you're talking to them on the phone, how does their voice sound?
Doing usability tests where the user's face is actually recorded is a great way to experience this first hand. While the participant is doing something with your software or website, their emotions are recorded on their face—and it's not hard to tell whether they're frustrated, or finding something easy to deal with.
Honestly, the best way to find this information to ask for it. Talk to your customers—ACTUALLY TALK TO THEM. Not surveys, not "fill this out for a chance at a gift card." Pick a few representative customers and have a conversation with them about what works and what doesn't. If you're building a software application or a web app, get people using it and see where they trip up—and where they say, "ooh! That's [easy, neat, cool, insert positive adjective here]." If you're building something that is designed to help people accomplish a specific task, talk to people who have to accomplish that task about how they do it normally. The biggest problem I see happening in design/dev teams is a focus on quantitative metrics over qualitative understanding, and a focus on talking to users about a project only after something's been built, without doing any research up front to decide whether something's even worth building.
Best case scenario, the user grunts and bears it for a while; however, they're unlikely to stick around once they realize there's a different option. This is especially true with apps; for example, I'm addicted to to-do list applications. I've tried at least a half-dozen of them, but it took me forever to find the right one. Despite this, for me, it wasn't a big deal. I'd download a new one I'd heard about, load some tasks into it, think "can I see myself doing this all the time?" And if the app made it too hard for me to easily add a bunch of tasks at once, or didn't let me easily sort them, or I couldn't sync the tasks to my calendar, I abandoned it and moved on to the next thing. For me, it wasn't a big deal; to the companies who made these applications, they lost a potential customer and a piece of their market share. The same thing happens with retail stores, restaurants and online shops. While we might be willing to handle a little bit of a wait for the right place, many of us have no problem taking off and heading someplace else if we feel like the person we deal with first doesn't want to bother with us—even if we have a bunch of stuff in our hands.
We've reached a point, particularly in tech, where someone who does almost exactly what you do is right around the corner, and easily accessible if you aren't doing that thing in a way that meets a customer need. And customers are increasingly unwilling to put up with clunky interfaces, steep learning curves, or company representatives that write off poor customer treatment as "policy." All of these things are part of the user experience of your company, and all of them have the potential to cost you customers if they aren't handled thoughtfully.
Good UX, in all its forms, balances thoughtful research and empathy for your company's stakeholders (not only customers, but also internal business groups, shareholders, etc.) with an intuition of what will meet the various needs of these groups best. Passion plays a role, yes, but not in the way a visual designer or artist might think of the word. The best UX designers are not lone visionaries working away in isolation towards some magical ground-breaking product; that's a romantic notion at best, and the marketplace is littered with lone visionaries whose products failed because they wouldn't allow themselves to be wrong. UX designers expect to be wrong, and acknowledge what they don't know yet. The passion, then, comes from understanding why you're wrong, and moving towards the right solution. It also comes into play when trying to sell an iterative, continuous learning process to stakeholders who believe that everything needs to be exactly right out of the park, and doesn't want to put energy into multiple iterations.