I don't know if it's a matter of not being yourself but maybe more so not being comfortable being yourself. I do sense a lack of confidence. Not saying that to put you down or anything, I've been in that boat.
Don't worry about the alpha thing. It's actually been disproved across multiple species, let alone humans. Alpha/beta male dichotomy doesn't really exist. There's a ton of nuance, context, and individual experiences at play here that this false dichotomy is missing.
Yes, aggressive males do succeed because arrogance and dominance is better than submissiveness and lack of confidence, but genuine self-esteem and comfort in one's own skin rates much higher than arrogance and dominance. This is according to social science research. I think genuine confidence is something you may need to work on.
Assertiveness has shown to be one of the most attractive traits. That's not the same as aggressiveness, self-centeredness, or "edge." For example, if there's genuine interest, the assertive thing to do is go for it, not try to hide it to fit an archetype. That signals non-assertiveness and discomfort with what you're doing.
I get the "mess with girls" thing because my most successful interactions have been with girls where we can push each other's buttons. That's the "game" as it were. But that's really a chemistry thing. Different girls have different buttons and you have to find what makes her tick; you have to find little "memes" between the two of you. That requires presenting yourself wholly and assertively, observing her instead of being so self-conscious (this is advice I've gotten a lot), and genuineness (which is also a highly-rated trait). My lady friends have told me that guys who try to mess around in an inorganic way come off insulting and weird. So I would work on it as a mindset as opposed to as a skill. Be confident.
As far as leaving, if you think it will help, you do you. That's kind of the whole point. But don't expect external changes to fix everything without doing the hard internal work of having genuine self-esteem.
With that, I leave you with a real dating expert:
I agree with a lot of this.
I definitely click better with women now that I'm older and wiser. I am happy in marriage, but I do wish I knew then what I know now. I have more game now than I did back then. I have mostly women friends. Even though I'm married, I've had a "work wife" at every job I've had. These women were married or had relationships of their own, but mutual chemistry and attraction definitely were present with all of them during work hours -- and each had a unique dynamic. In my case, we were just friends. In yours, work up a friendship and build from that foundation.
Definitely be genuine, be comfortable in your own skin, exude self-esteem. You need to ignore your deficiencies and negative thoughts, throw them aside, throw caution to the wind, and put your best foot/best self forward no matter what. Be assertive but not aggressive -- let the lady lead, but step in when you sense the need.
And most of all, be funny. Win them over with humor. Looks do help -- I ain't no looker; I work out, do my best with the face and body I've got -- but if you are damn funny, quick witted, and are good with a little verbal gamesmanship, then you have a chance. Women are attracted to funnymen. It's proven.
Translate what we all do here to the person you're going after, which is share funny memes and gifs, share funny articles, have a laugh over something happening that you're both aware of. If you make it comfortable and offer fun and wit, they will respond.
Don't play hard to get for too long. It doesn't work. Didn't Billy Joel teach you anything? Don't do what Andy in The Office did with Erin. After a few days, if you don't hear from her, reach out, don't keep her waiting, don't keep her confused or guessing. That's being assertive. She's probably busy with life, so you need to step in and make yourself known.
Recently I had an unsolicited recruiter ask me if I was interested in a job. I told a lady friend at work, and she admitted to me that while she was being supportive, she was metaphorically going to cry in a corner as I was telling her about the job offer.
Earlier, I had a Zoom call with another person, and she jokingly told me not to comment about her tank top. I was like, "no judgment zone here." Then she reminded me that I had made a tank-top comment about another girl at a previous all-team meeting. Why would she even make a comment like that, or put a comment I made in her memory bank, if she wasn't comfortable with me?
There's yet this other girl at work, super pretty, who I joke with. As we got to talking over the course of a few months, her natural goofiness came out, which is great because I'm also a goofball.
She and I have had some deep conversations, she shares her feelings and opinions about everything with me. Be a good listener.
She calls me her PIC at work, partner in crime. I made some salty-sweet analogy about this other person with her, and she quipped to me, "you can catch flies with honey, but you catch more honeys by being fly." lol
So build a comfort level and a rapport with your muse before doing anything else. That's the foundation. Attractive traits then naturally come out of that.
That's my spiel. Just sharing what has worked for me.