@Yaiyan - my sister, when she started teaching in Londond was constantly questioned about the meaning of words with the long e / ai sound (like airport or care), because her Welsh accent included a sound that was totally foreign to the kids she was speaking, and they constantly mistook it for an "ey". If she said hair, they thought she was saying hay.
I suspect that is what leads to the "Abertaway" etc amongst second language speakers - trying to make a dipthong of the more pure Welsh e.
I'm going to be a bit un-useful and say - don;t worry about it for now, however furstrating it might seem. The reason I say that is that if you can;t hear the difference, then any attempt at producing the different sound is likely to lead to an incorrect but different sound. As you get more used to hearing and using Welsh, you may start to develop an ear for the subtle differences. and that is when you need to start trying to pronounce the differrent sound.
I wonder about people saying they can't understand you if your e is the English sound of where / wear / rare rather than the Welsh sound. I now a lot of Welsh speakers - mostly second language or Welsh-school-produced - who pronounce the English e sound, and on it's own, it doesn't make a person difficult to understand. I'm wondering whether this is a south Wales valleys thing, in that people who speak Welsh in South Wales tend to only speak amongst themselves, which means that they tend not to be familiar with other accents / dialects. There are often confidence issues as well - Until recently, Welsh was very much frowned upon, even when I was growing up in the 80s. If a person is unfamiliar with some of the words you are using, you have an unfamiliar accent, and they have been told since forever that their Welsh isn't very goodm it may just be that they just lack the confidence to make the effort to understand unfamiliar sounds.
If this is the case, then you need to do two things - one is to keep doing what youare doing, and building your range of Welsh and you confidence in using it. Once you are more confident in yourself, you will be happier to take the occasional put-down / misunderstanding for what it is - the product of ignorance (as in a lack of knowledge rather than out and out rudeness), or the joshing of someone who is a bit out of their depth themselves.
The second things is, to get into more situations where you can speak Welsh, and find the situations that do build your confidence and comfort with your new language. You've put a lot of work in to get where you are, and there are a lot of people out there who will really appreciate that effort and really want to help you. Find them, and you'll be laughing!