Speaking from the other end of the career road: Widen your options. Don't restrict yourself. There are LOTS more fields out there than aerospace. In fact, "aerospace engineering" could be described as a subset or derivative from basic mechanical engineering. The same is true of many other disciplines.
There are large projects and products to manage in many other fields. For example, my particular specialty is machine design, but within that field I have significant experience and have managed projects in all the following **********: electronic products, tire manufacturing, heavy truck manufacturing, auto manufacturing, steel mills, engineering consulting (both large and small), water jet cutting machines (for aerospace), high-speed winding, and many others. I have patents in two of those fields.
Those working in aerospace now may interpret my remarks as demeaning or insulting. They are not intended that way at all. They are just honest statements of opinion based on direct experience.
Having always been fascinated with all things aviation, I naturally gravitated in the same direction as you - but I never got a direct job in aerospace. Looking from the other end of the career now, to be honest with you, frankly I am glad that it worked out that way.
Here's why: Over the years I have literally had hundreds of interfaces with aerospace customers and clients. I have had many friends in the industry. My impression of an engineer's life inside aerospace, though it could be completely wrong, is one governed by regulations, bureaucracy, ANSI and ISO drawing standards (often completely misunderstood or over-used), corporate policies, endless drawing checking and correction, etc. Life in a corporate maze doesn't really spur innovation, experimentation, risk-taking. I understand WHY all those factors are there, and why they are necessary. I just know it would not work for me personally.
Personal anecdote: A friend from college graduated with an aerospace eng. degree the same year I did with a BSME. He went right to work for Pratt and Whitney. When we met again four years later he told me he spent that entire time in a room with 300 other engineers writing specs for turbine engines. He never saw a turbine engine. In that same time I had designed several fixtures and parts for a small electronics manufacturer, and had progressed to the point where I oversaw the design, fabrication, and installation of a complete new manufacturing line. I was enjoying my job MUCH more than he did his.
One thing that all us gray-hairs have learned is that the things you learn about project management, detailed design, team communication, and other important skills in one field are easily transferable to any other field.
Go for it. There's a big world out there. Take your limitations off and go exploring.