You're using experience points, and the players farmed too much of it. Your player asked if they could use a homebrew class that you didn't check very well before saying yes. You gave them legendary items a bit too early. Or maybe your players are just experienced, and they've organized their classes well in session zero to optimize their efficiency.
Regardless of the reason, your players are just too powerful, and they're destroying everything you throw at them from Dungeons & Dragons' Monster Manual and even some homebrew beasts. What do you do when the players are overpowered? Tweaking the monsters they'll face usually works, but how specifically should you tweak them?
5 Boost The Basic Numbers
Sometimes, basic ability boosts can make all the difference. You can do whatever you want with a creature's stat block, so take advantage of that. If minions are being taken out too easily, just double or triple their health points and increase their armor class. If the players ask, they're elite minions. Still, keep in mind that minions aren't supposed to be too challenging, but this general idea applies to all enemies.
If you see they usually hit a creature with an AC of 17 or 18, don't be afraid to go above the 20+ margin. The same goes for health points; See how quickly they kill your minions and how much health they're taking in a round, and use that information to tweak the BBEG's health - in fact, as a bonus tip: Use minions or weaker encounters to see how your players are handling fights in general so that you can counter them appropriately when a big battle arrives.
You should also do that to ability scores; don't be afraid to give a boss 30 on their strength score if necessary. You can also provide them with resistance or immunities to certain damage types and conditions, but we'll discuss that shortly.
Just be careful to avoid overcorrecting the situation here. You can boost some stuff, but don't boost all of them - don't give your boss 30 on every ability score and immunity to all conditions (unless the party is that broken in terms of overpower). You want to challenge your players but not obliterate them. A DM can easily create an unkillable monster, so avoid doing that.
4 Increase The Number Of Creatures Or Turns
A big reason groups can overpower a boss is the action economy. Suppose everyone is effectively using their action, bonus action, and reaction every round. In that case, you have multiple people hitting them, for you to only give a few attacks during your turn - some that can even miss or be annulled by things such as Counterspell and whatnot.
Your monster needs to do more in a round to constantly challenge a group of four to five people. You can do so by giving your baddie a group of their own; A party vs. party is an exciting dynamic, and it can be pretty challenging, especially because you control everyone from your evil group, so they can coordinate as if they are one - because they are. And having multiple people to handle will make things more difficult for the players. Still, if you don't want or don't have time to plan that many baddies, you can use minions with the boss. Sure, they're weaker, but that only means you can put many of them.
Another way to strengthen this monster is by giving them multiple turns per round. "Wouldn't that be cheating?" No, because you control the rules! Legendary Actions are a perfect way to have extra attacks outside your turn, and nothing stops you from having a Legendary Action that grants you an extra turn or at least an extra main action during the round. The more your creature can do in a single round - regardless if it's during their turn or not, the better. We won't write this on every entry because it'll get tedious, but again, just be careful not to exaggerate how many things you give your monster.
3 Play Smart
What does your baddie know about the group? If the characters know each other, you can roleplay smart thinking, or you can make your enemy an overall smart guy. Usually, when DMing, we may go 'dumb' to prevent strategies that destroy parties - like when we scatter minions to attack everybody. But if that's too easy for them, then start strategizing.
Do they have a healer? Focus on taking them down first. Counterspell their healing and send your forces toward it. Isolate the tank with bait so they're too far to protect the group. Use ranged combatants, so you can focus on a target without putting all minions close to one another, so area spells won't hit everybody. Use stealth and get yourself a surprise round.
The players will have to focus on protecting their healer. They'll have to bait your Counterspells and see which spells are worth losing. The tank will have to find a way back to their friends, the spellcasters will have to be smart about their targets, and they'll start encounters at a disadvantage. Whatever the case is, they'll have more concerning goals. "Should I charge, or should I stay by the healer?" Whether they choose to attack or protect, the adrenaline will kick in and make things more difficult or at least seem more difficult.
2 Counter Players In Particular
If you see specific techniques during fights, create something that counters them. Or analyze the weak spots in the player's build - there's always one - and poke that wound in particular. If the Evocation Wizard is setting everything ablaze with their Fireballs, what happens if they fight a creature immune to fire?
Use attacks that make people fall against flying players. Target the strong Barbarian with mind-control spells. Force the high AC character into saving throws. Create enemies with straight-up anti-magic fields around them, or have them immune to conditions your players often use, such as immunity from being paralyzed.
Poke their weaknesses and prominent strategies, forcing them to think differently about how to handle the challenge at hand, creating difficulty and having some fun with the situation.
1 Tweak The Battle As It Goes
If your plans fail, don't give up on the boss just yet. This is a game of improvisation, and nothing stops you from changing details while the fight is already happening. If they took almost all your health in one round, increase it. Have them use a special power that increases their AC for the rest of the battle. Fudge a few rolls if necessary. You're good to go as long as the fight stays dramatic and challenging.
Improvise a whole second phase while giving the boss more stat boosts, some immunities, and Legendary Actions. Avoid the dull and anticlimactic moment of the BBEG themself losing in a mere round of combat due to poor balancing. That won't be a fun end to a story.
Bonus: Let Them Have Their Fun
Your mission is to make your players have fun. We can't tell you what your players enjoy; you know them, so you have to learn that from their behavior. Yes, maybe your players are destroying everything in their path, but are they enjoying that?
If not, sure, then raise the challenge. But if the power fantasy is entertaining your players, perhaps you don't need to challenge them. Some people enjoy playing games in easy mode; they don't see difficulty in a game as fun and just want to have a good time with friends. However, if the lack of challenge is boring them, our tips may be necessary.
On another side note, you also don't need to worry about making every single fight challenging. Otherwise, you may overwhelm your group. Fighting the BBEG and fighting minions from a side quest are two very different things. It's okay for the occasional bad-guy annihilation here and there. Focus on making the essential encounters challenging, but not necessarily all of them. Unless your group wants to go 'survivor mode,' then, by all means, break their spirits.