The Madhyamaka attack on Brahman-style views
Nagarjuna's core weapon is the concept of svabhava, usually translated as "inherent existence" or "self-nature." His argument in the Mulamadhyamakakarika is that nothing, not atoms, not causation, not the self, not even nirvana, can be shown to have svabhava once you trace out its dependencies. Everything arises in dependence on something else (pratityasamutpada), and anything that depends on something else cannot be a self-sufficient, unconditioned ground. Brahman, by definition in Vedanta, is supposed to be exactly that: unconditioned, self-existent, not dependent on anything.
So when later Madhyamikas like Chandrakirti and Bhavaviveka went after Vedantic and Samkhya positions directly, their move was to say: any positive metaphysical absolute, no matter how subtle you make it, smuggles in svabhava, and svabhava cannot survive analysis. They'd run the same four-cornered logic (catuskoti) against Brahman that they ran against everything else: does Brahman exist, not exist, both, or neither? Every option generates contradiction under analysis. Their conclusion wasn't "here's a better description of ultimate reality," it was "the entire project of positing an ultimate substrate is itself the error." That's a direct shot at the Upanishadic move of tat tvam asi.
The specific target: atman as subtly reintroduced
The Buddhist anatta doctrine isn't just "there's no permanent soul," it's a rejection of any subject that stands outside the five aggregates (skandhas) as their owner or witness. Vedanta, especially in its Advaita form, does something that looks superficially similar (the empirical ego, ahamkara, is also declared unreal) but then reintroduces a witness-consciousness (sakshin) as the true self, identical to Brahman. Buddhist critics specifically targeted this move. Their argument: you've just relocated the atman one level up and called it something fancier. Nagarjuna's compatriots would say if there's a real "seer behind the seeing," you haven't refuted the self, you've hidden it more skillfully.
Shankara's counter, and the historical irony
Here's the interesting twist. Shankara (the great systematizer of Advaita Vedanta, roughly 8th century CE) was so influenced by Madhyamaka logic that his opponents within Vedanta literally called him a "crypto-Buddhist" (pracchanna bauddha). He borrowed Buddhist tools of negation (neti neti, "not this, not this") to strip away everything the self is not, but where Nagarjuna's negation terminates in emptiness with no positive residue, Shankara's negation terminates in a positive residue: pure undifferentiated consciousness, which he identifies as Brahman. So Shankara essentially took the Buddhist scalpel and used it to carve out space for exactly the kind of ground Buddhists said couldn't survive the scalpel.
That's the real fault line, and it's precise: both traditions use negation as method, but Buddhism's negation is meant to be exhaustive (all the way down, no remainder), while Vedanta's negation is meant to be a filter (strip the false to reveal the true remainder). One gives you groundlessness as the final word. The other gives you Ground, capital G, as the final word.
An analogy that might land for you given your interest in physics: it's a bit like the difference between a theory that says "spacetime itself is emergent, there's no fundamental container" versus one that says "there's a deeper field, and what we perceive as separate objects are excitations of that one field." Both deny that ordinary objects are fundamental. Only one keeps a fundamental substrate at all.