The method of negation in Advaita Vedanta. Every description of Brahman is negated — not to arrive at nihilism, but to exhaust the mind's habit of locating Brahman within objects and concepts.
Yājñavalkya is asked: what is Brahman? He gives the answer twice in the Bṛhadāraṇyaka Upaniṣad: neti neti — not this, not this. He is not being evasive. He is being precise.
Every description of Brahman uses concepts, and concepts work by defining and limiting — by saying what a thing is by saying what it is not. But Brahman is the ground of all concepts, prior to all distinction. The moment you say 'Brahman is consciousness,' you have made consciousness a predicate of Brahman — and Brahman becomes the subject of a sentence, an object of thought. But Brahman is not an object. It is the ground of all subjects and all objects.
So neti neti is not the end of inquiry. It is the engine of inquiry. Say: Brahman is this — neti. Whatever you just thought of, not that. Say: Brahman is beyond thought — neti. Whatever 'beyond thought' just meant to you, not that either. Keep going. The mind exhausts every category it can generate. What remains when the generating stops — that cannot itself be negated, because negation requires a mind, and the mind is what it is happening in. That remainder is what is being pointed at.
The phrase appears at two key points in the Bṛhadāraṇyaka. In 2.3.6, Yājñavalkya distinguishes two forms of Brahman: the formed and the formless, the mortal and the immortal — and concludes that the true form is neti neti, the entity about which one can only say 'not this, not this.' In 3.9.26, in response to the direct question 'what is Brahman?', he says only: neti neti ity ācakṣate — it is described as neti neti.
Śaṅkara's reading: neti neti is not agnosticism. The negation is followed in both passages by a positive statement — Brahman is the unseen seer, the unheard hearer, the unthought thinker. The function of neti neti is to clear away inadequate positive descriptions so that the positive indicators that follow land correctly: without the preceding negations, 'the unseen seer' would be heard as another object-description. After neti neti, it is heard as pointing past all objects to the witness of all objects.
Advaita's technical vocabulary for neti neti is the method of adhyāropa-apavāda — superimposition and subsequent negation. The teacher first applies positive descriptions to Brahman (satyam jñānam anantam brahma, etc.) to orient the student. Then, having oriented the student, the teacher negates each description: not this, not this. The superimposition was pedagogically necessary; the negation is philosophically necessary. Together they produce the recognition that is neither the positive descriptions nor their absence, but the ground in which both arise. This method is central to the Vivekacūḍāmaṇi (Śaṅkara, verses 254–260) and is the structural principle behind both the Pañcakośa discrimination and the Māṇḍūkya's four-state analysis.
Sources: Bṛhadāraṇyaka Upaniṣad 2.3.6, 3.9.26, trans. Swami Mādhavānanda (Advaita Ashrama, 2010); S. Radhakrishnan, The Principal Upaniṣads (Allen & Unwin, 1953).