Neurons are a natural starting point for understanding neural networks, but interpreting them can be difficult: an individual neuron may activate in seemingly unrelated contexts, while a single prompt activates many neurons whose contributions are hard to disentangle.
Recent evidence, however, suggests that neurons are more interpretable than previously believed: with better tools and a closer look at their computational roles, individual neurons begin to make sense. Here, we study how small groups of neurons work together to implement common model behaviors, from grammatical agreement to copying and arithmetic.
Removing these groups substantially changes the model’s predictions, while retaining them can largely preserve predictions even after ablating almost all other MLP neurons in the layers involved. Tests on held-out prompts show that these communities – often fewer than ten neurons – are both necessary and sufficient to explain the contributions of the relevant layers.
We discover these groups without specifying their functions in advance. Across millions of prompts, we use gradient-based attribution to identify influential neurons, connect neurons repeatedly attributed together in a graph, and apply community detection. We then test the resulting communities for necessity and sufficiency and examine the behaviors they support.
These results add to the evidence that neural networks already contain interpretable structure if we know how to look for it. Neuron communities can serve as computational building blocks for model behaviors, offering a level of abstraction between individual neurons and higher-order computations. Because our discovery method is unsupervised, we can find communities at scale, providing concrete starting points for agents to investigate their functions and test explanations of how they work. This offers a path from individual discoveries toward a fuller understanding of neural networks.
Background
Distributed representation offers an explanation for why neurons are hard to interpret: features may overlap across neurons, giving each neuron multiple roles. This motivates methods such as transcoders and parameter decomposition, whose interpretable units need not align with individual neurons.
Yet Arora et al. (2025) recover neuron circuits as sparse and faithful as learned-feature circuits on subject–verb agreement, and Feucht et al. (2026) identify 28 neurons implementing reusable base-10 addition. Shabalin and Piotrowski (2026) also find that many active transcoder features align with individual neurons. In our recent work (Tigges et al., 2026), we showed that models can be significantly sparsified in the neuron basis while preserving many model behaviors. Together, these results suggest that some concepts and computations can be captured by surprisingly few neurons.
Method
We want to discover groups of neurons that repeatedly work together, without deciding in advance what behaviors to look for.
ADiscover recurring neuron communities
Step 1
Neuron attribution
Score how much each neuron drives each prediction
Step 2
Build neuron graph
Link neurons that repeatedly drive the same predictions
Step 3
Discover communities
Groups of neurons that keep working together
BTest causal effects
- Community neuron
- Other neuron
- Ablated
Necessity
high KL vs. originalAblate the community → prediction changes
Sufficiency
low KL vs. originalAblate everything else → prediction is preserved
Step 1: Neuron attribution. We run the model on millions of prompts and use gradient-based attribution to estimate which neurons contribute most to each next-token prediction.
Step 2: Build a neuron graph. We build a graph, representing neurons that repeatedly contribute to the same predictions. Each node is a neuron, and each edge quantifies how often two neurons co-occur together, adjusted for each neuron’s individual frequency.
Step 3: Discover communities. We use graph community detection to find groups of neurons that repeatedly work together.
We then test these candidate communities through causal interventions on held-out prompts:
- Necessity: Removing the community substantially changes the model’s predictions.
- Sufficiency: Keeping the community1 largely preserves predictions even after removing the other MLP neurons in those layers.
For the results below, we run the algorithm on Llama-3.1-8B with 10M prompts from FineWeb-Edu, English Wikipedia and FineMath. We later replicated this on Qwen 3.6 35B-A3B.2
Case study: an + vowel agreement
We start with one example found by our algorithm above: a community of six neurons spread across six layers: L2:5664, L5:3846, L8:13390, L12:14182, L13:3604, and L14:11888. Below, we call a community active when it is necessary for the prediction: removing its neurons substantially changes the model’s output.
Agreement: 'an' → vowel-initial next word
After the indefinite article "an" it keeps the next word vowel-initial ("an issue", "an excellent", "an upper"). Ablated, the model picks the consonant-initial word it would use after "a" ("a new", "a great", "a limited").
…ific part of the visual cortex – the extrastriate body area (EBA) – that is specially activated when perceiving human shapes and positions. These participants had never before been able to perceive an entire human body. They
…greenhouse effect. Both can be used: some people say that ‘global warming’ is incorrect, because the warming is not the same everywhere – some places are warming more than others, and there is even an area of the North Atlantic
…can be validated through correlation studies based on summative and formative data. Managing Academic Data Given high-quality data, districts must learn how to properly use hardware and software as an instructional tool. Educators
…ries. We need to look at social causes. The private experiences usually thought to be the proximate causes of suicide have only the influence borrowed from the victim’s moral predisposition, itself an echo of the moral state
…g-taking-- 4. Abortion-- 5. Euthanasia-- 6. Sex and the body. (source: Nielsen Book Data) Profoundly important ethical and political controversies turn on the question of whether biological life is an essential aspect of a human
…os[x] x, {x, 1}, StepMonitor :> Sow[x]]] Out[2]= This counts the steps. In[3]:= Block[{ct = 0}, {FindRoot[Cos[x] x, {x, 1}, StepMonitor :> ct++], ct}] Out[3]= To take a successful step towards an answer, iterative numerical algorithms
…frequently conflicting perspectives of the stakeholders and actors. Extensive public consultation may be required, potentially creating delays. The information must be disseminated and explained so an informed public can participate in
…he strongly recommended Botany Bay as a suitable place for a penal settlement. He was created a baronet in 1781, appointed K.C.B. in 1795, and a member of the Privy Council in 1797. Banks attracted an inner circle of accomplished collectors
…of waste — they eat our garbage, our roadkill, and our feces. None is perhaps more impressive than rat-tailed maggots, the larvae of certain kinds of hoverflies. On one end of their body, they have an extendable “breathing
…doesn’t utilize a holistic identity lens when reviewing and analyzing data, it can be difficult to understand the nuanced challenges that are inhibiting student success. Dive into this process with an understanding of the campus culture
…robably know the route from your house to your school. Assume that you moved to a new house in a different place, so you had to take a different route to school. What if following the old route was an innate behavior? You would
…, Manganese and Niacin. Note: adding a little cayenne pepper to your meals, besides the tasty "zing" it offers, is reported to act as a circulatory tune-up for your body (when eaten regularly, over an extended period of time).
…urriculum into snappy 15-minute lessons - Makes spelling fun! Aliens from an exciting online spelling planet clearly introduce spelling rules and concepts - Read Write Inc. Spelling Online includes an Extra Practice Zone including over
…could have such educational benefits. First Lego League as well as programs such as the camps and classes are all made possible through the support of sponsors, generally corporate sponsorships. As an example, to celebrate the
…helial and connective tissues form its lining, and nervous tissue extends throughout both its wall and its lining. Systems are the most complex of the component units of the human body. A system is an organization of varying numbers and
…son to master. That world view posits guns as inherently and irrefutably dangerous. That world view posits guns as serving no tangible, societally acceptable purpose. That world view posits guns as anachronistic. That
…maximize anabolism.9 These ranges were based on targeting 1.6 g/kg/d – 2.2 g/kg/d protein intakes for the purposes of muscle building. Reap the Benefits of an Ideal Protein Intake Now that you have an idea of what your daily
…, “For the wages of sin is death; but the gift of God is eternal life through Jesus Christ our Lord.” A Good Trade Sometimes I have made what would be considered a “poor” trade to give someone else an advantage: taking the smaller
…s would definitely make top of my list: when I cycle through early in the day when the sun is low, the morning dew sometimes creates a layer of fog over the park that looks mystical. Also, there is an army of ducks still sleeping
…tion task in which some word stems (e.g., "ki _ _") could be completed with either aggressive (e.g., "kill") or nonaggressive (e.g., "kiss") words. In Experiment 2, participants were presented with an ambiguous description of a person
…mpact. On April 1, the newly built post office began issuing and paying money orders. On April 11, the town council approved building a bridge across the stream on South Forest Avenue. On April 15, an application was made to the
In September 1998, Iraq reported an outbreak of peste des
… Organization as well. Ekhtiyari is licensed to practice as a basketball coach and referee by the Islamic Republic of Iran Basketball Federation. He also heads the Yadegar e Bastan Research Center, an institution supported by the Iranian
…o protect and aid the civilians impacted by the terrorist organization. Election as Catholicos-Patriarch On 6 September 2021, Mar Gewargis III formally stepped down as Catholicos-Patriarch during an Extraordinary Session of the
… numbers and awards prizes to people who match those numbers. In many states, the proceeds from the lottery are used to fund various public projects. However, some critics argue that the lottery is an unfair means of raising revenue
…omes to life in a split screen effect. - A rather unique combination of this effect and a Wipe occurs when a spy is shown running down a sidewalk, with the left side of the screen blacked out, like an unfinished wipe. The wipe
…(37.8% decrease) than for fuel cell electric buses (23.1% decrease). Since battery electric buses typically have a smaller range than fuel cell electric buses even under optimal conditions, this is an important consideration for transit agencies
… fruitless six week search, local detective Scott McDaniel decided to inspect the Mount Solo area near the Columbia River, which Kondro was known to frequent. There, police found Rudd's body inside an abandoned Volkswagen; she had
…inly within the rest of the Jewish people, there’s a vast amount of genetic variation that simply contradicts MacDonald’s most basic assertion that Jewish genetic sameness is a sign that Judaism is an evolutionary group strategy.”15
… them. At a time when firecrackers are being temporarily banned, and schoolchildren, conservationists and animal lovers have initiated social media campaigns against bursting them, Dr Samrat Ghosh, an Assistant Professor at the Department
…ng of items to anyone may need access globally. Traditional Identification Systems New Digital Identification Systems With traditional business systems, as a product or rolls off the factory floor, an identification number is generated simultaneously
…This led to his deep mistrust of white people – something that would later prove to be a key asset in his activism. After joining the African American Muslim Association at age 20, Malcolm X became an outspoken critic of white supremacy
… economics and bragging rights. Gorman concludes, “Sporting your rainbow in the gym or on a run became a way to feel like you belonged to something larger—you were connected to being ‘out’ or being an ally.” Sports marketing during
…r flowers of the Honeysuckle are lovely to admire and give off a sweet floral scent. What’s more, this perennial is low maintenance, making it a well-loved favorite with many gardeners. Watering is an important part of plant care
…. But, it was during the inter-war years that battery performance was greatly enhanced. This was achieved through better selection of materials and methods of manufacture. Batteries have now become an essential part of everyday life
…ily moved to Cork soon after Hogg's birth, and lived at Highfield West, College Road. She attended St Aloysius School, Cork, and Le Bon Sauveur Convent, Holyhead, Wales. After school, she worked as an insurance official in Cork.
… and state laws in NY and NJ make it illegal to kill bats, even when they have established a colony inside your home. Proper removal of bats depends on understanding these laws that protect them as an endangered animal. In some
…uences the inventory of homes for sale. Housing inventory and house prices Figure 1 shows the inventory and price level of vacant homes for sale as reported by the Census Bureau. These data are not an exact measure of total homes
…n of oxygen to H2O, which is accomplished by the respiratory chain in the mitochondria. During these reactions, NADH releases its electrons (another oxidation reaction), becoming available again as an electron acceptor (see
…bundant food and feed supply that has provided the stability for our nation to grow and prosper. 2004 is the International Year of Rice and the 50th anniversary of the establishment of ARS. This is an opportunity to reflect on some
… to present how time can be thought of in philosophy – that is, time reflected out and through man’s experience and then put back into the experience to transform or modify it. If time is viewed as an objective essence beyond all consciousness
…equation to express an even number as a sum of two equal addends. 2.OA.4 Use addition to find the total number of objects arranged in rectangular arrays with up to 5 rows and up to 5 columns, write an equation to express the total
# Intermediate Geometry : Prisms ## Example Questions ← Previous 1 3 4 5 6 7 8 9 10 11 ### Example Question #1 : How To Find The Length Of An Edge Of A Prism
Stanley Merton Lawler (19 November 1909 – 26 March 1958) was an Australian rules footballer who
The purpose of this assignment is to creatively demonstrate an understanding of infant developmental milestones
…as known for his Twitter and Facebook satire communities with followers over 270,000. He was the president of Student Union in Birla Institute of Technology and Science, Pilani – Goa Campus. He was an influential social media personality and
… by greg1313 $\displaystyle \sum_{k=1}^{n}(2k-1)=n^2$ Greg, could you please explain the procedure? May 8th, 2015, 05:09 AM #7 Global Moderator Joined: Dec 2006 Posts: 20,942 Thanks: 2210 It's an arithmetic progression (A.P
…res, pour your mixture into a spoon then gently submerge into the bath. Carefully flip the sphere a few times while it’s submerged — this will help keep the shape round and allow the bath to create an even gelling around the
…ted tradition and to our future group survival. What are Reform Jews and others to learn from this agenda-driven reading of the Torah? When Reform rabbis (and others) hold up Moses as an example of an intermarried Jew, we
…f of which was destroyed and all its guns lost. This monumental victory allowed the lifting of the siege further to the south where Ahmad Pasha- having heard of Topal Osman's victory -came out with an enthusiastic garrison to chase
…, indoor climate. For example, areas closest to an energy-production and delivery source could receive excess flow, resulting in excessive heating or cooling. On the other hand, areas farthest from an energy-production and delivery source
…teria (relative or absolute). This is the nature of the remeshing algorithm - it is extremely difficult, mathematically, to find a mesh that has tightly-bounded edge lengths. We can at best provide an approximation, with roughly
… Carroll said. However, all electricity customers statewide can look forward to a monthly charge due to the state’s Clean Energy Standard. The governor has said the charge won’t be more than $2 for an average household. Reason to
…ndian Malayalam-language period drama film directed by P. K. Joseph, written by Pappanamkodu Lakshmanan and produced by E. K. Thyagarajan. It is based on the life of Adimakannu / Mulamoottil Adima, an outlaw active in the Central
…2009 Frontline/World report, hard drives containing information about U.S. government contracts with the Defense Intelligence Agency, NASA, and the Department of Homeland Security were uncovered in an e-waste dump in
…e of the Four Corners area. If, instead, you opt to camp at Chaco Culture National Historical Park, you will be in for quite a treat on nights the clouds cooperate. In 2016, Chaco was designated as an International Dark Sky Park for
… demolished. Archaeological excavations on the site between 1974 and 1978 uncovered the remains of the Roman quay and revetment. The demolished warehouse was eventually replaced by St Magnus House, an office building designed by Richard
…wave period) for 32,000 counts/rev, which is high enough to let the controls command smooth motion output. What the coils measure Let’s take a closer look at hybrid-servomotor operation to see what an emulated resolver measures.
…ings, though this method proved to be inefficient and could risk failure. In modern aircraft, pilots can now rely on aircraft parts such as ailerons, which are one of the primary flight controls of an aircraft that allow for the
…on from certain property taxes for veterans with 100% service and total service-connected disabilities. In-state Tuition Rates for Nonresident Veterans: This benefit allows all veterans to receive an in-state tuition rate if
…inion (contrary to the solvency declaration made by the board of directors), then the liquidator must either (a) apply to the Court for the company to be wound-up on an insolvent basis, (b) appoint an administrator to the company,
…t is necessary to consider usage environment including working temperature and working load of the magnet and costs and to make a selection according to the objective. In addition, in a case where an integrated magnet attached by ins
…n: How Many Kilometers Is 100 Miles? In conclusion, converting miles into kilometers can be a tricky task if you don’t know the basics of the metric system. But with this blog post, we’ve provided an easy-to-follow explanation of
… 1. May 5, 2006 ### eldrick An excellent puzzle for you all - not overly hard, but a good tester : We have a ball made of pure gold. 2 Oxford Professor had obtained it in illicit fashion during an archaeological dig in Peru.
…ement along the demand curve. If one of the other factors changes that will cause a shift to the left or right of the whole demand curve. This latter is called a “change in demand”. ## What causes an increase in price and quantity
… being carried out the world over. Harriet Beecher Stowe wrote Uncle Tom’s Cabin, one of the most popular books in the United States in the 19th century (second only to the Bible). The book sparked an unquenchable national
…e 56 picked up the 1998 Chicago Fire, also coached by Bob Bradley, as well as the post-shootout era record of 55 set in 2017 by Atlanta United. LAFC's seven road wins also tied for the most ever by an expansion team in the pre
…what is happening . Obviously, simply because machines may practically autonomously produce artwork, it does not mean that they will replace artists. It only suggests that artists are going to have an additional innovative instrument at their
Iarlaith Daly (born 2000) is an Irish hurler who plays
… points which show the different combinations of two commodities a consumer is indifferent about. Since all these points render equal utility to the consumer, an indifference curve is also known as an isoutility (“iso
…he State government. FSSAI State license Food operators like small to medium medium-sized manufacturers, retailers, transporters, marketers, distributors, storage units, etc. are required to obtain an FSSAI State license
…ry. Most ontological frameworks do not consider classes to be universals, although some prominent philosophers do, such as John Bigelow. The term universality also refers to the medieval concept of an absolute, all-encompass
WhiskeyDick is an American acoustic, country-metal
…O standard of 1852m. One nm is taken to be equal to 6080 feet in calculations. · For all practical purposes while calculating distances, it is assumed that one nm is equal to the length of an arc of a great circle
… cargo, but they had cabins which enabled them to be used for long voyages of a month or so. One of his designs was the 15m cutter rig yacht Tally Ho. Designs Albert Strange Association There is an active Association which has as
… more powerful integrals like the Henstock–Kurzweil integral, every function for which an antiderivative exists is integrable, and its general integral coincides with its antiderivative. • If f has an antiderivative F on
…lt is 140.0376288224 pounds. How to convert 63.52 kg to lbs quicker and easier? You can also use the 63.52 kg to lbs converter , which will do whole mathematical operation for you and you will get an exact result . #### Kil
…ore responsibility in the hands of families, regarding the subjects they like to be occupied with and the methods they choose to advance in such subjects. Of course, the very concept of subjects is an invention of school, so
…ors, and the second matrix, usually called the R-matrix, being an upper triangular non-singular matrix. In fitting curves to data one often encounters overdetermined systems of linear equations. (An overdetermined system is where
…e for example, would it require more volts and the same current. So: 2@3v = 6v @700ma. Question by Lord_Tryndamere | last reply ### math/school help I need some help in school. today, we had an exam. our teacher did
Suhas Pandurang Sukhatme is an Indian scientist, teacher,
…ose proximity. Other examples of triggered events include natural disasters like bush fire or flood, war or torture, a serious motor vehicle accident and physical or sexual assault. The severity of an episode can be intensified if
…ip around the sun? M. Poarch – 2007 science-class.net 4. 4. Calculating Speed 43. You drive 150 miles in 3 hours before stopping for 30 minutes for lunch and gas.After lunch you travel 100 miles in an hour and a half.
…th sugarcane juice, tender coconut water, well ripened bananas has numerous benefits on plants. The cost is very low compared to chemical fertilizers. Panchagavya benefits for Plants Panchagavya is an effective manure. P
…pproximately fair. Then ${\bar p\approx\frac{1}{2}}$ so ${\bar p(1-\bar p)\approx \frac{1}{4}}$. If the confidence interval comprises values within two standard errors (${\kappa=2}$) and we require an accuracy of three significant figures
…y indicate that your pet is not acting or feeling normal are a loss of appetite, acting listless or sluggish, having an overly swollen stomach, and missing or deformed limbs. Another problem can be an infestation of mites
…o the town of Al-Baghuz Fawqani to carry out medical assistance tasks for the fighters and mobilize volunteers to fight against the last stronghold of the Islamic State in Syria. On March 18, 2019, an ISIS-related media announced on
Giampetro Puppi (20 November 1917 – 25 December 2006) was an Italian physicist who is known
…ns opposite the hotel. Davis published Mineral Baths of Bath; the Bath of Bathes Ayde in the Reign of Charles II (Bath, 1883), and other several pamphlets on the same subject. In 1863 he designed an escritoire, Bath
…’s heir and led an uprising against Henry in 1487 that ended in his death. Despite this, Edmund had been allowed to inherit the dukedom of Suffolk after his father’s death, though it was demoted to an earldom in
…on paper — makes a contribution. They learn something about the world, and about art, even if they don’t have that name for it. Art is not merely what hangs on gallery walls. Its primary purpose is an interaction with the world,
… team). As a senior, he made 11-of-18 field goals and 40-of-43 extra points, finishing with 73 points (second on the team). Professional career Dallas Cowboys (first stint) Hilbert was signed as an undrafted free agent by
General Mohsen Vezvaei (Persian: محسن وزوایی ; 29 July 1960 – 30 April 1982) was an officer of the Islamic Revolutionary
… of Weld Quay (C. Y. Choy Rd.) It was the last clan jetty to be built. They found that they need support from mainstream Chinese community in Penang. The Koay settlement in Penang saw themselves as an extension of their original clan
…ot had been a civilian. In London, Charlie Chaplin's last film, A Countess from Hong Kong, is screened in world premiere The White House confirmed that U.S. President Lyndon B. Johnson had rejected an official portrait painted of him
…s and their duties,” he said. To boost morale, Somali President Sharif Sheikh Ahmed promoted 141 police officials on May 11th of this year. The weapons, however, are not without controversy. Due to an archaic 1992
… belong to a new generation of voters which is at its core more liberal than previous generations were at that age (a “cohort effect”). (Another possible cause for differences in voting patterns is an event that affects everyone regardless
…ular interaction predicates a set of normative literacies that are grounded in an awareness of cultural power and bias. Knowledge allocates privilege and status. A critical pedagogy for literacy is an approach “that questions the
…und of body weight. This means protein recommendations depend on body weight as well as age and gender. For example an adult male who weighs 190 lb would need to consume 68.4 g of protein a day and an adult female who weighs
…ng and ranking Amazon uses matching to identify the products that match the search query and ranking to sort them. We want to go into both in more detail in the following. Find relevant products In an article published in 201
This community is active on prompts ending in the article “an.” The example prompts show a consistent pattern: the model originally predicts a word starting with a vowel. Zero-ablating these six neurons shifts its prediction toward words that still fit the context but start with a consonant. Conversely, keeping these six neurons (alongside four “always-on” neurons) while removing the remaining MLP neurons in those six layers (L2, L5, L8, L12, L13, L14) largely preserves the original prediction.

It turns out that this community is active on all “an” predictions. In the figure above we test a random set of prompts ending in the article “an,” without selecting them based on whether the community is active. We compare three interventions:
- Remove the community (orange). Removing just six neurons produces large changes in the next-token distribution, measured by KL divergence on the left, and in the probability of the model’s original top prediction on the right.
- Remove the MLP computation in the six layers (green). This produces similarly large changes.
- Keep the community within those layers (purple). Removing the other MLP neurons while retaining the community largely preserves both the prediction distribution and the original top token’s probability.
It is worth noting that our sufficiency claims are limited to the layers which contain community neurons, in this case six of them. The rest of the model still contributes to the prediction. These tests establish that the relevant computation in the tested layers is neuron-sparse, but don’t say anything about how the community interacts with the rest of the model and exactly what it computes to favor vowel-starting words.
Further examples
The “an” community is one of thousands discovered by our method. The interactive examples above show the range of computations these groups support, together with the model’s predictions before and after removing each community.
- Grammar. 17 neurons across eight layers help select “is” or “are” after “there.” Removing them changes the continuation of “Ten cards are put into five envelopes so that there” from “are” to “is.”
- Retrieval and computation. For instance, community supporting unit conversion and more generic multiplication. Some communities assist with in-context retrieval: e.g. seven neurons across four layers help retrieve names mentioned earlier in a prompt.
- Topic. Some communities are active on a specific topic, e.g. 3-neuron community activates on different trigonometric functions or 6-neuron community that activates on terms related to elevation.
- Formatting. These communities encode formatting rules and conventions, e.g. 7-neuron community responsible for continuing with ALL CAPS in a phrase.
- Common phrases. Many communities encode common phrases or a specific use of a word, e.g. 37-neuron community that’s active when predicting ‘out’ as a part of a phrasal verb (carry out, find out, figure out)
- Token suppression. This is one of the most common communities by type. Usually 1-2 neurons suppress one specific word or token.
You can explore hundreds more communities in our neuron community explorer.
The discovered communities also reveal patterns in where different computations occur. The heatmap shows how each category’s neurons are distributed across layers. Grammar-related neurons are more concentrated from layer 18 onward, while retrieval and computation—including arithmetic—peak around layers 17–22. Token suppression is concentrated in the final layers. These patterns suggest that neuron communities can help us map not only individual computations, but also how different kinds of computation are organized across the model’s depth.
Conclusion
Prior work has shown that individual neurons can be interpretable. In this work, we show that neurons form small groups that act as reusable units of computation, supporting behaviors from grammatical agreement to copying and arithmetic. We can discover these groups at scale without specifying their functions in advance, then test whether they are necessary and sufficient for predictions within selected layers.
These communities offer a level of abstraction between individual neurons and model behaviors, helping us investigate how neurons combine to implement higher-order computations. The explanation remains partial: many predictions remain unexplained, and identifying a necessary and sufficient community does not yet tell us exactly how its neurons compute or interact with the rest of the model. But unsupervised discovery gives us concrete starting points for investigating these questions at scale, including through agents that interpret communities and test hypotheses about their mechanisms. Connecting these findings could help us build a fuller understanding of how neural networks work—and how to control their behavior.
Citation
To cite this work, please cite the paper.
Footnotes
-
We also identify 17 “always-on” neurons across the model’s 32 layers and keep them on for all sufficiency tests. ↩
-
We identified hundreds of thousands of these neuron communities in Qwen 3.6 35B-A3B as well, including many circuits that encode pieces of knowledge—e.g., state capitals or famous painters associated with post-Impressionism. ↩
LLMs
Life Sciences