Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0314· OpenAIAi2 WildChat-4.8M (AI2)

Summarize this and list important information as bullet points: AI OpenAI’s new tool…

  1. User

    Summarize this and list important information as bullet points:

    AI OpenAI’s new tool attempts to explain language models’ behaviors Kyle Wiggers @kyle_l_wiggers / 6:00 PM GMT+1•May 9, 2023

    OpenAI's logo Image Credits: OpenAI It’s often said that large language models (LLMs) along the lines of OpenAI’s ChatGPT are a black box, and certainly, there’s some truth to that. Even for data scientists, it’s difficult to know why, always, a model responds in the way it does, like inventing facts out of whole cloth.

    In an effort to peel back the layers of LLMs, OpenAI is developing a tool to automatically identify which parts of an LLM are responsible for which of its behaviors. The engineers behind it stress that it’s in the early stages, but the code to run it is available in open source on GitHub as of this morning.

    “We’re trying to [develop ways to] anticipate what the problems with an AI system will be,” William Saunders, the interpretability team manager at OpenAI, told TechCrunch in a phone interview. “We want to really be able to know that we can trust what the model is doing and the answer that it produces.”

    To that end, OpenAI’s tool uses a language model (ironically) to figure out the functions of the components of other, architecturally simpler LLMs — specifically OpenAI’s own GPT-2.

    OpenAI explainability OpenAI’s tool attempts to simulate the behaviors of neurons in an LLM. Image Credits: OpenAI

    How? First, a quick explainer on LLMs for background. Like the brain, they’re made up of “neurons,” which observe some specific pattern in text to influence what the overall model “says” next. For example, given a prompt about superheros (e.g. “Which superheros have the most useful superpowers?”), a “Marvel superhero neuron” might boost the probability the model names specific superheroes from Marvel movies.

    OpenAI’s tool exploits this setup to break models down into their individual pieces. First, the tool runs text sequences through the model being evaluated and waits for cases where a particular neuron “activates” frequently. Next, it “shows” GPT-4, OpenAI’s latest text-generating AI model, these highly active neurons and has GPT-4 generate an explanation. To determine how accurate the explanation is, the tool provides GPT-4 with text sequences and has it predict, or simulate, how the neuron would behave. In then compares the behavior of the simulated neuron with the behavior of the actual neuron.

    “Using this methodology, we can basically, for every single neuron, come up with some kind of preliminary natural language explanation for what it’s doing and also have a score for how how well that explanation matches the actual behavior,” Jeff Wu, who leads the scalable alignment team at OpenAI, said. “We’re using GPT-4 as part of the process to produce explanations of what a neuron is looking for and then score how well those explanations match the reality of what it’s doing.”

    The researchers were able to generate explanations for all 307,200 neurons in GPT-2, which they compiled in a dataset that’s been released alongside the tool code.

    Tools like this could one day be used to improve an LLM’s performance, the researchers say — for example to cut down on bias or toxicity. But they acknowledge that it has a long way to go before it’s genuinely useful. The tool was confident in its explanations for about 1,000 of those neurons, a small fraction of the total.

    A cynical person might argue, too, that the tool is essentially an advertisement for GPT-4, given that it requires GPT-4 to work. Other LLM interpretability tools are less dependent on commercial APIs, like DeepMind’s Tracr, a compiler that translates programs into neural network models.

    Wu said that isn’t the case — the fact the tool uses GPT-4 is merely “incidental” — and, on the contrary, shows GPT-4’s weaknesses in this area. He also said it wasn’t created with commercial applications in mind and, in theory, could be adapted to use LLMs besides GPT-4.

    OpenAI explainability The tool identifies neurons activating across layers in the LLM. Image Credits: OpenAI

    “Most of the explanations score quite poorly or don’t explain that much of the behavior of the actual neuron,” Wu said. “A lot of the neurons, for example, are active in a way where it’s very hard to tell what’s going on — like they activate on five or six different things, but there’s no discernible pattern. Sometimes there is a discernible pattern, but GPT-4 is unable to find it.”

    That’s to say nothing of more complex, newer and larger models, or models that can browse the web for information. But on that second point, Wu believes that web browsing wouldn’t change the tool’s underlying mechanisms much. It could simply be tweaked, he says, to figure out why neurons decide to make certain search engine queries or access particular websites.

    “We hope that this will open up a promising avenue to address interpretability in an automated way that others can build on and contribute to,” Wu said. “The hope is that we really actually have good explanations of not just what neurons are responding to but overall, the behavior of these models — what kinds of circuits they’re computing and how certain neurons affect other neurons.”

    More TechCrunch

    Google's 'Sidekick' offers contextual prompts to get better results from AI

    Theranos founder Elizabeth Holmes isn't headed to jail tomorrow after all

    American psychology group issues recommendations for kids' social media use

    OpenAI’s ChatGPT is shaking up the edtech markets

    Please login to comment Login / Create Account

    TechCrunch Disrupt Sept 19-21 San Francisco, CA Register Now Sign up for Newsletters See all newsletters(opens in a new window)

    Daily

    Week in Review

    Startups Weekly

    Event Updates

    Advertising Updates

    TechCrunch+ Announcements

    TechCrunch+ Events

    TechCrunch+ Roundup

    Email Subscribe (opens in a new window)(opens in a new window)(opens in a new window)(opens in a new window)(opens in a new window) AI OpenAI’s new tool attempts to explain language models’ behaviors Kyle Wiggers 6:00 PM GMT+1•May 9, 2023 OpenAI's logo AI EU lawmakers back transparency and safety rules for generative AI Natasha Lomas 11:18 AM GMT+1•May 11, 2023 Artificial Intelligence for Deep Learning Technology over Top view scene of Motion blurred Crowd unrecognizable pedestrians walking internal subway intersection in rush hour before working hour, central Hong Kong. Gadgets Canon sidles up to vloggers with PowerShot V10 Haje Jan Kamps 10:50 AM GMT+1•May 11, 2023

    Apps Twitter launches encrypted DMs for verified users with security drawbacks Ivan Mehta 10:34 AM GMT+1•May 11, 2023 distorted, painterly twitter logo with a background of gears Gadgets Fairphone gets its audio groove on with repairable over-ear BT headphones Natasha Lomas 10:00 AM GMT+1•May 11, 2023 Fairphone Fairbuds XL modular headphones in black - exploded view Startups Microsoft makes strategic investment into Builder.ai, integrates its services into Teams Mike Butcher 9:00 AM GMT+1•May 11, 2023

    Apps Google’s new Labs page lets you sign up for its AI experiments Ivan Mehta 8:49 AM GMT+1•May 11, 2023 Google Labs Venture SoftBank Vision Fund yearly loss widens to 32BonstartupsvaluationcutManishSingh8:06AMGMT+1•May11,2023SoftBankVisionFundloses32B on startups valuation cut Manish Singh 8:06 AM GMT+1•May 11, 2023 SoftBank Vision Fund loses 32 billion in a year as startups confront valuation cut AI Everseen raises over $70M for AI tech to spot potential retail theft Kyle Wiggers 8:00 AM GMT+1•May 11, 2023

    Transportation Wingcopter, Germany’s drone delivery startup, raises another $44M from the EIB Ingrid Lunden 7:20 AM GMT+1•May 11, 2023

    Media & Entertainment Disney’s Hotstar loses 8.4 million subscribers as Ambani’s JioCinema ascends Manish Singh 6:16 AM GMT+1•May 11, 2023 Ambani bats for cricket glory as Disney scales back TechCrunch Disrupt 2023 APAC startups: Apply to Startup Battlefield 200 Lauren Simonds 1:00 AM GMT+1•May 11, 2023

    Startups Daily Crunch: OpenAI, Anthropic and Stability AI receive half of Sound Ventures’ $240M AI fund Christine Hall, Haje Jan Kamps 11:05 PM GMT+1•May 10, 2023 CVC, corporate venture capital, Media & Entertainment Disney+ and Hulu content to combine into one streaming app Lauren Forristal 10:10 PM GMT+1•May 10, 2023 Disney+ app on phone AI Google I/O 2023 is a wrap — here’s a list of everything announced Christine Hall 9:39 PM GMT+1•May 10, 2023 Google CEO Sundar Pichai presents at Google I/O 2023 Apps Google ‘Perspectives’ integrates Reddit, YouTube, TikTok and more in search results Sarah Perez 9:31 PM GMT+1•May 10, 2023 google search app ios Startups The Mint, started by Better Tomorrow Ventures, wants to be the accelerator fintech needs Natasha Mascarenhas 9:30 PM GMT+1•May 10, 2023 deal pace picking up Media & Entertainment Disney+ loses subscribers for second quarter in a row, drops 4M subs Lauren Forristal 9:18 PM GMT+1•May 10, 2023

    Transportation Google is bringing YouTube to more cars, starting with Polestar Rebecca Bellan 9:01 PM GMT+1•May 10, 2023 YouTube on display screen of Polestar vehicle Transportation Google adding Cisco, Microsoft and Zoom (audio) conferencing to Android Auto Rebecca Bellan 9:00 PM GMT+1•May 10, 2023 Zoom android auto google Transportation All the ways Google is driving deeper into the automotive world Rebecca Bellan, Kirsten Korosec 9:00 PM GMT+1•May 10, 2023 Waze Google built in car About TechCrunch Staff Contact Us Advertise Crunchboard Jobs Site Map Legal Terms of Service Privacy Policy TechCrunch+ Terms Privacy Dashboard Code of Conduct About Our Ads Trending Tech Topics Tech Layoffs ChatGPT Facebook(opens in a new window)Twitter(opens in a new window)YouTube(opens in a new window)Instagram(opens in a new window)LinkedIn(opens in a new window)Mastodon(opens in a new window) © 2023 Yahoo. All rights reserved. Powered by WordPress VIP(opens in a new window).

  2. Assistant
    • OpenAI is developing a tool to identify which parts of large language models (LLMs) are responsible for their behaviors.
    • The tool uses a language model to figure out the functions of OpenAI's own GPT-2 language model's components.
    • The tool identifies highly active neurons, uses GPT-4 to generate an explanation, and then simulates neuron behavior to determine accuracy.
    • It successfully generated explanations for all 307,200 neurons in GPT-2, though it was only confident in its explanations for around 1,000 of them.
    • The tool could potentially be used to improve LLMs, including reducing bias or toxicity.