/explore
|
You are here 
|
www.lesswrong.com | ||
| | | | |
transformer-circuits.pub
[AI summary] The text discusses the interpretability of features in a machine learning model, focusing on how features like Arabic, base64, and Hebrew are used in interpretable ways. It explores the extent to which these features explain the model's behavior, noting that features with higher activations are more interpretable. The text also addresses the limitations of current methods, such as the computational cost of simulating features and the potential for dataset correlations to influence feature interpretations. Finally, it concludes that the model's learning process creates a richer structure in its activations than the dataset alone, suggesting that feature-based interpretations provide meaningful insights into the model's behavior. |
|
| | | | | ||
| | | | |
goodfire.ai
Goodfire is an AI research company building practical interpretability tools for safe and reliable generative models. |
|
| | | | | ||
| | | | |
www.greaterwrong.com
Produced as part of the ML Alignment & Theory Scholars Program - Summer 2024 Cohort To recover all the relevant features from a superintelligent language model, we will likely need to scale sparse autoencoders (SAEs) to billions of features. Using current architectures, training extremely wide SAEs across multiple layers and sublayers at various sparsity levels is computationally intractable. Conditional computation has been used to scale transformers (Fedus et al.) to trillions of parameters while retaining computational efficiency. We introduce the Switch SAE, a novel architecture that leverages conditional computation to efficiently scale SAEs to many more features. |
|
| | | | | ||
| | | | |
www.jamesserra.com
[AI summary] This technical blog post explains the concepts behind LLMs and Generative AI, details their architecture, and guides developers on using Azure and Microsoft Copilot Studio to apply these models to enterprise data via RAG techniques. |
|
| | | |||
