Transform the following into a linkedin post rather than a thread: How should the…
- User
Transform the following into a linkedin post rather than a thread: How should the humanities leverage LLMs?
Domain-specific pretraining! Pretraining models can be a research tool, it's cheaper than LoRA, and allows studying
- grammatical change
- emergent word senses
- and who knows what more… Train on your data with our pipeline or use ours! #AI #LLM #linguistics Typical Large Language Models (LLMs) are trained on massive, mixed datasets, so the model's behaviour can't be linked to a specific subset of the pretraining data. Or in our case, to time eras. Historical analysis is a good example, as historical periods can get lost in blended information from different eras. Finetuning large models isn't enough, they “leak” future/modern concepts, making historical analysis impossible. Did you know cars existed in the 1800s? 🤦 But what if we pretrain instead? You can get a unique LLM trained on small time corpora (10M tokens). “Are you crazy? It will be so costly” Oh no, it is even more efficient than training those monster LoRAs. Using pretraining, we are able to track historical shifts, like evolving negative polarity item patterns in "only...ever" and "even...ever" - something not seen in the finetuned models. It was also cool to explore word sense change, using simple differences in surprisal metrics.
Assistant
How Humanities Can Leverage LLMs: The Power of Domain-Specific Pretraining As a linguist exploring the intersection of AI and humanities, I'm excited to share how domain-specific pretraining of LLMs can revolutionize humanities research! Most Large Language Models are trained on massive mixed datasets where specific time periods and cultural contexts get blended together. This creates challenges for historical analysis - traditional LLMs "leak" modern concepts into historical contexts (yes, some think cars existed in the 1800s! 🤦). While fine-tuning helps, it's not enough for precise historical work. That's where domain-specific pretraining shines - and it's more cost-effective than developing large LoRAs! By pretraining models on targeted corpora (just 10M tokens), we've successfully:
Tracked historical grammatical shifts like evolving negative polarity patterns in "only...ever" and "even...ever" constructions Explored word sense change using differences in surprisal metrics Created era-specific language models that maintain historical integrity
Our team has developed a pipeline that allows researchers to train models on their specialized datasets. Interested in exploring what domain-specific LLMs could reveal in your humanities field? #AI #LLM #DigitalHumanities #Linguistics #HistoricalAnalysis #ResearchTools
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.