Natural Language Processing

• Courses and Books
• Scientific NLP
• Tools
• Pretrained embeddings and models
• Links
• Lexicons
• Data
• Shared tasks, competitions
• Research Groups
• Mailing lists

Table of contents generated with markdown-toc

Courses and Books

• Statistical NLP Book: https://github.com/uclmr/stat-nlp-book/blob/python/overview.ipynb
• Course on word embeddings, variants, and applications: http://people.ds.cam.ac.uk/iv250/esslli2018.html
• A Course in Machine Learning: http://ciml.info/
• Introduction to Natural Language Processing by Jacob Eisenstein - https://github.com/jacobeisenstein/gt-nlp-class
• Information extraction: https://web.stanford.edu/~jurafsky/slp3/17.pdf covers main ideas of information extraction
• 601.765 Machine Learning: Linguistic & Sequence Modeling - https://seq2class.github.io/
• Variational Inference for NLP - https://github.com/philschulz/VITutorial#general
• Probabilistic NLP course - https://uva-slpl.github.io/nlp2/syllabus.html
• Oxford Deep Learning NLP - https://github.com/oxford-cs-deepnlp-2017/lectures
• CMU NLP - http://phontron.com/class/nn4nlp2017/schedule.html
• David Bamman's Course on Applied NLP - https://github.com/dbamman/anlp19 - Course Page
• NLP Course | For You by Lena Voita - https://lena-voita.github.io/nlp_course.html
• Course material for Machine Translation from UIUC, JHU - http://mt-class.org/
• Jason Eisner - http://www.cs.jhu.edu/~jason/465/
• Introduction to Cultural Analytics & Python - https://melaniewalsh.github.io/Intro-Cultural-Analytics/
• Computational Sociolinguistics by David Jurgens - https://docs.google.com/document/d/1Ouyqz-emtOI-ohwTOdOZpcjcEFtPPilhJDso8sjgByU/edit

Scientific NLP

• https://github.com/NIHOPA/NLPre
• https://shubhanshu.com/awesome-scholarly-data-analysis/

Tools

• Using spacy and flashtext for fast lookup of dictionary items in text. https://github.com/mpuig/spacy-lookup
• Open IE tool - https://github.com/dair-iitd/OpenIE-standalone
• Different evaluation techniques for NER: https://github.com/davidsbatista/NER-Evaluation
• Phrase extraction using POS patterns: https://github.com/slanglab/phrasemachine
• Whats wrong with my NLP in Python diff visualizer for NLP tasks - https://github.com/ppke-nlpg/whats-wrong-python
• Inform - interactive fiction based on natural language: http://inform7.com/
• Linguistic Knowledge and Transferability of Contextual Representations - https://github.com/nelson-liu/contextual-repr-analysis
• Generalized brown clusters - https://github.com/sean-chester/generalised-brown
• COGCOMP NLP: Online demo with multiple tasks - http://nlp.cogcomp.org/ Code: https://github.com/CogComp/cogcomp-nlp
• Python keyphrase extraction library - https://github.com/boudinfl/pke
• Neural Relation Extration - https://github.com/thunlp/OpenNRE
• List of tools for corpus analysis - https://corpus-analysis.com/
• WordMapper, evolution of words on Twitter - https://sites.google.com/site/wordmapperinfo/
• Document subject indexing and thesaura linking (also works for Finnish) - http://annif.org/
• BPEmb: Pre-trained Subword Embeddings in 275 Languages (LREC 2018) - https://doi.org/10.11588/data/V9CXPR
• Cornell Conversational Analysis Toolkit - https://zissou.infosci.cornell.edu/convokit/documentation/index.html
• Python based text summarization evaluation - https://github.com/chakki-works/sumeval
• Python based sequence tagging evaluation - https://github.com/chakki-works/seqeval
• NLP annotation tool - https://github.com/doccano/doccano
• Yake Single document keyword extraction - https://github.com/LIAAD/yake
• Open IE Tool - https://github.com/dair-iitd/OpenIE-standalone Other OpenIE tools
• Derive named entities from Wikipedia - https://github.com/kno10/WikipediaEntities
• Bengali NLP - https://github.com/sagorbrur/bnlp
• Py Readability metrics - https://github.com/cdimascio/py-readability-metrics
• iNLTK - https://inltk.readthedocs.io/en/latest/
• Fast spell correction and word segmentation - https://github.com/wolfgarbe/symspell
• NERD evaluation - https://nerd.readthedocs.io/en/latest/evaluation.html
• MedCAT can be used to extract information from Electronic Health Records (EHRs) and link it to biomedical ontologies like SNOMED-CT and UMLS - https://github.com/CogStack/MedCAT
• PyTerrier - A Python framework for performing information retrieval experiments, building on http://terrier.org/ - https://github.com/terrier-org/pyterrier
• JamSpell - Spelling correction (works in python) - https://github.com/bakwc/JamSpell

Pretrained embeddings and models

• ELMO for Many Languages - https://github.com/HIT-SCIR/ELMoForManyLangs
• Flair for many languages - https://github.com/flairNLP/flair-lms
• Pretrained transformer models for many languages and domains - https://huggingface.co/models
• FastText embeddings - https://fasttext.cc/

Links

• Bayesian NLP: https://homepages.inf.ed.ac.uk/sgwater/resources.html
• Rules for forming questions using text: https://ell.stackexchange.com/questions/1156/when-converting-a-statement-to-a-question-where-in-the-sentence-should-i-put-t/1198#1198
• Intel NLP library lots of great features using Tensorflow: http://nlp_architect.nervanasys.com/publications.html
• Information extraction from unstructured text: https://sites.google.com/site/keit2018kdd/
• Tutorial on deep latent variable models for NLP with pytorch code: https://nlp.seas.harvard.edu/latent-nlp-tutorial.html
• On the viability of crowdsourcing NLP annotations in healthcare: https://roamanalytics.com/2018/07/26/on-the-viability-of-crowdsourcing-nlp-annotations-in-healthcare/
• Contextual Word Representations: A Contextual Introduction - https://arxiv.org/abs/1902.06006
• Emotion vocabularies - https://www.w3.org/TR/emotion-voc/
• Emotion markup language - https://www.w3.org/TR/emotionml/
• NAACL 2019 tutorial on Tutorial on Modeling Language Change - https://github.com/jacobeisenstein/language-change-tutorial
• NAACL 2019 tutorial on Transfer Learning in Natural Language Processing - https://github.com/huggingface/naacl_transfer_learning_tutorial
• Papers on Textual Adversarial Attack and Defense: https://github.com/thunlp/TAADpapers
• Resources for NRE: Neural Relation Extraction: https://github.com/thunlp/NREPapers
• Online self-learning platform on vocabulary: http://vlearn.fed.cuhk.edu.hk/
• Awesome sentiment analysis - https://github.com/xiamx/awesome-sentiment-analysis
• List of papers on Textual Adversarial Attack and Defense - https://github.com/thunlp/TAADpapers
• Tutorial on NLP Approaches to Computational Argumentation - http://acl2016tutorial.arg.tech/index.php/tutorial-materials/
• Tutorial on Argumentation Mining - http://www.i3s.unice.fr/~villata/tutorialIJCAI2016.html
• ACL2019 Tutorial: Advances in Argument Mining - http://arg.tech/~chris/acl2019tut/index.html
• International Linguistic Olympiad - https://ioling.org/ - Covers a lot of low resource and multilingual problems
• Tutorial by Xavier Carreras on Structured Prediction - https://www.youtube.com/watch?v=f6Gqr2UCG9k&list=PLSWgH7JB2-1G2h8wj-ecK8FfpX72Z80_B&index=3&t=0s
• Visual Sentiment Ontology & Dataset - http://visual-sentiment-ontology.appspot.com/?
• NLP Roadmap - https://github.com/graykode/nlp-roadmap
• Pretrained LM model papers - https://github.com/thunlp/PLMpapers
• EmojiMap - http://kt.ijs.si/data/Emoji_sentiment_ranking/emojimap.html
• Literature review and datasets on Text-Summarization - https://github.com/neulab/Text-Summarization-Papers
• Causal Inference using text data - https://github.com/jaeyk/ITS-Text-Classification
• Tutorial on topic models and its extensions with animations of Chinese restaurant process and Poyla's Urn - http://topicmodels.west.uni-koblenz.de/
• Relation extraction resources - https://github.com/roomylee/awesome-relation-extraction
• Wordbank - An open database of children's vocabulary development - http://wordbank.stanford.edu/
• Comparison of web (text) annotation editors
• North American Computational Linguistics Open Competition - https://www.nacloweb.org/
• Frequency lists of words in all languages - https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists
• The NLP Pandect - https://github.com/ivan-bilan/The-NLP-Pandect
• Unicode text segmentation - http://www.unicode.org/reports/tr29/
• Datasets for Text Analysis - https://docs.google.com/spreadsheets/d/1I7cvuCBQxosQK2evTcdL3qtglaEPc0WFEs6rZMx-xiE/edit#gid=0
• Multilingual stopwords - https://github.com/stopwords-iso/stopwords-iso

Lexicons

• HistEmo lexicon for valance, arousal and dominance scores of words across time - https://github.com/JULIELab/HistEmo
• Sentiment lexicon - https://github.com/juliasilge/tidytext/blob/master/data-raw/sentiments.csv
• Polarity shifter lexicons - https://github.com/uds-lsv/polarity-shifter-resources
• Bootstrapped polarity shifter lexicons - https://github.com/uds-lsv/bootstrapped-lexicon-of-english-polarity-shifters
• Lexicon of abusive words - https://github.com/uds-lsv/lexicon-of-abusive-words
• Finnish General Ontology YSO, several domain-specific vocabularies, and the KOKO ontology - https://github.com/NatLibFi/Finto-data
• Entitypedia is an Extended Named Entity Dictionary from Wikipedia - https://github.com/chakki-works/entitypedia
• Extended Open Multilingual Wordnet - http://compling.hss.ntu.edu.sg/omw/summx.html
• Open Multilingual Wordnet - http://compling.hss.ntu.edu.sg/omw/index.html
• Open License English Wordnet - https://github.com/globalwordnet/english-wordnet
• HurtLex a lexicon of offensive, aggressive, and hateful words in over 50 languages - https://github.com/valeriobasile/hurtlex
• The Evaluative Lexicon - emotionality, valence, and extremity - http://www.evaluativelexicon.com/

Data

• Sentiment corpora: https://www.w3.org/community/sentiment/wiki/Datasets
• Relation Extraction Corpora: https://github.com/davidsbatista/Annotated-Semantic-Relationships-Datasets
• Summarization corpora: https://gist.github.com/napsternxg/2750479273e0621c5aa697bf89843428
• Text normalization corpora: https://github.com/rwsproat/text-normalization-data
• Sentihood data - Aspect based sentiment for neighborhood text - https://github.com/uclmr/jack/tree/master/data/sentihood
• Pretrained emoji embeddings and Twitter sentiment data - https://github.com/uclmr/emoji2vec
• Fake News Corpus: https://github.com/several27/FakeNewsCorpus
• EmoBank - Sentiment from perspective of reader's v/s writer's emotion - https://github.com/JULIELab/EmoBank
• Lot of argumentation mining corpora - https://www.informatik.tu-darmstadt.de/ukp/research_6/research_areas/argumentation_mining/index.en.jsp
• NER datasets - https://github.com/juand-r/entity-recognition-datasets
• More NER datasets - http://www.cs.technion.ac.il/~gabr/resources/data/ne_datasets.html
• Moral foundations twitter corpus - https://psyarxiv.com/w4f72
• Keyphrase extraction - https://github.com/titipata/keyphrase_extraction https://github.com/LIAAD/KeywordExtractor-Datasets#theses
• Multi-target-specific sentiment recognition on Twitter - https://github.com/bluemonk482/tdparse
• Multi (700 languages) lingual speech and text - https://github.com/festvox/datasets-CMU_Wilderness
• Large Scale text classification - http://lshtc.iit.demokritos.gr/
• Multiple text classification data - https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html
• Data for lifelong ML for sentiment classification - https://www.aclweb.org/anthology/papers/P/P15/P15-2123/
• Aspect extraction data for Amazon reviews for 36 domains (1000 per domain) - https://www.aclweb.org/anthology/papers/P/P14/P14-1033/
• Perspectrum: dataset of claims and supporting and opposing sentences and paragraphs - https://github.com/CogComp/perspectrum
• Multiple datasets curated by CogComp group - https://cogcomp.org/page/corpora/ and https://cogcomp.org/page/data/
• 204,135 articles from 18 American publications. Includes date, title, publication, article text, publication name, year, month, and URL (2013-2018)- https://components.one/datasets/all-the-news-articles-dataset/
• Legal Entity Detection - https://github.com/hockeyjudson/Legal-Entity-Detection
• New Yorker Cartoon Captions Funny v/s not-Funny dataset - https://github.com/nextml/caption-contest-data/tree/master/contests
• Semantic Text Similarity datasets - https://github.com/brmson/dataset-sts
• Another list of NLP datasets - https://github.com/shashankg7/Deep-Learning-for-NLP-Resources
• Few-Shot Relation Classification Dataset (FewRel), consisting of 70,000 sentences on 100 relations derived from Wikipedia and annotated by crowdworkers - https://github.com/thunlp/FewRel
• Named dataset - https://github.com/philipperemy/name-dataset
• Aspect/Target based sentiment prediction - https://github.com/apmoore1/Bella
• Relation classification - https://github.com/zhangdongxu/kbp37 https://github.com/deepakn97/relationPrediction
• Multimodal knowledge graphs - https://github.com/nle-ml/mmkb
• Distantly supervised relation extraction without false positives - https://github.com/pvthuy/distantly-supervised-RE
• Part whole relationship data - https://github.com/pvthuy/part-whole-relations
• Supersense-Tagged Repository of English with a Unified Semantics for Lexical Expressions for Web reviews from English Treebank - https://github.com/nert-nlp/streusle
• English WikiHow instructional guides semantically annotated with Universal Conceptual Cognitive Annotation (UCCA) - https://github.com/nert-nlp/Whow_UCCA
• English Keyphrase generation dataset - https://github.com/memray/seq2seq-keyphrase and https://github.com/kenchan0226/keyphrase-generation-rl
• Reverse dictionary evaluation data - https://github.com/uds-lsv/Multi-Sense-Embeddings-Reverse-Dictionaries
• Verbal shifter disambiguation - https://github.com/uds-lsv/disambiguation-of-verbal-shifters
• TextGraphs-13 Shared Task on Multi-Hop Inference Explanation Regeneration - https://competitions.codalab.org/competitions/20150
• Named Entities in European Newspapers (German, French, and Dutch) - https://github.com/EuropeanaNewspapers/ner-corpora
• Document and subject indexing corpora - https://github.com/NatLibFi/Annif-corpora
• GENETAG - https://github.com/openbiocorpora/genetag
• Open Biomedical corpora - https://github.com/openbiocorpora
• Multilingual NER - https://github.com/afshinrahimi/mmner - PANX - https://www.amazon.com/clouddrive/share/d3KGCRCIYwhKJF0H3eWA26hjg2ZCRhjpEQtDL70FSBN?_encoding=UTF8&%2AVersion%2A=1&%2Aentries%2A=0&mgh=1
• WikiANN 282 languages NER - http://nlp.cs.rpi.edu/wikiann/ - Gdrive
• Amazon QA review based question answering corpus - https://github.com/amazonqa/amazonqa
• Arabic NER - http://www.cs.cmu.edu/~ark/ArabicNER/
• NERWebpagesColumns from CogComp UIUC - https://cogcomp.seas.upenn.edu/page/resource_view/28
• Wikification evaluation data - https://cogcomp.seas.upenn.edu/page/resource_view/4
• Named Entity Coreference resolution across documents data - https://cogcomp.seas.upenn.edu/page/resource_view/44
• Seminars and Job posting Data (Named Entities) - https://cogcomp.seas.upenn.edu/page/resource_view/31
• List of various English NER datasets - https://github.com/dice-group/FOX/tree/master/input
• Document level relation extraction data using English Wikipedia - https://github.com/thunlp/DocRED
• Medinify dataset of medical comments tagged with drug mentions and ease of use - https://github.com/NLPatVCU/medinify-datasets
• Nano particles entity annotated data (in brat format) - https://github.com/NLPatVCU/medaCy_dataset_end
• Named entities dataset in brat format from Systematic Review Information Extraction (SRIE) 2018 - https://github.com/NLPatVCU/medaCy_dataset_tac_2018
• Ontonotes and FIGER gold data - http://nlp.cs.rpi.edu/kbp/2019/data.html - Wayback link
• FIGER training data - https://github.com/xiaoling/figer
• KNET fine grained named entity data - https://github.com/thunlp/KNET
• Drug Drug interaction data for TAC 2019 task (NER and RE) - https://bionlp.nlm.nih.gov/tac2019druginteractions/
• Model Sense classification - https://github.com/amarasovic/modal-sense-classifcation
• Entity Sentiment TAC KBP - https://tac.nist.gov//2014/KBP/data.html
• MPQA 3.0 Entity Sentiment data - https://mpqa.cs.pitt.edu/corpora/mpqa_corpus/
• Cross-Lingual Sentiment (CLS) dataset comprises about 800.000 Amazon product reviews - https://webis.de/data/webis-cls-10.html
• MLDoc: A Corpus for Multilingual Document Classification in Eight Languages - https://github.com/facebookresearch/MLDoc
• Topical Chat: human-human dataset of open-domain conversations - https://github.com/alexa/alexa-prize-topical-chat-dataset/
• Human to Human actionable request dataset - https://github.com/NervanaSystems/nlp-architect/tree/master/datasets/H2H%20requests%20detection
• Human written weather summaries - https://ehudreiter.files.wordpress.com/2016/12/sumtime.zip
• List of datasets for Natural Language Generation (NLG) - https://aclweb.org/aclwiki/Data_sets_for_NLG
• EmotionX:emotions induced by dialogue utterances - https://sites.google.com/view/emotionx2019/
• VU Amsterdam Metaphor Corpus - http://ota.ahds.ac.uk/headers/2541.xml Codalab
• Abusive language datasets - https://sites.google.com/view/alw3/resources?authuser=0
• Multilingual Surface Realization - http://taln.upf.edu/pages/msr2019-ws/SRST.html#data
• Argument mining microtexts (not social media; available in English and German) - https://github.com/peldszus/arg-microtexts
• List of resources in argument mining - https://www.informatik.tu-darmstadt.de/ukp/research_6/data/index.en.jsp
• Unibo corpora for argument mining - http://argumentationmining.disi.unibo.it/resources.html
• Datasets on natural argumentation - http://www-sop.inria.fr/NoDE/NoDE-xml.html
• Dataset on natural arguments with emotions - https://project.inria.fr/seempad/datasets/
• IBM debator dataset (multi task dataset) - http://www.research.ibm.com/haifa/dept/vst/debating_data.shtml
• El Capitan corpus for sentiment and topic at both the review (document) and sentence level - https://github.com/eibeke/El-Capitan-Dataset
• Argument Mining European Court of Human Rights - https://github.com/PLN-FaMAF/ArgumentMiningECHR
• Question Classification Labels for Science Questions - http://cognitiveai.org/explanationbank/
• Wikipedia Biographies data (for assessing text generation algorithms) - https://github.com/DavidGrangier/wikipedia-biography-dataset
• TASKMASTER-1 DIALOG CORPUS: TOWARD A REALISTIC AND DIVERSE DATASET - https://mila.quebec/en/publication/taskmaster-1-dialog-corpus-toward-a-realistic-and-diverse-dataset/
• Parallelly Annotated Stylistic Language Dataset with Multiple Personas - https://github.com/dykang/PASTEL
• A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking (UKP Snopes Corpus) - https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2081 Paper
• Argument Aspect Similarity (UKP ASPECT) Corpus - https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/1998 Paper
• VU Amsterdam Metaphor Corpus - https://github.com/UKPLab/acl2019-GPPL-humour-metaphor/tree/master/data/VU%20Amsterdam%20Metaphor%20Corpus/2541
• VUAMC_crowd Dataset for evaluating humour (containing 28,210 pairwise comparisons of 4030 texts) - https://github.com/UKPLab/acl2019-GPPL-humour-metaphor/tree/master/data/vuamc_crowd
• Argument convincingness from crowdsourced data - https://github.com/UKPLab/tacl2018-preference-convincing
• Discourse-level argumentation annotations - https://github.com/UKPLab/naacl2019-argument-annotations
• CNN/Daily Mail summarization - https://github.com/UKPLab/emnlp2019-summary-reward/tree/master/data
• IBM Debator Evidence Detection data - https://github.com/UKPLab/fever2019-interactive-evidence-detection/tree/master/data
• QA data for benchmarking entity linking systems - https://public.ukp.informatik.tu-darmstadt.de/starsem18-entity-linking/EntityLinkingForQADatasets.zip Code
• Wikipedia-Wikidata sentence-level relation annotations - https://www.informatik.tu-darmstadt.de/ukp/research_6/data/lexical_resources/wikipedia_wikidata_relations/index.en.jsp
• Live Blog Corpus for Summarization - https://github.com/UKPLab/lrec2018-live-blog-corpus
• Wikidata/FrameNet Alignment - https://www.informatik.tu-darmstadt.de/ukp/research_6/data/lexical_resources/wikidata_framenet_alignments/index.en.jsp
• Event time extraction - https://github.com/UKPLab/tacl2017-event-time-extraction/tree/master/input
• MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims - https://copenlu.github.io/publication/2019_emnlp_augenstein/
• Conversation datasets in ConvoKit - https://zissou.infosci.cornell.edu/convokit/documentation/datasets.html
• Dataset of personal narratives with Advice-Seeking Questions - https://github.com/CornellNLP/ASQ
• Ontonotes documents labeled using Freebase entity types (included as part of paper source files) - https://arxiv.org/format/1412.1820
• Google book corpora catalogue - http://storage.googleapis.com/books/
• Multiple AIDA datasets for Named Entity Linking - https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/aida/downloads/
• Named Entity Disambiguation for Noisy Text - https://github.com/yotam-happy/NEDforNoisyText
• Multiple datasets for wikipedia based disambiguation - http://rali.iro.umontreal.ca/rali/?q=en/wikipedia-ds-cont-emb
• VideoStory: Text summaries of videos - https://zenodo.org/record/2383739#.Xe5mO-hKh3g
• German Named Entity Linking dataset - https://github.com/linkedtv/videocorpus
• TAC KBP English Entity Linking Comprehensive Training and Evaluation Data 2010 - https://github.com/hellojet/TAC_KBP_English_EL_2010/tree/e274785013be529f0acee347b75d6076bda11126
• Temporal fact extraction datasets - https://github.com/dmsquare/tfwin
• Updated and fixed Fake news challenge dataset - https://github.com/UKPLab/coling2018_fake-news-challenge
• Document Ranking datasets - https://microsoft.github.io/TREC-2019-Deep-Learning/
• Japanese aspect based sentiment analysis corpora - https://github.com/chakki-works/chABSA-dataset
• Political scaling dataset - https://bitbucket.org/gg42554/cl-scaling/src/master/
• Cross lingual text similarity - https://bitbucket.org/gg42554/cl-sts/src/master/data/
• Topical segmentation of text - https://bitbucket.org/gg42554/graphseg/src/master/data/
• Lexico-semantic relatedness (7 datasets) - https://bitbucket.org/gg42554/dual-tensors/src/master/data/
• Text simiplification - https://bitbucket.org/gg42554/embesimp/src/master/Data/
• Ultra-fine entity typing - https://homes.cs.washington.edu/~eunsol/open_entity.html
• Capturing Discriminative Attributes - https://competitions.codalab.org/competitions/17326#participate
• IBM Debating datasets with labels on Sentiment, Argumentation - https://www.research.ibm.com/haifa/dept/vst/debating_data.shtml
• LitBank is an entity annotated dataset of 100 works of English-language fiction - https://github.com/dbamman/litbank - https://github.com/dbamman/NAACL2019-literary-entities
• Nested Named Entity database - https://github.com/nickyringland/nested_named_entities
• WikiNER - http://web.archive.org/web/20141104075416/http://schwa.org/projects/resources/wiki/Wikiner
• English Web Treebank - https://github.com/UniversalDependencies/UD_English-EWT
• PAWS: Paraphrase Adversaries from Word Scrambling - https://github.com/google-research-datasets/paws PAWS-X multilingual
• Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus - https://traces1.inria.fr/oscar/
• IRC Conversation Disentanglement - https://jkk.name/irc-disentanglement/
• Multiple NER datasets including corrected CoNLL 2003 data - https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers
• The Big Bad NLP Database - https://quantumstat.com/dataset/dataset.html
• LRE Map A database containing around 6,000 language resources and tools published at LREC conferences - http://lremap.elra.info/
• OpenIE datasets - https://github.com/gabrielStanovsky/supervised-oie/
• Entity linking dataset in German based on news broadcasts transcripts - https://github.com/linkedtv/videocorpus
• Text summarization datasets - http://pfliu.com/pl-summarization/summ_data.html
• Multi-ling summarization datasets from shared task - http://multiling.iit.demokritos.gr/pages/view/1666/multiling-2019-call-for-papers
• TED-Parallel-Corpus based on translations of TED talks - https://github.com/ajinkyakulkarni14/TED-Multilingual-Parallel-Corpus
• Audio books corpora for speech to text - https://github.com/ajinkyakulkarni14/Audio-Book-Corpus-for-European-Languages-
• Target Based Speech Act Classification in Political Campaign Text - https://github.com/shivashankarrs/Speech-Acts
• Datasets for Aspect Level Sentiment Analysis - https://github.com/12190143/Deep-Learning-for-Aspect-Level-Sentiment-Classification-Baselines
• Curation Corpus for Abstractive Text Summarisation - https://github.com/CurationCorp/curation-corpus
• List of NER datasets curated as split of train, dev, and test - https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers
• Portmanteau corpus of 1600 items - https://github.com/vgtomahawk/Charmanteau-CamReady
• NLP with human traits corpus - https://osf.io/bm9gn/
• Categorized Entity Linking Corpus - https://github.com/henryrosalesmendez/categorized_EMNLP_datasets - Outputs of various EL systems on the datasets
• Bio2RDF: Linked data which can be used for Entity Linking - https://download.bio2rdf.org/files/release/3/release.html - https://download.bio2rdf.org/#/current/
• Clickbait headlines dataset - https://github.com/bhargaviparanjape/clickbait/tree/master/dataset
• Named Entities based on gaze prediction - https://github.com/DS3Lab/ner-at-first-sight/tree/master/data
• Nerwip Corpus (Manually annotated 408 Wikipedia biographies for Named Entities) - https://doi.org/10.6084/m9.figshare.1289791.v17
• Serial Speakers: a Dataset of TV Series - https://figshare.com/articles/TV_Series_Corpus/3471839
• Wikipedia Abusive Conversations - https://figshare.com/articles/Wikipedia_Abusive_Conversations/11299118
• Extracting Semantic Network Data from Newspaper Articles - https://doi.org/10.7910/DVN/GBUL0K
• ChroniclItaly 2.0. A corpus of Italian American newspapers annotated for entities, 1898-1920 - https://doi.org/10.24416/uu01-4mecro
• English/Turkish Wikipedia Named-Entity Recognition and Text Categorization Dataset - https://data.mendeley.com/datasets/cdcztymf4k/1
• Various German NLP datasets - https://www.inf.uni-hamburg.de/en/inst/ab/lt/resources/data.html
• Hierarchical Patent Classification - https://dublin.zhaw.ch/~benf/HPC/
• LSHTC: A Benchmark for Large-Scale Text Classification - http://lshtc.iit.demokritos.gr/
• GSCL Shared Task: Automatic Linguistic Annotation of Computer-Mediated Communication / Social Media - https://sites.google.com/site/empirist2015/home
• EU News Summary Dataset for 2006-2013 - http://lexhub.org/data_sets/14
• NER on historical newspapers - https://github.com/impresso/CLEF-HIPE-2020/tree/master/data
• Salient information from news articles and tweets - http://data.crowdtruth.org/salience-news-tweets/
• The Upworthy Research Archive dataset of headline A/B tests conducted by Upworthy from early 2013 into April 2015 - https://upworthy.natematias.com/about-the-archive
• Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts - https://github.com/NUSTM/ECPE More unified version
• Aspect Based Sentiment Analysis (Twitter, Product Reviews) - https://github.com/NUSTM/ABSC/tree/master/data/absa
• Fewshot relation extraction corpus - https://thunlp.github.io/1/fewrel1.html
• SemEval 2010 Task 8 - Multi-Way Classification of Semantic Relations Between Pairs of Nominals - https://drive.google.com/file/d/0B_jQiLugGTAkMDQ5ZjZiMTUtMzQ1Yy00YWNmLWJlZDYtOWY1ZDMwY2U4YjFk/view?sort=name&layout=list&num=50
• Wordnet Annotated Corpora - http://globalwordnet.org/resources/wordnet-annotated-corpora/
• (RC)^2 dataset for Review Conversational Reading Comprehension (RCRC) - data repo
• Review Reading Comprehension (RRC) - data repo
• Complementary Entity Recognition (CER) - QA large PCQA Reviews
• A Multilingual Multi-Target Dataset for Stance Detection - https://github.com/ZurichNLP/xstance
• Argument Mining manual annotation of judgments of the European Court on Human Rights - https://github.com/PLN-FaMAF/ArgumentMiningECHR
• SemCor and Masc documents annotated with NOAD (New Oxford American Dictionary) word senses - https://github.com/google-research-datasets/word_sense_disambigation_corpora
• Polish NLP datasets (NER, IE, WSD) - http://poleval.pl/tasks/
• PoKi: A Large Dataset of Poems by Children (divided by grade and gender inferred from name) - https://github.com/whipson/PoKi-Poems-by-Kids
• Privacy policies of US companies judged by legal experts - https://github.com/ansgarw/privacy
• OpenSubtitles - subtitles in various languages - http://opus.nlpl.eu/OpenSubtitles-v2018.php
• OpenParallel corpus - http://opus.nlpl.eu/index.php
• A Dataset of Petitions from Avaaz.org - https://dataverse.mpi-sws.org/dataset.xhtml?persistentId=doi:10.5072/FK2/CUSKCS&version=1.0
• NLP Datasets on Indian Languages - https://github.com/indicnlpweb/indicnlp_catalog
• A Survey and Experiments on Annotated Corpora for Emotion Classification in Text - https://github.com/sarnthil/unify-emotion-datasets/tree/master/datasets
• Github Typo Corpus (A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors) - https://github.com/mhagiwara/github-typo-corpus
• Sentence Alignment in Text Simplification (Wiki and Newsela) - https://github.com/chaojiang06/wiki-auto
• Media Frame Corpus - https://github.com/dallascard/media_frames_corpus
• DAWT: Densely Annotated Wikipedia Texts across multiple languages - https://github.com/klout/opendata/blob/master/wiki_annotation/README.md
• News corpus of Police Killings which are sentence segmented, mention-level, distantly labeled data used in experiments. - http://slanglab.cs.umass.edu/PoliceKillingsExtraction/
• Biographical Structure in Text (Wikipedia event data, inferred gender and date of birth) - http://www.cs.cmu.edu/~ark/bio/
• NER for South and South East Asian Languages (Hindi, Bengali, Oriya, Telugu, Urdu) - http://ltrc.iiit.ac.in/ner-ssea-08/index.cgi?topic=5
• Summarization datasets - https://github.com/recitalAI/summarizing_summarization
• Same Side Stance Classification - https://events.webis.de/sameside-19/#task
• Information Extraction from Chemical Patents - https://chemu-patent-ie.github.io/
• Open Advancement of Question Answering Systems - https://oaqa.github.io/
• Medical term similarity datasets based on SNOMED-CT. - https://github.com/babylonhealth/medisim
• Large-Scale Multi-Label Text Classification on EU Legislation - EURLEX57K - http://nlp.cs.aueb.gr/software_and_datasets/EURLEX57K/index.html Paper
• O*NET® 25.0 Database - Jobtitle, Job Description to Job Codes - https://www.onetcenter.org/database.html#individual-files https://github.com/afshinrahimi/jobdescription2jobtitle
• WikiUMLS: Aligning UMLS to Wikipedia - https://github.com/afshinrahimi/wikiumls
• Microsoft Research Paraphrase Corpus - https://www.microsoft.com/en-us/download/details.aspx?id=52398
• Microsoft Research Paraphrase Phrase Tables - https://www.microsoft.com/en-us/download/details.aspx?id=52536
• SimplePPDB++ paraphrases with readability scores - https://github.com/mounicam/lexical_simplification/tree/master/SimplePPDBpp
• Word Complexity Lexicon - https://github.com/mounicam/lexical_simplification/tree/master/word_complexity_lexicon
• Multilingual paraphrase corpus - http://paraphrase.org/
• chakki's Aspect-Based Sentiment Analysis dataset - https://github.com/chakki-works/chABSA-dataset
• Sentiment Analysis in Russian - https://github.com/sismetanin/sentiment-analysis-in-russian
• Similarity and relatedness dataset for Wikipedia entities (WikiSRS) - https://slate.cse.ohio-state.edu/WikiSRS/
• Argumentation corpora - http://corpora.aifdb.org/
• A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos - https://github.com/frankxu2004/cooking-procedural-extraction
• End to End Entity Linking datasets (WebQSPEL and GraphQEL)- http://dl.fbaipublicfiles.com/elq/EL4QA_data.tar.gz Blink paper
• Entity Typing Dataset and WikilinksNED Unseen-Mentions - https://github.com/yasumasaonoe/ET4EL
• Zero Shot Entity Linking dataset using Wikia - https://github.com/lajanugen/zeshel
• Norwegian NER - https://github.com/ljos/navnkjenner
• Case Law Project (US Cases Open Text as well as Case Citation Networks) - https://case.law/download/
• Multilingual LibriSpeech (MLS) - A large multilingual corpus derived from LibriVox audiobooks - http://www.openslr.org/94/
• Event prediction from WikiHow - https://github.com/daiquocnguyen/EventPrediction
• IBM Debating data - https://www.research.ibm.com/haifa/dept/vst/debating_data.shtml
• T-Rex : A Large Scale Alignment of Natural Language with Knowledge Base Triples - https://github.com/hadyelsahar/RE-NLG-Dataset
• Hindi Translated datasets - http://www.cfilt.iitb.ac.in/iitb_parallel/
• Ontonotes CoNLL format data - https://github.com/yuchenlin/OntoNotes-5.0-NER-BIO
• Japanese NLP datasets for multiple tasks - http://nlp.ist.i.kyoto-u.ac.jp/EN/?NLPresources#d40a0717
• Japanese parallel text data - http://phontron.com/japanese-translation-data.php?lang=en
• Ted transcripts - https://wit3.fbk.eu/
• WikiConv - Wikipedia Talk Page conversations in 5 languages - https://github.com/conversationai/wikidetox/tree/master/wikiconv
• NER on historic newspaper text - https://github.com/dbmdz/historic-ner
• Persuasion Techniques Annotation - https://propaganda.math.unipd.it/index.html
• Temporal Privacy Policy dataset - https://github.com/citp/privacy-policy-historical
• Groningen Meaning Bank (GMB) Publc Domain English NER + other tags - https://gmb.let.rug.nl/
• Parallel Meaning Bank (PMB) - https://pmb.let.rug.nl/
• Finnish NLP corpora - https://www.kielipankki.fi/corpora/
• Finnish English Emotion Annotation Movie dialogues from OPUS - https://github.com/Helsinki-NLP/XED
• Event coref bank data - http://www.newsreader-project.eu/results/data/the-ecb-corpus/
• Newsreader project data (Wikinews) - http://www.newsreader-project.eu/results/data/
• NER transliteration dataset in multiple languages - http://workshop.colips.org/news2018/dataset.html
• Various Paraphrase corpus - https://github.com/wasiahmad/paraphrase_identification/tree/master/dataset
• Keyphrase Generation datasets - https://github.com/wasiahmad/NeuralKpGen/blob/master/data/README.md
• Corpus of Russian documents (for LM training) - https://ruscorpora.ru/new/en/index.html
• Dakshina corpus of South Asian languages in Latin Script - https://github.com/google-research-datasets/dakshina
• Economic Sentiment dataset - https://github.com/vanatteveldt/ecosent
• Entity Linking datasets - https://github.com/dice-group/gerbil/wiki/Licences-for-datasets
• Multilingual Political Scaling - https://bitbucket.org/gg42554/cl-scaling/src/master/
• Entity Aspect Linking - https://federiconanni.com/eal-d/
• TREC Complex Answer Retrieval - http://trec-car.cs.unh.edu/
• Multilingual Text Similarity - https://bitbucket.org/gg42554/cl-sts/src/master/data/
• TREC News (Wapost related doc and wikification) - http://trec-news.org/
• CrossNER - Multi Domain NER data - https://github.com/zliucr/CrossNER
• Geoparsing datasets - https://github.com/milangritta/Pragmatic-Guide-to-Geoparsing-Evaluation
• ParCOR - Parallel EN-DE coreference corpus - https://github.com/chardmeier/parcor-full
• DWIE (Deutsche Welle corpus for Information Extraction) for document-level multi-task Information Extraction (IE) (NER, NED, Coref, RelEx) - https://github.com/klimzaporojets/DWIE/
• ParSent - paragraph level entity centric sentiment - https://stonybrooknlp.github.io/PerSenT/
• Targeted Sentiment Analysis - https://www.research.ibm.com/haifa/dept/vst/debating_data.shtml#Targeted%20Sentiment%20Analysis
• African Languages NER - https://github.com/masakhane-io/masakhane-ner
• Politcal Ads - http://lig-membres.imag.fr/gogao/www21.html
• Propublica politcal ads - https://www.propublica.org/datastore/dataset/political-advertisements-from-facebook
• Speech NER - https://github.com/mdredze/speech_ner_entity_linking_data
• Russian Corpora 20+ datasets - https://github.com/natasha/corus#reference
• WNED Corpora as reported in Paper footnote 7 - https://www.dropbox.com/s/987hmjdoq0cql9z/WNED.tar.gz
• Deep - ED Entity Disambiguation Dataset - https://github.com/dalab/deep-ed - https://drive.google.com/uc?id=0Bx8d3azIm_ZcbHMtVmRVc1o5TWM&export=download
• Large scale dataset of multi-lingual aligned NER annotations from common crawl - http://data.statmt.org/xlent/
• ViralTexts project - identify why old newspaper text went viral - https://viraltexts.org/
• Multimodal Knowledge Graph Completion - https://public.ukp.informatik.tu-darmstadt.de/starsem18-multimodalKB/ Code
• Entity Linking on Question Answering Data - https://github.com/UKPLab/starsem2018-entity-linking
• WebQuestions Semantic Parses Dataset - https://www.microsoft.com/en-us/download/details.aspx?id=52763 Code
• WebQuestions Full dataset - https://nlp.stanford.edu/software/sempre/
• Question Answering over Linked Data - https://project-hobbit.eu/challenges/qald2017/
• Romanian language datasets - https://github.com/eemlcommunity/ro_benchmark_leaderboard
• Entity Linking datasets - https://github.com/kermitt2/entity-fishing/tree/master/data
• GDELT webngrams - https://blog.gdeltproject.org/announcing-the-web-news-ngram-datasets-web-ngram/
• GDELT new similarity graph - https://blog.gdeltproject.org/announcing-the-global-similarity-graph-television-news-sentence-embeddings-using-the-universal-sentence-encoder/
• Web Data Commons - Schema.org Table Corpus - http://webdatacommons.org/structureddata/schemaorgtables/
• QuoteBank A corpus of quotations from a decade of news - https://zenodo.org/record/4277311
• Indian Court Judgements annotated with Gender - https://www.devdatalab.org/judicial-data
• Sentiment in Firm Risk Reports - https://www.firmlevelrisk.com/home
• Terms of services tracked over time from various websites - https://github.com/ambanum/OpenTermsArchive-versions
• News Haiku Dataset - https://www.kaggle.com/newshaikus/dataset/version/3
• India Police Events about Gujrat 2002 riots - https://github.com/slanglab/IndiaPoliceEvents
• Wikipedia Entity Linking Editor Reccomendations - Code Datasets Paper
• WikiCheck: Wikipedia based Fact Checking - https://github.com/trokhymovych/WikiCheck Paper
• AIDA Entity Linking (Mapped to Wikidata) - https://github.com/aaaton/herd/tree/master/src/main/resources/evaluation
• CMU Movie Summary Corpus - https://github.com/ucinlp/GenderQuant/tree/master/data/raw/summaries
• MS Marco Keyphrase Extraction - https://microsoft.github.io/msmarco/
• Keyphrase Extraction datasets - https://github.com/memray/OpenNMT-kpg-release
• JTubeSpeech: Corpus of speech collected from YouTube - https://github.com/sarulab-speech/jtubespeech
• WikiNews - Annotated at multiple levels - http://www.newsreader-project.eu/results/data/wikinews/
• Linked Hypernym dataset attaches entity articles in English, German and Dutch Wikipedia linked to DBPedia - https://ner.vse.cz/datasets/linkedhypernyms/
• Entity Linking in Queries Resources - http://hasibi.com/resources/
• WNUT 2020 NER on wet lab protocol data - https://github.com/jeniyat/WNUT_2020_NER
• WNUT 2020 Relation Extraction on wet lab protocol data - https://github.com/jeniyat/WNUT_2020_RE
• Stack Overflow NER - https://github.com/jeniyat/StackOverflowNER
• Kensho derived wikimedia data [Wikipedia + Wikidata] - https://www.kaggle.com/kenshoresearch/kensho-derived-wikimedia-data
• Abstract Meaning Representation Corpus - https://github.com/IBM/transition-amr-parser/tree/master
• NER on Material Science Papers - https://github.com/olivettigroup/annotated-materials-syntheses

Shared tasks, competitions

• IN PURSUIT OF HAPPINESS - https://sites.google.com/view/affcon2019/cl-aff-shared-task
• NIST TAC Knowledge Base Population (KBP2019) Entity Discovery and Linking Track - http://nlp.cs.rpi.edu/kbp/2019/index.html
• Multilingual Surface Realization Shared Task (SR'19) - http://taln.upf.edu/pages/msr2019-ws/SRST.html#data
• SHARED TASK ON FINE-GRAINED PROPAGANDA DETECTION @NLP4IF 2019 - https://propaganda.qcri.org/nlp4if-shared-task/
• PolEval - http://poleval.pl/tasks/
• GERMEVAL - German NLP shared tasks - https://projects.fzai.h-da.de/iggsa/germeval/
• Hobbit: Holistic Benchmarking of Big Linked Data - https://project-hobbit.eu/
• CLEF Shared tasks - http://clef2021.clef-initiative.eu/index.php?page=Pages/labs.html
• SemEval shared tasks - https://semeval.github.io/
• TREC shared tasks - https://trec.nist.gov/pubs/call2021.html

Research Groups

• https://supernlp.github.io/ - University of Copenhagen
• http://slanglab.cs.umass.edu/ - Statistical Social Language Analysis - Brendan O'Connor
• https://nlp.stanford.edu/ - Stanford
• http://nlp.seas.harvard.edu/ - Harvard
• https://mr.cs.ucl.ac.uk/ - University College London
• http://nlpg.itk.ppke.hu/projects - Pázmány Péter Catholic University
• http://www.ark.cs.washington.edu/ - Noah Smith NLP Lab at University of Washington
• https://www.cs.washington.edu/research/nlp - University of Washington
• http://wiki.clsp.jhu.edu/view/NLP_Reading_Group - John Hopkins University
• https://github.com/thunlp - Natural Language Processing Lab at Tsinghua University
• https://github.com/OSU-slatelab - OSU Speech and Language Technologies Laboratory

Mailing lists

• https://mailman.uib.no/public/corpora/ - Corpora mailing list