Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models

Name: Work Video:
Duration: 3 min 30 s
Description: Tyler Chang, Yifan Xu, Weijian Xu, Zhuowen Tu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Tyler A. Chang; Yifan Xu; Weijian Xu; Zhuowen Tu
{'id': 'https://openalex.org/W3176887068', 'doi': 'https://doi.org/10.18653/v1/2021.acl-long.333', 'title': 'Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models', 'display_name': 'Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models', 'publication_year': 2021, 'publication_date': '2021-01-01', 'ids': {'openalex': 'https://openalex.org/W3176887068', 'doi': 'https://doi.org/10.18653/v1/2021.acl-long.333', 'mag': '3176887068'}, 'language': 'en', 'primary_location': {'is_oa': True, 'landing_page_url': 'https://doi.org/10.18653/v1/2021.acl-long.333', 'pdf_url': 'https://aclanthology.org/2021.acl-long.333.pdf', 'source': None, 'license': 'cc-by', 'license_id': 'https://openalex.org/licenses/cc-by', 'version': 'publishedVersion', 'is_accepted': True, 'is_published': True}, 'type': 'article', 'type_crossref': 'proceedings-article', 'indexed_in': ['crossref'], 'open_access': {'is_oa': True, 'oa_status': 'hybrid', 'oa_url': 'https://aclanthology.org/2021.acl-long.333.pdf', 'any_repository_has_fulltext': True}, 'authorships': [{'author_position': 'first', 'author': {'id': 'https://openalex.org/A5046456944', 'display_name': 'Tyler A. Chang', 'orcid': None}, 'institutions': [{'id': 'https://openalex.org/I36258959', 'display_name': 'University of California, San Diego', 'ror': 'https://ror.org/0168r3w48', 'country_code': 'US', 'type': 'education', 'lineage': ['https://openalex.org/I36258959']}], 'countries': ['US'], 'is_corresponding': True, 'raw_author_name': 'Tyler Chang', 'raw_affiliation_strings': ['Halıcıoglu Data Science Institute', 'University of California San Diego'], 'affiliations': [{'raw_affiliation_string': 'University of California San Diego', 'institution_ids': ['https://openalex.org/I36258959']}, {'raw_affiliation_string': 'Halıcıoglu Data Science Institute', 'institution_ids': []}]}, {'author_position': 'middle', 'author': {'id': 'https://openalex.org/A5067206559', 'display_name': 'Yifan Xu', 'orcid': 'https://orcid.org/0000-0003-1591-0384'}, 'institutions': [{'id': 'https://openalex.org/I36258959', 'display_name': 'University of California, San Diego', 'ror': 'https://ror.org/0168r3w48', 'country_code': 'US', 'type': 'education', 'lineage': ['https://openalex.org/I36258959']}], 'countries': ['US'], 'is_corresponding': False, 'raw_author_name': 'Yifan Xu', 'raw_affiliation_strings': ['University of California San Diego'], 'affiliations': [{'raw_affiliation_string': 'University of California San Diego', 'institution_ids': ['https://openalex.org/I36258959']}]}, {'author_position': 'middle', 'author': {'id': 'https://openalex.org/A5102717169', 'display_name': 'Weijian Xu', 'orcid': 'https://orcid.org/0009-0009-5306-4247'}, 'institutions': [{'id': 'https://openalex.org/I36258959', 'display_name': 'University of California, San Diego', 'ror': 'https://ror.org/0168r3w48', 'country_code': 'US', 'type': 'education', 'lineage': ['https://openalex.org/I36258959']}], 'countries': ['US'], 'is_corresponding': False, 'raw_author_name': 'Weijian Xu', 'raw_affiliation_strings': ['University of California San Diego'], 'affiliations': [{'raw_affiliation_string': 'University of California San Diego', 'institution_ids': ['https://openalex.org/I36258959']}]}, {'author_position': 'last', 'author': {'id': 'https://openalex.org/A5001760915', 'display_name': 'Zhuowen Tu', 'orcid': 'https://orcid.org/0000-0002-1900-2124'}, 'institutions': [{'id': 'https://openalex.org/I36258959', 'display_name': 'University of California, San Diego', 'ror': 'https://ror.org/0168r3w48', 'country_code': 'US', 'type': 'education', 'lineage': ['https://openalex.org/I36258959']}], 'countries': ['US'], 'is_corresponding': False, 'raw_author_name': 'Zhuowen Tu', 'raw_affiliation_strings': ['University of California San Diego'], 'affiliations': [{'raw_affiliation_string': 'University of California San Diego', 'institution_ids': ['https://openalex.org/I36258959']}]}], 'institution_assertions': [], 'countries_distinct_count': 1, 'institutions_distinct_count': 1, 'corresponding_author_ids': ['https://openalex.org/A5046456944'], 'corresponding_institution_ids': ['https://openalex.org/I36258959'], 'apc_list': None, 'apc_paid': None, 'fwci': 0.814, 'has_fulltext': True, 'fulltext_origin': 'pdf', 'cited_by_count': 8, 'citation_normalized_percentile': {'value': 0.542866, 'is_in_top_1_percent': False, 'is_in_top_10_percent': False}, 'cited_by_percentile_year': {'min': 84, 'max': 86}, 'biblio': {'volume': None, 'issue': None, 'first_page': '4322', 'last_page': '4333'}, 'is_retracted': False, 'is_paratext': False, 'primary_topic': {'id': 'https://openalex.org/T10028', 'display_name': 'Topic Modeling', 'score': 0.9997, 'subfield': {'id': 'https://openalex.org/subfields/1702', 'display_name': 'Artificial Intelligence'}, 'field': {'id': 'https://openalex.org/fields/17', 'display_name': 'Computer Science'}, 'domain': {'id': 'https://openalex.org/domains/3', 'display_name': 'Physical Sciences'}}, 'topics': [{'id': 'https://openalex.org/T10028', 'display_name': 'Topic Modeling', 'score': 0.9997, 'subfield': {'id': 'https://openalex.org/subfields/1702', 'display_name': 'Artificial Intelligence'}, 'field': {'id': 'https://openalex.org/fields/17', 'display_name': 'Computer Science'}, 'domain': {'id': 'https://openalex.org/domains/3', 'display_name': 'Physical Sciences'}}, {'id': 'https://openalex.org/T10181', 'display_name': 'Natural Language Processing Techniques', 'score': 0.9996, 'subfield': {'id': 'https://openalex.org/subfields/1702', 'display_name': 'Artificial Intelligence'}, 'field': {'id': 'https://openalex.org/fields/17', 'display_name': 'Computer Science'}, 'domain': {'id': 'https://openalex.org/domains/3', 'display_name': 'Physical Sciences'}}, {'id': 'https://openalex.org/T11714', 'display_name': 'Multimodal Machine Learning Applications', 'score': 0.989, 'subfield': {'id': 'https://openalex.org/subfields/1707', 'display_name': 'Computer Vision and Pattern Recognition'}, 'field': {'id': 'https://openalex.org/fields/17', 'display_name': 'Computer Science'}, 'domain': {'id': 'https://openalex.org/domains/3', 'display_name': 'Physical Sciences'}}], 'keywords': [{'id': 'https://openalex.org/keywords/computational-linguistics', 'display_name': 'Computational linguistics', 'score': 0.53553873}], 'concepts': [{'id': 'https://openalex.org/C41008148', 'wikidata': 'https://www.wikidata.org/wiki/Q21198', 'display_name': 'Computer science', 'level': 0, 'score': 0.61911196}, {'id': 'https://openalex.org/C155092808', 'wikidata': 'https://www.wikidata.org/wiki/Q182557', 'display_name': 'Computational linguistics', 'level': 2, 'score': 0.53553873}, {'id': 'https://openalex.org/C204321447', 'wikidata': 'https://www.wikidata.org/wiki/Q30642', 'display_name': 'Natural language processing', 'level': 1, 'score': 0.5303408}, {'id': 'https://openalex.org/C20556612', 'wikidata': 'https://www.wikidata.org/wiki/Q4469374', 'display_name': 'Volume (thermodynamics)', 'level': 2, 'score': 0.5121236}, {'id': 'https://openalex.org/C41895202', 'wikidata': 'https://www.wikidata.org/wiki/Q8162', 'display_name': 'Linguistics', 'level': 1, 'score': 0.49027398}, {'id': 'https://openalex.org/C154945302', 'wikidata': 'https://www.wikidata.org/wiki/Q11660', 'display_name': 'Artificial intelligence', 'level': 1, 'score': 0.45032704}, {'id': 'https://openalex.org/C188147891', 'wikidata': 'https://www.wikidata.org/wiki/Q147638', 'display_name': 'Cognitive science', 'level': 1, 'score': 0.44680467}, {'id': 'https://openalex.org/C18555067', 'wikidata': 'https://www.wikidata.org/wiki/Q8375051', 'display_name': 'Joint (building)', 'level': 2, 'score': 0.4398155}, {'id': 'https://openalex.org/C15744967', 'wikidata': 'https://www.wikidata.org/wiki/Q9418', 'display_name': 'Psychology', 'level': 0, 'score': 0.22273663}, {'id': 'https://openalex.org/C138885662', 'wikidata': 'https://www.wikidata.org/wiki/Q5891', 'display_name': 'Philosophy', 'level': 0, 'score': 0.16984203}, {'id': 'https://openalex.org/C127413603', 'wikidata': 'https://www.wikidata.org/wiki/Q11023', 'display_name': 'Engineering', 'level': 0, 'score': 0.12409663}, {'id': 'https://openalex.org/C121332964', 'wikidata': 'https://www.wikidata.org/wiki/Q413', 'display_name': 'Physics', 'level': 0, 'score': 0.06729397}, {'id': 'https://openalex.org/C62520636', 'wikidata': 'https://www.wikidata.org/wiki/Q944', 'display_name': 'Quantum mechanics', 'level': 1, 'score': 0.0}, {'id': 'https://openalex.org/C170154142', 'wikidata': 'https://www.wikidata.org/wiki/Q150737', 'display_name': 'Architectural engineering', 'level': 1, 'score': 0.0}], 'mesh': [], 'locations_count': 2, 'locations': [{'is_oa': True, 'landing_page_url': 'https://doi.org/10.18653/v1/2021.acl-long.333', 'pdf_url': 'https://aclanthology.org/2021.acl-long.333.pdf', 'source': None, 'license': 'cc-by', 'license_id': 'https://openalex.org/licenses/cc-by', 'version': 'publishedVersion', 'is_accepted': True, 'is_published': True}, {'is_oa': True, 'landing_page_url': 'https://arxiv.org/abs/2106.05505', 'pdf_url': 'https://arxiv.org/pdf/2106.05505', 'source': {'id': 'https://openalex.org/S4306400194', 'display_name': 'arXiv (Cornell University)', 'issn_l': None, 'issn': None, 'is_oa': True, 'is_in_doaj': False, 'is_core': False, 'host_organization': 'https://openalex.org/I205783295', 'host_organization_name': 'Cornell University', 'host_organization_lineage': ['https://openalex.org/I205783295'], 'host_organization_lineage_names': ['Cornell University'], 'type': 'repository'}, 'license': None, 'license_id': None, 'version': 'submittedVersion', 'is_accepted': False, 'is_published': False}], 'best_oa_location': {'is_oa': True, 'landing_page_url': 'https://doi.org/10.18653/v1/2021.acl-long.333', 'pdf_url': 'https://aclanthology.org/2021.acl-long.333.pdf', 'source': None, 'license': 'cc-by', 'license_id': 'https://openalex.org/licenses/cc-by', 'version': 'publishedVersion', 'is_accepted': True, 'is_published': True}, 'sustainable_development_goals': [{'score': 0.84, 'id': 'https://metadata.un.org/sdg/4', 'display_name': 'Quality education'}], 'grants': [], 'datasets': [], 'versions': [], 'referenced_works_count': 37, 'referenced_works': ['https://openalex.org/W1566289585', 'https://openalex.org/W2892009249', 'https://openalex.org/W2898700502', 'https://openalex.org/W2908336025', 'https://openalex.org/W2923014074', 'https://openalex.org/W2946794439', 'https://openalex.org/W2952809536', 'https://openalex.org/W2963250244', 'https://openalex.org/W2963310665', 'https://openalex.org/W2963341956', 'https://openalex.org/W2963403868', 'https://openalex.org/W2963494889', 'https://openalex.org/W2963925437', 'https://openalex.org/W2964110616', 'https://openalex.org/W2970597249', 'https://openalex.org/W2978670439', 'https://openalex.org/W2979826702', 'https://openalex.org/W2980433389', 'https://openalex.org/W2981413347', 'https://openalex.org/W2994759459', 'https://openalex.org/W3044284384', 'https://openalex.org/W3047171714', 'https://openalex.org/W3082274269', 'https://openalex.org/W3083429640', 'https://openalex.org/W3094502228', 'https://openalex.org/W3098824823', 'https://openalex.org/W3101278968', 'https://openalex.org/W3106210592', 'https://openalex.org/W3106504817', 'https://openalex.org/W3122515622', 'https://openalex.org/W4287824654', 'https://openalex.org/W4292779060', 'https://openalex.org/W4298422451', 'https://openalex.org/W4299802238', 'https://openalex.org/W4322614701', 'https://openalex.org/W4385245566', 'https://openalex.org/W4394668313'], 'related_works': ['https://openalex.org/W435830328', 'https://openalex.org/W4242592912', 'https://openalex.org/W2888483922', 'https://openalex.org/W2798121181', 'https://openalex.org/W2748574964', 'https://openalex.org/W2728221973', 'https://openalex.org/W2328676785', 'https://openalex.org/W2087896742', 'https://openalex.org/W2016805743', 'https://openalex.org/W1996130883'], 'abstract_inverted_index': {'Tyler': [0], 'Chang,': [1], 'Yifan': [2], 'Xu,': [3, 5], 'Weijian': [4], 'Zhuowen': [6], 'Tu.': [7], 'Proceedings': [8], 'of': [9, 14], 'the': [10, 15, 21], '59th': [11], 'Annual': [12], 'Meeting': [13], 'Association': [16], 'for': [17], 'Computational': [18], 'Linguistics': [19], 'and': [20], '11th': [22], 'International': [23], 'Joint': [24], 'Conference': [25], 'on': [26], 'Natural': [27], 'Language': [28], 'Processing': [29], '(Volume': [30], '1:': [31], 'Long': [32], 'Papers).': [33], '2021.': [34]}, 'cited_by_api_url': 'https://api.openalex.org/works?filter=cites:W3176887068', 'counts_by_year': [{'year': 2023, 'cited_by_count': 4}, {'year': 2022, 'cited_by_count': 4}], 'updated_date': '2024-12-11T03:11:47.404888', 'created_date': '2021-07-05'}
Publication Information

Basic Information

Access and Citation

AI Researcher Chatbot

Primary Location

Authors

Topics

Keywords

Related Works
