AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models

Name: Work Video:
Duration: 3 min 30 s
Description: Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Yichun Yin; Cheng Chen; Lifeng Shang; Xin Jiang; Xiao Chen; Qun Liu
{'id': 'https://openalex.org/W3174730533', 'doi': 'https://doi.org/10.18653/v1/2021.acl-long.400', 'title': 'AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models', 'display_name': 'AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models', 'publication_year': 2021, 'publication_date': '2021-01-01', 'ids': {'openalex': 'https://openalex.org/W3174730533', 'doi': 'https://doi.org/10.18653/v1/2021.acl-long.400', 'mag': '3174730533'}, 'language': 'en', 'primary_location': {'is_oa': True, 'landing_page_url': 'https://doi.org/10.18653/v1/2021.acl-long.400', 'pdf_url': 'https://aclanthology.org/2021.acl-long.400.pdf', 'source': None, 'license': 'cc-by', 'license_id': 'https://openalex.org/licenses/cc-by', 'version': 'publishedVersion', 'is_accepted': True, 'is_published': True}, 'type': 'article', 'type_crossref': 'proceedings-article', 'indexed_in': ['crossref'], 'open_access': {'is_oa': True, 'oa_status': 'hybrid', 'oa_url': 'https://aclanthology.org/2021.acl-long.400.pdf', 'any_repository_has_fulltext': True}, 'authorships': [{'author_position': 'first', 'author': {'id': 'https://openalex.org/A5043780805', 'display_name': 'Yichun Yin', 'orcid': None}, 'institutions': [{'id': 'https://openalex.org/I4210116209', 'display_name': 'The Ark', 'ror': 'https://ror.org/02dnv4714', 'country_code': 'IE', 'type': 'archive', 'lineage': ['https://openalex.org/I4210116209']}], 'countries': ['IE'], 'is_corresponding': False, 'raw_author_name': 'Yichun Yin', 'raw_affiliation_strings': [' Ark Lab'], 'affiliations': [{'raw_affiliation_string': ' Ark Lab', 'institution_ids': ['https://openalex.org/I4210116209']}]}, {'author_position': 'middle', 'author': {'id': 'https://openalex.org/A5100420600', 'display_name': 'Cheng Chen', 'orcid': 'https://orcid.org/0000-0003-3662-0263'}, 'institutions': [{'id': 'https://openalex.org/I99065089', 'display_name': 'Tsinghua University', 'ror': 'https://ror.org/03cve4549', 'country_code': 'CN', 'type': 'education', 'lineage': ['https://openalex.org/I99065089']}], 'countries': ['CN'], 'is_corresponding': False, 'raw_author_name': 'Cheng Chen', 'raw_affiliation_strings': ['Department of Computer Science and Technology, Tsinghua University'], 'affiliations': [{'raw_affiliation_string': 'Department of Computer Science and Technology, Tsinghua University', 'institution_ids': ['https://openalex.org/I99065089']}]}, {'author_position': 'middle', 'author': {'id': 'https://openalex.org/A5046228314', 'display_name': 'Lifeng Shang', 'orcid': None}, 'institutions': [{'id': 'https://openalex.org/I4210116209', 'display_name': 'The Ark', 'ror': 'https://ror.org/02dnv4714', 'country_code': 'IE', 'type': 'archive', 'lineage': ['https://openalex.org/I4210116209']}], 'countries': ['IE'], 'is_corresponding': False, 'raw_author_name': 'Lifeng Shang', 'raw_affiliation_strings': [' Ark Lab'], 'affiliations': [{'raw_affiliation_string': ' Ark Lab', 'institution_ids': ['https://openalex.org/I4210116209']}]}, {'author_position': 'middle', 'author': {'id': 'https://openalex.org/A5086603207', 'display_name': 'Xin Jiang', 'orcid': 'https://orcid.org/0000-0002-9117-8247'}, 'institutions': [{'id': 'https://openalex.org/I4210116209', 'display_name': 'The Ark', 'ror': 'https://ror.org/02dnv4714', 'country_code': 'IE', 'type': 'archive', 'lineage': ['https://openalex.org/I4210116209']}], 'countries': ['IE'], 'is_corresponding': False, 'raw_author_name': 'Xin Jiang', 'raw_affiliation_strings': [' Ark Lab'], 'affiliations': [{'raw_affiliation_string': ' Ark Lab', 'institution_ids': ['https://openalex.org/I4210116209']}]}, {'author_position': 'middle', 'author': {'id': 'https://openalex.org/A5100373674', 'display_name': 'Xiao Chen', 'orcid': 'https://orcid.org/0000-0001-5561-6572'}, 'institutions': [{'id': 'https://openalex.org/I4210116209', 'display_name': 'The Ark', 'ror': 'https://ror.org/02dnv4714', 'country_code': 'IE', 'type': 'archive', 'lineage': ['https://openalex.org/I4210116209']}], 'countries': ['IE'], 'is_corresponding': False, 'raw_author_name': 'Xiao Chen', 'raw_affiliation_strings': [' Ark Lab'], 'affiliations': [{'raw_affiliation_string': ' Ark Lab', 'institution_ids': ['https://openalex.org/I4210116209']}]}, {'author_position': 'last', 'author': {'id': 'https://openalex.org/A5100426170', 'display_name': 'Qun Liu', 'orcid': 'https://orcid.org/0000-0002-7000-1792'}, 'institutions': [{'id': 'https://openalex.org/I4210116209', 'display_name': 'The Ark', 'ror': 'https://ror.org/02dnv4714', 'country_code': 'IE', 'type': 'archive', 'lineage': ['https://openalex.org/I4210116209']}], 'countries': ['IE'], 'is_corresponding': False, 'raw_author_name': 'Qun Liu', 'raw_affiliation_strings': [' Ark Lab'], 'affiliations': [{'raw_affiliation_string': ' Ark Lab', 'institution_ids': ['https://openalex.org/I4210116209']}]}], 'institution_assertions': [], 'countries_distinct_count': 2, 'institutions_distinct_count': 2, 'corresponding_author_ids': [], 'corresponding_institution_ids': [], 'apc_list': None, 'apc_paid': None, 'fwci': 2.339, 'has_fulltext': True, 'fulltext_origin': 'pdf', 'cited_by_count': 23, 'citation_normalized_percentile': {'value': 0.999914, 'is_in_top_1_percent': True, 'is_in_top_10_percent': True}, 'cited_by_percentile_year': {'min': 94, 'max': 95}, 'biblio': {'volume': None, 'issue': None, 'first_page': None, 'last_page': None}, 'is_retracted': False, 'is_paratext': False, 'primary_topic': {'id': 'https://openalex.org/T10028', 'display_name': 'Topic Modeling', 'score': 0.9996, 'subfield': {'id': 'https://openalex.org/subfields/1702', 'display_name': 'Artificial Intelligence'}, 'field': {'id': 'https://openalex.org/fields/17', 'display_name': 'Computer Science'}, 'domain': {'id': 'https://openalex.org/domains/3', 'display_name': 'Physical Sciences'}}, 'topics': [{'id': 'https://openalex.org/T10028', 'display_name': 'Topic Modeling', 'score': 0.9996, 'subfield': {'id': 'https://openalex.org/subfields/1702', 'display_name': 'Artificial Intelligence'}, 'field': {'id': 'https://openalex.org/fields/17', 'display_name': 'Computer Science'}, 'domain': {'id': 'https://openalex.org/domains/3', 'display_name': 'Physical Sciences'}}, {'id': 'https://openalex.org/T10181', 'display_name': 'Natural Language Processing Techniques', 'score': 0.9991, 'subfield': {'id': 'https://openalex.org/subfields/1702', 'display_name': 'Artificial Intelligence'}, 'field': {'id': 'https://openalex.org/fields/17', 'display_name': 'Computer Science'}, 'domain': {'id': 'https://openalex.org/domains/3', 'display_name': 'Physical Sciences'}}, {'id': 'https://openalex.org/T11714', 'display_name': 'Multimodal Machine Learning Applications', 'score': 0.9922, 'subfield': {'id': 'https://openalex.org/subfields/1707', 'display_name': 'Computer Vision and Pattern Recognition'}, 'field': {'id': 'https://openalex.org/fields/17', 'display_name': 'Computer Science'}, 'domain': {'id': 'https://openalex.org/domains/3', 'display_name': 'Physical Sciences'}}], 'keywords': [{'id': 'https://openalex.org/keywords/chen', 'display_name': 'Chen', 'score': 0.9221041}, {'id': 'https://openalex.org/keywords/computational-linguistics', 'display_name': 'Computational linguistics', 'score': 0.61039245}, {'id': 'https://openalex.org/keywords/association', 'display_name': 'Association (psychology)', 'score': 0.48621646}], 'concepts': [{'id': 'https://openalex.org/C2776085556', 'wikidata': 'https://www.wikidata.org/wiki/Q183361', 'display_name': 'Chen', 'level': 2, 'score': 0.9221041}, {'id': 'https://openalex.org/C41008148', 'wikidata': 'https://www.wikidata.org/wiki/Q21198', 'display_name': 'Computer science', 'level': 0, 'score': 0.6480851}, {'id': 'https://openalex.org/C155092808', 'wikidata': 'https://www.wikidata.org/wiki/Q182557', 'display_name': 'Computational linguistics', 'level': 2, 'score': 0.61039245}, {'id': 'https://openalex.org/C204321447', 'wikidata': 'https://www.wikidata.org/wiki/Q30642', 'display_name': 'Natural language processing', 'level': 1, 'score': 0.5539054}, {'id': 'https://openalex.org/C20556612', 'wikidata': 'https://www.wikidata.org/wiki/Q4469374', 'display_name': 'Volume (thermodynamics)', 'level': 2, 'score': 0.49562633}, {'id': 'https://openalex.org/C142853389', 'wikidata': 'https://www.wikidata.org/wiki/Q744778', 'display_name': 'Association (psychology)', 'level': 2, 'score': 0.48621646}, {'id': 'https://openalex.org/C18555067', 'wikidata': 'https://www.wikidata.org/wiki/Q8375051', 'display_name': 'Joint (building)', 'level': 2, 'score': 0.48233482}, {'id': 'https://openalex.org/C154945302', 'wikidata': 'https://www.wikidata.org/wiki/Q11660', 'display_name': 'Artificial intelligence', 'level': 1, 'score': 0.4813874}, {'id': 'https://openalex.org/C41895202', 'wikidata': 'https://www.wikidata.org/wiki/Q8162', 'display_name': 'Linguistics', 'level': 1, 'score': 0.35655868}, {'id': 'https://openalex.org/C138885662', 'wikidata': 'https://www.wikidata.org/wiki/Q5891', 'display_name': 'Philosophy', 'level': 0, 'score': 0.14410532}, {'id': 'https://openalex.org/C127413603', 'wikidata': 'https://www.wikidata.org/wiki/Q11023', 'display_name': 'Engineering', 'level': 0, 'score': 0.110358685}, {'id': 'https://openalex.org/C111472728', 'wikidata': 'https://www.wikidata.org/wiki/Q9471', 'display_name': 'Epistemology', 'level': 1, 'score': 0.060944617}, {'id': 'https://openalex.org/C121332964', 'wikidata': 'https://www.wikidata.org/wiki/Q413', 'display_name': 'Physics', 'level': 0, 'score': 0.056648284}, {'id': 'https://openalex.org/C170154142', 'wikidata': 'https://www.wikidata.org/wiki/Q150737', 'display_name': 'Architectural engineering', 'level': 1, 'score': 0.0}, {'id': 'https://openalex.org/C151730666', 'wikidata': 'https://www.wikidata.org/wiki/Q7205', 'display_name': 'Paleontology', 'level': 1, 'score': 0.0}, {'id': 'https://openalex.org/C62520636', 'wikidata': 'https://www.wikidata.org/wiki/Q944', 'display_name': 'Quantum mechanics', 'level': 1, 'score': 0.0}, {'id': 'https://openalex.org/C86803240', 'wikidata': 'https://www.wikidata.org/wiki/Q420', 'display_name': 'Biology', 'level': 0, 'score': 0.0}], 'mesh': [], 'locations_count': 2, 'locations': [{'is_oa': True, 'landing_page_url': 'https://doi.org/10.18653/v1/2021.acl-long.400', 'pdf_url': 'https://aclanthology.org/2021.acl-long.400.pdf', 'source': None, 'license': 'cc-by', 'license_id': 'https://openalex.org/licenses/cc-by', 'version': 'publishedVersion', 'is_accepted': True, 'is_published': True}, {'is_oa': True, 'landing_page_url': 'https://arxiv.org/abs/2107.13686', 'pdf_url': 'https://arxiv.org/pdf/2107.13686', 'source': {'id': 'https://openalex.org/S4306400194', 'display_name': 'arXiv (Cornell University)', 'issn_l': None, 'issn': None, 'is_oa': True, 'is_in_doaj': False, 'is_core': False, 'host_organization': 'https://openalex.org/I205783295', 'host_organization_name': 'Cornell University', 'host_organization_lineage': ['https://openalex.org/I205783295'], 'host_organization_lineage_names': ['Cornell University'], 'type': 'repository'}, 'license': None, 'license_id': None, 'version': 'submittedVersion', 'is_accepted': False, 'is_published': False}], 'best_oa_location': {'is_oa': True, 'landing_page_url': 'https://doi.org/10.18653/v1/2021.acl-long.400', 'pdf_url': 'https://aclanthology.org/2021.acl-long.400.pdf', 'source': None, 'license': 'cc-by', 'license_id': 'https://openalex.org/licenses/cc-by', 'version': 'publishedVersion', 'is_accepted': True, 'is_published': True}, 'sustainable_development_goals': [{'display_name': 'Quality education', 'id': 'https://metadata.un.org/sdg/4', 'score': 0.81}], 'grants': [], 'datasets': [], 'versions': [], 'referenced_works_count': 52, 'referenced_works': ['https://openalex.org/W1566289585', 'https://openalex.org/W1690739335', 'https://openalex.org/W1821462560', 'https://openalex.org/W2748513770', 'https://openalex.org/W2782417188', 'https://openalex.org/W2896457183', 'https://openalex.org/W2912521296', 'https://openalex.org/W2923014074', 'https://openalex.org/W2955425717', 'https://openalex.org/W2963137684', 'https://openalex.org/W2963310665', 'https://openalex.org/W2963341956', 'https://openalex.org/W2963403868', 'https://openalex.org/W2963748441', 'https://openalex.org/W2964118293', 'https://openalex.org/W2964259004', 'https://openalex.org/W2964299589', 'https://openalex.org/W2965373594', 'https://openalex.org/W2970454332', 'https://openalex.org/W2970557265', 'https://openalex.org/W2970597249', 'https://openalex.org/W2975429091', 'https://openalex.org/W2978017171', 'https://openalex.org/W2979826702', 'https://openalex.org/W2996035354', 'https://openalex.org/W2996428491', 'https://openalex.org/W2998183051', 'https://openalex.org/W3008374555', 'https://openalex.org/W3015042199', 'https://openalex.org/W3015609966', 'https://openalex.org/W3034457371', 'https://openalex.org/W3034560159', 'https://openalex.org/W3034971973', 'https://openalex.org/W3035251378', 'https://openalex.org/W3035281298', 'https://openalex.org/W3095851346', 'https://openalex.org/W3098576111', 'https://openalex.org/W3098824823', 'https://openalex.org/W3099547366', 'https://openalex.org/W3101731278', 'https://openalex.org/W3104216863', 'https://openalex.org/W3105966348', 'https://openalex.org/W3109946440', 'https://openalex.org/W3125341084', 'https://openalex.org/W3158342816', 'https://openalex.org/W3184738308', 'https://openalex.org/W3202316938', 'https://openalex.org/W4287824654', 'https://openalex.org/W4288256350', 'https://openalex.org/W4292779060', 'https://openalex.org/W4297813615', 'https://openalex.org/W4385245566'], 'related_works': ['https://openalex.org/W3176693244', 'https://openalex.org/W3174119180', 'https://openalex.org/W3169305685', 'https://openalex.org/W2789919619', 'https://openalex.org/W2728221973', 'https://openalex.org/W2351428524', 'https://openalex.org/W2351267244', 'https://openalex.org/W2293457016', 'https://openalex.org/W2030530201', 'https://openalex.org/W1551406738'], 'abstract_inverted_index': {'Yichun': [0], 'Yin,': [1], 'Cheng': [2], 'Chen,': [3, 9], 'Lifeng': [4], 'Shang,': [5], 'Xin': [6], 'Jiang,': [7], 'Xiao': [8], 'Qun': [10], 'Liu.': [11], 'Proceedings': [12], 'of': [13, 18], 'the': [14, 19, 25], '59th': [15], 'Annual': [16], 'Meeting': [17], 'Association': [20], 'for': [21], 'Computational': [22], 'Linguistics': [23], 'and': [24], '11th': [26], 'International': [27], 'Joint': [28], 'Conference': [29], 'on': [30], 'Natural': [31], 'Language': [32], 'Processing': [33], '(Volume': [34], '1:': [35], 'Long': [36], 'Papers).': [37], '2021.': [38]}, 'cited_by_api_url': 'https://api.openalex.org/works?filter=cites:W3174730533', 'counts_by_year': [{'year': 2024, 'cited_by_count': 5}, {'year': 2023, 'cited_by_count': 11}, {'year': 2022, 'cited_by_count': 5}, {'year': 2021, 'cited_by_count': 1}], 'updated_date': '2024-12-13T05:42:18.748703', 'created_date': '2021-07-05'}
Publication Information

Basic Information

Access and Citation

AI Researcher Chatbot

Primary Location

Authors

Topics

Keywords

Related Works
