third_party.pylibs.pylint.src/pylint/testutils/tokenize_str.py

# Licensed under the GPL: https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
# For details: https://github.com/PyCQA/pylint/blob/main/LICENSE

import tokenize
from io import StringIO
from tokenize import TokenInfo
from typing import List


def _tokenize_str(code: str) -> List[TokenInfo]:
    return list(tokenize.generate_tokens(StringIO(code).readline))
Burst what remain of utils into files 2020-11-29 10:17:56 +00:00			`# Licensed under the GPL: https://www.gnu.org/licenses/old-licenses/gpl-2.0.html`
Fix copyright links (#4647) * Fix link in license header * Update link to astroid bump_changelog 2021-07-01 10:47:58 +00:00			`# For details: https://github.com/PyCQA/pylint/blob/main/LICENSE`
Burst what remain of utils into files 2020-11-29 10:17:56 +00:00
			`import tokenize`
			`from io import StringIO`
Add small bits of typing annotation (#4954) 2021-09-25 04:23:32 +00:00			`from tokenize import TokenInfo`
			`from typing import List`
Burst what remain of utils into files 2020-11-29 10:17:56 +00:00

Add small bits of typing annotation (#4954) 2021-09-25 04:23:32 +00:00			`def _tokenize_str(code: str) -> List[TokenInfo]:`
Burst what remain of utils into files 2020-11-29 10:17:56 +00:00			`return list(tokenize.generate_tokens(StringIO(code).readline))`