mirror of
https://fuchsia.googlesource.com/third_party/pigweed.googlesource.com/pigweed/pigweed
synced 2024-09-20 13:51:05 +00:00
aa52c4b31f
If it's present on the system, run buildifier on Bazel files. If it's not present, continue to check for trailing spaces. Not getting buildifier from CIPD because the one version in CIPD doesn't have proper platform strings and it looks hard to add buildifier to CIPD the correct way. Also run `pw format --fix` on all Bazel files. Bug: 191 Change-Id: I5d33cfce6a94ad12bc812e72431f4e0e6baccfd5 Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/54380 Pigweed-Auto-Submit: Rob Mohr <mohrr@google.com> Commit-Queue: Auto-Submit <auto-submit@pigweed.google.com.iam.gserviceaccount.com> Reviewed-by: Wyatt Hepler <hepler@google.com> Reviewed-by: Keir Mierle <keir@google.com>
283 lines
9.3 KiB
ReStructuredText
283 lines
9.3 KiB
ReStructuredText
.. _module-pw_presubmit:
|
||
|
||
============
|
||
pw_presubmit
|
||
============
|
||
The presubmit module provides Python tools for running presubmit checks and
|
||
checking and fixing code format. It also includes the presubmit check script for
|
||
the Pigweed repository, ``pigweed_presubmit.py``.
|
||
|
||
Presubmit checks are essential tools, but they take work to set up, and
|
||
projects don’t always get around to it. The ``pw_presubmit`` module provides
|
||
tools for setting up high quality presubmit checks for any project. We use this
|
||
framework to run Pigweed’s presubmit on our workstations and in our automated
|
||
building tools.
|
||
|
||
The ``pw_presubmit`` module also includes ``pw format``, a tool that provides a
|
||
unified interface for automatically formatting code in a variety of languages.
|
||
With ``pw format``, you can format Bazel, C, C++, Python, GN, and Go code
|
||
according to configurations defined by your project. ``pw format`` leverages
|
||
existing tools like ``clang-format``, and it’s simple to add support for new
|
||
languages. (Note: Bazel formatting requires ``buildifier`` to be present on your
|
||
system. If it's not Bazel formatting passes without checking.)
|
||
|
||
.. image:: docs/pw_presubmit_demo.gif
|
||
:alt: ``pw format`` demo
|
||
:align: left
|
||
|
||
The ``pw_presubmit`` package includes presubmit checks that can be used with any
|
||
project. These checks include:
|
||
|
||
* Check code format of several languages including C, C++, and Python
|
||
* Initialize a Python environment
|
||
* Run all Python tests
|
||
* Run pylint
|
||
* Run mypy
|
||
* Ensure source files are included in the GN and Bazel builds
|
||
* Build and run all tests with GN
|
||
* Build and run all tests with Bazel
|
||
* Ensure all header files contain ``#pragma once``
|
||
|
||
-------------
|
||
Compatibility
|
||
-------------
|
||
Python 3
|
||
|
||
-------------------------------------------
|
||
Creating a presubmit check for your project
|
||
-------------------------------------------
|
||
Creating a presubmit check for a project using ``pw_presubmit`` is simple, but
|
||
requires some customization. Projects must define their own presubmit check
|
||
Python script that uses the ``pw_presubmit`` package.
|
||
|
||
A project's presubmit script can be registered as a
|
||
:ref:`pw_cli <module-pw_cli>` plugin, so that it can be run as ``pw
|
||
presubmit``.
|
||
|
||
Setting up the command-line interface
|
||
-------------------------------------
|
||
The ``pw_presubmit.cli`` module sets up the command-line interface for a
|
||
presubmit script. This defines a standard set of arguments for invoking
|
||
presubmit checks. Its use is optional, but recommended.
|
||
|
||
pw_presubmit.cli
|
||
~~~~~~~~~~~~~~~~
|
||
.. automodule:: pw_presubmit.cli
|
||
:members: add_arguments, run
|
||
|
||
Presubmit checks
|
||
----------------
|
||
A presubmit check is defined as a function or other callable. The function must
|
||
accept one argument: a ``PresubmitContext``, which provides the paths on which
|
||
to run. Presubmit checks communicate failure by raising an exception.
|
||
|
||
Presubmit checks may use the ``filter_paths`` decorator to automatically filter
|
||
the paths list for file types they care about.
|
||
|
||
Either of these functions could be used as presubmit checks:
|
||
|
||
.. code-block:: python
|
||
|
||
@pw_presubmit.filter_paths(endswith='.py')
|
||
def file_contains_ni(ctx: PresubmitContext):
|
||
for path in ctx.paths:
|
||
with open(path) as file:
|
||
contents = file.read()
|
||
if 'ni' not in contents and 'nee' not in contents:
|
||
raise PresumitFailure('Files must say "ni"!', path=path)
|
||
|
||
def run_the_build(_):
|
||
subprocess.run(['make', 'release'], check=True)
|
||
|
||
Presubmit checks functions are grouped into "programs" -- a named series of
|
||
checks. Projects may find it helpful to have programs for different purposes,
|
||
such as a quick program for local use and a full program for automated use. The
|
||
:ref:`example script <example-script>` uses ``pw_presubmit.Programs`` to define
|
||
``quick`` and ``full`` programs.
|
||
|
||
Existing Presubmit Checks
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
A small number of presubmit checks are made available through ``pw_presubmit``
|
||
modules.
|
||
|
||
Code Formatting
|
||
===============
|
||
Formatting checks for a variety of languages are available from
|
||
``pw_presubmit.format_code``. These include C/C++, Java, Go, Python, GN, and
|
||
others. All of these checks can be included by adding
|
||
``pw_presubmit.format_code.presubmit_checks()`` to a presubmit program. These
|
||
all use language-specific formatters like clang-format or yapf.
|
||
|
||
These will suggest fixes using ``pw format --fix``.
|
||
|
||
#pragma once
|
||
============
|
||
There's a ``pragma_once`` check that confirms the first non-comment line of
|
||
C/C++ headers is ``#pragma once``. This is enabled by adding
|
||
``pw_presubmit.pragma_once`` to a presubmit program.
|
||
|
||
Inclusive Language
|
||
==================
|
||
.. inclusive-language: disable
|
||
|
||
The inclusive language check looks for words that are typical of non-inclusive
|
||
code, like using "master" and "slave" in place of "primary" and "secondary" or
|
||
"sanity check" in place of "consistency check".
|
||
|
||
.. inclusive-language: enable
|
||
|
||
These checks can be disabled for individual lines with
|
||
"inclusive-language: ignore" on the line in question or the line above it, or
|
||
for entire blocks by using "inclusive-language: disable" before the block and
|
||
"inclusive-language: enable" after the block.
|
||
|
||
.. In case things get moved around in the previous paragraphs the enable line
|
||
.. is repeated here: inclusive-language: enable.
|
||
|
||
pw_presubmit
|
||
~~~~~~~~~~~~
|
||
.. automodule:: pw_presubmit
|
||
:members: filter_paths, call, PresubmitFailure, Programs
|
||
|
||
.. _example-script:
|
||
|
||
Example
|
||
-------
|
||
A simple example presubmit check script follows. This can be copied-and-pasted
|
||
to serve as a starting point for a project's presubmit check script.
|
||
|
||
See ``pigweed_presubmit.py`` for a more complex presubmit check script example.
|
||
|
||
.. code-block:: python
|
||
|
||
"""Example presubmit check script."""
|
||
|
||
import argparse
|
||
import logging
|
||
import os
|
||
from pathlib import Path
|
||
import re
|
||
import sys
|
||
from typing import List, Pattern
|
||
|
||
try:
|
||
import pw_cli.log
|
||
except ImportError:
|
||
print('ERROR: Activate the environment before running presubmits!',
|
||
file=sys.stderr)
|
||
sys.exit(2)
|
||
|
||
import pw_presubmit
|
||
from pw_presubmit import build, cli, environment, format_code, git_repo
|
||
from pw_presubmit import inclusive_language, python_checks, filter_paths
|
||
from pw_presubmit import PresubmitContext
|
||
from pw_presubmit.install_hook import install_hook
|
||
|
||
# Set up variables for key project paths.
|
||
PROJECT_ROOT = Path(os.environ['MY_PROJECT_ROOT'])
|
||
PIGWEED_ROOT = PROJECT_ROOT / 'pigweed'
|
||
|
||
#
|
||
# Initialization
|
||
#
|
||
def init_cipd(ctx: PresubmitContext):
|
||
environment.init_cipd(PIGWEED_ROOT, ctx.output_dir)
|
||
|
||
|
||
def init_virtualenv(ctx: PresubmitContext):
|
||
environment.init_virtualenv(PIGWEED_ROOT,
|
||
ctx.output_dir,
|
||
setup_py_roots=[PROJECT_ROOT])
|
||
|
||
|
||
# Rerun the build if files with these extensions change.
|
||
_BUILD_EXTENSIONS = frozenset(
|
||
['.rst', '.gn', '.gni', *format_code.C_FORMAT.extensions])
|
||
|
||
|
||
#
|
||
# Presubmit checks
|
||
#
|
||
def release_build(ctx: PresubmitContext):
|
||
build.gn_gen(PROJECT_ROOT, ctx.output_dir, build_type='release')
|
||
build.ninja(ctx.output_dir)
|
||
|
||
|
||
def host_tests(ctx: PresubmitContext):
|
||
build.gn_gen(PROJECT_ROOT, ctx.output_dir, run_host_tests='true')
|
||
build.ninja(ctx.output_dir)
|
||
|
||
|
||
# Avoid running some checks on certain paths.
|
||
PATH_EXCLUSIONS = (
|
||
re.compile(r'^external/'),
|
||
re.compile(r'^vendor/'),
|
||
)
|
||
|
||
|
||
# Use the upstream pragma_once check, but apply a different set of path
|
||
# filters with @filter_paths.
|
||
@filter_paths(endswith='.h', exclude=PATH_EXCLUSIONS)
|
||
def pragma_once(ctx: PresubmitContext):
|
||
pw_presubmit.pragma_once(ctx)
|
||
|
||
|
||
#
|
||
# Presubmit check programs
|
||
#
|
||
QUICK = (
|
||
# List some presubmit checks to run
|
||
pragma_once,
|
||
host_tests,
|
||
# Use the upstream formatting checks, with custom path filters applied.
|
||
format_code.presubmit_checks(exclude=PATH_EXCLUSIONS),
|
||
# Include the upstream inclusive language check.
|
||
inclusive_language.inclusive_language,
|
||
)
|
||
|
||
FULL = (
|
||
QUICK, # Add all checks from the 'quick' program
|
||
release_build,
|
||
# Use the upstream Python checks, with custom path filters applied.
|
||
python_checks.all_checks(exclude=PATH_EXCLUSIONS),
|
||
)
|
||
|
||
PROGRAMS = pw_presubmit.Programs(quick=QUICK, full=FULL)
|
||
|
||
|
||
def run(install: bool, **presubmit_args) -> int:
|
||
"""Process the --install argument then invoke pw_presubmit."""
|
||
|
||
# Install the presubmit Git pre-push hook, if requested.
|
||
if install:
|
||
install_hook(__file__, 'pre-push', ['--base', 'HEAD~'],
|
||
git_repo.root())
|
||
return 0
|
||
|
||
return cli.run(root=PROJECT_ROOT, **presubmit_args)
|
||
|
||
|
||
def main() -> int:
|
||
"""Run the presubmit checks for this repository."""
|
||
parser = argparse.ArgumentParser(description=__doc__)
|
||
cli.add_arguments(parser, PROGRAMS, 'quick')
|
||
|
||
# Define an option for installing a Git pre-push hook for this script.
|
||
parser.add_argument(
|
||
'--install',
|
||
action='store_true',
|
||
help='Install the presubmit as a Git pre-push hook and exit.')
|
||
|
||
return run(**vars(parser.parse_args()))
|
||
|
||
if __name__ == '__main__':
|
||
pw_cli.log.install(logging.INFO)
|
||
sys.exit(main())
|
||
|
||
---------------------
|
||
Code formatting tools
|
||
---------------------
|
||
The ``pw_presubmit.format_code`` module formats supported source files using
|
||
external code format tools. The file ``format_code.py`` can be invoked directly
|
||
from the command line or from ``pw`` as ``pw format``.
|