# msgid "" msgstr "" msgid "" "Inspired by Fred Herbert's \"[Awk in 20 Minutes](https://ferd.ca/awk-" "in-20-minutes.html)\", here's a problem I just solved with a line of Awk: " "run ShellCheck in all scripts of a repository." msgstr "" msgid "" "In my repositories I usually have Bash and POSIX scripts, which I want to " "keep tidy with [ShellCheck](https://www.shellcheck.net/). Here's the first " "version of `assert-shellcheck.sh`:" msgstr "" msgid "" "#!/bin/sh\n" "set -eu\n" "\n" "find . -type f -name '*.sh' -print0 | xargs -0 shellcheck\n" msgstr "" msgid "" "This is the type of script that I copy around to all repositories, and I " "want it to be capable of working on any repository, without requiring a list" " of files to run ShellCheck on." msgstr "" msgid "" "This first version worked fine, as all my scripts had the '.sh' ending. But " "I recently added some scripts without any extension, so `assert-" "shellcheck.sh` called for a second version. The first attempt was to try " "grepping the shebang line:" msgstr "" msgid "" "$ grep '^#!/' assert-shellcheck.sh\n" "#!/usr/sh\n" msgstr "" msgid "" "Good, we have a grep pattern on the first try. Let's try to find all the " "matching files:" msgstr "" msgid "" "$ find . -type f | xargs grep -l '^#!/'\n" "./TODOs.org\n" "./.git/hooks/pre-commit.sample\n" "./.git/hooks/pre-push.sample\n" "./.git/hooks/pre-merge-commit.sample\n" "./.git/hooks/fsmonitor-watchman.sample\n" "./.git/hooks/pre-applypatch.sample\n" "./.git/hooks/pre-push\n" "./.git/hooks/prepare-commit-msg.sample\n" "./.git/hooks/commit-msg.sample\n" "./.git/hooks/post-update.sample\n" "./.git/hooks/pre-receive.sample\n" "./.git/hooks/applypatch-msg.sample\n" "./.git/hooks/pre-rebase.sample\n" "./.git/hooks/update.sample\n" "./build-aux/with-guile-env.in\n" "./build-aux/test-driver\n" "./build-aux/missing\n" "./build-aux/install-sh\n" "./build-aux/install-sh~\n" "./bootstrap\n" "./scripts/assert-todos.sh\n" "./scripts/songbooks\n" "./scripts/compile-readme.sh\n" "./scripts/ci-build.sh\n" "./scripts/generate-tasks-and-bugs.sh\n" "./scripts/songbooks.in\n" "./scripts/with-container.sh\n" "./scripts/assert-shellcheck.sh\n" msgstr "" msgid "" "This approach has a problem, though: it includes files ignored by Git, such " "as `builld-aux/install-sh~`, and even goes into the `.git/` directory and " "finds sample hooks in `.git/hooks/*`." msgstr "" msgid "To list the files that Git is tracking we'll try `git ls-files`:" msgstr "" msgid "" "$ git ls-files | xargs grep -l '^#!/'\n" "TODOs.org\n" "bootstrap\n" "build-aux/with-guile-env.in\n" "old/scripts/assert-docs-spelling.sh\n" "old/scripts/build-site.sh\n" "old/scripts/builder.bats.sh\n" "scripts/assert-shellcheck.sh\n" "scripts/assert-todos.sh\n" "scripts/ci-build.sh\n" "scripts/compile-readme.sh\n" "scripts/generate-tasks-and-bugs.sh\n" "scripts/songbooks.in\n" "scripts/with-container.sh\n" msgstr "" msgid "" "It looks to be almost there, but the `TODOs.org` entry shows a flaw in it: " "grep is looking for a `'^#!/'` pattern on any part of the file. In my case, " "`TODOs.org` had a snippet in the middle of the file where a line started " "with `#!/bin/sh`." msgstr "" msgid "" "So what we actually want is to match the **first** line against the pattern." " We could loop through each file, get the first line with `head -n 1` and " "grep against that, but this is starting to look messy. I bet there is " "another way of doing it concisely..." msgstr "" msgid "" "Let's try Awk. I need a way to select the line numbers to replace `head -n " "1`, and to stop processing the file if the pattern matches. A quick search " "points me to using `FNR` for the former, and `{ nextline }` for the latter. " "Let's try it:" msgstr "" msgid "" "$ git ls-files | xargs awk 'FNR>1 { nextfile } /^#!\\// { print FILENAME; nextfile }'\n" "bootstrap\n" "build-aux/with-guile-env.in\n" "old/scripts/assert-docs-spelling.sh\n" "old/scripts/build-site.sh\n" "old/scripts/builder.bats.sh\n" "scripts/assert-shellcheck.sh\n" "scripts/assert-todos.sh\n" "scripts/ci-build.sh\n" "scripts/compile-readme.sh\n" "scripts/generate-tasks-and-bugs.sh\n" "scripts/songbooks.in\n" "scripts/with-container.sh\n" msgstr "" msgid "" "Great! Only `TODOs.org` is missing, but the script is much better: instead " "of matching against any part of the file that may have a shebang-like line, " "we only look for the first. Let's put it back into the `assert-" "shellcheck.sh` file and use `NULL` for separators to accommodate files with " "spaces in the name:" msgstr "" msgid "" "#!/usr/sh\n" "set -eu\n" "\n" "git ls-files -z | \\\n" " xargs -0 awk 'FNR>1 { nextfile } /^#!\\// { print FILENAME; nextfile }' | \\\n" " xargs shellcheck\n" msgstr "" msgid "" "This is where I've stopped, but I imagine a likely improvement: match " "against only `#!/bin/sh` and `#!/usr/bin/env bash` shebangs (the ones I use " "most), to avoid running ShellCheck on Perl files, or other shebangs." msgstr "" msgid "" "Also when reviewing the text of this article, I found that `{ nextfile }` is" " a GNU Awk extension. It would be an improvement if `assert-shellcheck.sh` " "relied on the POSIX subset of Awk for working correctly." msgstr "" msgid "title: 'Awk snippet: ShellCheck all scripts in a repository'" msgstr "" msgid "date: 2020-12-15" msgstr "" msgid "layout: post" msgstr "" msgid "lang: en" msgstr "" msgid "ref: awk-snippet-shellcheck-all-scripts-in-a-repository" msgstr "" msgid "*Update*" msgstr "" msgid "" "After publishing, I could remove `{ nextfile }` and even make the script " "simpler:" msgstr "" msgid "" "#!/usr/sh\n" "set -eu\n" "\n" "git ls-files -z | \\\n" " xargs -0 awk 'FNR==1 && /^#!\\// { print FILENAME }' | \\\n" " xargs shellcheck\n" msgstr "" msgid "Now both the shell and Awk usage are POSIX compatible." msgstr "" msgid "eu_categories: shell" msgstr "" #~ msgid "" #~ "title: 'Awk snippet: ShellCheck all scripts in a repository'\n" #~ "date: 2020-12-15\n" #~ "layout: post\n" #~ "lang: en\n" #~ "ref: awk-snippet-shellcheck-all-scripts-in-a-repository" #~ msgstr ""