Advanced Fish: Pipes and Filters Techniques

The Fish shell (Friendly Interactive SHell) is known for its user-friendly defaults, powerful autosuggestions, and clean syntax. However, beneath its approachable surface lies a robust set of tools for composing complex data pipelines. This tutorial dives into advanced pipe and filter techniques in Fish — moving beyond simple command1 | command2 to harness the full expressive power of the shell. You will learn how to chain transformations, process streams efficiently, and write reusable filter scripts that integrate seamlessly with Fish's builtins and external tools.

What Are Pipes and Filters?

In Unix philosophy, a pipe connects the standard output of one command to the standard input of another. A filter is a command that reads from stdin, transforms the data, and writes to stdout. Fish fully supports this paradigm with the familiar | operator, but it also adds several unique features that extend the concept:

Command substitution with () captures output as a list or string.
Process substitution via psub to treat command output as a file.
Built-in string, math, and test commands that act as powerful filters without external dependencies.
Control flow operators like and, or, and not that can be chained with pipes.
Named pipes (FIFOs) for inter-process communication.

Understanding these techniques allows you to write more concise, efficient, and readable shell scripts that avoid temporary files and unnecessary subshells.

Why It Matters

Modern development often involves processing logs, parsing configuration files, transforming JSON or CSV, and orchestrating microservices. Fish's advanced piping capabilities let you:

Reduce latency by keeping data streams flowing without disk I/O.
Improve readability by composing small, well-defined filters instead of monolithic scripts.
Leverage Fish's syntax to avoid common pitfalls like quoting issues or word splitting.
Integrate with external tools like jq, awk, sed, and grep while using Fish's builtins for performance.

Mastering these techniques is essential for anyone who regularly works at the command line or writes automation scripts in Fish.

How to Use Advanced Techniques

1. Command Substitution as a Pipe Element

Command substitution in Fish is performed with (). Unlike Bash's $(), Fish splits the result into a list of words (or lines depending on context). You can pipe the output of a command substitution directly:

# Count the number of unique file extensions in a directory
ls | string match -r '\..+$' | string trim --chars '.' | sort | uniq -c

But you can also capture the result and use it as an argument to another command that expects a file path or a list. For example, to delete all files whose names contain a number:

# Find files with digits and delete them
rm (ls | string match -r '.*\d.*')

This is equivalent to piping ls into xargs rm, but more idiomatic in Fish because it avoids the extra process. However, be cautious with spaces in filenames — Fish's command substitution splits on newlines, not spaces, so it is safer than Bash's default word splitting.

2. Using `string` as a Swiss-Army Filter

Fish's built-in string command can replace many uses of sed, awk, grep, and tr. It is fast and avoids subshell overhead. Key subcommands:

string match – filter lines matching a pattern (like grep)
string replace – substitute text (like sed)
string split – split a line into multiple lines (like tr)
string join – join lines into one (like paste)
string length – report character count
string lower/upper – change case

Example: extract email addresses from a log file, convert to lowercase, and count duplicates:

cat server.log \
  | string match -r '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' \
  | string lower \
  | sort | uniq -c | sort -rn

The pipe is clear: each string invocation acts as a filter. No external processes needed.

3. Process Substitution with `psub`

Fish provides psub to create a temporary named pipe (FIFO) that can be used as a file argument. This is invaluable when a command expects a file path but you want to feed it the output of another pipeline.

# Compare two sorted lists of files
diff (ls dir1 | sort | psub) (ls dir2 | sort | psub)

The psub command returns the path to a FIFO that contains the sorted output. diff reads from these FIFOs as if they were regular files. This avoids writing temporary files to disk.

Another common use case: feeding output into vim or less for interactive filtering:

# Edit the result of a pipeline without saving to file
vim (git log --oneline -5 | psub)