Python os.pipe Function

Last modified April 11, 2025

This comprehensive guide explores Python’s os.pipe function, which creates a pipe for interprocess communication. We’ll cover pipe creation, data flow, parent-child processes, and practical IPC examples.

Basic Definitions

The os.pipe function creates a pipe - a unidirectional data channel for interprocess communication. It returns a pair of file descriptors (read, write).

Pipes provide a way for processes to communicate by writing to and reading from the pipe. Data written to the write end can be read from the read end.

Creating a Simple Pipe

This basic example demonstrates creating a pipe and using it within a single process. While not practical, it shows the fundamental pipe operations.

simple_pipe.py

import os

Create a pipe

read_fd, write_fd = os.pipe()

Write data to the pipe

os.write(write_fd, b"Hello from pipe!")

Read data from the pipe

data = os.read(read_fd, 100) print(f"Received: {data.decode()}")

Close file descriptors

os.close(read_fd) os.close(write_fd)

This creates a pipe, writes data to it, then reads it back. Note we must close both file descriptors when done to free system resources.

The pipe is unidirectional - data flows from write_fd to read_fd. Attempting to read from write_fd or write to read_fd will fail.

Parent-Child Communication

A common use case for pipes is communication between parent and child processes. This example shows how to fork a process and exchange data.

parent_child_pipe.py

import os

Create pipe before forking

read_fd, write_fd = os.pipe()

pid = os.fork()

if pid > 0: # Parent process os.close(read_fd) # Close unused read end

# Send message to child
message = "Hello child process!"
os.write(write_fd, message.encode())
os.close(write_fd)

print("Parent sent message to child")

else: # Child process os.close(write_fd) # Close unused write end

# Receive message from parent
data = os.read(read_fd, 1024)
print(f"Child received: {data.decode()}")
os.close(read_fd)

The parent closes its read end and writes to the pipe. The child closes its write end and reads from the pipe. This demonstrates one-way communication.

Remember to close unused ends in both processes to prevent resource leaks and potential deadlocks.

Bidirectional Communication

For two-way communication between processes, we need two pipes. This example shows how to set up bidirectional communication between parent and child.

bidirectional_pipe.py

import os

Create two pipes

parent_read, child_write = os.pipe() child_read, parent_write = os.pipe()

pid = os.fork()

if pid > 0: # Parent process os.close(child_read) os.close(child_write)

# Send to child
os.write(parent_write, b"Parent says hello")

# Receive from child
data = os.read(parent_read, 1024)
print(f"Parent received: {data.decode()}")

os.close(parent_read)
os.close(parent_write)

else: # Child process os.close(parent_read) os.close(parent_write)

# Receive from parent
data = os.read(child_read, 1024)
print(f"Child received: {data.decode()}")

# Send to parent
os.write(child_write, b"Child says hi back")

os.close(child_read)
os.close(child_write)

This creates two pipes - one for parent-to-child and another for child-to-parent communication. Each process closes the ends it doesn’t use.

Bidirectional communication requires careful management of file descriptors to avoid deadlocks and ensure proper cleanup.

Pipe with Subprocess

Pipes can connect Python processes with external programs. This example shows how to use a pipe with subprocess.Popen for interprocess communication.

subprocess_pipe.py

import os import subprocess

Create pipe

read_fd, write_fd = os.pipe()

Start subprocess with pipe

proc = subprocess.Popen([“cat”], stdin=subprocess.PIPE, stdout=write_fd, close_fds=True)

Write to subprocess stdin

proc.stdin.write(b"Data for subprocess\n") proc.stdin.close()

Read from pipe

data = os.read(read_fd, 1024) print(f"Received from subprocess: {data.decode()}")

Clean up

os.close(read_fd) os.close(write_fd) proc.wait()

This creates a pipe and connects it to a subprocess’s stdout. The parent writes to the subprocess’s stdin and reads from the pipe connected to its stdout.

The close_fds=True ensures file descriptors don’t leak to the child process. This technique works with any command-line program.

Non-blocking Pipe Reads

By default, pipe reads block until data is available. This example shows how to make non-blocking reads using fcntl and handle partial reads.

nonblocking_pipe.py

import os import fcntl

Create pipe

read_fd, write_fd = os.pipe()

Set read end to non-blocking

flags = fcntl.fcntl(read_fd, fcntl.F_GETFL) fcntl.fcntl(read_fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)

Try reading (will fail with EAGAIN if no data)

try: data = os.read(read_fd, 1024) print(f"Got data: {data.decode()}") except BlockingIOError: print(“No data available yet”)

Write some data

os.write(write_fd, b"Now there’s data")

Read again

try: data = os.read(read_fd, 1024) print(f"Got data: {data.decode()}") except BlockingIOError: print(“No data available”)

Clean up

os.close(read_fd) os.close(write_fd)

This sets the O_NONBLOCK flag on the read end of the pipe. Reads will now raise BlockingIOError (EAGAIN) instead of blocking when no data is available.

Non-blocking I/O is useful for event loops or when polling multiple pipes, but requires careful error handling for partial reads and writes.

Pipe with select

The select module can monitor multiple pipes for readability. This example shows how to use select with pipes for efficient I/O multiplexing.

select_pipe.py

import os import select

Create two pipes

pipe1_r, pipe1_w = os.pipe() pipe2_r, pipe2_w = os.pipe()

Write to second pipe

os.write(pipe2_w, b"Data in pipe 2")

Use select to monitor pipes

readable, _, _ = select.select([pipe1_r, pipe2_r], [], [], 1.0)

for fd in readable: if fd == pipe1_r: print(“Pipe 1 has data”) data = os.read(pipe1_r, 1024) elif fd == pipe2_r: print(“Pipe 2 has data”) data = os.read(pipe2_r, 1024) print(f"Received: {data.decode()}")

Clean up

os.close(pipe1_r) os.close(pipe1_w) os.close(pipe2_r) os.close(pipe2_w)

Select efficiently monitors multiple file descriptors for readability. It returns only those descriptors that have data available for reading.

This pattern is useful for servers or programs that need to handle multiple communication channels simultaneously without busy waiting.

Large Data Transfer

Pipes can handle large data transfers, but require proper buffering. This example demonstrates chunked reading and writing for large data.

large_data_pipe.py

import os

read_fd, write_fd = os.pipe()

Writer process

pid = os.fork() if pid == 0: # Child (writer) os.close(read_fd)

large_data = b"X" * 1000000  # 1MB of data
chunk_size = 4096

for i in range(0, len(large_data), chunk_size):
    chunk = large_data[i:i+chunk_size]
    os.write(write_fd, chunk)

os.close(write_fd)
os._exit(0)

Reader process

os.close(write_fd) received = bytearray()

while True: chunk = os.read(read_fd, 4096) if not chunk: break received.extend(chunk)

os.close(read_fd) print(f"Received {len(received)} bytes")

This example transfers 1MB of data in chunks. The writer sends data in 4KB blocks, and the reader accumulates the chunks until the pipe is closed.

For large transfers, chunking prevents buffer overflows and allows progress tracking. The pipe automatically handles flow control between processes.

Security Considerations

• Data integrity: Pipes provide reliable in-order delivery

• Access control: Pipe file descriptors inherit to child processes

• Buffer limits: Pipes have system-defined capacity limits

• Deadlocks: Improperly closed pipes can cause hangs

• Platform differences: Behavior may vary between OSes

Best Practices

• Close unused ends: Always close pipe ends you’re not using

• Handle errors: Check for EPIPE and other pipe errors

• Use chunking: For large data, read/write in reasonable chunks

• Consider alternatives: For complex IPC, look at multiprocessing

• Clean up: Ensure all file descriptors are properly closed

Source References

• Python os.pipe Documentation

• Linux pipe(2) man page

Author

My name is Jan Bodnar, and I am a passionate programmer with extensive programming experience. I have been writing programming articles since 2007. To date, I have authored over 1,400 articles and 8 e-books. I possess more than ten years of experience in teaching programming.

List all Python tutorials.