Python Lazy Evaluation

last modified February 24, 2025

Lazy evaluation is a programming technique where the evaluation of an expression is delayed until its value is actually needed. This can lead to significant performance improvements, especially when working with large datasets or computationally expensive operations. In this tutorial, we will explore lazy evaluation in Python using generators and compare it with non-lazy approaches using profiling.

Generating Fibonacci Sequence

This example demonstrates the difference between lazy and non-lazy approaches for generating a Fibonacci sequence.

fibonacci.py

import time import itertools from memory_profiler import memory_usage

Non-lazy approach

def fibonacci_non_lazy(n): result = [] a, b = 0, 1 for _ in range(n): result.append(a) a, b = b, a + b return result

Lazy approach

def fibonacci_lazy(n): a, b = 0, 1 for _ in range(n): yield a a, b = b, a + b

Profiling

def profile_non_lazy(): start_time = time.time() result = fibonacci_non_lazy(100_000)

# Print the first 20 elements
for e in result[:20]:
    print(e, end=' ')
print()

duration = time.time() - start_time
return duration

def profile_lazy(): start_time = time.time() slice = itertools.islice(fibonacci_lazy(100_000), 20)

# Print the first 20 elements
for e in slice:
    print(e, end=' ')
print()

duration = time.time() - start_time
return duration

def profile_non_lazy_memory(): result = fibonacci_non_lazy(100_000)

# Monitor memory usage in the loop
for e in memory_usage((print, [result[:100]])):
    pass

def profile_lazy_memory(): slice = itertools.islice(fibonacci_lazy(100_000), 100)

# Monitor memory usage in the loop
for e in memory_usage((print, [list(slice)])):
    pass

if name == “main”: # Profile non-lazy and lazy approaches with print non_lazy_me = memory_usage((profile_non_lazy_memory, )) print(’————————————-’) lazy_mem = memory_usage((profile_lazy_memory, ))

# Profile without print statements
non_lazy_delta = profile_non_lazy()
lazy_delta = profile_lazy()

print(f"Non-lazy approach: {non_lazy_me[0]} MiB used in {non_lazy_delta:.2f} seconds")
print('-------------------------------------')
print(f"Lazy approach: {lazy_mem[0]} MiB used in {lazy_delta:.2f} seconds")

In this example, the non-lazy approach generates the entire Fibonacci sequence and stores it in a list, while the lazy approach uses a generator to yield values on-the-fly. The lazy approach is more memory-efficient and faster for large sequences.

Note: be careful with large fibonacci sequences; this could overwhelm your OS.

The range function

The built-in range function is evaluated lazily.

range_fun.py

import time from memory_profiler import memory_usage

Non-lazy custom range function

def custom_non_lazy_range(start, end): result = [] current = start while current < end: result.append(current) current += 1 return result

Profiling functions

def profile_builtin_range(): start_time = time.time() result = range(1_500_000)

# Print the first 3000 elements
for e in result[:3000]:
    print(e, end=' ')
print()

duration = time.time() - start_time
return duration

def profile_custom_non_lazy_range(): start_time = time.time() result = custom_non_lazy_range(0, 1_500_000)

# Print the first 3000 elements
for e in result[:3000]:
    print(e, end=' ')
print()

duration = time.time() - start_time
return duration

if name == “main”: # Profile built-in range and custom non-lazy range builtin_range_memory = memory_usage((profile_builtin_range, )) print(’————————————-’) custom_non_lazy_range_memory = memory_usage((profile_custom_non_lazy_range, ))

# Print memory usage and durations
builtin_range_duration = profile_builtin_range()
custom_non_lazy_range_duration = profile_custom_non_lazy_range()

print(f"Built-in range: {builtin_range_memory[0]} MiB used in {builtin_range_duration:.2f} seconds")
print('-------------------------------------')
print(f"Custom non-lazy range: {custom_non_lazy_range_memory[0]} MiB used in {custom_non_lazy_range_duration:.2f} seconds")

In the example, we compare the built-in function with a custom one, which is non-lazy. We create a sequence of 1.5 mil values lazily and non-lazily. Then we pick up the first 3000. In the end, we compare the time and memory usage of both approaches.

Reading Large Files

This example compares lazy and non-lazy approaches for reading large files.

read_file.py

import time

Non-lazy approach

def read_file_non_lazy(filename): with open(filename, ‘r’) as file: return file.readlines()

Lazy approach

def read_file_lazy(filename): with open(filename, ‘r’) as file: for line in file: yield line

Profiling

start_time = time.time() read_file_non_lazy(’large_file.txt’) print(f"Non-lazy approach: {time.time() - start_time} seconds")

start_time = time.time() list(read_file_lazy(’large_file.txt’)) print(f"Lazy approach: {time.time() - start_time} seconds")

The non-lazy approach reads the entire file into memory, which can be inefficient for large files. The lazy approach reads the file line-by-line, reducing memory usage and improving performance.

Filtering Data

This example demonstrates lazy evaluation for filtering data.

filter_data.py

import time import itertools

Non-lazy approach

def filter_non_lazy(data): return [x for x in data if x % 2 == 0]

Lazy approach

def filter_lazy(data): for x in data: if x % 2 == 0: yield x

Profiling

data = range(10_000_000)

start_time = time.time() res = filter_non_lazy(data)

for e in res[:10]: print(e)

print(f"Non-lazy approach: {time.time() - start_time} seconds")

start_time = time.time() res = filter_lazy(data) for e in itertools.islice(res, 10): print(e)

print(f"Lazy approach: {time.time() - start_time} seconds")

The non-lazy approach filters the entire dataset at once, while the lazy approach filters elements on-the-fly. The lazy approach is more memory-efficient and faster for large datasets.

Infinite Sequences

This example demonstrates lazy evaluation for generating infinite sequences.

infinite_sequence.py

import time

Non-lazy approach (not feasible for infinite sequences)

Lazy approach

def infinite_sequence(): num = 0 while True: yield num num += 1

Profiling

start_time = time.time() sequence = infinite_sequence() for _ in range(1000000): next(sequence) print(f"Lazy approach: {time.time() - start_time} seconds")

The lazy approach allows us to generate an infinite sequence without consuming infinite memory. This is not feasible with a non-lazy approach.

Chaining Iterators

This example demonstrates lazy evaluation for chaining iterators.

chain_iterators.py

import time from itertools import chain

Non-lazy approach

def chain_non_lazy(iter1, iter2): return list(iter1) + list(iter2)

Lazy approach

def chain_lazy(iter1, iter2): return chain(iter1, iter2)

Profiling

iter1 = range(1000000) iter2 = range(1000000)

start_time = time.time() chain_non_lazy(iter1, iter2) print(f"Non-lazy approach: {time.time() - start_time} seconds")

start_time = time.time() list(chain_lazy(iter1, iter2)) print(f"Lazy approach: {time.time() - start_time} seconds")

The non-lazy approach combines two iterators into a single list, while the lazy approach chains them without creating an intermediate list. The lazy approach is more memory-efficient.

Processing Large Datasets

This example demonstrates lazy evaluation for processing large datasets.

process_data.py

import time

Non-lazy approach

def process_non_lazy(data): return [x * 2 for x in data]

Lazy approach

def process_lazy(data): for x in data: yield x * 2

Profiling

data = range(1000000)

start_time = time.time() process_non_lazy(data) print(f"Non-lazy approach: {time.time() - start_time} seconds")

start_time = time.time() list(process_lazy(data)) print(f"Lazy approach: {time.time() - start_time} seconds")

The non-lazy approach processes the entire dataset at once, while the lazy approach processes elements on-the-fly. The lazy approach is more memory-efficient and faster for large datasets.

Source

Python itertools Documentation

In this article, we have explored lazy evaluation in Python and demonstrated its effectiveness through practical examples and profiling comparisons.

Author

My name is Jan Bodnar, and I am a passionate programmer with extensive programming experience. I have been writing programming articles since 2007. To date, I have authored over 1,400 articles and 8 e-books. I possess more than ten years of experience in teaching programming.

List all Python tutorials.