Explain File Handling in Python: A Step-by-Step Guide with Real-World Examples

What Is File Handling in Python?

File handling is how a Python program talks to files stored on a disk, SD card, or flash memory. Python gives you a file object, also called a file handle, that works like a pipe between your code and the file.

In simple words, you tell Python which file to use, what you want to do with it, and when you are finished.

Think of RAM as a whiteboard and a file as a notebook. The whiteboard is fast but gets wiped when the power goes off. The notebook is slower but keeps everything you wrote.

Why File Handling Matters

• Persistence: Data survives after the program ends or the device restarts.
• Input: Read configuration, datasets, logs, and text-based build artifacts such as .hex and map files.
• Output: Save reports, results, and sensor readings.
• Exchange: Share data with spreadsheets and other tools through CSV and JSON.

Text Files vs Binary Files

Rule of thumb: Choose text for data people will read, and binary for compact or byte-exact data such as sensor frames.

Coming from C? A Quick Comparison

If you have written C code with FILE *, Python will feel familiar, with far less manual bookkeeping.

The biggest win is the with statement. It behaves like RAII in C++: the file is released when the block ends, so you cannot leak a file descriptor by forgetting fclose().

How File Handling in Python Works Behind the Scenes

• Open: open() asks the operating system for access and returns a file object.
• Buffer: Reads and writes pass through a memory buffer, so data may not reach storage immediately.
• Flush: flush() pushes the buffer to the operating system, and os.fsync() asks the OS to write it to storage.
• Close: close() flushes the buffer and releases the file descriptor.
• Limits: A process can keep only a limited number of files open, so unclosed files eventually cause errors.

Step-by-Step Guide to File Handling in Python

Work through these steps in order and you will have the complete workflow covered.

Step 1: Open the File with open()

file_object = open(file_path, mode='r', encoding='utf-8')

• file_path: The name or path of the file, as a string or a Path object.
• mode: What you plan to do with the file. The default is 'r'.
• encoding: Set it for text files, for example 'utf-8', so your code behaves the same on Windows, Linux, and a Raspberry Pi.

If you open a missing file for reading, Python raises FileNotFoundError.

Step 2: Choose the Right File Mode

Watch out: 'w' is the most common cause of accidental data loss. If you only want to add data, use 'a'.

Step 3: Write Data to the File

with open('report.txt', 'w', encoding='utf-8') as f:
    f.write('Device: ESP32\n')
    f.write('Status: Online\n')
    f.writelines(['Temp: 27.4 C\n', 'Humidity: 61 %\n'])

• write() and writelines() do not add newline characters, so add \n yourself.
• Text mode accepts str only, and binary mode accepts bytes only.

Step 4: Append Data Without Erasing

with open('report.txt', 'a', encoding='utf-8') as f:
    f.write('Uptime: 3 hours\n')

• Append mode keeps the old content and always writes at the end.
• It is the natural choice for logs.

Step 5: Read Data Back from the File

with open('report.txt', 'r', encoding='utf-8') as f:
    text = f.read()          # whole file as one string
    print(text)

• read() returns everything, and read(n) returns the next n characters.
• readline() returns one line each time you call it.
• readlines() returns a list of all lines, so use it only on small files.

Looping over the file object is best for big files because it reads one line at a time:

with open('report.txt', 'r', encoding='utf-8') as f:
    for line in f:
        print(line.strip())

Pro tip: line.strip() removes the trailing newline, so your output does not show blank lines between entries.

Step 6: Close the File and Why with Is Better

# Manual way: works, but easy to get wrong
f = open('data.txt', 'r', encoding='utf-8')
try:
    data = f.read()
finally:
    f.close()

# Recommended way: the file closes automatically
with open('data.txt', 'r', encoding='utf-8') as f:
    data = f.read()

• close() flushes the buffer and frees the file descriptor.
• A with block closes the file even if an exception occurs inside it.
• Unclosed files can lose buffered data if the program crashes or the device resets.
• You can open several files in one statement: with open('a.txt') as src, open('b.txt', 'w') as dst:

Step 7: Move Around with seek() and tell()

with open('data.bin', 'wb') as f:
    f.write(bytes(range(10)))       # bytes 0 to 9

with open('data.bin', 'rb') as f:
    f.seek(4)                       # jump to byte 4
    chunk = f.read(2)               # reads bytes 4 and 5
    print(list(chunk), f.tell())    # [4, 5] 6

• tell() returns the current position.
• seek(offset, whence) moves it: 0 is the start, 1 is the current position, and 2 is the end.
• For byte-exact jumps, like fseek in C, use binary mode, because text mode allows only limited offsets.

Step 8: Handle Errors Gracefully

try:
    with open('config.json', 'r', encoding='utf-8') as f:
        data = f.read()
except FileNotFoundError:
    print('Config file is missing, loading defaults.')
except PermissionError:
    print('No permission to read the file.')
except OSError as err:
    print(f'File error: {err}')

• Put specific exceptions first, because FileNotFoundError and PermissionError are both subclasses of OSError.
• Falling back to defaults lets a device still boot when a file is missing.

How to Use File Handling in Python: 5 Practical Patterns

Reading and writing plain text is only the start. These are the patterns you will reach for again and again.

1. Log Sensor Readings to a CSV File

import csv
from datetime import datetime

def log_reading(temp_c, humidity):
    with open('readings.csv', 'a', newline='', encoding='utf-8') as f:
        writer = csv.writer(f)
        writer.writerow([datetime.now().isoformat(), temp_c, humidity])

log_reading(27.4, 61)

• newline='' stops the csv module from adding blank rows on Windows.
• csv.writer takes care of commas and quotes inside values.
• Use csv.reader or csv.DictReader to read the data back.

2. Save and Load Settings with JSON

import json

settings = {'device_id': 'node-07', 'sample_rate_hz': 10}

with open('config.json', 'w', encoding='utf-8') as f:
    json.dump(settings, f, indent=4)

with open('config.json', 'r', encoding='utf-8') as f:
    loaded = json.load(f)

• JSON is readable, easy to edit, and ideal for configuration and calibration values.
• It works on MicroPython too, so one format can serve both your PC tools and your board.

3. Read and Write Binary Frames with struct

import struct

FRAME = '<Hhf'                    # uint16 id, int16 temp x100, float32 volts
size = struct.calcsize(FRAME)     # 8 bytes

with open('frames.bin', 'ab') as f:
    f.write(struct.pack(FRAME, 7, 2740, 3.29))

with open('frames.bin', 'rb') as f:
    node_id, temp_raw, volts = struct.unpack(FRAME, f.read(size))
    print(node_id, temp_raw / 100, round(volts, 2))

• Every frame is exactly 8 bytes, so frame number n starts at byte n * 8 and seek() can jump straight to it.
• The < prefix fixes the byte order and removes padding, so the layout matches a packed C struct.
• It is the Python counterpart of calling fwrite() on a struct, with the layout spelled out explicitly.

4. Handle Paths the Modern Way with pathlib

from pathlib import Path

log_dir = Path('logs')
log_dir.mkdir(parents=True, exist_ok=True)

log_file = log_dir / 'boot.log'
log_file.write_text('System started\n', encoding='utf-8')
print(log_file.read_text(encoding='utf-8'))
print(log_file.exists())

• The / operator joins paths safely on Windows, Linux, and macOS.
• exists(), is_file(), and iterdir() replace many os.path calls.
• write_text() replaces the file content, so treat it like mode 'w'.

5. Rename, Copy, and Delete Files

os.rename('old.txt', 'new.txt')  # Renames a file
shutil.copy('a.txt', 'b.txt')     # Copies a file
os.remove('a.txt')                # Deletes a file
os.listdir('.')                   # Lists the contents of a folder

Mini Project: Build a Simple Sensor Data Logger

Time to combine everything. This script creates a log folder, writes a CSV header once, and appends a new reading every second.

import csv
import random
import time
from pathlib import Path

LOG_FILE = Path('logs/sensor_log.csv')
LOG_FILE.parent.mkdir(parents=True, exist_ok=True)

def read_sensor():
    # Replace this with a real sensor read on your board
    return round(random.uniform(24.0, 30.0), 1), random.randint(55, 70)

is_new_file = not LOG_FILE.exists()

with open(LOG_FILE, 'a', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    if is_new_file:
        writer.writerow(['timestamp', 'temp_c', 'humidity_pct'])
    for _ in range(5):
        temp, hum = read_sensor()
        writer.writerow([time.strftime('%Y-%m-%d %H:%M:%S'), temp, hum])
        f.flush()
        time.sleep(1)

What This Script Demonstrates

• pathlib creates the folder and checks whether the file already exists.
• Mode 'a' keeps old readings every time you run the script.
• The header row is written only once.
• flush() after every row keeps data loss small if the power drops.

Ways to Extend It

• Swap read_sensor() for a DHT22 or BME280 reading on a Raspberry Pi.
• Rotate the file when it passes a size limit.
• Upload the CSV to a server when Wi-Fi comes back.

File Handling in Embedded Systems and IoT with Python

File handling looks the same on a laptop and on a small board, but the stakes are higher. Storage is limited, power can vanish at any moment, and flash memory wears out. A classic field failure is a logger that works perfectly on the desk and corrupts its file the first time someone unplugs it.

Where You Will Use It on Real Hardware

• Raspberry Pi: Data loggers for sensors such as the DHT22 and BME280, plus system and application logs on the SD card.
• MicroPython boards (ESP32, Raspberry Pi Pico): open(), read(), write(), and os.listdir() work on the board’s flash file system.
• IoT gateways: Buffer readings in a file when Wi-Fi is down and upload them later.
• Calibration data: Store sensor offsets in JSON and load them at boot.
• Hardware test automation: Save pass or fail results as CSV for every test run.

Protect Your Data from Power Loss

Writes sit in buffers first. If power drops before the data reaches storage, it is gone.

import os

with open('readings.csv', 'a', encoding='utf-8') as f:
    f.write('2026-10-17T10:30:00,27.4,61\n')
    f.flush()                # push the Python buffer to the OS
    os.fsync(f.fileno())     # ask the OS to write it to storage

For files that must never be half-written, such as configuration, write to a temporary file and then swap it in:

import json
import os

def save_config(path, data):
    tmp = path + '.tmp'
    with open(tmp, 'w', encoding='utf-8') as f:
        json.dump(data, f)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)    # atomic swap on Linux: the old file stays valid until this line

If power fails before os.replace(), the old configuration is still intact.

On MicroPython, call f.flush() and close files promptly, because its os module is a reduced version of the CPython one.

Respect Flash and SD Card Limits

• Flash memory and SD cards survive a limited number of erase and write cycles, so avoid writing every few milliseconds.
• Collect readings in a list and write them in batches, for example once a minute.
• Check free space before writing. On MicroPython, os.statvfs('/') reports block size and free blocks.
• Rotate logs: start a new file after a size limit and delete the oldest one.
• Use compact formats such as binary frames or JSON Lines when space is tight.

When to Use File Handling in Python

File handling is the right tool when your data is small, local, and simple. It is the wrong tool when many users or devices need to read and write the same data at the same time.

Use File Handling When You Need To

• Store logs, readings, or results locally without extra software.
• Keep configuration or calibration values between runs.
• Process text data such as logs, CSV exports, datasets, and reports.
• Exchange data with spreadsheets and other tools using CSV or JSON.
• Buffer data on a device that may be offline before uploading it.
• Parse or generate binary formats such as sensor frames and firmware images.

Pick Something Else When

SQLite is worth remembering: Python ships with the sqlite3 module, and the whole database lives in a single file.

Python File Handling Methods at a Glance

Common Mistakes in Python File Handling and How to Avoid Them

• Forgetting to close the file: Use a with block every time.
• Opening with 'w' by accident: It erases the file, so use 'a' to add data or 'x' to refuse overwriting.
• Skipping encoding: Set encoding='utf-8' so behavior matches across systems.
• Reading huge files with read() or readlines(): Loop over the file object instead.
• Forgetting newline characters: write() does not add \n.
• Mixing text and binary: 'wb' expects bytes, not str.
• Hard-coding Windows paths: Use pathlib, because a backslash in a normal string acts as an escape character.
• Ignoring exceptions: Wrap risky file operations in try / except.
• Trusting the buffer on unstable power: Call flush() and os.fsync() for critical data.

Key Takeaways

• File handling lets a Python program store and retrieve data permanently.
• The workflow is open, read or write, then close, and the with statement handles the closing for you.
• Your mode choice matters, because 'w' erases existing content.
• CSV, JSON, struct, and pathlib cover most real-world needs.
• On embedded and IoT devices, flush critical data, write in batches, and save configuration atomically.
• Use files for small, local data, and move to a database or MQTT when many writers or readers are involved.

Ready to take Python beyond scripts and into real hardware? Combine these file handling skills with C, microcontrollers, and IoT projects through the hands-on embedded systems programs at IIES, Bangalore.