feature/library

Provider-agnostic email ingestion with reliability built in. Toggle a feature and watch its real code drop into the program on the right — toggle it off and it's gone. Every snippet is the actual API; the guarantees in the always-on row come for free.

added to program
mailflow brand / selectable
always-on (no code needed)
provider
source
parse
envelope
filter
which mail
extract
CleanEmail
clean
html→text
stages
transform
deliver
to you
always on: dedupe cursor size guard retry → DLQ at-least-once 401 → refresh-once durable DLQ + redrive ttl purge attachment cap blob dedupe scope check metrics + health postgres retry

Compose

// click to add / remove
Source
Delivery (graph only)
State
Consume
Secrets

Your program

quickstart.py — built live
$ python quickstart.py · 0 features composed

Installing mailflow

// not on PyPI — install straight from the company repo

Not published to PyPI. Two install paths depending on what you're doing: pull it in as a dependency for your own app, or clone it to work on mailflow itself.

git+sshrecommended

Needs your GitHub SSH key registered on your account — no separate token to manage. Best for individual developers.

git+httpsCI / no SSH

Uses a GitHub personal access token via an env var — never hardcode it. Best for CI/CD pipelines or machines without an SSH key set up.

git clonecontributing

Editable install (pip install -e) — for anyone actually changing mailflow's code, not just consuming it.

# Using mailflow in your own project — pick ONE:

# A) SSH (recommended — needs your GitHub SSH key registered)
$ pip install "mailflow[gmail] @ git+ssh://git@github.com/Jetrix-TJ/Email-Intelligence.git@main"

# B) HTTPS + token (CI/CD, or no SSH set up) — token from an env var, never hardcoded
$ pip install "mailflow[gmail] @ git+https://${GITHUB_TOKEN}@github.com/Jetrix-TJ/Email-Intelligence.git@main"

# swap [gmail] for whichever extras you need, or combine them:
#   [graph]  [servicebus]  [postgres]  [gcpsm]   e.g. "mailflow[gmail,postgres] @ git+ssh://..."
# Contributing to mailflow itself (not just using it):
$ git clone git@github.com:Jetrix-TJ/Email-Intelligence.git
$ cd Email-Intelligence
$ pip install -e ".[dev,gmail,graph,servicebus,postgres,gcpsm]"   # or just the extras you need

$ python -m pytest -m "not slow"   # fast tier, ~15s, no cloud needed
$ python -m mypy               # strict type check
Pin to a tag once one exists, not @main. main moves as the team pushes to it — a bare @main install can silently change under you. Once a release is tagged (git tag v0.1.0 && git push --tags), reference @v0.1.0 instead for reproducible installs.

Configuring attachments

// connect(attachments=…) — types · sizes · scanning

Attachments stream through a per-class byte cap, are content-addressed (identical bytes are stored once), and pass a safety seam before they're persisted. Pass attachments= to connect() to activate strip-and-deliver: offending parts are removed, recorded on email.stripped_attachments, and the message is still delivered. Without a policy the default remains fail-closed — a single blocked or over-cap part dead-letters the entire message.

max_bytesdefault 25 MB

Hard per-attachment cap, enforced while streaming. With a policy: over-cap parts are stripped and recorded (reason="oversize"). Without a policy: a single over-cap part dead-letters the entire message — nothing partial is stored.

allowlistdefault ∅ = allow all

A frozenset of permitted content-types or lowercased file extensions — e.g. "application/pdf" or "pdf". Either one matching admits the part. Non-matching parts are stripped with reason="not_allowlisted".

scannerdefault no-op

A swappable AttachmentScanner port — your .scan(attachment) returns a ScanResult (allow / block). The default allows everything; drop in an AV / CDR check here. Blocked parts are stripped with reason="scanner".

Two attachment classes — real vs. inline. AttachmentPolicy has two slots: real (parts with Content-Disposition: attachment, or with a filename and no inline disposition) and inline (parts with Content-Disposition: inline or a Content-ID header that are not explicitly dispositioned as attachment — CID-referenced logos, tracking pixels, and embedded images). Pass a bare dict or AttachmentRule to apply the same policy to both classes at once.

from mailflow import connect, AttachmentPolicy, AttachmentRule

# Simple: a bare dict applies to BOTH classes (real + inline)
mf = connect("gmail", credentials=creds,
    attachments={"max_bytes": 10_000_000, "allowlist": ["application/pdf"]},
)

# Per-class: different rules for real vs. inline parts
mf = connect("gmail", credentials=creds,
    attachments=AttachmentPolicy(
        real=AttachmentRule(max_bytes=10_000_000,
                           allowlist=["application/pdf", "image/png"]),
        inline=AttachmentRule(max_bytes=2_000_000),  # size-cap only
    ),
)

# Stripped parts are recorded on the delivered CleanEmail:
for email in mf.stream():
    for s in email.stripped_attachments:
        print(s.filename, s.content_type, s.size_bytes, s.reason.value)
        # reason.value: "not_allowlisted" | "oversize" | "unreadable" | "scanner"
        # (s.reason itself is the StripReason enum member, not the bare string)
from mailflow.core.models import ScanResult, ScanVerdict

class PdfOnlyScanner:                       # optional AV / CDR hook
    def scan(self, attachment):
        if attachment.content_type == "application/pdf":
            return ScanResult()                 # verdict defaults to allow
        return ScanResult(verdict=ScanVerdict.block, reason="not a pdf")

mf = connect("gmail", credentials=creds,
    attachments=AttachmentRule(
        max_bytes=10_000_000,
        allowlist=["application/pdf"],
        scanner=PdfOnlyScanner(),
    ),
)
Strip-and-deliver is opt-in; the default remains fail-closed. Without an attachments= policy, a single over-cap, scanner-blocked, or unreadable part dead-letters the entire message — it is quarantined, not dropped per-part. With a policy, the offending part is stripped and its metadata appended to email.stripped_attachments; the message is then delivered normally. A non-empty allowlist runs on every non-body part (real and inline) — include the inline types you expect in ordinary mail (e.g. image/png, image/jpeg), or a routine logo will be stripped.

Command-line operations

// the `mailflow` CLI — auth · check · redrive · purge

Everything above is the connect() library surface for your own process. A few things are operational tasks instead — a human or a cron job running them against a state= location, not something your ingestion code calls. The mailflow CLI (installed alongside the package) covers these.

CommandWhat it does
mailflow auth gmailOne-time OAuth consent flow → writes GMAIL_REFRESH_TOKEN to .env.
mailflow auth graphOne-time Microsoft delegated OAuth consent → writes GRAPH_REFRESH_TOKEN.
mailflow check gmailVerifies the saved Gmail credentials (+ Pub/Sub) actually work.
mailflow check graph [--app-only]Verifies Graph credentials — delegated /me, or app-only client-credentials against a named mailbox.
mailflow redrive --state … --tenant … [--limit N]Re-submits every durable dead-letter at that state= location through a fresh pipeline. Still-failing messages stay in the DLQ; recovered ones are deleted from it.
mailflow purge --state …Deletes dedupe records past their done_ttl_seconds at that state= location — the CLI form of mf.purge_expired(), meant for a cron/scheduled job.
# one-time setup
$ mailflow auth gmail
$ mailflow check gmail

# operational housekeeping — run redrive after fixing a root cause,
# run purge on a schedule (e.g. daily cron) to bound dedupe-store growth
$ mailflow redrive --state sqlite:///mailflow.db --tenant acme --limit 50
examined=3 resubmitted=2 still_dead_lettered=1

$ mailflow purge --state postgresql://user:pass@host:5432/mailflow
purged 214 expired dedupe record(s)

Same state= location, both sides. Point the CLI at the exact state= string your connect() call uses (or its state_ref=-resolved equivalent) — the CLI reads and writes the same cursor/dedupe/DLQ tables your running process does.