Security Logging and Monitoring Failures

Logging enough to answer "who did what, when"

A generic access log, one line per HTTP request, method, path, status code, tells you traffic happened. It rarely tells you anything useful about a security incident six months later: which authenticated user took the action, whether it succeeded, and from where.

1@app.route("/login", methods=["POST"])
2def login():
3    user = authenticate(request.form["email"], request.form["password"])
4    if user:
5        session["user_id"] = user.id
6        return redirect("/dashboard")
7    return "Invalid credentials", 401

A thousand failed login attempts against this endpoint, spread across an hour, leave nothing behind. Not a trace, not a count, nothing to look at after the fact even if someone knew to look.

1@app.route("/login", methods=["POST"])
2def login():
3    user = authenticate(request.form["email"], request.form["password"])
4    log_security_event(
5        action="login_attempt",
6        actor=request.form.get("email"),
7        outcome="success" if user else "failure",
8        source_ip=request.remote_addr,
9    )
10    if user:
11        session["user_id"] = user.id
12        return redirect("/dashboard")
13    return "Invalid credentials", 401

The specific fields matter less than the discipline of always including them: who (the actor), what (the action), on what (the target, where applicable), when, from where, and whether it succeeded. Apply that same structure to authorization failures, password and email changes, admin actions, and data exports, the events that actually matter when someone's reconstructing what happened, not just that something did.

Logs are a copy of your data, with fewer safeguards

Logging too little leaves you blind. Logging too much of the wrong thing creates a second copy of your most sensitive data, usually shipped to a third-party log aggregator, retained for years by default, and readable by a much larger set of engineers than the production database ever is.

1log.info(f"Login attempt: {request.form}")

request.form includes the raw password field. It's now in plaintext, in your logging pipeline, wherever that pipeline sends it, for as long as your retention policy keeps it. The same habit applied elsewhere puts session tokens, full API keys, and card numbers into a system that was never built with the same access controls or encryption as the database those values came from.

1log.info("Login attempt", extra={
2    "email": request.form.get("email"),
3    "outcome": outcome,
4})

Decide what belongs in a log line the same way you'd decide what belongs in an API response: name the fields explicitly. Logging the whole request object, the whole response body, or the whole exception context is convenient right up until one of those objects happens to contain something that should never have left the process it was created in.

Having logs isn't the same thing as having detection

Every fix in this article so far produces logs. Logs sitting in storage, queried for the first time after someone else tells you something went wrong, aren't monitoring. Monitoring means something is actively watching for a specific pattern and telling a human before the damage is done, not after.

The patterns worth alerting on are usually more specific than "an error occurred." A few that map directly onto attacks covered elsewhere in this series:

1ALERT IF failed_logins(account=X, window=10m) > 5
2  # one account, many attempts: brute force
3
4ALERT IF failed_logins(source_ip=Y, window=10m) > 20 across distinct accounts
5  # many accounts, one source: credential stuffing
6
7ALERT IF admin_action performed AND (hour outside 06:00-22:00 OR new_device = true)
8  # a privileged action at an unusual time, from somewhere new
9
10ALERT IF GET /users/{id}(actor=Z, window=1h) > 200 distinct ids
11  # one account reading far more records than a real user ever would

None of these require anything exotic, they're counts and thresholds over the structured events from the first section of this article. What they require is deciding, ahead of time, which specific patterns actually matter, instead of hoping someone eventually reads the raw log stream and happens to notice something wrong in it.

Closing the gap between the breach and noticing it

Year after year, industry breach investigation reports keep finding the same shape: median detection time measured in months, not hours, and a large share of breaches are first noticed by someone outside the company, a customer, a journalist, a law enforcement notification, rather than by the company's own monitoring. Logging and alerting exist specifically to close that gap, and three things usually keep it open anyway.

Retention shorter than your detection time is a contradiction

Logs retained for seven days are worthless for investigating something discovered three months later, and if your own numbers say detection realistically takes longer than a week, seven-day retention was never actually protecting you, just satisfying a checkbox. Set retention based on how long an investigation might realistically need to look back, not on what's cheapest to store.

An alert nobody has ever tested is a hypothesis, not a control

A detection rule that's never fired might be working perfectly. It might also have been broken by a refactor eighteen months ago and nobody has noticed, because it's never had a reason to fire. Regular incident response drills, deliberately triggering the condition an alert is supposed to catch, are the only way to find out which one it is before a real incident does.

Correlating a dozen separate log files during an active incident is too late

If the application, the database, the load balancer, and the auth provider each keep logs in their own separate system, reconstructing one attacker's path across all four means building that correlation pipeline for the first time, under pressure, during the incident itself. Centralizing logs into one searchable place is infrastructure work that has to happen before there's anything to investigate, not work that gets improvised once there is.