Security Monitoring and Alerting
Explain how monitoring turns log lines into alerts, and tune an alert rule by weighing false positives against false negatives.
Logs nobody reads protect nobody
Tulsi Health's systems write tens of thousands of log lines a day, and Neha's team has two people. If they only read the logs after something goes wrong, an attacker could sit inside Sameer's mailbox for weeks before anyone looked. Continuous security monitoring means watching systems and their logs all the time, mostly with software, so that trouble is noticed within minutes instead of months. The software reads every line as it arrives, compares it with rules, and raises a security alert, a message to a person saying "this matched something worrying, please look", when something stands out. The people then decide what the alert really means.
A smoke detector is the everyday version. It watches the air in the kitchen all day and night, so nobody has to stand there sniffing for smoke. When it detects smoke it beeps, and a person comes to check whether the house is on fire or someone is making rotis.
From log line to alert
Monitoring is a chain, and every link matters. The logs must be collected from every system; the rules must check them quickly; the alert must reach someone who is awake; and that person must be able to act.
A broken link anywhere breaks the chain: a system that sends no logs, a rule that never matches, or an alert sent to a phone nobody checks at night.
A good alert also saves the person time. "Unusual activity" says almost nothing. "Seven failed logins to sameer.j from Bhopal at 02:09, then a successful login at 02:11; Sameer normally logs in from Pune" says who, what, when, where, and why it is odd, so Neha can start checking immediately.
Two ways to spot suspicious behaviour
Monitoring tools find suspicious behaviour in two main ways. The first uses rules that describe known warning signs: alert if any account has more than five failed logins in ten minutes, if a laptop contacts an address on a list of criminal servers, or if anyone exports more than 10,000 customer records at once. Rules are easy to understand and explain, but they only catch what someone thought of in advance.
The second compares behaviour with what is normal for that person or system, often called a baseline. Farida logs in from the Kothrud store between 9 am and 8 pm, six days a week. A login to her account at 3 am from another state is unusual for her, even though no fixed rule mentions it. The watchman of a housing society works the same way: he knows the milkman comes at 6 and the cook at 8, so a stranger at 2 am stands out at once. Baselines catch new tricks that no rule describes, but they also flag innocent changes, such as Farida's first late stock-taking night.
Most monitoring uses both: rules for the attacks everyone knows about, and baselines for the ones nobody predicted.
The cell below shows the simplest rule there is. It counts the failed logins in seven log lines for Sameer's account and raises an alert if the count reaches a limit, called the threshold. You can press Run without knowing any Python. Then change the number on the threshold line, which is marked with a comment, and run it again.
Expected output:
Failed logins for sameer.j: 6
ALERT: possible password guessing on sameer.j
| Line | What it does |
|---|---|
log = [...] |
Stores the seven log lines as a list of text, in the order they happened |
threshold = 5 |
The limit: this many failures or more raises an alert |
failed = 0 |
Starts a counter at zero |
for line in log: |
Repeats the next two lines once for each log line |
if "FAILED" in line: |
Checks whether this line records a failed login |
failed = failed + 1 |
Adds one to the counter for each failure |
if failed >= threshold: |
Compares the count with the limit and prints an alert or "No alert" |
Six failures reach the limit of 5, so the rule raises an alert. Change the threshold to 10 and the same attack passes silently, which is exactly the kind of miss the next section is about.
False positives and false negatives
Every alerting system makes two kinds of mistake. A false positive is an alert when nothing bad is happening: the smoke detector screaming at rotis, or Farida's account flagged because she kept typing her old password after changing it. A false negative is the opposite and more dangerous mistake: something bad is happening and no alert is raised, like a smoke detector with a dead battery during a real fire.
Both detectors have failed, in opposite directions. One wastes attention; the other misses the thing it exists to catch.
False positives sound harmless, but they cause alert fatigue. When most alerts turn out to be nothing, people start to ignore or close them without reading, just as a whole colony ignores a car alarm that goes off every night. Then the one real alert is dismissed with the rest. False negatives give no warning at all: the attacker keeps working, and the organisation finds out much later, often from a customer or the police.
Tuning a rule moves outcomes between boxes. Making it stricter turns false positives into silence, but some real alerts slide into false negatives too.
Sort these Tulsi Health outcomes into the right box below. Ask two questions: was there an alert, and was something really wrong?
Tuning the threshold
The threshold in a rule decides the balance between the two mistakes. Set "failed logins in ten minutes" to 3, and staff who mistype their password a few times trigger alerts all week. Set it to 15, and the alerts stop, but so does any warning about an attacker who tries only a handful of passwords. No single number is right for every attacker, which is why careful teams combine a sensible threshold with a second, different kind of check.
The model below holds one week at Tulsi Health. Staff mistype passwords about 110 times in short bursts, mostly once or twice, occasionally up to seven times after a password change. Four real attacks also happen: automated guessing of 40 and 12 passwords, a person trying 8, and a careful attacker who tries only 3 before logging in from Bhopal. Start with the slider alone and see whether any threshold manages both goals in the mission.