Research

I'm building an evaluation environment to test whether security controls can contain autonomous coding agents across multi-step Linux tasks. The core question is whether actions that are individually allowed can become unsafe when an agent combines them over a longer trajectory.

I evaluated the released EMBER2024 LightGBM malware detector on 540,000 Windows PE records without retraining or recalibrating it. The model reached 98.0% accuracy, 0.9982 ROC AUC, and 0.0031 ECE overall. Those numbers suggest a highly reliable detector.

Slice-level evaluation changed the picture. Malicord had a 0.602 ECE and an 85.2% false-negative rate. Lazzzy and rugmi also had false-negative rates above 61%. This is a concrete evaluation failure mode: strong population metrics can miss concentrated errors in smaller groups.

Inspired by NIST's update on how it is handling record CVE growth, I built a reproducible baseline that predicts severity, CWE, CVSS score, and eight vector components from CVE descriptions. It trains on records published in 2022 and 2023 and tests on 36,539 records published in 2024. I also tested how the results change with split design, duplicates, label frequency, and similarity to the training data.

TF-IDF reached 66.05% four-class severity accuracy, 67.15% closed-set CWE accuracy, 0.997 CVSS mean absolute error, and 86.13% average component accuracy. The aggregate scores were incomplete: LOW-severity recall was zero, and unseen CWEs and labels seen one to nine times in training had zero original-label recovery. Predicted score and severity also disagreed on 19.63% of test records. I see the model as a baseline or decision-support system, not a replacement for security analysts.

I built Linux and Microsoft 365 checks to compare declared security configuration with account-level exceptions and runtime state. The checks looked beyond whether a policy existed and tested whether it was actually enforced. The main result was that configured and enforced were not always the same.

I studied whether standard toxicity detectors capture the kinds of harmful language that appear in academic peer review. Using nearly 1000 ICLR 2023 reviews, I compared GPT-based measures of emotive tone, lack of constructive feedback, personal attacks, and speculation with VADER sentiment scores and Perspective web-toxicity scores. Pearson and Spearman correlations showed that these academic-toxicity measures aligned more closely with general negative sentiment than with conventional web toxicity, suggesting that harmful academic feedback has context-specific characteristics that standard toxicity detectors may miss.