log_ingest/README.md

# log_ingest

A Rust CLI tool for loading log files into a SQLite database for analysis.

## Overview

Parses application logs containing signature messages and loads them into SQLite for querying. Designed to handle large log volumes (10GB+ per day) with batched inserts and efficient parsing.

## Features

- Parse `signature:` messages extracting app info, device details, and feature flags
- Support for both plain `.log` and gzip compressed `.log.gz` files
- File discovery by date range using `YYYY/mm/dd` directory structure
- Batched inserts for performance with large files
- Indexed columns (`session_id`, `version`) for efficient queries
- Extensible parser architecture for adding new message types

## Installation

```bash
cargo build --release
```

## Usage

### Process a single file

```bash
log_ingest --file /path/to/logs.log --output output.db
```

### Process a date range

```bash
log_ingest \
  --from 2026/01/20 \
  --to 2026/01/21 \
  --base-dir /var/log/myapp \
  --filename app.log \
  --output output.db
```

The tool will look for files at `<base-dir>/YYYY/MM/DD/<filename>.gz` or `<base-dir>/YYYY/MM/DD/<filename>` for each day in the range.

### Options

| Option | Description |
|--------|-------------|
| `--file <PATH>` | Single log file to process |
| `--from <DATE>` | Start date (YYYY/mm/dd) |
| `--to <DATE>` | End date (YYYY/mm/dd) |
| `--base-dir <PATH>` | Base directory containing log files |
| `--filename <NAME>` | Log filename (e.g., `app.log`) |
| `-o, --output <PATH>` | Output SQLite database path |
| `--batch-size <N>` | Batch size for inserts (default: 10000) |

## Database Schema

```sql
CREATE TABLE signature_entries (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT NOT NULL,
    timestamp TEXT NOT NULL,
    app TEXT NOT NULL,
    version TEXT NOT NULL,
    offline_login_usage INTEGER NOT NULL,
    is_password_autofill_enabled INTEGER NOT NULL,
    camera_roll_usage INTEGER NOT NULL,
    os TEXT NOT NULL,
    app_name TEXT NOT NULL,
    touch_id INTEGER NOT NULL,
    is_offline_login_enabled INTEGER NOT NULL,
    model TEXT NOT NULL,
    device TEXT NOT NULL,
    password_autofill_usage INTEGER NOT NULL
);

CREATE INDEX idx_session_id ON signature_entries(session_id);
CREATE INDEX idx_version ON signature_entries(version);
```

## Example Queries

```sql
-- Percentage of users with password autofill enabled
SELECT
    ROUND(100.0 * SUM(is_password_autofill_enabled) / COUNT(*), 2) as pct
FROM signature_entries;

-- Count by app version
SELECT version, COUNT(*) as cnt
FROM signature_entries
GROUP BY version
ORDER BY cnt DESC;

-- Device breakdown
SELECT device, COUNT(*) as cnt
FROM signature_entries
GROUP BY device;
```

## Development

```bash
# Build
cargo build

# Run tests
cargo test

# Format
cargo fmt

# Lint
cargo clippy
```

## License

MIT