diff --git a/README.md b/README.md index 013aa3c..86e8574 100644 --- a/README.md +++ b/README.md @@ -1,120 +1,120 @@ -# Log ingest +# log_ingest -This repo is created with cargo generate --git https://github.com/skanehira/rust-cli-template (start with cargo install cargo-generate ) +A Rust CLI tool for loading log files into a SQLite database for analysis. ## Overview -This repository serves as a template for quickly bootstrapping Rust command-line -interface (CLI) applications using `cargo-generate`. It provides a minimal yet -comprehensive foundation with the following features: +Parses application logs containing signature messages and loads them into SQLite for querying. Designed to handle large log volumes (10GB+ per day) with batched inserts and efficient parsing. -- CLI argument parsing using [clap](https://github.com/clap-rs/clap) with derive - macros -- GitHub Actions workflow for CI/CD - - Code coverage reporting with [octocov](https://github.com/k1LoW/octocov) - - Automatic benchmark result visualization and deployment with - [github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark) - - Security audit checks for dependencies - - Automated release workflow for publishing - - Automated dependency updates with Dependabot +## Features -## Project Structure +- Parse `signature:` messages extracting app info, device details, and feature flags +- Support for both plain `.log` and gzip compressed `.log.gz` files +- File discovery by date range using `YYYY/mm/dd` directory structure +- Batched inserts for performance with large files +- Indexed columns (`session_id`, `version`) for efficient queries +- Extensible parser architecture for adding new message types -Generated projects will have the following structure: +## Installation +```bash +cargo build --release ``` -. -├── .github/ # GitHub Actions workflows -│ ├── workflows/ # CI/CD workflows for testing, benchmarking, and releasing -│ │ ├── ci.yml # Main CI workflow (tests, linting, coverage) -│ │ ├── audit.yml # Security audit workflow -│ │ └── release.yml # Release automation workflow -│ └── dependabot.yaml # Automated dependency update configuration file -├── benches/ # Benchmark code (requires nightly Rust) -├── src/ # Source code -├── .gitignore # Git ignore file -├── .octocov.yml # Code coverage configuration -├── goreleasser.yaml # GoReleaser configuration file for cross-platform builds and distribution -├── Cargo.toml # Project manifest -├── Cargo.lock # Dependency lock file -└── rust-toolchain.toml # Rust toolchain configuration -``` - -## Benchmark visualization - -The benchmark results are automatically deployed to GitHub Pages for easy -visualization and performance tracking. You need to create a `gh-pages` branch -in your repository before first push. - -image - -image - -## Coverage - -This project uses [octocov](https://github.com/k1LoW/octocov) to measure code -coverage. During CI execution, coverage reports are automatically generated and -displayed as comments on PRs or commits. The coverage history is also tracked, -allowing you to see changes over time. - -The coverage reports are deployed to GitHub Pages for easy visualization. -Coverage information can also be displayed in the README as a badge. - -image - -The detailed configuration for octocov is managed in the `.octocov.yml` file. ## Usage -### Prerequisites - -- [cargo-generate](https://github.com/cargo-generate/cargo-generate) -- [gh](https://github.com/cli/cli) - -### Creating a New Project - -Create a new project using this template: +### Process a single file ```bash -cargo generate --git https://github.com/skanehira/rust-cli-template.git +log_ingest --file /path/to/logs.log --output output.db ``` -Follow the prompts to customize your project. - -### Running Tests +### Process a date range ```bash +log_ingest \ + --from 2026/01/20 \ + --to 2026/01/21 \ + --base-dir /var/log/myapp \ + --filename app.log \ + --output output.db +``` + +The tool will look for files at `/YYYY/MM/DD/.gz` or `/YYYY/MM/DD/` for each day in the range. + +### Options + +| Option | Description | +|--------|-------------| +| `--file ` | Single log file to process | +| `--from ` | Start date (YYYY/mm/dd) | +| `--to ` | End date (YYYY/mm/dd) | +| `--base-dir ` | Base directory containing log files | +| `--filename ` | Log filename (e.g., `app.log`) | +| `-o, --output ` | Output SQLite database path | +| `--batch-size ` | Batch size for inserts (default: 10000) | + +## Database Schema + +```sql +CREATE TABLE signature_entries ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + session_id TEXT NOT NULL, + timestamp TEXT NOT NULL, + app TEXT NOT NULL, + version TEXT NOT NULL, + offline_login_usage INTEGER NOT NULL, + is_password_autofill_enabled INTEGER NOT NULL, + camera_roll_usage INTEGER NOT NULL, + os TEXT NOT NULL, + app_name TEXT NOT NULL, + touch_id INTEGER NOT NULL, + is_offline_login_enabled INTEGER NOT NULL, + model TEXT NOT NULL, + device TEXT NOT NULL, + password_autofill_usage INTEGER NOT NULL +); + +CREATE INDEX idx_session_id ON signature_entries(session_id); +CREATE INDEX idx_version ON signature_entries(version); +``` + +## Example Queries + +```sql +-- Percentage of users with password autofill enabled +SELECT + ROUND(100.0 * SUM(is_password_autofill_enabled) / COUNT(*), 2) as pct +FROM signature_entries; + +-- Count by app version +SELECT version, COUNT(*) as cnt +FROM signature_entries +GROUP BY version +ORDER BY cnt DESC; + +-- Device breakdown +SELECT device, COUNT(*) as cnt +FROM signature_entries +GROUP BY device; +``` + +## Development + +```bash +# Build +cargo build + +# Run tests cargo test + +# Format +cargo fmt + +# Lint +cargo clippy ``` -### Running Benchmarks +## License -Benchmarks require the nightly Rust channel: - -```bash -cargo +nightly bench -``` - -### Release Process - -This template includes an automated release workflow. Follow these steps to -create a release: - -1. Push a tag with your changes: - ```bash - git tag v0.1.0 # Replace with the appropriate version number - git push origin v0.1.0 - ``` - -2. When the tag is pushed, the GitHub Actions `release.yml` workflow will - automatically execute. This workflow: - - Builds cross-platform binaries (Linux, macOS, Windows) - - Creates a GitHub Release - - Uploads binaries and changelog - -The release configuration is managed in the `.github/workflows/release.yml` and -`goreleasser.yaml` files. - ---- - -Feel free to customize this template to fit your specific needs! +MIT