README
This commit is contained in:
196
README.md
196
README.md
@@ -1,120 +1,120 @@
|
||||
# Log ingest
|
||||
# log_ingest
|
||||
|
||||
This repo is created with cargo generate --git https://github.com/skanehira/rust-cli-template (start with cargo install cargo-generate )
|
||||
A Rust CLI tool for loading log files into a SQLite database for analysis.
|
||||
|
||||
## Overview
|
||||
|
||||
This repository serves as a template for quickly bootstrapping Rust command-line
|
||||
interface (CLI) applications using `cargo-generate`. It provides a minimal yet
|
||||
comprehensive foundation with the following features:
|
||||
Parses application logs containing signature messages and loads them into SQLite for querying. Designed to handle large log volumes (10GB+ per day) with batched inserts and efficient parsing.
|
||||
|
||||
- CLI argument parsing using [clap](https://github.com/clap-rs/clap) with derive
|
||||
macros
|
||||
- GitHub Actions workflow for CI/CD
|
||||
- Code coverage reporting with [octocov](https://github.com/k1LoW/octocov)
|
||||
- Automatic benchmark result visualization and deployment with
|
||||
[github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark)
|
||||
- Security audit checks for dependencies
|
||||
- Automated release workflow for publishing
|
||||
- Automated dependency updates with Dependabot
|
||||
## Features
|
||||
|
||||
## Project Structure
|
||||
- Parse `signature:` messages extracting app info, device details, and feature flags
|
||||
- Support for both plain `.log` and gzip compressed `.log.gz` files
|
||||
- File discovery by date range using `YYYY/mm/dd` directory structure
|
||||
- Batched inserts for performance with large files
|
||||
- Indexed columns (`session_id`, `version`) for efficient queries
|
||||
- Extensible parser architecture for adding new message types
|
||||
|
||||
Generated projects will have the following structure:
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cargo build --release
|
||||
```
|
||||
.
|
||||
├── .github/ # GitHub Actions workflows
|
||||
│ ├── workflows/ # CI/CD workflows for testing, benchmarking, and releasing
|
||||
│ │ ├── ci.yml # Main CI workflow (tests, linting, coverage)
|
||||
│ │ ├── audit.yml # Security audit workflow
|
||||
│ │ └── release.yml # Release automation workflow
|
||||
│ └── dependabot.yaml # Automated dependency update configuration file
|
||||
├── benches/ # Benchmark code (requires nightly Rust)
|
||||
├── src/ # Source code
|
||||
├── .gitignore # Git ignore file
|
||||
├── .octocov.yml # Code coverage configuration
|
||||
├── goreleasser.yaml # GoReleaser configuration file for cross-platform builds and distribution
|
||||
├── Cargo.toml # Project manifest
|
||||
├── Cargo.lock # Dependency lock file
|
||||
└── rust-toolchain.toml # Rust toolchain configuration
|
||||
```
|
||||
|
||||
## Benchmark visualization
|
||||
|
||||
The benchmark results are automatically deployed to GitHub Pages for easy
|
||||
visualization and performance tracking. You need to create a `gh-pages` branch
|
||||
in your repository before first push.
|
||||
|
||||
<img width="1165" alt="image" src="https://github.com/user-attachments/assets/333631e2-dee0-48f9-bc8e-d72c583857de" />
|
||||
|
||||
<img width="874" alt="image" src="https://github.com/user-attachments/assets/6a07ea77-1294-422f-abd6-cb3e4281c26e" />
|
||||
|
||||
## Coverage
|
||||
|
||||
This project uses [octocov](https://github.com/k1LoW/octocov) to measure code
|
||||
coverage. During CI execution, coverage reports are automatically generated and
|
||||
displayed as comments on PRs or commits. The coverage history is also tracked,
|
||||
allowing you to see changes over time.
|
||||
|
||||
The coverage reports are deployed to GitHub Pages for easy visualization.
|
||||
Coverage information can also be displayed in the README as a badge.
|
||||
|
||||
<img width="936" alt="image" src="https://github.com/user-attachments/assets/8471d58a-06b3-4fd5-85e6-916959704c69" />
|
||||
|
||||
The detailed configuration for octocov is managed in the `.octocov.yml` file.
|
||||
|
||||
## Usage
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- [cargo-generate](https://github.com/cargo-generate/cargo-generate)
|
||||
- [gh](https://github.com/cli/cli)
|
||||
|
||||
### Creating a New Project
|
||||
|
||||
Create a new project using this template:
|
||||
### Process a single file
|
||||
|
||||
```bash
|
||||
cargo generate --git https://github.com/skanehira/rust-cli-template.git
|
||||
log_ingest --file /path/to/logs.log --output output.db
|
||||
```
|
||||
|
||||
Follow the prompts to customize your project.
|
||||
|
||||
### Running Tests
|
||||
### Process a date range
|
||||
|
||||
```bash
|
||||
log_ingest \
|
||||
--from 2026/01/20 \
|
||||
--to 2026/01/21 \
|
||||
--base-dir /var/log/myapp \
|
||||
--filename app.log \
|
||||
--output output.db
|
||||
```
|
||||
|
||||
The tool will look for files at `<base-dir>/YYYY/MM/DD/<filename>.gz` or `<base-dir>/YYYY/MM/DD/<filename>` for each day in the range.
|
||||
|
||||
### Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--file <PATH>` | Single log file to process |
|
||||
| `--from <DATE>` | Start date (YYYY/mm/dd) |
|
||||
| `--to <DATE>` | End date (YYYY/mm/dd) |
|
||||
| `--base-dir <PATH>` | Base directory containing log files |
|
||||
| `--filename <NAME>` | Log filename (e.g., `app.log`) |
|
||||
| `-o, --output <PATH>` | Output SQLite database path |
|
||||
| `--batch-size <N>` | Batch size for inserts (default: 10000) |
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE signature_entries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_id TEXT NOT NULL,
|
||||
timestamp TEXT NOT NULL,
|
||||
app TEXT NOT NULL,
|
||||
version TEXT NOT NULL,
|
||||
offline_login_usage INTEGER NOT NULL,
|
||||
is_password_autofill_enabled INTEGER NOT NULL,
|
||||
camera_roll_usage INTEGER NOT NULL,
|
||||
os TEXT NOT NULL,
|
||||
app_name TEXT NOT NULL,
|
||||
touch_id INTEGER NOT NULL,
|
||||
is_offline_login_enabled INTEGER NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
device TEXT NOT NULL,
|
||||
password_autofill_usage INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_session_id ON signature_entries(session_id);
|
||||
CREATE INDEX idx_version ON signature_entries(version);
|
||||
```
|
||||
|
||||
## Example Queries
|
||||
|
||||
```sql
|
||||
-- Percentage of users with password autofill enabled
|
||||
SELECT
|
||||
ROUND(100.0 * SUM(is_password_autofill_enabled) / COUNT(*), 2) as pct
|
||||
FROM signature_entries;
|
||||
|
||||
-- Count by app version
|
||||
SELECT version, COUNT(*) as cnt
|
||||
FROM signature_entries
|
||||
GROUP BY version
|
||||
ORDER BY cnt DESC;
|
||||
|
||||
-- Device breakdown
|
||||
SELECT device, COUNT(*) as cnt
|
||||
FROM signature_entries
|
||||
GROUP BY device;
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Build
|
||||
cargo build
|
||||
|
||||
# Run tests
|
||||
cargo test
|
||||
|
||||
# Format
|
||||
cargo fmt
|
||||
|
||||
# Lint
|
||||
cargo clippy
|
||||
```
|
||||
|
||||
### Running Benchmarks
|
||||
## License
|
||||
|
||||
Benchmarks require the nightly Rust channel:
|
||||
|
||||
```bash
|
||||
cargo +nightly bench
|
||||
```
|
||||
|
||||
### Release Process
|
||||
|
||||
This template includes an automated release workflow. Follow these steps to
|
||||
create a release:
|
||||
|
||||
1. Push a tag with your changes:
|
||||
```bash
|
||||
git tag v0.1.0 # Replace with the appropriate version number
|
||||
git push origin v0.1.0
|
||||
```
|
||||
|
||||
2. When the tag is pushed, the GitHub Actions `release.yml` workflow will
|
||||
automatically execute. This workflow:
|
||||
- Builds cross-platform binaries (Linux, macOS, Windows)
|
||||
- Creates a GitHub Release
|
||||
- Uploads binaries and changelog
|
||||
|
||||
The release configuration is managed in the `.github/workflows/release.yml` and
|
||||
`goreleasser.yaml` files.
|
||||
|
||||
---
|
||||
|
||||
Feel free to customize this template to fit your specific needs!
|
||||
MIT
|
||||
|
||||
Reference in New Issue
Block a user