eghuro/log_ingest
1
0
Fork 0
Code Pull Requests Releases Activity

README

Browse Source
This commit is contained in:
Alexandr Mansurov
2026-01-21 22:37:13 +01:00
parent 30cc4fae01
commit f0a390d802
1 changed files with 98 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files

196
README.md
View File

@@ -1,120 +1,120 @@
# Log ingest # log_ingest
This repo is created with cargo generate --git https://github.com/skanehira/rust-cli-template (start with cargo install cargo-generate ) A Rust CLI tool for loading log files into a SQLite database for analysis.
## Overview ## Overview
This repository serves as a template for quickly bootstrapping Rust command-line Parses application logs containing signature messages and loads them into SQLite for querying. Designed to handle large log volumes (10GB+ per day) with batched inserts and efficient parsing.
interface (CLI) applications using `cargo-generate`. It provides a minimal yet
comprehensive foundation with the following features:
- CLI argument parsing using [clap](https://github.com/clap-rs/clap) with derive ## Features
macros
- GitHub Actions workflow for CI/CD
- Code coverage reporting with [octocov](https://github.com/k1LoW/octocov)
- Automatic benchmark result visualization and deployment with
[github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark)
- Security audit checks for dependencies
- Automated release workflow for publishing
- Automated dependency updates with Dependabot
## Project Structure - Parse `signature:` messages extracting app info, device details, and feature flags
- Support for both plain `.log` and gzip compressed `.log.gz` files
- File discovery by date range using `YYYY/mm/dd` directory structure
- Batched inserts for performance with large files
- Indexed columns (`session_id`, `version`) for efficient queries
- Extensible parser architecture for adding new message types
Generated projects will have the following structure: ## Installation
```bash
cargo build --release
``` ```
.
├── .github/ # GitHub Actions workflows
│ ├── workflows/ # CI/CD workflows for testing, benchmarking, and releasing
│ │ ├── ci.yml # Main CI workflow (tests, linting, coverage)
│ │ ├── audit.yml # Security audit workflow
│ │ └── release.yml # Release automation workflow
│ └── dependabot.yaml # Automated dependency update configuration file
├── benches/ # Benchmark code (requires nightly Rust)
├── src/ # Source code
├── .gitignore # Git ignore file
├── .octocov.yml # Code coverage configuration
├── goreleasser.yaml # GoReleaser configuration file for cross-platform builds and distribution
├── Cargo.toml # Project manifest
├── Cargo.lock # Dependency lock file
└── rust-toolchain.toml # Rust toolchain configuration
```
## Benchmark visualization
The benchmark results are automatically deployed to GitHub Pages for easy
visualization and performance tracking. You need to create a `gh-pages` branch
in your repository before first push.
<img width="1165" alt="image" src="https://github.com/user-attachments/assets/333631e2-dee0-48f9-bc8e-d72c583857de" />
<img width="874" alt="image" src="https://github.com/user-attachments/assets/6a07ea77-1294-422f-abd6-cb3e4281c26e" />
## Coverage
This project uses [octocov](https://github.com/k1LoW/octocov) to measure code
coverage. During CI execution, coverage reports are automatically generated and
displayed as comments on PRs or commits. The coverage history is also tracked,
allowing you to see changes over time.
The coverage reports are deployed to GitHub Pages for easy visualization.
Coverage information can also be displayed in the README as a badge.
<img width="936" alt="image" src="https://github.com/user-attachments/assets/8471d58a-06b3-4fd5-85e6-916959704c69" />
The detailed configuration for octocov is managed in the `.octocov.yml` file.
## Usage ## Usage
### Prerequisites ### Process a single file
- [cargo-generate](https://github.com/cargo-generate/cargo-generate)
- [gh](https://github.com/cli/cli)
### Creating a New Project
Create a new project using this template:
```bash ```bash
cargo generate --git https://github.com/skanehira/rust-cli-template.git log_ingest --file /path/to/logs.log --output output.db
``` ```
Follow the prompts to customize your project. ### Process a date range
### Running Tests
```bash ```bash
log_ingest \
--from 2026/01/20 \
--to 2026/01/21 \
--base-dir /var/log/myapp \
--filename app.log \
--output output.db
```
The tool will look for files at `<base-dir>/YYYY/MM/DD/<filename>.gz` or `<base-dir>/YYYY/MM/DD/<filename>` for each day in the range.
### Options
| Option | Description |
|--------|-------------|
| `--file <PATH>` | Single log file to process |
| `--from <DATE>` | Start date (YYYY/mm/dd) |
| `--to <DATE>` | End date (YYYY/mm/dd) |
| `--base-dir <PATH>` | Base directory containing log files |
| `--filename <NAME>` | Log filename (e.g., `app.log`) |
| `-o, --output <PATH>` | Output SQLite database path |
| `--batch-size <N>` | Batch size for inserts (default: 10000) |
## Database Schema
```sql
CREATE TABLE signature_entries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id TEXT NOT NULL,
timestamp TEXT NOT NULL,
app TEXT NOT NULL,
version TEXT NOT NULL,
offline_login_usage INTEGER NOT NULL,
is_password_autofill_enabled INTEGER NOT NULL,
camera_roll_usage INTEGER NOT NULL,
os TEXT NOT NULL,
app_name TEXT NOT NULL,
touch_id INTEGER NOT NULL,
is_offline_login_enabled INTEGER NOT NULL,
model TEXT NOT NULL,
device TEXT NOT NULL,
password_autofill_usage INTEGER NOT NULL
);
CREATE INDEX idx_session_id ON signature_entries(session_id);
CREATE INDEX idx_version ON signature_entries(version);
```
## Example Queries
```sql
-- Percentage of users with password autofill enabled
SELECT
ROUND(100.0 * SUM(is_password_autofill_enabled) / COUNT(*), 2) as pct
FROM signature_entries;
-- Count by app version
SELECT version, COUNT(*) as cnt
FROM signature_entries
GROUP BY version
ORDER BY cnt DESC;
-- Device breakdown
SELECT device, COUNT(*) as cnt
FROM signature_entries
GROUP BY device;
```
## Development
```bash
# Build
cargo build
# Run tests
cargo test cargo test
# Format
cargo fmt
# Lint
cargo clippy
``` ```
### Running Benchmarks ## License
Benchmarks require the nightly Rust channel: MIT
```bash
cargo +nightly bench
```
### Release Process
This template includes an automated release workflow. Follow these steps to
create a release:
1. Push a tag with your changes:
```bash
git tag v0.1.0 # Replace with the appropriate version number
git push origin v0.1.0
```
2. When the tag is pushed, the GitHub Actions `release.yml` workflow will
automatically execute. This workflow:
- Builds cross-platform binaries (Linux, macOS, Windows)
- Creates a GitHub Release
- Uploads binaries and changelog
The release configuration is managed in the `.github/workflows/release.yml` and
`goreleasser.yaml` files.
---
Feel free to customize this template to fit your specific needs!