Skip to content
FinFocus Documentation

E2E Testing Guide

Overview

Section titled “Overview”

This guide details how to set up and run end-to-end (E2E) tests for FinFocus to ensure the system works correctly with real cloud resources. E2E tests validate the complete cost calculation pipeline using actual AWS services.

Table of Contents

Section titled “Table of Contents”

Prerequisites

Section titled “Prerequisites”

Before running end-to-end tests, ensure your environment meets the following requirements:

  • Go 1.26.4+: Required for building the core and plugins.
  • AWS Credentials: A valid AWS account with read permissions (for Cost Explorer) and resource creation permissions (for infrastructure tests).
  • FinFocus Core: Installed locally.

Quick Start

Section titled “Quick Start”

  1. Install Required Plugins The E2E tests rely on the aws-public plugin for fallback pricing and aws-costexplorer for actual billing data validation.

    Terminal window
    # Install public plugin
    finfocus plugin install github.com/rshade/finfocus-plugin-aws-public
    

    # (Optional) Install Cost Explorer plugin if testing actual costs
    finfocus plugin install github.com/rshade/finfocus-plugin-aws-costexplorer

  2. Configure Environment Export your AWS credentials.

    Terminal window
    export AWS_ACCESS_KEY_ID="testing-key"
    export AWS_SECRET_ACCESS_KEY="testing-secret"
    export AWS_REGION="us-east-1"

  3. Run Tests Execute the E2E test suite using the Makefile target.

    Terminal window
    make test-e2e

  4. View Results Test results are summarized in JSON format.

    Terminal window
    cat test-results/e2e-summary.json

Running Tests

Section titled “Running Tests”

The make test-e2e command orchestrates the entire testing process. It:

  1. Builds the core binary and plugins.
  2. Sets up a temporary Pulumi project.
  3. Runs the test suite which interacts with AWS.
  4. Cleans up resources after tests complete.

Custom Test Arguments

Section titled “Custom Test Arguments”

You can pass custom arguments to the test runner using the TEST_ARGS variable:

Terminal window
make test-e2e TEST_ARGS="-v -run TestProjectedCosts"

Test Scenarios

Section titled “Test Scenarios”

The E2E suite covers several critical scenarios to ensure system reliability.

1. Projected Costs

Section titled “1. Projected Costs”

Calculates estimated monthly costs based on the Pulumi plan before any resources are deployed. This validates the aws-public plugin’s pricing data lookup.

2. Actual Costs

Section titled “2. Actual Costs”

After resources have been running (or using historical data), this scenario queries the AWS Cost Explorer API via the aws-costexplorer plugin to verify actual spending matches expectations.

3. Cost Validation

Section titled “3. Cost Validation”

Compares projected costs against actual costs (where applicable) to ensure accuracy within a defined tolerance.

4. Cleanup Verification

Section titled “4. Cleanup Verification”

Ensures that all resources created during the test are destroyed, preventing accidental costs.

Section titled “Related Documentation”

Troubleshooting

Section titled “Troubleshooting”

”Plugin not found”

Section titled “”Plugin not found””

Ensure you ran finfocus plugin install before testing. You can list installed plugins with:

Terminal window
finfocus plugin list

“Access Denied” or Credential Errors

Section titled ““Access Denied” or Credential Errors”

Verify your AWS credentials have the necessary permissions. The test suite requires permissions to create/destroy EC2 instances, S3 buckets, and query Cost Explorer.

Test Timeouts

Section titled “Test Timeouts”

E2E tests can be slow due to cloud resource provisioning. If tests timeout, try increasing the timeout duration:

Terminal window
go test -timeout 30m ./test/e2e/...

Subprocess Hangs (Leaked Plugin Processes)

Section titled “Subprocess Hangs (Leaked Plugin Processes)”

Plugin subprocesses can outlive a test and hold inherited stdout/stderr pipes open, causing cmd.Wait() to block indefinitely. All E2E command execution uses a newCommand helper that sets exec.Cmd.WaitDelay (30 seconds) so a leaked subprocess fails fast with real output instead of hanging the runner.

If a test still hangs, check for orphaned plugin processes:

Terminal window
# List any lingering finfocus plugin processes
ps aux | grep finfocus-plugin

Windows Cross-Platform Notes

Section titled “Windows Cross-Platform Notes”

The E2E and unit test suite is designed to run on Linux, macOS, and Windows. Key platform-aware behaviors:

  • Plugin binaries use the .exe suffix on Windows (handled automatically by exec.LookPath).
  • Home directory detection uses USERPROFILE on Windows (os.UserHomeDir resolves correctly).
  • Golden file comparisons normalize CRLF line endings on Windows checkouts.
  • Permission-bit assertions (Unix chmod modes) are skipped on Windows where they are not applicable.

Integration Test Log Isolation

Section titled “Integration Test Log Isolation”

Trace integration tests set an isolated FINFOCUS_HOME directory (via t.TempDir()) to prevent tests from reading or writing the real ~/.finfocus/ directory. Log assertions inspect both the captured stderr and the log file at $FINFOCUS_HOME/logs/finfocus.log, because the CLI routes structured logs to the log file by default rather than stderr.