docs
Search docsCmd K
Docs/operations

Operations Docs

Phase 3 QA Runbook

/operations/phase3-qa-runbook

Phase 3 QA Runbook

Scope: publisher portal, webhook delivery, API key management, analytics Last updated: 2026-04-12

목적

이 문서는 로컬 또는 staging 환경에서 Phase 3 핵심 흐름을 반복 검증하기 위한 seed + QA 시나리오를 정리한다.

검증 대상:

  • Publisher module metadata
  • API key issuance and revoke
  • Webhook creation and delivery logs
  • Analytics summary and daily chart

1. 준비

필수:

  • membloc-app-engine DB 마이그레이션 완료
  • membloc-developer-portal 로컬 환경값 설정 완료
  • DATABASE_URL 사용 가능

권장:

  • backend: http://localhost:8080
  • portal: http://localhost:3000

주의:

  • 위 localhost URL은 로컬 QA 예시다.
  • public production 또는 staging endpoint로 간주하면 안 된다.

2. Seed 데이터 주입

다음 스크립트는 반복 실행 가능한 demo 데이터를 넣는다.

cd /Users/jeongmokhwang/jmcunst/project/membloc-project/membloc-app-engine
SEED_REQUESTED_BY=<human-or-ticket-id> \
SEED_RUN_MARKER=<unique-non-secret-marker> \
DRY_RUN=false \
SEED_CONFIRM=seed-phase3-demo \
DATABASE_URL=postgres://... \
./scripts/seed_phase3_demo.sh

SEED_RUN_MARKER is an audit marker, not a secret. Keep it ticket/date based and never paste tokens, passwords, or production DSNs into it.

Seed 결과:

  • approved publisher 1개
  • family 1개
  • published server module 1개
  • screenshots / privacy / support metadata
  • API key 1개
  • webhook 2개
  • delivery log 샘플
  • 최근 14일 module_daily_stats

데모 기준 module key:

  • com.membloc.budget-pro

3. 수동 QA 시나리오

A. Dashboard analytics

확인 위치:

  • portal /dashboard

기대 결과:

  • publisher summary 카드 표시
  • installs / uninstalls 차트 표시
  • active installs, period installs, errors 숫자 표시

실패 시 확인:

  • /api/publisher/analytics/overview?days=14
  • module_daily_stats seed 여부

B. Module detail analytics

확인 위치:

  • portal /modules/com.membloc.budget-pro

기대 결과:

  • review readiness 패널 표시
  • uploaded assets 표시
  • API key 패널 표시
  • webhook 패널 표시
  • installs / uninstalls 차트 표시

실패 시 확인:

  • /api/publisher/modules/com.membloc.budget-pro/analytics
  • /api/publisher/modules/com.membloc.budget-pro/analytics/daily

C. API key flow

시나리오:

  1. module detail에서 issue api key
  2. plaintext key 1회 표시 확인
  3. 페이지 refresh 후 plaintext 재노출 안 되는지 확인
  4. revoke 버튼으로 제거

기대 결과:

  • DB에는 hash만 저장
  • UI에는 plaintext 재조회 불가
  • revoke 후 목록에서 제거

D. Webhook management

시나리오:

  1. module detail에서 webhook 생성
  2. secret 1회 표시 확인
  3. 기존 seeded webhook logs 확인
  4. delete 동작 확인

기대 결과:

  • newly created secret only once
  • existing logs 표시
  • delete 후 목록 제거

E. Runtime analytics increment

시나리오:

  1. runtime token 준비
  2. /api/runtime/:moduleKey/families/:familyId/data GET/POST/PUT/DELETE 실행
  3. 일부 호출은 의도적으로 실패시킴
    • invalid body
    • wrong permission
  4. analytics 재확인

기대 결과:

  • 모든 runtime endpoint hit는 api_calls 증가
  • 4xx/5xx는 errors 증가

4. 빠른 API 확인 쿼리

아래 curl 예시는 모두 local QA 기준 URL을 사용한다. 운영/preview URL이 따로 확정되지 않았다면 그대로 외부 공유하거나 public endpoint처럼 안내하지 않는다.

publisher analytics:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8080/api/publisher/analytics/overview?days=14"

module analytics summary:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8080/api/publisher/modules/com.membloc.budget-pro/analytics?days=14"

module analytics daily:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8080/api/publisher/modules/com.membloc.budget-pro/analytics/daily?days=14"

module api keys:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8080/api/publisher/modules/com.membloc.budget-pro/api-keys"

module webhooks:

curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8080/api/publisher/modules/com.membloc.budget-pro/webhooks"

admin reject reason required:

# expect 400 invalid input when reason is missing
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}' \
  "http://localhost:8080/api/admin/modules/com.membloc.budget-pro/reject"

# expect 200 when reason is present
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason":"Insufficient privacy policy details"}' \
  "http://localhost:8080/api/admin/modules/com.membloc.budget-pro/reject"

5. 테스트용 runtime 호출 예시

runtime list:

curl -H "Authorization: Bearer $MODULE_TOKEN" \
  "http://localhost:8080/api/runtime/com.membloc.budget-pro/families/fam-demo-membloc/data"

runtime set:

curl -X POST \
  -H "Authorization: Bearer $MODULE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"key":"budget/monthly","value":{"month":"2026-04","budget":3200000}}' \
  "http://localhost:8080/api/runtime/com.membloc.budget-pro/families/fam-demo-membloc/data"

expected invalid request:

curl -X POST \
  -H "Authorization: Bearer $MODULE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}' \
  "http://localhost:8080/api/runtime/com.membloc.budget-pro/families/fam-demo-membloc/data"

6. QA 완료 기준

  • seed 스크립트 실행 후 portal dashboard/module detail이 비어 있지 않음
  • API key 발급/폐기 가능
  • webhook 생성/삭제 가능
  • webhook delivery logs 확인 가능
  • runtime endpoint hit 이후 analytics 숫자 변화 확인 가능

7. 남은 공백

현재 runbook은 다음을 아직 자동화하지 않는다.

  • Firebase login token 발급 자동화
  • webhook 실제 외부 endpoint mock server
  • scheduled retry completion 검증
  • analytics api_calls/errors에 대한 장기 부하 시나리오

다음 단계에서 고려:

  • local webhook mock receiver script
  • runtime token issue helper script
  • end-to-end smoke script

8. Authenticated request lifecycle smoke handoff

The request lifecycle smoke is intentionally separate from the Phase 3 publisher/admin checks. Run it only when a disposable signed-in user/session and matching disposable family/channel/message fixture already exist in a local or explicitly non-production target.

Required shell-only inputs for scripts/curl_smoke.sh:

  • AUTH_TOKEN or TOKEN: short-lived Firebase ID token for the disposable user. Prefer read -rsp 'Firebase ID token: ' AUTH_TOKEN; echo so the token is not echoed, and keep it out of tracked .env, markdown, screenshots, tickets, and shell transcripts. TOKEN overrides AUTH_TOKEN if both are set.
  • BASE: target engine URL. Use http://localhost:8080 by default. For a disposable stage target, set ALLOW_NON_LOCAL_REQUEST_SMOKE=1 only after confirming it is not production and contains no shared user data.
  • REQUEST_SMOKE_ONLY=1 and REQUEST_SMOKE=1: select the guarded request path.
  • REQUEST_SMOKE_MARKER=request-smoke-YYYYMMDD: cleanup marker. Keep this exact shape; put worker/run uniqueness in IDs, not the marker.
  • REQUEST_SMOKE_FAMILY_ID, REQUEST_SMOKE_CHANNEL_ID, and REQUEST_SMOKE_MESSAGE_ID: disposable fixture IDs readable by the token user.
  • Optional: REQUEST_SMOKE_EXPECT_REQUESTED=1 for an existing requested marker; REQUEST_SMOKE_MUTATE=1 REQUEST_SMOKE_CLEANUP=1 for mutation-and-cleanup; REQUEST_SMOKE_DUE_AT, REQUEST_SMOKE_ASSIGNEE_UID, and REQUEST_SMOKE_STATUS_REPLY for mutation metadata.

Safe one-shot pattern:

# From the membloc-project checkout:
cd membloc-app-engine
read -rsp 'Firebase ID token for disposable user: ' AUTH_TOKEN; echo
AUTH_TOKEN="$AUTH_TOKEN" \
REQUEST_SMOKE_ONLY=1 \
REQUEST_SMOKE=1 \
REQUEST_SMOKE_MARKER=request-smoke-YYYYMMDD \
REQUEST_SMOKE_FAMILY_ID=<disposable-family-id> \
REQUEST_SMOKE_CHANNEL_ID=<disposable-channel-id> \
REQUEST_SMOKE_MESSAGE_ID=<disposable-message-id> \
BASE=http://localhost:8080 \
./scripts/curl_smoke.sh
unset AUTH_TOKEN TOKEN ADMIN_TOKEN REQUEST_SMOKE_MARKER REQUEST_SMOKE_FAMILY_ID \
  REQUEST_SMOKE_CHANNEL_ID REQUEST_SMOKE_MESSAGE_ID REQUEST_SMOKE_DUE_AT \
  REQUEST_SMOKE_ASSIGNEE_UID REQUEST_SMOKE_STATUS_REPLY

Terminal history caveats:

  • Do not type a real token directly into a command line; many shells persist that line. Use read -rsp, a temporary environment variable in the current shell, or a one-shot command whose secret value is expanded from memory.
  • A local untracked .env.local may document placeholder variable names only; never commit token/session values. Prefer updating .env.example with placeholders if discoverability is needed.
  • Firebase ID tokens are short-lived. If the smoke returns 401, refresh the signed-in disposable session and obtain a new token instead of saving the old token anywhere.

Evidence and cleanup:

  • Capture only non-secret evidence: marker, disposable target name, sanitized command name, HTTP result summary, and post-cleanup zero-count queries.
  • Cleanup is complete only when the requested marker, message, channel, family membership/family, disposable user, and any request-detail state rows matching the disposable IDs are removed.
  • If cleanup cannot be proven, leave the live smoke status as blocked/incomplete and record the exact blocker without pasting tokens or real user data.

SDK assets

Static SDK files

JavaScript runtime and TypeScript definitions are served from this docs app.
membloc-sdk.jsmembloc-sdk.d.ts