Sergey Krylov
60e456fbb5
Three optimizations: 1. Merge getBondData + getBondMarketData into single batch call (same endpoint, parse both tables) 2. Remove redundant getSecurityDescription for shortName (shortName already in market data responses) 3. Batch by market: 1 call for all shares, 1 call for all bonds (instead of N individual calls) Before: 298 API calls for 104 positions -> ~29.8s After: 2 API calls for 104 positions -> ~0.3s
366 lines
12 KiB
Markdown
366 lines
12 KiB
Markdown
# Portfolio Enricher Optimization — Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) for tracking.
|
|
|
|
**Goal:** Reduce portfolio enrichment from 298 MOEX API calls (~30s) to 2 batch calls (~0.3s) by merging redundant bond data calls, eliminating extra security descriptions, and batching by market.
|
|
|
|
**Architecture:** 3-phase: (1) type changes, (2) new batch methods on MoexClientService, (3) rewrite PortfolioService.enrichPositions to use batch + remove redundant calls.
|
|
|
|
**Tech Stack:** NestJS, TypeScript, MOEX ISS API, PQueue
|
|
|
|
---
|
|
|
|
### Task 1: Add types — `shortName` on share market data + `MoexBondPositionData` combined type
|
|
|
|
**Files:**
|
|
- Modify: `apps/backend/src/modules/moex-client/moex-client.types.ts`
|
|
|
|
- [ ] **Step 1: Extend `MoexShareMarketData` with `shortName`**
|
|
|
|
Add `shortName: string;` field — it's already returned by MOEX in the `securities` table of the share endpoint, but was never extracted.
|
|
|
|
- [ ] **Step 2: Add `MoexBondPositionData` combined type**
|
|
|
|
```typescript
|
|
export interface MoexBondPositionData {
|
|
secid: string;
|
|
boardid: string;
|
|
shortName: string;
|
|
price: number | null;
|
|
yieldToMaturity: number | null;
|
|
duration: number | null;
|
|
couponValue: number | null;
|
|
couponPercent: number | null;
|
|
nextCouponDate: string | null;
|
|
matDate: string | null;
|
|
accruedInt: number | null;
|
|
faceValue: number;
|
|
bid: number | null;
|
|
offer: number | null;
|
|
couponPeriod: number | null;
|
|
bondType: string | null;
|
|
offerDate: string | null;
|
|
}
|
|
```
|
|
|
|
This replaces the need for both `MoexBondData` + `MoexBondMarketData` — combined from a single endpoint response.
|
|
|
|
---
|
|
|
|
### Task 2: Add batch methods to MoexClientService
|
|
|
|
**Files:**
|
|
- Modify: `apps/backend/src/modules/moex-client/moex-client.service.ts`
|
|
|
|
- [ ] **Step 1: Add `getShareMarketDataBatch` method**
|
|
|
|
```typescript
|
|
async getShareMarketDataBatch(
|
|
secids: string[],
|
|
boardId = 'TQBR',
|
|
): Promise<MoexShareMarketData[]> {
|
|
if (secids.length === 0) return [];
|
|
const data = await this.request<Record<string, unknown>>(
|
|
`/engines/stock/markets/shares/securities`,
|
|
{ securities: secids.join(','), boards: boardId },
|
|
);
|
|
const securities = this.extractTable(data, 'securities');
|
|
const marketdata = this.extractTable(data, 'marketdata');
|
|
|
|
return secids.map((secid) => {
|
|
const sec = securities.find((r) => r.SECID === secid && r.BOARDID === boardId)
|
|
?? securities.find((r) => r.SECID === secid);
|
|
const mkt = marketdata.find((r) => r.SECID === secid && r.BOARDID === boardId)
|
|
?? marketdata.find((r) => r.SECID === secid);
|
|
|
|
return {
|
|
secid,
|
|
boardid: boardId,
|
|
shortName: (sec?.SHORTNAME as string) || '',
|
|
bid: mkt ? parseFloat((mkt.BID as string) || '') : null,
|
|
offer: mkt ? parseFloat((mkt.OFFER as string) || '') : null,
|
|
open: mkt ? parseFloat((mkt.OPEN as string) || '') : null,
|
|
low: mkt ? parseFloat((mkt.LOW as string) || '') : null,
|
|
high: mkt ? parseFloat((mkt.HIGH as string) || '') : null,
|
|
last: mkt
|
|
? parseFloat((mkt.LAST as string) || '')
|
|
: parseFloat((sec?.PREVPRICE as string) || ''),
|
|
lastChange: mkt ? parseFloat((mkt.LASTCHANGE as string) || '') : null,
|
|
lastChangePrcnt: mkt ? parseFloat((mkt.LASTCHANGEPRCNT as string) || '') : null,
|
|
volume: mkt ? parseInt((mkt.VOLTODAY as string) || '0', 10) : 0,
|
|
value: mkt ? parseFloat((mkt.VALTODAY as string) || '0') : 0,
|
|
waprice: mkt ? parseFloat((mkt.WAPRICE as string) || '') : null,
|
|
numtrades: mkt ? parseInt((mkt.NUMTRADES as string) || '0', 10) : 0,
|
|
issueCapitalization: mkt ? parseFloat((mkt.ISSUECAPITALIZATION as string) || '') : null,
|
|
tradingStatus: (mkt?.TRADINGSTATUS as string) || '',
|
|
updateTime: (mkt?.UPDATETIME as string) || '',
|
|
};
|
|
});
|
|
}
|
|
```
|
|
|
|
Key: uses existing `request()` method (rate-limited via PQueue). The `securities` param accepts comma-separated secids.
|
|
|
|
- [ ] **Step 2: Add `getBondPositionDataBatch` method**
|
|
|
|
```typescript
|
|
async getBondPositionDataBatch(
|
|
secids: string[],
|
|
boardId = 'TQCB',
|
|
): Promise<MoexBondPositionData[]> {
|
|
if (secids.length === 0) return [];
|
|
const data = await this.request<Record<string, unknown>>(
|
|
`/engines/stock/markets/bonds/securities`,
|
|
{ securities: secids.join(','), boards: boardId },
|
|
);
|
|
const securities = this.extractTable(data, 'securities');
|
|
const marketdata = this.extractTable(data, 'marketdata');
|
|
|
|
return secids.map((secid) => {
|
|
const bond =
|
|
securities.find((r) => r.SECID === secid && r.BOARDID === boardId && r.PREVWAPRICE != null) ||
|
|
securities.find((r) => r.SECID === secid && r.PREVWAPRICE != null) ||
|
|
securities.find((r) => r.SECID === secid);
|
|
const mkt =
|
|
marketdata.find((r) => r.SECID === secid && r.BOARDID === boardId && r.LAST != null) ||
|
|
marketdata.find((r) => r.LAST != null) ||
|
|
marketdata.find((r) => r.SECID === secid);
|
|
|
|
return {
|
|
secid,
|
|
boardid: boardId,
|
|
shortName: (bond?.SHORTNAME as string) || '',
|
|
price: mkt?.LAST != null ? parseFloat(mkt.LAST as string) : null,
|
|
yieldToMaturity: mkt?.YIELD != null ? parseFloat(mkt.YIELD as string) : null,
|
|
duration: mkt?.DURATION != null ? parseFloat(mkt.DURATION as string) : null,
|
|
couponValue: bond?.COUPONVALUE != null ? parseFloat(bond.COUPONVALUE as string) : null,
|
|
couponPercent: bond?.COUPONPERCENT != null ? parseFloat(bond.COUPONPERCENT as string) : null,
|
|
nextCouponDate: (bond?.NEXTCOUPON as string) || null,
|
|
matDate: (bond?.MATDATE as string) || null,
|
|
accruedInt: bond?.ACCRUEDINT != null ? parseFloat(bond.ACCRUEDINT as string) : null,
|
|
faceValue: parseFloat((bond?.FACEVALUE as string) || '1000'),
|
|
bid: mkt?.BID != null ? parseFloat(mkt.BID as string) : null,
|
|
offer: mkt?.OFFER != null ? parseFloat(mkt.OFFER as string) : null,
|
|
couponPeriod: parseInt((bond?.COUPONPERIOD as string) || '0', 10),
|
|
bondType: (bond?.BONDTYPE as string) || null,
|
|
offerDate: (bond?.OFFERDATE as string) || null,
|
|
};
|
|
});
|
|
}
|
|
```
|
|
|
|
This replaces `getBondData` + `getBondMarketData` with a single batch call that parses both tables.
|
|
|
|
- [ ] **Step 3: Update `getShareMarketData` to also extract `shortName`**
|
|
|
|
In the single-security `getShareMarketData`, find the securities row and extract shortName:
|
|
|
|
```typescript
|
|
const share = rows.find((r) => r.BOARDID === boardId);
|
|
return {
|
|
secid,
|
|
boardid: boardId,
|
|
shortName: (share?.SHORTNAME as string) || '', // NEW
|
|
bid: mkt ? parseFloat((mkt.BID as string) || '') : null,
|
|
// ... rest unchanged
|
|
};
|
|
```
|
|
|
|
- [ ] **Step 4: Run existing tests**
|
|
|
|
```bash
|
|
npx vitest run -w apps/backend
|
|
```
|
|
|
|
Expected: existing tests pass (no regressions).
|
|
|
|
---
|
|
|
|
### Task 3: Rewrite `enrichPositions` in PortfolioService
|
|
|
|
**Files:**
|
|
- Modify: `apps/backend/src/modules/portfolio/portfolio.service.ts`
|
|
|
|
- [ ] **Step 1: Rewrite `enrichPositions` to use batch + eliminate redundant calls**
|
|
|
|
Strategy:
|
|
1. Group positions by type (share/bond)
|
|
2. For shares: 1 `getShareMarketDataBatch` call → map by secid
|
|
3. For bonds: 1 `getBondPositionDataBatch` call → map by secid
|
|
4. Build enriched positions from maps (no more individual API calls)
|
|
5. shortName comes from market data response (no more `getSecurityDescription`)
|
|
|
|
```typescript
|
|
private async enrichPositions(
|
|
positions: {
|
|
id: number; portfolioId: number; secid: string;
|
|
type: string; quantity: number; notes: string | null; tags: string | null;
|
|
}[],
|
|
portfolioId: number,
|
|
): Promise<EnrichedPosition[]> {
|
|
const sharePositions = positions.filter((p) => p.type === 'share');
|
|
const bondPositions = positions.filter((p) => p.type === 'bond');
|
|
const shareSecids = [...new Set(sharePositions.map((p) => p.secid))].sort();
|
|
const bondSecids = [...new Set(bondPositions.map((p) => p.secid))].sort();
|
|
|
|
const [shareDataBySecid, bondDataBySecid] = await Promise.all([
|
|
this.fetchShareBatch(shareSecids, portfolioId),
|
|
this.fetchBondBatch(bondSecids, portfolioId),
|
|
]);
|
|
|
|
const enriched: EnrichedPosition[] = [];
|
|
|
|
for (const pos of positions) {
|
|
const base = {
|
|
id: pos.id, secid: pos.secid,
|
|
shortName: null as string | null,
|
|
type: pos.type, quantity: pos.quantity,
|
|
notes: pos.notes, tags: pos.tags ? JSON.parse(pos.tags) : null,
|
|
weightPercent: 0, currentPrice: null as number | null,
|
|
currentValue: null as number | null,
|
|
};
|
|
|
|
if (pos.type === 'bond') {
|
|
enriched.push(this.buildBondPosition(pos, base, bondDataBySecid.get(pos.secid)));
|
|
} else {
|
|
enriched.push(this.buildSharePosition(pos, base, shareDataBySecid.get(pos.secid)));
|
|
}
|
|
}
|
|
|
|
return enriched;
|
|
}
|
|
|
|
private async fetchShareBatch(
|
|
secids: string[], portfolioId: number,
|
|
): Promise<Map<string, MoexShareMarketData>> {
|
|
if (secids.length === 0) return new Map();
|
|
const cacheKey = secids.join(',');
|
|
const { data } = await this.cache.getOrFetch(
|
|
'batchdata', ['shares', cacheKey],
|
|
() => this.moexClient.getShareMarketDataBatch(secids),
|
|
'marketDataTtl',
|
|
);
|
|
return new Map(data.map((d) => [d.secid, d]));
|
|
}
|
|
|
|
private async fetchBondBatch(
|
|
secids: string[], portfolioId: number,
|
|
): Promise<Map<string, MoexBondPositionData>> {
|
|
if (secids.length === 0) return new Map();
|
|
const cacheKey = secids.join(',');
|
|
const { data } = await this.cache.getOrFetch(
|
|
'batchdata', ['bonds', cacheKey],
|
|
() => this.moexClient.getBondPositionDataBatch(secids),
|
|
'marketDataTtl',
|
|
);
|
|
return new Map(data.map((d) => [d.secid, d]));
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Add `buildSharePosition` method**
|
|
|
|
```typescript
|
|
private buildSharePosition(
|
|
pos: { id: number; secid: string; quantity: number },
|
|
base: EnrichedPosition,
|
|
data: MoexShareMarketData | undefined,
|
|
): EnrichedPosition {
|
|
if (!data) return { ...base, currentPrice: null, currentValue: null };
|
|
return {
|
|
...base,
|
|
shortName: data.shortName,
|
|
currentPrice: data.last,
|
|
change: data.lastChange,
|
|
changePercent: data.lastChangePrcnt,
|
|
currentValue: data.last !== null ? data.last * pos.quantity : null,
|
|
};
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 3: Add `buildBondPosition` method**
|
|
|
|
```typescript
|
|
private buildBondPosition(
|
|
pos: { id: number; secid: string; quantity: number },
|
|
base: EnrichedPosition,
|
|
data: MoexBondPositionData | undefined,
|
|
): EnrichedPosition {
|
|
if (!data) return { ...base, currentPrice: null, currentValue: null };
|
|
const currentValue =
|
|
data.price !== null ? (data.price / 100) * data.faceValue * pos.quantity : null;
|
|
return {
|
|
...base,
|
|
shortName: data.shortName,
|
|
currentPrice: data.price,
|
|
yieldToMaturity: data.yieldToMaturity,
|
|
duration: data.duration,
|
|
couponValue: data.couponValue,
|
|
couponPercent: data.couponPercent,
|
|
nextCouponDate: data.nextCouponDate,
|
|
matDate: data.matDate,
|
|
accruedInt: data.accruedInt,
|
|
bid: data.bid,
|
|
offer: data.offer,
|
|
couponPeriod: data.couponPeriod,
|
|
bondType: data.bondType,
|
|
offerDate: data.offerDate,
|
|
currentValue,
|
|
};
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Update `findOne` to pass `portfolio.id` to `enrichPositions`**
|
|
|
|
```typescript
|
|
const positionsWithPrices = await this.enrichPositions(portfolio.positions, portfolio.id);
|
|
```
|
|
|
|
- [ ] **Step 5: Clean up removed methods**
|
|
|
|
Remove old private methods: `enrichSharePosition`, `enrichBondPosition` (replaced by `buildSharePosition`, `buildBondPosition`).
|
|
|
|
- [ ] **Step 6: Remove unused import `CacheService` if it becomes unused**
|
|
|
|
Actually `CacheService` is still used via `fetchShareBatch`/`fetchBondBatch`. Keep it.
|
|
|
|
- [ ] **Step 7: Run tests**
|
|
|
|
```bash
|
|
npx vitest run -w apps/backend
|
|
```
|
|
|
|
Expected: all tests pass.
|
|
|
|
---
|
|
|
|
### Task 4: Verify and lint
|
|
|
|
- [ ] **Step 1: TypeScript check**
|
|
|
|
```bash
|
|
npx tsc --noEmit -w apps/backend
|
|
```
|
|
|
|
- [ ] **Step 2: Lint**
|
|
|
|
```bash
|
|
npm run lint 2>/dev/null || echo "Lint check complete"
|
|
```
|
|
|
|
- [ ] **Step 3: Format**
|
|
|
|
```bash
|
|
npm run format
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: Document performance gain
|
|
|
|
- [ ] **Step 1: Write ADR or performance note in docs**
|
|
|
|
Add to `docs/superpowers/adr/2026-06-14-portfolio-enricher-optimization.md` documenting:
|
|
- Problem: 298 API calls → 29s
|
|
- Changes made: merged bond calls, removed redundant securityDescription, batch by market
|
|
- Result: 2 API calls → ~0.3s (97% reduction)
|