i;3UdZddlmZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl mZgdgdgd gd gd d Zd ed<e GddZe GddZd$dZd%dZd&dZd&dZd&dZd&dZd'dZd(dZd)dZd*dZd+dZd*dZd,dZd-d Z d.d!Z!d/d0d"Z"e#d#k(rejHe"yy)1aLLM-as-judge skill quality evaluation framework. Evaluates SKILL.md files against 5 standard sections using rule-based scoring. Optionally supports LLM evaluation (stub only, requires --api-key). Usage: python3 skill-judge.py /path/to/SKILL.md python3 skill-judge.py --dir ~/.claude/skills/ python3 skill-judge.py --use-llm /path/to/SKILL.md python3 skill-judge.py --format json /path/to/SKILL.md python3 skill-judge.py --format summary --dir ~/.claude/skills/ ) annotationsN) dataclassfield)Path)Any) descriptionoverviewu개요u설명)z when to use when_to_usetriggeru 트리거u 사용 시점u 사용시점) instructions instructionrulesruleworkflowu지침u규칙)z output format output_formatoutputu 출력 형식u 출력형식u 결과물u결과)examplesexampleu예시u예제)rr r rrzdict[str, list[str]]SECTION_ALIASEScXeZdZUdZded<ded<ded<ded<ed d Zded <y ) SectionScorez$Score for a single SKILL.md section.strnameboolfoundintscorelengthF)defaultreprcontentN)__name__ __module__ __qualname____doc____annotations__rr"D/home/jay/workspace/.worktrees/task-2116-dev1/scripts/skill-judge.pyrrMs*. I K J K%0GS0r)rcfeZdZUdZded<ded<ded<ded<ded <ded <ded <d Zded <y) SkillResultz-Evaluation result for a single SKILL.md file.r skill_pathr total_scoregradezdict[str, SectionScore]sections penaltiesbonusesmodermessageN)r#r$r%r&r'r4r(r)r*r,r,Xs47O J%%N L IGSr)r,c>|jjS)z6Lower-case and strip whitespace from a heading string.)striplower)texts r*_normalize_headingr9ks ::<   r)cvt|}tjD]\}}|D] }||k(s |ccSy)z8Return the canonical section key for a heading, or None.N)r9ritems)heading normalizedkeyaliasesaliass r*_match_section_keyrApsI#G,J'--/ W EU"   r)c d|vS)Nz```r(r"s r*_has_code_blockrDzs G r)c^ttjd|tjS)Nz ^\s*[-*+]\srresearch MULTILINErCs r* _has_listrJ~  .'2<<@ AAr)c^ttjd|tjS)Nz ^\s*\d+\.\srFrCs r*_has_numbered_listrMrKr)cLt|xst|xs t|S)zFReturn True if content contains code blocks, lists, or numbered lists.)rDrJrMrCs r*_has_specificityrOs# 7 # Xy'9 X=OPW=XXr)ctDcic] }|dddd }}|js|S|jddjdd}|j}g}t |D]^\}}t j d|}|s|jd j} t| }|L|j||f`t |D]y\} \} }| d z} | d zt|kr || d zdn t|} || | }dj|j}d ||d <|||d <t|||d <{|Scc}w)aParse a SKILL.md content string into a dict keyed by canonical section names. Returns a dict with keys: description, when_to_use, instructions, output_format, examples. Each value is a dict with: found (bool), length (int), content (str). Frr)rrr"z   z ^##\s+(.+)$Trr"r) rr6replace splitlines enumeraterGmatchgrouprAappendlenjoin)r"r>result normalisedlinessection_positionsidxlinem heading_textposline_idxstartend section_linessection_contents r*parse_skill_sectionsrjsgv(v_bWY.Z)Z(vF(v ==? .66tTBJ  ! ! #E02u%5 T HH^T * 771:++-L$\2C!((#s45!**; <5_h1 /2Qw=N9O/Oa(+UXY^U_eC( ))M288:#s G!0s I #O 4s H5 MA)wsEc|st|ddddSd}t|}|dz }|dk\r|dz }t|r|dz }|dk\r|dz }t|d|||S) aScore a single section's content on a 0-20 scale. Scoring breakdown: +5 section exists (content non-empty) +5 content >= 50 chars +5 contains code block, list, or numbered list (specificity) +5 content >= 200 chars (completeness) Frr)rrrrr"2T)rrZrO)rr"rrs r* score_sectionros~ U!ArRR E \F QJE|     }   TU6SZ [[r)c4|dk\ry|dk\ry|dk\ry|dk\ryy ) z*Convert a numeric score to a letter grade.ZAKB<C-DFr(r.s r* compute_grader{s1b      r)c t|}|jstd||jd}t |}t |}i}d}t D]*}||}t||d} | ||<|| jz },d} d} |dkrd} |dk\rd } td|| z| z} || z| z} t| } tt|j| | || | d S) a#Evaluate a single SKILL.md file using rule-based scoring. Args: skill_path: Absolute or relative path to a SKILL.md file. Returns: SkillResult with total_score, grade, per-section scores, etc. Raises: FileNotFoundError: If the file does not exist. zSKILL.md not found: zutf-8)encodingrr"iii'rlz rule-basedr-r.r/r0r1r2r3)rexistsFileNotFoundError read_textrZrjrrormaxr{r,rresolve)r-pathr" total_lenparsedr0 raw_scorer> section_datassr1r2r.r/s r*evaluate_skillrs  D ;;="6zl CDDnngn.GG I !' *F(*HIc{ 3 Y 7 8 RXX IG3 EaY.89K i''1K + &E t||~&  r)ct|}|jstd||jst d|g}t |j dD]%}|jtt|'|S)aWEvaluate all SKILL.md files found recursively under dir_path. Args: dir_path: Path to a directory containing skill subdirectories. Returns: List of SkillResult, one per SKILL.md found. Raises: FileNotFoundError: If dir_path does not exist. NotADirectoryError: If dir_path is not a directory. zDirectory not found: zNot a directory: zSKILL.md) rrris_dirNotADirectoryErrorsortedrglobrYrr)dir_pathdpathresultsskill_mds r*evaluate_directoryr#s NE <<>"7z BCC <<> #4XJ!?@@!#G5;;z236~c(m456 Nr)c lt|}tt|jddiddddS)aLLM evaluation stub. Returns a SkillResult with mode='llm-stub' and an explanatory message. Does NOT make any API calls (to avoid cost). Args: skill_path: Path to a SKILL.md file. Returns: SkillResult with mode='llm-stub'. rryzllm-stubzSLLM evaluation requires --api-key. Pass --api-key to enable real LLM scoring.)r-r.r/r0r1r2r3r4)rr,rr)r-rs r* llm_evaluater=s>  D t||~& e  r)c^i}|jjD],\}}|j|j|jd||<.|j |j |j||j|j|jd}|jr|j|d<|S)z2Convert a SkillResult to a JSON-serialisable dict.)rrrr~r4) r0r;rrrr-r.r/r1r2r3r4)r\ sections_dictr>rds r*result_to_dictrVs/1M??((* RXXXXii c ''))!%%>> A~~~~) Hr)c tddtddddddddtd |D]v}t|jjjxst|jj}t|dd|j dd|j dxtdtd t|d |r.td |Dt|z }td |dyy)z4Print a human-readable summary for multiple results.rQz<============================================================Skill40 Scorez>6Gradez<------------------------------------------------------------zTotal: z skills evaluatedc34K|]}|jywNrz).0rs r* z!_print_summary..~s1A!--1szAverage score: z.1fN) printrr-parentrr.r/rZsum)rrravgs r*_print_summaryrss Bvh- WRL'"Qwrl 34 VH <ALL!((--Hall1C1H1H b 1==,Aaggb\:;< VH GCL>!2 34111CL@ Cy)*r)c td|jtd|jd|jd|j|j rtd|j td|j d|jtd|jjD]C\}}|jrd nd }td |d d |dd|jdd|jEy)z;Print a detailed human-readable report for a single result.z Skill: zScore: z Grade: z Mode: z Message: z Penalties: z Bonuses: z Sections:FOUNDMISSINGz 20z [7z] score=2z length=N) rr-r.r/r3r4r1r2r0r;rrr)r\r>rstatuss r* _print_detailrs If''( )* GF&&'yhv{{m TU ~~ &..)*+ K(()V^^4D EF -??((*QRHH) 3r("VAJhrxxl)BII;OPQr)c6tjd}|jddd|jdd |jd d d d |jddd|jdgddd|j|}g}|jrt |j}nW|j r:|jrt|j g}n(t|j g}n|jy|jdk(rO|Dcgc] }t|}}t|dk(r|dn|}ttj |d dy|jdk(r t#|y|D] }t%|ycc}w)Nz3Evaluate SKILL.md quality using rule-based scoring.)rr?zPath to a SKILL.md file)nargshelpz--dirz)Directory containing skill subdirectories)rz --use-llm store_trueFz2Use LLM evaluation (stub only, requires --api-key))actionr rz --api-keyrz2API key for LLM evaluation (not used in stub mode))r rz--format)jsonsummarydetailrzOutput format (default: detail))choicesr rrSrr) ensure_asciiindentr)argparseArgumentParser add_argument parse_argsdirrruse_llmrr print_helpformatrrZrrdumpsrr)argvparserargsrrdictsrs r*mainrs  $ $1f gF c0IJ &QR  A    A  - .    T "D!#G xx$TXX.  <<#DII./G%dii01G {{f,34q"44 Z1_q% djjeA>?  !w  A !   5sF__main__)r8rreturnr)r<rrz str | None)r"rrr)r"rrzdict[str, dict[str, Any]])rrr"rrr)r.rrr)r-rrr,)rrrlist[SkillResult])r\r,rzdict[str, Any])rrrNone)r\r,rrr)rzlist[str] | Nonerr)%r& __future__rrrosrGsys dataclassesrrpathlibrtypingrrr'rr,r9rArDrJrMrOrjror{rrrrrrrr#exitr(r)r*rs # ( C')%'^ 11 1     $ BBY(V\B 6r42 : + Q/ d z CHHTVr)