KiFdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl m Z m Z ddl mZddlmZddlmZmZmZej*eZGd d e Zd ZGd d eZGddeZGddeZddhZd dZ d! d"dZd#dZ dZ!d$dZ"d%dZ#d&dZ$d'dZ%GddZ&d(dZ' d) d*dZ(y)+a-Vector embedding support for semantic code search. Supports multiple providers: 1. Local (sentence-transformers) - Private, fast, offline. 2. Google Gemini - High-quality, cloud-based. Requires explicit opt-in. 3. MiniMax (embo-01) - High-quality 1536-dim cloud embeddings. Requires MINIMAX_API_KEY. ) annotationsN)ABCabstractmethod)Path)Any) GraphNode GraphStore node_to_dictcheZdZeddZeddZeeddZeed dZy) EmbeddingProvidercyN)selftextss h/home/jay/workspace/scripts/.codegraph-venv/lib/python3.12/site-packages/code_review_graph/embeddings.pyembedzEmbeddingProvider.embed s cy)zCEmbed a search query (may use a different task type than indexing).Nrrtexts r embed_queryzEmbeddingProvider.embed_query$ rcyrrrs r dimensionzEmbeddingProvider.dimension)rrcyrrrs rnamezEmbeddingProvider.name.rrNr list[str]returnlist[list[float]]rstrr" list[float]r"intr"r%) __name__ __module__ __qualname__rrrpropertyrrrrrr r s`        rr zall-MiniLM-L6-v2cPeZdZdd dZdZd dZd dZed dZed dZ y)LocalEmbeddingProviderNcl|xs$tjjdt|_d|_y)NCRG_EMBEDDING_MODEL)osenvirongetLOCAL_DEFAULT_MODEL _model_name_model)r model_names r__init__zLocalEmbeddingProvider.__init__8s-%  !#6*  rc|j/ ddlm}||jdddi|_|jS|jS#t$r t dwxYw)Nr)SentenceTransformerTtrust_remote_code)r< model_kwargszSsentence-transformers not installed. Run: pip install code-review-graph[embeddings])r7sentence_transformersr;r6 ImportError)rr;s r _get_modelz!LocalEmbeddingProvider._get_model>sm ;;  E1$$&*"5t!< {{t{{  !E s "AAc|j}|j|d}|Dcgc]}|jc}Scc}w)NF)show_progress_bar)r@encodetolist)rrmodelvectorsvs rrzLocalEmbeddingProvider.embedNs:!,,u,>$+,q ,,,sAc,|j|gdS)Nr)rrs rrz"LocalEmbeddingProvider.embed_querySszz4&!!$$rcB|j}|jSr)r@ get_sentence_embedding_dimension)rrEs rrz LocalEmbeddingProvider.dimensionVs!5577rc d|jS)Nzlocal:)r6rs rrzLocalEmbeddingProvider.name[s(()**rr)r8 str | Noner"Noner r$r'r)) r*r+r,r9r@rrr-rrrrrr/r/7s>  - %88++rr/c^eZdZdd dZd dZed d dZd dZeddZ eddZ y)GoogleEmbeddingProviderc ddlm}|j||_||_d|_y#t $r t dwxYw)Nr)genaiapi_keyzXgoogle-generativeai not installed. Run: pip install code-review-graph[google-embeddings])googlerQClient_clientrE _dimensionr?)rrSrErQs rr9z GoogleEmbeddingProvider.__init__asL  $ <<<8DLDJ*.DO H  s +.Ac4d}g}tdt||D]S}||||z}j|ffd }|j|jDcgc]}|j c}Uj |rt|d_|Scc}w)Ndrcljjjj|ddiS)N task_typeRETRIEVAL_DOCUMENTrEcontentsconfigrVmodels embed_contentrE)brs rz/GoogleEmbeddingProvider.embed..ss5 3 3 A A**')=>!B!r)rangelen_call_with_retryextend embeddingsvaluesrW)rr batch_sizeresultsibatchresponsees` rrzGoogleEmbeddingProvider.embedms q#e*j1 DA!A N+E,,H NNh.A.ABAHHB C D ?? "w!'!*oDOCsB c t|D] } |cSy#t$rf}t|}d|vxs d|vxsd|v}|r||dz k(rd|z}tj d|dz|||t j |Yd}~yd}~wwxYw)z9Call fn with exponential backoff on transient API errors.429500503rz5Gemini API error (attempt %d/%d), retrying in %ds: %sN)re Exceptionr%loggerwarningtimesleep)fn max_retriesattemptrperr_str is_retryablewaits rrgz(GoogleEmbeddingProvider._call_with_retry~s[) !G !t  ! !a&$/W5G3CWuPWGW #w+/'AG|V&{KqB 4   !s B ABB cjfd}|jdj}jt |_|S)NcnjjjjgddiS)Nr[RETRIEVAL_QUERYr]r`rsrrdz5GoogleEmbeddingProvider.embed_query..s6DLL''55jj#%676rr)rgrirjrWrf)rrrovecs`` rrz#GoogleEmbeddingProvider.embed_querysJ((  !!!$++ ?? "!#hDO rc4|j |jSy)Ni)rWrs rrz!GoogleEmbeddingProvider.dimensions ?? &?? "rc d|jS)Nzgoogle:rErs rrzGoogleEmbeddingProvider.names %%rN)zgemini-embedding-001)rSr%rEr%r"rMr ))r|r(r$r'r)) r*r+r,r9r staticmethodrgrr-rrrrrrOrO`sM "!!  &&rrOc`eZdZdZdZdZdZd dZd dZddZ ddZ e dd Z e dd Z y )MiniMaxEmbeddingProviderzMiniMax embo-01 embedding provider (1536 dimensions). Uses the MiniMax Embeddings API (https://api.minimax.io/v1/embeddings) with the embo-01 model. Requires the MINIMAX_API_KEY environment variable. z$https://api.minimax.io/v1/embeddingszembo-01ic||_yr)_api_key)rrSs rr9z!MiniMaxEmbeddingProvider.__init__s  rc nddl}ddl}|j|j||dj d}|j j |j|dd|jd}d}t|D]} ddl } | j} |j j|d | 5} |j| jjd} ddd j!d i} | j!d ddk7rt#d | j!dd| dcSgS#1swYXxYw#t$$rg}t'|}d|vxs d|vxsd|v}|r||dz k(rd|z}t(j+d|dz|||t-j.|Yd}~3d}~wwxYw)Nr)rErtypezutf-8zapplication/jsonzBearer )z Content-Type Authorization)dataheadersr<)timeoutcontext base_resp status_codezMiniMax API error: status_msgunknownrFrrrsrtrruz6MiniMax API error (attempt %d/%d), retrying in %ds: %s)jsonurllib.requestdumps_MODELrCrequestRequest _ENDPOINTrresslcreate_default_contexturlopenloadsreaddecoder4 RuntimeErrorrvr%rwrxryrz)rrr[_jsonurllibpayloadreqr|r}r_ssl_ctxrespbodyrrpr~rrs r _call_apiz"MiniMaxEmbeddingProvider._call_apis++[[   6'?  nn$$ NN 2#*4==/!:%  [) !G !557^^++CX+NDRV ;;tyy{'9'9''BCDD!HH["5 ==2a7&-immL).T-UVI& !4 -DD !a&$/W5G3CWuPWGW #w+/'AG|LaKdA 4   !s22E2/D8!AE8E =E F4 AF//F4cd}g}tdt||D]+}||||z}|j|j|d-|S)NrYrdb)rerfrhr)rrrkrlrmrns rrzMiniMaxEmbeddingProvider.embedsV %'q#e*j1 8A!A N+E NN4>>%6 7 8rc.|j|gddS)Nqueryr)rrs rrz$MiniMaxEmbeddingProvider.embed_querys~~tfg.q11rc|jSr) _DIMENSIONrs rrz"MiniMaxEmbeddingProvider.dimensions rc d|jS)Nzminimax:)rrs rrzMiniMaxEmbeddingProvider.names$++''rN)rSr%r"rM)rr!r[r%r"r#r r$r'r))r*r+r,__doc__rrrr9rrrr-rrrrrrrsU 7I FJ .`2((rrrTminimaxctjjddjdk(ryt d|dt j y)uPrint a stderr warning before a cloud embedding provider is used. The warning is suppressed when ``CRG_ACCEPT_CLOUD_EMBEDDINGS=1`` is set in the environment, so scripted / CI workloads can acknowledge once and move on. Use stderr (never stdin/input) to stay compatible with the MCP stdio transport — anything we write to stdout would corrupt the JSON-RPC stream. See: #174 CRG_ACCEPT_CLOUD_EMBEDDINGS1Nu9 ⚠️ code-review-graph: about to embed code via the 'az' cloud provider. Your source code (function names, docstrings, file paths) will be sent to an external API. This is necessary for semantic search with the cloud provider you selected. To skip this warning in future runs, set CRG_ACCEPT_CLOUD_EMBEDDINGS=1 in your environment. To stay fully offline, use the default 'local' provider instead (no API key needed). )file)r2r3r4stripprintsysstderr) provider_names r_warn_cloud_egressrsK zz~~3R8>>@CG D]O T! !ZZ rc|dk(rCtjjd}|s tdt dt |S|dk(rMtjjd}|s tdt d t d d|i|rd |iSiS t| S#t$rYy wxYw#t$rYy wxYw) aGet an embedding provider by name. Args: provider: Provider name. One of "local", "google", "minimax", or None for local. Google requires GOOGLE_API_KEY env var and explicit opt-in. MiniMax requires MINIMAX_API_KEY env var and explicit opt-in. Cloud providers emit a one-time stderr warning before use unless ``CRG_ACCEPT_CLOUD_EMBEDDINGS=1`` is set. See: #174 model: Model name/path to use. For local provider this is any sentence-transformers compatible model. Falls back to CRG_EMBEDDING_MODEL env var, then to all-MiniLM-L6-v2. For Google provider this is a Gemini model ID. rMINIMAX_API_KEYzTMINIMAX_API_KEY environment variable is required for the MiniMax embedding provider.rRrTGOOGLE_API_KEYzRGOOGLE_API_KEY environment variable is required for the Google embedding provider.rSrEN)r8r) r2r3r4 ValueErrorrrrOr?r/)providerrErSs r get_providerrs"9**..!232  9%'888**..!121  8$ *',GU# 24 %77    s*B(B( B7( B43B47 CCc, ddl}y#t$rYywxYw)z3Check whether local embedding support is available.rNTF)r>r?)r>s r_check_availablerIs$ s  z CREATE TABLE IF NOT EXISTS embeddings ( qualified_name TEXT PRIMARY KEY, vector BLOB NOT NULL, text_hash TEXT NOT NULL, provider TEXT NOT NULL DEFAULT 'unknown' ); cFtjt|dg|S)z/Encode a float vector as a compact binary blob.f)structpackrf)rs r_encode_vectorr`s ;;#c(1~ , ,,rcbt|dz}ttj|d|S)z,Decode a binary blob back to a float vector.r)rflistrunpack)blobns r_decode_vectorres+ D QA  1gt, --rct|t|k7rytdt||D}td|Ddz}td|Ddz}|dk(s|dk(ry|||zz S)z.Compute cosine similarity between two vectors.gc3,K|] \}}||zywrr).0xys r z%_cosine_similarity..os *1a!e *sc3&K|] }||z ywrrrrs rrz%_cosine_similarity..p"1Q"g?c3&K|] }||z ywrrrs rrz%_cosine_similarity..qrrr)rfsumzip)arcdotnorm_anorm_bs r_cosine_similarityrkst 1vQ *Aq * *C "" "c )F "" "c )F {fk &6/ ""rc|jg}|jdk7r)|j|jj|jr|jd|j|j r|j|j |j r|jd|j |jr|j|jdj|S)z3Convert a node to a searchable text representation.Filezin zreturns  ) rkindappendlower parent_nameparams return_typelanguagejoin)nodepartss r _node_to_textrws YYKE yyF TYY__&'  s4++,-. {{ T[[!  x 0 0123 }} T]]# 88E?rcXeZdZdZ d d dZd dZd d dZdddZddZddZ y)EmbeddingStorez4Manages vector embeddings for graph nodes in SQLite.Nc t|||_|jdu|_t||_t j t|jddd|_t j|j_ |jjt |jjd|jjy#t j$r|jjdYKwxYw)NrF)rcheck_same_threadisolation_levelz'SELECT provider FROM embeddings LIMIT 1zJALTER TABLE embeddings ADD COLUMN provider TEXT NOT NULL DEFAULT 'unknown')rr availablerdb_pathsqlite3connectr%_connRow row_factory executescript_EMBEDDINGS_SCHEMAexecuteOperationalErrorcommit)rrrrEs rr9zEmbeddingStore.__init__s %XU; d2G} __  rU   ")    !34  JJ  H I  ''  JJ  2  s&C.D  D c8|jjyr)rclosers rr zEmbeddingStore.closes rc|jsyg}|jj}|D]}|jdk(rt|}t j |j j}|jjd|jfj}|r|d|k(r |d|k(r|j|||f|sy|D cgc]\} } } |  } } } |jj| } t|| D]>\\}} }}t|}|jjd|j|||f@|jj!t#|Scc} } w)z1Compute and store embeddings for a list of nodes.rrzCSELECT text_hash, provider FROM embeddings WHERE qualified_name = ? text_hashrzvINSERT OR REPLACE INTO embeddings (qualified_name, vector, text_hash, provider) VALUES (?, ?, ?, ?))rrrrhashlibsha256rC hexdigestrrqualified_namefetchonerrrrr rf)rnodesrkto_embedrrrr existing_trrF_textrrs r embed_nodeszEmbeddingStore.embed_nodessw}}68 **  5DyyF" &Dt{{}5??AIzz))U$$&hj  Xk2i? , = OOT43 4 5"#++wq!Q++--%%e,-07-C  ) $T5)c!#&D JJ  *$$dI}E   8},s E:c|jsgS|jj}|jj|}g}|jj d|f}d} |j |}|sn8|D]2} t | d} t|| } |j| d| f4L|jdd|d|S) z(Search for nodes by semantic similarity.z@SELECT qualified_name, vector FROM embeddings WHERE provider = ?iTvectorrc |dS)Nrr)rs rrdz'EmbeddingStore.search..s !A$r)keyreverseN) rrrrr fetchmanyrrrsort) rrlimitr query_vecscoredcursor chunk_sizerowsrowrsims rsearchzEmbeddingStore.searchs}}I ** MM--e4 +-## N    ##J/D <$S]3(C8 s#34c:; <    5fu~rcr|jjd|f|jjy)Nz/DELETE FROM embeddings WHERE qualified_name = ?)rrr )rrs r remove_nodezEmbeddingStore.remove_nodes.  =?P  rcZ|jjdjdS)NzSELECT COUNT(*) FROM embeddingsr)rrrrs rcountzEmbeddingStore.counts'zz!!"CDMMOPQRRrNN)rz str | PathrrLrErLr"rM)r"rM)@)rzlist[GraphNode]rkr(r"r()rr%r!r(r"zlist[tuple[str, float]])rr%r"rMr') r*r+r,rr9r rr)r+r-rrrrrsS> $    6*X6 Srrc|jsy|j}g}|D]"}|j|j|$|j |S)z&Embed all non-file nodes in the graph.r)r get_all_filesrhget_nodes_by_filer) graph_storeembedding_store all_files all_nodesrs rembed_all_nodesr9s\  $ $))+I!#I ;66q9:;  & &y 11rcb|jrs|jdkDr`|j||}g}|D]D\}}|j|}|st |} t |d| d<|j | F|S|j||} | D cgc] } t | c} Scc} w)zESearch nodes using vector similarity, falling back to keyword search.r)r!rsimilarity_score)rr-r)get_noder roundr search_nodes) rr5r6r!rloutputqnscorerdrrs rsemantic_searchrCs  _%:%:%rOs-#  #66   8 $  *)&+.&+RE&/E&PO(0O(dY'6 // //d- . #  oSoSd 2" , ,,$,  ,  ,r