i]dZddlmZddlZddlmZddlmZmZm Z ddl Z ddl m Z m Z mZ d ddZdddZGd d ZGd d ZGd dZGddZGddZGddZGddZGddZe j4j6GddZy#e$r dZ dZdZ YwxYw)u Lightpanda 크롤러 래퍼 테스트 단위 테스트: mock 사용 (네트워크 불필요) 통합 테스트: @pytest.mark.integration — 실제 lightpanda 서버(ws://127.0.0.1:9333) 필요 ) annotationsN)fields) AsyncMock MagicMockpatch) CrawlError CrawlResultLightpandaCrawlerc&t}t||_t||_ttd|_t|_d|_td|_td|_tg|_ |S)u$playwright Page 모의 객체 생성 return_value)statushttps://example.com Test PageN) rrtitlecontentgotocloseurlevaluatequery_selectorquery_selector_all)rrpages L/home/jay/workspace/.worktrees/task-2116-dev1/tools/tests/test_lightpanda.py_make_mock_pagerso ;D.DJ'2DLy'<=DIDJ$DH;7DM#6D'R8D Kc| t}t}t||_t|_td|_|S)u'playwright Browser 모의 객체 생성r T)rrrnew_pager is_connected)rbrowsers r_make_mock_browserr"/s> | kG d3GKGM$$7G Nrc,eZdZddZddZddZddZy)TestLightpandaCrawlerInitcTtJdt}|jdk(sJy)u;CDP endpoint 기본값이 ws://127.0.0.1:9333 인지 확인NuLightpandaCrawler import 실패zws://127.0.0.1:9333r cdp_endpointselfcrawlers rtest_default_cdp_endpointz3TestLightpandaCrawlerInit.test_default_cdp_endpoint@s1 ,O.OO,#%##'<<<CrawlResult 에 필수 필드가 모두 존재하는지 확인NuCrawlResult import 실패> rhtmlmetatextlinksrenginer elapsed_msu누락 필드: )r rnameissubset)r)f field_namesrequireds rtest_required_fields_existz3TestCrawlResultDataclass.test_required_fields_exist_s`&C(CC&'-k':;!qvv; ;f  -YKAW@X/YY-zhttps://example.com/page descriptiontestrgY@ lightpanda rrr@r>rAr?rrCrB)r rrB isinstancerAlistr?dict)r)results rtest_instantiationz+TestCrawlResultDataclass.test_instantiationfs&&&% -.(  zz2222}} ,,,&,,---&++t,,,rc ltJdD]'}tddddgidd| }|j|k(r'Jy)u8engine 필드는 'lightpanda' 또는 'chrome' 중 하나NrNchromez https://x.comrgrO)r rB)r)rBrs rtest_engine_field_valuesz1TestCrawlResultDataclass.test_engine_field_valuesysV&&&. &F# A88v% %% &rNr4)r7r8r9rIrTrZr:rrr<r<^sZ-&&rr<ceZdZejdZej jddZej jddZ ej jddZ ej jddZ y) TestFetchUnitc#:Ktd5}t}t||j_td|j_t }t |}t||j_|||fdddy#1swYyxYww)z playwright connect_over_cdp mock)tools.lightpanda_crawler.async_playwrightr FN rrr __aenter__ __aexit__rr"chromiumconnect_over_cdpr)mock_pw mock_instancerr!s rmock_playwright_connectz%TestFetchUnit.mock_playwright_connects> ? )7%KM.7].SG +-6E-JG *"$D(.G6?W6UM " " 37D( ( ) ) ) BA8B BBBcKtJ|\}}}t4d{}|jdd{}dddd{ttsJy7@7)7#1d{7swY+xYww)u2fetch() 가 CrawlResult 를 반환하는지 확인Nr)r fetchrPr r)rg_r!rr*rSs rtest_fetch_returns_crawl_resultz-TestFetchUnit.test_fetch_returns_crawl_results!,,,27D$& @ @'"==)>??F @ @&+... @? @ @ @ @sSA;A A;A&A"A& A;A$ A;"A&$A;&A8,A/ -A84A;cFKtJ|\}}}t4d{}|jdd{}dddd{tdsJt|dsJt|dsJt|dsJy7f7O7A#1d{7swYQxYww)uBfetch() 결과에 title/text/html/links 필드가 있는지 확인Nrrr@r>rA)r rjhasattrrks r%test_fetch_result_has_required_fieldsz3TestFetchUnit.test_fetch_result_has_required_fieldss!,,,27D$& @ @'"==)>??F @ @vw'''vv&&&vv&&&vw'''  @? @ @ @ @sSB!BB!B BB  B!B  >B!B  B! BB BB!cKtJ|\}}}t4d{}|jdd{}dddd{jdk(sJy7?7(7#1d{7swY*xYww)u6fetch() 결과에 요청 URL이 저장되는지 확인Nr)r rjrrks rtest_fetch_url_stored_correctlyz-TestFetchUnit.test_fetch_url_stored_correctlys!,,,27D$& @ @'"==)>??F @ @zz2222 @? @ @ @ @SA:AA:A%A!A% A:A# A:!A%#A:%A7+A. ,A73A:cKtJ|\}}}t4d{}|jdd{}dddd{jdk\sJy7?7(7#1d{7swY*xYww)u*fetch() elapsed_ms 가 양수인지 확인Nrr)r rjrCrks rtest_fetch_elapsed_ms_positivez,TestFetchUnit.test_fetch_elapsed_ms_positives!,,,27D$& @ @'"==)>??F @ @  A%%% @? @ @ @ @rsNr4) r7r8r9pytestfixturergmarkasynciormrprrrur:rrr\r\s ^^ ) ) [[// [[ ( ( [[33 [[&&rr\cReZdZejdZej jddZej jddZ ej jddZ ej jddZ ej jddZ y) TestFetchManyUnitc#:Ktd5}t}t||j_td|j_t }t |}t||j_|||fdddy#1swYyxYww)Nr^r Fr_rds rrgz)TestFetchManyUnit.mock_playwright_connects > ? )7%KM.7].SG +-6E-JG *"$D(.G6?W6UM " " 37D( ( ) ) )rhcKtJgd}t4d{}|j|d{}dddd{ttsJy7@7)7#1d{7swY+xYww)u4fetch_many() 가 리스트를 반환하는지 확인Nrhttps://example.orgzhttps://example.net)r fetch_manyrPrQr)rgurlsr*resultss rtest_fetch_many_returns_listz.TestFetchManyUnit.test_fetch_many_returns_listsy!,,,T$& 5 5'#..t44G 5 5'4((( 54 5 5 5 5sSA9AA9A$A A$ A9A"A9 A$"A9$A6*A- +A62A9cKtJgd}t4d{}|j|d{}dddd{tdk(sJy7>7'7#1d{7swY)xYww)uCfetch_many() 결과 수가 입력 URL 수와 일치하는지 확인Nr~r rlenrs r#test_fetch_many_count_matches_inputz5TestFetchManyUnit.test_fetch_many_count_matches_inputs{!,,,T$& 5 5'#..t44G 5 57|q    54 5 5 5 5sSA7AA7A"AA" A7A A7A" A7"A4(A+ )A40A7cKtJgd}t4d{}|j|d{}dddd{tdDsJy7B7+7#1d{7swY-xYww)uCfetch_many() 결과가 모두 CrawlResult 인스턴스인지 확인Nr~c3<K|]}t|tywNrPr .0rYs r zFTestFetchManyUnit.test_fetch_many_all_crawl_results..?!:a-?)r rallrs r!test_fetch_many_all_crawl_resultsz3TestFetchManyUnit.test_fetch_many_all_crawl_resultss}!,,,T$& 5 5'#..t44G 5 5?w???? 54 5 5 5 5sSA;A A;A&A"A& A;A$A;"A&$A;&A8,A/ -A84A;cKtJt4d{}|jgd{}dddd{gk(sJy7577#1d{7swY xYww)u:fetch_many() 빈 리스트 입력 시 빈 리스트 반환N)r r)r)rgr*rs rtest_fetch_many_empty_listz,TestFetchManyUnit.test_fetch_many_empty_listsp!,,,$& 3 3'#..r22G 3 3"}} 32 3 3 3 3sSA*AA*AAA A*A A*AA*A'A A'#A*cKtJddg}t4d{}|j|dd{}dddd{tdk(sJy7@7'7#1d{7swY)xYww)u?fetch_many() concurrency 파라미터가 수용되는지 확인Nrr concurrencyrrs r!test_fetch_many_concurrency_paramz3TestFetchManyUnit.test_fetch_many_concurrency_param s!,,,%'<=$& D D'#..t.CCG D D7|q    DC D D D DsSA9AA9A$A A$ A9A" A9 A$"A9$A6*A- +A62A9Nr4) r7r8r9rvrwrgrxryrrrrrr:rrr{r{s ^^ ) ) [[)) [[!! [[@@ [[ [[!!rr{ceZdZejdZej jddZej jddZ y)TestEvaluateUnitc#\Ktd5}t}t||j_td|j_t }td|_t|}t||j_ |||fdddy#1swYyxYww)Nr^r FExample Domain) rrr r`rarrr"rbrcrds rrgz(TestEvaluateUnit.mock_playwright_connects > ? )7%KM.7].SG +-6E-JG *"$D%3CDDM(.G6?W6UM " " 37D( ( ) ) )s B,B B  B, B)%B,cKtJ|\}}}t4d{}|jddd{}dddd{dk(sJy7677#1d{7swY xYww)u9evaluate() 가 JS 실행 결과를 반환하는지 확인Nrdocument.titler)r rrks rtest_evaluate_returns_valuez,TestEvaluateUnit.test_evaluate_returns_value-s!,,,27D$& U U'"++,ACSTTF U U)))) UT U U U UsSA1AA1AAA A1 A A1AA1A."A% #A.*A1c KtJ|\}}}t4d{}|jddd{dddd{|jjy7I717##1d{7swY3xYww)u7evaluate() 가 page.evaluate 를 호출하는지 확인Nrr)r rassert_called_once)r)rgrlr!rr*s r!test_evaluate_calls_page_evaluatez2TestEvaluateUnit.test_evaluate_calls_page_evaluate8s!,,,27D$& L L'""#8:JK K K L L ((* L K L L L LsSBA)BA/A+A/ B A- B+A/-B/B5A8 6B=BNr4) r7r8r9rvrwrgrxryrrr:rrrrsT ^^ ) ) [[** [[++rrceZdZejdZej jddZej jddZ ej jddZ ej jddZ y)TestExtractStructuredUnitc#Ktd5}t}t||j_td|j_t }td|_td|_t}t||_ t||g|_ t|}t||j_ ||||fdddy#1swYyxYww)Nr^r Fz Hello World attr_value)rrr r`rar inner_text get_attributerrrr"rbrc)r)rerfmock_elrr!s rrgz1TestExtractStructuredUnit.mock_playwright_connectJs > ? 27%KM.7].SG +-6E-JG *kG!* !FG $-<$HG !"$D"+"AD &/gw=O&PD #(.G6?W6UM " " 37D'1 1 2 2 2s C,C C  C, C)%C,cKtJ|\}}}}t4d{}|jdddid{}dddd{ttsJy7C7)7#1d{7swY+xYww)u8extract_structured() 가 dict 를 반환하는지 확인Nrrh1)r extract_structuredrPrRr)rgrlr!rrr*rSs r$test_extract_structured_returns_dictz>TestExtractStructuredUnit.test_extract_structured_returns_dict]s!,,,$;!7D'$& ^ ^'"556KgW[_]]F ^ ^&$''' ^] ^ ^ ^ ^sTA?A$A?A*A&A* A? A( A?&A*(A?*A<0A3 1A<8A?cKtJ|\}}}}t4d{}|jdddid{}dddd{dvsJy7777#1d{7swYxYww)u@extract_structured() 결과에 요청한 키가 있는지 확인Nrrrr rrs r#test_extract_structured_key_presentz=TestExtractStructuredUnit.test_extract_structured_key_presenths!,,,$;!7D'$& ^ ^'"556KgW[_]]F ^ ^&    ^] ^ ^ ^ ^TA3AA3AAA A3 A A3AA3A0$A' %A0,A3cKtJ|\}}}}t4d{}|jdddid{}dddd{dvsJy7777#1d{7swYxYww)u;::attr(name) 문법으로 속성 추출 가능한지 확인Nrhrefz a::attr(href)rrs r#test_extract_structured_attr_syntaxz=TestExtractStructuredUnit.test_extract_structured_attr_syntaxss!,,,$;!7D'$& h h'"556KfVeMfggF h h hg h h h hrc KtJ|\}}}}t4d{}|jdddid{}dddd{tjdtsJy7R787*#1d{7swY:xYww)u;복수형 키(links)는 리스트를 반환하는지 확인NrrAa)r rrPgetrQrs r/test_extract_structured_plural_key_returns_listzITestExtractStructuredUnit.test_extract_structured_plural_key_returns_list~s!,,,$;!7D'$& ] ]'"556KgWZ^\\F ] ]&**W-t444 ]\ ] ] ] ]sTBA3BA9A5A9 B A7 'B5A97B9B ?B B BNr4) r7r8r9rvrwrgrxryrrrrr:rrrrIs ^^22$ [[(( [[!! [[   [[55rrceZdZejj ddZejj ddZejj ddZy)TestChromeFallbackcKtJtd5}t}t||j_td|j_t }t|}ttd|g|j_ t4d{}|jdk(sJdddd{dddy7.7#1d{7swYxYw#1swYyxYww)uCLightpanda 연결 실패 시 Chrome으로 fallback 되는지 확인Nr^r FzLightpanda connection refused side_effectrW) r rrr r`rarr" Exceptionrbrc_engine)r)rerfrchrome_browserr*s r*test_chrome_fallback_on_lightpanda_failurez=TestChromeFallback.test_chrome_fallback_on_lightpanda_failures!,,, > ? 37%KM.7].SG +-6E-JG *"$D/5N7@=>"7M " " 3)* 3 3g(222 3 3! 3 3  3 3 3 3 3! 3 3seC4B C( C!C($C6 C(CC( C4C(C(C% CC% !C((C1-C4cKtJtd5}t}t||j_td|j_t }t|}t||j_ t4d{}|jdk(sJdddd{dddy7.7#1d{7swYxYw#1swYyxYww)uBLightpanda 연결 성공 시 engine 이 'lightpanda' 인지 확인Nr^r FrN) r rrr r`rarr"rbrcr)r)rerfr lp_browserr*s r$test_engine_is_lightpanda_on_successz7TestChromeFallback.test_engine_is_lightpanda_on_successs!,,, > ? 77%KM.7].SG +-6E-JG *"$D+D1J6?Z6XM " " 3(* 7 7g,666 7 7 7 7 7 7 7 7 7 7 7seC)A?CCCC+ C6C7C; C)CCC CC CC&"C)c KtJtd5}t}t||j_td|j_t }t|}t||j_ t4d{}|jdd{}|j|jk(sJdddd{dddy7Q7:7#1d{7swY!xYw#1swYyxYww)uECrawlResult.engine 이 실제 사용 엔진과 일치하는지 확인Nr^r Fr) r rrr r`rarr"rbrcrjrBr)r)rerfrrr*rSs r(test_crawl_result_engine_matches_crawlerz;TestChromeFallback.test_crawl_result_engine_matches_crawlers!,,, > ? 87%KM.7].SG +-6E-JG *"$D+D1J6?Z6XM " " 3(* 8 8g&}}-BCC}}777 8 8 8 8 8C 8 8 8 8 8 8swDA?DC'DC-.C)/C- DC+D D'D)C-+D-C? 3C64C? ;DD DNr4) r7r8r9rvrxryrrrr:rrrrsZ [[33. [[77  [[88rrceZdZddZej j ddZej j ddZej j ddZ y)TestErrorHandlingcJtJdtttsJy)u5CrawlError 커스텀 예외가 존재하는지 확인NuCrawlError import 실패)r issubclassr)r)s rtest_crawl_error_existsz)TestErrorHandling.test_crawl_error_existss$%A'AA%*i000rc  KtJtJtd5}t}t||j_td|j_ttd|j_ tjt5t4d{} dddd{ddddddy7&7#1d{7swY'xYw#1swY+xYw#1swYyxYww)u7Lightpanda + Chrome 모두 실패 시 CrawlError 발생Nr^r FzConnection refusedr) r rrrr r`rarrbrcrvraises)r)rerfr*s r+test_both_endpoints_fail_raises_crawl_errorz=TestErrorHandling.test_both_endpoints_fail_raises_crawl_errors!,,,%%% > ? 7%KM.7].SG +-6E-JG *6?IVjLk6lM " " 3z* ,.'       s}DA>C7C++C ,C+/C1 C+<C=C+C7 DC+C+C( CC( $C++C4 0C77D<DcKtJtd5}t}t||j_td|j_t }t|}t||j_ t4d{}tjt5}|jdd{ddddtjvsdt|jvsJdddd{dddy77W#1swYVxYw7#1d{7swY-xYw#1swYyxYww)u2screenshot() 호출 시 NotImplementedError 발생Nr^r FrzPlaywright+Chromeu 스크린샷)r rrr r`rarr"rbrcrvrNotImplementedError screenshotstrvalue)r)rerfrr!r*exc_infos r&test_screenshot_raises_not_implementedz8TestErrorHandling.test_screenshot_raises_not_implementeds<!,,, > ? k7%KM.7].SG +-6E-JG *"$D(.G6?W6UM " " 3(* k kg]]#67D8!,,-BCCCD*c(...AA^WZ[c[i[iWjEjjj k k k k kCDD k k k k k ksEA?EDED03D"D D" 8D0 ED.E EE D""D+ 'D0.E0E 6D97E >EE EcFKtJtd5}t}t||j_td|j_t }t|}t||j_ t4d{}tjt5|jdd{ddddddd{dddy7X7'#1swY&xYw7#1d{7swY-xYw#1swYyxYww)u+pdf() 호출 시 NotImplementedError 발생Nr^r Fr)r rrr r`rarr"rbrcrvrrpdf)r)rerfrr!r*s rtest_pdf_raises_not_implementedz1TestErrorHandling.test_pdf_raises_not_implementeds!,,, > ? =7%KM.7].SG +-6E-JG *"$D(.G6?W6UM " " 3(* = =g]]#67=!++&;<<<= = = = = =<== = = = = = =sD!A?DC.DD3C2C0 C2 D D C>!D% D!.D0C22C; 7D>DD D D DDD!Nr4) r7r8r9rrvrxryrrrr:rrrrsc1  [[  [[kk$ [[==rrceZdZejj ddZejj ddZejj ddZejj ddZ y)TestIntegrationcKtJt4d{}|jdd{}dddd{ttsJ|jsJ|j sJ|j dk(sJt|jtsJ|jdkDsJ|jdvsJy777#1d{7swYxYww)uK실제 example.com 크롤링 — title/text/html/links 필드 존재 확인NrrrV) r rjrPr rr>rrArQrCrBr)r*rSs rtest_fetch_example_comz&TestIntegration.test_fetch_example_coms!,,,$& @ @'"==)>??F @ @&+...|||{{{zz2222&,,---  1$$$}} 8888 @? @ @ @ @sTCCCC CC  CCBCC C CC CCcKtJgd}t4d{}|j|dd{}dddd{tdk(sJtd|DsJy7T7;7-#1d{7swY=xYww)u/3개 URL 병렬 크롤링 — 결과 수 확인Nr~rrc3<K|]}t|tywrrrs rrz=TestIntegration.test_fetch_many_three_urls..1rr)r rrr)r)rr*rs rtest_fetch_many_three_urlsz*TestIntegration.test_fetch_many_three_urls$s!,,,  %& D D'#..t.CCG D D7|q   ?w????  DC D D D DsSB A2B A8A4A8 B A6 *B 4A86B 8B >B ?B B cKtJt4d{}|jddd{}dddd{ttsJt |dkDsJy7Q797+#1d{7swY;xYww)u+evaluate() 로 document.title 반환 확인Nrrr)r rrPrr)r)r*rs rtest_evaluate_document_titlez,TestIntegration.test_evaluate_document_title3s!,,,$& T T'!**+@BRSSE T T%%%%5zA~~  TS T T T TsSBA+BA1A-A1 BA/(B-A1/B1B7A: 8B?Bc8KtJt4d{}|jddddd{}dddd{ttsJd|vsJd|vsJt|dtsJy7e7J7<#1d{7swYLxYww)u+CSS 셀렉터로 구조화 데이터 추출Nrrr)headingrArrA)r rrPrRrQrs rtest_extract_structured_cssz+TestIntegration.test_extract_structured_css=s!,,,$&  '"55% 3/F   &$'''F"""&   &/4000     sSBA?BBBB BB9BBBB B BBNr4) r7r8r9rvrxryrrrrr:rrrrsz [[ 9 9 [[ @ @ [[ [[ 1 1rr)rzxTest Page

Hello

link)rrrrr5rr)rzMagicMock | Noner5r)__doc__ __future__rry dataclassesr unittest.mockrrrrvtools.lightpanda_crawlerrr r ImportErrorrr"r$r<r\r{rrrrrx integrationrr:rrrs#55 SSN  " @@>*&*&d<&<&HB!B!T#+#+V>5>5L:8:8D:=:=D616161CKJs B11 B?>B?