Ti{ *UdZddlZddlmcmZddlZddlZ ddl Z ddl Z ddl Z ddl mZddlmZmZddlZeej)j*j*j*Ze j.j1deedz dZeeed< e j8j;deedz dz ZeBej>6e j8jAeZej>jCeddl#Z#ddl$Z%ddl&Z'dd l#m(Z(m)Z)m*Z*m+Z+m,Z,m-Z-dd l$m.Z.dd l&m/Z/m0Z0d Z1d Z2dZ3dZ4dZ5 d#dedede6defdZ7dedede8efdZ9GddZ:GddZ;GddZ<Gdd Z=Gd!d"Z>y#e"$rYwxYw)$ui 통합 테스트: test_crawl_integration.py browser.py + crawl_utils + insurance_crawler + insurance_spider 모듈 간 통합 호환성을 검증하는 테스트 스위트. 주의: - 외부 네트워크 호출 금지 (모든 테스트는 로컬 HTML fixture 사용). - 합법적 공개 데이터(보험사 공시 페이지 등)만을 대상으로 합니다. N)Path)AnyOptionalscripts _browser_modbrowserz browser.py) ProxyRotator clean_htmlfetch_with_retryget_resource_block_typeshtml_to_markdownis_proxy_error)InsuranceCrawler)InsuranceSpiderResponseHistoryu
화재보험 50000
자동차보험 100000
생명보험 30000
다음 u
보험명보험료가입기간
화재보험500001년
자동차보험1000001년
u

보험 공시 정보

화재보험
u.

보험 정보

htmlurlstatusreturncDddlm}|||d}||_i|_|S)uGSelector를 Response 대용으로 사용 (Response는 Selector 상속).r)SelectorF)radaptive)scrapling.parserrrmeta)rrrrsels ;/home/jay/workspace/scripts/tests/test_crawl_integration.py_make_mock_responserjs( * 4S5 1CCJCH JspiderresponsecpKg}|j|23d{}|j|76|Sw)u9parse() async generator의 결과를 리스트로 수집.N)parseappend)r r!resultsitems r_collect_parse_resultsr'xs=Gll8,dt, Ns6313636ceZdZdZddZddZddZddZddZddZ dd Z dd Z dd Z dd Z dd ZddZddZddZddZddZddZddZy)TestImportCompatibilityu@모든 모듈이 충돌 없이 import 가능한지 검증한다.Ncd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}tjddzd|iz}ttj|dx}}y) u<browser.py가 importlib으로 정상 로드되어야 한다.Nis notz%(py0)s is not %(py3)srpy0py3u+browser.py를 로드하지 못했습니다.z >assert %(py5)spy5) r @pytest_ar_call_reprcompare @py_builtinslocals_should_repr_global_name _saferepr_format_assertmsgAssertionError_format_explanationself @py_assert2 @py_assert1 @py_format4 @py_format6s rtest_browser_module_loadedz2TestImportCompatibility.test_browser_module_loadedsd#'V|4'VVV|4VVVVVV|VVV|VVV4VVV)VVVVVVVrcZd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}d}tt|}|sd d tjvstj trtj tnd dtjvstj trtj tndtj |tj |d z}ttj|dx}}y) u9browser.py에 STEALTH_ARGS 변수가 존재해야 한다.Nr+r-rr.assert %(py5)sr1 STEALTH_ARGS5assert %(py5)s {%(py5)s = %(py0)s(%(py1)s, %(py3)s) }hasattrr/py1r0r1 rr2r3r4r5r6r7r9r:rFr<r=r>r?r@ @py_assert4s r test_browser_stealth_args_existsz8TestImportCompatibility.test_browser_stealth_args_exists#''|4''''|4''''''|'''|'''4'''''''%34w|^44444444w444w444444|444|444^4444444444rcZd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}d}tt|}|sd d tjvstj trtj tnd dtjvstj trtj tndtj |tj |d z}ttj|dx}}y) u9browser.py에 HARMFUL_ARGS 변수가 존재해야 한다.Nr+r-rr.rCr1 HARMFUL_ARGSrErFrGrIrJs r test_browser_harmful_args_existsz8TestImportCompatibility.test_browser_harmful_args_existsrMrcld}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|}|sdd tjvstj trtj tnd dtjvstj trtj tndtj |tj |d z}ttj|dx}}y) uEbrowser.py의 generate_stealth_headers가 호출 가능해야 한다.Nr+r-rr.rCr1zZassert %(py5)s {%(py5)s = %(py0)s(%(py3)s {%(py3)s = %(py1)s.generate_stealth_headers }) }callablerG) rr2r3r4r5r6r7r9r:generate_stealth_headersrRrJs r.test_browser_generate_stealth_headers_callablezFTestImportCompatibility.test_browser_generate_stealth_headers_callables#''|4''''|4''''''|'''|'''4'''''''$==>x=>>>>>>>>x>>>x>>>>>> >>> >>>=>>>>>>>>>>rcld}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|}|sdd tjvstj trtj tnd dtjvstj trtj tndtj |tj |d z}ttj|dx}}y) u?browser.py의 get_google_referer가 호출 가능해야 한다.Nr+r-rr.rCr1zTassert %(py5)s {%(py5)s = %(py0)s(%(py3)s {%(py3)s = %(py1)s.get_google_referer }) }rRrG) rr2r3r4r5r6r7r9r:get_google_refererrRrJs r(test_browser_get_google_referer_callablez@TestImportCompatibility.test_browser_get_google_referer_callables#''|4''''|4''''''|'''|'''4'''''''$778x788888888x888x888888 888 88878888888888rcld}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|}|sdd tjvstj trtj tnd dtjvstj trtj tndtj |tj |d z}ttj|dx}}y) uDbrowser.py의 create_resource_blocker가 호출 가능해야 한다.Nr+r-rr.rCr1zYassert %(py5)s {%(py5)s = %(py0)s(%(py3)s {%(py3)s = %(py1)s.create_resource_blocker }) }rRrG rr2r3r4r5r6r7r9r:create_resource_blockerrRrJs r-test_browser_create_resource_blocker_callablezETestImportCompatibility.test_browser_create_resource_blocker_callables#''|4''''|4''''''|'''|'''4'''''''$<<=x<========x===x====== === ===<==========rcZd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}d}tt|}|sd d tjvstj trtj tnd dtjvstj trtj tndtj |tj |d z}ttj|dx}}y) uCbrowser.py에 BLOCKED_RESOURCE_TYPES 변수가 존재해야 한다.Nr+r-rr.rCr1BLOCKED_RESOURCE_TYPESrErFrGrIrJs r*test_browser_blocked_resource_types_existszBTestImportCompatibility.test_browser_blocked_resource_types_existss#''|4''''|4''''''|'''|'''4'''''''%=>w|%=>>>>>>>>w>>>w>>>>>>|>>>|>>>%=>>>>>>>>>>rcnd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}y)u7crawl_utils.ProxyRotator가 import 가능해야 한다.Nr+r-r r.rCr1) r r2r3r4r5r6r7r9r:r;s r)test_crawl_utils_proxy_rotator_importablezATestImportCompatibility.test_crawl_utils_proxy_rotator_importables[#''|4''''|4''''''|'''|'''4'''''''rctt}|sddtjvst j trt j tnddtjvst j trt j tndt j |dz}tt j|d}y)u9crawl_utils.is_proxy_error가 import 가능해야 한다.,assert %(py3)s {%(py3)s = %(py0)s(%(py1)s) }rRrr/rHr0N) rRrr4r5r2r6r7r9r:r<r=r?s r*test_crawl_utils_is_proxy_error_importablezBTestImportCompatibility.test_crawl_utils_is_proxy_error_importablesi''''''''x'''x'''''''''''''''''''rctt}|sddtjvst j trt j tnddtjvst j trt j tndt j |dz}tt j|d}y)u;crawl_utils.fetch_with_retry가 import 가능해야 한다.rbrRr rcN) rRr r4r5r2r6r7r9r:rds r,test_crawl_utils_fetch_with_retry_importablezDTestImportCompatibility.test_crawl_utils_fetch_with_retry_importablel())))))))x)))x))))))()))())))))))))rctt}|sddtjvst j trt j tnddtjvst j trt j tndt j |dz}tt j|d}y)uCcrawl_utils.get_resource_block_types가 import 가능해야 한다.rbrRr rcN) rRr r4r5r2r6r7r9r:rds r4test_crawl_utils_get_resource_block_types_importablezLTestImportCompatibility.test_crawl_utils_get_resource_block_types_importablesl011111111x111x111111011101111111111rctt}|sddtjvst j trt j tnddtjvst j trt j tndt j |dz}tt j|d}y)u;crawl_utils.html_to_markdown가 import 가능해야 한다.rbrRr rcN) rRr r4r5r2r6r7r9r:rds r,test_crawl_utils_html_to_markdown_importablezDTestImportCompatibility.test_crawl_utils_html_to_markdown_importablerhrctt}|sddtjvst j trt j tnddtjvst j trt j tndt j |dz}tt j|d}y)u5crawl_utils.clean_html이 import 가능해야 한다.rbrRr rcN) rRr r4r5r2r6r7r9r:rds r&test_crawl_utils_clean_html_importablez>TestImportCompatibility.test_crawl_utils_clean_html_importablesi ########x###x###### ### ##########rcnd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}y)uAinsurance_crawler.InsuranceCrawler가 import 가능해야 한다.Nr+r-rr.rCr1) rr2r3r4r5r6r7r9r:r;s r!test_insurance_crawler_importablez9TestImportCompatibility.test_insurance_crawler_importables_'++t++++t++++++++++++t+++++++rcnd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}y)u?insurance_spider.InsuranceSpider가 import 가능해야 한다.Nr+r-rr.rCr1) rr2r3r4r5r6r7r9r:r;s r test_insurance_spider_importablez8TestImportCompatibility.test_insurance_spider_importable[&**d****d************d*******rcnd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}y)u?insurance_spider.ResponseHistory가 import 가능해야 한다.Nr+r-rr.rCr1) rr2r3r4r5r6r7r9r:r;s r test_response_history_importablez8TestImportCompatibility.test_response_history_importablersrctj}tj}||u}|stjd|fd||fdt j vstj|rtj|nddt j vstj|rtj|nddz}dd|iz}ttj|d}y) uMcrawl_utils와 insurance_crawler 사이에 이름 충돌이 없어야 한다.is)z%(py0)s is %(py2)scu_ProxyRotatoric_ProxyRotatorr/py2assert %(py4)spy4N) crawl_utilsr _insurance_crawler_modr2r3r4r5r6r7r9r:)r<ryrzr> @py_format3 @py_format5s r%test_no_name_conflict_between_modulesz=TestImportCompatibility.test_no_name_conflict_between_moduless&220==/1111/111111111111111/111/1111111rcttu}|stjd|fdttfdt j vstj trtjtnddt j vstj trtjtnddz}dd|iz}ttj|d}ttu}|stjd|fdttfdt j vstj trtjtndd t j vstj trtjtnd dz}dd|iz}ttj|d}ttu}|stjd|fdttfdt j vstj trtjtndd t j vstj trtjtnd dz}dd|iz}ttj|d}y) u7각 모듈은 서로 다른 모듈 객체여야 한다.r+)z%(py0)s is not %(py2)sr_insurance_spider_modr{r}r~Nr) rrr2r3r4r5r6r7r9r:r)r<r>rrs r!test_all_modules_distinct_objectsz9TestImportCompatibility.test_all_modules_distinct_objectssG%-BBBBB%-BBBBBBB%BBB%BBBBBB-BBBB-BBBBBBBB%[8888%[888888%888%888888[888[8888888$K7777$K777777$777$777777K777K7777777rrN)__name__ __module__ __qualname____doc__rArLrPrTrWr[r^r`rergrjrlrnrprrrurrrrr)r)s`JW5 5 ? 9 > ? ((*2*$,++28rr)cpeZdZdZddZddZddZddZddZddZ dd Z dd Z dd Z dd Z dd ZddZy)TestInterfaceConsistencyuV함수 인터페이스가 모듈 간에 일관성 있게 연결되는지 검증한다.Nc d}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|t}|s'dd tjvstj trtj tnd dtjvstj trtj tndtj |d tjvstj trtj tnd tj |d z}ttj|dx}}y) u4browser.py의 STEALTH_ARGS는 tuple이어야 한다.Nr+r-rr.rCr1zWassert %(py6)s {%(py6)s = %(py0)s(%(py3)s {%(py3)s = %(py1)s.STEALTH_ARGS }, %(py4)s) } isinstancetupler/rHr0r~py6) rr2r3r4r5r6r7r9r:rDrrr<r=r>r?r@ @py_assert5 @py_format7s rtest_stealth_args_is_tuplez3TestInterfaceConsistency.test_stealth_args_is_tuple#''|4''''|4''''''|'''|'''4'''''''&33;z3U;;;;;;;;z;;;z;;;;;;,;;;,;;;3;;;;;;U;;;U;;;;;;;;;;rc d}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|t}|s'dd tjvstj trtj tnd dtjvstj trtj tndtj |d tjvstj trtj tnd tj |d z}ttj|dx}}y) u4browser.py의 HARMFUL_ARGS는 tuple이어야 한다.Nr+r-rr.rCr1zWassert %(py6)s {%(py6)s = %(py0)s(%(py3)s {%(py3)s = %(py1)s.HARMFUL_ARGS }, %(py4)s) }rrr) rr2r3r4r5r6r7r9r:rOrrrs rtest_harmful_args_is_tuplez3TestInterfaceConsistency.test_harmful_args_is_tuplerrc d}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|t}|s'dd tjvstj trtj tnd dtjvstj trtj tndtj |d tjvstj trtj tnd tj |d z}ttj|dx}}y) u<browser.py의 BLOCKED_RESOURCE_TYPES는 set이어야 한다.Nr+r-rr.rCr1zaassert %(py6)s {%(py6)s = %(py0)s(%(py3)s {%(py3)s = %(py1)s.BLOCKED_RESOURCE_TYPES }, %(py4)s) }rsetr) rr2r3r4r5r6r7r9r:r]rrrs r"test_blocked_resource_types_is_setz;TestInterfaceConsistency.test_blocked_resource_types_is_sets#''|4''''|4''''''|'''|'''4'''''''&==Cz=sCCCCCCCCzCCCzCCCCCC,CCC,CCC=CCCCCCsCCCsCCCCCCCCCCrc6ddg}t|}|j}d}||u}|stjd|fd||fdt j vstj |rtj|ndtj|tj|dz}d d |iz}ttj|dx}x}}|j}t|t} | sd d t j vstj trtjtnd dt j vstj |rtj|ndtj|d t j vstj trtjtnd tj| dz} ttj| dx}} y)u`InsuranceCrawler가 proxy_list로 생성할 때 crawl_utils.ProxyRotator를 사용해야 한다.zhttp://proxy1:8080zhttp://proxy2:8080 proxy_listNr+z9%(py2)s {%(py2)s = %(py0)s.proxy_rotator } is not %(py5)scrawlerr/r|r1assert %(py7)spy7zXassert %(py6)s {%(py6)s = %(py0)s(%(py3)s {%(py3)s = %(py1)s.proxy_rotator }, %(py4)s) }rr r) r proxy_rotatorr2r3r4r5r6r7r9r:rr ) r<proxiesrr>rK @py_assert3r@ @py_format8r=rrs r:test_insurance_crawler_uses_proxy_rotator_from_crawl_utilszSTestInterfaceConsistency.test_insurance_crawler_uses_proxy_rotator_from_crawl_utilss)')=>"g6$$0D0$D0000$D000000w000w000$000D0000000!//>z/>>>>>>>>z>>>z>>>>>>'>>>'>>>/>>>>>>>>>>>>>>>>>>>rct}d}|j|}t|t}|s ddt j vst jtrt jtnddt j vst j|rt j|nddt j vst jtrt jtndt j|dz}tt j|d}d}||v}|st jd |fd ||ft j|dt j vst j|rt j|ndd z}d d |iz} tt j| dx}}d}||v}|st jd|fd||ft j|dt j vst j|rt j|ndd z}d d |iz} tt j| dx}}y)uaInsuranceCrawler.to_llm_input()은 clean_html→html_to_markdown 파이프라인을 사용한다.u3

보험 공시 정보

5assert %(py4)s {%(py4)s = %(py0)s(%(py1)s, %(py2)s) }rresultstrr/rHr|r~Nu보험 공시 정보inz%(py1)s in %(py3)srHr0rCr1evil()not inz%(py1)s not in %(py3)s) r to_llm_inputrrr4r5r2r6r7r9r:r3) r<rrrrr @py_assert0r=r?r@s r?test_insurance_crawler_clean_html_and_html_to_markdown_pipelinezXTestInterfaceConsistency.test_insurance_crawler_clean_html_and_html_to_markdown_pipelines\"$D%%d+&#&&&&&&&&z&&&z&&&&&&&&&&&&&&&&&#&&&#&&&&&&&&&&%/%////%///%////////////////%xv%%%%xv%%%x%%%%%%v%%%v%%%%%%%rcvtdg}d}t||}|sddtjvst j trt j tnddtjvst j |rt j |ndt j |t j |dz}tt j|dx}}|j}t|t}|sd d tjvst j trt j tnd dtjvst j |rt j |ndt j |d tjvst j trt j tnd t j |d z}tt j|dx}}y) uVInsuranceSpider 인스턴스에 _crawler 속성(InsuranceCrawler)이 있어야 한다.https://example.com start_urls_crawlerrErFr rGNSassert %(py6)s {%(py6)s = %(py0)s(%(py3)s {%(py3)s = %(py1)s._crawler }, %(py4)s) }rrr rrFr4r5r2r6r7r9r:rrrr<r r=rKr@rrs r+test_insurance_spider_has_crawler_attributezDTestInterfaceConsistency.test_insurance_spider_has_crawler_attribute( -B,CD)*wvz********w***w******v***v***z********** //rrs r$test_response_history_save_signaturez=TestInterfaceConsistency.test_response_history_save_signaturesb!#,c2  ( ( * %f4<*<<=H LL "> *>(( *(* ** * * * * *4 * * *4 * * * * * * * * * * * *> * * *( * * ** * * * * * *F|&99H LL "?? $?$ $$ $ $ $ $ $8 $ $ $8 $ $ $? $ $ $$ $ $ $ $ $ $ % % %s G>H88IcBd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}td}tj}||k(}|stjd |fd ||fd tjvstj |rtj |nd d tjvstj |rtj |nd d z}dd|iz}ttj|d}y)ufcrawl_utils.get_resource_block_types('default')와 browser.BLOCKED_RESOURCE_TYPES가 같아야 한다.Nr+r-rr.rCr1defaultr)z%(py0)s == %(py2)s default_types browser_typesr{r}r~) rr2r3r4r5r6r7r9r:r r]) r<r=r>r?r@rrrrs r=z0%(py3)s {%(py3)s = %(py0)s(%(py1)s) } >= %(py6)sritemsrrrNrr HTML_PRODUCTSasynciorunr'rdictrr2r3r4r5r6r7r9r: r<configr r!r%rrr=rrKrrs r6test_insurance_spider_extract_with_selector_delegationzOTestInterfaceConsistency.test_insurance_spider_extract_with_selector_delegation2s&w'  !-.$ '}:OP++4VXFG#;qz!T':;;5zQzQzQss55zQ< E8!E8cddd}tdg|}ttd}tjt ||}|Dcgc]}t |ts|}}t|}d}||k\} | stjd| fd||fd tjvstjtrtjtnd d tjvstj|rtj|nd tj|tj|d z} d d | iz} ttj | dx}x} }ycc}w)u\InsuranceSpider.parse()에서 table 모드는 _crawler.extract_table을 호출해야 한다.tablertable_selectorrrrrrrrrrrrN)rr HTML_TABLErrr'rrrr2r3r4r5r6r7r9r:rs r.test_insurance_spider_extract_table_delegationzGTestInterfaceConsistency.test_insurance_spider_extract_table_delegationBs!W= -.$ 'z7LM++4VXFG#;qz!T':;;5zQzQzQss55zQr@rs r2test_html_parse_extract_with_selector_returns_dictzJTestDataFlowIntegration.test_html_parse_extract_with_selector_returns_dictWs|"E2}}]+.. ##h7/  &$''''''''z'''z''''''&'''&''''''$'''$''''''''''6{a{a{ass66{a )*z)T********z***z***)******T***T**********ay 2N2 N2222 N222 222N2222222ay!,W,!W,,,,!W,,,!,,,W,,,,,,,rctd}|jt}|j|}t |t }|s ddt jvstjtrtjtnddt jvstj|rtj|nddt jvstjt rtjt ndtj|dz}ttj|d}t|}d }||k(}|stjd |fd ||fd t jvstjtrtjtnd dt jvstj|rtj|ndtj|tj|d z} dd| iz} ttj| dx}x}}|d} t | t}|sddt jvstjtrtjtndtj| dt jvstjtrtjtndtj|dz} ttj| dx} }|dd} d}| |k(}|sltjd |fd| |ftj| tj|dz}dd|iz} ttj| dx} x}}|dd} d}| |k(}|sltjd |fd| |ftj| tj|dz}dd|iz} ttj| dx} x}}y)uVHTML → parse() → extract_table() → 결과 dict 플로우가 동작해야 한다.Fr rrrrrNrrrrrrrrrr 보험명rrrrru 보험료r)rr#r extract_tablerrr4r5r2r6r7r9r:rr3rrs r*test_html_parse_extract_table_returns_dictzBTestDataFlowIntegration.test_html_parse_extract_table_returns_dictfsg"E2}}Z(&&t,&$''''''''z'''z''''''&'''&''''''$'''$''''''''''6{a{a{ass66{a )*z)T********z***z***)******T***T**********ay%77%7777%777%7777777777ay%00%0000%000%0000000000rct}|jt}t|t}|s ddt j vstjtrtjtnddt j vstj|rtj|nddt j vstjtrtjtndtj|dz}ttj|d}t|}d}||kD}|stjd|fd ||fd t j vstjtrtjtnd dt j vstj|rtj|ndtj|tj|d z}d d |iz} ttj| dx}x}}g} d}||v}|} |sd} | |v} | } | sXtjd|fd||ftj|dt j vstj|rtj|nddz}dd|iz}| j||stjd fd |ftj| dt j vstj|rtj|nddz}dd|iz}| j|tj| diz}dd|iz}ttj|dx} x} x}x}x} } y)uPHTML → to_llm_input() → markdown 문자열 플로우가 동작해야 한다.rrrrrNr)>)z/%(py3)s {%(py3)s = %(py0)s(%(py1)s) } > %(py6)srrrrru보험r)z%(py3)s in %(py5)s)r0r1z%(py7)sr)z%(py10)s in %(py12)s)py10py12z%(py14)spy14rzassert %(py17)spy17)rrrrrr4r5r2r6r7r9r:rr3r$_format_boolop)r<rrrrr=rrKrrr>r @py_assert9 @py_assert11r@r @py_format13 @py_format15 @py_format16 @py_format18s r.test_html_to_llm_input_returns_markdown_stringzFTestDataFlowIntegration.test_html_to_llm_input_returns_markdown_stringqs"$%%m4&#&&&&&&&&z&&&z&&&&&&&&&&&&&&&&&#&&&#&&&&&&&&&&6{Q{Q{Qss66{Q=~=~'=8=8v+=====~===~================8v===8======v===v==============rct}|jdddg|jdd|j}t|}d}||k(}|st j d|fd ||fd t jvst jtrt jtnd d t jvst j|rt j|nd t j|t j|d z}d d|iz}tt j|dx}x}}|jd}t|}d}||k\}|st j d|fd||fd t jvst jtrt jtnd dt jvst j|rt j|ndt j|t j|d z}d d|iz}tt j|dx}x}}|dd} d} | | k(}|slt j d|fd| | ft j| t j| dz} dd| iz}tt j|dx} x}} |jd} t| }d}||k\}|st j d|fd||fd t jvst jtrt jtnd dt jvst j| rt j| ndt j|t j|d z}d d|iz}tt j|dx}x}}tj5} t| dz }|j!||j"}|} | sddt jvst j|rt j|ndt j|t j| dz} tt j| dx}} t%j&|j)d }t+|t,} | s d!d"t jvst jt*rt jt*nd"d#t jvst j|rt j|nd#d$t jvst jt,rt jt,nd$t j| d%z} tt j| d} t|}d}||k(}|st j d|fd ||fd t jvst jtrt jtnd d#t jvst j|rt j|nd#t j|t j|d z}d d|iz}tt j|dx}x}}dddy#1swYyxYw)&uPrecord → get_history → get_chain → save(tmpfile) → 파일 존재 확인.zhttps://example.com/finalrzhttps://example.com/old)rzhttps://example.com/otheri-r rrrall_histrrrNrrr chain_finalrrrrrr chain_oldz history.jsonrpathrutf-8encodingrrdatarr)rrrrr2r3r4r5r6r7r9r: get_chainrrrrrjsonloads read_textrr)r<rr3r=rrKrrr4rrrr5rr6r>r:s r7test_response_history_record_get_history_get_chain_savezOTestDataFlowIntegration.test_response_history_record_get_history_get_chain_savezs*!# ' 01  2C8&&(8}!!}!!!!}!!!!!!s!!!s!!!!!!8!!!8!!!}!!!!!!!!!!''(CD ;$1$1$$$$1$$$$$$s$$$s$$$$$$;$$$;$$$$$$1$$$$$$$1~e$C(CC$(CCCCC$(CCCC$CCC(CCCCCCCC%%&?@ 9~""~""""~""""""s"""s""""""9"""9"""~""""""""""  ( ( * "f<.0D LL ;; ;= = 4 4 ; = ::dnngn>?DdD) )) ) ) ) ) ): ) ) ): ) ) ) ) ) )d ) ) )d ) ) ) ) ) )D ) ) )D ) ) )) ) ) ) ) ) )t9 ! !9> ! ! !9 ! ! ! ! ! !3 ! ! !3 ! ! ! ! ! !t ! ! !t ! ! !9 ! ! ! ! ! ! ! ! ! !  " " "s +L]]c d}t|}d}||v}|stjd|fd||ftj|dt j vstj |rtj|nddz}dd|iz}ttj|d x}}t|d }t|t}|s d d t j vstj trtjtnd dt j vstj |rtj|nddt j vstj trtjtndtj|dz} ttj| d }d}||v}|stjd|fd||ftj|dt j vstj |rtj|nddz}dd|iz}ttj|d x}}y )uRclean_html() → html_to_markdown() 파이프라인이 정상 동작해야 한다.u?

보험 정보

rrrcleanedrrCr1NF) remove_noiserrmarkdownrru 보험 정보rr) r r2r3r7r4r5r6r9r:r rr) r<raw_htmlrArr=r?r@rCrrs r.test_clean_html_then_html_to_markdown_pipelinezFTestDataFlowIntegration.test_clean_html_then_html_to_markdown_pipelinesWTX&&xw&&&&xw&&&x&&&&&&w&&&w&&&&&&&#G%@(C((((((((z(((z(((((((((((((((((C(((C((((((((((*(****(*********(***(*******rcddg}t|}|j}d}||u}|stjd|fd||fdt j vstj |rtj|ndtj|tj|dz}d d |iz}ttj|dx}x}}|jj}|jj} d} || k(}|stjd |fd || fd t j vstj |rtj|nd tj| dz} dd| iz}ttj|dx}} d} | | k(}|stjd |fd | | fdt j vstj | rtj| ndtj| dz} dd| iz}ttj|dx}} y)u[InsuranceCrawler에 proxy_list 전달 시 ProxyRotator가 올바르게 동작해야 한다.zhttp://p1:8080zhttp://p2:8080rNr+rrrrrrz%(py0)s == %(py3)sfirstr.rCr1second) rrr2r3r4r5r6r7r9r:get_next) r<rrr>rKrr@rrHrIr=r?s r*test_proxy_rotator_round_trip_with_crawlerzBTestDataFlowIntegration.test_proxy_rotator_round_trip_with_crawlersi#%56"g6$$0D0$D0000$D000000w000w000$000D0000000%%..0&&//1((u(((((u(((((((u(((u((((((((((())v)))))v)))))))v)))v)))))))))))rc`dddddd}tdg|}ttd }tjt ||}|Dcgc]}t |ts|}}t|}d }||k\} | stjd | fd ||fdtjvstjtrtjtnddtjvstj|rtj|ndtj|tj|dz} dd| iz} ttj | dx}x} }d|D} t#| } | sddtjvstjt"rtjt"ndtj| tj| dz}ttj |dx} } |dd}d } || k(}|sltjd|fd|| ftj|tj| dz}dd|iz} ttj | dx}x}} ycc}w)uHSpider parse CSS 모드 전체 플로우: HTML → parse → dict items.rrrrrrrrzhttps://example.com/productsrrrrrrrrrNc3$K|]}d|v yw) _source_urlNr).0r&s r zOTestDataFlowIntegration.test_spider_parse_css_mode_full_flow..s;T=D(;s,assert %(py4)s {%(py4)s = %(py0)s(%(py2)s) }allrrrNrrrrr)rrrrrr'rrrr2r3r4r5r6r7r9r:rR)r<rr r!r%rrr=rrKrrr>rrrs r$test_spider_parse_css_mode_full_flowz":+*Irr cPeZdZdZd dZd dZd dZd dZd dZd dZ d d Z d d Z y) TestSpiderCrawlerIntegrationu>InsuranceSpider와 InsuranceCrawler의 연동을 검증한다.Ncvtdg}d}t||}|sddtjvst j trt j tnddtjvst j |rt j |ndt j |t j |dz}tt j|dx}}|j}t|t}|sd d tjvst j trt j tnd dtjvst j |rt j |ndt j |d tjvst j trt j tnd t j |d z}tt j|dx}}y) uPInsuranceSpider 초기화 시 InsuranceCrawler가 내부 생성되어야 한다.rrrrErFr rGNrrrrrrs r5test_spider_init_creates_insurance_crawler_internallyzRTestSpiderCrawlerIntegration.test_spider_init_creates_insurance_crawler_internallyrrctdg}|j}|j}d}||u}|stjd|fd||fdt j vstj|rtj|ndtj|tj|tj|dz}dd |iz}ttj|d x}x}x}}y ) uRInsuranceSpider 내부의 _crawler는 adaptive=False로 초기화되어야 한다.rrFrw)zN%(py4)s {%(py4)s = %(py2)s {%(py2)s = %(py0)s._crawler }.adaptive } is %(py7)sr r/r|r~rassert %(py9)spy9N) rrrr2r3r4r5r6r7r9r:)r<r r>rrrr @py_format10s r"test_spider_crawler_adaptive_falsez?TestSpiderCrawlerIntegration.test_spider_crawler_adaptive_falses -B,CD0''050'50000'5000000v000v000000'00050000000rcddddid}tdg|}|j}d}||u}|stjd |fd ||fd t j vstj |rtj|nd tj|tj|d z}d d|iz}ttj|dx}x}}|jd}d}||k(} | sltjd| fd||ftj|tj|dz} dd| iz} ttj| dx}x} }tt} tjt|| } | Dcgc]}t|t s|}}t#|} d}| |k\}|stjd|fd| |fdt j vstj t"rtjt"nddt j vstj |rtj|ndtj| tj|dz} dd| iz}ttj|dx} x}}d}|d}||v} | sltjd| fd||ftj|tj|dz} dd| iz} ttj| dx}x} }ycc}w) udextraction_config mode=css 전달 시 parse()에서 extract_with_selector가 선택되어야 한다.rrrrrrrNr+)z=%(py2)s {%(py2)s = %(py0)s.extraction_config } is not %(py5)sr rrrrrrrrrrrrrrrrrrr)z%(py1)s in %(py4)s)rrr2r3r4r5r6r7r9r:rrrrr'rrr)r<rr r>rKrr@rrr=rrr!r%rrrrs rDtest_spider_extraction_config_css_mode_selects_extract_with_selectorzaTestSpiderCrawlerIntegration.test_spider_extraction_config_css_mode_selects_extract_with_selectors&w'  !-.$ ''3t3't3333't333333v333v333'333t3333333''/858/58888/5888/88858888888&}5++4VXFG#;qz!T':;;5zQzQzQss55zQ!q!v!!!!v!!!v!!!!!!!!!!r @py_assert10 @py_assert13 @py_assert12r. @py_format17r0 @py_format20s r>test_spider_extraction_config_table_mode_selects_extract_tablez[TestSpiderCrawlerIntegration.test_spider_extraction_config_table_mode_selects_extract_tables!W= -.$ 'z2++4VXFG#;qz!T':;;5zQzQzQss55zQ={=eAh={h&='=U1X='X*===={h==={===h======='X==='===X==============. sP11U8>>PsrQanyr)rrrrrr'rrrr2r3r4r5r6r7r9r:rs) r<rr r!rr=rrKrrr>rrs r-test_spider_response_history_records_on_parsezJTestSpiderCrawlerIntegration.test_spider_response_history_records_on_parsesIPWGXY -.$ '}:YZ *68<=))5577| q |q    |q      s   s      7   7   |   q       PPPsPPPPPPPPPsPPPsPPPPPPPPPPPPPPrc6tj5}t|dz }tdgt |}t j |jd|j}|}|sddtjvstj|rtj|ndtj|tj|dz}ttj|d x}}d d d y #1swYy xYw) u;on_start() 실행 후 output_dir이 생성되어야 한다. spider_outrr output_dirFresumingrrxrN)rrrrrrron_startrr4r5r2r6r7r9r:)r<rrxr r>rrs r'test_spider_on_start_creates_output_dirzDTestSpiderCrawlerIntegration.test_spider_on_start_creates_output_dir s  ( ( * 'ff 4J$12z?F KK7 8$$ &$& && & & & & &: & & &: & & &$ & & && & & & & & & ' ' 's C1DDctj5}tdg|}|jj ddt j |jdt j |jt|dz }|j}|}|sddtjvstj|rtj|ndtj|tj|d z}t!tj"|d x}}d d d y #1swYy xYw) u[on_close() 실행 후 response_history.json 파일이 output_dir에 저장되어야 한다.rrwrFryzresponse_history.jsonr history_filerN)rrrrrrrr{on_closerrr4r5r2r6r7r9r:)r<rr r~r>rrs r'test_spider_on_close_saves_history_filezDTestSpiderCrawlerIntegration.test_spider_on_close_saves_history_files  ( ( * )f$12!F  # # * *+@# F KK7 8 KK) *<*AAL&& (&( (( ( ( ( ( (< ( ( (< ( ( (& ( ( (( ( ( ( ( ( ( ) ) )s D'EEr) rrrrrWr]r_rjrnrtr|rrrrrUrUs.H= 1 "& > ) Q ' )rrUceZdZdZddZddZddZddZddZddZ dd Z dd Z dd Z dd Z dd ZddZddZddZddZddZy) TestEdgeCasesu^경계 조건 및 예외 상황에서 각 모듈이 안전하게 동작하는지 검증한다.Nctd}|jt}|j|dddi}g}||k(}|st j d|fd||fd t jvst j|rt j|nd t j|d z}d d |iz}tt j|d x}}y )uO빈 HTML에서 extract_with_selector()는 빈 리스트를 반환해야 한다.Fr rrrrrrGrr.rCr1N) rr# HTML_EMPTYrr2r3r4r5r6r7r9r:r<rrrr=r>r?r@s r8test_empty_html_extract_with_selector_returns_empty_listzFTestEdgeCases.test_empty_html_extract_with_selector_returns_empty_list.s"E2}}Z(.. #G$/  v|vvvrctd}|jt}|j|}g}||k(}|st j d|fd||fdt jvst j|rt j|ndt j|dz}dd|iz}tt j|d x}}y ) uG빈 HTML에서 extract_table()은 빈 리스트를 반환해야 한다.Fr rrGrr.rCr1N) rr#rr"r2r3r4r5r6r7r9r:rs r0test_empty_html_extract_table_returns_empty_listz>TestEdgeCases.test_empty_html_extract_table_returns_empty_list9s"E2}}Z(&&t,v|vvvrcLt}|jt}t|t}|s ddt j vstjtrtjtnddt j vstj|rtj|nddt j vstjtrtjtndtj|dz}ttj|d}|j}|}d}||k(}|stjd|fd ||fdt j vstj|rtj|ndtj|tj|tj|d z}d d |iz} ttj| dx}x}x}}y) uF빈 HTML에서 to_llm_input()은 빈 문자열을 반환해야 한다.rrrrrNrr)zD%(py4)s {%(py4)s = %(py2)s {%(py2)s = %(py0)s.strip }() } == %(py7)srYrZr[)rrrrrr4r5r2r6r7r9r:stripr3) r<rrrrr>rrrr\s r1test_empty_html_to_llm_input_returns_empty_stringz?TestEdgeCases.test_empty_html_to_llm_input_returns_empty_string@s!"$%%j1&#&&&&&&&&z&&&z&&&&&&&&&&&&&&&&&#&&&#&&&&&&&&&&||#|~##~####~######v###v###|###~##########rctd}|j}d}||u}|stjd|fd||fdt j vstj |rtj|ndtj|tj|dz}dd|iz}ttj|dx}x}}y) uMproxy_list=None이면 InsuranceCrawler.proxy_rotator는 None이어야 한다.Nrrw)z5%(py2)s {%(py2)s = %(py0)s.proxy_rotator } is %(py5)srrrr) rrr2r3r4r5r6r7r9r:r<rr>rKrr@rs r2test_none_proxy_list_crawler_proxy_rotator_is_nonez@TestEdgeCases.test_none_proxy_list_crawler_proxy_rotator_is_noneGs~"d3$$,,$,,,,$,,,,,,w,,,w,,,$,,,,,,,,,,rctd}|j}d}||u}|stjd|fd||fdt j vstj |rtj|ndtj|tj|dz}dd|iz}ttj|d x}x}}y ) uEadaptive=False이면 InsuranceCrawler.adaptive가 False여야 한다.Fr rw)z0%(py2)s {%(py2)s = %(py0)s.adaptive } is %(py5)srrrrN) rrr2r3r4r5r6r7r9r:rs r+test_adaptive_false_disables_smart_matchingz9TestEdgeCases.test_adaptive_false_disables_smart_matchingLs~"E2(5(5((((5((((((w(((w((((((5(((((((rcddddid}tdg|}ttd}tjt ||}|Dcgc]}t |ts|}}t|}d }||k(} | stjd | fd ||fd tjvstjtrtjtnd d tjvstj|rtj|nd tj|tj|dz} dd| iz} ttj | dx}x} }ycc}w)uK빈 HTML에서 Spider.parse()는 아이템을 yield하지 않아야 한다.rrrrrrrrrrrrrrrrN)rrrrrr'rrrr2r3r4r5r6r7r9r:rs r-test_empty_html_spider_parse_returns_no_itemsz;TestEdgeCases.test_empty_html_spider_parse_returns_no_itemsQsPWGXY -.$ 'z7LM++4VXFG#;qz!T':;;5zQzQzQss55zQTestEdgeCases.test_nonexistent_css_selector_returns_empty_list]s"E2}}]+.. *G$/  v|vvvrctd}|jt}|j|}g}||k(}|st j d|fd||fdt jvst j|rt j|ndt j|dz}dd|iz}tt j|d x}}y ) uW테이블이 없는 HTML에서 extract_table()은 빈 리스트를 반환해야 한다.Fr rrGrr.rCr1N) rr# HTML_SIMPLEr"r2r3r4r5r6r7r9r:rs r test_no_table_returns_empty_listz.TestEdgeCases.test_no_table_returns_empty_lisths"E2}}[)&&t,v|vvvrct}|jd}g}||k(}|stjd|fd||fdt j vstj |rtj|ndtj|dz}dd|iz}ttj|dx}}y) uN빈 ResponseHistory에서 get_chain()은 빈 리스트를 반환해야 한다.rrrGrr.rCr1N) rr;r2r3r4r5r6r7r9r:)r<rrr=r>r?r@s r3test_response_history_empty_get_chain_returns_emptyzATestEdgeCases.test_response_history_empty_get_chain_returns_emptyoss!#""#89v|vvvrct}|jdd|jtj5}t |dz }|j ||j}|}|sddtjvstj|rtj|ndtj|tj|dz}ttj|dx}}tj |j#d }g}||k(}|stj$d |fd ||fd tjvstj|rtj|nd tj|d z} dd| iz} ttj| dx}}dddy#1swYyxYw)uNclear() 후 save()하면 빈 배열([]) JSON 파일이 생성되어야 한다.rrz empty.jsonrr6rNr7r8rrGr:r.rCr1)rrclearrrrrrr4r5r2r6r7r9r:r<r=r>r3) r<rrr6r>rrr:r=r?r@s r0test_response_history_clear_then_save_empty_filez>TestEdgeCases.test_response_history_clear_then_save_empty_fileus'!#,c2   ( ( * f<,.D LL ;; ;= = 4 4 ; = ::dnngn>?D 42:   42      4   4   2           s FGG$cd}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|t}|s dd tjvstj trtj tnd d tjvstj |rtj |nd d tjvstj trtj tnd tj |d z}ttj|d}y) uBbrowser.generate_stealth_headers()는 dict를 반환해야 한다.Nr+r-rr.rCr1rrrrr) rr2r3r4r5r6r7r9r:rSrr)r<r=r>r?r@rrrs r*test_generate_stealth_headers_returns_dictz8TestEdgeCases.test_generate_stealth_headers_returns_dicts#''|4''''|4''''''|'''|'''4'''''''668&$''''''''z'''z''''''&'''&''''''$'''$''''''''''rc d}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|t}|s dd tjvstj trtj tnd d tjvstj |rtj |nd d tjvstj trtj tnd tj |d z}ttj|d}d }||v}|stjd|fd||ftj |d tjvstj |rtj |nd dz}dd|iz}ttj|dx}}y)uLbrowser.get_google_referer()는 Google URL 문자열을 반환해야 한다.Nr+r-rr.rCr1rrrrrz google.comrrr) rr2r3r4r5r6r7r9r:rVrr) r<r=r>r?r@rrrrs r*test_get_google_referer_returns_google_urlz8TestEdgeCases.test_get_google_referer_returns_google_urlsJ#''|4''''|4''''''|'''|'''4'''''''002&#&&&&&&&&z&&&z&&&&&&&&&&&&&&&&&#&&&#&&&&&&&&&&%|v%%%%|v%%%|%%%%%%v%%%v%%%%%%%rc8d}t|u}|stjd|fdt|fdtjvstj trtj tndtj |dz}dd|iz}ttj|dx}}tj}t|}|sdd tjvstj trtj tnd d tjvstj |rtj |nd tj |d z}ttj|d}y) uTbrowser.create_resource_blocker()는 호출 가능한 함수를 반환해야 한다.Nr+r-rr.rCr1rbrRhandlerrcrY)r<r=r>r?r@rs r-test_create_resource_blocker_returns_callablez;TestEdgeCases.test_create_resource_blocker_returns_callables#''|4''''|4''''''|'''|'''4'''''''668        x   x                   rctd}|jt}|j|d}g}||k(}|st j d|fd||fdt jvst j|rt j|ndt j|dz}d d |iz}tt j|d x}}y ) ul존재하지 않는 reference_selector로 extract_similar() 호출 시 빈 리스트를 반환해야 한다.Fr z.nonexistent-ref)rmrrGrr.rCr1N) rr#rextract_similarr2r3r4r5r6r7r9r:rs r8test_extract_similar_nonexistent_reference_returns_emptyzFTestEdgeCases.test_extract_similar_nonexistent_reference_returns_emptys"E2}}]+((BT(Uv|vvvrctg}|j}|}d}||u}|stjd|fd||fdt j vstj |rtj|ndtj|tj|tj|dz}dd|iz}ttj|dx}x}x}}y)uT빈 proxy_list로 ProxyRotator 생성 시 get_next()는 None을 반환해야 한다.Nrw)zG%(py4)s {%(py4)s = %(py2)s {%(py2)s = %(py0)s.get_next }() } is %(py7)srotatorrYrZr[) r rJr2r3r4r5r6r7r9r:)r<rr>rrrrr\s r*test_proxy_rotator_empty_list_returns_nonez8TestEdgeCases.test_proxy_rotator_empty_list_returns_nonesr")!)T)!T))))!T))))))w)))w))))))!)))T)))))))rctdgd}tt}tjt ||}|Dcgc]}t |ts|}}t|}d}||k(}|stjd|fd||fdtjvstjtrtjtnddtjvstj|rtj|ndtj|tj|d z} d d | iz} ttj | dx}x}}ycc}w) uRextraction_config=None이면 parse()는 아이템을 yield하지 않아야 한다.rNrrrrrrrrrr) r<r r!r%rrr=rrKrrs r1test_spider_extraction_config_none_yields_nothingz?TestEdgeCases.test_spider_extraction_config_none_yields_nothings -." '}5++4VXFG#;qz!T':;;5zQzQzQss55zQrs   (^ # # % , , 3 3 : : 3zI-./ # hsm" NN 2 29c*yBX[gBg>h iE U\\5 ~~66u=    .20/=  (  ?  %        S T8T8xkkfgIgI^`)`)P~~Q  sA,F FF